HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: avoiding BL's
Author: William Roeder
Date: 12/03/2008 15:10
> Also; Is there a way to bypass the /robots.txt check
options -> spider
> using HTTRACK? 
HTTrack Users Guide By Fred Cohen - <>
-sN follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always)

Change the browser ID to a normal browser not google. I use Mozilla/4.0
(compatible; MSIE 5.5; Windows NT 5.0)
Reply Create subthread

All articles

Subject Author Date
avoiding BL's

12/02/2008 09:41
Re: avoiding BL's

12/02/2008 10:09
Re: avoiding BL's

12/02/2008 10:15
Re: avoiding BL's

12/03/2008 15:10


Created with FORUM 2.0.11