HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Disabling robots.txt file
Author: D_A
Date: 04/18/2004 10:29
 
set options/Spider/Spider select there,
but don't forget to limit bandwith usage,
number of connections, max tranfer rate
and try to limit the downloads to what you need by setting
scan rules
in order to be "gentle" with the site server.
robots.txt are there for good reasons
 
Reply Create subthread


All articles

Subject Author Date
Disabling robots.txt file

04/18/2004 09:10
Re: Disabling robots.txt file

04/18/2004 10:29
Re: Disabling robots.txt file

04/19/2004 00:41
Re: Disabling robots.txt file

04/19/2004 09:16
Re: Disabling robots.txt file

04/26/2004 00:08
Re: Disabling robots.txt file

07/10/2009 13:22
Re: Disabling robots.txt file

04/26/2013 06:06




f

Created with FORUM 2.0.11