HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Disabling robots.txt file
Author: Rebekah
Date: 04/19/2004 00:41
 
> set options/Spider/Spider select there,

Thx. I recognise it now, I was looking for cgi-bin, images 
and looked right past it.

> but don't forget to limit bandwith usage, number of 
connections, max tranfer rate 

Is there recommended settings?
> and try to limit the downloads to what you need by 
setting scan rules in order to be "gentle" with the site 
server. robots.txt are there for good reasons

Still trying to take in the entire help folder. Can you 
make suggestions for setting scan rules pls?
 
Reply Create subthread


All articles

Subject Author Date
Disabling robots.txt file

04/18/2004 09:10
Re: Disabling robots.txt file

04/18/2004 10:29
Re: Disabling robots.txt file

04/19/2004 00:41
Re: Disabling robots.txt file

04/19/2004 09:16
Re: Disabling robots.txt file

04/26/2004 00:08
Re: Disabling robots.txt file

07/10/2009 13:22
Re: Disabling robots.txt file

04/26/2013 06:06




b

Created with FORUM 2.0.11