HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: follow robots.txt rules does not work
Author: D_A
Date: 03/10/2005 10:03
 
By default, HTTrack obeys robots.txt rules.
You have to explicitly ask if you don't want it (unless you
saved the option as default).
Is the problem with a local site or with a URL?Following robots.txt will not
avoid falling in a spider trap.
Can you give us more precisions?
 
Reply Create subthread


All articles

Subject Author Date
follow robots.txt rules does not work

03/09/2005 23:48
Re: follow robots.txt rules does not work

03/10/2005 10:03
Re: follow robots.txt rules does not work

03/13/2005 10:23
Re: follow robots.txt rules does not work

03/14/2005 07:13




7

Created with FORUM 2.0.11