| Subject: Re: follow robots.txt rules does not work |
Author: D_A |
Date: 03/10/2005 10:03 |
| | By default, HTTrack obeys robots.txt rules.
You have to explicitly ask if you don't want it (unless you
saved the option as default).
Is the problem with a local site or with a URL?Following robots.txt will not
avoid falling in a spider trap.
Can you give us more precisions? | |
|
|
|
|
7
Created with FORUM 2.0.11
