| Subject: Re: How to completely ignore 'robots.txt'? |
Author: Xavier Roche |
Date: 11/30/2001 17:43 |
| | > Ok, I'm trying to mirror a site that tells engines
> like httrack to not go down to certain directories.
> Which version of httrack allows me to complete
ignore
> these files and go down into this certain directory?
See options/spider/spider: robots.txt -> 'never'
But also ensure that you set proper bandwidth limiter
if you are crawling big files or a large number of
generated pages (robots.txt are often used to avoid
server overload)
| |
|
|
|
|
b
Created with FORUM 2.0.11