| > Ok, I'm trying to mirror a site that tells engines
> like httrack to not go down to certain directories.
> Which version of httrack allows me to complete
ignore
> these files and go down into this certain directory?
See options/spider/spider: robots.txt -> 'never'
But also ensure that you set proper bandwidth limiter
if you are crawling big files or a large number of
generated pages (robots.txt are often used to avoid
server overload)
| |