| Subject: How to completely ignore 'robots.txt'? |
Author: Chetwood |
Date: 11/30/2001 11:41 |
| | Ok, I'm trying to mirror a site that tells engines
like httrack to not go down to certain directories.
Which version of httrack allows me to complete ignore
these files and go down into this certain directory?
TIA! | |
|
|
|
|
a
Created with FORUM 2.0.11
