| Subject: Re: a precision about robots.txt when mirroring lo |
Author: Xavier Roche |
Date: 08/15/2003 09:10 |
| | > When robots.txt is in the HTML root folder, Winhttrack
reads
> it, mentions it in the log file but doesn't follow the
rules.
Err, what are these rules ? If limited to the robots-rfc,
rules should work..
| |
|
|
|
|
3
Created with FORUM 2.0.11
