Subject: Re: a precision about robots.txt when mirroring lo
Author: Xavier Roche
Date: 08/15/2003 09:10
> When robots.txt is in the HTML root folder, Winhttrack
reads
> it, mentions it in the log file but doesn't follow the
rules.
Err, what are these rules ? If limited to the robots-rfc,
rules should work..
