HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: a precision about robots.txt when mirroring lo
Author: Dan
Date: 08/10/2003 22:51
 
Thanks for your help

When robots.txt is in the HTML root folder, Winhttrack reads
it, mentions it in the log file but doesn't follow the rules.
(DocumentRoot "d:/site" with in "d:/site/robots.txt" // URL
<http://127.0.0.1/site/index.php>)

My other problem is I have a few local sites with different
robots.txt but only one HTML root.

I couldn't find any solution with the alias section, I must
be misunderstanding something.
 
Reply Create subthread


All articles

Subject Author Date
a precision about robots.txt when mirroring locall

08/05/2003 11:50
Re: a precision about robots.txt when mirroring locall

08/07/2003 20:49
Re: a precision about robots.txt when mirroring lo

08/08/2003 10:16
Re: a precision about robots.txt when mirroring lo

08/10/2003 11:51
Re: a precision about robots.txt when mirroring lo

08/10/2003 22:51
Re: a precision about robots.txt when mirroring lo

08/15/2003 09:10
Re: a precision about robots.txt when mirroring lo

08/15/2003 16:07




f

Created with FORUM 2.0.11