HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: a precision about robots.txt when mirroring locall
Author: Xavier Roche
Date: 08/07/2003 20:49
 
> a precision about robots.txt when mirroring locally 
> (http://127.0.0.1/site/index.php) :
> Winhttrack looks for <http://127.0.0.1/robots.txt> instead 
of 
> <http://127.0.0.1/site/robots.txt>

Err I am actually missing something, or it is the normal 
behaviour? Robots should look for the /robots.txt fiename 
(root file) even for local hosts ?
 
Reply Create subthread


All articles

Subject Author Date
a precision about robots.txt when mirroring locall

08/05/2003 11:50
Re: a precision about robots.txt when mirroring locall

08/07/2003 20:49
Re: a precision about robots.txt when mirroring lo

08/08/2003 10:16
Re: a precision about robots.txt when mirroring lo

08/10/2003 11:51
Re: a precision about robots.txt when mirroring lo

08/10/2003 22:51
Re: a precision about robots.txt when mirroring lo

08/15/2003 09:10
Re: a precision about robots.txt when mirroring lo

08/15/2003 16:07




3

Created with FORUM 2.0.11