| Subject: Re: a precision about robots.txt when mirroring locall |
Author: Xavier Roche |
Date: 08/07/2003 20:49 |
| | > a precision about robots.txt when mirroring locally
> (http://127.0.0.1/site/index.php) :
> Winhttrack looks for <http://127.0.0.1/robots.txt> instead
of
> <http://127.0.0.1/site/robots.txt>
Err I am actually missing something, or it is the normal
behaviour? Robots should look for the /robots.txt fiename
(root file) even for local hosts ?
| |
|
|
|
|
3
Created with FORUM 2.0.11