| Subject: Re: robots.txt wanted |
Author: Xavier Roche |
Date: 02/04/2002 11:38 |
| | > How do I get the mirror to have the robots.txt. I
> include it in the rules:
> +*.png +*.gif +*.jpg +*.css +*.js +*/Dist/mb/*
> +*/Eaglpph.htm
> +*/robots.txt
> Nevertheless it doesn't show up in the mirror and I
> know it is there because I can see it via FTP.
robots.txt file is generally NOT linked within the
mirror, therefore you will never have the change to
get it. BESIDES, the engine is filtering /robots.txt
file, for internal purpose (this file is generally
downloaded and stored internally to check remote
directory restrictions).
| |
|
|
|
|
a
Created with FORUM 2.0.11