HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: robots.txt wanted
Author: Xavier Roche
Date: 02/04/2002 11:38
> How do I get the mirror to have the robots.txt.  I 
> include it in the rules:
> +*.png +*.gif +*.jpg +*.css +*.js +*/Dist/mb/*
> +*/Eaglpph.htm
> +*/robots.txt
> Nevertheless it doesn't show up in the mirror and I 
> know it is there because I can see it via FTP.

robots.txt file is generally NOT linked within the 
mirror, therefore you will never have the change to 
get it. BESIDES, the engine is filtering /robots.txt 
file, for internal purpose (this file is generally 
downloaded and stored internally to check remote 
directory restrictions).

Reply Create subthread

All articles

Subject Author Date
robots.txt wanted

02/03/2002 22:25
Re: robots.txt wanted

02/04/2002 11:38


Created with FORUM 2.0.11