HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt wanted
Author: Glen
Date: 02/03/2002 22:25
 
How do I get the mirror to have the robots.txt.  I 
include it in the rules:
+*.png +*.gif +*.jpg +*.css +*.js +*/Dist/mb/*
+*/Eaglpph.htm
+*/robots.txt

Nevertheless it doesn't show up in the mirror and I 
know it is there because I can see it via FTP.
 
Reply


All articles

Subject Author Date
robots.txt wanted

02/03/2002 22:25
Re: robots.txt wanted

02/04/2002 11:38




b

Created with FORUM 2.0.11