HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt wanted
Author: Glen
Date: 02/03/2002 22:25
How do I get the mirror to have the robots.txt.  I 
include it in the rules:
+*.png +*.gif +*.jpg +*.css +*.js +*/Dist/mb/*

Nevertheless it doesn't show up in the mirror and I 
know it is there because I can see it via FTP.

All articles

Subject Author Date
robots.txt wanted

02/03/2002 22:25
Re: robots.txt wanted

02/04/2002 11:38


Created with FORUM 2.0.11