HTTrack Website Copier
Free software offline browser - FORUM
Subject: Robots problem
Author: Lebazard
Date: 04/19/2002 04:38
with some site i have a serious problem.. it only 
starts the donload and then stop (after just a couple 
of seconds).. Explaining in the log file that "File 
has moved from my_targeturl/robots.txt to 
<http://another_url/index.html> and that" 
mytarget_url/index.htm not scanned (follow robots 
meta tag)"  ..  as i already red it in the help file 
i should certainly ask httrack not to follow 
robots.txt rules for this site.. but where can i set 
this option exactly ? i simply can't find it. Thanks 
for your help.

All articles

Subject Author Date
Robots problem

04/19/2002 04:38
Re: Robots problem

04/19/2002 04:47


Created with FORUM 2.0.11