with some site i have a serious problem.. it only
starts the donload and then stop (after just a couple
of seconds).. Explaining in the log file that "File
has moved from my_targeturl/robots.txt to
<http://another_url/index.html> and that"
mytarget_url/index.htm not scanned (follow robots
meta tag)" .. as i already red it in the help file
i should certainly ask httrack not to follow
robots.txt rules for this site.. but where can i set
this option exactly ? i simply can't find it. Thanks
for your help. |