HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: what to do with "robots.txt"
Author: D_A
Date: 06/15/2006 22:11
 
I had no problem once I set the URL to 
<http://www.moreart4u.de/internet/index.php>

<http://www.moreart4u.de/index.html> has a nasty javascript redirection towards
the online copy of the site.
As for robots.txt, you don't need one ; you can set the Spider option to no
robots.txt rules, but it won't change anything.
 
Reply Create subthread


All articles

Subject Author Date
what to do with "robots.txt"

06/15/2006 10:57
Re: what to do with "robots.txt"

06/15/2006 22:11




b

Created with FORUM 2.0.11