HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: what to do with "robots.txt"
Author: D_A
Date: 06/15/2006 22:11
I had no problem once I set the URL to 

<> has a nasty javascript redirection towards
the online copy of the site.
As for robots.txt, you don't need one ; you can set the Spider option to no
robots.txt rules, but it won't change anything.
Reply Create subthread

All articles

Subject Author Date
what to do with "robots.txt"

06/15/2006 10:57
Re: what to do with "robots.txt"

06/15/2006 22:11


Created with FORUM 2.0.11