HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: robots.txt
Author: William Roeder
Date: 05/07/2012 15:24
> I changed to [Mozilla/4.0 (compatible; MSIE 6.0;
> Windows NT 5.0)] under Browser ID but it still
> fails. Tried a few other options too to no avail.

> Any idea how to fix this?
I ran with log=debug and found it immediately. It was looking at (org not com)
Look at the .com source and you'll see .com is nothing but a shell for .org: <
frame src=

Since HTT stays on site you get nothing.

Just change the url to .org
Reply Create subthread

All articles

Subject Author Date

05/06/2012 15:25
Re: robots.txt

05/06/2012 21:28
Re: robots.txt

05/06/2012 23:06
Re: robots.txt

05/07/2012 15:24
Re: robots.txt

05/07/2012 22:19


Created with FORUM 2.0.11