HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: robots.txt
Author: William Roeder
Date: 05/07/2012 15:24
 
> I changed to [Mozilla/4.0 (compatible; MSIE 6.0;
> Windows NT 5.0)] under Browser ID but it still
> fails. Tried a few other options too to no avail.

> Any idea how to fix this?
I ran with log=debug and found it immediately. It was looking at
www.spelljammer.org (org not com)
Look at the .com source and you'll see .com is nothing but a shell for .org: <
frame src=http://www.spelljammer.org/...

Since HTT stays on site you get nothing.

Just change the url to .org
 
Reply Create subthread


All articles

Subject Author Date
robots.txt

05/06/2012 15:25
Re: robots.txt

05/06/2012 21:28
Re: robots.txt

05/06/2012 23:06
Re: robots.txt

05/07/2012 15:24
Re: robots.txt

05/07/2012 22:19




4

Created with FORUM 2.0.11