HTTrack Website Copier
Free software offline browser - FORUM
Subject: local robot.txt
Author: Dan
Date: 08/03/2003 11:17
 
WinHTTrack Website Copier 3.30-RC-12 (+swf)

Still a small problem with robot.txt rules: 
When I use Winhttrack locally 
<http://127.0.0.1/site/index.php>) it follows meta tags, but 

seems not to read robots.txt.

The new forum uses many cookies from many folders. As I'm 
prompted for what to do with them, 

I must say I prefered the previous cookie.   

Each time I update, I unzip a file to correct a few 
spelling mistakes in the language files. 

I will send it as it may be used by others.
Regards
 
Reply


All articles

Subject Author Date
local robot.txt

08/03/2003 11:17
Re: local robot.txt

08/03/2003 13:14
Re: local robot.txt

08/03/2003 17:38




a

Created with FORUM 2.0.11