HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: local robot.txt
Author: Xavier Roche
Date: 08/03/2003 13:14
 
> WinHTTrack Website Copier 3.30-RC-12 (+swf)
> Still a small problem with robot.txt rules: 
> When I use Winhttrack locally 
> <http://127.0.0.1/site/index.php>) it follows meta tags, 
but 
> seems not to read robots.txt.

Strange.. did you select any specific option when running 
httrack? Any robots.txt reference in hts-log.txt ?

 
Reply Create subthread


All articles

Subject Author Date
local robot.txt

08/03/2003 11:17
Re: local robot.txt

08/03/2003 13:14
Re: local robot.txt

08/03/2003 17:38




0

Created with FORUM 2.0.11