HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: local robot.txt
Author: Xavier Roche
Date: 08/03/2003 13:14
> WinHTTrack Website Copier 3.30-RC-12 (+swf)
> Still a small problem with robot.txt rules: 
> When I use Winhttrack locally 
> <>) it follows meta tags, 
> seems not to read robots.txt.

Strange.. did you select any specific option when running 
httrack? Any robots.txt reference in hts-log.txt ?

Reply Create subthread

All articles

Subject Author Date
local robot.txt

08/03/2003 11:17
Re: local robot.txt

08/03/2003 13:14
Re: local robot.txt

08/03/2003 17:38


Created with FORUM 2.0.11