HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: local robot.txt
Author: Dan
Date: 08/03/2003 17:38
 
No problem with the same settings online,
I didn't select any specific option,
There's no mention of robots.txt reference in hts-log.txt,
in winprofile.ini I have 
FollowRobotsTxt=2
WildCardFilters=+*.png +*.gif +*.jpg +*.css +*.js -
ad.doubleclick.net/*%0d%0a-*.exe -*.zip -*.pdf -*.hqx

I had the same problem with the previous versions.

 
Reply Create subthread


All articles

Subject Author Date
local robot.txt

08/03/2003 11:17
Re: local robot.txt

08/03/2003 13:14
Re: local robot.txt

08/03/2003 17:38




2

Created with FORUM 2.0.11