HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: local robot.txt
Author: Dan
Date: 08/03/2003 17:38
No problem with the same settings online,
I didn't select any specific option,
There's no mention of robots.txt reference in hts-log.txt,
in winprofile.ini I have 
WildCardFilters=+*.png +*.gif +*.jpg +*.css +*.js -*%0d%0a-*.exe -*.zip -*.pdf -*.hqx

I had the same problem with the previous versions.

Reply Create subthread

All articles

Subject Author Date
local robot.txt

08/03/2003 11:17
Re: local robot.txt

08/03/2003 13:14
Re: local robot.txt

08/03/2003 17:38


Created with FORUM 2.0.11