HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt alwyas obeyed. FTW, why?
Author: Yoyo
Date: 08/29/2007 18:03
 
httrack always obeys the robots.txt rules.
I tried -s0 with httrack under Windows/Linux
No matter what I do, I can't grab the content because of the f.. robots.txt
file. 

Is there a precompiled version that NEVER obeys robots.txt?Tried privoxy but
hat problems with it.

Any suggestions?
 
Reply


All articles

Subject Author Date
robots.txt alwyas obeyed. FTW, why?

08/29/2007 18:03




4

Created with FORUM 2.0.11