HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt alwyas obeyed. FTW, why?
Author: Yoyo
Date: 08/29/2007 18:03
httrack always obeys the robots.txt rules.
I tried -s0 with httrack under Windows/Linux
No matter what I do, I can't grab the content because of the f.. robots.txt

Is there a precompiled version that NEVER obeys robots.txt?Tried privoxy but
hat problems with it.

Any suggestions?

All articles

Subject Author Date
robots.txt alwyas obeyed. FTW, why?

08/29/2007 18:03


Created with FORUM 2.0.11