| Subject: robots.txt alwyas obeyed. FTW, why? |
Author: Yoyo |
Date: 08/29/2007 18:03 |
| |
httrack always obeys the robots.txt rules.
I tried -s0 with httrack under Windows/Linux
No matter what I do, I can't grab the content because of the f.. robots.txt
file.
Is there a precompiled version that NEVER obeys robots.txt?Tried privoxy but
hat problems with it.
Any suggestions? | |
|
|
|
|
f
Created with FORUM 2.0.11
