httrack always obeys the robots.txt rules.
I tried -s0 with httrack under Windows/Linux
No matter what I do, I can't grab the content because of the f.. robots.txt
file.
Is there a precompiled version that NEVER obeys robots.txt?Tried privoxy but
hat problems with it.
Any suggestions? |