| No I did not test it... and it did not work for me either, to just disable
robots.txt rules.
However, I succesfully mirrored the web page in question by telling HTTrack
to:
1: Ignore robots.txt rules
2: Set the user agent to "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"
From the command line it is done with these parameters:
-s0 -F "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" | |