> Httrack does not appear to obey robots.txt exclusion 
rules.
> We've tried both WinHttrack and the command line httrack
> with -sN2 option and in neither case does httrack obey the
> robots.txt rules.

HTTrack obeys to robot exclusion rules (except the too 
restrictive '/' rule), but does not takes in account the 
name of the robot ('*', 'foo' or whatever). In the future 
I'll change this behaviour to follow only two set of rules: 
the 'catch all' (*) and a specific httrack rule 
(an "httrack" entry)