I have a similar problem in WinHTTrack. It is probably not my "browser
identity", which is set to 
"Mozilla/4.78 [en] (Windows NT 5.0; U)"
When I try to mirror the site, I get a .gif file with the words "stop wasting
bandwidth".

I want to follow robots.txt in general. I think the site is rejecting me,
because I request a robots.txt and I am telling it I am using a browser.
 
Can I include a filter "-*site-name.com/robots.txt"?Would it be better to use
"googlebot" as my browser identity?
Thanks in advance for any advice.