> User-agent: HTTrack
> Disallow: /media
>
> or is it possible to override the httrack user-agent
> with Mozilla 5.0 bla bla ... and fake it?
There are options to override both the user agent (options->BrowserID) and to
ignore some or all of robots.txt (options->spider)
Users overriding robots.txt are advised that not abuse the site
<http://www.httrack.com/html/abuse.html> |