| This is a good question. Many websites include anti-
offline browsers measures, because some users abused
the bandwidth, and caused network overload. This is
good, and bad - good because it limit bandwidth abuse,
and bad because "good" users have problems when they
want to make mirror of these websites.
First, if you want to mirror such sites, ensure that
you use reasonnable settings:
* use ONE simultaneous connection
* limit the maximum bandwidth to something like 4KB
* prefer to avoid download during business hours
Then, you can try to change the browser-id, or disable
robots.txt, BUT only if you follow the rules above!
And check the abuse FAQ at:
<http://www.httrack.com/HelpHtml/abuse.html#USERS>
| |