| > So, in the site's robot.txt it shows the following:
> User-agent: *
> Crawl-delay: 5
Yep, this isn't processed by httrack.
> How do I set a delay in HTTrack?
It is not really directly possible per request, but you can lower the transfer
rate to something low (such as 10KB/s) and also limit the number of
connections per second:
Options / Scan Rules / Max connections/seconds
0.2
| |