| Subject: Re: Set crawl delay? |
Author: Xavier Roche |
Date: 09/07/2013 14:50 |
| | > So, in the site's robot.txt it shows the following:
> User-agent: *
> Crawl-delay: 5
Yep, this isn't processed by httrack.
> How do I set a delay in HTTrack?
It is not really directly possible per request, but you can lower the transfer
rate to something low (such as 10KB/s) and also limit the number of
connections per second:
Options / Scan Rules / Max connections/seconds
0.2
| |
|
|
|
|
4
Created with FORUM 2.0.11