HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Set crawl delay?
Author: Xavier Roche
Date: 09/07/2013 14:50
 
> So, in the site's robot.txt it shows the following:
> User-agent:  *
> Crawl-delay:  5

Yep, this isn't processed by httrack.

> How do I set a delay in HTTrack?
It is not really directly possible per request, but you can lower the transfer
rate to something low (such as 10KB/s) and also limit the number of
connections per second:

Options / Scan Rules / Max connections/seconds
0.2

 
Reply Create subthread


All articles

Subject Author Date
Set crawl delay?

09/06/2013 13:27
Re: Set crawl delay?

09/07/2013 14:50




4

Created with FORUM 2.0.11