HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to control hits / minute?
Author: Xavier Roche
Date: 11/20/2001 19:56
 
> I am trying to mirror a site that does not allow 
> crawling at faster than 8 hits / minute (average).  
I 
> have tried various settings, even setting the 
> simultaneous connections to 1, but still get errors 
on 
> my mirror indicating that I am crawling too fast.

You can limit the hits at 1 hit/second (60/minute) - 
8/minute is quite LOW, and therefore you may want to 
use the bandwidth limiter (to something like 100 
bytes/s), and 1 connection, this may be sufficient.
If not, you may have to use the -V option (execute an 
external command after each file being stored) with 
something like -V "sleep 8" - for example in the 
filter list, add:
-V "C:\winnt\sleep.exe 1"

(the sleep.exe program must be present)
 
Reply Create subthread


All articles

Subject Author Date
How to control hits / minute?

11/19/2001 21:53
Re: How to control hits / minute?

11/20/2001 19:56




4

Created with FORUM 2.0.11