| > I am trying to mirror a site that does not allow
> crawling at faster than 8 hits / minute (average).
I
> have tried various settings, even setting the
> simultaneous connections to 1, but still get errors
on
> my mirror indicating that I am crawling too fast.
You can limit the hits at 1 hit/second (60/minute) -
8/minute is quite LOW, and therefore you may want to
use the bandwidth limiter (to something like 100
bytes/s), and 1 connection, this may be sufficient.
If not, you may have to use the -V option (execute an
external command after each file being stored) with
something like -V "sleep 8" - for example in the
filter list, add:
-V "C:\winnt\sleep.exe 1"
(the sleep.exe program must be present)
| |