HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: how to have maximum performance and not hammer URL
Author: Jim Zaba
Date: 01/14/2005 16:47
 
This works
1 link per 10 minutes == 1 link per 600 seconds == 1/600 
links per second == 0.0017 links per second

However it is not done the right way.
I have 100 URLs (domains)
I have 10 threads.
This should not be global parameter.

1 link per minute 
should be 1 link per minut per domain
This way I can spider large amount of data and not hammer 
the site.
 
Reply Create subthread


All articles

Subject Author Date
how to have maximum performance and not hammer URL

01/05/2005 01:27
Re: how to have maximum performance and not hammer URL

01/08/2005 11:08
Re: how to have maximum performance and not hammer URL

01/14/2005 16:47




4

Created with FORUM 2.0.11