HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: how to have maximum performance and not hammer URL
Author: Jim Zaba
Date: 01/14/2005 16:47
 
This works
1 link per 10 minutes == 1 link per 600 seconds == 1/600 
links per second == 0.0017 links per second

However it is not done the right way.
I have 100 URLs (domains)
I have 10 threads.
This should not be global parameter.

1 link per minute 
should be 1 link per minut per domain
This way I can spider large amount of data and not hammer 
the site.
 
Reply Create subthread


All articles

Subject Author Date
how to have maximum performance and not hammer URL

Jim Zaba

01/05/2005 01:27
Re: how to have maximum performance and not hammer URL

Xavier Roche

01/08/2005 11:08
Re: how to have maximum performance and not hammer URL

Jim Zaba

01/14/2005 16:47




4

Created with FORUM 2.0.11