| Subject: Re: how to have maximum performance and not hammer URL |
Author: Jim Zaba |
Date: 01/14/2005 16:47 |
| | This works
1 link per 10 minutes == 1 link per 600 seconds == 1/600
links per second == 0.0017 links per second
However it is not done the right way.
I have 100 URLs (domains)
I have 10 threads.
This should not be global parameter.
1 link per minute
should be 1 link per minut per domain
This way I can spider large amount of data and not hammer
the site. | |
|
|
|
|
4
Created with FORUM 2.0.11