This works
1 link per 10 minutes == 1 link per 600 seconds == 1/600
links per second == 0.0017 links per second
However it is not done the right way.
I have 100 URLs (domains)
I have 10 threads.
This should not be global parameter.
1 link per minute
should be 1 link per minut per domain
This way I can spider large amount of data and not hammer
the site. |