| Subject: Re: Crawl-Delay and Honored robots.txt lines |
Author: William Roeder |
Date: 03/30/2010 21:48 |
| | > iftop was showing me a pull of 110-140Kb/s, with
> only one connection, so the throughput ratelimiter
> might need some tweaking.
Nothing wrong with the ratelimiter.
But I did say that's the default, I didn't say the security limits couldn't be
overrode. | |
|
|
|
|
a
Created with FORUM 2.0.11