HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Crawl-Delay and Honored robots.txt lines
Author: William Roeder
Date: 03/30/2010 21:48
 
> iftop was showing me a pull of 110-140Kb/s, with
> only one connection, so the throughput ratelimiter
> might need some tweaking.
Nothing wrong with the ratelimiter.
But I did say that's the default, I didn't say the security limits couldn't be
overrode.
 
Reply Create subthread


All articles

Subject Author Date
Crawl-Delay and Honored robots.txt lines

03/30/2010 17:12
Re: Crawl-Delay and Honored robots.txt lines

03/30/2010 19:47
Re: Crawl-Delay and Honored robots.txt lines

03/30/2010 20:40
Re: Crawl-Delay and Honored robots.txt lines

03/30/2010 21:48
Re: Crawl-Delay and Honored robots.txt lines

04/07/2010 18:42




a

Created with FORUM 2.0.11