HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Silent Crawl?
Author: D_A
Date: 09/28/2005 22:42
 
Htrrack acts like an offline browser.
If you change the User Agent, do not ask for robots.txt and limit the tranfers
to the slowest possible, you may appear like a regular user.
However, humans do not browse a site like a robot (they load 1 to 5 pages,
then come back later and do not explore each link...). Thus any webmaster will
spot the unusual behaviour, will be warned if you fall in a bot traps, if you
download too many pages ...
 
Reply Create subthread


All articles

Subject Author Date
Silent Crawl?

09/28/2005 20:13
Re: Silent Crawl?

09/28/2005 22:42




2

Created with FORUM 2.0.11