HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: More Spider Behavior Options
Author: Xavier Roche
Date: 02/25/2001 10:06
 
>Without this, with a large link list to scan, there 
>may be a damn long wait before anything of interest 
>is actually pulled form the web, as the spider 
>scans 1 level at a time from each link... :(

Well, sound interesting, but there are too many cases 
to handle, I mean spidering behaviour would then 
require some basic spidering-language, and I doubt 
that many people will be able to use it :)
In your case, what you want to do is somethink like 
mirroring several sites, as if you were mirroring the 
next site at the end of the current one. If you don't 
rely upon links within these sites, you can do some 
batch-script that will allow to fire several mirrors 
(not together, but fire 1st mirror, the 2nd one ..)

 
Reply Create subthread


All articles

Subject Author Date
More Spider Behavior Options

02/25/2001 02:47
Re: More Spider Behavior Options

02/25/2001 10:06
Re: More Spider Behavior Options

02/25/2001 11:16




2

Created with FORUM 2.0.11