| >Without this, with a large link list to scan, there
>may be a damn long wait before anything of interest
>is actually pulled form the web, as the spider
>scans 1 level at a time from each link... :(
Well, sound interesting, but there are too many cases
to handle, I mean spidering behaviour would then
require some basic spidering-language, and I doubt
that many people will be able to use it :)
In your case, what you want to do is somethink like
mirroring several sites, as if you were mirroring the
next site at the end of the current one. If you don't
rely upon links within these sites, you can do some
batch-script that will allow to fire several mirrors
(not together, but fire 1st mirror, the 2nd one ..)
| |