HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Can winhttrack get certain pages before others?
Author: Xavier Roche
Date: 03/13/2005 11:03
 
> Thanks for the reply. I guess the order in which I have 
my 
> scan rules wouldn't have an effect, right?
An effect on the mirror scope, yes. But not on the link 
fetch order ; which is really simple:

- an empty heap of links is created at the begining of the 
program
- for each page, all links are added if suitable (not yet 
know, and within the mirror scope) in the heap, in page 
link order
- the crawler fetches pages one by one on the heap

This allows to crawl the "level N-1" completely, then go 
in "level N" and so on..

 
Reply Create subthread


All articles

Subject Author Date





8

Created with FORUM 2.0.11