| > Thanks for the reply. I guess the order in which I have
my
> scan rules wouldn't have an effect, right?
An effect on the mirror scope, yes. But not on the link
fetch order ; which is really simple:
- an empty heap of links is created at the begining of the
program
- for each page, all links are added if suitable (not yet
know, and within the mirror scope) in the heap, in page
link order
- the crawler fetches pages one by one on the heap
This allows to crawl the "level N-1" completely, then go
in "level N" and so on..
| |