> So for example if I need to crawl all the pages at
> depth 3 only then I would need all the links at
> depth 2 which I can only get at depth 1. So I need
> to go down the tree to get all the links. thats ok.
> But I want to 'download' only the pages at depth 3
> and not the ones at depth 2 and depth 1. How can I
> do it? 
How do you think you can get the links at depth 1 and 2 WITHOUT downloading
them?