So for example if I need to crawl all the pages at depth 3 only then I would
need all the links at depth 2 which I can only get at depth 1. So I need to go
down the tree to get all the links. thats ok. But I want to 'download' only
the pages at depth 3 and not the ones at depth 2 and depth 1. How can I do it?