> Setting maximum number of links scanned to 30 000
> 000 (30 million) generates a nice memory hole
Why are you surprised. Every link is stored in ram and you increased the max
from 100K go 30M.

> Sadly there are some pages that have so many
> links(dictionaries)
Do you really want to mirror 30 million pages to your machine?