> > > Is the -#L1,000,000,000 an issue here? 
> > I have pretty much proven to myself that it was
> the
> > culprit for the memory use. 
> 
> Are you REALLY going to mirror a Billion links? If
> each page averages 50KB, do you REALLY have 50
> terabytes of disk storage to store them on?> 
> It's an issue for you because of you. Get real.

I guess that the issue from a Software Engineering perspective is that it's
really bad news to statically allocate memory unless you *really* need to do
so.  Sure, a billion links is affordable, but that's not the issue.   I've
written many web crawlers and never had to specify the maximum number of links
that would be crawled and doing so can easily incur many penalties such as
poor cache performance.


Tony