HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: memory eating bug
Author: William Roeder
Date: 02/01/2009 17:46
 
> It works fine with 10 million links. Maximum memory
> amount of used during scanning (300000 links
> found)and memory usage is not over 100mb. With 30
> million links : it eats up 1 gb memory(all free
> memory i have) when starting scanning(scanning
> itself wont start). 
> 
> I really hope that I can handle with 5 million
> links, but a little bit overhead won't be bad,
> because running mirroring twice, because there were
> some more links to scan is a ridiculous thing

Your numbers don't make sense to me and you don't say why it wont start.

I ran a site with 47K links. It ran in 304MB virtual.  That's ruffly 7KB per
link.

Where did you get that 100mb figure? If it was task manager/Mem Usage that's
what's in RAM, not total (look at VM Size instead.)

300K links should require 2100MB and 30 million should require 21GB. What is
your virtual memory settings?
 
Reply Create subthread


All articles

Subject Author Date
memory eating bug

01/30/2009 16:59
Re: memory eating bug

01/30/2009 18:15
Re: memory eating bug

01/30/2009 22:08
Re: memory eating bug

02/01/2009 17:46




5

Created with FORUM 2.0.11