HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Multiple instances, single instance hogs CPU
Author: Niels van Hecke
Date: 12/16/2013 21:46
> > I am running windows server 2012 and am currently
> > trying to back up a very large website for my
> > employer and have run into an issue with using
> > multiple instances (each instance is responsible
> for
> > one section of the site). For several days the
> > usage of each instance is within reason, but then
> a
> > single instance's CPU usage spikes to over 70% of
> > the CPU of my computer. The CPU spike doesn't
> seem
> > to be correlated to file parsing and doesn't seem
> to
> > be linked to large file fetching.
> How many links are being processed ? httrack is
> known not to be good beyond 100,000 links (millions
> links will probably cause performance issues in
> various places), and this might be the cause.

The total number of seeds is 6161 split into 616/instance. I don't know how
many links there are per seed, but it varies from seed to seed. Is there a way
to figure out the average or some sort of information about the state of the
crawler?Some additional info: I have only one connection/instance to keep
traffic down (we cannot sacrifice speed or processing time on the host
server). The action status is 'ready'.
Reply Create subthread

All articles

Subject Author Date
Multiple instances, single instance hogs CPU

11/22/2013 17:41
Re: Multiple instances, single instance hogs CPU

11/23/2013 11:25
Re: Multiple instances, single instance hogs CPU

12/16/2013 21:46


Created with FORUM 2.0.11