HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Multithreaded page downloading
Author: JamesL
Date: 10/02/2013 00:07
 
I have a list of 7 million URLs to download, httrack just won't even load this
list, it's 1.5 GB in size as well.

I think multiple instances is the way to go, there is no mutex on the exe so
this is legal in process terms.

Bearing in mind, when I say multiple instances I mean by the CLI, no caching,
and by using multiple batch files.

If this doesn't work then I will have to go about coding my own crawler,
should not be too much hassle IMO.
 
Reply Create subthread


All articles

Subject Author Date
Multithreaded page downloading

09/22/2013 23:40
Re: Multithreaded page downloading

09/24/2013 02:21
Re: Multithreaded page downloading

09/24/2013 04:14
Re: Multithreaded page downloading

09/24/2013 04:15
Re: Multithreaded page downloading

09/24/2013 07:45
Re: Multithreaded page downloading

09/24/2013 20:24
Re: Multithreaded page downloading

09/24/2013 23:39
Re: Multithreaded page downloading

10/02/2013 00:07




2

Created with FORUM 2.0.11