| I have a list of 7 million URLs to download, httrack just won't even load this
list, it's 1.5 GB in size as well.
I think multiple instances is the way to go, there is no mutex on the exe so
this is legal in process terms.
Bearing in mind, when I say multiple instances I mean by the CLI, no caching,
and by using multiple batch files.
If this doesn't work then I will have to go about coding my own crawler,
should not be too much hassle IMO. | |