HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: large websites
Author: Xavier Roche
Date: 05/08/2004 19:30
 
> I was not able to control 
> the transfer rate using the transfer rate limit option. 

This issue is really weird, and I still can not reproduce 
it.. might be related to a specific site, or because of 
multiple small files being received very quickly.

> was doing was attepting to load all the .jpg files in one 
> folder. Windows cannot handle directories that large and 
> when the threshold was reached the remaining files failed 
> to download.

You can try to use the build options to select an user-
defined structure mask ; if the URLs allow that.

Example:
<http://www.example.com/images/image.cgi?sect=flowers&id=45>
<http://www.example.com/images/image.cgi?sect=animals&id=98>
..

Then, use:
%h%p/%[sect::/::]%n%[id:-:::].%t

(note: the %[param:before:after:notfound:empty] syntax will 
include the 'param' query string parameter, 
prepending 'before' and appending 'after' if found, and 
using 'notfound' and 'empty' as replacement if the 
parameter could not be found, or was empty)

But depending on the structure, it can be impossible to do 
so. An alternative would be to use a mirror driven on an 
Linux/Unix box, and then split the image directory.

 
Reply Create subthread


All articles

Subject Author Date
large websites

05/08/2004 19:02
Re: large websites

05/08/2004 19:30




3

Created with FORUM 2.0.11