HTTrack Website Copier
Free software offline browser - FORUM
Subject: problem making a mirror of a VERY large site
Author: Charles Whittle
Date: 04/23/2006 02:10
 
Howdy!

   I have been attempting to make a mirror of a VERY large
web site ( <http://community.webshots.com> ) which as of
April 22 2006 has 365,600,739 photos (522,459 new in 24 hours).  Even with the
option for downloading html files
first I only get about 49,000 files.  Aaron's WebVacuum
will get several hundred thousand jpegs, but then it gets
bogged down (even with 2.6 gigabytes of RAM with a Pentium
4 processor (3.3 gigahertz speed).  The large format images
are located in sites <http://image##.webshots.com> and the
thumbnails are in sites <http://thumb##.webshots.com> ( # is
a single digit number).  If you visit the parent URL and
link through it a bit you will find that it has recursive
links in abundance that lead back to earlier pages, so the
option to go down only does not seem to work well (could
be wrong about that).  Even setting the download speed to
dialup rates does not help.  

   This problem also occurs with other huge web sites.  I'm
in no hurry to download the entire site, so how do I set up
the options so that the entire site is mirrored?  Even
excluding all files except html and image files does not 
make a difference.  HTTRACK locks up, the elapsed time
counter and download displays freeze, and the program can
only be ended with the task manager.  I am downloading the
files onto a freshly formatted hard drive, so a lack of
defragging is not the problem.  (nothing else is on the
drive.)  Any help would be most welcome.  Thanks!

           Aloha,
              Charles Whittle

PS:  I am running HTTRACK on a GateWay 550GR, Pentium 4
(3.3 GHz) with 2.5 GB RAM using Windows XP Home Edition
with all updates, primary drive has 250 GB, and secondary
drive has 200 GB (location of My Web Sites [target folder
for HTTRACK]).
 
Reply


All articles

Subject Author Date
problem making a mirror of a VERY large site

04/23/2006 02:10
Re: problem making a mirror of a VERY large site

04/23/2006 10:22
Re: problem making a mirror of a VERY large site

04/23/2006 18:40




f

Created with FORUM 2.0.11