HTTrack Website Copier
Free software offline browser - FORUM
Subject: Continuing after a crash (crash recovery)
Author: David G
Date: 05/11/2002 22:43
Hi Xavier and others,

The program is outstanding in quality overall, but I
have found one apparent shortcoming which detracts
from its robustness.

In particular, when my computer crashes while doing
a mirror, the program seems often unable to take up
where it left off, even when using the "Continue
Interrupted Download" option.  It seems to want to
reload most or all of the previously downloaded files,
even if they were completely downloaded previously.
This is in contrast to the situation in which the
user _manually_ interrupts a download by pressing
"Cancel".  Then, the program remembers its spot well.
The next time it is run, "Continue Interrupted..."
continues the download from the precise spot that
the user interrupted it.

I would really, really appreciate having the ability
to do that with crashed downloads (i.e., to continue
from the point at which the crash occurred, except
maybe the partially downloaded files at that point,
which can be reloaded).

One way of doing that might be to have the program
automatically save "checkpoints", for example, every
half an hour or so.  These "checkpoints" would record
the portion of the web site which has been successfully
downloaded, as if the user pressed the "Cancel" button
at that point.  That way, when the computer crashed,
you would only lose the downloads made up to the last
checkpoint.  If the checkpoint was saved, for example,
every half an hour, you would only lose half an hour
of downloading.

As the program currently works, I find that after a 
crash, I lose the "download effort" for sites that 
are even 99% complete before the crash occurs.  I
often set up my computer to download series of large
files over a standard dialup connection during the
night in the hope that they will have been downloaded
to my hard drive by morning time (and if not, I can
interrupt the download manually, with "Cancel", and
"Continue..." the transfer in the _next_ night while
I sleep).  However, sometimes I wake up in the morning
to find that my computer has crashed some time during
the night.  As a result, HTTrack does not remember the
point at which the download was "interrupted" (by the
crash).  Because of the apparent incompleteness of its
old internal mirror operation tracking information,
HTTrack therefore starts re-downloading _everything_
in the mirror, including files which were downloaded
completely before the crash.  The end result is a huge
waste of time and bandwidth.  

I don't know if I have explained the issue clearly.
If not, feel free to ask questions.  I am running
version 3.16-2 of HTTrack.  The phenomenon which I
describe has happened to me many times, not just
once or twice.  I would love to find a way to avoid
it, or if a new feature was added to HTTrack to avoid
it (something like a "Continue crashed download..."
option, which overwrites only files downloaded since
the last "checkpoint", and files which had not been
yet downloaded before the crash).

Thanks very much for a most useful utility!

David G


All articles

Subject Author Date
Continuing after a crash (crash recovery)

05/11/2002 22:43
Re: Continuing after a crash (crash recovery)

05/12/2002 01:39


Created with FORUM 2.0.11