HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Update of large dynamic sites.
Author: Xavier Roche
Date: 07/15/2003 21:57
 
> I am controlling the updates with 
> “httrack -O $site --update -E5 -%c7200” 
> but this only gets a portion of the site.  I want to be 
> able to get the remaining data by restarting the 
following 
> day with something like 
> “httrack -O $site --continue -E5 -%c7200 –i “ 
> but it doesn’t appear to be working,  The size of the 
data 
> isn’t increasing.  I just appears to be restarting the 
> update from the beginning each time.

Yes - httrack will always either quickly rescans the mirror 
from the begining (--continue) or update it (--update), but 
you can not start exactly from the last point.

A solution is to create after these 2 hours an empty file 
called "hts-stop.lock" which will force the engine to 
finish pending transfers and wait until the file is being 
erased (a file, hts-paused.lock, is created once httrack 
has finished pending transfers). The next day you can erase 
the file and the mirror will continue.

Example: updated between 20:00 and 22:00

while true; do

# wait until 20 o'clock
while test `date +%H` -lt 20; do sleep 60; done
# let httrack continue if necessary
test -f hts-paused.lock && echo "httrack was paused, let's 
continue" && rm -f hts-paused.lock
# wait until 22 o'clock
while test `date +%H` -lt 22; do sleep 60; done
# pause httrack
>hts-stop.lock

done
 
Reply Create subthread


All articles

Subject Author Date
Update of large dynamic sites.

07/15/2003 21:44
Re: Update of large dynamic sites.

07/15/2003 21:57
Re: Update of large dynamic sites.

07/29/2003 18:37




9

Created with FORUM 2.0.11