HTTrack Website Copier
Free software offline browser - FORUM
Subject: amother suggestion
Author: Matt
Date: 08/18/2003 19:29
 
How about an option that discontinues the site mirroring 
after x amount of "forbidden" replies are received? Several 
sites lock out the user for 30+ minutes if they detect 
spidering, etc. During this time, some of the sites give 
out "dummy" pages that are 1kb in size with nothing on 
them. So to Httrack, it completed the mirror, but it simply 
downloaded the "forbidden" error pages instead.

Httrack ending the process will allow one to recontinue 
where it left off at a later time when the account is 
reactivated.
 
Reply Create subthread


All articles

Subject Author Date
simple suggestion

08/17/2003 20:43
Re: simple suggestion

08/18/2003 09:48
amother suggestion

08/18/2003 19:29




4

Created with FORUM 2.0.11