HTTrack Website Copier
Free software offline browser - FORUM
Subject: Cloudflare DDOS protection thwarts HTTrack
Author: Rob
Date: 04/15/2017 08:46
 
Hi All,
I'm trying to mirror the site www.bigdave44.com
When I surf to it on a browser it spends a few seconds on a page saying
"Cloudflare DDOS protection is checking your  browser, please wait." Then
after 5 seconds or so the site appears.

When I try to mirror the site with HTTrack it fails, in about 2 seconds (much
shorter than the wait time).

HTTrack log file:
HTTrack3.47-27+htsswf+htsjava launched on Sat, 15 Apr 2017 14:20:43 at
<http://bigdave44.com> +*.png +*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/*
-mime:application/foobar
(winhttrack -qwC2%Ps2u1%B%s%uN0%I0p3DaK0T60H1%kf2A25000%f#f -F "Mozilla/4.5
(compatible; HTTrack 3.0x; Windows 98)" -%F "<!-- Mirrored from %s%s by
HTTrack Website Copier/3.x [XR&CO'2013], %s -->" -%l "en, *"
<http://bigdave44.com> -O1 "D:\working\low\big dave\big dave" +*.png +*.gif
+*.jpg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar )
Information, Warnings and Errors reported for this mirror:
note: the hts-log.txt file, and hts-cache folder, may contain sensitive
information,
 such as username/password authentication for websites mirrored in this
project
 do not share these files/folders if you want these information to remain
private
14:20:45 Error:  "Service Temporarily Unavailable" (503) at link
bigdave44.com/ (from primary/primary)
14:20:45 Warning:  No data seems to have been transferred during this session!
: restoring previous one!


Contents of the mirror directory:
* * MIRROR ERROR * * 
HHTrack has detected that the current mirror is empty. If it was an
update, the previous mirror was restored.
Reason: the first page(s) either could not be found, or a connection
problem occurred.
=> Ensure that the website still exits, and/or check your proxy settings!
<=



I manually set timeout to 60 seconds, no change.
I set no robots.txt option, no change.

Is there any way to make HTTrack wait a work on this site? Or is Cloudflare
rejecting it as a DDOS bot?
 
Reply


All articles

Subject Author Date
Cloudflare DDOS protection thwarts HTTrack 04/15/2017 08:46
Re: Cloudflare DDOS protection thwarts HTTrack 04/15/2017 14:47
Re: Cloudflare DDOS protection thwarts HTTrack 04/17/2017 06:12




b

Created with FORUM 2.0.11