I'm trying to mirror the site www.bigdave44.com
When I surf to it on a browser it spends a few seconds on a page saying
"Cloudflare DDOS protection is checking your browser, please wait." Then
after 5 seconds or so the site appears.
When I try to mirror the site with HTTrack it fails, in about 2 seconds (much
shorter than the wait time).
HTTrack log file:
HTTrack3.47-27+htsswf+htsjava launched on Sat, 15 Apr 2017 14:20:43 at
<http://bigdave44.com> +*.png +*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/*
(winhttrack -qwC2%Ps2u1%B%s%uN0%I0p3DaK0T60H1%kf2A25000%f#f -F "Mozilla/4.5
(compatible; HTTrack 3.0x; Windows 98)" -%F "<!-- Mirrored from %s%s by
HTTrack Website Copier/3.x [XR&CO'2013], %s -->" -%l "en, *"
<http://bigdave44.com> -O1 "D:\working\low\big dave\big dave" +*.png +*.gif
+*.jpg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar )
Information, Warnings and Errors reported for this mirror:
note: the hts-log.txt file, and hts-cache folder, may contain sensitive
such as username/password authentication for websites mirrored in this
do not share these files/folders if you want these information to remain
14:20:45 Error: "Service Temporarily Unavailable" (503) at link
bigdave44.com/ (from primary/primary)
14:20:45 Warning: No data seems to have been transferred during this session!
: restoring previous one!
Contents of the mirror directory:
* * MIRROR ERROR * *
HHTrack has detected that the current mirror is empty. If it was an
update, the previous mirror was restored.
Reason: the first page(s) either could not be found, or a connection
=> Ensure that the website still exits, and/or check your proxy settings!
I manually set timeout to 60 seconds, no change.
I set no robots.txt option, no change.
Is there any way to make HTTrack wait a work on this site? Or is Cloudflare
rejecting it as a DDOS bot?