HTTrack Website Copier
Free software offline browser - FORUM
Subject: HTTrack not obeying robots.txt
Author: Aaron
Date: 11/27/2019 19:34
Hi guys,

My site is behind a DSL line and just got hammered hard by an HTTrack user. I
notice that HTTrack does a GET on robots.txt right as it starts crawling. Can
you please make HTTrack obey
the value specified in Crawl-delay?

All articles

Subject Author Date
HTTrack not obeying robots.txt

11/27/2019 19:34


Created with FORUM 2.0.11