HTTrack Website Copier
Free software offline browser - FORUM
Subject: Set crawl delay?
Author: LordPorker
Date: 09/06/2013 13:27
Trying to copy a site, and having issues with trying to clone it. I suspect
it's to do with a rule in robot text.

So, in the site's robot.txt it shows the following:

User-agent:  *
Crawl-delay:  5

How do I set a delay in HTTrack?

All articles

Subject Author Date
Set crawl delay?

09/06/2013 13:27
Re: Set crawl delay?

09/07/2013 14:50


Created with FORUM 2.0.11