HTTrack Website Copier
Free software offline browser - FORUM
Subject: Set crawl delay?
Author: LordPorker
Date: 09/06/2013 13:27
 
Trying to copy a site, and having issues with trying to clone it. I suspect
it's to do with a rule in robot text.

So, in the site's robot.txt it shows the following:

User-agent:  *
Crawl-delay:  5

How do I set a delay in HTTrack?
Thanks.
 
Reply


All articles

Subject Author Date
Set crawl delay?

09/06/2013 13:27
Re: Set crawl delay?

09/07/2013 14:50




b

Created with FORUM 2.0.11