| Subject: Set crawl delay? |
Author: LordPorker |
Date: 09/06/2013 13:27 |
| | Trying to copy a site, and having issues with trying to clone it. I suspect
it's to do with a rule in robot text.
So, in the site's robot.txt it shows the following:
User-agent: *
Crawl-delay: 5
How do I set a delay in HTTrack?
Thanks. | |
|
|
|
|
b
Created with FORUM 2.0.11
