I am trying to mirror a message board and I am occasionally getting the
following message in testing:
"The server is temporarily unable to service your request due to maintenance
downtime or capacity problems. Please try again later."
Is this something that can be cleared up with expanding the timeout or retry
count? I am obeying robots.txt >
User-agent: MJ12bot
Disallow: /
User-agent: Slurp
Crawl-delay: 1
User-agent: msnbot
Crawl-delay: 1
User-agent: BecomeBot
Disallow: /
User-agent: *
Crawl-delay: 1
Disallow:
|