| | If the articles are in the /assetRoot/ directory on this site, here is the
reason: (www.direct.gov.uk robots.txt file)
User-agent: UKOnline
Disallow:
User-agent: *
Disallow: /assetRoot/
You need to Mirror, Modify Options and click the Spider tab. in the
robots.txt combo box, select "no robots.txt rules" | |
