HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: www.Direct.gov.uk
Author: William
Date: 09/26/2005 21:50
 
If the articles are in the /assetRoot/ directory on this site, here is the
reason:  (www.direct.gov.uk robots.txt file)

User-agent: UKOnline
Disallow:

User-agent: *
Disallow: /assetRoot/

You need to Mirror, Modify Options and click the Spider tab.  in the
robots.txt combo box, select "no robots.txt rules"
 
Reply Create subthread


All articles

Subject Author Date
www.Direct.gov.uk

09/23/2005 10:32
Re: www.Direct.gov.uk

09/26/2005 21:50
b020dde283@lpv.lv

04/23/2006 15:53




9

Created with FORUM 2.0.11