| Subject: Re: Trouble to download wordpress MU website |
Author: William Roeder |
Date: 06/02/2008 21:48 |
| | > On the websites I deleted it <meta name='robots'
> content='noindex,nofollow' />
Tell httrack to ignore it: options -> Spider -> Spider = "no robots.txt except
wizard" or "no robots.txt rules" | |
|
|
|
|
4
Created with FORUM 2.0.11
