| Subject: Re: Robot Error In Wordpress Site |
Author: William Roeder |
Date: 02/21/2012 16:02 |
| | > site(http://digitalmarketer.com/) but only one page
> is downloaded "index.html".
>
> The Log file reads the following:
> :
> Please Help!!!
Always post the command line used (log file line two) which you did, plus
anything helpful in the log (there isn't.)
Looking at the page source I see < meta name="robots" content="index,
nofollow" />
So the site doesn't want robots to follow the links and you have options ->
spider -> spider = follow robots.txt, so you get nothing. | |
|
|
|
|
7
Created with FORUM 2.0.11