| > site(http://digitalmarketer.com/) but only one page
> is downloaded "index.html".
>
> The Log file reads the following:
> :
> Please Help!!!
Always post the command line used (log file line two) which you did, plus
anything helpful in the log (there isn't.)
Looking at the page source I see < meta name="robots" content="index,
nofollow" />
So the site doesn't want robots to follow the links and you have options ->
spider -> spider = follow robots.txt, so you get nothing. | |