HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Robot Error In Wordpress Site
Author: William Roeder
Date: 02/21/2012 16:02
 
> site(http://digitalmarketer.com/) but only one page
> is downloaded "index.html".
> 
> The Log file reads the following: 
> :
> Please Help!!!
Always post the command line used (log file line two) which you did, plus
anything helpful in the log (there isn't.)

Looking at the page source I see < meta name="robots" content="index,
nofollow" />
So the site doesn't want robots to follow the links and you have options ->
spider -> spider = follow robots.txt, so you get nothing.
 
Reply Create subthread


All articles

Subject Author Date
Robot Error In Wordpress Site

02/21/2012 05:37
Re: Robot Error In Wordpress Site

02/21/2012 16:02
Re: Robot Error In Wordpress Site

02/22/2012 08:33
Re: Robot Error In Wordpress Site

02/22/2012 08:34




7

Created with FORUM 2.0.11