HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: NoIndex, NoFollow
Author: William Roeder
Date: 04/09/2009 16:22
> I have some trouble getting httrack to recognize the
> robots meta tag. 
> From what i understand from the documentation is
> that the command line tool by default reads the
> robots.txt and the appropiate tags.
> But still sites are indexed.

I don't know about the meta tag but httrack does recognize and follows
robots.txt by default.

That setting can be overridden by an option.

Reply Create subthread

All articles

Subject Author Date
NoIndex, NoFollow

04/09/2009 15:28
Re: NoIndex, NoFollow

04/09/2009 16:22
Re: NoIndex, NoFollow

04/14/2009 09:59


Created with FORUM 2.0.11