HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: NoIndex, NoFollow
Author: Oliver
Date: 04/14/2009 09:59
 
But the robots.txt has some shortcomings that can be overcome with the robots
metatag. 

For example there are Typo3 installations that don't have a discernible URL
Pattern for pages offered for pretty printing a page, so HTTrack indexes these
sites (although the robots tag is set correctly)

Hopefully the developer or someone with knowledge about the inner workings can
shed some light on this?
 
Reply Create subthread


All articles

Subject Author Date
NoIndex, NoFollow

04/09/2009 15:28
Re: NoIndex, NoFollow

04/09/2009 16:22
Re: NoIndex, NoFollow

04/14/2009 09:59




f

Created with FORUM 2.0.11