Subject: Re: Why do you allow robots.txt to be overriden?
Author: jack crow
Date: 04/04/2010 08:06
As long as hosts choose to nuke entire domains like geocities at will; I will
continue to ignore robots.txt. I archive pages for the same I take pictures on
a trip; because the next time I go back it's usually not there anymore.
