| Subject: Re: Why do you allow robots.txt to be overriden? |
Author: jack crow |
Date: 04/04/2010 08:06 |
| | As long as hosts choose to nuke entire domains like geocities at will; I will
continue to ignore robots.txt. I archive pages for the same I take pictures on
a trip; because the next time I go back it's usually not there anymore. | |
|
|
|
|
1
Created with FORUM 2.0.11
