| Subject: Re: PHP based websites |
Author: Xavier Roche |
Date: 11/23/2003 18:33 |
| | > I am frustrated with innormous settings and a night spent
> in figuring out how to get past the damn robot.txt...
Set Options / Spider / Spider: no robots.txt
Please use bandwith limits, too, to avoid to clobber the
website, THANKS.
| |
|
|
|
|
0
Created with FORUM 2.0.11
