| Subject: Re: How to HTTR these site |
Author: William Roeder |
Date: 10/25/2008 15:32 |
| | http://www.diaryproject.com/robots.txt says
User-agent: *
Disallow:
# /robots.txt file for <http://www.developerdotstar.com/>
User-agent: *
Disallow: /h0neyp0t/i_would_skip_this_link.php
Disallow: /printable
...
set options no robots.txt | |
|
|
|
|
3
Created with FORUM 2.0.11