| Subject: Re: Spider identification in robot.txt |
Author: Jack Hughes |
Date: 05/21/2009 18:14 |
| | It may have been talked about, but there were no satisfactory answers...
HTTrack should have be capable of being banned from a website using robots.txt
and should be clearly documented in the FAQ. | |
|
|
|
|
b
Created with FORUM 2.0.11
