HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Spider identification in robot.txt
Author: Jack Hughes
Date: 05/21/2009 18:14
 
It may have been talked about, but there were no satisfactory answers...
HTTrack should have be capable of being banned from a website using robots.txt
and should be clearly documented in the FAQ.
 
Reply Create subthread


All articles

Subject Author Date
Spider identification in robot.txt

05/21/2009 11:09
Re: Spider identification in robot.txt

05/21/2009 15:30
Re: Spider identification in robot.txt

05/21/2009 18:14
Re: Spider identification in robot.txt

05/21/2009 19:08
Re: Spider identification in robot.txt

05/04/2011 11:04




b

Created with FORUM 2.0.11