HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Spider identification in robot.txt
Author: William Roeder
Date: 05/21/2009 19:08
 
> It may have been talked about, but there were no
> satisfactory answers...
If you don't want users to peruse your site, don't make it public. 

What you called "stolen" I see a user who liked what he saw and wanted to see
the entire site. It's only stolen, when he puts it up as his site.

Httrack and the others, allows a simple way what the users can do with their
browsers anyway.
 
Reply Create subthread


All articles

Subject Author Date
Spider identification in robot.txt

05/21/2009 11:09
Re: Spider identification in robot.txt

05/21/2009 15:30
Re: Spider identification in robot.txt

05/21/2009 18:14
Re: Spider identification in robot.txt

05/21/2009 19:08
Re: Spider identification in robot.txt

05/04/2011 11:04




b

Created with FORUM 2.0.11