HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Why do you allow robots.txt to be overriden?
Author: Xavier Roche
Date: 12/27/2005 22:00
 
> Why do you offer and explain to users the option to
> override this?
Because some users will still want to bypass remote rules, for various
reasons: private copy of small parts of the site (which is perfectly legal in
Europe), web preservation (done by libraries), legal (people who want to copy
part of a site as proof for anything)

This isn't an excuse for abusing, but this is the user's task to ensure that
they aren't doing anything wrong.

> By blocking your user-agent, it is my express wish
> to stop this behavior in my site. 

And this is what httrack does by default.

> Otherwise you are putting a tool out there that
> contributes to copyright infringement, and are not
> doing all you can to avoid this?
This is not copyright infringement.

> As the lawful owner of my website assets, my wish to
> not have them copied without my permission

And this is perfectly normal. And if someone makes a live backup, he'll
probably be liable for damages.

> I understand this would
> make your software less effective, but I believe you
> should err on this side.

This will not make the software less effective, this will only annoy regular
users. "Bad guys" would patch the software to bypass robots rules (as free
software, the sources are public)
 
Reply Create subthread


All articles

Subject Author Date
Why do you allow robots.txt to be overriden?

12/27/2005 21:13
Re: Why do you allow robots.txt to be overriden?

12/27/2005 22:00
Re: Why do you allow robots.txt to be overriden?

12/27/2005 22:39
Re: Why do you allow robots.txt to be overriden?

12/28/2005 00:47
Re: Why do you allow robots.txt to be overriden?

12/30/2005 11:18
Re: Why do you allow robots.txt to be overriden?

01/13/2006 12:23
Re: Why do you allow robots.txt to be overriden?

12/25/2006 19:24
Re: Why do you allow robots.txt to be overriden?

06/26/2007 19:49
Re: Why do you allow robots.txt to be overriden?

04/04/2010 08:06




a

Created with FORUM 2.0.11