HTTrack Website Copier
Free software offline browser - FORUM
Subject: Why do you allow robots.txt to be overriden?
Author: Maria
Date: 12/27/2005 21:13
 
Hi,
I just found out about this tool via my site logs. Someone used it to copy my
pages, and whether it was for good purposes, or not, I would want to stop this
in the future.

I have read your posts and understand that you have several good intentions by
developing this tool. HOWEVER, downloading somebody's web pages and assets and
creating copies of them outside of where they live (hosting server) is
copyright infringement, and against U.S. and international law.

I believe, as such, you should err on the side of protecting webmasters who
explicitly choose to ban this software via robots.txt.

Why do you offer and explain to users the option to override this?By blocking
your user-agent, it is my express wish to stop this behavior in my site. 

I have looked on the forum and read your ABUSE FAQ and can't find an effective
way to stop your tool. You offer several options that fail with dynamic IPs,
and although I understand you can't control someone who really wants to abuse
the tool, I don't understand why you make it so easy for abusers when
robots.txt would perfectly do should you not include a feature that overrides
it.

Could you please consider removing this feature from your software?Otherwise
you are putting a tool out there that contributes to copyright infringement,
and are not doing all you can to avoid this?
As the lawful owner of my website assets, my wish to not have them copied
without my permission should be protected over the will of others, no matter
how good their intentions are. I understand this would make your software less
effective, but I believe you should err on this side.

Thanks.
 
Reply


All articles

Subject Author Date
Why do you allow robots.txt to be overriden?

12/27/2005 21:13
Re: Why do you allow robots.txt to be overriden?

12/27/2005 22:00
Re: Why do you allow robots.txt to be overriden?

12/27/2005 22:39
Re: Why do you allow robots.txt to be overriden?

12/28/2005 00:47
Re: Why do you allow robots.txt to be overriden?

12/30/2005 11:18
Re: Why do you allow robots.txt to be overriden?

01/13/2006 12:23
Re: Why do you allow robots.txt to be overriden?

12/25/2006 19:24
Re: Why do you allow robots.txt to be overriden?

06/26/2007 19:49
Re: Why do you allow robots.txt to be overriden?

04/04/2010 08:06




d

Created with FORUM 2.0.11