HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: HTTrack Crashed Our Server Last Night
Author: imacat
Date: 05/30/2005 17:46
 
    That is fine.  First friendly answer I have ever get from here. ^_^  Glad
to hear that your tool will enforce a hard limit for resources used.  I wish
you can keep your words.

    I have downloaded HTTrack and tried it.  I was frightened that it can
disable respecting robots.txt rules, setting the User-Agent string.  It
doesn't work for "Disallow: /" (disallow everything), either.  Those are not
polite for a spider, too.  Please remove those options, and fix the "Disallow:
/" problem.  Not all contents are GPL or GDL.  Please respect the content
providers' intension on how to provide or distribute our content, be honest
about your user-agent identity and allow content providers to have special
treatment on you.

    You may refer to the robots.txt standard at:
<http://www.robotstxt.org/wc/norobots.html> .  The "Disallow: " is only a
prefix.  Anything matching that prefix should be excluded, not only
directories.

    I have double checked the wget manual.  No, there is have no option for
simutanuous connections.  The user cannot disable robots.txt rules, too.

    I'm writing GNU GPL softwares, too.  I believe GNU GPL softwares are made
to help people, but not to hurt people.

    A small suggestion:  Allowing abuse only within the same IP network, or
localhost.  That is not hard.
 
Reply Create subthread


All articles

Subject Author Date
HTTrack Crashed Our Server Last Night

05/30/2005 07:49
Re: HTTrack Crashed Our Server Last Night

05/30/2005 08:03
Re: HTTrack Crashed Our Server Last Night

05/30/2005 12:41
Re: HTTrack Crashed Our Server Last Night

05/30/2005 16:41
Re: HTTrack Crashed Our Server Last Night

05/30/2005 17:32
Re: HTTrack Crashed Our Server Last Night

05/30/2005 17:46
Re: HTTrack Crashed Our Server Last Night

05/30/2005 17:50
Re: HTTrack Crashed Our Server Last Night

05/30/2005 17:55
Re: HTTrack Crashed Our Server Last Night

05/30/2005 18:00
Re: HTTrack Crashed Our Server Last Night

05/30/2005 20:56
Re: HTTrack Crashed Our Server Last Night

05/30/2005 21:20
Re: HTTrack Crashed Our Server Last Night

05/31/2005 08:51
Re: HTTrack Crashed Our Server Last Night

05/31/2005 11:37
Re: HTTrack Crashed Our Server Last Night

05/31/2005 12:52
Re: HTTrack Crashed Our Server Last Night

06/02/2005 10:46
Re: to imacat : solution found for u

06/02/2005 10:50
Re: HTTrack Crashed Our Server Last Night

06/02/2005 16:07
My does MySql crash?

06/06/2005 14:59
Re: My does MySql crash?

06/06/2005 17:41
Re: HTTrack Crashed Our Server Last Night

06/09/2005 13:02
Re: HTTrack Crashed Our Server Last Night

06/13/2005 21:57
Re: HTTrack Crashed Our Server Last Night

10/02/2006 10:51




9

Created with FORUM 2.0.11