HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Won't archive site at...
Author: Leto
Date: 11/24/2002 22:59
 
> <http://mostgraveconcern.com/freebsd/>

For some reason robots.txt is blocking HTTrack. You can turn off robots.txt
under "Options > Spider > Spider: No robots.txt rules".  Please also define
some bandwidth limiting options ("Options > Limits") so HTTrack does not
hammer the website.

Xavier, one question is WHY robots.txt is affecting HTTrack. Here's the file
below, but it is not disallowing /freebsd/ !!

<http://mostgraveconcern.com/robots.txt>
# Anti-robot file
User-agent: *
Disallow: /quotes/
Disallow: /pics/
Disallow: /midi/
Disallow: /start/
Disallow: /friends/
Disallow: /747/
 
Reply Create subthread


All articles

Subject Author Date
Won't archive site at...

11/22/2002 16:40
Re: Won't archive site at...

11/24/2002 22:59
Re: Won't archive site at...

11/24/2002 23:15
Re: Won't archive site at...

11/25/2002 00:06
Re: Won't archive site at...

11/29/2002 00:06




c

Created with FORUM 2.0.11