HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Won't archive site at...
Author: Xavier Roche
Date: 11/24/2002 23:15
 
> Xavier, one question is WHY robots.txt is affecting
> HTTrack. Here's the file below, but it is not
> disallowing /freebsd/ !!

That's right, but the page at 
<http://mostgraveconcern.com/freebsd/> contains the following 
HTML code:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">

... and httrack checks both robots.txt AND the robots meta-
tag :)
 
Reply Create subthread


All articles

Subject Author Date
Won't archive site at...

11/22/2002 16:40
Re: Won't archive site at...

11/24/2002 22:59
Re: Won't archive site at...

11/24/2002 23:15
Re: Won't archive site at...

11/25/2002 00:06
Re: Won't archive site at...

11/29/2002 00:06




4

Created with FORUM 2.0.11