HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Won't archive site at...
Author: Leto
Date: 11/25/2002 00:06
 
> That's right, but the page at 
> <http://mostgraveconcern.com/freebsd/> contains the following 
> HTML code:
> <META NAME='ROBOTS' CONTENT='NOFOLLOW'>
> ... and httrack checks both robots.txt AND the robots meta-
> tag :)

Aha! :)
 
Reply Create subthread


All articles

Subject Author Date
Won't archive site at...

11/22/2002 16:40
Re: Won't archive site at...

11/24/2002 22:59
Re: Won't archive site at...

11/24/2002 23:15
Re: Won't archive site at...

11/25/2002 00:06
Re: Won't archive site at...

11/29/2002 00:06




7

Created with FORUM 2.0.11