| Subject: Re: Won't archive site at... |
Author: Leto |
Date: 11/25/2002 00:06 |
| | > That's right, but the page at
> <http://mostgraveconcern.com/freebsd/> contains the following
> HTML code:
> <META NAME='ROBOTS' CONTENT='NOFOLLOW'>
> ... and httrack checks both robots.txt AND the robots meta-
> tag :)
Aha! :) | |
|
|
|
|
7
Created with FORUM 2.0.11
