HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: follow robots meta tag??
Author: Leto
Date: 06/04/2002 07:34
 
> 06:33:45 Warning:  Link xxx not scanned (follow robots meta 
> tag).
> I try to change Browser identity to IE Explorer 5.50, but 
> result is still the same.

You can tell HTTrack to not obey those rules and act more like the browser. 
Go to the options:

Spider > Spider: > change to "no robots.txt rules"

Please note that some sites use robots.txt to stop programs like HTTrack from
hammering their sites, so set your options to also have few "Number of
connections" and also limit the "Max transfer rate"
 
Reply Create subthread


All articles

Subject Author Date
follow robots meta tag??

06/04/2002 07:04
Re: follow robots meta tag??

06/04/2002 07:34
Re: follow robots meta tag??

03/11/2022 17:32




4

Created with FORUM 2.0.11