HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Help with mirroring
Author: Kumar
Date: 02/22/2003 12:58
 
Set spider option not to follow robots.txt rules.

In Winhttrack, it is in the Set Options/ Spider tab.

sN follow robots.txt and meta robots tags 
(0=never,1=sometimes,* 2=always) (--robots[=N])
 
Reply Create subthread


All articles

Subject Author Date
Help with mirroring

02/22/2003 04:21
Re: Help with mirroring

02/22/2003 12:58




4

Created with FORUM 2.0.11