| Subject: Re: Help with mirroring |
Author: Kumar |
Date: 02/22/2003 12:58 |
| | Set spider option not to follow robots.txt rules.
In Winhttrack, it is in the Set Options/ Spider tab.
sN follow robots.txt and meta robots tags
(0=never,1=sometimes,* 2=always) (--robots[=N])
| |
|
|
|
|
4
Created with FORUM 2.0.11
