HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Can't find the option to ignore the robots.txt!
Author: Xavier Roche
Date: 11/04/2001 11:37
 
> Quite possible I'm stupid or blind but I can't find 
> the option where I can tell Httrack to ignore the 
> robots.txt file. Thuis the site I'm trying to 
download 
> can't be accessed. Where can I turn this off?
Options/Spider/Spider: "no robots.txt rules"
or, using commandline release: option -s0
 
Reply Create subthread


All articles

Subject Author Date
Can't find the option to ignore the robots.txt!

11/04/2001 11:23
Re: Can't find the option to ignore the robots.txt!

11/04/2001 11:37




6

Created with FORUM 2.0.11