HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Using Options to disable remote robots.txt rul
Author: William Roeder
Date: 09/17/2007 04:28
 
> robots.txt rules, links begining with these path

options -> Spider -> Spider = no robots txt rules

Also try 3.32
 
Reply Create subthread


All articles

Subject Author Date
Using Options to disable remote robots.txt ruls 09/17/2007 00:11
Re: Using Options to disable remote robots.txt rul 09/17/2007 04:28
Re: Using Options to disable remote robots.txt rul 04/20/2016 20:34
Re: Using Options to disable remote robots.txt rul 09/24/2007 01:51
Re: Using Options to disable remote robots.txt rul 04/20/2016 18:57




8

Created with FORUM 2.0.11