HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Using Options to disable remote robots.txt rul
Author: William Roeder
Date: 09/17/2007 04:28
 
> robots.txt rules, links begining with these path

options -> Spider -> Spider = no robots txt rules

Also try 3.32
 
Reply Create subthread


All articles

Subject Author Date
Using Options to disable remote robots.txt ruls 09/17/2007 00:11
Re: Using Options to disable remote robots.txt rul 09/17/2007 04:28
Re: Using Options to disable remote robots.txt rul 09/24/2007 01:51




8

Created with FORUM 2.0.11