Re: Using Options to disable remote robots.txt rul

Subject: Re: Using Options to disable remote robots.txt rul

Author: William Roeder

Date: 09/17/2007 04:28

> robots.txt rules, links begining with these path

options -> Spider -> Spider = no robots txt rules

Also try 3.32

All articles

Subject	Author	Date
Using Options to disable remote robots.txt ruls		09/17/2007 00:11
Re: Using Options to disable remote robots.txt rul		09/17/2007 04:28
Re: Using Options to disable remote robots.txt rul		04/20/2016 20:34
Re: Using Options to disable remote robots.txt rul		09/24/2007 01:51
Re: Using Options to disable remote robots.txt rul		04/20/2016 18:57
Re: Using Options to disable remote robots.txt ruls		11/26/2017 14:03
Re: Using Options to disable remote robots.txt ruls		05/08/2020 20:43