HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Using Options to disable remote robots.txt ruls
Author: SV
Date: 11/26/2017 14:03
 
Okay it seems, you are not actually looking deeply into the SET OPTION after
entering the URL in the Box.

Click on set option -> Go to Spider ->Do Not Follow Robots.txt 

Done.

Hope it should be easy and helpful.
 
Reply Create subthread


All articles

Subject Author Date
Using Options to disable remote robots.txt ruls

09/17/2007 00:11
Re: Using Options to disable remote robots.txt rul

09/17/2007 04:28
Re: Using Options to disable remote robots.txt rul

09/24/2007 01:51
Re: Using Options to disable remote robots.txt rul

04/20/2016 18:57
Re: Using Options to disable remote robots.txt ruls

11/26/2017 14:03




b

Created with FORUM 2.0.11