HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to get rid of anti-robots protection
Author: Silvia
Date: 05/02/2019 07:35
 
use httrack options to ignore the robots.txt

when you start one project, you can see the options setting page. There is one
option "get the web-pages even forbidden" which is unchecked. Checked this
option then you can success.


 
Reply Create subthread


All articles

Subject Author Date
How to get rid of anti-robots protection

11/19/2008 09:02
Re: How to get rid of anti-robots protection

11/19/2008 17:40
Re: How to get rid of anti-robots protection

11/21/2008 15:45
Re: How to get rid of anti-robots protection

11/21/2008 16:52
Re: How to get rid of anti-robots protection

02/17/2009 09:52
Re: How to get rid of anti-robots protection

05/02/2019 07:35
Re: How to get rid of anti-robots protection

10/18/2019 03:12




0

Created with FORUM 2.0.11