HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to get rid of anti-robots protection
Author: Elias
Date: 11/21/2008 15:45
 
> Your log shows your mirroring blackviper.com not www.blackviper.com.

Doesn't matter. I also tried with www, but with no luck.

> You override this with options -> spider.

Could you please tell me which exactly option overrides robots.txt protection?
I tried to use --robots and --spider, but also had no luck. I'm still gettings
"403" error which says access to the sites is forbidden. Still I can access
the website from my web browser (Konqueror).

If I can access the web site using my web browser, may be I should try to
change httrack's browser ID to that one Konqueror uses?
 
Reply Create subthread


All articles

Subject Author Date
How to get rid of anti-robots protection

11/19/2008 09:02
Re: How to get rid of anti-robots protection

11/19/2008 17:40
Re: How to get rid of anti-robots protection

11/21/2008 15:45
Re: How to get rid of anti-robots protection

11/21/2008 16:52
Re: How to get rid of anti-robots protection

02/17/2009 09:52
Re: How to get rid of anti-robots protection

05/02/2019 07:35




4

Created with FORUM 2.0.11