HTTrack Website Copier
Free software offline browser - FORUM
Subject: ...it's NOT hard... so... :)
Author: Bandit
Date: 11/30/2009 04:55
 
> And where i can set this , witch option , i search ,
> but nothing ;) 

In WinHTTrack, the setting can be found in the options under the spider tab:
"no robots.txt rules"
(default is "follow robots.txt rules")

Use this setting at your own risk!  I know nothing about the site you are
trying to mirror so I can't tell you if ignoring the robots.txt rules is
acceptable or unacceptable.

Some sites just don't want their pages crawled and cached by search engines
but others may have limited bandwidth that mirroring may exceed and make the
site inaccessible to other users.

PLEASE READ THE USER GUIDE AND FILTERS GUIDE before changing this setting.  It
is best if you know what you are doing before it's too late.  You will
probably get banned (unless you control the banning) if you abuse the site, so
be warned!
 
Reply Create subthread


All articles

Subject Author Date
....its hard or i`m stupid ?:))

11/29/2009 22:49
Re: ....its hard or i`m stupid ?:))

11/30/2009 00:45
Re: ....its hard or i`m stupid ?:))

11/30/2009 03:35
...it's NOT hard... so... :)

11/30/2009 04:55
Re: ...it's NOT hard... so... :)

11/30/2009 07:06




6

Created with FORUM 2.0.11