| > And where i can set this , witch option , i search ,
> but nothing ;)
In WinHTTrack, the setting can be found in the options under the spider tab:
"no robots.txt rules"
(default is "follow robots.txt rules")
Use this setting at your own risk! I know nothing about the site you are
trying to mirror so I can't tell you if ignoring the robots.txt rules is
acceptable or unacceptable.
Some sites just don't want their pages crawled and cached by search engines
but others may have limited bandwidth that mirroring may exceed and make the
site inaccessible to other users.
PLEASE READ THE USER GUIDE AND FILTERS GUIDE before changing this setting. It
is best if you know what you are doing before it's too late. You will
probably get banned (unless you control the banning) if you abuse the site, so
be warned!
| |