| > what's the correct user-agent-string in robots.txt,
> because I wan't to exclude HTTrack from my website,
> because of traffic problems.
'Disallow: /' won't be recognized (too general), use
at least folders (even multiple ones). Any agent will
work, as HTTrack restrict the path using all found
paths in the robots.txt file.
But note that robots can be disabled by the user using
advanced options, and therefore this may not be
sufficient.
See also related documentation information at:
<http://www.httrack.com/HelpHtml/abuse.html>
| |