HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: User-agent in robots.txt for HTTrack
Author: Xavier Roche
Date: 03/06/2002 20:53
 
> what's the correct user-agent-string in robots.txt,
> because I wan't to exclude HTTrack from my website,
> because of traffic problems.

'Disallow: /' won't be recognized (too general), use 
at least folders (even multiple ones). Any agent will 
work, as HTTrack restrict the path using all found 
paths in the robots.txt file.

But note that robots can be disabled by the user using 
advanced options, and therefore this may not be 
sufficient.

See also related documentation information at:
<http://www.httrack.com/HelpHtml/abuse.html>
 
Reply Create subthread


All articles

Subject Author Date
User-agent in robots.txt for HTTrack

03/06/2002 13:19
Re: User-agent in robots.txt for HTTrack

03/06/2002 20:53
Re: User-agent in robots.txt for HTTrack

03/09/2002 00:00




7

Created with FORUM 2.0.11