HTTrack Website Copier
Free software offline browser - FORUM
Subject: User-agent in robots.txt for HTTrack
Author: Stefan Einspender
Date: 03/06/2002 13:19
 
hi!

what's the correct user-agent-string in robots.txt,
because I wan't to exclude HTTrack from my website,
because of traffic problems.

User-agent: HTTrack
Disallow: /

or

User-agent: IDentity
Disallow: /

?
greetings from germany, stefan
 
Reply


All articles

Subject Author Date
User-agent in robots.txt for HTTrack

03/06/2002 13:19
Re: User-agent in robots.txt for HTTrack

03/06/2002 20:53
Re: User-agent in robots.txt for HTTrack

03/09/2002 00:00




7

Created with FORUM 2.0.11