HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How does work the robot.txt ?
Author: Xavier Roche
Date: 04/17/2002 20:58
 
> How does work the robot.txt ?> Is it all the configuations of HTTrack process
?> Where can I find a file robot.txt to know it is 
> implemented ?
See <http://www.robotstxt.org/wc/robots.html>

HTTrack, by default, always check the robots.txt 
file ; and catch ALL disallow paths. You can disable 
the robots.txt use, as offline browsers are something 
between browsers and spiders, but thsis is generally 
not a good idea.
 
Reply Create subthread


All articles

Subject Author Date
How does work the robot.txt ?

04/17/2002 15:58
Re: How does work the robot.txt ?

04/17/2002 20:58




3

Created with FORUM 2.0.11