| Subject: Re: How does work the robot.txt ? |
Author: Xavier Roche |
Date: 04/17/2002 20:58 |
| | > How does work the robot.txt ?> Is it all the configuations of HTTrack process
?> Where can I find a file robot.txt to know it is
> implemented ?
See <http://www.robotstxt.org/wc/robots.html>
HTTrack, by default, always check the robots.txt
file ; and catch ALL disallow paths. You can disable
the robots.txt use, as offline browsers are something
between browsers and spiders, but thsis is generally
not a good idea.
| |
|
|
|
|
3
Created with FORUM 2.0.11