| > How does work the robot.txt ?> Is it all the configuations of HTTrack process
?> Where can I find a file robot.txt to know it is
> implemented ?
See <http://www.robotstxt.org/wc/robots.html>
HTTrack, by default, always check the robots.txt
file ; and catch ALL disallow paths. You can disable
the robots.txt use, as offline browsers are something
between browsers and spiders, but thsis is generally
not a good idea.
| |