| > Xavier, one question is WHY robots.txt is affecting
> HTTrack. Here's the file below, but it is not
> disallowing /freebsd/ !!
That's right, but the page at
<http://mostgraveconcern.com/freebsd/> contains the following
HTML code:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
... and httrack checks both robots.txt AND the robots meta-
tag :)
| |