| > <http://mostgraveconcern.com/freebsd/>
For some reason robots.txt is blocking HTTrack. You can turn off robots.txt
under "Options > Spider > Spider: No robots.txt rules". Please also define
some bandwidth limiting options ("Options > Limits") so HTTrack does not
hammer the website.
Xavier, one question is WHY robots.txt is affecting HTTrack. Here's the file
below, but it is not disallowing /freebsd/ !!
<http://mostgraveconcern.com/robots.txt>
# Anti-robot file
User-agent: *
Disallow: /quotes/
Disallow: /pics/
Disallow: /midi/
Disallow: /start/
Disallow: /friends/
Disallow: /747/
| |