| > For this project we have to download and archive
sites
> of Dutch political parties. To perform the download
we
> use Httrack version 3.05 (Unix-version).
> With this Unix version we have encountered a problem
> we cant explain ourselves:
> Httrack seems to interpret internal links sometimes
as
> external links without any obvious reason.
Always look at the hts-txt.log file:
16:07:32 Info: Note: due to www.cda.nl remote
robots.txt rules, links begining with these path will
be forbidden: /articles/ (see in the options to
disable this)
Therefore, use -s0 to disable robots.txt behaviour
| |