| > I have som problem whit downloading from this site:
> <http://www.stud.bygg.ntnu.no/sia4005/a400501/>
> Can you tell me whats wrong - I shall just download pages
> in this path and underpaths
Hum, the error log is self-explainatory:
19:13:25 Info: Note: due to www.stud.bygg.ntnu.no remote
robots.txt rules, links begining with these path will be
forbidden: /sib3011/, /sia4004/, /sia4005/, /sib3015/, /sib5
003/, /_old/, /admin/, /Templates/ (see in the options to
disable this)
Therefore, disable robots, and it should work (but ask
first the webmaster permission to copy the site, especially
for bandwidth reasons, or limit the number of simultaneous
connections to 2)
| |