| > I am getting a regular broken link message in the log
file.
> However the paths work fine in a browser.
> Here are the lines:
> 09:51:14 Error: "Not Found" (404) at link
> www.thenation.com/directory/ (from primary/primary)
The site is filtering offline browsers (including httrack).
To avoid that, change the Browser identity (Set Option /
Browser ID / Browser "identity" : set it to IE or
Netscape) - but *first* ensure that you have setup
reasonnable download settings, such as:
Set Options / Flow Control / Number of connections: 2
Set Options / Limits / Max transfer rate: 10000
Don't override robots and/or user-agent without setting up
reasonnable bandwidth limits!
| |