| > Does the spider consider all the 403.* error as fatal
> ones, like the 404 ones?
As fatal errors ; 403 (unauthorized) is used to indicate
that the page access was refused (auhtentication)
> I am having problem downloading some pages from a website
> with high traffic, the ones you get 'HTTP 403.9 - Access
> Forbidden: Too many users are connected
> Internet Information Services' from.
Argh.. this is a problem indeed, because this will
generate "random holes" in the website you are crawling..
try not to update the website during working hours ; the
server will maybe be slightly less overloaded.
| |