> Thank you, then this was the reason why i was not
> able to crawl it completly.

Had you bothered to look at the bottom of the log file, you would have seen
the reason.