Hi, I think you helped me as well with your question.  Perhaps you should view
the warning and error messages.  I would think this is similar to what I've
been studying.  Your post may even be a bit abnormal considering I obtained
similar results and posted here only a few minutes before... weird!

Although I know very little about IT specifics, I think I'm getting the
basics+.  Upon my completion, or should I say failure of a scan, I checked
tosee if I was BL'd at any of the well-known sites.  Nope! This could infact
take a few hours for me to query the dnsbl.httpbl.org NS for a poor rating,
but I think it's more likely that I've attempted to spoof my user-agent.

This spoof may trigger a deny to occur if the UA was BL'd!  What if we tried
to decrease the threads and increase the TTL/WAIT time, while using a browser
UA.

It may also help increase our chances of success if we can find a proxy that
will work for our attempts.  But can you proxy this crawler even if you could
mask your IP address via proxy?
This is a good RE: to get us both started I think.

P.S. It's also funny that I was crawling a media site as well!  Perhaps
coincidence... But I'll find the cash_icon.jpg or 138117.jpg first! Muhahahaha
(diabolical-laugh)  

~danday3953