| Subject: Re: avoiding BL's |
Author: danday3953 |
Date: 12/02/2008 10:09 |
| | WOW!!!
I think I was just blacklisted! I attempted to crawl the site, and everything
just stopped. I was returned to HTTRACK Application, and no index was built,
and everything seemed to be working at first?
Could this be because I changed the user-agent to represent myself as
GoogleBot while crawling? Perhaps I should try a browser user-agent, and
decrease the threads to a slower level?
Any ideas guys??? | |
|
|
|
|
5
Created with FORUM 2.0.11