| WOW!!!
I think I was just blacklisted! I attempted to crawl the site, and everything
just stopped. I was returned to HTTRACK Application, and no index was built,
and everything seemed to be working at first?
Could this be because I changed the user-agent to represent myself as
GoogleBot while crawling? Perhaps I should try a browser user-agent, and
decrease the threads to a slower level?
Any ideas guys??? | |