HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: avoiding BL's
Author: danday3953
Date: 12/02/2008 10:09
 
WOW!!!

I think I was just blacklisted!  I attempted to crawl the site, and everything
just stopped.  I was returned to HTTRACK Application, and no index was built,
and everything seemed to be working at first?
Could this be because I changed the user-agent to represent myself as
GoogleBot while crawling?  Perhaps I should try a browser user-agent, and
decrease the threads to a slower level?
Any ideas guys???
 
Reply Create subthread


All articles

Subject Author Date
avoiding BL's 12/02/2008 09:41
Re: avoiding BL's 12/02/2008 10:09
Re: avoiding BL's 12/02/2008 10:15
Re: avoiding BL's 12/03/2008 15:10




5

Created with FORUM 2.0.11