Hi guys, My site is behind a DSL line and just got hammered hard by an HTTrack user. I notice that HTTrack does a GET on robots.txt right as it starts crawling. Can you please make HTTrack obey the value specified in Crawl-delay? Thanks
Aaron
Created with FORUM 2.0.11