This problem used to happen to me when i used to leave all settings at default.
I now use "no robots.txt", and change the browser useragent string (Browser
ID) to a Mozilla one, and every website i try has worked so far.