You have to set "no robots.txt" rules to do that, you may have to set a browser
ID.
Don't overload the server.