Don't follow the robot rules:

in "Set Options..." go to "Spider," select "no robots.txt rules."