If you are using the Windows GUI, go to the options for your project, go to the
"Spider" tab, find the second drop-down box and change "Follow robots.txt
rules" to "no robots.txt rules"