Set options > Spider > Spider: no robots.txt rules
Zaid
Diaz
William Roeder
Created with FORUM 2.0.11