I tried turning off robots.txt in spider and also adding some of the urls as
inclusion links +*webdata.soc.hawaii.edu/* (for example).

Any other suggestions?  i'm running it with these new settings now.