I have 2 suggestions that I think would be great and easy to
impliment.

1)  Make it possible to specify cookies and their values. 
So if you want to spider a forum that requires to be logged
in, you can just make it send the same cookie.  Most modern
browsers let you see the info on cookies so it would just be
a copy and paste thing for the user id and password hash,
etc.  While spidering it could also accept cookies and store
then to send as well.

2) Make it possible to specify a delay between page
requests, since at this moment it just goes as fast as it
can, which is not good for spidering online sites since it
bogs down the server.  It would be nice to be able to
specify a number of seconds.  So 0 for no delay and anything
else for a delay, since going at high speed is good for
local intranets.  

I hope these will be taken into consideration, thanks!