| > set options/Spider/Spider select there,
Thx. I recognise it now, I was looking for cgi-bin, images
and looked right past it.
> but don't forget to limit bandwith usage, number of
connections, max tranfer rate
Is there recommended settings?
> and try to limit the downloads to what you need by
setting scan rules in order to be "gentle" with the site
server. robots.txt are there for good reasons
Still trying to take in the entire help folder. Can you
make suggestions for setting scan rules pls? | |