| I want to spider this site:
Using this starting url:
<https://foo:bar@www.instructables.com/recent>
I added a scan rule to ignore zip files:
-*.zip
But the spider keeps trying to go all over the world to other hosts. It's
trying to also spider tumber, twitter, google, everything else under the sun.
How to keep it focused only on the host I asked for? | |