| > Unlike most of you, I am trying to delimit rather
> than limit my spidering
:p
> -C0N1003s0K%e9999r9999zI0b1nBe
One big advice: do not use depth settings if possible (e
and r options), but use scan rules instead. This is much
more reliable, and should not cause as many problems as
depth settings can.
Example 1: download two sites with cross links, but nothing
more:
-* +www.example.com/index.html +www.anotherexample.com/*
Example 2: same as #1, but download only /articles/ section
in site 2
-* +www.example.com/index.html
+www.anotherexample.com/articles/*
| |