| Hello,
I have an interesting case.
I would like to scan an entire website, but download only folders that contain
"xyz" for example.
The problem is links with these folders are not linked from the front page so
if I do a rule like this
-* +mysite.com/*/xyz/
It wont download anything as none of the front page links contain this
pattern. A few more links away this rule would work. | |