Is there a way to allow it to scan the entire site but only save files with a
certain format?
I tried doing this:
-*
+users/*/
I'm trying to save all the files in the users directory BUT the index page
(the start location) does NOT have a list of all the users. It should crawl
throughout all the pages and save only when it hits a page with that url
format. Is it possible to do this? |