| Subject: Scanning the entire site but saving specific files |
Author: Bahamut |
Date: 03/21/2008 03:29 |
| | Is there a way to allow it to scan the entire site but only save files with a
certain format?
I tried doing this:
-*
+users/*/
I'm trying to save all the files in the users directory BUT the index page
(the start location) does NOT have a list of all the users. It should crawl
throughout all the pages and save only when it hits a page with that url
format. Is it possible to do this? | |
|
|
|
|
5
Created with FORUM 2.0.11