> Can't I just tell
> the rule to scan everything down from
> www.site.com/dir/ and get the folder/files that have
> the last subdirectory in the URL named "subdir"?
Unless /dir contains direct links to /dir/X/subdir you MUST let it spider X to
get the links to X/subdir

You can filter to only get the html and not other files except in subdir: -*
+*/subdir/* +mime:text/html