> If I use a scan rule like so:
> +*filename_a*
> -A's top level domain
> 
> nullifies the first rule by not allowing download of any 
> links on the top level domain in the second rule.

The order you list the scan rules will affect behavior.  
Try reversing the order of the two rules.
See this for more info and examples:
<http://www.httrack.com/HelpHtml/filters.html>