Is there a way to get this or any program to crawl the pages as if it was
someone viewing it in a browser, rather than getting directory listings and
otherwise identifying the connection as an attempt to mirror the site?