| I'm trying to create a static HTML archive of my own forum here:
<http://theknightsofsumeria.com>
It's been running for 9 hours, and it's downloaded over 93,000 files. There's
probably only 150 posts on the whole forum! I've filtered out as many pages as
I can using this list;
-*delete_cookies*
-*faq*
-*login*
-*post*
-*print*
-*privacy*
-*quote*
-*register*
-*reply*
-*sendpassword*
-*terms*
-*#top*
-*viewprofile*
Which in theory should just leave the index pages, subforums, each page, and
each post to be archived. There are no external links being downloaded, just
this domain. I can't understand how that can be 93,000 files and counting!?
The first time I tried, I had allowed guests to see attached images, and I
stopped it at 30GB, as I could see that it was downloading many multiple
copies of each image and wasn't stopping. I thought I'd disable attached
images, as a test to see how many HTML-only pages it was trying to download. | |