| I should also add... I just noticed this note from the log file. It seems the
robot.txt file for the site has already removed most of the "extra" links
(like member info) from being mirrored.
"Note: due to www.sawmillcreek.org remote robots.txt rules, links begining
with these path will be forbidden: /admincp/, /archive/, /attachments/,
/chat/, /cpstyles/, /install/, /modcp/, /phpbb/, /subscriptions/, /usage/,
/chat2.php, /chat.php, /cron.php, /editpost.php, /external.php,
/IRCApplet.class, /irc.cab, /irc.jar, /Jicra.class, /joinrequests.php,
/memberlist.php, /member.php, /misc.php, /moderator.php, /Net.class,
/newattachment.php, /newreply.php, /newthread.php, /postings.php,
/Protocol.class, /report.php, /reputation.php, /search.php, /sendmessage.php,
/subscription.php, /subscriptions.php, /threadrate.php, /usercp.php,
/usernote.php" | |