HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Grabing all pages of a SUBforum
Author: Dan
Date: 12/16/2008 18:55
 
I should also add... I just noticed this note from the log file.  It seems the
robot.txt file for the site has already removed most of the "extra" links 
(like member info) from being mirrored.

"Note: due to www.sawmillcreek.org remote robots.txt rules, links begining
with these path will be forbidden: /admincp/, /archive/, /attachments/,
/chat/, /cpstyles/, /install/, /modcp/, /phpbb/, /subscriptions/, /usage/,
/chat2.php, /chat.php, /cron.php, /editpost.php, /external.php,
/IRCApplet.class, /irc.cab, /irc.jar, /Jicra.class, /joinrequests.php,
/memberlist.php, /member.php, /misc.php, /moderator.php, /Net.class,
/newattachment.php, /newreply.php, /newthread.php, /postings.php,
/Protocol.class, /report.php, /reputation.php, /search.php, /sendmessage.php,
/subscription.php, /subscriptions.php, /threadrate.php, /usercp.php,
/usernote.php"
 
Reply Create subthread


All articles

Subject Author Date
Grabing all pages of a SUBforum 12/10/2008 21:19
Re: Grabing all pages of a SUBforum 12/11/2008 16:38
Re: Grabing all pages of a SUBforum 12/11/2008 17:45
Re: Grabing all pages of a SUBforum 12/11/2008 19:06
Re: Grabing all pages of a SUBforum 12/12/2008 13:59
Re: Grabing all pages of a SUBforum 12/12/2008 20:17
Re: Grabing all pages of a SUBforum 12/16/2008 18:51
Re: Grabing all pages of a SUBforum 12/16/2008 18:55
Re: Grabing all pages of a SUBforum 12/18/2008 00:18




2

Created with FORUM 2.0.11