HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Need help
Author: D_A
Date: 01/28/2004 14:52
 
You can disable the reading of robots.txt in
Set Options / Spider / then select no robots.txt rules.

I think it may not safe to do so without filtering in the
scan rules.
add
-*/_vti*
-*/_private*
etc
 in order to download what's necessary and avoid overloading
the server (you may need /images/ and /_fpclass/ )
also limit the number of connections and bandwidth usage
(otherwise you may be blocked if the webmaster is upset by
brutal mirroring of his site)
 
Reply Create subthread


All articles

Subject Author Date
Need help

01/28/2004 04:39
Re: Need help

01/28/2004 14:52
Re: Need help

01/29/2004 03:19




2

Created with FORUM 2.0.11