| Subject: Re: How to download ALL the files in directory |
Author: Xavier Roche |
Date: 04/06/2003 19:02 |
| | > There is this site I am downloading which was made using
PHP
> nuke.
> Based on the robots there, admin folder can not be
> downloaded. and few other folders.
> What is the command line to download EVERYTHING, and what
> the options do I have.
*FIRST* ensure that you can download this site (ask the
administrator permission!), and how (during non-working
hours? using a bandwidth limit (see httrack options))
Then, you can safely disable robots.txt:
Set Options / Spider / Spider : do not follow robots.txt
| |
|
|
|
|
8
Created with FORUM 2.0.11