| > There is this site I am downloading which was made using
PHP
> nuke.
> Based on the robots there, admin folder can not be
> downloaded. and few other folders.
> What is the command line to download EVERYTHING, and what
> the options do I have.
*FIRST* ensure that you can download this site (ask the
administrator permission!), and how (during non-working
hours? using a bandwidth limit (see httrack options))
Then, you can safely disable robots.txt:
Set Options / Spider / Spider : do not follow robots.txt
| |