| Subject: Re: How to completely ignore 'robots.txt'? |
Author: The Dro |
Date: 05/23/2015 22:13 |
| | You would basically just need to use the command to ignore robots.txt. If
you're using command line to run HTTRACK just type in:
httrack <http://website.com/directory-you-want-to-rip> -O "/path-to-save-files"
-%v -s0
the "-s0" parameter that you pass should block out the application from
listening to robots.txt | |
|
|
|
|
3
Created with FORUM 2.0.11