| Subject: Re: Does httrack respect the robots.txt? |
Author: William Roeder |
Date: 01/17/2009 14:17 |
| | > User-agent: HTTrack
> Disallow: /media
>
> or is it possible to override the httrack user-agent
> with Mozilla 5.0 bla bla ... and fake it?
There are options to override both the user agent (options->BrowserID) and to
ignore some or all of robots.txt (options->spider)
Users overriding robots.txt are advised that not abuse the site
<http://www.httrack.com/html/abuse.html> | |
|
|
|
|
8
Created with FORUM 2.0.11