HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Does httrack respect the robots.txt?
Author: William Roeder
Date: 01/17/2009 14:17
 
> User-agent: HTTrack
> Disallow: /media
> 
> or is it possible to override the httrack user-agent
> with Mozilla 5.0 bla bla ... and fake it?
There are options to override both the user agent (options->BrowserID) and to
ignore some or all of robots.txt (options->spider)

Users overriding robots.txt are advised that not abuse the site
<http://www.httrack.com/html/abuse.html>
 
Reply Create subthread


All articles

Subject Author Date
Does httrack respect the robots.txt?

01/17/2009 12:39
Re: Does httrack respect the robots.txt?

01/17/2009 14:17




8

Created with FORUM 2.0.11