| Subject: Does httrack respect the robots.txt? |
Author: bot7 |
Date: 01/17/2009 12:39 |
| | Like this:
User-agent: HTTrack
Disallow: /media
or is it possible to override the httrack user-agent with Mozilla 5.0 bla bla
... and fake it? | |
|
|
|
|
0
Created with FORUM 2.0.11