| Thanks for your help
When robots.txt is in the HTML root folder, Winhttrack reads
it, mentions it in the log file but doesn't follow the rules.
(DocumentRoot "d:/site" with in "d:/site/robots.txt" // URL
<http://127.0.0.1/site/index.php>)
My other problem is I have a few local sites with different
robots.txt but only one HTML root.
I couldn't find any solution with the alias section, I must
be misunderstanding something.
| |