But the robots.txt has some shortcomings that can be overcome with the robots
metatag. 

For example there are Typo3 installations that don't have a discernible URL
Pattern for pages offered for pretty printing a page, so HTTrack indexes these
sites (although the robots tag is set correctly)

Hopefully the developer or someone with knowledge about the inner workings can
shed some light on this?