> I have some trouble getting httrack to recognize the
> robots meta tag. 
> 
> From what i understand from the documentation is
> that the command line tool by default reads the
> robots.txt and the appropiate tags.
> 
> But still sites are indexed.

I don't know about the meta tag but httrack does recognize and follows
robots.txt by default.

That setting can be overridden by an option.

<http://www.httrack.com/html/abuse.html#WEBMASTERS>