Is there any way for HTTrack to differentiate between restrictions imposed by a
robots.txt file and those imposed by a META tag? I'd like to be able to
automatically ignore restrictions imposed through META tags while still
following the robots.txt protocol.