HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt vs ROBOT META tag
Author: David
Date: 02/28/2007 18:59
Is there any way for HTTrack to differentiate between restrictions imposed by a
robots.txt file and those imposed by a META tag? I'd like to be able to
automatically ignore restrictions imposed through META tags while still
following the robots.txt protocol.

All articles

Subject Author Date
robots.txt vs ROBOT META tag

02/28/2007 18:59


Created with FORUM 2.0.11