By default, HTTrack obeys robots.txt rules.
You have to explicitly ask if you don't want it (unless you
saved the option as default).
Is the problem with a local site or with a URL?Following robots.txt will not
avoid falling in a spider trap.
Can you give us more precisions? |