HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Robots problem
Author: Leto
Date: 04/19/2002 04:47
If you are using the Windows GUI, go to the options for your project, go to the
"Spider" tab, find the second drop-down box and change "Follow robots.txt
rules" to "no robots.txt rules"
Reply Create subthread

All articles

Subject Author Date
Robots problem

04/19/2002 04:38
Re: Robots problem

04/19/2002 04:47


Created with FORUM 2.0.11