HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Robots.txt
Author: Dan_A
Date: 04/20/2005 19:26
When you set up a mirror using the GUI, on the page where
you set the URL, click the button Set Options... | Spider
and select the option from the drop-down list.
Reply Create subthread

All articles

Subject Author Date
Robots.txt 04/20/2005 15:09
Re: Robots.txt 04/20/2005 19:26
Re: Robots.txt 04/23/2005 11:01


Created with FORUM 2.0.11