HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: robots.txt
Author: Mike
Date: 01/19/2013 13:32
Hi Guys again,
I found the option I needed. The path of option is "Set Options", "tab Spider"
and select the option in the right listbox of text "Spider".
Insted of using listbox, I suggest to change this listbox with 3 radiobuttons
to get more visibility
1. no robots.txt rules
2. robots.txt except wizard
3. follow robots.txt rules

version 3.46

Thans again.
Reply Create subthread

All articles

Subject Author Date

01/19/2013 12:56
Re: robots.txt

01/19/2013 13:32
Re: robots.txt

01/19/2013 13:42


Created with FORUM 2.0.11