HTTrack Website Copier
Free software offline browser - FORUM
Subject: robots.txt
Author: Mike
Date: 01/19/2013 12:56
 
Hi Guys,
I noticed HTTrack obeys robots.txt but I have a doubt. 
I need HTTrack doens't obey this file to mirror a folder included in
robots.txt and I tried the option "include links" in Options, Scan Rules
without success.
It's possible configure HTTtrack to skip robots.txt?
Thanks in advance
Mike 
 
Reply


All articles

Subject Author Date
robots.txt

01/19/2013 12:56
Re: robots.txt

01/19/2013 13:32
Re: robots.txt

01/19/2013 13:42




e

Created with FORUM 2.0.11