HTTrack Website Copier
Free software offline browser - FORUM
Subject: Too many robots.txt
Author: Jacruth
Date: 06/29/2008 16:51
 
Hi, i'm downloading a webpage but Httracks get too many robots.txt files and
downloading the web takes a lot of time. I think that a photo could help to
understand me:
<http://img176.imageshack.us/my.php?image=35249545sm6.jpg>

How could I avoid to download that large amount of robots.txt?
 
Reply


All articles

Subject Author Date
Too many robots.txt

06/29/2008 16:51
Re: Too many robots.txt

06/29/2008 20:26
Re: Too many robots.txt

07/23/2008 11:41
Re: Too many robots.txt

07/28/2008 04:24
Re: Too many robots.txt

04/18/2009 17:40




a

Created with FORUM 2.0.11