HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Verifying robots.txt...
Author: Xavier Roche
Date: 11/01/2003 11:09
 
> I have one question. I am using httrack a list of urls 
> disburdens, but he does not verify the file robots.txt of 
> these urls.

robots.txt, is not disabled in the commandline of in the 
GUI, are normally checked automatically for html pages 
(separated files such as external gif files but without any 
other html data do not trigger robots.txt)
 
Reply Create subthread


All articles

Subject Author Date
Verifying robots.txt...

10/30/2003 13:28
Re: Verifying robots.txt...

11/01/2003 11:09




8

Created with FORUM 2.0.11