HTTrack Website Copier
Free software offline browser - FORUM
Subject: Verifying robots.txt...
Author: Piti
Date: 10/30/2003 13:28

I have one question. I am using httrack a list of urls 
disburdens, but he does not verify the file robots.txt of 
these urls. Is there any form in order that he verifies it 
when I spend to him a file with the urls?
Thank you very much in advance

All articles

Subject Author Date
Verifying robots.txt...

10/30/2003 13:28
Re: Verifying robots.txt...

11/01/2003 11:09


Created with FORUM 2.0.11