| Subject: Verifying robots.txt... |
Author: Piti |
Date: 10/30/2003 13:28 |
| | Hi,
I have one question. I am using httrack a list of urls
disburdens, but he does not verify the file robots.txt of
these urls. Is there any form in order that he verifies it
when I spend to him a file with the urls?
Thank you very much in advance
:) | |
|
|
|
|
f
Created with FORUM 2.0.11