HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Robot.txt what is it about
Author: BT
Date: 04/18/2007 03:45
Robots.txt is a file that admins use to keep automatic spiders from abusing
their bandwidth as well as to keep people from getting certain content
especially if it is proprietary. You can enable httrack to ignore the
robots.txt file and you should be able to dl the files you want. Cheers!
Reply Create subthread

All articles

Subject Author Date
Robot.txt what is it about

04/08/2007 01:26
Re: Robot.txt what is it about

04/18/2007 03:45


Created with FORUM 2.0.11