HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Thousands of files and I don't understand why
Author: William Roeder
Date: 07/02/2012 14:25
 
> Directory Listing:
> <http://www.freeimagehosting.net/3afnf>
Shows lots of search pages. 

The robots.txt said not to get them:
User-agent: *
Disallow: /search
Allow: /

But you overrode robots: -s0
So you got them.
 
Reply Create subthread


All articles

Subject Author Date
Thousands of files and I don't understand why

07/02/2012 00:02
Re: Thousands of files and I don't understand why

07/02/2012 14:25
Re: Thousands of files and I don't understand why

07/03/2012 03:17




8

Created with FORUM 2.0.11