HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to HTTR these site
Author: William Roeder
Date: 10/25/2008 15:32
 
http://www.diaryproject.com/robots.txt says
User-agent: *
Disallow:

# /robots.txt file for <http://www.developerdotstar.com/>
User-agent: *
Disallow: /h0neyp0t/i_would_skip_this_link.php
Disallow: /printable
...

set options no robots.txt
 
Reply Create subthread


All articles

Subject Author Date
How to HTTR these site

10/25/2008 15:04
Re: How to HTTR these site

10/25/2008 15:32




3

Created with FORUM 2.0.11