HTTrack Website Copier
Free software offline browser - FORUM
Subject: Robots.txt and metatag
Author: Constantine
Date: 06/22/2007 00:08

I read your FAQ, search (e.g. "robot") and read forum.

Debian, httrack command-line mode, use urllist and ruleslist files. Robots
rules: --robots=0 (or --robots=N0, or sN0, or s0).

I load site (blog), to urllist add

httrack load <>, read this file,,, and

OK, paste link <>
Httrack load link.html, load robots.txt ,,, and finish.

In robots.txt me read:
User-Agent: *
Disallow: /


1) Other blog this domain very good loading (robots.txt read only "User-Agent:
*") to httrack, very good.
2) Use <> from browser - good loaded and viewed.

Uses specific user-agent browser? or other method?

All articles

Subject Author Date
Robots.txt and metatag

06/22/2007 00:08
Re: Robots.txt and metatag

07/03/2007 11:17
Re: Robots.txt and metatag

07/03/2007 18:22


Created with FORUM 2.0.11