HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: "noindex,nofollow" and relative links
Author: D_A
Date: 12/03/2003 11:03
As for css (when <META NAME='robots'
CONTENT='noindex,nofollow'> is found) and javascript linked
files, I meant http:// links, thus the pages cannot be
properly browsed offline.

For javascript, php extensions are not modified even if the
html file has been downloaded and thus there is no linking
with the javascript routines and the local mirror.

> if robots.txt are enabled, "nofollow" in meta will force
httrack to skip ALL links in the page.

I understand that the links to other pages mustn't be
followed and you may be right in skipping all the links in
the page,
but if you have 
<link rel="SHORTCUT ICON"
<link rel="stylesheet"
href= type="text/css">
<style type="text/css">
@import url(;
<script src=

(same problem with java applets, Flash etc...)

THe page (not to be followed) won't be correctly displayed
Reply Create subthread

All articles

Subject Author Date
"noindex,nofollow" and relative links

11/26/2003 14:32
Re: "noindex,nofollow" and relative links

11/26/2003 17:10
Re: "noindex,nofollow" and relative links

12/02/2003 20:40
Re: "noindex,nofollow" and relative links

12/03/2003 11:03


Created with FORUM 2.0.11