HTTrack Website Copier
Free software offline browser - FORUM
Subject: Using Options to disable remote robots.txt ruls
Author: Stan Bucci
Date: 09/17/2007 00:11
 
I keep getting the following error; tried a number of configurations but can't
seem to be able to get the excluded information. Any assistance would be
appreciated.

Note: due to www.jewishencyclopedia.com remote robots.txt rules, links
begining with these path will be forbidden: /images/, /pages/, /pages_thumb/,
/volfp/, /volume1/, /volume2/, /volume3/, /volume4/, /volume5/, /volume6/,
/volume7/, /volume8/, /volume9/, /volume10/, /volume11/, /volume12/,
/WEB-INF/, /ratearticle.jsp, /post.jsp, /view_friendly.jsp, /mp_art_list.jsp,
/rv_art_list.jsp, /hr_art_list.jsp (see in the options to disable this)

Thanks,

Stan
 
Reply


All articles

Subject Author Date
Using Options to disable remote robots.txt ruls

09/17/2007 00:11
Re: Using Options to disable remote robots.txt rul

09/17/2007 04:28
Re: Using Options to disable remote robots.txt rul

09/24/2007 01:51
Re: Using Options to disable remote robots.txt rul

04/20/2016 18:57
Re: Using Options to disable remote robots.txt ruls

11/26/2017 14:03
Re: Using Options to disable remote robots.txt ruls

05/08/2020 20:43




8

Created with FORUM 2.0.11