HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Repeated scanning same page.
Author: Xavier Roche
Date: 04/10/2002 19:01
 
> I an trying to scan a site hosted on IBM/Lotus 
Domino.
> 
> Most of the pages will have a link back to home 
page, 
> and the home page is a frame page contain three 
> dynamically generated pages, one of it just a 
counter.
> 
> When I download this page, it will repeatly scan 
this 
> homepage, and all threads were waiting for that.  So 
> if this site have 10,000 pages, this pages might 
have 
> download and scan 10,000 times.  

If the homepage link can be skipped, use filters:
-www.foo.com/bar/homepage.cgi*

If you have to get this homepage, except when the 
counter variable is used (I don't know exacly, but 
this may be something like that):
-www.foo.com/bar/homepage.cgi?*counter=*

If you have to get the homepage, only for a specific 
counter value (example: 1) :
-www.foo.com/bar/homepage.cgi?*counter=*
+www.foo.com/bar/homepage.cgi?*counter=1&*

Note: if the homepage is the #1 page, given in the URL 
list, you can safely use the first filter which 
exclude it, as all URLs given as "starting pages" are 
taken whatever the filters are.

 
Reply Create subthread


All articles

Subject Author Date
Repeated scanning same page.

04/10/2002 18:25
Re: Repeated scanning same page.

04/10/2002 19:01
Re: Repeated scanning same page.

04/10/2002 20:02
Re: Repeated scanning same page.

04/10/2002 20:57
Re: Repeated scanning same page.

04/10/2002 21:10
Re: Repeated scanning same page.

04/12/2002 01:29




6

Created with FORUM 2.0.11