HTTrack Website Copier
Free software offline browser - FORUM
Subject: Workaround for www.foo.com <-> foo.com and //
Author: Xavier Roche
Date: 03/23/2003 16:24
 
Two problems are often encountered with bogus sites:

1- loop problems with multiple // in urls; such as:
www.foo.com/bar/foobar
www.foo.com/bar//foobar
www.foo.com/bar///foobar
..

2- duplicate site copy; such as
www.foo.com/bar/foobar
foo.com/bar/foobar
www-45.foo.com/bar/foobar

3- multidomain duplicate site copy; such as
www.foo.com/bar/foobar
www.foo.net/bar/foobar
www.foo.fr/bar/foobar

I have implemented a workaround for 1. and 2. (maybe 3. in 
the future when I have some time) in the current 3.30-alpha-
05 ; which detects such cases, and merge "similar" links 
together:

These links will be merged with the "urlhack" option:
www.foo.com/bar.html
foo.com/bar.html
www-42.foo.com/bar.html

And these links too:
www.foo.com/bar.html
www.foo.com//bar.html
www.foo.com///bar.html

If you have problems similar to 1. and 2., please test the 
current alpha-05 and give me some feedback, aspecially:
- is the workaround working?- is the workaround causing "bad side effects" on
other 
links/sites?
 
Reply


All articles

Subject Author Date
Workaround for www.foo.com <-> foo.com and //

03/23/2003 16:24




4

Created with FORUM 2.0.11