HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: d/ling images (site doesn't show, URLs exist)
Author: William Roeder
Date: 12/14/2011 14:57
 
> without trouble. However, there are many pages
> missing, which I can access if I manually type the
Always post the command line used (log file line two)
Did you read the log file about the robots.txt restrictions:
User-agent: *
Disallow: /cgi-bin/viewvc/
Disallow: /~mdalmau/prototypes/
Disallow: /variations/reserves/
Disallow: /dml/pilot/
Disallow: /internal/
Disallow: /cgi-bin/var/
Disallow: /cgi-bin/dido/
Disallow: /collections/dido/classes/
Disallow: /variations/scores/aaz2552/
Disallow: /variations/scores/abf1877/
Disallow: /variations/scores/abg6909/
Disallow: /variations/scores/abm0994/
:
and override that with options -> spider -> spider = ignore
or simply add a filter +*/scores/*
 
Reply Create subthread


All articles

Subject Author Date
d/ling images (site doesn't show, URLs exist)

12/14/2011 00:58
Re: d/ling images (site doesn't show, URLs exist)

12/14/2011 14:57




8

Created with FORUM 2.0.11