| > without trouble. However, there are many pages
> missing, which I can access if I manually type the
Always post the command line used (log file line two)
Did you read the log file about the robots.txt restrictions:
User-agent: *
Disallow: /cgi-bin/viewvc/
Disallow: /~mdalmau/prototypes/
Disallow: /variations/reserves/
Disallow: /dml/pilot/
Disallow: /internal/
Disallow: /cgi-bin/var/
Disallow: /cgi-bin/dido/
Disallow: /collections/dido/classes/
Disallow: /variations/scores/aaz2552/
Disallow: /variations/scores/abf1877/
Disallow: /variations/scores/abg6909/
Disallow: /variations/scores/abm0994/
:
and override that with options -> spider -> spider = ignore
or simply add a filter +*/scores/* | |