> I can't get this site
> <http://www.chicagospankingreview.org/homepage/home_p>
> age.html
> Also - how can I get the spider to keep a copy of
> robot.txt for me to look at?http://www.chicagospankingreview.org/robots.txt
does not exist
The problem I see is most of the links are at
/doc...
/gallery...
etc
but you are starting at /homepage/...
By default httrack does not go up. Either override with a filter +site/* or
experts page allow either direction |