HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Slow .html indexing
Author: Paul
Date: 09/07/2009 20:12
> what is the staring url, what does robots.txt say,
> what is the url with the actual content, what does
> the log file say about the image url. did you set
> the log file to debug.
> You don't give any information, so you can't get any
> answers.

Sorry about that, went to look at the log after it finally completed the
mirror and realized what happened. httrack apparently mirrors the site level
by level and due to the speed of the html connection, had yet to reach that
depth when I was looking at it. However, before it can reach that depth, it
ran into the default 100,000 links limit and "completed".

Resuming with a higher link limit appears to make it work fine with regards to
doc/pics/zip since I can see it downloading some at this point. 

I'm back to the -%N0 problem though. It's still only using one connection for
html. Overall transfer speed has gone up to reasonable levels once the actual
content mirroring started but html's still crawling.

Is the default simultaneous connections max 2? If so, then I could see why it
seems like -%N0 has no effect. 
Reply Create subthread

All articles

Subject Author Date
Slow .html indexing

09/05/2009 07:01
Re: Slow .html indexing

09/05/2009 14:22
Re: Slow .html indexing

09/05/2009 16:30
Re: Slow .html indexing

09/05/2009 19:14
Re: Slow .html indexing

09/07/2009 13:18
Re: Slow .html indexing

09/07/2009 14:54
Re: Slow .html indexing

09/07/2009 20:12
Re: Slow .html indexing

09/07/2009 21:07
Re: Slow .html indexing

09/07/2009 21:08
Re: Slow .html indexing

09/08/2009 17:25
Re: Slow .html indexing

04/15/2010 03:26


Created with FORUM 2.0.11