HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Too many URLs
Author: Christian
Date: 04/18/2015 09:41
 
I know this is a very old thread and despite having been using HTTrack for
almost 7 years, I hit the frustrating "Panic: too many links" error that
prevents pages from being mirrored all the way this week (April 2015)!

***I am going to post here the fix to this problem, easy step-by-step for
n00bs and experts alike. :) ***

I confess, I almost went crazy with frustration. HTTrack works so dang well, I
NEVER look at the error log! So for the first time I hit the "Too many URLs,
giving up" error. 

I tried everything: adding that line (example: -#L1000000) to "Rules" to no
avail. Tried the line in the "add url" and "web addresses" to no avail. I
tried deleting the dang .tmp files, not work either. I spent hours creating +*
Rules for each major link artery... nothing. 

Well... After banging my head against the wall over 3 days, I checked error
logs carefully. Ta-da!

So here is the easy way to fix this. The whole "use #L option for more links
(example: -#L1000000)" was not making sense. 

So here is what I did:

1) Do NOT delete hts-cache or .tmp files like I did! Otherwise, the download
will take a long time again from scratch. Go back to the project and open the
existing project. Select action: "Continue interrupted download";

2. Click on "Set Options" --> Limits (tab) --> "Maximum Number of links"
(bottom field) --> Just go for highest available of threads or write
999999999.

But be warned! You need to constantly check the In Progress/action download
screen to be sure you are not copying the entire Internet! lol  One way to do
that is to put "0" under the "Maximum external depth" field (Limits tab) and
checkmark "No external pages" under "Build" tab. I know this can cause you to
miss some files, but that can be fixed manually under "Rules".

Hope this helps! :) Be blessed mirroring and thanks the crew at Httrack for
this awesome project! :)
Christian
 
Reply Create subthread


All articles

Subject Author Date
Too many URLs

07/21/2005 21:56
Re: Too many URLs

07/22/2005 01:22
Re: Too many URLs

04/18/2015 09:41




b

Created with FORUM 2.0.11