HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to increase download speed?
Author: Sergei Kulagin
Date: 12/09/2020 20:13
 
> I've seen in the forum, the last post is about
> informing the members about the closing of that
> website, but I wasn't able to even open that page in
> the web-browser. Some sort of overload information
> was displayed. I did not spent much time on the
> website,  tried to reload that webpage, and finally
> the website crashed with the following error
> response: 
> 
> "Warning: mysql_connect(): Too many connections in
> /var/www/html/cms_admin/db_mysql.php on line 30"

Yes, that happened back then because I started crawling with wget and it was
parsing one page in 1 second. My guess is that other people did the same
trying to archive with different programs. wget was doing it at 200-500 kb
speed. Which is still slow compared to browsers like Google Chrome, which do
exactly the same job saving everything locally and do load pages much
quicker.

The problem with wget is that there's no way to make it ignore links and not
follow them. I wrote this command:

> wget --continue --wait=1 --waitretry=10 -4 --recursive --page-requisites
--adjust-extension --no-clobber --convert-links --random-wait -e robots=off
--force-directories --load-cookies tekkenzaibatsu.com_cookies.txt
--user-agent='Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101
Firefox/30.0' <http://www.tekkenzaibatsu.com/> -R
"*/editpost.php?action=editpost&postid=*,*/newreply.php?action=newreply&postid=*,*/showthread.php?postid=*#post*,*/online.php/*,*/search.php?s=&action=showresults&searchid=*,*/search.php?s=*,*/private.php?action=newmessage&userid=*,*/member2.php?action=addlist&userlist=buddy&userid=*,*/newreply.php?action=newreply&threadid=*,*/newthread.php?action=newthread&forumid=*"

But it still follows all the patterns in the -R list(it's the reject list),
downloads all the pages and saves it on the disk. Here are all the pages that
follow the */editpost.php?action=editpost&postid=* pattern which I set in the
-R list: <https://i.imgur.com/KCcVCLO.png>

Which makes it unusable for the forum because there are all these buttons:
<https://i.imgur.com/sXfBUmG.png> . Which result in wget trying to download
millions of pages that I don't need to archive: edit post, quote post, new
thread, new reply and all other.
 
Reply Create subthread


All articles

Subject Author Date
How to increase download speed?

12/05/2020 14:08
Re: How to increase download speed?

12/05/2020 15:11
Re: How to increase download speed?

12/05/2020 21:31
Re: How to increase download speed?

12/06/2020 19:51
Re: How to increase download speed?

12/07/2020 02:07
Re: How to increase download speed?

12/07/2020 04:00
Re: How to increase download speed?

12/09/2020 20:13
Re: How to increase download speed?

12/09/2020 21:29
Re: How to increase download speed?

12/14/2020 02:31
Re: How to increase download speed?

12/14/2020 02:57
Re: How to increase download speed?

12/14/2020 02:58
Re: How to increase download speed?

12/14/2020 03:00
Re: How to increase download speed?

12/16/2020 10:59
Re: Digital Marketing Experts in India

01/11/2021 17:40
Re: How to increase download speed?

01/11/2021 17:41
Re: How to increase download speed?

01/11/2021 17:41




4

Created with FORUM 2.0.11