HTTrack Website Copier
Free software offline browser - FORUM
Subject: Single-threaded communications with slow servers
Author: Vyacheslav
Date: 08/01/2021 00:41
Hello. I am trying to download quite large forum topic, that has many dead or
"slow-server" image links.
What HTTrack does, for example:
1) Scans HTML page, finds 5 images, and puts links in queue.
2) Starts downloading first image, but server is slow, socket is in "connect"
3) Connection took ~10 sec, connection established, request sent. Socket now
in "receive" state. Server still slow, we are waiting for response.
4) Server "thinking" about ~15 sec and starts transmission.
5) After HTTP headers received (<1 sec), HTTrack starts downloading second
image (first image's data downloading in background).

So, the question: while processing steps 3, 4 and 5 (before HTTP headers
received) of first image, why HTTrack just waits for them? On 400+ pages with
many slow links it take some weeks. Maybe I missed a configuration parameter
for this?

All articles

Subject Author Date
Single-threaded communications with slow servers

08/01/2021 00:41


Created with FORUM 2.0.11