HTTrack Website Copier
Free software offline browser - FORUM
Subject: select() on ssl connection problem
Author: Patrick Pfeifer
Date: 03/06/2009 09:26
Dear Xavier et al

I finally managed to find out what is causing the enormous delay I'm seeing
when receiving 304/Not Modified replies over ssl connection. To read less than
200 Bytes httrack would take at least 20 seconds (200 times select() timeout
of 1/10th second, i guess). The reason seems to be that select() is not
working properly on ssl connections. Some measures are taken against this
misbehavior already, in htsback.c in back_wait(). I'm talking about these

-----------code snip from htsback.c-------
           // données dispo?          //## if
          if (back[i].r.is_file)
          else if (back[i].r.ssl)

This is necessary because select() on the ssl fd would not set the read fd in
fds even though there is data available. The problem now is, that when parsing
http headers, where data is read byte by byte i think, select() waits until
the timeout _for each byte_ which is really annoying because it slows down the
transfer-rate to 10 bytes per second!

I hope you could fix that somehow. For a demonstration try run httrack like
this _twice_ and you will notice the annoyingly slow transfer-rate for
recieving the 304 response for <> the second

$ httrack --update -%H -Z -s0 -%s -%v2 -A100000
<> -* +*.gif

I could proove that the problem lies as I described it by introducing a
conditional select in htsback.c like this:

+ if (back[0].status==STATUS_WAIT_HEADERS) {
+     FD_ZERO(&fds);
+     FD_ZERO(&fds_c);
+     FD_ZERO(&fds_e);
+ } else
        DEBUG_W("select done\n");

If you add these five lines the problem is not occuring any more. But it's not
a real fix of course.



All articles

Subject Author Date
select() on ssl connection problem

03/06/2009 09:26
Re: select() on ssl connection problem

03/06/2009 10:03
Re: select() on ssl connection problem

03/17/2009 21:11
Re: select() on ssl connection problem

03/19/2009 22:01


Created with FORUM 2.0.11