Dear Xavier et al

I finally managed to find out what is causing the enormous delay I'm seeing
when receiving 304/Not Modified replies over ssl connection. To read less than
200 Bytes httrack would take at least 20 seconds (200 times select() timeout
of 1/10th second, i guess). The reason seems to be that select() is not
working properly on ssl connections. Some measures are taken against this
misbehavior already, in htsback.c in back_wait(). I'm talking about these
lines:

-----------code snip from htsback.c-------
           // données dispo?          //## if
(back[i].url_adr[0]!=lOCAL_CHAR)
          if (back[i].r.is_file)
            dispo=1;
          else if (back[i].r.ssl)
            dispo=1;
          else
            dispo=FD_ISSET(back[i].r.soc,&fds);
--------------------------------------------

This is necessary because select() on the ssl fd would not set the read fd in
fds even though there is data available. The problem now is, that when parsing
http headers, where data is read byte by byte i think, select() waits until
the timeout _for each byte_ which is really annoying because it slows down the
transfer-rate to 10 bytes per second!

I hope you could fix that somehow. For a demonstration try run httrack like
this _twice_ and you will notice the annoyingly slow transfer-rate for
recieving the 304 response for <https://moodle.fhnw.ch/pix/i/key.gif> the second
time:

$ httrack --update -%H -Z -s0 -%s -%v2 -A100000
<https://moodle.fhnw.ch/course/category.php?id=6> -* +*.gif

I could proove that the problem lies as I described it by introducing a
conditional select in htsback.c like this:

  #if HTS_WIDE_DEBUG
        DEBUG_W("select\n");
  #endif
+ if (back[0].status==STATUS_WAIT_HEADERS) {
+     FD_ZERO(&fds);
+     FD_ZERO(&fds_c);
+     FD_ZERO(&fds_e);
+ } else
        select(nfds,&fds,&fds_c,&fds_e,&tv);
  #if HTS_WIDE_DEBUG
        DEBUG_W("select done\n");
  #endif

If you add these five lines the problem is not occuring any more. But it's not
a real fix of course.

Regards

Patrick