GNUTLS handshake errors and memory leaks (ECDHE related?)

[Yaroslav]

Hi!

I have added SSL support to my web server (nxweb) using GNUTLS. The server
has asynchronous architecture (based on epoll), where 4 threads run almost
independently accepting and servicing client http connections. Therefore I
am using GNUTLS in non-blocking mode.

Platform: Ubuntu 11.10, x64, 4-core CPU.
Linked with latest GNUTLS 3.0.11.

Most of the time everything works fine. Benchmarks show significant speed
advantage over nginx (which is build on OpenSSL). But sometimes there are
handshake errors and also memory leaks, which might be bound to those
errors.

I am attaching full valgrind log. Here are some excerpts:

% valgrind --leak-check=full --show-reachable=yes
dist/Debug/GNU-Linux-x86/nxweb
...
2012-01-10 13:46:51 [28704:0x71b9700]: gnutls handshake error -24 fatal=1
2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -110 fatal=1
2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -10 fatal=1
...
==28704== 47,952 bytes in 999 blocks are definitely lost in loss record 11
of 11
==28704==    at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
==28704==    by 0x5F28908: __gmp_default_allocate (in
/usr/lib/libgmp.so.10.0.1)
==28704==    by 0x5F3CF39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)
==28704==    by 0x50ED444: ecc_projective_check_point
(ecc_projective_check_point.c:60)
==28704==    by 0x50E692A: _wrap_nettle_pk_derive (pk.c:144)
==28704==    by 0x50CD4A2: calc_ecdh_key (ecdh_common.c:64)
==28704==    by 0x50CD569: _gnutls_proc_ecdh_common_client_kx
(ecdh_common.c:124)
==28704==    by 0x50615E6: _gnutls_recv_client_kx_message (gnutls_kx.c:549)
==28704==    by 0x505ED25: _gnutls_handshake_server
(gnutls_handshake.c:2801)
==28704==    by 0x505F622: gnutls_handshake (gnutls_handshake.c:2368)
==28704==    by 0x4088FD: do_handshake (nxd_ssl_socket.c:158)
==28704==    by 0x408A62: handshake_stub_is_do_write (nxd_ssl_socket.c:188)

There are also other leaks related to gnutls_dh_params_generate2 and some
error messages related to gnutls_certificate_set_x509_key_file but those
are of less importance to me as they happen only once on server startup.

The log has been collected while running apache-bench (ab) on the
client-side. Here is the log:

% ab -c 100 -n 1000 'https://test.domain.ru:8056/benchmark'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking test.domain.ru (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nxweb/3.0.0-dev
Server Hostname:        test.domain.ru
Server Port:            8056
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES256-SHA,2048,256

Document Path:          /benchmark
Document Length:        20 bytes

Concurrency Level:      100
Time taken for tests:   159.929 seconds
Complete requests:      1000
Failed requests:        3
   (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
Write errors:           0
Total transferred:      164505 bytes
HTML transferred:       19940 bytes
Requests per second:    6.25 [#/sec] (mean)
Time per request:       15992.885 [ms] (mean)
Time per request:       159.929 [ms] (mean, across all concurrent requests)
Transfer rate:          1.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0 13464 1915.7  13581   16479
Processing:   115 2526 1904.2   2416   15897
Waiting:      115 2487 1758.1   2411   15785
Total:      15870 15989 212.0  15912   16645


The problems might be related to ECDHE algorithm. There were no handshake
errors when 'PERFORMANCE:%SERVER_PRECEDENCE' priority string was used.
There were no handshake errors when using GNUTLS 2.10.5 (from Ubuntu repo)
-- I switched to 3.0 version as it supports significantly faster ECDHE
algorithm, to be on par with nginx/openssl.

And there seemed to be no such leaks when there were no handshake erros.

And, yes, I do call gnutls_deinit(session) when closing every connection
regardless of errors on it. I am ready to provide full source code (I plan
to publish it soon as it is open source).

Regards,

Yaroslav Stavnichiy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/attachments/20120110/4bd9b598/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind_log_120110-2.log
Type: text/x-log
Size: 31668 bytes
Desc: not available
URL: </pipermail/attachments/20120110/4bd9b598/attachment.bin>