[Help-gnutls] Re: OpenSSL <-> GnuTLS imcompatibilities

[Simon Josefsson]

David Herrmann <dh.herrmann at googlemail.com> writes:

> Hi
> I know that OpenSSL only implements SSL2/3 and partly TLS1 but is
> there a way to connect with an OpenSSL
> client to a simple GnuTLS server?

Hi.  Sure, you can try the 'gnutls-serv' tool that comes with GnuTLS.
See the manual:

http://www.gnu.org/software/gnutls/manual/html_node/Invoking-gnutls_002dserv.html

> I used the following code to initialize my ssl listener:
>     gnutls_dh_params_init(&ssl_dhparams);
>     gnutls_dh_params_generate2(ssl_dhparams, SSL_DH_BITS);

This is one problem, you are generating DH parameters which is a slow
cryptographic process.  There is a sample client in the manual:

http://www.gnu.org/software/gnutls/manual/html_node/Simple-client-example-with-X_002e509-certificate-support.html

> And then if I connect with a simple SSLv3 OpenSSL connection, my
> gnutls_handshake returns either
> GNUTLS_E_UNEXPECTED_PACKET_LENGTH or GNUTLS_E_UNKNOWN_CIPHER_SUITE.

Please enable debug logging and post more information.

> However, the weird thing is, the OpenSSL handshake call blocks and
> does not return any error.
> Even when I kill the process of the GnuTLS listener the OpenSSL
> handshake still blocks.

This is because the DH issue.

> It would be nice to hear whether there are known compatibility
> problems between GnuTLS and
> OpenSSL and whether there are ways to force SSLv3 on GnuTLS to connect
> to OpenSSL
> without any problems.

I'm not aware of any compatibility problems with OpenSSL.

You can force GnuTLS to use SSLv3 with a
NORMAL:-VERS-TLS1.0:-VERS-TLS1.1 priority string.  But if you need that,
there is most likely some other problem that should be fixed instead.

> Another problem is the following code:
> static gnutls_rsa_params_t ssl_rsaparams;
> gnutls_rsa_params_init(&ssl_rsaparams);
> gnutls_rsa_params_generate2(ssl_rsaparams, 512);

These functions are for export ciphers, I would recommend you to not use
them at all.

> The last function blocks and does not return. gdb gives me something like:
>  (gdb) backtrace
> #0  0xb800e430 in __kernel_vsyscall ()
> #1  0xb7ea577d in select () from /lib/tls/i686/cmov/libc.so.6
> #2  0xb7d7f782 in ?? () from /lib/libgcrypt.so.11
> #3  0xb7d5334b in ?? () from /lib/libgcrypt.so.11
> #4  0xb7d54946 in ?? () from /lib/libgcrypt.so.11
> #5  0xb7d54cf9 in ?? () from /lib/libgcrypt.so.11
> #6  0xb7d862dc in ?? () from /lib/libgcrypt.so.11
> #7  0xb7d51bc7 in ?? () from /lib/libgcrypt.so.11
> #8  0xb7d531c9 in ?? () from /lib/libgcrypt.so.11
> #9  0xb7d719c4 in ?? () from /lib/libgcrypt.so.11
> #10 0xb7d4920f in ?? () from /lib/libgcrypt.so.11
> #11 0xb7d3dc24 in gcry_pk_genkey () from /lib/libgcrypt.so.11
> #12 0xb7f6a545 in _gnutls_rsa_generate_params () from /usr/lib/libgnutls.so.26
> #13 0xb7f8b6e9 in gnutls_x509_privkey_generate () from /usr/lib/libgnutls.so.26
> #14 0xb7f6a3b4 in gnutls_rsa_params_generate2 () from /usr/lib/libgnutls.so.26
> #15 0x080b878d in ssl_init () at ssl.c:73
>  #16 0x0805fa28 in main (argc=4, argv=0xbfa29a64) at ircd.c:730
> However, I had to interrupt the process myself. It looks like gcrypt calls some
> function which actually is no function. The select() syscall may be some random
> data in the memory.
> I hope someone can help me

Generating a key is a slow process, and it takes time.  Just give the
process a few minutes to finish.

/Simon