Problem using the server name extension

[Simon Josefsson]

Sam Varshavchik <mrsam at courier-mta.com> writes:

> Simon Josefsson writes:
>
>> Sam Varshavchik <mrsam at courier-mta.com> writes:
>>
>>> My client is compiled against gnutls 2.8.5. I am connecting to a
>>> server that's built against OpenSSL 1.0.0.
>>>
>>> The OpenSSL server is failing the handshake with the following error
>>> message:
>>>
>>> error:1408A0E3:SSL routines:SSL3_GET_CLIENT_HELLO:parse tlsext
>>>
>>> After some Googling around, I remove my client's call to
>>> gnutls_server_name_set( .. GNUTLS_NAME_DNS .. ), and that makes
>>> OpenSSL happy.
>>>
>>> If I do not invoke gnutls_server_name_set(), we have a happy
>>> conversation. If I invoke gnutls_server_name_set(), OpenSSL bombs out
>>> during the handshake.
>>>
>>> Has anyone seen this before?
>>
>> We've seen it for very old implementations, notably some IBM-derived
>> variant of OpenSSL, that cannot handle any extensions.  But it is very
>> surprising to see it for a recent OpenSSL.  Are you sure OpenSSL 1.0.0
>> is used?  Can you reproduce this using 'openssl s_server'?  Maybe the
>> application server is requesting SSLv2 from OpenSSL?
>
> The application is the client, and since the application is GnuTLS, it
> can't be asking for SSLv2.
>
> Yes, Fedora 12, OpenSSL 1.0.0 is the server side. It's configured to
> accept all protocols (SSLv23_method() in OpenSSL's API), but I also
> tried TLSv1_method() as well, no difference.
>
> On the GnuTLS client side, I'm specifying  GNUTLS_TLS1_1,
> GNUTLS_TLS1_0, and GNUTLS_SSL3 in that order. This is not a direct
> SSL/TLS connection, this is IMAP STARTTLS, so I can't easily drop in
> s_server in the server's place.
>
> I'll explore what debugging messages are available on the OpenSSL
> side. I gave up on the debugger. Debugging optimized code, on either
> the server or the client side, just doesn't work very well.

Maybe you can reproduce this using 'gnutls-cli'?  It supports STARTTLS
by using --starttls and entering ^D when you want to start the TLS
handshake.  Please post output from 'gnutls-cli -d 4711' if you can
reproduce it.

Maybe the server name you provide is simply the wrong one, and that's
why the server refuses to talk with you?

/Simon