Problem using the server name extension

[Sam Varshavchik]

Simon Josefsson writes:

> Sam Varshavchik <mrsam at courier-mta.com> writes:
> 
>> Simon Josefsson writes:
>>
>>> Sam Varshavchik <mrsam at courier-mta.com> writes:
>>>
>>>> My client is compiled against gnutls 2.8.5. I am connecting to a
>>>> server that's built against OpenSSL 1.0.0.
>>>>
>>>> The OpenSSL server is failing the handshake with the following error
>>>> message:
>>>>
>>>> error:1408A0E3:SSL routines:SSL3_GET_CLIENT_HELLO:parse tlsext
>>>>
>>>> After some Googling around, I remove my client's call to
>>>> gnutls_server_name_set( .. GNUTLS_NAME_DNS .. ), and that makes
>>>> OpenSSL happy.
>>>>
>>>> If I do not invoke gnutls_server_name_set(), we have a happy
>>>> conversation. If I invoke gnutls_server_name_set(), OpenSSL bombs out
>>>> during the handshake.
>>>>
>>>> Has anyone seen this before?
>>>
>>> We've seen it for very old implementations, notably some IBM-derived
>>> variant of OpenSSL, that cannot handle any extensions.  But it is very
>>> surprising to see it for a recent OpenSSL.  Are you sure OpenSSL 1.0.0
>>> is used?  Can you reproduce this using 'openssl s_server'?  Maybe the
>>> application server is requesting SSLv2 from OpenSSL?
>>
>> The application is the client, and since the application is GnuTLS, it
>> can't be asking for SSLv2.
>>
>> Yes, Fedora 12, OpenSSL 1.0.0 is the server side. It's configured to
>> accept all protocols (SSLv23_method() in OpenSSL's API), but I also
>> tried TLSv1_method() as well, no difference.
>>
>> On the GnuTLS client side, I'm specifying  GNUTLS_TLS1_1,
>> GNUTLS_TLS1_0, and GNUTLS_SSL3 in that order. This is not a direct
>> SSL/TLS connection, this is IMAP STARTTLS, so I can't easily drop in
>> s_server in the server's place.
>>
>> I'll explore what debugging messages are available on the OpenSSL
>> side. I gave up on the debugger. Debugging optimized code, on either
>> the server or the client side, just doesn't work very well.
> 
> Maybe you can reproduce this using 'gnutls-cli'?  It supports STARTTLS
> by using --starttls and entering ^D when you want to start the TLS
> handshake.  Please post output from 'gnutls-cli -d 4711' if you can
> reproduce it.

Well now, that seems to work. The handshake appears to complete successfully 
with gnutls-cli. Perusing gnutls-cli's source, it does seem to use 
gnutls_server_name_set() by default, so this is a valid, working baseline.

So, it appears that I'm doing something on the client side, with GnuTLS, 
that's making OpenSSL on the server side crap out. Looks like I have 
something that I can dig into. Stay tuned.

> Maybe the server name you provide is simply the wrong one, and that's
> why the server refuses to talk with you?

I don't think that's it. It's "localhost" in both cases. I'm not certain 
that OpenSSL implements this extension.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: </pipermail/attachments/20100430/9688ff3b/attachment.pgp>