TLS1.1 handshake problem (demonstrated with gnutls-cli)

[Simon Josefsson]

Vivek Dasmohapatra <vivek at collabora.co.uk> writes:

> While investigating a report of a problem connecting to a particular
> server when using the gnutls backend of our XMPP library, we
> (collabora)
> stumbled across this problem :
>
> /usr/local/src/gnutls26-2.8.4/src/gnutls-cli --priority "NORMAL:%COMPAT" \
>     -p 5223 stbeehive.oracle.com
> Resolving 'stbeehive.oracle.com'...
> Connecting to '141.146.118.11:5223'...
> proto  [0] = 3 TLS1.1
>        [1] = 2 TLS1.0
>        [2] = 1 SSL3.0
> *** Fatal error: A TLS fatal alert has been received.
> *** Received alert [10]: Unexpected message
> *** Handshake has failed
> GNUTLS ERROR: A TLS fatal alert has been received.
>
> (the proto lines are some extra debugging I added myself)
> Whereas:
>
> /usr/local/src/gnutls26-2.8.4/src/gnutls-cli \
>     --priority "NORMAL:%COMPAT:-VERS-TLS1.1"  -p 5223 stbeehive.oracle.com
> Resolving 'stbeehive.oracle.com'...
> Connecting to '141.146.118.11:5223'...
> proto  [0] = 1 SSL3.0
>        [1] = 2 TLS1.0
> - The hostname in the certificate matches 'stbeehive.oracle.com'.
> - Peer's certificate issuer is unknown
> - Peer's certificate is NOT trusted
> - Version: TLS1.0
> - Key Exchange: RSA
> - Cipher: AES-128-CBC
> - MAC: SHA1
> - Compression: NULL
> - Handshake was completed
>
> Now I'm not sure who's in the wrong here, the server or the client
> library: fwiw openssl negotiates TLS1 with the server out of the box,
> but s_client doesn't state which explicit TLS minor version it's using,
> so it could just be that it doesn't implement TLS 1.1 or 1.2 yet.

OpenSSL doesn't support TLS 1.1 or 1.2.

> Enabling 1.2 and disablinng 1.1 also causes the error above
> (Unexpected message).
>
> Disabling %COMPAT (padding) triggers:
>
> *** Received alert [20]: Bad record MAC
>
> instead.
>
> So, is this a bug in the TLS1.1/TLS1.2 implementation in gnutls, or
> is it the server doing something wrong, or both?

The symptom indicates a fairly common TLS server problem.  To know for
sure requires debugging the server side.  But if you cannot get it to
work with any other TLS client (that supports TLS > 1.0) I would suspect
a server bug rather than a GnuTLS bug.

What you need to do in your application is to expose an interface to
provide a GnuTLS priority string, preferrably per IP address, to allow
users to work around buggy servers.

/Simon