TLSv1.2 interop issue (was: Re: gnutls 3.0.9)

[Florian Weimer]

* Nikos Mavrogiannopoulos:

>> We're seeing interop issues with a TLSv1.2 server which advertises are
>> fairly restricted list of cipher suites.

> What do you see?

Well, the cipher suite thing was a different bug, on the server side,
not caused by GNUTLS.  Fixing that didn't make a dent in the original
issue.

The issue is triggered when I use GNTULS 2.12.14 to connect to an
OpenJDK 7u2 server which requires client certificates.

Here's output from "gnutls-cli --debug 255":

|<3>| HSK[0x163a450]: SERVER HELLO DONE was received [4 bytes]
|<6>| BUF[HSK]: Peeked 36 bytes of Data
|<6>| BUF[HSK]: Emptied buffer
|<6>| BUF[HSK]: Inserted 4 bytes of Data
|<3>| HSK[0x163a450]: CERTIFICATE was sent [742 bytes]
|<6>| BUF[HSK]: Peeked 4 bytes of Data
|<6>| BUF[HSK]: Emptied buffer
|<7>| HWRITE: enqueued 742. Total 742 bytes.
|<3>| HSK[0x163a450]: CLIENT KEY EXCHANGE was sent [262 bytes]
|<6>| BUF[HSK]: Peeked 0 bytes of Data
|<6>| BUF[HSK]: Emptied buffer
|<7>| HWRITE: enqueued 262. Total 1004 bytes.
|<2>| sign handshake cert vrfy: picked RSA-SHA512 with SHA512
|<2>| ASSERT: gnutls_sig.c:630
|<2>| ASSERT: auth_cert.c:1562
|<2>| ASSERT: gnutls_kx.c:336
|<2>| ASSERT: gnutls_handshake.c:2831
|<6>| BUF[HSK]: Cleared Data from buffer
*** Fatal error: GnuTLS internal error.
|<4>| REC: Sending Alert[2|80] - Internal error
|<4>| REC[0x163a450]: Sending Packet[1] Alert(21) with length: 2

gnutls_sig.c:630 says:

|    return gnutls_assert_val(GNUTLS_E_INTERNAL_ERROR); /* too bad we only support SHA1 and SHA256 */

This is a bit puzzling.  Why does GNUTLS pick RSA-SHA512 if it doesn't
support the algorithm?

I remove RSA-SHA384 and RSA-SHA512 from gnutls_algorithm.c, and now I
end up with:

|<2>| sign handshake cert vrfy: picked RSA-SHA256 with SHA256

And the handshake completes.

The next task is to figure out how to disable SHA-384 and SHA-512 in the
server and client code. *sigh*

-- 
Florian Weimer                <fweimer at bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99