[gnutls-devel] RSA-SHA512 signature support for gnutls 2.12.x

[Nikos Mavrogiannopoulos]

On Thu, Mar 20, 2014 at 7:06 PM, Nikos Mavrogiannopoulos
<nmav at gnutls.org> wrote:
>> GnuTLS 2.12.x seems to fail to connect to servers using a cert signed
>> with RSA-SHA512. Since cacert.org seems to be using RSA-SHA512 this
>> has become more important.
>> More details in abovementioned bugreport.
>> Ivan Shmakov <ivan at siamics.net> has provided attached patch for
>> GnuTLS 2.12.x. Could you please review it (and if this is successful
>> integrate to GIT).
> Hello Andreas,
>  From a quick glimpse I don't think that this would solve the problem.
> This code does not restrict the signature algorithms available for
> certificate verification, but rather the signature algorithms that will
> be used during the TLS handshake. As I understand (but cannot deduce
> because the logs available are very limited) the client advertises only
> support for SHA512 hash in the signature algorithms extension.
> Unfortunately that version of gnutls could only work with either SHA1 or
> SHA256 in the TLS 1.2 handshake and this is what this check takes care
> of.

It seems I was wrong on that assessment. It is a different issue. That
code tries to enforce the TLS 1.2 (rfc5246) requirement:
"If the client provided a "signature_algorithms" extension, then all
certificates provided by the server MUST be signed by a hash/signature
algorithm pair that appears in that extension."
and it seems is buggy when the server has a sha512 certificate. That
code was removed in the end as the reporter noticed [*], so I guess
re-applying that patch would fix the issue (I cannot check whether
there were followups in that fix or other related changes).
Alternatively you could disable TLS 1.2 in that version of gnutls.

regards,
Nikos

[*]. The reason was that the server has no say in which algorithm the
CA will sign its certificate with, so there was an implementer
consensus in the TLS WG to ignore this protocol requirement.