[gnutls-devel] GnuTLS | Incorrect error returned in TLS 1.3 when an unsupported signature algorithm is used by a client for Certificate VErify message signatures (#682)

[Development of GNU's TLS library]

New Issue was created.

Issue 682: https://gitlab.com/gnutls/gnutls/issues/682
Author:    Simo Sorce
Assignee:  

## Description of problem:
As I was writing tlsfuzzer tests to probe the correctness of client certificate handling by server implementations, it stood out that GNUTLS is returning a handshake_failure error when a client sends an RSA pkcs1 signature that the server should not accept.
The error returned should be illegal_parameter in this case (openssl and tlslite conform).

Here is the description of the 2 errors from the RFC:
```
   handshake_failure:  Receipt of a "handshake_failure" alert message
      indicates that the sender was unable to negotiate an acceptable
      set of security parameters given the options available.

   illegal_parameter:  A field in the handshake was incorrect or
      inconsistent with other fields.  This alert is used for errors
      which conform to the formal protocol syntax but are otherwise
      incorrect.
```

The second correctly describes the situation, the client misbehaved sending a field (signature algorithm selected) that is inconsistent with other fields (the server sent proper support signature algorithms lists in the CertificateRequest message).

A handshake_failure is improper because it is applicable only when the server, after parsing a list of permissible options, discovers it can use none. It is not the case here as the server is the *receiver*, and the client sent an invalid parameter, not a field to negotiate upon.

HTH.
Simo

## Version of gnutls used:
gnutls-3.6.5-2.fc29.x86_64

## Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
Fedora

## How reproducible:
Run the tests introduced here:
https://github.com/tomato42/tlsfuzzer/pull/496
using the following command line to run a GnuTLS server:
`$ gnutls-serv --http --priority NORMAL:-VERS-ALL:+VERS-TLS1.3 -p 4433 --x509keyfile=tests/serverX509Key.pem --x509certfile=tests/serverX509Cert.pem`

Steps to Reproduce:

 * run the server
 * run the test
 * observe the errors reported by the test

## Actual results:
Invalid pkcs1 signatures produce a handshake_failure error

## Expected results:
Invalid pkcs1 signatures produce an illegal_paramter error

-- 
Reply to this email directly or view it on GitLab: https://gitlab.com/gnutls/gnutls/issues/682
You're receiving this email because of your account on gitlab.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnutls-devel/attachments/20190123/073ab677/attachment.html>