[gnutls-devel] GnuTLS | gnutls_pubkey_verify_data2 calls fail erroneously with GNUTLS_E_INVALID_REQUEST when GNUTLS_VERIFY_DISABLE_CA_SIGN flag is set (#754)

[Development of GNU's TLS library]

New Issue was created.

Issue 754: https://gitlab.com/gnutls/gnutls/issues/754
Author:    Ken
Assignee:  

## Description of problem:

When calling gnutls_pubkey_verify_data2 with the GNUTLS_VERIFY_DISABLE_CA_SIGN flag set, GNUTLS_E_INVALID_REQUEST is asserted due to an erroneous comparison of the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA define (previously defined as GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA in gnutls_pubkey_flags_t) with the gnutls_certificate_verify_flags passed to the function.

This occurs when the bitwise AND comparison of the gnutls_certificate_verify_flags against OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA returns '1', as the GNUTLS_VERIFY_DISABLE_CA_SIGN gnutls_certificate_verify_flag and the deprecated GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA gnutls_pubkey_flag share the least significant bit of flags.

The bug was introduced in version 3.4.2 with this specific commit:
https://gitlab.com/gnutls/gnutls/commit/0422a67eb0935e1dd96d602445ee179ef9dd993d

I assume the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA define was added to maintain backwards compatibility with the old calling of gnutls_pubkey_verify_data2 that utilized gnutls_pubkey_flags_t.
Looking at internal gnutls calls of gnutls_pubkey_verify_data2, I don't see any call where this would be necessary any more. Perhaps this comparison is deprecated as well and could be removed?

gnutls_pubkey_verify_hash2 has the same comparison and the same issue might be present there as well, though I have not tested this.

## Version of gnutls used:

Tested in 3.6.7, issue is not present in versions prior to 3.4.2.

## Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)

N/A, built directly from gnutls source.

## How reproducible:

Steps to Reproduce:

 * Call gnutls_pubkey_verify_data2 with a valid signature and the GNUTLS_VERIFY_DISABLE_CA_SIGN flag set

## Actual results:

gnutls_pubkey_verify_data2 asserts GNUTLS_E_INVALID_REQUEST

## Expected results:

gnutls_pubkey_verify_data2 returns 0

-- 
Reply to this email directly or view it on GitLab: https://gitlab.com/gnutls/gnutls/issues/754
You're receiving this email because of your account on gitlab.com.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnutls-devel/attachments/20190415/9b717b85/attachment.html>