[gnutls-devel] gnutls_pubkey_get_verify_algorithm fails with RSA keys of over 1536 bits

[Nikos Mavrogiannopoulos]

On 02/05/2013 01:33 AM, Jensen, Erik A wrote:

> Calling gnutls_pubkey_get_verify_algorithm with an RSA signature of more than 192 bytes results in GNUTLS_E_PK_SIG_VERIFY_FAILED. This means that usage with keys with a modulus of over 1536 bits is impossible.
> 
> I tracked the problem down to line 1064 of lib/nettle/pk.c:
> uint8_t digest_info[MAX_HASH_SIZE*3]
> which, along with 1011:
> if (key->size == 0 || *length < key->size)
> results in the error.
> 
> MAX_HASH_SIZE is 64, so only 192 bytes are allocated. I would expect the function to be able at least to handle the 15424 bit (1928 byte) RSA keys generated by certtool when sec-param is set to ultra (and their corresponding signatures), but it'd be nice not to have a restriction at all.
> 
> This bug also causes the deprecated gnutls_pubkey_verify_hash function always to fail when using RSA keys of over 1536 bits.


Hello,
 Thank you for the report. I have committed the attached fix to the
issue. However, you should switch to verify_hash2() instead because the
verify_hash() function relies on the algorithm information in the
signature to be correct (and you never know if there is some possible
cross-algorithm collision attack).

regards,
Nikos
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch.txt
URL: </pipermail/attachments/20130205/83c6a54f/attachment.txt>