[gnutls-devel] gnutls_pubkey_get_verify_algorithm fails with RSA keys of over 1536 bits

[Jensen, Erik A]

Calling gnutls_pubkey_get_verify_algorithm with an RSA signature of more than 192 bytes results in GNUTLS_E_PK_SIG_VERIFY_FAILED. This means that usage with keys with a modulus of over 1536 bits is impossible.

I tracked the problem down to line 1064 of lib/nettle/pk.c:
uint8_t digest_info[MAX_HASH_SIZE*3]
which, along with 1011:
if (key->size == 0 || *length < key->size)
results in the error.

MAX_HASH_SIZE is 64, so only 192 bytes are allocated. I would expect the function to be able at least to handle the 15424 bit (1928 byte) RSA keys generated by certtool when sec-param is set to ultra (and their corresponding signatures), but it'd be nice not to have a restriction at all.

This bug also causes the deprecated gnutls_pubkey_verify_hash function always to fail when using RSA keys of over 1536 bits.