[gnutls-devel] RSAES-PKCS1-v1_5 incompatible behavior

KIKUCHI Masashi kikuchi at lepidum.co.jp
Fri Dec 28 12:03:43 CET 2012


I sent this mail to bugs at gnutls.org yesterday but I received following
error message.  So, I re-send it to this ML.

> You need to subscribe in order to post to this mailing list.
> http://lists.gnutls.org/mailman/listinfo/gnutls-devel

GnuTLS server randomly sends illformed 'public-key-encrypted' data.

RSAES-PKCS1-v1_5(RFC2437 7.2.1 Encryption operation) says:
> 2. Convert the encoded message EM to an integer message
>   representative m: m = OS2IP (EM)
>  3. Apply the RSAEP encryption primitive (Section 5.1.1) to the public
>    key (n, e) and the message representative m to produce an integer
>    ciphertext representative c: c = RSAEP ((n, e), m)
> 4. Convert the ciphertext representative c to a ciphertext C of
>   length k octets: C = I2OSP (c, k)

The implementation in GnuTLS uses "length m" instead of "length k".
When the leading byte of "m" is zero, these two length does not match.


This results illformed 'public-key-encrypted' data and random handshake

Next, the implementation of decryption conversely uses "length C"
instead of "length k" and does not check RSAES-PKCS1-V1_5-DECRYPT Step 1:

> 1. If the length of the ciphertext C is not k octets, output
>   "decryption error" and stop.


This results in hiding the former bug when client and server are both

There are same problems in sign/verify functions.

To reproduce this handshake failure, use GnuTLS client and OpenSSL
server and run many handshakes (> 256 times).

This is OpenSSL log message when GnuTLS sends wrong signature.
> 3075704456:error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature
> 3075704456:error:1408807B:SSL routines:SSL3_GET_CERT_VERIFY:bad


More information about the Gnutls-devel mailing list