[gnutls-devel] RSAES-PKCS1-v1_5 incompatible behavior

[KIKUCHI Masashi]

Hello,

I sent this mail to bugs at gnutls.org yesterday but I received following
error message.  So, I re-send it to this ML.

> You need to subscribe in order to post to this mailing list.
>
> http://lists.gnutls.org/mailman/listinfo/gnutls-devel


GnuTLS server randomly sends illformed 'public-key-encrypted' data.

RSAES-PKCS1-v1_5(RFC2437 7.2.1 Encryption operation) says:
> 2. Convert the encoded message EM to an integer message
>   representative m: m = OS2IP (EM)
>
>  3. Apply the RSAEP encryption primitive (Section 5.1.1) to the public
>    key (n, e) and the message representative m to produce an integer
>    ciphertext representative c: c = RSAEP ((n, e), m)
>
> 4. Convert the ciphertext representative c to a ciphertext C of
>   length k octets: C = I2OSP (c, k)

The implementation in GnuTLS uses "length m" instead of "length k".
When the leading byte of "m" is zero, these two length does not match.

http://git.savannah.gnu.org/cgit/gnutls.git/tree/lib/nettle/pk.c?id=gnutls_3_1_5#n224

This results illformed 'public-key-encrypted' data and random handshake
failure.


Next, the implementation of decryption conversely uses "length C"
instead of "length k" and does not check RSAES-PKCS1-V1_5-DECRYPT Step 1:

> 1. If the length of the ciphertext C is not k octets, output
>   "decryption error" and stop.

http://git.savannah.gnu.org/cgit/gnutls.git/tree/lib/nettle/pk.c?id=gnutls_3_1_5#n270

This results in hiding the former bug when client and server are both
GnuTLS.

There are same problems in sign/verify functions.

To reproduce this handshake failure, use GnuTLS client and OpenSSL
server and run many handshakes (> 256 times).

This is OpenSSL log message when GnuTLS sends wrong signature.
> 3075704456:error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature
length:rsa_sign.c:175:
> 3075704456:error:1408807B:SSL routines:SSL3_GET_CERT_VERIFY:bad
signature:s3_srvr.c:3049:


Regards,