RSA sign/verify and hash generation functions

[Nikos Mavrogiannopoulos]

On 12/08/2010 12:30 AM, Murray S. Kucherawy wrote:
> I got a fair bit further, but I'm stuck now with GnuTLS not generating the same signature as OpenSSL under the same circumstances.  Anyone that can spot what I've missed, please do let me know.
> 
> Here's what I'm doing:
> 
> gnutls_datum_t rsa_out; /* output signature */
> gnutls_datum_t dd; /* SHA256 digest; size = 20 */
> gnutls_datum_t key; /* private key in PEM form */
> gnutls_x509_privkey_t privkey;
> gnutls_privkey_t rsa_key;
> 
> key.data = <buffer containing PEM formatted private key>
> key.len = strlen(key.data);
> 
> dd.data = (SHA256 of object to sign);
> dd.size = 20 (size of a SHA256 digest);

Why 20? SHA-1 is 20 bytes. SHA256 is 32 bytes.

> assert(gnutls_privkey_init(&rsa_key) == GNUTLS_E_SUCCESS);
> assert(gnutls_x509_privkey_init(&privkey) == GNUTLS_E_SUCCESS);
> assert(gnutls_x509_privkey_import(privkey, &key, GNUTLS_X509_FMT_PEM) == GNUTLS_E_SUCCESS);
> assert(gnutls_privkey_import_x509(rsa_key, privkey, 0) == GNUTLS_E_SUCCESS);
> assert(gnutls_privkey_sign_hash(rsa_key, &dd, &rsa_out == GNUTLS_E_SUCCESS);
> At this point, comparing "rsa_out" in the GnuTLS case to the OpenSSL case reveals that rsa_out.size is the same, but the data in rsa_out.data is not.
> I haven't tried the public key verification code nor the public key extraction (In fact I haven't even found that yet).

Do you mean the gnutls_x509_crt_verify_hash()?
Which signing method do you use with openssl? In gnutls we support only
PKCS #1 1.5 signatures (that one required by TLS).

regards,
Nikos