2.5.7 gnutls_x509_privkey_generate() returns GNUTLS_E_INVALID_REQUEST

[Nikos Mavrogiannopoulos]

Fix commited. Thank you!

On Sun, Sep 21, 2008 at 8:30 PM, Sam Varshavchik <mrsam at courier-mta.com> wrote:
> Nikos Mavrogiannopoulos writes:
>
>> Sam Varshavchik wrote:
>>
>>> Unfortunately, there appears to be more problems with private key
>>> functions in 2.5.7. I'm now getting segfaults in
>>> gnutls_x509_privkey_import(). My debugging results are as follows:
>>>
>> [...]
>>>
>>> ==3339== 5,886 (168 direct, 5,718 indirect) bytes in 3 blocks are
>>> definitely lost in loss record 8 of 9
>>> ==3339==    at 0x4A05174: calloc (vg_replace_malloc.c:397)
>>> ==3339==    by 0x3158C0A211: (within /usr/lib64/libtasn1.so.3.0.14)
>>> ==3339==    by 0x3158C0A3F2: (within /usr/lib64/libtasn1.so.3.0.14)
>>> ==3339==    by 0x3158C0A7BD: asn1_create_element (in
>>> /usr/lib64/libtasn1.so.3.0.14)
>>> ==3339==    by 0x4C829B2: _gnutls_asn1_encode_rsa (privkey.c:1075)
>>> ==3339==    by 0x4C83E0B: gnutls_x509_privkey_export (privkey.c:739)
>>
>> Hello Sam,
>>  I've commited fixes for these issues, so that latest code in git should
>> not have these issues. All except the last one (quoted). It would be
>> very helpful to give me some pointers on how to reproduce it.
>>
>> As far as I understand it should happening when
>> gnutls_rsa_params_export_raw() is called?
>
> The following minimal program triggers the leak:
>
> #include <gcrypt.h>
> #include <gnutls/gnutls.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <errno.h>
> #include <pthread.h>
>
> GCRY_THREAD_OPTION_PTHREAD_IMPL;
>
> int main()
> {
>        int i;
>        gnutls_rsa_params_t rsa;
>        size_t buflen;
>
>        gcry_control(GCRYCTL_SET_THREAD_CBS, &gcry_threads_pthread);
>        gcry_control(GCRYCTL_ENABLE_QUICK_RANDOM);
>        gnutls_global_init();
>
>        gnutls_rsa_params_init(&rsa);
>
>        printf("vers %s %s\n", LIBGNUTLS_VERSION, gnutls_check_version
> (NULL));
>
>        gnutls_rsa_params_generate2(rsa, 1024);
>
>        gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, NULL,
> &buflen);
>
>        char b[buflen];
>
>        gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, b, &buflen);
>
>        gnutls_rsa_params_deinit(rsa);
>
>        return(0);
> }
>
>
> The leak gets triggered only when gnutls_rsa_params_export_pkcs1() gets
> invoked twice, once with a NULL buffer pointer, to compute the buffer size,
> and a second time with the real buffer pointer. Invoking it just once is not
> sufficient.
>
> What looks like is happening is that in gnutls_x509_privkey_export():
>
>       case GNUTLS_PK_RSA:
>         ret = _gnutls_asn1_encode_rsa (&key->key, key->params);
>
> First time through, key->key is NULL, and _gnutls_asn1_encode_rsa()
> allocates it using asn1_create_element(). On the second call, key->key holds
> the previously allocated buffer, but _gnutls_asn1_encode_rsa() allocates a
> new one, without freeing the first one, and leaks it.
>
> I also see similar logic for the GNUTLS_PK_DSA case, and
> _gnutls_asn1_encode_dsa(), so there may also be a similar leak there.
>
>
>