[gnutls-devel] GnuTLS | crash in 32-bit gnutls_privkey_sign_data() when using MD2 digest algorithm (#1447)

[Read-only notification of GnuTLS library development activities]

Andrew Meadows created an issue: https://gitlab.com/gnutls/gnutls/-/issues/1447



## Description of problem:
Crash in gnutls_privkey_sign_data() when trying to create RSA signature with MD2 digest algorithm.

## Version of gnutls used:
32-bit 3.6.7-4+deb10u7

## Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
debian buster, using 32-bit package libgnutls28-dev:i386

## How reproducible:
100%

Steps to Reproduce:

 * Create a foo.cpp file with the following code:
```
// foo.cpp -- crash in gnutls_privkey_sign_data()
//
// This program demonstrates what appears to be a bug in 32-bit
// gnutls 3.6.7 when trying to create an RSA signature using the
// MD2 algorithm.  These are the tested versions and their results:
//
// 64-bit    gnutls-3.7.3    FAIL
// 64-bit    gnutls-3.6.7    SUCCESS
// 32-bit    gnutls-3.6.7    CRASH
//
// These tests were done using some variety of debian or ubuntu.

#include <string>
#include <iostream>
#include <gnutls/abstract.h>
#include <gnutls/crypto.h>

gnutls_digest_algorithm_t digest_algorithm = GNUTLS_DIG_MD2;

std::string pem_key = "-----BEGIN PRIVATE KEY-----"
"MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDj1ejV7Aax85LJ"
"yevp8C5bsinRyFGEGK+oVXVzQUKGUFI8fjW0j0uGGfezkwKvNo0UUg4RjoORCsjn"
"xTVh0WFUnQd2S9VCmngak41jSR7g7Zg5et2nLOLIKhUcq5N9bdYkFKFF6Tcij20Z"
"ekkHJtjPDYAemux26kViqALKvS/n0T+d3MedsVUrGVxCDjGvyrbNF0PaJXfW/ZEz"
"C01LcyqCUUMRLdGQCuG7EbV9xKOKajj2nuSeg/aPDAZ5tHJ/ZtBbmyV+5mF8UU5e"
"QJKYXCg16xeFoQstdRtk3xyOvfA9D0MPewIpoZL1xj8grMpEigJXH2DbMcuAwFHl"
"PL2OVPNZAgMBAAECggEATOqRYW7+T6NIYgFeOKsSK3i5b44uHudqnezD6vYULF3/"
"dVaycNOTjqFDo4c098v10u9kun6fYh7+9mf4fp5+Ol+M/l3ZkFOvymEX8XXlC9CK"
"fJVDMUYRBbxXRkeSUlz7V1DzpTN3np5HGbvT3as6tKP9JdftjrPGWukXkDRQcPTr"
"aTT1s205Bc7ph0G3nRBfNJz077XUJUD+O17r8ILPjMFnQkWOilx9rZtdnUvXwnei"
"WUfitibsuZRmNxldQVkeBlM3BS8sMAXodBnDIarQxkbFiRkOUxMNNjhB2oUF1bUI"
"zob7S0pAqXpnp6wawvscuwLn5Ik8vGicqYUgoC9UpQKBgQD2SpQyKtSiK6ccR1bH"
"divXmRbdu2tXBHVOxxNscH21aCMJ3RExFnQHMTZI2E0zmW2dYUQrqfN+OoXYkq/h"
"NfHGiXN059dOXU543UCymtMkxv9UqmkNQxuvhTlP+aBuVOsg6/vNBSUkxH+pyAhV"
"kKP70aCzai7xFV6+a3jAi5XQ5wKBgQDs0RYr8Xr0/BdwlekN7cOGRYGLOCOMwI+9"
"cq1t61iAUbhXf8wh2NaYg/GsEZ+ZJv5tLLT8KDVOeTCj2/AkKvDuqNhbjY6i8RMY"
"xoBx7Pd5GAfkOj3LoMRyQelY+6TbQh2l2iRAxJIjMeBpyyXnKIgoB09R75SmxbV/"
"Obx3OkNRvwKBgQCwdZCmPO+p3VLWvPoc4LratrGeXgizCNuvfIybFiHLFnh8Oap2"
"nBq3iHfXSzpM7PGPX8AC9LojR3TcTYS7+/VlBNf2+6DUYZ+2kvcmwCzlXbpUg9/O"
"g7VTVUVR2o0qX6czKEB5jOUm6a56C3qFK6OB6ZiU01AcpO4DymZytc+6IwKBgQDO"
"p673beTkgFBcGRWk6AUl2OWw/a5YF3nX0ojxUNyMLcP15znVl4m80fdFNqPVXCfO"
"UaMmIFhUQIFnnHg0t5xER+d2DVOZcVI+3oob4LQcYhIQ4ZonZgXHxyjAU+iKxWll"
"37SfKGp2eAHjkDllPQOEGsznLXDFD+bQPSvvx1ITtQKBgGCNB92wMTiLc0GdlEtq"
"vOUZBm/ImvmfM52Z7Xgk5x86Nb5kUq39+3udMT/7G0MklGy0gHrX6mjp5rnFAzi6"
"eeUxQBQJTOxateiaMrh8CLjrBth0NfTI61bkDNIV/EFAXa7Ou3jjzSDhPW/xzPmm"
"jA4eMgGDOpi6f8hdAjDPgrys"
"-----END PRIVATE KEY-----";

std::string message = "Hello world!";

int32_t main() {
    gnutls_datum_t key_dat = { (uint8_t*)(pem_key.data()), (uint32_t)(pem_key.size()) };
    gnutls_x509_privkey_t key;
    gnutls_privkey_t private_key;
    gnutls_datum_t msg_dat = { (uint8_t*)(message.data()), (uint32_t)(message.size()) };
    gnutls_datum_t sig_dat;

    int32_t sig64_len = 0;
    bool success = true;
    std::string error_message = "";
    if (gnutls_x509_privkey_init(&key)) {
        error_message = "out of memory";
        success = false;
    }
    if (success && gnutls_x509_privkey_import(key, &key_dat, GNUTLS_X509_FMT_PEM)) {
        error_message = "failed to import key";
        success = false;
    }
    if (success && gnutls_privkey_init(&private_key)) {
        error_message = "out of memory";
        success = false;
    }
    if (success) {
        if (gnutls_privkey_import_x509(private_key, key, 0)) {
            error_message = "failed to import private key";
            success = false;
        }
        if (success && GNUTLS_PK_RSA != gnutls_privkey_get_pk_algorithm(private_key, nullptr)) {
            // key not compatible with RSA?
            error_message = "private key incompatible with RSA";
            success = false;
        }
        // sign data
        if (gnutls_privkey_sign_data(private_key, digest_algorithm, 0, &msg_dat, &sig_dat) != 0) {
            error_message = "failed to sign";
            success = false;
        }
        if (success) {
            std::cout << "success!" << std::endl;
            gnutls_free(sig_dat.data);
        }
        gnutls_privkey_deinit(private_key);
    }
    gnutls_x509_privkey_deinit(key);
    if (!success) {
        std::cout << "fail: error='" << error_message << "'" << std::endl;
    }
    return 0;
}
```

 * Install the libgnutls28-dev:i386 package
 * Compile foo.cpp: `g++ -ofoo -m32 foo.cpp -l gnutls -L /usr/lib/i386-linux-gnu/`
 * Run the foo executable: `./foo`

## Actual results:
 * Executable will crash with SIGABRT signal


## Expected results:
 * Executable should print `success!`

-- 
Reply to this email directly or view it on GitLab: https://gitlab.com/gnutls/gnutls/-/issues/1447
You're receiving this email because of your account on gitlab.com.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnutls-devel/attachments/20230124/764d8774/attachment-0001.html>