[gnutls-devel] GnuTLS | x86(_64): CPU feature detection broken (#1282)

[Read-only notification of GnuTLS library development activities]

Rafael Gieschke created an issue: https://gitlab.com/gnutls/gnutls/-/issues/1282



If you try to run GnuTLS on a Linux kernel with the `noxsave` [command-line parameter](https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html) set, it will (on CPUs with AVX2) be terminated with `SIGILL` at the `verzoupper` instruction in <https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s#L4241>.

The reason is that `_gnutls_x86_cpuid_s` is not calculated like [`sha256_block_data_order`](https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s#L48) expects it to be.

In OpenSSL, `OPENSSL_ia32cap_P[4]` is essentially `= {CPUID.1:EDX, CPUID.1:ECX, CPUID.7:EBX, CPUID.7:ECX}` **but** with some heavy modifications (in assembly) done afterwards: <https://github.com/openssl/openssl/blob/d5d95daba59adc41ab60ea86acd513f255fca3c0/crypto/x86_64cpuid.pl#L73>. There is a more readable C version of the same code with explanations in BoringSSL: <https://github.com/google/boringssl/blob/bb88f52261f3231005c7fa43e55cc888d2f9f582/include/openssl/cpu.h#L75>, <https://github.com/google/boringssl/blob/bb88f52261f3231005c7fa43e55cc888d2f9f582/crypto/cpu-intel.c#L154>.

## Bugs in GnuTLS

1. [`read_cpuid_vals()`](https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/x86-common.c#L130) mixes up CPUID.1:EDX with CPUID.1:EBX, so that `_gnutls_x86_cpuid_s[0] = CPUID.1:EBX` instead of `= CPUID.1:EDX`.

2. [`read_cpuid_vals()`](https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/x86-common.c#L125) does neither check the OSXSAVE bit nor does it apply the other modifications done by the upstream code, e.g., it does not set `_gnutls_x86_cpuid_s[0] & (1 << 30)` on Intel CPUs (this bit originally was the ["IA64 processor emulating x86" bit](https://en.wikipedia.org/wiki/CPUID#EAX=1:_Processor_Info_and_Feature_Bits), is currently reserved (0) on current Intel CPUs and is (ab)used by the upstream code to indicate **any** Intel CPU).

3. This results in [`sha256_block_data_order`](https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s#L48) (and other function) not following Intel's specified way to check for AVX(2/-512) support (i.e., check the OSXSAVE bit first): <https://www.intel.com/content/dam/develop/external/us/en/documents-tps/325462-sdm-vol-1-2abcd-3abcd.pdf#page=353>. If a CPU supports AVX(2/-512) but the operating system does not (e.g., Linux with `noxsave`), this will, thus, cause SIGILL.

4. Not setting the Intel CPU bit (even if `read_cpuid_vals()` was fixed to set `_gnutls_x86_cpuid_s[0] = CPUID.1:EDX`), probably results in the AVX (without AVX2) code path's never being taken (https://gitlab.com/gnutls/gnutls/-/blob/9571f3a9e202ca2eeb369bb320bb93b638bb718c/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s#L57 checks `_gnutls_x86_cpuid_s[0]` for `1073741824 == 1 << 30`).

-- 
Reply to this email directly or view it on GitLab: https://gitlab.com/gnutls/gnutls/-/issues/1282
You're receiving this email because of your account on gitlab.com.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnutls-devel/attachments/20211017/253b76cc/attachment.html>