Trouble with ElGamal pubkey_decrypt()

[Kevin Pulo]

Hi,

I'm having a LOT of trouble trying to use the ElGamal method in my own
program for authentication...

Attached to this email are two .c files and sample output illustrating the
problem.  First copy the gnupg libraries into the directory with the .c
files:

cd /usr/local/src/gnupg-1.0.1
cp `find -name *.a` ~/gpgtest

Then compile up the files:

cd ~/gpgtest
gcc encrypt.c -o encrypt -ldl *.a -I/usr/local/src/gnupg-1.0.1/include/
gcc decrypt.c -o decrypt -ldl *.a -I/usr/local/src/gnupg-1.0.1/include/

Then run the two:

./encrypt | ./decrypt

Encrypt simply encrypts the test string and outputs it to standard output.
Decrypt will read the encrypted data from standard input and attempt to
decrypt it.  It also encrypts the same test string and tries to decrypt
that too.

Put briefly, a pubkey_decrypt() appears to work only if it's done
immediately after the corresponding pubkey_encrypt(), otherwise it's
doomed.

If the decryption is first done on other_encrypted_data, and then
encrypted_data, then other_unencrypted_data will be correct but
unencrypted_data will not be.  If it's done the other way around,
encrypted_data first and other_encrypted_data second, then both
unencrypted_data and other_unencrypted_data will be wrong.  I can never
get the encrypted_data which is read in from stdin to decrypt properly.

I also notice that the encrypted data is a lot bigger than just twice the
original data, as it should be for ElGamal.  Further, when the decryption
fails it returns "unencrypted" data which is the same length as the
encrypted data, which definately isn't right.

Lastly, if the encryption and other_* variables are commented out of
decrypt.c, then the decryption returns -0.

I've traced this problem for many hours now, to no avail.  I've come to
the conclusion that either I'm doing something wrong (if so, _please_ tell
me what! :) ) or there's a bug somewhere in GnuPG.  The way I can destroy
the decryption of other_encrypted_data simply by doing it after the
decryption of encrypted_data is definately making me suspicious of memory
management.

The problem is identical in gnupg-1.0.1 and gnupg-1.1.0.  Compiling GnuPG
with M_GUARD doesn't yeild any noticable changes.  I can't compile with
M_DEBUG, for some reason...  Also winding the optimisation right back to
-O0 doesn't help.  'make check' reports no problems.

My system is Linux 2.2.12, libc5.4.46, gcc 2.7.2.3, for what it's worth.  
A friend on Linux 2.2.13, glibc2.1, gcc 2.95 gets similar results.

Bye 4 now,
  Kev.

--
.----------------------------------------------------------------------.
| Kevin Pulo                Quidquid latine dictum sit, altum viditur. |
| kev at zip.com.au                _ll l_ng__g_e_ _r_ hi__ly p__d_ct__le. |
| http://www.zip.com.au/~kev/         God casts the die, not the dice. |
`--------------- Linux: The choice of a GNU generation. ---------------'
-------------- next part --------------
A non-text attachment was scrubbed...
Name: encrypt.c
Type: application/octet-stream
Size: 3051 bytes
Desc: 
Url : /pipermail/attachments/20000109/36c6b739/encrypt.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: decrypt.c
Type: application/octet-stream
Size: 4179 bytes
Desc: 
Url : /pipermail/attachments/20000109/36c6b739/decrypt.obj
-------------- next part --------------
encrypt: data is:
456E6372797074696F6E205465737420737472696E670A00
?: Warning: using insecure memory!
encrypt: write successful
encrypt: encrypted_data is:
E2CED37EF356A45C04D395CEBD4DB2D2CAFBDC20FD145D107F19C1D92BE2663614C56B9D7AAD776D2033E3E7734CA9524A00FFF5D547700220A2598AA097CFA232EB9AB2B2E6DD5B45D3E44137D3A793A5B6ECE5A94C2EF7A60922349FACFE2397980194C24E7C6C5D6F0F4446100518EAEF8B286F23FE22067A95A0A39B063AF8BA5A58AA0D1024CD00999E4EDF3C62D20D2D70D54FA099CACAADECAC8B071D56B6C067A613C072389433E6E8CA71E2DB4E1D17950B89DBFCEBA31961193391B14A56006837F9417C73A1C45A25E4C5D30BB72B545DB4C643BD5EC51A4FD3B7511DE3930430903FACDA1F99ABC0AB0A51EE33AD1F4592A3D5AE4D90E23CE683
?: Warning: using insecure memory!
decrypt: Encrypted_data is:
E2CED37EF356A45C04D395CEBD4DB2D2CAFBDC20FD145D107F19C1D92BE2663614C56B9D7AAD776D2033E3E7734CA9524A00FFF5D547700220A2598AA097CFA232EB9AB2B2E6DD5B45D3E44137D3A793A5B6ECE5A94C2EF7A60922349FACFE2397980194C24E7C6C5D6F0F4446100518EAEF8B286F23FE22067A95A0A39B063AF8BA5A58AA0D1024CD00999E4EDF3C62D20D2D70D54FA099CACAADECAC8B071D56B6C067A613C072389433E6E8CA71E2DB4E1D17950B89DBFCEBA31961193391B14A56006837F9417C73A1C45A25E4C5D30BB72B545DB4C643BD5EC51A4FD3B7511DE3930430903FACDA1F99ABC0AB0A51EE33AD1F4592A3D5AE4D90E23CE683
decrypt: pubkey_check_secret_key == 0
decrypt: pubkey_check_secret_key == 0
decrypt: pubkey_check_secret_key == 0
decrypt: pubkey_decrypt(other_encrypted_data) == 0
decrypt: other_unencypted_data is:
456E6372797074696F6E205465737420737472696E670A00
decrypt: pubkey_check_secret_key == 0
decrypt: pubkey_decrypt(encrypted_data) == 0
decrypt: unencrypted_data is:
CE925A6A7D92B7C48B421D6A34F47159841E700174E5980AF40D607E5403B7215B8AC8B4C56E02E63EFE8EFD89981FCE8644D6B15E0EDEDBF5A9B394831A1183BD961259B51954DB2EB5199AE1A9249EB850EAF1F11103B7B09BF8F74E3172D045171E27BCCEC33A7BC140E61B3E04A92E69DFF97C71F02B65B3ED975AF21C21F15779088FA26156C03FF5A6C98EC39B51D9762C7BBE15000F07BFAA358B83402CAA842F4578BD8883E7688B9115836EC7E02CDE3DBC4D38EECDC83B45A480C74B058BC7309D1312387E3449D91A51E2513810B5C3C5A5F1FF0C44C664C8BAEEB5026DE8E928746A31AE0643EE4C50B62DADDF5613C8A37607B46056426DB6F1
decrypt: pubkey_check_secret_key == 0