Too many open files caused by repeated decryption operations via a GPGME API

[Kazunori Kobayashi]

Hi,

On modern Linux, we can change the maximum number of file descriptors per process in some ways. This feature is a well-known way for long time operation without reboot in cases such as server machines.

When I tried many iteration test of decryption via gpgme_op_decrypt() with the maximum number of file descriptors increased, "Too many open files" was caught from its result. At that time, the number of used file descriptors in the process did not reached to the limit I set.

The following error log was observed with the debug level of 5 enabled.

GPGME 2021-10-21 16:18:26 <0x047e>  gpgme_op_decrypt:493: error: Too many open files <GPGME>

I investigated where this error was caused in the source code, then found the following if statement to see if a value of file descriptor is bigger than FD_SETSIZE assumed the passed file descriptor is invalid and set the error code as EMFILE.

https://github.com/gpg/gpgme/blob/gpgme-1.10.0/src/posix-io.c#L673

The file descriptor seemed to be counted up to 1024 by repeating decryption operations. This value is derived from the upper limit of select(), but poll() is currently recommended on Linux and no limiting values are determined. This code using select() seems incompatible with the current file descriptor management on Linux.

My failure case occurred in v1.10.0, but the logic in master has the same if statement. For this reason, this problem will occur in the latest version.

https://github.com/gpg/gpgme/blob/master/src/posix-io.c#L725

One of the solution to avoid this issue would be to export the I/O callback functions to the caller side. I think we can avoid using the internal select() code and do appropriate synchronization I hope outside the gpgme library.

I referred to the following document.

https://www.gnupg.org/documentation/manuals/gpgme/I_002fO-Callback-Example.html#I_002fO-Callback-Example

However I feel that way is different from the original purpose of exporting the I/O callbacks, so it's just a workaround. If there is a more appropriate way to handle file descriptors more than the system default in GPGME, please let me know.

Regards,

-- 
Kazunori Kobayashi
EDGEMATRIX Inc.
https://edgematrix.com