Why Operating Systems don't always upgrade GnuPG [was: Re: How can we utilize latest GPG from RPM repository?]

[Daniel Kahn Gillmor]

On Sat 2018-02-17 17:06:54 -0600, helices wrote:
> I will probably never understand why wanting to run the most current
> version of gnupg on a plethora of servers is controversial.

Here's one last try to explain the situation.

GnuPG (and the libraries it depends on) are used by (aka "depended on
by") other libraries and tools, both those integrated into the operating
system itself, and those that might be externally installed.  Some of
these dependencies are "brittle".

Brittle software dependencies
-----------------------------

GnuPG is under active development, and it has never had a fully-featured
stable API (Application Programming Interface).  What i mean is, there
are some capabilities that are only available from the user interface
(UI), and are not in the stable API.  GnuPG's UI has been constantly
improving; sometimes that means that older (worse) UI gets deprecated,
removed, or has its semantics change.

For historical reasons, there are a number of tools that were built
around some element of the GnuPG UI that was current at the time the
tool was built.  Even worse, there are a number of tools that assume
certain behaviors and features of GnuPG's internal storage (e.g. what
goes on in ~/.gnupg/), which has never been explicitly part of its API
(confusingly, there are some exceptions where GnuPG upstream *has*
encouraged other tools to programmatically interact with some elements
within its internal storage).  Newer versions of GnuPG do different
things with its internal storage (and as users we get benefits from
those improvements).

Simply upgrading GnuPG to the latest available version on a server that
also ships other complex software is likely to lead to breakage
elsewhere in the OS because of these brittle assumptions and
dependencies around GnuPG's UI and internal storage.

A case study
------------

For example, the current stable version of the Debian operating system
is Debian 9 ("stretch"), and it ships a version of the "modern" branch
of GnuPG.

As one of the GnuPG maintainers for Debian, i was hoping at one point to
backport the "modern" version of GnuPG to the previous version of Debian
(Debian 8, "jessie"), which some people still run on servers.  I found
that such an upgrade would break at least a half-dozen other packages in
Debian jessie *that i knew of* [0] -- and their breakage would in turn
likely affect some number of other packages.  This was not an exhaustive
survey of all possible bugs, just the most visible ones. :/

I have personally given up on the project of backporting modern GnuPG to
"jessie", because i think what time i can devote to GnuPG maintenance is
better-spent elsewhere.  I don't have the bandwidth to cope with the
resultant bug reports in other packages that such a backport would
produce.  Generally, i encourage users of "jessie" to uprade their
entire OS to the current version of debian stable, and to take advantage
of the improvements to GnuPG that way.

What can we do?
---------------

The problems described above point to problems in the ecosystem *around*
GnuPG, but it also points to concerns about GnuPG's presentation of its
capabilities *to* the rest of the ecosystem.  To the extent that GnuPG
offers features that other tools might want to use, when those features
are not part of an explicit, documented API, the ecosystem apparently
*does* try to manipulate them anyway, with all the attendant brittleness
that you can imagine.

How can GnuPG contribute to fixing this problem?  The traditional way
that many other projects have taken is to define their core programmatic
functionality into a library with a strict interface guarantees, and
have explicitly deprecated other use.  The closest that GnuPG comes to
this technique is GPGME, which is not feature-complete (as compared to
the gpg executable), and has its own history of both difficult upgrades
and unclear/surprising semantics.  It also doesn't have bindings in many
popular programming languages.  When programmers in those language want
to use GnuPG, their shortest path to "something that works" often
involves shelling out to gpg, rather than binding to GPGME. :/ 

Another thing that would help would be to explicitly and succinctly
document the preferred ways of interacting with GnuPG in ways that other
developers find useful.  Perhaps GnuPG could also itself try to detect
when it is being used programmatically in an unstable way and produce
warnings?

Yet another complementary approach might be to aggressively police the
ecosystem by finding other software that deends on GnuPG in any of the
aforementioned brittle ways, and either ask those developers to stop
using GnuPG entirely, or to provide them with stable, well-supported
ways to do what they're looking to do.

I welcome discussion/suggestions on how we can improve this situation,
and i *definitely* welcome help in doing the kind of
ecosystem-perspective work necessary to make it easier to maintain an
up-to-date branch of GnuPG.

But shrugging and suggesting it's uncontroversial to upgrade arbitrary
machines to the latest version of GnuPG doesn't appreciate the scope of
the problem involved with software maintenance in an active and
interdependent ecosystem.

Regards,

        --dkg

[0] examples of breakage:
  https://bugs.debian.org/834281
  https://bugs.debian.org/835075
  https://bugs.debian.org/835592
  https://bugs.debian.org/835465
  https://bugs.debian.org/834514
  https://bugs.debian.org/834600
  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <https://lists.gnupg.org/pipermail/gnupg-users/attachments/20180219/efbac9df/attachment.sig>