Why Operating Systems don't always upgrade GnuPG

[Daniel Kahn Gillmor]

On Tue 2018-02-20 16:08:35 +0100, Werner Koch wrote:
> On Mon, 19 Feb 2018 19:45, dkg at fifthhorseman.net said:
>
>> GnuPG is under active development, and it has never had a fully-featured
>> stable API (Application Programming Interface).  What i mean is, there
>> are some capabilities that are only available from the user interface
>> (UI), and are not in the stable API.  GnuPG's UI has been constantly
>
> You probably expected that I need to dissent: Modulo a few bugs there
> has always been a stable API in GnuPG and we have taken great caution to
> not break it.

We're not in disagreement, actually :) I explicitly said that what was
has been lacking was a "*fully-featured* stable API".  I didn't say that
the API was not stable.  But the UI has features that the API does not,
and this results in people using the UI as though it was an API, or
mucking about with the innards of the local storage.  As one example, I
know this because i've been through that when working on early versions
of monkeysphere; i was using bash around GnuPG and yeah, in a few cases
i ended up stuffing a pipeline of "commands" into a gpg invocation
because it was there and seemed to do roughly what i wanted. 

The GnuPG stable API is much more fully-featured today (thank you for
the --quick-* commands!), but that has not been the case historically.
And we're paying the cost for that with legacy applications that have
ossified around wrong assumptions about how to interact with GnuPG :/

> Granted some parts of the API are not easy to use but
> nevertheless, they are stable.  For example the --edit-key interactor
> requires the use a state machine to stay compatible with future versions
> of GnuPG.  This has been remarked over and over in the last ~20 years.

Sorry, but building a state machine in external code to model the
internal state of a tool is not a functional API that we can
realistically expect developers to use.  If a module has state, it needs
to express that state to the user and make the transitions clear and
functional, with comprehensible and easily-handlable error cases.  To
wit:

> Unfortunately some folks reject to read manuals, examples and
> description on proper use and just go ahead and do what seems to work.

Even the folks like me, who read manuals and examples and descriptions
of proper use will eventually just go ahead and do what seems to work :/
The goal of using someone else's code is to *relieve* yourself of
needing to know exactly how that code works.  I know that there are some
people that end up sending patches upstream for every project they
touch.  But most developers don't have time or energy for that, and they
need simple, clear interfaces.

This is why (for example) i've argued in the past for making the return
code for gpgv simple and closely aligned with the boolean tests what
people are likely to want (https://dev.gnupg.org/T1537#100523).  Not
everyone wants to write a parser for a text stream to find out if a
signature is valid.  Some folks just want to know whether a signature is
valid, and are happy to punt on the details.

And even if everyone *did* want to write a parser, how many different
parser implementations do you think would need to be created?  how many
of them would be bug-free?  checking a return code is much harder to get
wrong.

> Actually this is not just a gpg thing: I have seen too many scripts and
> programs which neglect to use LC_ALL=C and break as soon as they are
> running under a different locale.

very true.  it's quite difficult to write robust programs, even when
interfaces are simple and clear and don't have sneaky side-effects or
behavioral changes due to rarely-changed environment variables.  And
even if you test with LC_ALL=C, did you test with LC_ALL=C.UTF-8 ? :P

> To avoid such breakage but keeping friendly to the command line user,
> gpg provide a machine interface (--with-colons, --batch, --command-fd)
> to make automation easy.

yes, these are great things!  thank you for them!

> I can't count how often I wrote that the only defined way to access the
> content of keyrings are my means of the --export and --import commands.  

I know.  I can't count how many times i've said it either.  It's pretty
frustrating!  I think it's aggravated further by the fact that we don't
have a clear rule like "do not touch anything inside ~/.gnupg" -- those
kinds of rules end up riddled with caveats like "well, except for
sshcontrol and gpg.conf" or "oh, you want to do custom subkey
management? let me tell you how to poke around in private-keys-v1.d/" :/

> Please give examples.

did you look at the bug reports i already cited?  Those are definitely
examples of overly-brittle wrappers around GnuPG.  You might also be
interested in Request-Tracker's brittle bindings
(https://bugs.debian.org/832829), or pretty much anything in debian that
explicitly Depends: gnupg1 today.

I'd be game to cc you every time i hold a project's hand through some
tricky problem if you'd like.  I've even succeeded at convincing some
developers to post their problems to this mailing list, or to open
reports on https://dev.gnupg.org, though i've also sometimes seen people
discouraged by responses that come across as "the thing that bothers you
is intended to bother you, deal with it".  Those folks generally don't
come back to ask more questions, and just figure if they can make it
"seem to work" that's good enough. :(

Anyway, here's one concrete example (hinted at above) of a programmatic
gap that is much easier to achieve by mucking around with the internal
state rather than by the programmatic interface:

 * I want to introduce a new signing-capable subkey, and i want to
   distribute it widely, but i don't want to start signing with it just
   yet.  Maybe the subkey needs some time to "burn in", or it needs to
   be reviewed before importing into the debian keyring, or maybe it's a
   new algorithm that i don't think all of my correspondents can verify
   yet.  I'm willing to start using it at some specified future date D,
   but i need it to exist and be distributed *today* so that i can be
   confident that everyone will have a copy of it by the day D rolls
   around.

Here are the options i see:

 * the programmatic approach:
    a) export the secret key to a temporary staging ground
    b) generate the subkey
    c) export the new, augmented secret key to a "future backup" file
    d) delete the secret key entirely (leaving the public part)
    e) re-import the temporarily staged secret key
    f) at date D, re-import the "future backup" file
   
 (note that there's a point in this workflow (between d and e) where i
  actually don't have access to *any* of my secret keys!  what side
  effects will happen there if my key is also in use by other
  tools/processes in the background?  if the transition fail at that
  point (for whatever reason), what happens to me then?)

 (note also that i haven't even tried to write down how i would attempt
  to do this programmatically)

 (note also that if i'm doing this with multiple keys concurrently
  (e.g. today i add a new signing-capable subkey; tomorrow i add an
  authentication-capable subkey), then maybe i've introduced a
  collision.  have i?  and before the "modern" branch of GnuPG, it
  wasn't even possible to merge secret keys, so to be compatible there,
  i'd need to add a step between e and f where i delete the entire
  secret key again.  now we definitely have opportunities for
  collisions!)

 * the "bad" approach (mucking about with the local data store):
   1) generate the subkey, learning its $keygrip
   2) mv ~/.gnupg/private-keys-v1.d/$keygrip.key{,.restore-at-D}
   3) at date D, mv ~/.gnupg/private-keys-v1.d/$keygrip.key{.restore-at-D,}

Guess which approach i'm going to be tempted to take if i write a tool
that attempts to automates sensible secret key management with GnuPG?
And i'm someone who knows better!

Alternately, i might try to use the "disable" subcommand within
--edit-key, but gpg(1) says this controls the "entire key", and it
appears to refer specifically to "encryption", but i'm interested in
sigining.  so that probably isn't right.

What i'd really like is a way to programmatically, privately mark a
secret key as "do not use before date D".  As far as i know, this
interface doesn't exist.  And I'm not asking for it yet because i know
that developer time is limited, and there are so many other outstanding
API improvements (e.g. the gpgv return code example mentioned above)
that i don't want to distract from :/

> And even today, we see hackers who are breaking working integration of
> gpg by replacing the (correct) use of the machine interface by the
> supposed easier to use human interface of gpg.  And then encountering
> problems with newer versions of gpg.  :-(

Right.  they're using the UI as though it was an API, because we have
failed to effectively communicate what the API is actually capable of,
presented in a way that was appealing for developers to adopt :(

>> are not part of an explicit, documented API, the ecosystem apparently
>> *does* try to manipulate them anyway, with all the attendant brittleness
>> that you can imagine.
>
> Right.  That is a bad idea and that is why it took many years to get a
> modern version in widespread use.

yep :/  We're getting there, though! :)

> Library interfaces are as much abused as command line interfaces.  This
> is not really the point.  A library may have the advantage that it
> exposes only a smaller set of interfaces and thus lessens the risk of
> wrong usage.

It's definitely possible to write bad library interfaces, or a library
interface that makes it easy for the developer to make mistakes.  Good
library API design is an art, and not a settled one at that.

I like many of the suggestions made by the libabc developers:

   https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git/plain/README

I disagree with some of them too :) But they do present a predictable,
orderly approach that lessens the cognitive load on the developer, and
lets the developer "encapsulate" the behavior of the library in their
mind.

One example of a terrible library interface is the classic OpenSSL
interface.  We have many concrete examples in the free software
ecosystem of people who finally "just got it working" in OpenSSL and
then have refused to ever touch that part of the code again, despite the
fact that they actually didn't manage to communicate their actual
programmatic intent to the library properly.

To their credit, the OpenSSL team has made significant improvements in
the interface that they encourage developers to use over the last few
years, and are now in a position to actively deprecate many of their
older mistakes.  It's still going to take us years to rip out the old
cruft from the ecosystem and replace it with "good" use of OpenSSL, or
other TLS implementations, though.  These things have a long shelf life.

The point is, usable programmatic interface design that encourages
stable/sane use (and actually guides developers into doing the right
thing) is a non-trivial task.  It is also potentially exacerbated when
the programmatic use is quite close to other common ways of interacting
with a tool (e.g. a UI).  In that case, developers have a tempting
"quick fix" of just pretending the UI is an API.  Do we really expect
them not to reach for that cookie, when it's just sitting there on the
shelf next to their head?

>> Yet another complementary approach might be to aggressively police the
>> ecosystem by finding other software that deends on GnuPG in any of the
>> aforementioned brittle ways, and either ask those developers to stop
>
>  [ ... ] this is still high on my task list.

Some of us have been doing this kind of work for a few years now, even
before the 2.2 release.  If you'd like to collaborate or strategize
about how we can do that better, i'd be happy to continue to work on it
with you.

Regards,

        --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <https://lists.gnupg.org/pipermail/gnupg-users/attachments/20180220/f5a6316a/attachment.sig>