Unattended use of gpg across a wide range of gpg versions, Ubuntu edition. --debug-quick-random taking evasive action.

[Dan Kegel]

addendum: demo of how to delete a key unattended with gpg2
Documented in earlier thread,
http://marc.info/?l=gnupg-users&m=146287358008663&w=2

-- snip --
#!/bin/sh
# Script to demonstrate unattended creation, export, and deletion of a
secret key with gpg 2.x
set -ex

cat > test-script.sh << "_EOF_"
set -e
set -x
passphrase=""
gpg --batch --passphrase "$passphrase" --quick-gen-key 'test user
<test at example.org>'

gpg --batch --passphrase "$passphrase" --pinentry-mode loopback
--export-secret-key --armor 'test user <test at example.org>' > key.dat

# 1st fingerprint is for the primary, 2nd is for the secondary?
fingerprint=$(gpg -k --with-colons test at example.org | awk -F: '/^fpr:/
{print $10}' | head -n 1)

gpg  --batch --passphrase "$passphrase" --pinentry-mode loopback --yes
--delete-secret-and-public-key $fingerprint
_EOF_
chmod +x test-script.sh

rm -rf /tmp/gpgtest-*
export GNUPGHOME=$(mktemp -d /tmp/gpgtest-XXXXXXX.tmp)
echo "allow-loopback-pinentry" > $GNUPGHOME/gpg-agent.conf
gpg-agent --daemon ./test-script.sh
rm -rf $GNUPGHOME
-- snip --

On Sat, Apr 29, 2017 at 9:14 PM, Dan Kegel <dank at kegel.com> wrote:
> tl;dr: anyone know what's up with --debug-quick-random?  Also, handy
> script for unattended key generation across many versions of gpg.
>
> Hi all.  This topic has been beaten to death on many forums and in many
> bug reports, but here's a user report from the field that sums up what
> works.  It's mostly just stitching together known workarounds, plus
> one little mystery
> with --debug-quick-random in gpg 2.1.15 (the one on Ubuntu 17.04).
> I'll list the problems, then at the bottom show the full solution I'm using.
>
> I'm writing a test script that uses gpg, so I reviewed
> https://www.gnupg.org/documentation/manuals/gnupg/Unattended-Usage-of-GPG.html
> but it doesn't quite handle all the situations I ran into.
> This kind of test script has to satisfy requirements like:
> - work on current OS as well as last few LTS releases
> - use the OS's default gpg
> - work in both interactive and headless situations
> - leave the user's normal environment unchanged
> - work even in deeply nested directories
> That means I can't follow some of the advice in the manual (e.g. "use
> GPGME" or "use --quick-addkey").
>
> For the purposes of testing, let's say I want to generate a key with the command
>    gpg --gen-key
> for use with apt on an Ubuntu 17.04 desktop, as well as in freshly
> installed headless older systems.
> (For instance, containers created with the commands
>   lxc-create -n ubu1204 -t download -- --dist ubuntu --release precise
> --arch amd64
>   lxc-create -n ubu1404 -t download -- --dist ubuntu --release trusty
> --arch amd64
>   lxc-create -n ubu1604 -t download -- --dist ubuntu --release xenial
> --arch amd64
>   lxc-create -n ubu1704 -t download -- --dist ubuntu --release zesty
> --arch amd64 )
> Easy, right?
>
> Challenges and solutions I ran into, rearranged in a less embarassing
> order than I ran into them:
>
> 0. Googling for solutions to problems finds stale or incomplete info
> from random people
> Solution: RTFM.  Really.  Go find *the manual* for gpg and read it.
>
> 1. Running a test script that creates keys affects user's keyring
> Solution: follow
> https://www.gnupg.org/documentation/manuals/gnupg/Ephemeral-home-directories.html
> i.e. create a directory for the test, and set GNUPGHOME to the
> absolute path to that dir
> Works on all systems
>
> 2. 'gpg --gen-key' prompts user for key parameters, and aborts if
> /dev/tty can't be opened (e.g. with noninteractive ssh )
> Solution: follow
> https://www.gnupg.org/documentation/manuals/gnupg/Unattended-GPG-key-generation.html
> i.e. create a file foo.dat containing the responses, e.g.
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: My Real Name
> Name-Email: foo at example.com
> Expire-Date: 30
> and change the command to 'gpg --batch --gen-key foo.dat'
> Works on ubuntu 16.04 and below
>
> 3. On ubuntu 16.04, which straddles gpg and gpg2, the command
>  'gpg --export | gpg2 --import -'
> appears to be required to get apt to notice a key you've generated
> with gpg, but 'gpg2 --import' aborts with
> gpg: can't connect to the agent: Invalid value passed to IPC
> gpg: error getting the KEK: No agent running
>
> Solution: 'sudo apt-get install gnupg-agent', then
> use "gpg-agent --daemon -- gpgcommand..." to create a transient
> gpg-agent just for the duration of the gpg command.
> This works on Ubuntu 12.04 through 16.04.
>
> 4. also on ubuntu 17.04, the previous fix isn't quite enough.
> gpg-agent fails with
> gpg-agent[1631]: command 'GENKEY' failed: Inappropriate ioctl for
> device <Pinentry>
> gpg: agent_genkey failed: Inappropriate ioctl for device
> which sounds like https://dev.gnupg.org/T2680
> Evidently it wants a tty, which isn't going to be possible.
> Solution:
> echo allow-loopback-pinentry > $GNUPGHOME/gpg-agent.conf
> and add --pinentry-mode loopback to the gpg command.
> This requires ubuntu 17.04 and up; you can't use it with ubuntu 12.04
> through 16.04.
>
> 5. gpg hangs with message
> Not enough random bytes available.  Please do some other work...
> Solutions:
> a) stuff the system rng somewhat securely; e.g. on Ubuntu, 'sudo
> apt-get install haveged'
> b) tell gpg to use an insecure RNG, e.g.
> if gpg --quick-random --version >/dev/null 2>&1 ; then
>     echo quick-random >> "$GNUPGHOME"/gpg.conf
> elif gpg --debug-quick-random --version >/dev/null 2>&1 ; then
>     echo debug-quick-random >> "$GNUPGHOME"/gpg.conf
> fi
> Either works on all tested ubuntu versions up to ubuntu 16.04.
>
> 6. On Ubuntu 17.04, gpg (2.1.15) takes several minutes to run, complaining
> gpg-agent[6385]: can't connect my own socket: IPC connect call failed
> gpg-agent[6385]: this process is useless - shutting down
> even with --debug-quick-random in gpg.conf (or gpg-agent.conf).
> Oddly, the same two workarounds fix this, more or less:
> a) stuff the system rng somewhat securely; e.g. on Ubuntu, 'sudo
> apt-get install haveged'
> b) tell gpg-agent to use an insecure RNG; only way is to pass
> --debug-quick-random option on gpg-agent's commandline!
> Neither conf file will do anymore.  That socket error is very odd, and
> so is the fact
> that tweaking the rng in these two ways makes it go away.  Bug?  Feature?
>
> 7. When running tests in directories with long names, gpg aborts with
> socket name '/some/long/path is too long
> Solution: GNUPGHOME must be shorter than UNIX_PATH_MAX, which can be
> as short as 94 bytes on some systems.
> (Fixed in latest gpg, but you have to work around it for older ones.)
>
> 8. gpg might write to the tty at some point
> Solution: as documented, add --no-tty options on all gpg calls or in conf file
>
> Here's a consensus script for unattended key generation demonstrating
> most of the workarounds.
> --- snip ---
> #!/bin/sh
> set -x
> set -e
>
> # Check to see if gpg requires agent & supports loopback prompt
> if gpg --version | head -n 1 | grep ' 2\.'
> then
>    gpg_use_loopback=true
> else
>    gpg_use_loopback=false
> fi
>
> # Check to see if ubuntu 16.04-specific workaround needed for apt
> if test -x /usr/bin/gpg2
> then
>    gpg_copy_to_gpg2_needed=true
> else
>    gpg_copy_to_gpg2_needed=false
> fi
>
> # Avoid 'socket name too long' error with older gpg
> GNUPGHOME=/tmp/gpg-isolation-demo-unique-and-obscure-path
> export GNUPGHOME
> rm -rf $GNUPGHOME
> mkdir -m700 $GNUPGHOME
>
> # 1st half of workaround to allow bypassing pinentry prompts; insecure-ish
> if $gpg_use_loopback
> then
>     echo allow-loopback-pinentry > $GNUPGHOME/gpg-agent.conf
> fi
>
> # Never send output to tty
> echo no-tty >> $GNUPGHOME/gpg.conf
> # Never ask questions
> echo batch >> $GNUPGHOME/gpg.conf
>
> # Work around lack of entropy; we don't need it for this test.  Insecure.
> QUICK=""
> if gpg --quick-random --version >/dev/null 2>&1 ; then
>     echo quick-random >> "$GNUPGHOME"/gpg.conf
> elif gpg --debug-quick-random --version >/dev/null 2>&1 ; then
>     echo debug-quick-random >> "$GNUPGHOME"/gpg.conf
>     QUICK=--debug-quick-random
> fi
>
> keyemail=foo at example.com
> cat > gpg.in.tmp <<_EOF_
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: My Real Name
> Name-Email: $keyemail
> Expire-Date: 30
> _EOF_
>
> if $gpg_use_loopback
> then
>     time gpg-agent $QUICK --daemon -- \
>     gpg --pinentry-mode loopback --passphrase '' --gen-key gpg.in.tmp
> else
>     gpg --passphrase '' --gen-key gpg.in.tmp < /dev/null
>
>     # Extra step only needed for ubuntu 16.04's apt?
>     if $gpg_copy_to_gpg2_needed
>     then
>         gpg --passphrase '' --armor --export-secret-keys $keyemail \
>         | gpg-agent --daemon gpg2 --passphrase "" --import -
>     fi
> fi
> --- snip ---