GnuPG 1.4.11 v. VMS batch mode

[Steven M. Schweda]

   Greetings:

   A victim of my GnuPG 1.4.11 for VMS kit recently ran into an
interesting problem.  A DCL procedure (think: "shell script") works when
run interactively, but loops when run as a batch job (think: "at/cron
job"?).  I think that I've figured out what's happening, and I thought
that I'd get a consultation on a proposed fix.

   The (apparently harmless) problem command looks like this:

      pipe gpg -d -o bat.out --passphrase-fd 0 login.com-gpg < pp.txt

The problem seems (to me, at the moment) to be bad general code, but the
problem case may never occur on a UNIX(-like) system, so perhaps no one
else would have noticed it.  During the program initialization,
util/ttyio.c:init_ttyfp() gets called.  It uses ctermid() to get the
name of the controlling terminal.  For a batch job on VMS, this comes
back as the disk name (which is what I'd guess never happens elsewhere). 
The program tries to fopen() this, which fails, because it's a disk, not
a terminal.  The program then tries to write a "cannot open" error
message to the terminal, but it won't do that until it gets the terminal
I/O set up properly, and for that it calls init_ttyfp().  Lather, rinse,
...

   The following change seems to solve the looping in a
plausible/harmless way (slightly enhancing the message, because a
typical VMS user might not otherwise guess why anyone is trying to open
a disk device):

--- util/ttyio.c_orig	2010-09-28 04:39:05 -0500
+++ util/ttyio.c	2011-09-07 22:19:34 -0500
@@ -185,7 +185,8 @@
 #else
     ttyfp = batchmode? stderr : fopen( tty_get_ttyname (), "r+");
     if( !ttyfp ) {
-	log_error("cannot open `%s': %s\n",
+	initialized = 1;        /* Don't come back again, ever. */
+	log_error("cannot open tty, `%s': %s\n",
                   tty_get_ttyname (), strerror(errno) );
 	exit(2);
     }

The victim still needs to specify option(s) like "--batch" to avoid the
fatal "cannot open [tty]" error, but this way it dies promptly, emitting
a message like:

      gpg: cannot open tty, `_ALP$DKC0:': no such file or directory

which, while not wonderfully informative, certainly beats an endless
loop (and no message).

   On VMS, it should be relatively easy to check the process mode during
initialization, and set the "--batch" flag ("batchmode"?) automatically
for a batch job.  I haven't yet thought of a good reason not to do this,
but I'm always open to a good counter-argument.

   Any wisdom would be gratefully received.

------------------------------------------------------------------------

   Steven M. Schweda               sms at antinode-info
   382 South Warwick Street        (+1) 651-699-9818
   Saint Paul  MN  55105-2547