Yubikey Card Error "sign_and_send_pubkey: signing failed: agent refused operation"

[Lawrence Larabee]

I've got a new Yubikey NEO that I am trying to set up for SSH authentication. I've already personalized the card and loaded the keys, following all the creation rules (2048-bit max RSA, etc.) and loaded all the packages I am supposed to load. However I can't make it work. My platform is AMD64 GNU/Linux Ubuntu 16.04 running the Lubuntu flavor. I have tried it on two different machines with this same configuration.

I have verified that I am not running ssh-agent or gnome-keyring, as I have read these can interfere. 

"ssh-agent -L" shows my key 

I run 
export GPG_TTY="$(tty)" 
export SSH_AUTH_SOCK=/home/$USER/.gnupg/S.gpg-agent.ssh 
gpg - connect - agent updatestartuptty /bye 

I confirm that gpg-agent is running and that the auth sock environment variable is pointing to the correct place. 

gpg-agent.conf is: 

default-cache-ttl 36000 
pinentry-program /usr/bin/pinentry-gtk-2 
no-grab 
enable-ssh-support 

(tried disabling no-grab, no difference) 

scdaemon.conf: 

reader-port "Yubico Yubikey NEO OTP CCID 00 00" 
card-timeout 1 

(these don't make a difference, but some threads said to try it. it does same thing without the scdaemon options)

I turned on debugging, here is a dump of attempting to connect via SSH: 

<redacted>@<redacted>:~$ ssh -I /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so <redacted>@<redacted> 
no slots 
gpg-agent[24850]: ssh handler 0x7fa474d1a700 for fd 5 started 
gpg-agent[24850]: ssh request handler for request_identities (11) started 
gpg-agent[24850]: new connection to SCdaemon established (reusing) 
gpg-agent[24850]: DBG: chan_6 -> GETATTR $AUTHKEYID 
gpg-agent[24850]: DBG: chan_6 <- S $AUTHKEYID OPENPGP.3 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: DBG: chan_6 -> GETATTR SERIALNO 
gpg-agent[24850]: DBG: chan_6 <- S SERIALNO <redacted> 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: DBG: chan_6 -> READKEY OPENPGP.3 
gpg-agent[24850]: DBG: chan_6 <- [ <redacted> ...(286 byte(s) skipped) ] 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: DBG: chan_6 -> GETATTR $DISPSERIALNO 
gpg-agent[24850]: DBG: chan_6 <- S $DISPSERIALNO <redacted> 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: ssh request handler for request_identities (11) ready 
gpg-agent[24850]: ssh request handler for sign_request (13) started 
gpg-agent[24850]: DBG: chan_6 -> SERIALNO 
gpg-agent[24850]: DBG: chan_6 <- S SERIALNO <redacted> 0 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: DBG: detected card with S/N <redacted> 
gpg-agent[24850]: DBG: encoded hash: <redacted> 
gpg-agent[24850]: DBG: chan_6 -> SETDATA <redacted> 
gpg-agent[24850]: DBG: chan_6 <- OK 
gpg-agent[24850]: DBG: chan_6 -> PKAUTH OPENPGP.3 
gpg-agent[24850]: DBG: chan_6 <- INQUIRE NEEDPIN ||Please enter the PIN 
gpg-agent[24850]: starting a new PIN Entry 
gpg-agent[24850]: DBG: connection to PIN entry established 
gpg-agent[24850]: handler 0x7fa46f7fe700 for fd 10 started 
gpg-agent[24850]: DBG: chan_10 -> OK Pleased to meet you, process 24850 
gpg-agent[24850]: DBG: chan_8 <- OK Pleased to meet you, process 24850 
gpg-agent[24850]: DBG: chan_8 -> GETINFO pid 
gpg-agent[24850]: DBG: chan_10 <- GETINFO pid 
gpg-agent[24850]: DBG: chan_10 -> D 24850 
gpg-agent[24850]: DBG: chan_10 -> OK 
gpg-agent[24850]: DBG: chan_8 <- D 24850 
gpg-agent[24850]: DBG: chan_8 <- OK 
gpg-agent[24850]: DBG: chan_8 -> BYE 
gpg-agent[24850]: DBG: chan_10 <- BYE 
gpg-agent[24850]: DBG: chan_10 -> OK closing connection 
gpg-agent[24850]: handler 0x7fa46f7fe700 for fd 10 terminated 
gpg-agent[24850]: DBG: chan_6 -> [ <redacted> ...(76 byte(s) skipped) ] 
gpg-agent[24850]: DBG: chan_6 -> END 
gpg-agent[24850]: DBG: chan_6 <- ERR 100663404 Card error <SCD> 
gpg-agent[24850]: smartcard signing failed: Card error 
gpg-agent[24850]: ssh sign request failed: Card error <SCD> 
gpg-agent[24850]: ssh request handler for sign_request (13) ready 
sign_and_send_pubkey: signing failed: agent refused operation 
<redacted>@<redacted>'s password: 

As you can see, PIN entry works correctly, but after this everything fails with an error 100663404 and returns "signing failed: agent refused operation" 

I have Googled this extensively and have tried everything I can find to try to resolve this, but I've run out of things to try. 

Please help, 
LL