Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPG 2.3.1-2.3.2 don't show recipients #1977

Closed
xsteadfastx opened this issue Aug 17, 2021 · 27 comments
Closed

GPG 2.3.1-2.3.2 don't show recipients #1977

xsteadfastx opened this issue Aug 17, 2021 · 27 comments
Labels
bug Defects
Milestone

Comments

@xsteadfastx
Copy link

xsteadfastx commented Aug 17, 2021

Summary

i get failed to read recipient IDs from raw secret on gopass fsck on some passwords.

Steps To Reproduce

i run gopass fsck.

Environment

  • OS: ubuntu 20.04
  • OS version: Linux troy 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • gopass Version:
gopass 1.12.7 (2021-07-02 09:20:21) go1.16.6 linux amd64
<root>     -  gpg 2.3.1 - git 2.32.0
Available Crypto Backends: age, gpgcli, plain
Available Storage Backends: fs, gitfs
  • Installation method: brew install gopass

Additional context

i can decrypt the files manually with gpg --decrypt .password-store/websites/bla.gpg.
with gopass show they get shown. but with rofi-gopass i get empty secret in the debug logs. after that i tried the gopass fsck and ran in this. i have the feeling that this happens on passwords i added the last.
here is a snippet from the debug log:

2021/08/17 14:46:06.798912 leaf/store.go:68     leaf.New        Instantiated  at /home/marv/.password-store - storage: &gitfs.Git{fs:(*fs.Store)(0xc000098890)} - crypto: &cli.GPG{binary:"/usr/bin/gpg", args:[]string{"--quiet", "--yes", "--compress-algo=none", "--no-encrypt-to", "--no-auto-check-trustdb"}, pubKeys:gpg.KeyList(nil), privKeys:gpg.KeyList(nil), listCache:(*lru.TwoQueueCache)(0xc000319860), throwKids:false}
2021/08/17 14:46:06.798918 root/init.go:71      root.(*Store).initialize        Root Store initialized at /home/marv/.password-store
2021/08/17 14:46:06.798927 fs/store.go:109      fs.(*Store).Exists      Checking if .gpg-id exists at /home/marv/.password-store/.gpg-id: true
2021/08/17 14:46:06.798933 action/init.go:36    action.(*Action).IsInitialized  Store is already initialized
2021/08/17 14:46:06.798985 fs/store.go:156      fs.(*Store).IsDir       websites/amazon/[email protected] at /home/marv/.password-store/websites/amazon/[email protected] is a directory? false
2021/08/17 14:46:06.798993 fs/store.go:156      fs.(*Store).IsDir       websites/amazon/[email protected] at /home/marv/.password-store/websites/amazon/[email protected] is a directory? false
2021/08/17 14:46:06.799001 fs/store.go:41       fs.(*Store).Get Reading websites/amazon/[email protected] from /home/marv/.password-store/websites/amazon/[email protected]
2021/08/17 14:46:06.799017 cli/decrypt.go:22    cli.(*GPG).Decrypt      /usr/bin/gpg [/usr/bin/gpg --quiet --yes --compress-algo=none --no-encrypt-to --no-auto-check-trustdb --decrypt]
2021/08/17 14:46:07.025924 secparse/parse.go:20 secparse.Parse  failed to parse as MIME: EOF
2021/08/17 14:46:07.025951 secrets/yaml.go:95   secrets.ParseYAML       Parsing
2021/08/17 14:46:07.025965 secparse/parse.go:30 secparse.Parse  failed to parse as YAML: EOF

2021/08/17 14:46:07.025971 secparse/parse.go:37 secparse.Parse  failed to parse as KV: EOF
2021/08/17 14:46:07.025976 secparse/parse.go:40 secparse.Parse  parsed as plain:
2021/08/17 14:46:07.025992 action/show.go:147   action.(*Action).showHandleOutput       empty secret - stacktrace: empty secret
2021/08/17 14:46:07.026000 action/show.go:63    action.(*Action).Show   empty secret - stacktrace: empty secret

and all the newer passwords are also not readable by the classic pass command. did something changed in the format?

@xsteadfastx
Copy link
Author

xsteadfastx commented Aug 17, 2021

here is the decrypted formats of a working and non working entry:

nonworking (added today via gopass create as always):

mypassword
comment:
url: penexchange.de
username: myusername

working:

mypasswordComment:
Url: x17-shop.de
Username: myusername

both decrypted via gpg --decrypt websites/....

@xsteadfastx
Copy link
Author

both files show the same signature: PGP RSA encrypted session key - keyid: 4A22181A 9308417E RSA (Encrypt or Sign) 4096b

@xsteadfastx
Copy link
Author

ok... i copy pasted from the debug log the actual gpg call:

/usr/bin/gpg --quiet --yes --compress-algo=none --no-encrypt-to --no-auto-check-trustdb --decrypt .password-store/websites/penexchange.de/username.gpg

and it returns nothing.... if i try an older one... it works.

@xsteadfastx
Copy link
Author

xsteadfastx commented Aug 17, 2021

it looks like the newer entries got created with a newer gpg version.
can i enforce which gpg version to use?

~ on  master [!]
❯ gpg --version
gpg (GnuPG) 2.3.1
libgcrypt 1.9.3
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/marv/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
AEAD: EAX, OCB
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

~ on  master [!]
❯ /usr/bin/gpg --version
gpg (GnuPG) 2.2.19
libgcrypt 1.8.5
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/marv/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

@xsteadfastx
Copy link
Author

xsteadfastx commented Aug 18, 2021

here is the debug log for fsck. one working entry and one thats not:

021/08/18 08:36:25.900725 leaf/fsck.go:52      leaf.(*Store).Fsck      [] Checking WOBCOM/ldap
2021/08/18 08:36:25.900744 fs/store.go:41       fs.(*Store).Get Reading WOBCOM/ldap.gpg from /home/marv/.password-store/WOBCOM/ldap.gpg
2021/08/18 08:36:25.900789 cli/decrypt.go:34    cli.(*GPG).RecipientIDs /home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg [/home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg --batch --list-only
 --list-packets --no-default-keyring --secret-keyring /dev/null]
2021/08/18 08:36:25.904020 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: # off=0 ctb=85 tag=1 hlen=3 plen=524
2021/08/18 08:36:25.904069 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: :pubkey enc packet: version 3, algo 1, keyid 1A18224A7E410893
2021/08/18 08:36:25.904096 cli/keyring.go:31    cli.(*GPG).listKeys     /home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg [/home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg --with-colons --wit
h-fingerprint --fixed-list-mode --list-public-keys 1A18224A7E410893]
2021/08/18 08:36:25.907690 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: data: [4096 bits]
2021/08/18 08:36:25.907746 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: # off=527 ctb=d2 tag=18 hlen=2 plen=72 new-ctb
2021/08/18 08:36:25.907752 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: :encrypted data packet:
2021/08/18 08:36:25.907757 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: length: 72
2021/08/18 08:36:25.907762 cli/decrypt.go:44    cli.(*GPG).RecipientIDs GPG Output: mdc_method: 2
2021/08/18 08:36:25.907776 cli/keyring.go:31    cli.(*GPG).listKeys     /home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg [/home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg --with-colons --wit
h-fingerprint --fixed-list-mode --list-secret-keys 81ADC3FB6D362E3775A1CCB7D87BCB6EF82A0017]
2021/08/18 08:36:25.911796 fs/store.go:109      fs.(*Store).Exists      Checking if WOBCOM/ldap/.gpg-id exists at /home/marv/.password-store/WOBCOM/ldap/.gpg-id: false
2021/08/18 08:36:25.911817 fs/store.go:109      fs.(*Store).Exists      Checking if WOBCOM/.gpg-id exists at /home/marv/.password-store/WOBCOM/.gpg-id: false
2021/08/18 08:36:25.911826 fs/store.go:109      fs.(*Store).Exists      Checking if .gpg-id exists at /home/marv/.password-store/.gpg-id: true
2021/08/18 08:36:25.911833 fs/store.go:41       fs.(*Store).Get Reading .gpg-id from /home/marv/.password-store/.gpg-id
2021/08/18 08:36:25.911867 cli/keyring.go:31    cli.(*GPG).listKeys     /home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg [/home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg --with-colons --with-fingerprint --fixed-list-mode --list-secret-keys 81ADC3FB6D362E3775A1CCB7D87BCB6EF82A0017]
2021/08/18 08:36:25.915049 leaf/fsck.go:52      leaf.(*Store).Fsck      [] Checking WOBCOM/lsw
2021/08/18 08:36:25.915064 fs/store.go:41       fs.(*Store).Get Reading WOBCOM/lsw.gpg from /home/marv/.password-store/WOBCOM/lsw.gpg
2021/08/18 08:36:25.915094 cli/decrypt.go:34    cli.(*GPG).RecipientIDs /home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg [/home/linuxbrew/.linuxbrew/Cellar/gnupg/2.3.1_1/bin/gpg --batch --list-only --list-packets --no-default-keyring --secret-keyring /dev/null]
2021/08/18 08:36:25.917830 root/fsck.go:31      root.(*Store).Fsck      ERROR: fsck failed on root store: failed to check "WOBCOM/lsw": failed to read recipient IDs from raw secret: exit status 2
2021/08/18 08:36:25.917905 action/fsck.go:58    action.(*Action).Fsck   fsck found errors: 1 error occurred:
        * failed to check "WOBCOM/lsw": failed to read recipient IDs from raw secret: exit status 2

 - stacktrace: 1 error occurred:
        * failed to check "WOBCOM/lsw": failed to read recipient IDs from raw secret: exit status 2

@dominikschulz dominikschulz added the bug Defects label Aug 18, 2021
@dominikschulz dominikschulz added this to the 1.x.x milestone Aug 18, 2021
@xsteadfastx
Copy link
Author

i think i found the error. i read alot through the code and executed the gpg commands and checked the output. i got packet errors here and there. right now my i think my trustdb was corrupted. so nothing to do with this wonderful piece of software,

@AnomalRoil
Copy link
Member

Glad you could find the solution, on my side the only thing that came to mind was this:
if you created some secrets on a device where the throw-keyids GPG config option was set, maybe it messed up these secrets.

See e.g. #1083

@xsteadfastx
Copy link
Author

thats a pretty good hint. i dont have this set in my config but i wonder if its somewhere enabled in the default config... i wish gpg could just print out its enabled options.

@matthew-brett
Copy link

Please forgive me if it is the wrong thing to do, to comment on this closed issue, but I have the same issue, and the fixes suggested above do not work for me.

Specifically I have:

$ gopass version
gopass 1.12.8 (2021-08-28 17:36:37) go1.17 darwin amd64
<root>     -  gpg 2.3.2 - git 2.30.1
Available Crypto Backends: age, gpgcli, plain
Available Storage Backends: fs, gitfs

with everything installed via Homebrew.

With gopass fsck I get:

[] Checking all secrets in store
] 132 / 252 [Gooooooooooooooooooooooooooooooooooooooooooooopass                                             ]  52.38% 
❌ fsck failed on root store: failed to check "account.co-op.co.uk/[email protected]": failed to read recipient IDs from raw secret: exit status 2

I can decrypt the file via gpg:

$ gpg --decrypt account.co-op.co.uk/[email protected]
gpg: encrypted with rsa2048 key, ID <my key id>, created 2021-04-28
      "Matthew Brett <[email protected]>"
<correct password>

I have only the gpg binary installed, not gpg2 - where gpg is the version above.

gopass show account.co-op.co.uk/[email protected] works correctly.

I recreated my GPG trustdb with these commands with no change in behavior.

I have not set the throw-keyids GPG config option.

This is a single-user store - I haven't used recipients add. I use it on only two computers, both updated macOS with Homebrew installs. I get the same gopass fsck errors on both.

Is there anything further I can do to debug?

@AnomalRoil
Copy link
Member

@matthew-brett I think you could try and fix this by running:

gopass edit account.co-op.co.uk/[email protected]
<add a comment or something on the line below your secret and save&quit>

And see if now it works or not. This would basically "reencrypt" the secret. I think running fsck --decrypt would also work, but maybe that's what you're doing already?

Otherwise, could you run in DEBUG mode and give us the (redacted) part around this issue?

You can do so by running:

GOPASS_DEBUG_LOG=debug.log gopass fsck --decrypt

and then look at the debug.log file it would have created.

@matthew-brett
Copy link

gopass edit as above doesn't change the fsck output.

--decrypt gives the same error.

Is there a good private way to send you the debug log (sorry, I am somewhat paranoid about leaking GPG information)?

@matthew-brett
Copy link

The part relevant to the entry causing the error is:

2021/10/15 13:41:23.161772 leaf/fsck.go:52	leaf.(*Store).Fsck	[] Checking account.co-op.co.uk/[email protected]
2021/10/15 13:41:23.161805 fs/store.go:41	fs.(*Store).Get	Reading account.co-op.co.uk/[email protected] from /Users/mb312/.local/share/gopass/stores/root/account.co-op.co.uk/[email protected]
2021/10/15 13:41:23.161887 cli/decrypt.go:22	cli.(*GPG).Decrypt	/usr/local/Cellar/gnupg/2.3.2/bin/gpg [/usr/local/Cellar/gnupg/2.3.2/bin/gpg --quiet --yes --compress-algo=none --no-encrypt-to --no-auto-check-trustdb --decrypt]
2021/10/15 13:41:23.289890 secparse/parse.go:20	secparse.Parse	failed to parse as MIME: (elided)
2021/10/15 13:41:23.289923 secrets/yaml.go:95	secrets.ParseYAML	Parsing (elided)
2021/10/15 13:41:23.289936 secparse/parse.go:30	secparse.Parse	failed to parse as YAML: no YAML marker
(elided)
2021/10/15 13:41:23.289955 secrets/kv.go:223	secrets.ParseKV	no KV entries
2021/10/15 13:41:23.289964 secparse/parse.go:34	secparse.Parse	parsed as KV: (elided)
2021/10/15 13:41:23.289978 fs/store.go:41	fs.(*Store).Get	Reading account.co-op.co.uk/[email protected] from /Users/mb312/.local/share/gopass/stores/root/account.co-op.co.uk/[email protected]
2021/10/15 13:41:23.290090 cli/decrypt.go:34	cli.(*GPG).RecipientIDs	/usr/local/Cellar/gnupg/2.3.2/bin/gpg [/usr/local/Cellar/gnupg/2.3.2/bin/gpg --batch --list-only --list-packets --no-default-keyring --secret-keyring /dev/null]
2021/10/15 13:41:23.299027 root/fsck.go:31	root.(*Store).Fsck	ERROR: fsck failed on root store: failed to check "account.co-op.co.uk/[email protected]": failed to read recipient IDs from raw secret: exit status 2
2021/10/15 13:41:23.299127 action/fsck.go:59	action.(*Action).Fsck	fsck found errors: 1 error occurred:

However, for the first key it checks, just before this in the log, that does not raise an error on the command line, I get the same failed to parse as YAML: no YAML marker error, but instead of the recipients error, I get

2021/10/15 13:41:23.129486 cli/keyring.go:33	cli.(*GPG).listKeys	/usr/local/Cellar/gnupg/2.3.2/bin/gpg [/usr/local/Cellar/gnupg/2.3.2/bin/gpg --with-colons --with-fingerprint --fixed-list-mode --list-public-keys <my-key-id>]

followed by :unknown packet: and a hex dump, but no error about recipients ID.

@AnomalRoil
Copy link
Member

And what happens when you are running manually

/usr/local/Cellar/gnupg/2.3.2/bin/gpg --batch --list-only --list-packets --no-default-keyring --secret-keyring /dev/null /Users/mb312/.local/share/gopass/stores/root/account.co-op.co.uk/[email protected]

@matthew-brett ?
What is the output?

@matthew-brett
Copy link

I get:

gpg: invalid marker packet
# off=0 ctb=85 tag=1 hlen=3 plen=268
:pubkey enc packet: version 3, algo 1, keyid 5FD3368ACEA67DAB
	data: [2048 bits]
# off=271 ctb=d4 tag=20 hlen=2 plen=84 new-ctb
:aead encrypted packet: cipher=9 aead=2 cb=21
	length: 84
# off=277 ctb=aa tag=10 hlen=5 plen=2181150457
:marker packet: [invalid]

When I delete this entry in Gopass, and create a new one in Gopass, with the same password, then running the same command on the new entry file:

gpg: [don't know]: invalid packet (ctb=7e)
gpg: [don't know]: invalid packet (ctb=00)
gpg: [don't know]: invalid packet (ctb=2e)
gpg: packet(13) too large

@AnomalRoil
Copy link
Member

Wow, I'm now able to reproduce this with Gopass 1.12.8 on a test entry.
We have probably messed up something, let's open a new issue!

@xsteadfastx
Copy link
Author

this is exactly what i got. i created a fresh gpg dir and it worked again. so i thought it was an error with my keyring.

@AnomalRoil
Copy link
Member

I think GPG 2.3.2 is the culprit: I updated and am now trying to bissect the origin of the bug, but this isn't working even with 1.10.0, which makes no sense.

So Gopass isn't really at fault, but we will have to dig into what's changed in GPG 2.3.2 I guess.
I'll try to install multiple versions of GPG to find which one brought the issue.

@AnomalRoil
Copy link
Member

@matthew-brett A workaround for now is to stick to the stable GnuPG 2.2 using:
brew install [email protected]

And making sure you export the 2.2 version in your path first as brew will tell you.

@xsteadfastx
Copy link
Author

xsteadfastx commented Oct 20, 2021 via email

@AnomalRoil
Copy link
Member

On my side I downgraded to GnuPG 2.2 and everything is working correctly, as soon as I update to 2.3.2 again, gopass fsck is failing again and GPG is giving packet errors.
We will track this in #2014 from now on, but it is probably the same issue, truth to be told, as you noticed the problem was related to GnuPG versions in #1977 (comment).

@matthew-brett
Copy link

Thanks - yes - that also works for me - for fixing fsck.

I made a new secret called go, to test.

However, when I clone the store to another machine, also running GnuPG 2.2 (2.2.12) I have the same problem that originally set me to investigating, which is:

$ gopass show websites/account.co-op.co.uk/[email protected]

Error: empty secret

On the other hand, the new secret I just made on my GnuPG 2.2 mac gives:

$ gopass show go
⚠ Parsing is enabled. Use -n to disable.
Secret: go

WBuBar3RpRw3Z9zkWNprHNBP

(I'm not using that secret, so no leak of anything interesting).

Is there a way to rebuild the store so it works correctly on another machine?

@AnomalRoil
Copy link
Member

gopass fsck --decrypt

should decrypt and rencrypt all secrets, so I guess adding gpg 2.2 to your path and then running it could solve it?

@matthew-brett
Copy link

Tried that first - but that doesn't seem to rewrite the encrypted files in the store ...

@AnomalRoil
Copy link
Member

I'm afraid then the only solution is to add a new "test" recipient on the machine where it works and then remove that recipient: it will reencrypt everything for sure.

Make sure you create a new gpg key and then delete that key if you do so.

@matthew-brett
Copy link

Thanks for the suggestion - and yes - thanks - that fixes it ...

@AnomalRoil
Copy link
Member

I guess we might want to add something to do re-encryption on demand to fsck

@dominikschulz dominikschulz changed the title failed to read recipient IDs from raw secret on gopass fsck GPG 2.3.3+ doesn't show recipients Oct 30, 2021
@AnomalRoil AnomalRoil changed the title GPG 2.3.3+ doesn't show recipients GPG 2.3.1-2.3.2 don't show recipients Nov 11, 2021
@AnomalRoil
Copy link
Member

It appears that GPG 2.3.3 has solved the issue:
they no longer exit with error code 2 when reading a file encrypted with GPG 2.3 using gpg --list-only --list-packets file.gpg

Amusingly they didn't exit with an error code when reading files encrypted with older GPG versions.

Notice that GnuPG 2.2 is a LTS, so to avoid such issues it might still be best to stay on GnuPG 2.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Defects
Projects
None yet
Development

No branches or pull requests

4 participants