Can't run any podman command anymore - panic: runtime error: invalid memory address or nil pointer dereference

[dozenposture]

I've used to have about 20 containers running without issues.
While trying to update an image, I've found podman can't execute any command anymore.

It really doesn't matter which command I try to run, I always get the following error message:

podman@host:~> podman info
ERRO[0000] Refreshing container a2dea400161ecc6b8440e937e4fb745a6bda8fc87056ddc14c4289dfc560c6da: acquiring lock 2 for container a2dea400161ecc6b8440e937e4fb745a6bda8fc87056ddc14c4289dfc560c6da: file exists
WARN[0000] Found incomplete layer "6d1aab23cceb2aa47ce43e610416609023b6a0dc576c7b383b3eccf30ab6efd6", deleting it
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x5633e1fec837]

goroutine 1 [running]:
github.com/containers/storage.(*layerStore).load(0xc000460280, 0x1)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/layers.go:917 +0xff7
github.com/containers/storage.(*store).newLayerStore(0x3b?, {0xc000049ef0, 0x28}, {0xc00060f8c0, 0x3b}, {0x0, 0x0}, {0x5633e327af18, 0xc000146d80}, 0x0)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/layers.go:1156 +0x6e5
github.com/containers/storage.(*store).getLayerStoreLocked(0xc0005ac000)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:1125 +0x25d
github.com/containers/storage.(*store).getLayerStore(0xc0005ac000)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:1141 +0x8f
github.com/containers/storage.(*store).writeToAllStores(0xc0005ac000, 0xc0005244c0)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:1401 +0x4c
github.com/containers/storage.(*store).DeleteContainer(0x93?, {0xc0005dcb00?, 0x0?})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:2703 +0x38
github.com/containers/podman/v5/libpod.(*storageService).DeleteContainer(0xc000116600, {0xc00060f500?, 0xc000260fc0?})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/storage.go:177 +0x72
github.com/containers/podman/v5/libpod.(*Container).teardownStorage(0xc0004b8780)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/container_internal.go:603 +0x314
github.com/containers/podman/v5/libpod.(*Runtime).removeContainer(0xc000505180, {0x5633e3267d90, 0x5633e4032600}, 0xc0004b8780, {0x0, 0x1, 0x0, 0x0, 0x0, 0x0, ...})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime_ctr.go:981 +0x1888
github.com/containers/podman/v5/libpod.(*Runtime).refresh(0xc000505180, {0x5633e3267d90, 0x5633e4032600}, {0xc0005130c0, 0x1f})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime.go:871 +0x52d
github.com/containers/podman/v5/libpod.makeRuntime({0x5633e3267d90, 0x5633e4032600}, 0xc000505180)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime.go:637 +0x1bce
github.com/containers/podman/v5/libpod.newRuntimeFromConfig({0x5633e3267d90, 0x5633e4032600}, 0xc0000c4008, {0xc000525660, 0x0, 0xa?})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime.go:213 +0x30e
github.com/containers/podman/v5/libpod.NewRuntime({0x5633e3267d90, 0x5633e4032600}, {0xc000525660, 0x0, 0x0})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime.go:171 +0x5a
github.com/containers/podman/v5/pkg/domain/infra.getRuntime({0x5633e3267d90, 0x5633e4032600}, 0xc0004a4100, 0xc000525898)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/runtime_libpod.go:228 +0x1edc
github.com/containers/podman/v5/pkg/domain/infra.GetRuntime.func1()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/runtime_libpod.go:46 +0x65
sync.(*Once).doSlow(0x1c?, 0xc0004be400?)
	/usr/lib64/go/1.23/src/sync/once.go:76 +0xb4
sync.(*Once).Do(...)
	/usr/lib64/go/1.23/src/sync/once.go:67
github.com/containers/podman/v5/pkg/domain/infra.GetRuntime({0x5633e3267d90?, 0x5633e4032600?}, 0x5633e2a79cfa?, 0xe?)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/runtime_libpod.go:45 +0x55
github.com/containers/podman/v5/pkg/domain/infra.NewLibpodImageRuntime(0xc0005259b0?, 0xc0004bbab8?)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/runtime_proxy.go:24 +0x27
github.com/containers/podman/v5/pkg/domain/infra.NewImageEngine(0x5633e2a79cfa?)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/runtime_abi.go:31 +0x4b
github.com/containers/podman/v5/cmd/podman/registry.NewImageEngine(0x5633e3f3b3c0?, {0x5633e4009e00?, 0x9?, 0x5633e2da6e58?})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/registry/registry.go:50 +0x54
main.persistentPreRunE(0x5633e3f3b3c0, {0x5633e4032600, 0x0, 0x0})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/root.go:317 +0x6f9
github.com/spf13/cobra.(*Command).execute(0x5633e3f3b3c0, {0xc0000400d0, 0x0, 0x0})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/spf13/cobra/command.go:956 +0x931
github.com/spf13/cobra.(*Command).ExecuteC(0x5633e3f23e00)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/spf13/cobra/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/spf13/cobra/command.go:1034
main.Execute()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/root.go:116 +0xb4
main.main()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/main.go:61 +0x4b2

I've browsed for similar issues and I've found something about lingering. The podman user has lingering enabled (since the beginning)

Can someone please help me out understanding what went wrong and how to fix this?
Thanks!

[Luap99]

The fix for this is containers/storage#2185