After storage reset, still getting SIGSEGV due to incomplete layer that podman tries to delete

[dozenposture]

This is a follow up on this topic.

As suggested, as a temporary workaround, I deleted all content of ~/.local/share/containers/storage (no named volumes so I deleted everything as I have separate directories that I mount inside containers).

I pulled again all the images (I'm using quadlet files so a simple reboot did the trick) and all of the containers worked expect for one that still give me the same error:

WARN[0000] Found incomplete layer "66495ed88a239f0698c32616e4dc2ad0edf67e6805bdcd638ce04c472ba20e6e", deleting it
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x55c0804138d7]

goroutine 1 [running]:
github.com/containers/storage.(*layerStore).load(0xc0002daa00, 0x1)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/layers.go:917 +0xff7
github.com/containers/storage.(*store).newLayerStore(0x3b?, {0xc0002422d0, 0x28}, {0xc000045f40, 0x3b}, {0x0, 0x0}, {0x55c0816a3cf8, 0xc0000ccea0}, 0x0)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/layers.go:1156 +0x6e5
github.com/containers/storage.(*store).getLayerStoreLocked(0xc00052a340)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:1125 +0x25d
github.com/containers/storage.(*store).Shutdown(0xc00052a340, 0x0)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/storage/store.go:3582 +0xa5
github.com/containers/common/libimage.(*Runtime).Shutdown(...)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/vendor/github.com/containers/common/libimage/runtime.go:148
github.com/containers/podman/v5/libpod.(*Runtime).Shutdown(0xc0001fd6c0, 0x0)
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/libpod/runtime.go:805 +0x19a
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ImageEngine).Shutdown.func1()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/abi/images.go:680 +0x1c
sync.(*Once).doSlow(0x0?, 0x55c0808ee820?)
	/usr/lib64/go/1.23/src/sync/once.go:76 +0xb4
sync.(*Once).Do(...)
	/usr/lib64/go/1.23/src/sync/once.go:67
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ImageEngine).Shutdown(0x55c08234ce00?, {0x55c08245b620?, 0x55c080eaf83d?})
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/pkg/domain/infra/abi/images.go:679 +0x3a
main.Execute()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/root.go:138 +0x215
main.main()
	/home/abuild/rpmbuild/BUILD/podman-5.3.1/cmd/podman/main.go:61 +0x4b2

I tried to move the quadlet file of that container under another user on the same system and the container started immediately without any issue. This suggests that the image does not have any problem, AFAIK.

I tried looking for something (file/directory) that contains that string, nothing:

host:/home/podman # find . -name '*66495ed*'
host:/home/podman #

What could be causing that problem? Is there something else that should be deleted to completely get rid of that issue?

[giuseppe]

AFAICS you are already building podman from sources. Could you please verify if the last version of the containers/storage solve the issue for you?

Are you using any additional image store?

[dozenposture]

Hi @giuseppe,
I'm not building from source but using the package provided by the OS I use, openSUSE Tumbleweed.
I can't provide the output of podman info as the SIGSEGV issue prevents podman to run correctly.

However, this is the output of the package manager:

Information for package podman:
-------------------------------
Repository     : Main Repository (OSS)
Name           : podman
Version        : 5.3.1-3.1
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 52.7 MiB
Installed      : Yes
Status         : up-to-date
Source package : podman-5.3.1-3.1.src
Upstream URL   : https://github.com/containers/podman

I'm not using any additional image store. Or at least not that I know.
My configuration is: quadlet files and mounted directories inside containers.

[giuseppe]

ok thanks.

I am closing this issue because it is a duplicate of containers/storage#2184 and we can keep the conversion there if you've any follow up since there is nothing new here.

There is really not much more we can do upstream, the bug was fixed, so please try with the upstream version of podman