Weird error message after reboot: Error: current system boot ID differs from cached boot ID

[groovyman]

Dear friends,

i am running a Fedora F40 Server in order to run my apps, where some of them use podman to run rootless container.
In order to archive this, i created some technical users, that acts like domains, that run in turn some container apps.

One of these technical users shows a strange behaviour after some updates. Whenever i reboot my sever and i login
on my technical user i get the following message, whenever i call podman:

podman ps
Error: current system boot ID differs from cached boot ID; an unhandled reboot has occurred. Please delete directories "/run/user/1002/containers" and "/Home/techlxoffice/rteDataSpace/podmanTMP" and re-run Podman

Then i go into the podmanTMP directory and i see a couple of files:

$ ls -l /Home/techlxoffice/rteDataSpace/podmanTMP
insgesamt 4
-rw-r--r--. 1 techlxoffice techlxoffice  37  3. Jul 21:55 alive
-rw-r--r--. 1 techlxoffice techlxoffice   0  3. Jul 21:55 alive.lck
drwxr-x---. 1 techlxoffice techlxoffice   0  4. Jul 01:26 exits
drwxr-x---. 1 techlxoffice techlxoffice 128  3. Jul 21:55 persist
drwx------. 1 techlxoffice techlxoffice   0  3. Jul 21:55 rp

So when i remove all these files, podman is running, and the systemd --user is also able to run the application, that normally lauch up, when the host is starting.

Do you have an idea, what went wrong. This configuration suddently fails, maybe due to a update, that took place a week ago.

[Luap99]

https://github.com/containers/podman/releases/tag/v5.1.0

Podman now detects unhandled system reboots and advises the user on proper mitigations.

In general the podman tmpdir and rundir must be on a tmpfs or deleted on boot in order to allow for podman to function properly

[hakong]

Sounds contradictory to what /usr/lib/tmpfiles.d/podman.conf contains.

user@workstation:/usr/lib/tmpfiles.d$ rpm -qf podman.conf
podman-5.2.2-9.el9_5.x86_64

user@workstation:/usr/lib/tmpfiles.d$ cat podman.conf
# /tmp/podman-run-* directory can contain content for Podman containers that have run
# for many days. This following line prevents systemd from removing this content.
x /tmp/podman-run-*
x /tmp/storage-run-*
x /tmp/containers-user-*
x /tmp/run-*/libpod
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
# Remove /var/tmp/container_images* podman temporary directories on each
# boot which are created when pulling or saving images.
R! /var/tmp/container_images*

We've just had a lot of our services go down due to this issue. We had to put a workaround in place to remove these files on boot.

I think podman should be able to function properly if these temp files are still in place.

[groovyman]

Salut @Luap99

so the best strategy is to ignore the --tmpdir argument, when building the image and let podman decide to choose the correct default tmp-directory for this step.

Would that be the right conclusion from your answer?

Thank you for your help!

[Luap99]

you can use --tmpdir but you should make sure that this directory is on a tmpfs or deleted on boot before you launch podman.

[maweil]

Thank you for your help, this hint helped me a lot.

In case others have the same issue even without using the --tmpdir argument:
I also ran into this problem on my Almalinux 9 system after it upgraded to 9.5 (which includes Podman 5.2.2). At least on my machine the issue was that /tmp was not mounted on a tmpfs. I had to enable tmp.mount first.

[beeritis]

Have the same issue today as podman was upgraded to 5.2.2.

This broke some of our containers that run as a systemd user service.
We initially downgraded back to podman 4.0.2 (previously installed version before servers got patched) as that doesn't have this problem.

The current workaround is to remove these files before the service starts .
For example : ExecStartPre=/usr/bin/rm -rf /tmp/containers-user-<id>/containers /tmp/podman-run-<id>/libpod/tmp

Part of our problem is that /tmp exists as an xfs file system (don't ask) so the files persist upon reboot.
Having said that , I don't know what is classed as a "unhandled system reboot", given that reboots are usually a result of patching which I wouldn't class as "unhandled". A flag to disable this feature would be useful or if there is a cleaner way to start containers /pods on reboot that would be better.

[beeritis]

This works perfectly ! Thank you :) .

Means I can remove my temporary workaround of using ExecStartPre= in systemd units :) .

[shay1197]

I had also considered doing this but any configuration changes seem to get overwritten when podman next gets upgraded. (i.e podman.conf gets replaced). If podman respects it, a configuration file in ~/.config/user-tmpfiles.d/*.conf might take precedence over the global podman.conf so the configuration persists but that would need testing.

@beeritis and anyone else really According to the tmpfiles.d manual page

Files in /etc/tmpfiles.d are reserved for
the local administrator, who may use this logic to override the
configuration files installed by vendor packages

To resolve this issue we copied the settings from our /usr/lib/tmpfiles.d/podman.conf to /etc/tmpfiles.d/podman.conf and appended at the bottom

R! /tmp/storage-run-*/containers/
R! /tmp/storage-run-*/libpod/tmp/

This should persist if podman is updated and can happily report our service now withstands a reboot.

Hi,
I tried to configure this in the image I am building, but the path /etc/tmpfiles.d does not exist.
How can I handle this in the image build? :\

[beeritis]

@shay1197 - This isn't within an image, this is on the host itself (i.e. where podman is installed).

[shay1197]

Yes, but I am running on Kubernetes, so I create the path and file during the image build process.
when the pod is up I can see the folder with this file:

x /tmp/podman-run-*
R! /tmp/storage-run-*
x /tmp/containers-user-*
x /tmp/run-/libpod
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images

[beeritis]

Oh , interesting use case!

Can't say I've run podman within Kubernetes before :). I would assume you still get a unique boot id when the pod is run?

Sounds a bit tricky as tmpfiles.d is related to the systemd-tmpfiles/ systemd-tmpfiles-clean.service which clears out the files and I don't recall an instance where systemd is being utilised within a container.

If a boot ID doesn't change then it might not be an issue but otherwise someone else can hopefully answer that question