Warewulfv4 container read-only after dnf update -y

[Kostusas]

Hi everyone,

I followed the building recipe on Warewulf4/SLURM Edition for Rocky 9 (x86 64), but I encountered a problem in Section 3.8. Specifically after running the command:

wwctl container exec rocky-9.4 /bin/bash <<- EOF
dnf -y install http://repos.openhpc.community/OpenHPC/3/EL_9/x86_64/ohpc-release-3-1.el9.x86_64.rpm
dnf -y update
EOF

The container goes into read-only mode, and I cannot run subsequent commands successfully nor enable systemd services. This seems to be related to a recent Warewulf4 issue: warewulf/warewulf#1567.

Is there perhaps some temporary workaround?

[adrianreber]

Thanks for the report. This was also mentioned on one of the OpenHPC slack channel. We tested with 9.4 and it was not happening there. Seems to be some 9.5 change.

@MiddelkoopT do you know if there is an easy fix for this? Some chmod command?

Same question for @anderbubble: Is there a workaround or do we need the patch from warewulf/warewulf#1564

[Kostusas]

Hi @adrianreber, thanks for your response.

Just to clarify, I am running Rocky 9.4:

# cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

[mslacken]

You can always make the container directory writeable. The read only containers in warewulf are just a 'hidden' feature. The idea there is to mark container as ro so that these can only be update via a not yet implemented wwcti container update command.
So

chmod +w /var/lib/warewulf/chroots/$CONTAINERNAME/rootfs/

is always safe.

[adrianreber]

Hi @adrianreber, thanks for your response.

Just to clarify, I am running Rocky 9.4:

# cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

Yes, but if you run dnf update it will update to 9.5.

[adrianreber]

Thanks @mslacken. @Kostusas can you try that workaround?

[anderbubble]

@adrianreber

warewulf/warewulf#1564 Hasn’t been merged yet. If/when it is, how would you want/need that delivered to you? Would it be best to just incorporate the patch locally in the package?

[adrianreber]

If there is an easy way around I would just document it somewhere. What @mslacken said sounded doable I think.

If we need to do a fix up release, I would just take your patch and apply it for now. Also something to have a discussion in our technical steering committee.

We also have a problem that the 9.5 EL versions pull in selinux policies which also breaks booting the compute nodes currently. Personally I would not do a release if we have documented workarounds for now. Let's see what the TSC thinks.

[anderbubble]

Regarding the selinux issue, so far the feedback I've gotten from a contact in the Rocky community is that we should just remove the selinux packages as part of the container build process. I intend to do this for the images at https://github.com/warewulf/warewulf-node-images.

It's also entirely possible to boot an selinux-enabled container image with Warewulf; there are just more steps involved.

https://warewulf.org/docs/main/contents/security.html#selinux

Here's a patch for the read-only container issue that applies to v4.5.5, which appears to be the version in OHPC 3.2.1.

read-only-sentinel.patch

[anderbubble]

chmod +w /var/lib/warewulf/chroots/$CONTAINERNAME/rootfs/

Do chmod u+w.

[Kostusas]

Thanks everyone for your help.

I confirm that running:

chmod u+w /srv/warewulf/chroots/rocky-9.4/rootfs/

after dnf update -y fixes the issue.

[adrianreber]

Thanks everyone for your help.

I confirm that running:

chmod u+w /srv/warewulf/chroots/rocky-9.4/rootfs/

after dnf update -y fixes the issue.

Thanks for confirming. We will discuss this in the OpenHPC technical steering committee.

[anderbubble]

Be advised: warewulf/warewulf#1564 has been merged to Warewulf main now, matching read-only-sentinel.patch.

I've added the next TSC meeting to my calendar, and plan to be there, in case there's any follow-up questions or requests for Warewulf.

[adrianreber]

I've added the next TSC meeting to my calendar, and plan to be there, in case there's any follow-up questions or requests for Warewulf.

Thank you.

[MiddelkoopT]

The Rocky 9 image has been upgraded to Rocky 9.5 and that should fix the issue. Please confirm.