Implement the `bootc` provision plugin #3161

ckyrouac · 2024-08-21T20:16:45Z

This creates a new provision plugin that is built on top of the existing TestCloud (virtual) plugin. It adds new parameters to pass a Containerfile or container image. The plugin will then build a container image (if necessary) then build a bootc disk image from the container image using bootc image builder. Currently, bootc requires podman to be run as root when building a disk image. This is typically handled by running a podman machine as root.

An additional parameter "add-deps" toggles building a derived container image with the tmt test requirements.

Resolves #3013

Pull Request Checklist

This is a work in progress. I'm opening this PR early to get feedback on the high level design. I will add tests, docs, etc. after we solidify the higher level design.

If you want to try running the code here is an example fmf plan:

provision:
  how: bootc
  containerimage: quay.io/centos-bootc/centos-bootc:stream9
  disk: 20
summary: Testing bootc plugin
execute:
  how: tmt
  script: |
    echo "ok"

tmt/steps/provision/bootc.py

ckyrouac · 2024-08-21T20:20:47Z

@cgwalters fyi

tmt/steps/provision/bootc.py

cgwalters

Mostly a skim (I don't know the tmt codebase either) but looks sane to me!

tmt/steps/provision/bootc.py

happz · 2024-08-22T07:49:01Z

LGTM, I like the reuse of existing plugins, that's very nice.
Would it also make sense to support the container plugin? IIUIC, the image produced by podman can be run both with podman run, or, converted to qcow2, as a VM. I could imagine both ways being available to use, maybe through some runtime-plugin: container|virtual switch. Is it a bad idea?
I will have more low-level comments related to the actual implementation, but I won't bother you now till you receive the high-level review. Just ping when you're ready for the boring stuff :)

martinhoyer

I'm still trying to speedrun reading/learning all things bootc (and tmt provisioning plugins), but fwiw, looks cool to me.

@happz about the container plugin support - Fedora docs says:

for fully-fledged tests it is not recommended to run a bootable container via, for instance, podman-run. One reason among others is that the filesystem is writable when being executed as an OCI container while most of the filesystem is mounted read-only on a deployed bootc system. That means the running container behaves differently than a deployed system. Yet, if you desire to run some quick tests it is recommended to run the container in detached mode.

From what I understand, podman-bootc could be pretty cool to use, once available.

happz · 2024-08-22T12:20:00Z

I'm still trying to speedrun reading/learning all things bootc (and tmt provisioning plugins), but fwiw, looks cool to me.

@happz about the container plugin support - Fedora docs says:

for fully-fledged tests it is not recommended to run a bootable container via, for instance, podman-run. One reason among others is that the filesystem is writable when being executed as an OCI container while most of the filesystem is mounted read-only on a deployed bootc system. That means the running container behaves differently than a deployed system. Yet, if you desire to run some quick tests it is recommended to run the container in detached mode.

From what I understand, podman-bootc could be pretty cool to use, once available.

I agree, it's not a perfect 1:1 substitution, but, exactly: for quick tests or basic test development, it may give me results faster than VM. I for one work on binutils and C/C++ toolchain in general, and my area of focus is fairly simple - compile this, run objdump on that, grep for expected section names, this kind of stuff. Learning about a typo in my reproducer quickly is very valuable, and once I'm done, I can always use full VM. I do it today already: I develop tests with container, then I switch to beaker or 1minutetip to get a more real environment before committing the new test to git. So, podman-run, for my trivial component, would be very welcome, even with caveats like the one you mentioned :)

martinhoyer · 2024-08-22T13:30:35Z

I agree, it's not a perfect 1:1 substitution, but, exactly: for quick tests or basic test development, it may give me results faster than VM. I for one work on binutils and C/C++ toolchain in general, and my area of focus is fairly simple - compile this, run objdump on that, grep for expected section names, this kind of stuff. Learning about a typo in my reproducer quickly is very valuable, and once I'm done, I can always use full VM. I do it today already: I develop tests with container, then I switch to beaker or 1minutetip to get a more real environment before committing the new test to git. So, podman-run, for my trivial component, would be very welcome, even with caveats like the one you mentioned :)

Thank you for the insight in your development process. I hope I can see it in more detail one day.

ckyrouac · 2024-08-22T13:52:45Z

Would it also make sense to support the container plugin? IIUIC, the image produced by podman can be run both with podman run, or, converted to qcow2, as a VM. I could imagine both ways being available to use, maybe through some runtime-plugin: container|virtual switch. Is it a bad idea?

I think the existing container plugin will handle this case without any additional code. The bootc image is just another container that can be run like a typical image.

lukaszachy

--add-deps doesn't install require/recommend yet, right?

Either way, we have chicken&egg problem in the case 'dist-git-source' is used as the list of packages is known after provision :/ But IMO we can ignore that for now to have something working.

ckyrouac · 2024-09-10T14:01:57Z

@happz This is ready for a review. I added some docs, tests, and code to cleanup the container images.

tmt/steps/provision/bootc.py

happz · 2024-11-06T15:29:55Z

/packit build

tmt.spec

happz · 2024-11-06T16:05:48Z

/packit build

ckyrouac · 2024-11-11T15:10:02Z

The tests are failing with Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory. Any ideas why?

omertuc · 2024-11-11T15:15:58Z

The tests are failing with Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory. Any ideas why?

I ran into this before locally while trying to run the bootc tmt tests, I solved it with systemctl enable virtqemud.socket

happz · 2024-11-11T15:24:37Z

The tests are failing with Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory. Any ideas why?

Tests depend on additional setup, shouldn't we add a dedicated plan & let it run in the provision check? IIUIC, that's what we did with provision-virtual tests, they are now grouped under /plans/provision/virtual which tasks care of the necessary setup, starting libvirt is exactly one of the steps in plans/provision/virtual.fmf, and since bootc provisioning plugin depends on an extra setup as well, we should isolate them in the same manner.

@psss does it make sense? It does look like another provision-foo plan is needed, and I think we can add it to the same Github check as virtual tests.

ckyrouac · 2024-11-11T16:40:25Z

@happz added a prepare step to the bootc plan. Could you trigger another test run?

psss · 2024-11-12T15:55:13Z

/packit build

psss · 2024-11-12T16:27:42Z

@psss does it make sense? It does look like another provision-foo plan is needed, and I think we can add it to the same Github check as virtual tests.

Yes, this definitely makes sense. Let's have this handled in a consistent way.

ckyrouac · 2024-11-12T18:00:40Z

If I understand this right, I believe this was already in place. The prepare step just wasn’t in place yet but is now.

Now the bootc tests pass but there is an avc failure that I don’t understand. Any ideas?

This creates a new provision plugin that is built on top of the existing TestCloud (virtual) plugin. It adds new parameters to pass a Containerfile or container image. The plugin will then build a container image (if necessary) then build a bootc disk image from the container image using bootc image builder. Currently, bootc requires podman to be run as root when building a disk image. This is typically handled by running a podman machine as root. An additional parameter "add-deps" toggles building a derived container image with the tmt test requirements. Signed-off-by: Chris Kyrouac <[email protected]>

Signed-off-by: Chris Kyrouac <[email protected]>

When the podman connection is rootless=True, this will automatically start a new rootful podman-machine to be used for the bootc disk creation. Signed-off-by: Chris Kyrouac <[email protected]>

Ensures the /var/tmp/tmt directory exists before creating the temp directories. This directory might not exist in the CI environment. It usually exists when running locally. Signed-off-by: Chris Kyrouac <[email protected]>

Signed-off-by: Chris Kyrouac <[email protected]>

happz · 2024-11-12T18:20:01Z

If I understand this right, I believe this was already in place. The prepare step just wasn’t in place yet but is now.

Now the bootc tests pass but there is an avc failure that I don’t understand. Any ideas?

A rebase should help, the problem has been addressed in #3355

happz · 2024-11-12T18:20:10Z

/packit build

martinhoyer · 2024-11-13T14:48:17Z

If I understand this right, I believe this was already in place. The prepare step just wasn’t in place yet but is now.

Now the bootc tests pass but there is an avc failure that I don’t understand. Any ideas?

I'm guessing it's the usage of /var, nosuid transition:
https://danwalsh.livejournal.com/78312.html

I can see container-selinux package being installed in the log btw. Maybe a bug there on in podman-machine package? Not sure how to solve this. @psss @happz

happz · 2024-11-13T15:13:13Z

If I understand this right, I believe this was already in place. The prepare step just wasn’t in place yet but is now.
Now the bootc tests pass but there is an avc failure that I don’t understand. Any ideas?

I'm guessing it's the usage of /var, nosuid transition: https://danwalsh.livejournal.com/78312.html

I can see container-selinux package being installed in the log btw. Maybe a bug there on in podman-machine package? Not sure how to solve this. @psss @happz

It has been reported for bootc image builder, see osbuild/bootc-image-builder#645. It does look like something bootc either should not do, or ask selinux-policy maintainers to allow.

cgwalters · 2024-11-13T17:11:13Z

Hmm that probably relates to our initialization of containers/storage at bootc install time. But I am confused as to why we may only be seeing this here. It may relate somehow to the host policy? We do have some coverage for this in other CI tests… Long story short if this theory is right then we need to suppress the domain transition or maybe just switch to skopeo which we probably want to do anyways.

martinhoyer · 2024-11-14T15:33:26Z

Hmm that probably relates to our initialization of containers/storage at bootc install time. But I am confused as to why we may only be seeing this here. It may relate somehow to the host policy? We do have some coverage for this in other CI tests… Long story short if this theory is right then we need to suppress the domain transition or maybe just switch to skopeo which we probably want to do anyways.

Thanks,
@cgwalters @ckyrouac Would you prefer merging it "as is", perhaps like a tech preview or addressing the selinux issue first, most likely sliding to the next release cycle?

ckyrouac · 2024-11-14T18:56:31Z

I will be afk on paternity leave soon (most likely in a week), so I won't be able to work on this much for a few months. So, unless someone else can pick up the work, I think merging as-is makes the most sense.

martinhoyer · 2024-11-14T19:11:47Z

I will be afk on paternity leave soon (most likely in a week)

Congratulations in advance sir!

psss · 2024-11-15T12:45:34Z

So, unless someone else can pick up the work, I think merging as-is makes the most sense.

Ok, then I suggest to mark the avc failure as expected for now and include this in 1.39 as a tech preview. I've added a short release not in 44b0cd3. Does it look ok?

psss

Added a couple of comments. I tried the minimal example but ended up with the following error:

Command 'podman machine start podman-machine-tmt' returned 125.

# stdout (1/1 lines)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting machine "podman-machine-tmt"
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# stderr (2/2 lines)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
time="2024-11-15T14:33:33+01:00" level=error msg="process 419571 has not ended"
Error: failed to find virtiofsd: exec: "virtiofsd": executable file not found in $PATH
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I tried this under a regular user. Do I understand it correctly that it is necessary to always run this under root?

psss · 2024-11-15T12:48:40Z

tmt/steps/provision/bootc.py

+
+        provision:
+            how: bootc
+            containerimage: quay.io/fedora/fedora-bootc:40


Throughout the tmt specification we consistently use dashes to separate words in multi-word keys. Would it be ok if we used container-image here? Similar for other multiword keys.

psss · 2024-11-15T12:49:20Z

tmt/steps/provision/bootc.py

+            containerfile: "./my-custom-image.containerfile"
+            containerfile-workdir: .
+            image_builder: quay.io/centos-bootc/bootc-image-builder:stream9


Suggested change

containerfile: "./my-custom-image.containerfile"

containerfile-workdir: .

image_builder: quay.io/centos-bootc/bootc-image-builder:stream9

container-file: "./my-custom-image.containerfile"

container-file-workdir: .

image-builder: quay.io/centos-bootc/bootc-image-builder:stream9

psss · 2024-11-15T12:50:57Z

tmt/steps/provision/bootc.py

+
+        provision:
+            how: bootc
+            add_deps: false


Suggested change

add_deps: false

add-tmt-dependencies: true

Perhaps it would be good to make the name a bit more specific? And I guess the example should use true?

psss · 2024-11-15T13:08:05Z

plans/provision/bootc.fmf

@@ -0,0 +1,39 @@
+summary: Bootc virtual machine via testcloud


This plan is currently run under the full packit job. Seems to be working fine, but this means that it is executed for all targets. Should we move it under the provision packit job? @happz, I guess that's what you meant in your comment? Or are we good as it is now?

psss · 2024-11-15T13:15:50Z

tests/provision/bootc/test.sh

+    rlPhaseStartTest "Image that already includes dependencies"
+        rlRun "podman build . -f $CONTAINERFILE_INCLUDES_DEPS -t $IMAGE_INCLUDES_DEPS"
+        rlRun "cp $IMAGE_INCLUDES_DEPS_PLAN ."
+        rlRun "tmt -vvvvv run -i $run"


If reusing the same run --id it's better to use tmt run --scratch to remove previous run content before executing a new run. Otherwise this results into following tmt runs to just report done.

psss · 2024-11-15T13:21:04Z

tests/provision/bootc/test.sh

+rlJournalStart
+    rlPhaseStartSetup
+        # cleanup previous runs
+        test -d /var/tmp/tmt/testcloud && rlRun "rm -rf /var/tmp/tmt/testcloud"


If we need to do a complete cleanup, what about using TMT_WORKDIR_ROOT instead to choose a different workdir root? We're trying to write tests so that they can be easily/safely executed on user laptop as well. This would remove for example all fetched images.

psss · 2024-11-15T13:23:06Z

tmt.spec

+Provides:       tmt-provision-bootc == %{version}-%{release}
+%if 0%{?fedora} < 40
+Obsoletes:      tmt-provision-bootc < %{version}-%{release}
+%endif


There was no tmt-provision-bootc package before, so I think we can drop the Obsoletes here.

psss · 2024-11-15T13:27:12Z

tmt/steps/provision/bootc.py

+                ).run(cwd=self.workdir, stream_output=True, logger=self._logger)
+        except BaseException:
+            self._logger.debug(
+                "Unable to remove podman machine {PODMAN_MACHINE_NAME}, it might not exist")


Suggested change

"Unable to remove podman machine {PODMAN_MACHINE_NAME}, it might not exist")

"Unable to remove podman machine '{PODMAN_MACHINE_NAME}', it might not exist.")

ckyrouac requested review from psss, lukaszachy, happz, thrix and janhavlin as code owners August 21, 2024 20:16

ckyrouac marked this pull request as draft August 21, 2024 20:16

ckyrouac commented Aug 21, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

ckyrouac commented Aug 21, 2024

View reviewed changes

tmt/steps/provision/bootc.py Show resolved Hide resolved

cgwalters reviewed Aug 21, 2024

View reviewed changes

tmt/steps/provision/bootc.py Show resolved Hide resolved

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

martinhoyer mentioned this pull request Aug 22, 2024

tmt cannot run tests on Fedora CoreOS containers #2884

Open

martinhoyer reviewed Aug 22, 2024

View reviewed changes

ckyrouac force-pushed the bootc-provision branch from 641a411 to 78e86f3 Compare August 22, 2024 13:47

lukaszachy reviewed Aug 28, 2024

View reviewed changes

ckyrouac force-pushed the bootc-provision branch 2 times, most recently from e3058f9 to 44ff728 Compare September 10, 2024 14:00

ckyrouac marked this pull request as ready for review September 10, 2024 14:00

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Sep 10, 2024

View reviewed changes

tmt/steps/provision/bootc.py Outdated Show resolved Hide resolved

happz reviewed Nov 6, 2024

View reviewed changes

tmt.spec Show resolved Hide resolved

ckyrouac force-pushed the bootc-provision branch from 7d63c51 to 1b06357 Compare November 6, 2024 16:02

ckyrouac force-pushed the bootc-provision branch from 8fc5094 to 125f5d6 Compare November 11, 2024 16:39

ckyrouac force-pushed the bootc-provision branch from 125f5d6 to 98b8349 Compare November 12, 2024 18:16

ckyrouac added 5 commits November 12, 2024 13:16

bootc: Add schema for bootc plugin

400aea0

Signed-off-by: Chris Kyrouac <[email protected]>

bootc: Automatically start podman-machine

a9071ab

When the podman connection is rootless=True, this will automatically start a new rootful podman-machine to be used for the bootc disk creation. Signed-off-by: Chris Kyrouac <[email protected]>

bootc: Make temp directory in test setup

b2bc58f

Ensures the /var/tmp/tmt directory exists before creating the temp directories. This directory might not exist in the CI environment. It usually exists when running locally. Signed-off-by: Chris Kyrouac <[email protected]>

bootc: Start libvirtd during prepare step of tests

9d4d5f2

Signed-off-by: Chris Kyrouac <[email protected]>

ckyrouac force-pushed the bootc-provision branch from 98b8349 to 9d4d5f2 Compare November 12, 2024 18:16

Add a short release not, mark avc failure as expected

44b0cd3

psss requested changes Nov 15, 2024

View reviewed changes

		@@ -0,0 +1,39 @@
		summary: Bootc virtual machine via testcloud

	"Unable to remove podman machine {PODMAN_MACHINE_NAME}, it might not exist")
	"Unable to remove podman machine '{PODMAN_MACHINE_NAME}', it might not exist.")

Implement the bootc provision plugin #3161

Are you sure you want to change the base?

Implement the bootc provision plugin #3161

Conversation

ckyrouac commented Aug 21, 2024 • edited by psss Loading

ckyrouac commented Aug 21, 2024

cgwalters left a comment

Choose a reason for hiding this comment

happz commented Aug 22, 2024

martinhoyer left a comment

Choose a reason for hiding this comment

happz commented Aug 22, 2024

martinhoyer commented Aug 22, 2024

ckyrouac commented Aug 22, 2024

lukaszachy left a comment

Choose a reason for hiding this comment

ckyrouac commented Sep 10, 2024

happz commented Nov 6, 2024

happz commented Nov 6, 2024

ckyrouac commented Nov 11, 2024

omertuc commented Nov 11, 2024 • edited Loading

happz commented Nov 11, 2024

ckyrouac commented Nov 11, 2024

psss commented Nov 12, 2024

psss commented Nov 12, 2024

ckyrouac commented Nov 12, 2024

happz commented Nov 12, 2024

happz commented Nov 12, 2024

martinhoyer commented Nov 13, 2024

happz commented Nov 13, 2024

cgwalters commented Nov 13, 2024 via email

martinhoyer commented Nov 14, 2024

ckyrouac commented Nov 14, 2024

martinhoyer commented Nov 14, 2024

psss commented Nov 15, 2024

psss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Implement the `bootc` provision plugin #3161

Implement the `bootc` provision plugin #3161

ckyrouac commented Aug 21, 2024 •

edited by psss

Loading

omertuc commented Nov 11, 2024 •

edited

Loading