Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests/kola: adding the external kola test for luks + multipath case #3124

Open
wants to merge 1 commit into
base: testing-devel
Choose a base branch
from

Conversation

Poorna-Gottimukkula1
Copy link

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 commented Aug 27, 2024

Adding the external kola test for luks + multipath validation
ref: https://issues.redhat.com/browse/OCPBUGS-13123
coreos/fedora-coreos-tracker#1728

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 changed the title tests/kola:: Adding the external kola test for luks + multipath case tests/kola: adding the external kola test for luks + multipath case Aug 27, 2024
@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from a305f0c to 178e930 Compare August 28, 2024 09:37
@jlebon
Copy link
Member

jlebon commented Aug 28, 2024

CI is hitting:

boot mounted not by UUID

We need to update the logic here to instead be something like (pseudo-code):

if we have rd.multipath=default karg; then
  verify that boot.mount's What is /dev/disk/by-label/dm-mpath-boot
else if ... (same logic as before)
fi

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 178e930 to a19c7d0 Compare August 29, 2024 10:50
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the original bug was against PowerPC, can you verify that it also passes there?

Comment on lines 33 to 44
if grep -q "rd.multipath=default" /proc/cmdline; then
if ! systemctl cat boot.mount | grep -q "What=/dev/disk/by-label/dm-mpath-boot"; then
systemctl cat boot.mount
fatal "boot is not mounted by /dev/disk/by-label/dm-mpath-boot"
fi
ok "boot mounted from /dev/disk/by-label/dm-mpath-boot"
else
if ! systemctl cat boot.mount | grep -q "What=/dev/disk/by-uuid"; then
systemctl cat boot.mount
fatal "boot is not mounted by UUID as expected"
fi
ok "boot mounted by UUID"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good candidate to dedupe. E.g. something like

if grep ...; then
  expected_what=/dev/disk/by-label/dm-mpath-boot
else
  expected_what=/dev/disk/by-uuid
fi
if ! systemctl cat ...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the changes as per the above suggestion.

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from a19c7d0 to dd5c4e9 Compare September 2, 2024 09:27
@Poorna-Gottimukkula1
Copy link
Author

Since the original bug was against PowerPC, can you verify that it also passes there?

Yes, I have tested it on PowerPC and it got passed.

Sep 02 09:22:37 qemu0 kola-runext-test.sh[4044]: + reboot
-- Boot b4d12cf821e64eb6a0f4eedc164611f4 --
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1696]: ++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + parent_device=/dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1697]: + lsblk -pno NAME /dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1698]: + grep -qw /dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/luks-test.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1699]: +++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1700]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ blktype=crypt
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ crypt == crypt ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1701]: +++ findmnt -nvr /sysroot -o FSTYPE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ fstype=xfs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ xfs == xfs ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok source is XFS on LUKS device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1702]: +++ findmnt /sysroot -no OPTIONS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1703]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok root mounted with prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1704]: +++ dmsetup table myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ table='0 20137985 crypt aes-xts-plain64 :64:logon:cryptsetup:0680a9c4-9392-4038-a76d-7c932e386fa3-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q allow_discards
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q no_read_workqueue
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok discard and custom option enabled for root LUKS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q rd.multipath=default /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1708]: ++ systemctl cat boot.mount
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1709]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep root=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1710]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep rd.luks.name= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1711]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found root kargs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep boot=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1712]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found boot karg

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from dd5c4e9 to d52cfb9 Compare September 2, 2024 14:45
jlebon
jlebon previously approved these changes Sep 3, 2024
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment, but LGTM as is too.
Thanks for working on this!

## kola:
## # This test reprovisions the rootfs.
## tags: "reprovision"
## # This uses additionalDisks, which is QEMU only
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is incorrect. It's more like:

Suggested change
## # This uses additionalDisks, which is QEMU only
## # This uses appendKernelArgs and multipath, which is QEMU only

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, updated it.

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from 4c39fda to 79a976b Compare September 4, 2024 09:56
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Just one final sanity-check: did you test this on both RHCOS and FCOS?

@Poorna-Gottimukkula1
Copy link
Author

Poorna-Gottimukkula1 commented Sep 4, 2024

Just one final sanity-check: did you test this on both RHCOS and FCOS?

I tested this only on FCOS as the RHCOS having issues in boot(found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.

Shell i build the RHCOS image with custom coreos-installer and try this out.?

@jlebon
Copy link
Member

jlebon commented Sep 4, 2024

Just one final sanity-check: did you test this on both RHCOS and FCOS?

I tested this only on FCOS as the RHCOS having issues in boot(found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.

Ahh indeed. We're still on v0.21.0 there for some reason. Will inquire about that.

@jlebon jlebon added the hold label Sep 9, 2024
@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from 67b6c23 to 74efa3f Compare December 3, 2024 06:14
@Poorna-Gottimukkula1
Copy link
Author

Poorna-Gottimukkula1 commented Dec 3, 2024

I just tested the RHCOS with latest build and it got booted and the external kola luks+multipath test case got passed.

[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (170.96s)
PASS, output in tmp/kola

RHCOS kola test logs forluks+multipath:

cat tmp/kola/ext.config.root-reprovision.luks.multipath/76350551-f1ca-46d3-b0ee-adb7ab3ec484/ext.config.root-reprovision.luks.multipath.txt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3753]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + parent_device=/dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3755]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3754]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3757]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3758]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ blktype=crypt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ crypt == crypt ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3759]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ fstype=xfs
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ xfs == xfs ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok source is XFS on LUKS device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3760]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3761]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok root mounted with prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3762]: +++ dmsetup table myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q allow_discards
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q no_read_workqueue
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok discard and custom option enabled for root LUKS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3767]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3766]: ++ systemctl cat boot.mount
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ '[' '!' -e /run/ignition-ostree-growfs.stamp ']'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ /tmp/autopkgtest-reboot rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + /usr/local/bin/kolet reboot-request rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + reboot
-- Boot b883d68b3bec408780c3901ca29f539e --
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1566]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + parent_device=/dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1568]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1567]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1570]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1571]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ blktype=crypt
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ crypt == crypt ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1572]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ fstype=xfs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ xfs == xfs ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok source is XFS on LUKS device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1573]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1574]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok root mounted with prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1575]: +++ dmsetup table myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q allow_discards
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q no_read_workqueue
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok discard and custom option enabled for root LUKS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1580]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1579]: ++ systemctl cat boot.mount
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep root=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1581]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep rd.luks.name= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1582]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found root kargs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep boot=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1583]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found boot karg

However FCOS, which was previously functional, is now failing to boot as it is throwing the below error.

[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (72.18s)
        harness.go:1823: mach.Start() failed: machine aa6ae511-d5c1-421e-b957-8c3276a31782 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-03T09:49:36Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1

FCOS console logs :

[..snip..]
[   28.592836] ignition[1596]: disks: createLuks: op(3): [started]  creating "myluksdev"
[   34.831451] ignition[1596]: disks: createLuks: op(3): [finished] creating "myluksdev"
[   34.831931] ignition[1596]: disks: createLuks: op(4): [started]  opening luks device myluksdev
[   37.546373] ignition[1596]: disks: createLuks: op(4): [finished] opening luks device myluksdev
[   37.549355] ignition[1596]: disks: createLuks: op(5): [started]  Clevis bind
[   42.681896] ignition[1596]: disks: createLuks: op(5): [finished] Clevis bind
[   42.682286] ignition[1596]: disks: createLuks: op(6): [started]  closing clevis luks device myluksdev
[   42.750229] ignition[1596]: disks: createLuks: op(6): [finished] closing clevis luks device myluksdev
[   42.750533] ignition[1596]: disks: createLuks: op(7): [started]  reopening clevis luks device myluksdev
[   45.570754] ignition[1596]: disks: createLuks: op(7): [finished] reopening clevis luks device myluksdev
[   45.571009] ignition[1596]: disks: createLuks: op(8): [started]  removing key file for myluksdev
[   47.850124] ignition[1596]: disks: createLuks: op(8): [finished] removing key file for myluksdev
[   47.850353] ignition[1596]: disks: createLuks: op(9): [started]  waiting for triggered uevent
[   47.950205] ignition[1596]: disks: createLuks: op(9): [finished] waiting for triggered uevent
[   47.950423] ignition[1596]: disks: createFilesystems: op(a): [started]  waiting for devices [/dev/mapper/myluksdev]
[   47.978127] ignition[1596]: disks: createFilesystems: op(a): [finished] waiting for devices [/dev/mapper/myluksdev]
[   47.978366] ignition[1596]: disks: createFilesystems: created device alias for "/dev/mapper/myluksdev": "/run/ignition/dev_aliases/dev/mapper/myluksdev" -> "/dev/dm-5"
[   47.978550] ignition[1596]: disks: createFilesystems: op(b): [started]  determining filesystem type of "/dev/mapper/myluksdev"
[   48.084046] ignition[1596]: disks: createFilesystems: op(b): [finished] determining filesystem type of "/dev/mapper/myluksdev"
[   48.084285] ignition[1596]: disks: createFilesystems: found  filesystem at "/dev/mapper/myluksdev" with uuid "" and label ""
[   48.084450] ignition[1596]: disks: createFilesystems: op(c): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.129081] ignition[1596]: disks: createFilesystems: op(c): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.129320] ignition[1596]: disks: createFilesystems: op(d): [started]  creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.901003] ignition[1596]: disks: createFilesystems: op(d): [finished] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.901450] ignition[1596]: disks: createFilesystems: op(e): [started]  waiting for triggered uevent
[   49.014476] ignition[1596]: disks: createFilesystems: op(e): [finished] waiting for triggered uevent
[   49.014740] ignition[1596]: disks: disks passed
[   49.014956] ignition[1596]: Ignition finished successfully
[   49.022491] systemd[1]: Finished ignition-disks.service - Ignition (disks).
[   49.022944] systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[   49.024331] systemd[1]: Starting coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem...
[   49.556492] systemd[1]: Finished coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem.
[   49.558446] systemd[1]: ignition-ostree-uuid-root.service - Ignition OSTree: Regenerate Filesystem UUID (root) was skipped because of an unmet condition check (ConditionPathExists=!/run/ignition-ostree-transposefs).
[   49.562012] systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
[   49.588122] ignition-ostree-growfs[2335]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[   49.671835] ignition-ostree-growfs[2370]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[   49.672049] ignition-ostree-growfs[2370]:        dmesg(1) may have more information after failed mount system call.
[   49.672639] systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
[   49.673019] systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
[   49.673442] systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
[   49.673696] systemd[1]: Dependency failed for ignition-complete.target - Ignition Complete.
[   49.673868] systemd[1]: Dependency failed for initrd.target - Initrd Default Target.
[   49.674362] systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
[   49.674488] systemd[1]: initrd.target: Triggering OnFailure= dependencies.
[   49.674965] systemd[1]: ignition-complete.target: Job ignition-complete.target/start failed with result 'dependency'.
[..snip..]

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 74efa3f to 8fc820b Compare December 4, 2024 16:33
@jlebon
Copy link
Member

jlebon commented Dec 4, 2024

[   47.210473] ignition-ostree-growfs[2382]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[   47.267611] ignition-ostree-growfs[2417]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[   47.269927] ignition-ostree-growfs[2417]:        dmesg(1) may have more information after failed mount system call.

Hmm, definitely odd. This might be some weird udev issue. And this used to pass... so possibly some race that became more racy? Not sure.

I would try to reproduce this manually (do a cosa run -c, passing in the same Ignition config, and --qemu-multipath). Then you can poke around a bit in the emergency shell. E.g. lsblk would be useful.

@Poorna-Gottimukkula1 Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 8fc820b to e42329b Compare December 9, 2024 09:46
@Poorna-Gottimukkula1
Copy link
Author

Poorna-Gottimukkula1 commented Dec 9, 2024

I have manually reproduced this issue using below commands with the same ignition file.

# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2  --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -i ../test.ign

[..snip..]
Dec 09 10:22:02 systemd[1]: Failed to mount sysroot.mount - /sysroot.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
Dec 09 10:22:01 ignition-ostree-growfs[3826]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-growfs[3861]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-growfs[3861]:        dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Restoring rootfs from RAM...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Mounting /dev/disk/by-label/root rw (/dev/dm-4) to /sysroot
Dec 09 10:22:01 ignition-ostree-transposefs[3906]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-transposefs[3906]:        dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Triggering OnFailure= dependencies.

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


Press Enter for maintenance
(or press Control-D to continue):
:/root# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda               8:0    0   10G  0 disk
|-sda1            8:1    0    4M  0 part
|-sda2            8:2    0    1M  0 part
|-sda3            8:3    0  384M  0 part
|-sda4            8:4    0  2.2G  0 part
`-mpatha        253:0    0   10G  0 mpath
  |-mpatha1     253:1    0    4M  0 part
  |-mpatha2     253:2    0    1M  0 part
  |-mpatha3     253:3    0  384M  0 part
  `-mpatha4     253:4    0  2.2G  0 part
    `-myluksdev 253:5    0  2.2G  0 crypt
sdb               8:16   0   10G  0 disk
|-sdb1            8:17   0    4M  0 part
|-sdb2            8:18   0    1M  0 part
|-sdb3            8:19   0  384M  0 part
|-sdb4            8:20   0  2.2G  0 part
`-mpatha        253:0    0   10G  0 mpath
  |-mpatha1     253:1    0    4M  0 part
  |-mpatha2     253:2    0    1M  0 part
  |-mpatha3     253:3    0  384M  0 part
  `-mpatha4     253:4    0  2.2G  0 part
    `-myluksdev 253:5    0  2.2G  0 crypt

However, it is booted successfully with the below config.

# cat tmp.bu
variant: fcos
version: 1.4.0
boot_device:
  luks:
    tpm2: true
---
# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2  --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -B tmp.bu
[..snip..]
[   80.761209] systemd[1]: Starting systemd-hwdb-update.service - Rebuild Hardware Database...
[   80.764083] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[   80.769153] systemd[1]: Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
[   80.772613] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.

Fedora CoreOS 41.20241209.dev.0
Kernel 6.11.10-300.fc41.ppc64le on an ppc64le (hvc0)

SSH host key: SHA256:uJ/jYOQwgeH8/i/fjqY69mLbhr17zz28uRb7C9nwvKE (ECDSA)
SSH host key: SHA256:6jp/Kz7HX6HI1dpjQ2FeleL3iIvDr5QFhAS2u1SahLA (ED25519)
SSH host key: SHA256:KDG4F8xLPoR+nzBs6tEccy27qN1TYMSjsCEXe9P49lo (RSA)
enp0s3: 10.0.2.15 fe80::bc4:f482:8844:c646
Ignition: ran on 2024/12/09 12:36:52 UTC (this boot)
Ignition: user-provided config was applied
No SSH authorized keys provided by Ignition or Afterburn
cosa-devsh login: core (automatic login)

Fedora CoreOS 41.20241209.dev.0
[core@cosa-devsh ~]$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  4.3M  1 loop
sda           8:0    0   10G  0 disk
└─mpatha    253:0    0   10G  0 mpath
  ├─mpatha1 253:1    0    4M  0 part
  ├─mpatha2 253:2    0    1M  0 part
  ├─mpatha3 253:3    0  384M  0 part  /boot
  └─mpatha4 253:4    0  9.6G  0 part
    └─root  253:5    0  9.6G  0 crypt /var
                                      /sysroot/ostree/deploy/fedora-coreos/var
                                      /etc
                                      /sysroot
sdb           8:16   0   10G  0 disk
└─mpatha    253:0    0   10G  0 mpath
  ├─mpatha1 253:1    0    4M  0 part
  ├─mpatha2 253:2    0    1M  0 part
  ├─mpatha3 253:3    0  384M  0 part  /boot
  └─mpatha4 253:4    0  9.6G  0 part
    └─root  253:5    0  9.6G  0 crypt /var
                                      /sysroot/ostree/deploy/fedora-coreos/var
                                      /etc
                                      /sysroot

@Poorna-Gottimukkula1
Copy link
Author

Poorna-Gottimukkula1 commented Dec 11, 2024

@dustymabe I have run the coreos-builds-bisect.py with stable and testing-devel builds and found the failure introduced build.

With stable builds

[coreos-assembler]$ cat test.sh
#!/bin/bash
set -eux -o pipefail
build=$1
cosa buildfetch --force --stream=stable --build=$build --artifact=qemu
cosa decompress --build=$build
cosa kola run --build=$build 'ext.config.root-reprovision.luks.multipath'

[coreos-assembler]$ chmod 777 test.sh
[coreos-assembler]$ cosa buildfetch --force --stream=stable
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/ostree-commit-object
[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241109.3.0 --good 40.20240920.3.0 --testscript=./test.sh
Using data file at: ./coreos-builds-bisect-data.json
Targetting architecture: ppc64le
Executing test for new build: 40.20241019.3.0. ~2 steps left
+ build=40.20241019.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241019.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.3.0
Targeting build: 40.20241019.3.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.3.0/ppc64le
2024-12-11 12:53:32,129 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.3.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (160.98s)
        harness.go:1823: mach.Start() failed: machine db54928d-6b77-4ee6-bd36-8e5b494c7f77 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T12:56:24Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.3.0 failed
Executing test for new build: 40.20241006.3.0. ~1 steps left
+ build=40.20241006.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241006.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241006.3.0
Targeting build: 40.20241006.3.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241006.3.0/ppc64le
2024-12-11 12:56:55,356 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241006.3.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241006.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241006.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (277.93s)
PASS, output in tmp/kola
40.20241006.3.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241006.3.0
First known bad build: 40.20241019.3.0
[coreos-assembler]$

With testing-devel builds:

[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241108.20.0 --good 40.20241013.20.0 --testscript=./test.sh
[..snip..]
Executing test for new build: 41.20241030.20.0. ~4 steps left
+ build=41.20241030.20.0
+ cosa buildfetch --force --stream=testing-devel --build=41.20241030.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=41.20241030.20.0
Targeting build: 41.20241030.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/41.20241030.20.0/ppc64le
2024-12-11 14:16:51,398 INFO - Running command: ['xz', '-dc', '-T16', 'builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/41.20241030.20.0/ppc64le/meta.json
+ cosa kola run --build=41.20241030.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 41.20241030.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (121.29s)
        harness.go:1823: mach.Start() failed: machine 22e6b73b-ae84-411f-ac32-7150350eb73a entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:19:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
41.20241030.20.0 failed
Executing test for new build: 40.20241019.20.0. ~3 steps left
+ build=40.20241019.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241019.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.20.0
Targeting build: 40.20241019.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.20.0/ppc64le
2024-12-11 14:19:39,005 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.20.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (131.85s)
        harness.go:1823: mach.Start() failed: machine a3a6a267-bc52-43ea-931d-cc161287c30f entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:22:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.20.0 failed
Executing test for new build: 40.20241017.20.0. ~1 steps left
+ build=40.20241017.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241017.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241017.20.0
Targeting build: 40.20241017.20.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241017.20.0/ppc64le
2024-12-11 14:22:39,418 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241017.20.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241017.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241017.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (272.80s)
PASS, output in tmp/kola
40.20241017.20.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241017.20.0
First known bad build: 40.20241019.20.0

@dustymabe
Copy link
Member

BISECT TEST RESULTS:
Last known good build: 40.20241017.20.0
First known bad build: 40.20241019.20.0

That represents this diff:

[coreos-assembler]$ cosa diff --from=40.20241017.20.0 --to=40.20241019.20.0 --rpms
Extracting c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
   Downloading...                                                                                                                                                                                                                          1
3134 metadata, 20030 content objects imported; 1.7 GB content written                                                 
Extracting b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
   Downloading...                                                                                                                                                                                                                          7
388 metadata, 2295 content objects imported; 593.7 MB content written                                                 
2024-12-11 19:35:53,684 INFO - Running command: ['rpm-ostree', 'db', 'diff', '--repo', 'tmp/repo', 'c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81', 'b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087']
ostree diff commit from: c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
ostree diff commit to:   b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
Upgraded:
  amd-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  atheros-firmware 20240909-1.fc40 -> 20241017-1.fc40
  brcmfmac-firmware 20240909-1.fc40 -> 20241017-1.fc40
  bsdtar 3.7.2-4.fc40 -> 3.7.2-7.fc40
  containers-common 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
  containers-common-extra 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
  device-mapper 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-event 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-event-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
  fwupd 1.9.25-1.fc40 -> 1.9.26-1.fc40
  intel-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  libarchive 3.7.2-4.fc40 -> 3.7.2-7.fc40
  libjcat 0.2.1-2.fc40 -> 0.2.2-1.fc40
  libuv 1:1.49.0-1.fc40 -> 1:1.49.1-1.fc40
  libxmlb 0.3.19-2.fc40 -> 0.3.21-1.fc40
  linux-firmware 20240909-1.fc40 -> 20241017-1.fc40
  linux-firmware-whence 20240909-1.fc40 -> 20241017-1.fc40
  lvm2 2.03.23-1.fc40 -> 2.03.25-1.fc40
  lvm2-libs 2.03.23-1.fc40 -> 2.03.25-1.fc40
  mt7xxx-firmware 20240909-1.fc40 -> 20241017-1.fc40
  nvidia-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  nxpwireless-firmware 20240909-1.fc40 -> 20241017-1.fc40
  qed-firmware 20240909-1.fc40 -> 20241017-1.fc40
  realtek-firmware 20240909-1.fc40 -> 20241017-1.fc40
  shadow-utils 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
  shadow-utils-subid 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
  tiwilink-firmware 20240909-1.fc40 -> 20241017-1.fc40

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants