Sync before reboot?

[aledbf]

Hello there! I have noticed some unexpected behavior during the reboot process. I have an ext4 disk image and without running sync before rebooting, I am experiencing some issues with the disk. Specifically, I am seeing detected issues with e2fsck and/or the disk being in an unexpected state with changes in files.
I am not sure if this is an expected behavior, as I have not found any references to it. However, I did come across the old Kata containers agent that seems to do that.
Any pointers on what I might be doing wrong?
Thanks!

Guest: Ubuntu 22.04/23.10
firecracker: 1.4.1/1.5.0
kernel parameters: console=ttyS0 reboot=k panic=1 pci=off random.trust_cpu=on acpi=off

[kalyazin]

Hi @aledbf . Thanks for reaching out to us.
Just to doublecheck, do you run the reboot command or something else to trigger a reboot?
Could you please share the guest OS output after that.

[aledbf]

@kalyazin thank you for the quick reply
These are the last lines I see

Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
All filesystems, swaps, loop devices, MD devices and DM devices detached.
Rebooting.
[   14.727726] reboot: Restarting system

which suggests the shutdown ran as expected

But if I run e2fsck I see

# e2fsck disk.img e2fsck 1.46.5 (30-Dec-2021)
disk.img: recovering journal
e2fsck: Invalid argument while recovering journal of disk.img
disk.img was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (266590, counted=237652).
Fix<y>? yes
Free inodes count wrong (142464, counted=138754).
Fix<y>? yes

disk.img : ***** FILE SYSTEM WAS MODIFIED *****
disk.img : 25086/163840 files (0.1% non-contiguous), 414082/651734 blocks

[kalyazin]

@aledbf Thanks for sharing. Another question, do you use a sync or async IO engine for the block device?

[aledbf]

sync

[kalyazin]

thanks, we'll be looking into reproducing that on our side.

[csweichel]

We are using sync, and we have also seen this issue with the async IO engine.

[pb8o]

Hi @aledbf , I have been working on a repro for this issue, but I couldn't reproduce in our integration tests. Could you point out what do you do differently??

from pathlib import Path

import pytest


def test_reboot_ext4_is_clean(microvm_factory, guest_kernel_linux_5_10, rootfs_rw):
    """Since Firecracker cannot reboot, we test it by running in 2 VMs:

    vm1: boot, create a file and reboot (which kills Firecracker)
    vm2: boot using vm1 ext4 rootfs, and check for the file
    """

    filename = "/I_was_here"

    vm = microvm_factory.build(guest_kernel_linux_5_10, rootfs_rw, monitor_memory=False)
    vm.spawn()
    vm.basic_config()
    vm.add_net_iface()
    vm.start()
    res = vm.ssh.run(f"[ ! -e {filename} ] && touch {filename}")
    print(res)
    res = vm.ssh.run("reboot")
    print(res)

    vm2 = microvm_factory.build(guest_kernel_linux_5_10, rootfs_rw, monitor_memory=False)
    vm2.rootfs_file = vm.rootfs_file
    vm2.spawn()
    vm2.basic_config()
    vm2.add_net_iface()
    vm2.start()
    res = vm2.ssh.run(f"stat {filename}")
    print(res)
    assert res[0] == 0

Can be saved as tests/test_repro.py and run as:

./tools/devtool test_debug test_repro.py

[zulinx86]

Hello @aledbf , if you're still seeing this issue, we'd appreciate it if you could answer @pb8o asked in the previous thread. Thanks!

[aledbf]

@zulinx86 I am sorry for the delay. I am not looking at this issue now.
I will close the thread and open it again once I return to this.