Data loss after power outage

[brianjmurrell]

I had a power outage here yesterday, and the fallout, once everything was back up and running was that an fsck on a ext4 files system on a VDO device suffered data loss due to ext4 metadata issues.

All of the losses were due to has deleted/unused inode .... CLEARED. I'm trying to understand why this is.

The files that were lost due to the above were written to the device at least weeks ago, so there should not have been any ext4 metadata (or data even) that was not written through to physical disk.

Moreover ext4 is journaled so there shouldn't be any metadata loss due to power outage.

No other ext4 filesystem on the system, some of which are much more volatile (i.e. /var) and would most certainly be subject to having data in a cache that was not written through to disk (modulo ext4's journaling) did not suffer any has deleted/unused inode .... CLEARED. issues.

So my question is was this loss related to VDO in any way? Is VDO unsuitable in situations where a machine is subject to unexpected immediate power loss?

[raeburn]

Do you have kernel logs from the boot after the power outage? VDO should log some info if it runs into problems.

We do believe VDO should be robust against a power failure -- however, it does assume that the storage layer under it is robust as well, and VDO doesn’t maintain a lot of redundancy. If VDO sends data (or metadata) to the backing volume, sends a flush and has it acknowledged, and then that data disappears, VDO may lose data. For example, a battery-backed cache where the battery fails during a power outage may have indicated the data is in stable storage, and then it turns out not to be.

While it sounds like your file data should long since have been flushed to stable storage, changes like file access time or mode updates can cause the inode to be rewritten, and the new version of the file system block is written to new storage. If the backing storage lost those updates, it’s possible for data to be lost (though it might need an interesting loss pattern to lose data despite the ext4 and VDO journals).

What does your storage stack look like below VDO?