Make the mount points of the overlayfs visible via config option

[Ircama]

Considering that the Performance option of raspi-config allows a minimal configuration of the Overlay File System, this is a proof of concept for making the mount points of the OverlayFS visible via configuration option, based on the idea in this thread. A new step in the wizard is added for this; the default is to make the mount points invisible, as per the current setting; when they are visible, their root directory is /run, like the following (referring to a Raspberry Pi Zero W):

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             80M     0   80M   0% /dev
tmpfs            43M  796K   43M   2% /run
/dev/mmcblk0p7  6.5G  5.2G  948M  85% /run/lower
tmpfs           215M   28M  188M  13% /run/upper
overlay         215M   28M  188M  13% /
tmpfs           215M     0  215M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p8  488M   32M  421M   8% /data
/dev/mmcblk0p6  253M   65M  188M  26% /boot
tmpfs            43M   20K   43M   1% /run/user/1000

This configuration is still safe because the physical mount point (lower) is in read-only mode.
The /run/upper mount point allows inspection of the changes, for instance, to monitor their growth (via du or standard commands). Any changes here are not persistent, as the /run/upper filesystem is a tempfs in RAM. The /run/lower mount point is a read-only filesystem, so any possible access is safe for the SD card.

The new step in the wizard is the following:

┌──────────────────────────────────────────────────────────┐
│                                                          │
│ Would you like the mount points of the overlay file      │
│ system to be visible under '/run/upper' (rw) and         │
│ '/run/lower' (ro)?                                       │
│                                                          │
│                                                          │
...
│                                                          │
│               <Yes>                  <No>                │
│                                                          │
└──────────────────────────────────────────────────────────┘

If No is selected (default), the output (after the reboot) is the standard overlay mode, with no visible mount points:

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             80M     0   80M   0% /dev
tmpfs            43M  796K   43M   2% /run
overlay         215M   92M  123M  43% /
tmpfs           215M     0  215M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p8  488M   32M  421M   8% /data
/dev/mmcblk0p6  253M   65M  188M  26% /boot
tmpfs            43M   20K   43M   1% /run/user/1000

The following command will monitor the growth of the upper filesystem (e.g., files changed after a reboot) by listing the first 10 largest directories:

cd /run/upper/data && sudo du -h | sort -hr | head

This PR also includes some improvements to the batch (command line) mode:

sudo ./raspi-config nonint do_overlayfs status [ writable [ invisible ] ]

• "status" can be 0, to enable the overlayfs, or 1 to disable it (1 is the default);
• "writable" can be 0 for setting a read-only boot partition, or 1 if the boot partition has to be writable (0 is the default);
• "invisible" is used when enabling the overlays; it can be 0 for visible mounts or, 1 for invisible mounts (1 is the default).

If status is missing, a usage summary is printed.

(Notice that best practices suggest that the whole SD card is in read-only mode, including the boot partition, as physically consumer SD cards have no partition concept (and no fault tolerance), so any write inconsistency (also to the boot partition) can drive an SD corruption if not appropriately completed.)

Even if all preliminary tests of this patch appear to be fine, long-term stability needs monitoring for some additional time.

---

Usage notes

Formally, OverlayFS does not support merging changes from an upper filesystem to a lower filesystem and changes to the lower filesystem are not allowed. This premised, empirically I succeeded in mounting the lower filesystem in read-write via the following command:

sudo mount -o remount,rw /run/lower

Then I could make permanent changes to the SD card by copying (cp -p) files from the upper part to the lover part; if a file is changed in the lower part, it persists to a reboot. After making changes, I could return back to read-only mode with the following command:

sudo mount -o remount,ro /run/lower

Also, I could change the /run/upper filesystem, provided that after the changes the overlayfs is remounted through

sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /

Interesting to note that an adequate application of changes to the /run/upper filesystem followed by the remount of the OverlayFS allows reverting changes.

Example:

# Creating a new file under /home/pi (login pi):
cd

cat foo  # failure
ls -l /run/upper/data/home/pi/foo  # failure
ls -l /run/lower/home/pi/foo  # failure

echo test > foo

cat foo  # success
ls -l /run/upper/data/home/pi/foo  # success
ls -l /run/lower/home/pi/foo  # failure

mount | grep "lower type"  # read-only

# Making the new file persistent:
sudo mount -o remount,rw /run/lower
cp -p foo /run/lower/home/pi
sudo mount -o remount,ro /run/lower
rm /run/upper/data/home/pi/foo
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /

cat foo  # success
ls -l /run/upper/data/home/pi/foo  # failure
ls -l /run/lower/home/pi/foo  # success

# Updating the new file:
echo hello >> foo

cat foo  # success
ls -l /run/upper/data/home/pi/foo  # success
ls -l /run/lower/home/pi/foo  # success

# Reverting updates:
rm /run/upper/data/home/pi/foo
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /

cat foo  # previous data
ls -l /run/upper/data/home/pi/foo  # failure
ls -l /run/lower/home/pi/foo  # success

# Removing the file and making this persistent:
rm foo
ls -l /run/upper/data/home/pi/foo  # whiteout existing, as a character device with 0/0 device number
sudo mount -o remount,rw /run/lower
rm /run/lower/home/pi/foo
sudo mount -o remount,ro /run/lower
rm -f /run/upper/data/home/pi/foo  # remove the whiteout
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /

cat foo  # failure
ls -l /run/upper/data/home/pi/foo  # failure
ls -l /run/lower/home/pi/foo  # failure

Notice that the above methods have been empirically tested; they appear to work, even if the official documentation does not report them.

---

Maintenance notes

By periodically monitoring /run/upper with df -h|sort -k5 -h, cd /run/upper/data&&sudo du -h|sort -hr|head - and sudo tree -dh /run/upper I noticed that the growth of two resources have to be controlled: packagekit and journald: when the OverlayFS is active, the first one need to be stopped and disabled, while the second one might benefit from a configuration to limit its size.

The following script can be added to crontab @reboot:

if ! test -d /run/upper; then exit 0; fi  # Only run if the overlayfs is active
sudo systemctl stop packagekit.service
sudo systemctl mask packagekit
sudo systemctl stop apt-daily.service
sudo systemctl stop apt-daily.timer
sudo systemctl stop apt-daily-upgrade.timer
sudo systemctl stop apt-daily-upgrade.service
sudo systemctl mask apt-daily.service
sudo systemctl mask apt-daily.timer
sudo systemctl mask apt-daily-upgrade.timer
sudo systemctl mask apt-daily-upgrade.service
sudo apt-get clean
if ! grep -q '^SystemMaxUse=' /etc/systemd/journald.conf; then
    echo "SystemMaxUse=50M" | sudo tee -a /etc/systemd/journald.conf >/dev/null
    echo "SystemKeepFree=50M" | sudo tee -a /etc/systemd/journald.conf >/dev/null
    sudo service systemd-journald restart
fi

As per the journald service, in the hypothesis that only user-1000 (pi) and system user ids are used, the script should result in making journald produce up to 10 files under /var/log/journal/ (including rotated files), where each one has size of 6,3 MB, with a total consumption of max 63 MB.

More generally, it is necessary to avoid the memory full condition of the OverlayFS by periodically monitoring the “/run/upper” usage size. Typical elements to check are:

• journald (size of /var/log/journal directory)
• size of /tmp directory
• size of the user directories
• mailing system
• apt and /var/lib/dpkg
• packagekit

On a Raspberry Pi Zero W, the max size of /run/upper will be about 215 MB.

Notice that when the usage size reaches 100%, a crash might occur; on a Raspberry Pi Zero W, this can be detected by monitoring the ACT led, intercepting when it repeatedly blinks with the following pattern: about 200 short blinks within 0.5 secs + 0.5 sec blink (constantly repeated).

To avoid the OverlayFS saturation condition, a periodic reboot can be issued when the OverlayFS usage exceeds 90% via the following script, which can be added to crontab, to be run every day (e.g., 0 2 * * * /home/pi/OverlayFSMonitor):

USAGE_PERCENT=90
size=`df| awk '/\/run\/upper/{ gsub("%",""); print $5 }'`
test "$size" || size=0
if [ "$size" -gt "$USAGE_PERCENT" ] ; then
    echo "OverlayFS bigger than $size%: performing a reboot." | systemd-cat -t OverlayFSMonitor
    df -h | systemd-cat -t OverlayFSMonitor
    sudo shutdown -r now
fi

When the growth of the previously mentioned resources is correctly maintained, the above memory full condition in the script (producing the reboot) should occur very rarely, for example after many months of uninterrupted operation.