Skip to content

Latest commit

 

History

History
215 lines (154 loc) · 14 KB

README.md

File metadata and controls

215 lines (154 loc) · 14 KB
Raw

rclonemount

rclone mount is a rootless systemd integration allowing to seamlessly use one or several rclone mount commands as a systemd services with an optional directories and files structure cache warmup.

rootless means that root privileges will not be used during run (as they should not) and rclone mount execution will be compartimented on a dedicated user (rclonemount) while the files and directories can be mapped to specific and different UID/GID with valid linux permissions enforced by the kernel. In addition to avoid using root, this means that you can provide vfs to your services without giving them the ability to read your rclone backend credentials (as config files and rclone execution will be compartimented on a dedicated user).

The directories and files structure cache warmup allows to fully scan the structure of your backend and have it in memory before the systemd unit is actually ready allowing subsequent services whichs depends on the mount and its VFS to start with the full vfs structure already cached locally.

Installation

deb pkg (WIP)

Just install it and go straight to configuration. It will require that you have installed rclone as a deb pkg too.

manually

  • Copy rclone_vfsrcwarmup to /usr/local/bin
  • Copy [email protected] to /etc/systemd/system
  • Change the path of the ExecStartPost= entry in [email protected] from /usr/bin/rclone_vfsrcwarmup to /usr/local/bin/rclone_vfsrcwarmup
  • Create the /etc/rclonemount dir
  • Copy example.conf and example.env files to the /etc/rclonemount dir
  • Execute the following commands as root for a future rootless usage:
useradd --home-dir /var/lib/rclonemount --create-home --system --shell /usr/sbin/nologin rclonemount
chown rclonemount: /etc/rclonemount
chmod 750 /etc/rclonemount /var/lib/rclonemount
chmod 640 /etc/rclonemount/*

Configuration

Using the example provided, a single configuration is composed of 2 files:

  • the .conf file which corresponds to the rclone backend vanilla configuration (nothing new here)
  • the .env file which allows to configure this particular configuration as a systemd service

All configurations are stored in /etc/rclonemount in order for the service unit template to find them on activation.

As both file contains sensitive informations (auth tokens and password) I recommend to always have them readable by rclonemount only. One time fix/enforce can be found below:

chown rclonemount: /etc/rclonemount/*
chmod 640 /etc/rclonemount/*

Environment configuration

Using example.env as reference. Environment variables are used to configure rclone to avoid specifying these options on the command line which would lead us to edit the systemd unit service file and:

Command section

Used to configure not rclone directly but the command line in particular.

  • SOURCE reference the rclone backend you want to mount (it must exist in the corresponding .conf file, here example.conf)
  • DESTINATION is the mount point. It must be a valid directory path and the rclonemount user (or group) must be have the right permission ont it and its parent directory.
  • WARMUPCACHE will trigger a directories and files structure warmup during startup, set to false to deactive cache warmup.

Global section

Used to set up global rclone options.

Mount section

Used to set up mount rclone options.

  • RCLONE_ALLOW_OTHER this is important to have it set to true, all virtual filesystem operations will be executed by rclone as rclonemout but the files ownership will actually be mapped to a different user/group for compartmentalization. By default FUSE only allows the mounting user to access the virtual filesystem. More on the different mapped user/group below.
  • RCLONE_ATTR_TIMEOUT How often the kernel will refresh/ask rclone about file attributes. If the backend is not modified outside this mount, you can increase it to enhance performance (let's say 8760h, see rclone documentation for details.
  • RCLONE_CACHE_DIR where rclone will store its cache. Example use /var/lib/rclone as base directory but make sure you have enough space or put it elsewhere (check RCLONE_VFS_CACHE_MAX_SIZE option and see rclone documentation). Be carefull to target a dedicated sub folder for each configuration. Target must be writable by the rclonemount user.
  • RCLONE_DEFAULT_PERMISSIONS this is important to have it set to true as we allowed other users to access this FUSE mount, file security will be handled by kernel with regular rights (see rclone documentation for details).
  • RCLONE_DIR_CACHE_TIME directories structure cache (see rclone documentation for details).
  • RCLONE_DIR_PERMS default permissions for mounted directories, this is important as we have activated allow-other and default-permissions. Example value 0750 will allow the mapped UID to read/write/traverse and GID to read/traverse but will forbid any actions by others (see rclone documentation for details).
  • RCLONE_FILE_PERMS default permissions for mounted files, this is important as we have activated allow-other and default-permissions. Example value 0640 will allow the mapped UID to read/write and GID to read but will forbid any actions by others (see rclone documentation for details).
  • RCLONE_GID the GID to map the directories and files to. On this example, GID only has read right, so it could be a group (let's say mediaservers containing emby and plex users) which you want to access your files but deny them write (see rclone documentation for details).
  • RCLONE_POLL_INTERVAL request the backend for external changes (see rclone documentation for details)
  • RCLONE_UID the UID to map the directories and files to. On this example, UID has read/write rights, so it could be your own user in order for you to drop new files (see rclone documentation for details).
  • RCLONE_UMASK set the desired mask when creating new directories and files. Example value 027 is set accordingly with RCLONE_DIR_PERMS and RCLONE_FILE_PERMS (see rclone documentation for details).
  • RCLONE_VFS_CACHE_MAX_AGE trigger age of files to be purged of cache each time a cleanup is performed (check RCLONE_VFS_CACHE_POLL_INTERVAL and see rclone documentation for details).
  • RCLONE_VFS_CACHE_MODE cache mod, see rclone documentation for details.
  • RCLONE_VFS_CACHE_POLL_INTERVAL interval to cleanup the cache (see rclone documentation for details)
  • RCLONE_VFS_READ_AHEAD disk buffer in addition to the kernel buffer during read (see rclone documentation for details)
  • RCLONE_VFS_WRITE_BACK how many time to wait before starting to upload a file (see rclone documentation for details)
  • RCLONE_WRITE_BACK_CACHE send writes in batch to rclone, should increase performance especially if a lot of context switching is occuring (see rclone documentation for details)

RC section

Only used for cache warmup at start. Be carefull to set a different port in RCLONE_RC_ADDR for each configuration. See rclone_vfsrcwarmup script for details.

You can set RCLONE_RC to false if WARMUPCACHE in the Command section is set to false as well (it will prevent rclone from starting and binding the rc server).

FUSE configuration

FUSE needs to be configured to allow the usage of other users' mounts (in our case rclonemount and your others users). Add user_allow_other to the /etc/fuse.conf file. This need to be done only once, before your first mount.

Example:

# /etc/fuse.conf - Configuration file for Filesystem in Userspace (FUSE)

# Set the maximum number of FUSE mounts allowed to non-root users.
# The default is 1000.
#mount_max = 1000

# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other

systemd service unit template

This normally does not need configuration but values here will impact all yours configurations. You can tune them before registering your first configuration.

  • Type=notify as rclone supports systemd notify system, it will enhance systemd services scheduling. Should not be changed.
  • User=rclonemount do not use root to run rclone
  • Nice=-5 increase system priority for rclone mounts. Reduce time others services using the mount will spend their CPU time in iowait by scheduling rclone mount more often in order for data to be ready for them.
  • LimitNOFILE=32768 increase the maximum number of open files (default is 1024) to avoid rclone errors on large backends
  • TimeoutStartSec=infinity as the cache warmup can be quite long if there is thousands of thousands of files, we don't want systemd to consider the unit stalling. Note that cache warmup script using rc has also timeout deactivated.
  • ExecReload=/bin/kill -SIGHUP $MAINPID allow easy runtime dir cache purging, see Purge directories and files structure cache
  • ExecStopPost=-+/bin/umount -f $DESTINATION sometimes rclone will fail to unmount cleanly. In order to be able to start the unit again and not leave the system with a mount in an awkward state, we force an unmount just in case. Will fail when regular mount has succeeded (this is expected and valid as indicated to systemd with -). Because a forced unmount needs root privilege, this command only will run as root as indicated by +.

Usage

Once a configuration is ready, you can start manipulating the unit. Following examples will use anotherconf as configuration name, which means anotherconf.conf and anotherconf.env files both exist within /etc/rclonemount and are valid.

First start

systemctl enable --now [email protected]

Usual commands

systemctl start [email protected]
systemctl stop [email protected]
systemctl restart [email protected]
systemctl status [email protected]

Checking logs

Live logs

journalctl -f -u [email protected]

Full logs until now

journalctl -u [email protected]

Purge directories and files structure cache

systemctl reload [email protected]

Unregister

systemctl disable --now [email protected]

Service binding

Now that your virtual filesystem is perfectly integrated to your system as a systemd service unit, there is a big probability others services are meant to use it. It could be important to bind these services to it to prevent/delay their start if/when the mount has failed/is not ready.

For example for a plex web server:

~$ cat /etc/systemd/system/plexmediaserver.service.d/rclone.conf
[Unit]
[email protected]
[email protected]
~$ systemctl daemon-reload
~$ systemctl restart plexmediaserver.service

This way: