CODER_MOUNTS and /envbox docker --mounts CernVM-FS issue

[groundsada]

I am running coder on a k8s cluster. My outer container is the a modified envbox docker image that has cvmfs installed (without installing it mounting doesn't work). My inner container is an Ubuntu 20:04 image with cvmfs installed as well (just to be safe).

My outer container has cvmfs mount with a HostToContainer mount propagation. However, whether through CODER_MOUNTS or /envbox docker --mounts, I face issues mounting my cvmfs from the outer to the inner container.

The command:

/envbox docker --mounts /tools/Xilinx:/tools/Xilinx:ro

The output (trimmed):

{"output":"Failed to run envbox: remount: mount failed: exit status 2\nMounting command: mount\nMounting arguments:  -o remount,rw /tools/Xilinx\nOutput: The CernVM File System\nVersion 2.11.0\nCopyright (c) 2009- CERN, all rights reserved\n\nPlease visit http://cernvm.cern.ch for details.\n\nUsage: cvmfs2 [-h] [-V] [-s] [-f] [-d] [-k] [-o mount options] \u003crepository name\u003e \u003cmount point\u003e\n\nCernVM-FS general options:\n  --help|-h            Print Help output (this)\n  --version|-V         Print CernVM-FS version\n  -s                   Run singlethreaded\n  -f                   Run in foreground\n  -d                   Enable debugging\n  -k                   Parse options\nCernVM-FS mount options:\n  -o config=FILES      colon-separated path list of config files\n  -o uid=UID           Drop credentials to another user\n  -o gid=GID           Drop credentials to another group\n  -o system_mount      Indicate that mount is system-wide\n  -o grab_mountpoint   give ownership of the mountpoint to the user before mounting (required for autofs)\n  -o parse             Parse and print cvmfs parameters\n  -o cvmfs_suid        Enable suid mode\n\n  -o disable_watchdog  Do not spawn a post mortem crash handler\n  -o foreground        Run in foreground\n  -o libfuse=[2,3]     Enforce a certain libfuse version\nFuse mount options:\n  -o allow_other       allow access to other users\n  -o allow_root        allow access to root\n  -o nonempty          allow mounts over non-empty directory\n\n","time":"2023-10-14T12:44:13.421245546-07:00","type":"error"}
{"output":"Failed to run envbox: run: remount: mount failed: exit status 2\nMounting command: mount\nMounting arguments:  -o remount,rw /tools/Xilinx\nOutput: The CernVM File System\nVersion 2.11.0\nCopyright (c) 2009- CERN, all rights reserved\n\nPlease visit http://cernvm.cern.ch for details.\n\nUsage: cvmfs2 [-h] [-V] [-s] [-f] [-d] [-k] [-o mount options] \u003crepository name\u003e \u003cmount point\u003e\n\nCernVM-FS general options:\n  --help|-h            Print Help output (this)\n  --version|-V         Print CernVM-FS version\n  -s                   Run singlethreaded\n  -f                   Run in foreground\n  -d                   Enable debugging\n  -k                   Parse options\nCernVM-FS mount options:\n  -o config=FILES      colon-separated path list of config files\n  -o uid=UID           Drop credentials to another user\n  -o gid=GID           Drop credentials to another group\n  -o system_mount      Indicate that mount is system-wide\n  -o grab_mountpoint   give ownership of the mountpoint to the user before mounting (required for autofs)\n  -o parse             Parse and print cvmfs parameters\n  -o cvmfs_suid        Enable suid mode\n\n  -o disable_watchdog  Do not spawn a post mortem crash handler\n  -o foreground        Run in foreground\n  -o libfuse=[2,3]     Enforce a certain libfuse version\nFuse mount options:\n  -o allow_other       allow access to other users\n  -o allow_root        allow access to root\n  -o nonempty          allow mounts over non-empty directory\n\n","time":"2023-10-14T12:44:13.421327848-07:00","type":"error"}
{"output":"","time":"2023-10-14T12:44:13.421404419-07:00","type":"done"}
{"ts":"2023-10-14T19:44:13.421414299Z","level":"ERROR","msg":"send startup logs","caller":"/home/runner/work/envbox/envbox/buildlog/coder.go:116","func":"github.com/coder/envbox/buildlog.(*CoderLogger).processLogs.func1","fields":{"error":"Patch \"https://coder.nrp-nautilus.io/api/v2/workspaceagents/me/startup-logs\": context canceled"}}
run: remount: mount failed: exit status 2
Mounting command: mount
Mounting arguments:  -o remount,rw /tools/Xilinx
Output: The CernVM File System
Version 2.11.0
Copyright (c) 2009- CERN, all rights reserved

Please visit http://cernvm.cern.ch for details.

Usage: cvmfs2 [-h] [-V] [-s] [-f] [-d] [-k] [-o mount options] <repository name> <mount point>

CernVM-FS general options:
  --help|-h            Print Help output (this)
  --version|-V         Print CernVM-FS version
  -s                   Run singlethreaded
  -f                   Run in foreground
  -d                   Enable debugging
  -k                   Parse options
CernVM-FS mount options:
  -o config=FILES      colon-separated path list of config files
  -o uid=UID           Drop credentials to another user
  -o gid=GID           Drop credentials to another group
  -o system_mount      Indicate that mount is system-wide
  -o grab_mountpoint   give ownership of the mountpoint to the user before mounting (required for autofs)
  -o parse             Parse and print cvmfs parameters
  -o cvmfs_suid        Enable suid mode

  -o disable_watchdog  Do not spawn a post mortem crash handler
  -o foreground        Run in foreground
  -o libfuse=[2,3]     Enforce a certain libfuse version
Fuse mount options:
  -o allow_other       allow access to other users
  -o allow_root        allow access to root
  -o nonempty          allow mounts over non-empty directory

I also tried mounting without installing cvmfs. That also fails on command cvmfs2 is not installed.

[groundsada]

This issue is also consistent with seaweedfs.