Failure to find real path on volume binding

[spoutn1k]

Hello,

I am getting what I suspect is a bug while binding a directory to a container. I am trying to run a container with the /opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1 folder accessible inside.

jskutnik@nid002737:~> /usr/bin/shifter --image=ecpe4s/e4s-cuda:22.05 --volume=/opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1:/opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1 ls
shifter_realpath: failed to lstat /var/udiMount/opt/cray
FAILED to find real path for volume "from": /opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1
FAILED to setup user-requested mounts.
FAILED to setup image.

The error shows a failure to resolve a path on the host, while this path exists and is not a symlink:

jskutnik@perlmutter:login22:~> file /opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1
/opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1: directory
jskutnik@perlmutter:login22:~> readlink -f /opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1
/opt/cray/pe/mpich/8.1.24/ofi/gnu/9.1

I tested this behaviour on both the login and work nodes, and both are identical. The code snippets come from one of each as I took some time writing this error report.

I would have included the software version but shifter --version is not a thing.

[scanon]

Please file a NERSC ticket for this but the short answer is Shifter has some restrictions set by the site (eg NERSC) on which directories are allowed to be volume mounted.