How to get Podman, GitLab Runner with "docker" socket binding and SELinux to work in harmony?

[yangm97]

I have launched a RHEL8.5 VM to use as a GitLab runner with the docker executor pointing to the podman socket. It seems to work very well, thanks to the amazing work on the compat endpoints.

However there's one kind of job that doesn't work unless you disable SELinux, the one which requires binding the "docker" socket to the container. For instance, Code Quality without dind (dinp?).

I've found out that launching podman containers with --security-opt label=disable allows access to the podman socket but adding security_opt = ["label=disable"] to [runners.docker] doesn't, not sure how to debug that.

Still, changing SELinux from enforcing to permissive has allowed the pipeline to succeed so maybe all this setup needs is a custom selinux policy?

[rhatdan]

If you allow a container to talk to any UNIX domain socket on the host, SELinux will rightly block this and you will need to disable SELinux enforcement. We definitely do not want to allow SELinux confined containers to talk to the docker daemon or the podman system service.

[yangm97]

I see. But should this behavior also apply to "normal" bind mounts? (see my other comment for details)

[rhatdan]

security_opt = ["label=disable"] to [runners.docker] doesn't, not sure how to debug that.
Would be a gitlab running question, sadly I know nothing about gitlab runners.

[yangm97]

Ok, I think I see what's happening now. Looking into the code quality job yaml and SELinux audit log I can infer the security_opt got applied, because the docker pull from the job script works as intended. But when it tries to docker run it bind mounts $PWD and the "docker" socket.

What is funny is that, aside from denying access to the podman socket SELinux is also blocking access to files from that host mount:

type=AVC msg=audit(1638397303.521:2088): avc: denied { ioctl } for pid=51842 comm="ruby" path="/code/.codeclimate.yml" dev="dm-0" ino=86676155 ioctlcmd=0x5413 scontext=system_u:system_r:container_t:s0:c242,c346 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1638397303.521:2088): arch=c000003e syscall=16 success=no exit=-25 a0=7 a1=5413 a2=7fff259b02b0 a3=0 items=0 ppid=51822 pid=51842 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ruby" exe="/usr/bin/ruby" subj=system_u:system_r:container_t:s0:c242,c346 key=(null)

With $PWD being /home/gitlab-runner/builds/something-random-string/.... Setting the builds dir to /tmp/builds as suggested in the gitlab docs probably hits a different SELinux assertion but gets blocked too.

So I decided to do some sanity checks:

# /home/gitlab-runner/builds/scuZ-oAf/0/foo
# podman run -eDOCKER_HOST=unix:///var/run/docker.sock -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/code --rm --security-opt label=disable docker run --rm -v /code:/code alpine touch /code/bar
touch: /code/bar: Permission denied

# cd /tmp/builds/scuZ-oAf/0/foo
# podman run --rm -v$PWD:/foo -ti alpine touch /foo/bar
touch: /foo/bar: Permission denied

Interesting.

Looks like the container spawned by the job script won't get the security-opt flag and that is needed not only to access the podman socket but also any sort of bind mount.

If the mount behavior is working as intended then I guess there's nothing wrong from the podman side.

Actually, now that I took the time to look into this I'm wondering why that job uses dind at all...

[rhatdan]

Yes SELinux would block random volumes added to a container unless they are relabeled with the :Z or :z option. You would not want an escaped container reading $HOME/.ssh or secret passwords stored in ~/.firefox ...

[tymonx]

I was facing the same issue. I have posted solution on GitLab Runner issue tracker: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3491#note_2233675636

In short:

• GitLab must fix their GitLab Runner https://gitlab.com/gitlab-org/gitlab-runner/-/issues/11738 and add the :z or :Z flag for created volume /builds that is shared between CI jobs (mostly to reuse .git)
• Until fix, the only solution that is quite generic is to disable SELinux labeling for CI jobs in GitLab Runner configuration file config.toml:

[[runners]]
[runners.docker]
security_opt = ["label:disable"]

It is not a perfect solution because it lowers security layer (for volumes used by CI jobs as cache) but much better than disabling whole SELinux protection.