Containerssh kills container: Program exited with 137

[bmeekhof]

I'm having an issue with my container image running under containerssh. My containerssh installation works fine with the test image but using an image I've built I am having problems. The image in question works fine in docker run on the command line using the same options I pass into containerssh. When I use it for containerssh I get as far as the image being spun up and the user being dropped into the container with prompt, then shortly after containerssh or docker kill the container with error 'Program exited with 137'

Debug logs don't reveal any more detail than the exit code:
{"timestamp":"2024-07-22T14:56:06-04:00","level":"debug","code":"SSH_CHANNEL_REQUEST_SUCCESSFUL","message":"shell channel request from client successful","details" ... (user, connectionid, etc)
{"timestamp":"2024-07-22T14:56:08-04:00","level":"debug","code":"DOCKER_EXIT_CODE","message":"Program exited with 137","details"...
{"timestamp":"2024-07-22T14:56:08-04:00","level":"debug","code":"SSH_EXIT","message":"Program exited with status 137","details"

The system has 13G free memory and as far as I can tell the container isn't being killed due to an actual OOM situation (and it also has no issues running outside containerssh). I'm not setting any resource limitations in my config. I'm running containerssh from the RPM installation on Alma 9 with docker/podman backend. The image in question includes the containerssh-agent.

For an experiment I tried setting oomkilldisable: true in the containerssh backend config but my system won't allow me to do that: "crun: cannot disable OOM killer with cgroupv2: OCI runtime error". I also tried setting oomscoreadj to -1000 but same behaviour.

Any thoughts on further troubleshooting or debugging?

[tsipinakis]

Since you have a DOCKER_EXIT_CODE message I'm assuming you're using the docker backend. Is it podman rootless, rootful podman or an actual docker daemon?

I'm afraid I'm not too familiar with the Docker memory management options, this is a situation where the container runtime kills the container due to some contraints - containerssh can't really do much in this scenario. Did you take a look at the container runtiumes logs, and the full system logs as well if there's anything useful there?

Edit: Also what versions are you using? Of containerssh and docker/podman

[bmeekhof]

Thanks for the response! This is rootful podman. Podman version is 4.9.4-5 and containerssh is 0.5.0-1.

I did look at system logs and I see typical messages about container startup, the same error code ("container X exited with status 137"), and then container cleanup, nothing else to go on. I don't see anything in the container runtime logs either. Maybe I'll have to try paring down my image until I get to something that works and go from there.

[bmeekhof]

I finally did track down this issue to something with our login scripts running at startup. I have the container entrypoint as /bin/bash --login (I set HOME and bind mount a home dir). When our various profile.d scripts get sourced the container gets killed shortly after startup. If I just enter into /bin/bash and skip the login stuff then the container stays running. Unfortunately I can't replicate the problem at that point. Running bash --login sources our environment and works fine. It also works to run the image with login scripts outside containerssh on the command line (with same options). So it seems isolated to the context of bash login running our environment scripts at container entrypoint when it runs from containerssh. At this point I don't know which specific action/script is causing the issue.

[tsipinakis]

You can try running the container manually to see if you can replicate it, or run the login script using podman exec, in a started container it's the same API call ContainerSSH uses to get a shell.