Problem: Semaphore crashes when running as a systemd service

[phillf]

Issue

Semaphore is crashing right after

 2:22:34 PM
Task 58 added to queue
2:22:35 PM
Started: 58
2:22:35 PM
Run TaskRunner with template: Update apt packages
2:22:35 PM
Preparing: 58
2:22:36 PM
Updating Repository https://git.nti.dev/CT/ansible-playbooks.git
2:22:37 PM
From https://git.nti.dev/CT/ansible-playbooks
2:22:37 PM
 * branch            main       -> FETCH_HEAD
2:22:37 PM
Already up to date.
2:22:37 PM
cloning inventory repository
2:22:37 PM
Cloning Repository https://git.nti.dev/CT/ansible-playbooks.git
2:22:37 PM
Cloning into 'inventory_58'...
2:22:39 PM
No /tmp/semaphore/repository_1_2/playbooks/Update packages/collections/requirements.yml file found. Skip galaxy install process.
2:22:39 PM
No /tmp/semaphore/repository_1_2/playbooks/Update packages/requirements.yml file found. Skip galaxy install process.
2:22:39 PM
No /tmp/semaphore/repository_1_2/playbooks/Update packages/roles/requirements.yml file found. Skip galaxy install process.

without any output to console, task log, or other logs. This results in a hung task after Semaphore is restarted by systemd.

I only have two task templates built out, one for updating dnf pkgs and one for updating apt pkgs. These both work when running semaphore in server mode from CLI.

Impact

Web-Frontend (what users interact with), Service (scheduled tasks, alerts), Ansible (task execution), Other

Installation method

Binary

Database

MySQL

Browser

Chrome

Semaphore Version

2.10.43-e43af6b-1732277822

Ansible Version

config file = /home/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ansible/.local/pipx/venvs/ansible-core/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ansible/.local/bin/ansible
  python version = 3.9.19 (main, Sep 11 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2.0.1)] (/home/ansible/.local/pipx/venvs/ansible-core/bin/python)
  jinja version = 3.1.4
  libyaml = True

Logs & errors

 Loaded: loaded (/etc/systemd/system/semaphore.service; enabled; preset: disabled)
Drop-In: /run/systemd/system/service.d
         └─zzz-lxc-service.conf
 Active: activating (auto-restart) (Result: exit-code) since Sat 2024-11-30 14:22:39 EST; 292ms ago
   Docs: https://docs.ansible-semaphore.com/
Process: 2291 ExecStart=/usr/bin/semaphore server (code=exited, status=2)

Main PID: 2291 (code=exited, status=2)
CPU: 374ms

No longs anywhere else

Manual installation - system information

Linux semaphore.creativetech.me 6.8.8-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-4 (2024-07-26T11:15Z) x86_64 x86_64 x86_64 GNU/Linux

Oracle Linux Server release 9.5

Configuration

No response

Additional information

No response

[steadfasterX]

works fine here on Alma Linux 9.5.

pls share:

- /etc/systemd/system/semaphore.service
- /run/systemd/system/service.d/zzz-lxc-service.conf

[phillf]

I will share the contents of both files requested when I get home.

[phillf]

@steadfasterX, Sorry for the delay. Here is the contents of the files which were requested.
/etc/systemd/system/semaphore.service:

[Unit]
Description=Ansible Semaphore UI
Documentation=https://docs.ansible-semaphore.com/
Wants=network-online.target
After=network-online.target
ConditionPathExists=/usr/bin/semaphore

[Service]
ExecStart=/usr/bin/semaphore server
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=10s
User=ansible
Group=ansible

[Install]
WantedBy=multi-user.target

- /run/systemd/system/service.d/zzz-lxc-service.conf:

[Service]
ProcSubset=all
ProtectProc=default
ProtectControlGroups=no
ProtectKernelTunables=no
NoNewPrivileges=no
LoadCredential=

I don't believe this should effect anything but at this point the the smeaphore config is in /etc/semaphore/. The original issues still persists.

[steadfasterX]

I would disable ALL options in /run/systemd/system/service.d/zzz-lxc-service.conf, try again. if it starts it is one of them. so enable one by one again until it fails again.

Is this running within LXC ?

[phillf]

Is this running within LXC ?

Yes, for clarity, it does start. Semaphore just crashes while running a task.

I can work through what you suggested later.

Is there any reason why running this inside of an LXC container would cause an issues?

[steadfasterX]

yes the reduce /change of privileges you did might let it crash later first.

Is there any reason why running this inside of an LXC container would cause an issues?

not as I am aware of but it like to know as much as possible to understand the situation better