use ssh private key from KeyStore

[Korhm]

Hello,

I would like use an ssh key to connect to remote hosts, that is stored is the Key Store of Sempahore (like to one to connect to git).
Is it something possible ?

[fiftin]

Yes, you can.

[Korhm]

Thank you for the answer fiftin, but can you please elaboration on how to do it ?

On the Template Edit form, I do not find any way to select an ssh key from the KeyStore. The only thing about it is inside the "Advanced" field, but the key has to come from file system, not database

[fiftin]

It should be specified in the Inventory settings.

[Korhm]

The ssh key is set in the inventory, but it seems that ansible won't use it for the ssh connection to remote host.
Semaphore creates /tmp/semaphore/access_key_736139256 (which is deleted after the Play is terminated), but it still uses /tmp/semaphore/.ssh/id_rsa that does not exist...

In the debug log of the task run, I have :

5:06:26 PM
PLAYBOOK: copy_cert.yml ********************************************************
5:06:26 PM
Positional arguments: tasks/copy_file.yml
5:06:26 PM
verbosity: 4
5:06:26 PM
private_key_file: /tmp/semaphore/access_key_736139256
5:06:26 PM
connection: smart
5:06:26 PM
timeout: 10
5:06:26 PM
become_method: sudo

[...]

5:06:26 PM
TASK [Copy file] ***********************************************
5:06:26 PM
task path: /tmp/semaphore/repository_1_1/tasks/copy_file.yml:7
5:06:26 PM
<lab-server1> ESTABLISH SSH CONNECTION FOR USER: lab-user1
5:06:26 PM
<lab-server1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/tmp/semaphore/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o

[fiftin]

Latest version uses ssh-agent instead of file.

[fiftin]

Hi @Korhm,

Looks like you should specify username for your remote server(s).

You can do it via Environments:

Since this is not very convenient, I added the ability to set a username in Access Key settings. It will be available in next release:

[fiftin]

Available in v2.8.45

[bor7an]

Is there option to add environment for different inventory hosts group who are using same SSH key but different SSH users.

[chakib2002]

@bor7an did you find a solution? can we assign a unique ssh key for each hosts in an inventory file ?

[bverkron]

@fiftin I'm having some difficulty with this as well. It seems semaphore cannot make the connection and throws the following:
fatal: [nas]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.", "unreachable": true}

debug.log

However, I know the key is good because I can use it successfully from my local machine to that server. I have also copied the full SSH command semaphore uses from the debug output and run that locally using my key and it works (after removing the ControlPath option because I don't have that same temp file that semaphore generates)

SSH command from semaphore debug output
ssh -v -C -o ControlMaster=auto -o ControlPersist=60s -o Port=44 -o 'IdentityFile="~/.ssh/id_rsa_nas_maintenance"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maintenance"' -o ConnectTimeout=10 -tt 192.168.1.219 'touch ~/success.txt'

Result on remote host (success.txt file is there)

[maintenance@NAS-219 ~]$ ls -alh
drwxrwxrwx    3 maintena everyone      4.0k Sep 30 20:06 ./
drwxrwxrwx    9 admin    administ      4.0k Sep 30 17:42 ../
-rw-rw----    1 maintena everyone         0 Sep 30 20:06 success.txt

So, it seems semaphore isn't using the key correctly or perhaps I don't have some thing setup correctly but I cannot figure out what.

My configuration:

Full contents of inventory file

[storage]
nas ansible_host=192.168.1.219 ansible_user=maintenance ansible_port=44

I'm fairly new to Ansible and semaphore to my apologies if I have something misconfigured.

[bor7an]

This options to add ansible_user= next to host working only with first group.

[bverkron]

Interestingly there are a bunch errors related to the known_hosts files in the debug log right before it fails. Maybe it's a related?

debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts: No such file or directory
debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts does not exist
debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts2"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts2"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
debug1: read_passphrase: can\'t open /dev/tty: No such device or address
Host key verification failed.

[beingyash]

@Korhm Well, the issue pertains to the known_hosts file only. This file does not exist in the container, hence the ssh key based authentication fails as no known host is available. It works once you manually create this file inside the container in "/home/semaphore/.ssh" and add the public key of the server you were trying to connect with in your playbook.
@fiftin I believe this must be labeled as a bug.

[bverkron]

I shelled into the running container (I'm running this in docker via semaphoreui/semaphore:latest) and manually created the known_hosts file in the first location it looked and I got past that error but now it's one onto the following error.

fatal: [nas]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: [email protected]: Permission denied (publickey,password,keyboard-interactive).", "unreachable": true}

I see in the job log that it automatically adds bitbucket to know hosts (though not sure where because it doesn't exist where it's looking for the remote ssh server known host. So... why does it not auto-accept known hosts from other servers and where is there no option to get known hosts somewhere?

Warning: Permanently added 'bitbucket.org' (RSA) to the list of known hosts.

[bverkron]

Final comment for now I switched to a username and password credential for it under Inventory and it works. Just SSH key doesn't

[thre-z]

I came across this issue as well, with the same error messages about the known_hosts file in the debug output. I ended up finding two solutions:

1. As root in the Ansible Semaphore instance (in my case a Package manager installation in an LXC), add the host keys by running ssh host and accepting the key for each host. This is not ideal as it is manual and the host keys could change. There are probably other, better ways to manage this... but I went with the next option.
2. Disable host key checking in Ansible as described in the docs (https://docs.ansible.com/ansible/latest/user_guide/connection_details.html). This has security implications. In my case, I decided this was worth it for the fix and went this route.

I hope this ends up being helpful for someone out there.

[wilpig]

*2 here for me was the magic ticket. I just added that to the environment variables like.{ "ANSIBLE_HOST_KEY_CHECKING": "False" } and everything started connecting just fine.

[ScarUY]

I think either google doesn't index this comments, or I'm just awful at googling things, I spent 5 days trying to figure this out and since I could't find it anywhere not even mentioned I assumed it was just me doing something wrong, I tried so many useless thiungs and it turns out it's the feature that doesn't work! Thank you all for posting this here, you've saved my sanity.

[shallcrm]

FWIW ... I was experiencing similar symptoms to those reported by @bverkron above, where Semaphore seemed to ignore the SSH private key that I had placed in the Semaphore Key Store.
Instead, the ssh command was being issued with ... -o 'IdentityFile="/tmp/semaphore/.ssh/ansible_pwless_ssh_key"' pointing to a local key file that it could not find.

In my case this was being caused by my playbook having explicitly specified the private key file in a group variable...
ansible_ssh_private_key_file: ~/.ssh/ansible_pwless_ssh_key

As soon as I removed this explicitly specified variable, Ansible Semaphore correctly used the SSH private key associated with the Inventory and retrieved it from the Key Store.

[BakermanLP]

There is another, in my opinion, better solution for that.

Exec into the container and edit the .ssh/config of the user semaphore

docker exec -it ansible-semaphore-semaphore-1 sh

mkdir /home/semaphore/.ssh
echo "StrictHostKeyChecking    no" > /home/semaphore/.ssh/config
chown -R semaphore /home/semaphore/.ssh
chmod 700 /home/semaphore/.ssh
chmod 600 /home/semaphore/.ssh/config

After that change, the known_hosts file is written and everything works fine.

Could also be changed in the Docker Container Build by editing the /etc/ssh/config file.

[0xluc]

Saved my night, thank you

[avegy]

Same here. I knew I was doing it right and still couldn't figure out why it wouldn't work. Thanks!

[heiko-weber]

awesome hint, took only 5 days to find it ;-)

[segadora]

I did this instead

services:
  semaphore:
    restart: unless-stopped
    ports:
      - 3000:3000
    image: semaphoreui/semaphore:latest
    command: >
        bash -c "mkdir -p /home/semaphore/.ssh && echo 'StrictHostKeyChecking    no' > /home/semaphore/.ssh/config && chmod 700 /home/semaphore/.ssh && chmod 600 /home/semaphore/.ssh/config && /usr/local/bin/server-wrapper"
....

[kruscho]

Any solution for that topic? I tried to set up my environment with the ideas of @BakermanLP and @wilpig , but ended up like @bverkron. User/Password is working, but ssh keys (also adding them manually like @thre-z described) won´t work. Am I doing something wrong or is this feature not implemented?
Log -> here

[fiftin]

@riker09 do you use snap installation?

[riker09]

No, I'm using a containerized install with podman

[fiftin]

I just installed Semaphore v2.9.64 from Docker Hub.

I added SSH key with Passphrase. And successfully connected to the server. I need details of your issue.

[fiftin]

@riker09 is it Podman installation issue?

[riker09]

I will have to investigate that when I find the the time. Unfortunately, it's not a top priority issue (for me). I will report back here, though.

[ssh30]

its 2024 this is still a huge issue

anyways if you are using docker compose to your setup modify it and mount a volume sharing the local server ssh keys
like the way i did

  semaphore:
    restart: unless-stopped
    ports:
      - 3000:3000
    image: semaphoreui/semaphore:latest
    volumes:
      - /home/devops/.ssh/:/home/semaphore/.ssh:ro 

[xremoteuser]

A year later it still doesn't work. @ssh30 solution works, but if your user ID is not 1001, copy the ssh dir to another location and make chown 1001:1001

[fiftin]

I just installed Semaphore v2.9.64 from Docker Hub.

I added SSH key with Passphrase. And successfully connected to the server. I need details of your issue.

[jmcamputaro]

Same issue here, pulling my hair out. I tried @ssh30 fix but it did not work for me.

[ssh30]

first you need to create the ssh keys on the server that is going to run the docker compose using ssh-keygen (don't use a passphrase) just Enter Enter Enter and do not change the location for the ssh files leave it as it is.
After that use ssh-copy-id user@server_ip_address against every server that you need to control with Ansible.

Then within Semaphore go to the Environment tab and inside the Extra Variables paste this

{ "ANSIBLE_HOST_KEY_CHECKING": "False" }

On the Semaphore Keystore, create key of type SSH key and paste the content of the private key file id_rsa not the id_rsa.pub that is inside the .ssh/ folder

after that mount the volume the way i did , that will pass the ssh public and private key folder into the container

[ssh30]

that's my compose file

services:
  mysql:
    restart: unless-stopped
    image: mysql:8.0
    hostname: mysql
    volumes:
      - semaphore-mysql:/var/lib/mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: semaphore
      MYSQL_USER: semaphore
      MYSQL_PASSWORD: xxxx
  semaphore:
    restart: unless-stopped
    ports:
      - 3000:3000
    image: semaphoreui/semaphore:latest
    volumes:
      - /home/devops/.ssh/:/home/semaphore/.ssh:ro  # Add this line, replace ./path/to/local/.ssh with the actual path to your .ssh directory
    environment:
      SEMAPHORE_DB_USER: semaphore
      SEMAPHORE_DB_PASS: semaphore
      SEMAPHORE_DB_HOST: mysql
      SEMAPHORE_DB_PORT: 3306
      SEMAPHORE_DB_DIALECT: mysql
      SEMAPHORE_DB: semaphore
      SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
      SEMAPHORE_ADMIN_PASSWORD: xxxxxx
      SEMAPHORE_ADMIN_NAME: admin
      SEMAPHORE_ADMIN_EMAIL: admin@localhost
      SEMAPHORE_ADMIN: admin
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      SEMAPHORE_LDAP_ACTIVATED: 'no'
    depends_on:
      - mysql
volumes:
  semaphore-mysql:
    driver: local

[Kupro404]

For me it would not work when the variable "ANSIBLE_HOST_KEY_CHECKING": "False" was given through the ui I needed to put it in my docker compose file to work. Moreover another issue is that if the user is not specified within the key store when you create it, it will try to connect as semaphore @192.168.x.x and not root @192.168.x.x or any other user that you want to as expected. It says optional but it doesnt seem to work any other way except the ones i will mention bellow. So try putting root in the username field works for me.

Another way is to also include var ANSIBLE_REMOTE_USER: "root" in the compose file. This does the same job with above so you can skip the username part.

Lastly you can skip having to paste your keys in ui by either manually creating the ssh folder in /home/semaphore/.ssh/id_rsa or by creating a volume of your ssh keys to the same path through the compose file. Then in your inventory specify the the path for the keys:

[target]
x.x.x.x   ansible_ssh_private_key_file=/home/semaphore/.ssh/id_rsa

With this you do not have to create an ssh option for the inventory and you can use a none one.

The bellow compose file should work out of the box and you will only need to paste your private key in the key store or through the volumes method mentioned above.

services:
     # uncomment this section and comment out the mysql section to use postgres instead of mysql
     #postgres:
       #restart: unless-stopped
       #image: postgres:14
       #hostname: postgres
       #volumes:
       #  - semaphore-postgres:/var/lib/postgresql/data
       #environment:
      #  POSTGRES_USER: semaphore
      #  POSTGRES_PASSWORD: semaphore
      #  POSTGRES_DB: semaphore
    # if you wish to use postgres, comment the mysql service section below
  mysql:
    restart: unless-stopped
    image: mysql:8.0
    hostname: mysql
    volumes:
      - semaphore-mysql:/var/lib/mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: semaphore
      MYSQL_USER: semaphore
      MYSQL_PASSWORD: semaphore
  semaphore:
    restart: unless-stopped
    ports:
      - 3000:3000
    image: semaphoreui/semaphore:latest
    environment:
      SEMAPHORE_DB_USER: semaphore
      SEMAPHORE_DB_PASS: semaphore
      SEMAPHORE_DB_HOST: mysql # for postgres, change to: postgres
      SEMAPHORE_DB_PORT: 3306 # change to 5432 for postgres
      SEMAPHORE_DB_DIALECT: mysql # for postgres, change to: postgres
      SEMAPHORE_DB: semaphore
      SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
      SEMAPHORE_ADMIN_PASSWORD: changeme
      SEMAPHORE_ADMIN_NAME: admin
      SEMAPHORE_ADMIN_EMAIL: admin@localhost
      SEMAPHORE_ADMIN: admin
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: c3bHOwIvsBZCHoaKY1ubWj3VFyxXSJnLc72iIwO/Pl0=
      ANSIBLE_REMOTE_USER: "root"
      ANSIBLE_HOST_KEY_CHECKING: "false"
        #SEMAPHORE_LDAP_ACTIVATED: 'no' # if you wish to use ldap, set to: 'yes'
        #SEMAPHORE_LDAP_HOST: dc01.local.example.com
        #SEMAPHORE_LDAP_PORT: '636'
        #SEMAPHORE_LDAP_NEEDTLS: 'yes'
        #SEMAPHORE_LDAP_DN_BIND: 'uid=bind_user,cn=users,cn=accounts,dc=local,dc=shiftsystems,dc=net'
        #SEMAPHORE_LDAP_PASSWORD: 'ldap_bind_account_password'
        #SEMAPHORE_LDAP_DN_SEARCH: 'dc=local,dc=example,dc=com'        
    depends_on:
      - mysql # for postgres, change to: postgres
volumes:
  semaphore-mysql
       

[fiftin]

I just installed Semaphore v2.9.64 from Docker Hub.

I added SSH key with Passphrase. And successfully connected to the server. I need details of your issue.

[Kupro404]

Mine too. What is the issue with it.

…

On Fri, Mar 22, 2024, 03:24 sigma0100 ***@***.***> wrote: can't do that playbook lives in a repo, but i get your idea — Reply to this email directly, view it on GitHub <#847 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCWOISAAOAOCWBMXAOYAPKLYZOB45AVCNFSM5NHRZ7DKU5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQOBXGIZTKNY> . You are receiving this because you commented.Message ID: ***@***.*** com>

[fiftin]

@Kupro404 it is outdated docs, will update it. It must works. If not, it is a bug.

[geelenbert]

Well, changing to keys without passphrase seems to work:

2:55:01 PM
Task 6 added to queue
2:55:04 PM
Started: 6
2:55:04 PM
Run TaskRunner with template: update apt
2:55:04 PM
Preparing: 6
2:55:05 PM
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
2:55:06 PM
Updating Repository [email protected]:geelenbert/ansible.git
2:55:06 PM
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
2:55:07 PM
From github.com:geelenbert/ansible
2:55:07 PM
 * branch            main       -> FETCH_HEAD
2:55:07 PM
Already up to date.
2:55:07 PM
installing static inventory
2:55:07 PM
No collections/requirements.yml file found. Skip galaxy install process.
2:55:07 PM
No roles/requirements.yml file found. Skip galaxy install process.
2:55:08 PM
2:55:08 PM
PLAY [Update and upgrade apt packages] *****************************************
2:55:08 PM
2:55:08 PM
TASK [Gathering Facts] *********************************************************
2:55:09 PM
ok: [10.0.0.129]
2:55:10 PM
ok: [10.0.0.128]
2:55:10 PM
2:55:10 PM
TASK [Update packages with apt] ************************************************
2:55:10 PM
changed: [10.0.0.129]
2:55:11 PM
changed: [10.0.0.128]
2:55:11 PM
2:55:11 PM
TASK [Update packages with yum] ************************************************
2:55:11 PM
skipping: [10.0.0.128]
2:55:11 PM
skipping: [10.0.0.129]
2:55:11 PM
2:55:11 PM
TASK [Upgrade packages with apt] ***********************************************
2:55:12 PM
ok: [10.0.0.129]
2:55:12 PM
ok: [10.0.0.128]
2:55:12 PM
2:55:12 PM
TASK [Upgrade packages with yum] ***********************************************
2:55:12 PM
skipping: [10.0.0.128]
2:55:12 PM
skipping: [10.0.0.129]
2:55:12 PM
2:55:12 PM
PLAY RECAP *********************************************************************
2:55:12 PM
10.0.0.128                 : ok=3    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
2:55:12 PM
10.0.0.129                 : ok=3    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
2:55:12 PM

[fiftin]

@geelenbert I just created Semaphore+MySQL containers from you docker compose file. Added private repo, SSH key with Passphrase, and add its public key to the remote server. All works.

1. please check version of Semaphore (perhaps you installed older version from Docker cache).

2. check the passphrase and SSH private key.

[fiftin]

@geelenbert It looks like I was able to reproduce your issue. Did you backup/restore the project?

Backup don't contains secrets. All keys are empty. Yes, it was a bad idea, I will think how to solve it.

So, what we have as a result. SSH keys with Passphrase works out of box, if not:

1. Make sure you have the latest Semaphore version.
   
2. Make sure you have correct SEMAPHORE_ACCESS_KEY_ENCRYPTION: https://docs.semui.co/administration-guide/security#database-encryption.
3. Update your SSH keys if you restored the project from the backup or you changed the encryption key.
4. Update your SSH keys if all else fails.

[geelenbert]

My version is v2.9.64

I did not restore any backup or anything. Just a blank test device to get a proof of concept of semaphore running before deploying to my homelab.

After re-adding and creating new SSH key to the key store it seems to work.

[geelenbert]

It is still very precies. I've been struggling with it for quite some time now.
I haven't found the "always works" solution yet. So i keep on adding keys, updating keys.
Sometimes you realy need to remove a key and then re-add it, because updating doesn't work.

[fiftin]

Hi @geelenbert ,

Do you provide SEMAPHORE_ACCESS_KEY_ENCRYPTION env variable?

[RussB-626]

Running into an issue with v2.9.70 where I have a test task attempting to establish an SSH Connection, but it's saying the user is none, and then attempts to ssh using the [email protected] and just fails:

Task 80.txt
Task 81.txt
testing-playbook.txt

[rmotapar]

@RussB-626 I struggled with the same error for couple of hours before ending up here on this thread. Based on the screenshot provided above by @fiftin, I downgraded to v2.9.64 from latest (v2.9.70) and it started working.

[Kupro404]

include var ANSIBLE_REMOTE_USER: "root" in the compose file.

[RussB-626]

@rmotapar

@RussB-626 I struggled with the same error for couple of hours before ending up here on this thread. Based on the screenshot provided above by @fiftin, I downgraded to v2.9.64 from latest (v2.9.70) and it started working.

Thank you for that info!

[RussB-626]

@Kupro404

include var ANSIBLE_REMOTE_USER: "root" in the compose file.

Is this something new that has to be done from now on, or is there a plan to change it later?

• Edit: That fixed the issue, thank you

[Kupro404]

@RussB-626
This is the equivalent env for the field you filled russ in keystore. Its says it is optional but it always try to ssh with username semaphore if left empty. Regarding plans and fixes you will need to talk with the devs of the project about it. But i left a bigger comment and how i understand and use sema with a compose file earlier in this discussion so you can check it out if you want.