FIPS Problem

[sdurig1]

Having a problem with FIPS on an ipareplica.

I created 2 EC2 instances from the same AMI. Both have FIPS enabled.

I ran the cluster playbook with one of them as the ipaserver and the other as an ipareplica. The server installed just fine, but the replica fails at the step: "ipareplica : Install - Replica preparation"

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ipapython.admintool.ScriptError: Cannot join FIPS-enabled replica into existing topology: FIPS is not enabled on the master server. fatal: [IDM2]: FAILED! => {"changed": false, "module_stderr": "Shared connection to IDM2 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/home/myuser/.ansible/tmp/ansible-tmp-1691012352.7248142-16788-186937430347613/AnsiballZ_ipareplica_prepare.py\", line 107, in <module>\r\n _ansiballz_main()\r\n File \"/home/myuser/.ansible/tmp/ansible-tmp-1691012352.7248142-16788-186937430347613/AnsiballZ_ipareplica_prepare.py\", line 99, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/home/myuser/.ansible/tmp/ansible-tmp-1691012352.7248142-16788-186937430347613/AnsiballZ_ipareplica_prepare.py\", line 48, in invoke_module\r\n run_name='__main__', alter_sys=True)\r\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\r\n return _run_module_code(code, init_globals, run_name, mod_spec)\r\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\r\n mod_name, mod_spec, pkg_name, script_name)\r\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\r\n exec(code, run_globals)\r\n File \"/tmp/ansible_ipareplica_prepare_payload_68zpm0xc/ansible_ipareplica_prepare_payload.zip/ansible/modules/ipareplica_prepare.py\", line 944, in <module>\r\n File \"/tmp/ansible_ipareplica_prepare_payload_68zpm0xc/ansible_ipareplica_prepare_payload.zip/ansible/modules/ipareplica_prepare.py\", line 627, in main\r\n File \"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py\", line 548, in check_remote_fips_mode\r\n \"Cannot join FIPS-enabled replica into existing topology: \"\r\nipapython.admintool.ScriptError: Cannot join FIPS-enabled replica into existing topology: FIPS is not enabled on the master server.\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

On both the EC2 instances, "crypto.fips_enabled" is set to 1, and the same in /boot/grub2/grub.cfg. They are both RHEL 8.

I am running it by:

ansible-playbook -u myuser -b --private-key ~/.ssh/mykey -i inventory/hosts.cluster playbooks/install-cluster.yml

Inventory file

[ipaserver]
IDM1

[ipaserver:vars]
ipaserver_auto_forwarders=no

[ipareplicas_tier1]
IDM2

[ipareplicas_tier2]
IDM3

[ipareplicas_tier3]
IDM4 ipareplica_servers=IDM2,IDM3

[ipareplicas_tier1:vars]
ipaclient_force_join=yes
ipareplica_setup_ca=yes

[ipareplicas_tier2:vars]
ipaclient_force_join=yes

[ipareplicas_tier3:vars]
ipaclient_force_join=yes

[ipaclients]

[ipaclients:vars]
#ipaclient_use_otp=yes
#ipaclient_allow_repair=yes

[ipareplicas:children]
ipareplicas_tier1
ipareplicas_tier2
ipareplicas_tier3

[ipa:children]
ipaserver
ipareplicas
#ipaclients

[ipa:vars]
ipaadmin_password=MyPassword
ipadm_password=MyOtherPassword
ipaserver_domain=MyDomain
ipaserver_realm=MyRealm
ipaserver_setup_dns=no
ipaserver_setup_firewalld=no

Playbook

---
- name: Install IPA servers
  hosts: ipaserver
  become: true
  roles:
  - role: ipaserver
    state: present

- name: Install IPA replicas Tier1
  hosts: ipareplicas_tier1
  become: true
  roles:
  - role: ipareplica
    state: present

- name: Install IPA replicas Tier2
  hosts: ipareplicas_tier2
  become: true
  roles:
  - role: ipareplica
    state: present

- name: Install IPA replicas Tier3
  hosts: ipareplicas_tier3
  become: true
  roles:
  - role: ipareplica
    state: present

[t-woerner]

According to the error message "FIPS is not enabled on the master server" it seems that FIPS is not enabled on the server. Is this a fresh deployment or was there a non FIPS deployment before on the server?
Which ansible-freeipa and IPA version are you using?

[sdurig1]

So after turning FIPS off, it fails in the same step but with a different error. Before running the playbook, our servers hostname is fine. After running the playbook, the servers lose their hostname for some reason. This causes this error to appear:

2023-08-03T21:03:31Z DEBUG failed to find session_cookie in persistent storage for principal 'admin@OLD_DOMAIN_WE_ARE_RESUSING'
2023-08-03T21:03:31Z DEBUG trying https://OLD_MASTER_HOST/ipa/json
2023-08-03T21:03:31Z DEBUG New HTTP connection (OLD_MASTER_HOST)
2023-08-03T21:03:32Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=C5Y51AUoG4Aox5SLms8xfmwfWAZA9eKnszkXwkH%2bbNtw9h6HhwayiWZsr4vpZl0l3SAFlp51LDi%2fCE%2b5AzGGhVoBebIWHluCgpouoFj4ToSxP5R4BxD4jNL6xLNsUIRiNTtM%2fOl321Ja3cNgGX0WblbONp6QL6ZmEZP6vgfAfrjAzvs7VPampT7Bc%2bCzqssp;path=/ipa;httponly;secure;']'
2023-08-03T21:03:32Z DEBUG storing cookie 'ipa_session=MagBearerToken=C5Y51AUoG4Aox5SLms8xfmwfWAZA9eKnszkXwkH%2bbNtw9h6HhwayiWZsr4vpZl0l3SAFlp51LDi%2fCE%2b5AzGGhVoBebIWHluCgpouoFj4ToSxP5R4BxD4jNL6xLNsUIRiNTtM%2fOl321Ja3cNgGX0WblbONp6QL6ZmEZP6vgfAfrjAzvs7VPampT7Bc%2bCzqssp;' for principal admin@OLD_DOMAIN_WE_ARE_RESUSING
2023-08-03T21:03:32Z DEBUG Created connection context.rpcclient_140273225443032
2023-08-03T21:03:32Z DEBUG raw: ping(version='2.251')
2023-08-03T21:03:32Z DEBUG ping(version='2.251')
2023-08-03T21:03:32Z DEBUG [try 1]: Forwarding 'ping/1' to json server 'https://OLD_MASTER_HOST/ipa/json'
2023-08-03T21:03:32Z DEBUG HTTP connection keep-alive (OLD_MASTER_HOST)
2023-08-03T21:03:32Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=CoQICvlzt%2bs%2f8cHj8WrLbUq5mQYoHShbiPs3GR%2ftMh5cxI3%2f56IVK8ioYUuErRUcd9ycz4bc8mU7pEpnu5KL5U0QBPxYHNBl3MWhTZXaqoBeq6%2buCms2FWdtyx0%2fcGQ%2fjBxEemrm3hVoqTXvIW2ii8XoIZa6lUYL3pv5UtHMv2wrQRQxShG3ffNCdoRd4dHC;path=/ipa;httponly;secure;']'
2023-08-03T21:03:32Z DEBUG storing cookie 'ipa_session=MagBearerToken=CoQICvlzt%2bs%2f8cHj8WrLbUq5mQYoHShbiPs3GR%2ftMh5cxI3%2f56IVK8ioYUuErRUcd9ycz4bc8mU7pEpnu5KL5U0QBPxYHNBl3MWhTZXaqoBeq6%2buCms2FWdtyx0%2fcGQ%2fjBxEemrm3hVoqTXvIW2ii8XoIZa6lUYL3pv5UtHMv2wrQRQxShG3ffNCdoRd4dHC;' for principal admin@OLD_DOMAIN_WE_ARE_RESUSING
2023-08-03T21:03:32Z INFO Execute check on remote master
2023-08-03T21:03:32Z DEBUG [try 1]: Forwarding 'server_conncheck' to json server 'https://OLD_MASTER_HOST/ipa/json'
2023-08-03T21:03:32Z DEBUG HTTP connection keep-alive (OLD_MASTER_HOST)
2023-08-03T21:03:32Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=watAInwXIdcmWPtLSOOW2XqShlaIfxGksJgnAZfRFRG00r7oRcq5VDj4vwZOXu6VDtaVRV613A6oX12BVCIm07llKYItEzGHGpLeFlx7RW%2faUwWrGup6%2bdE3l%2bioJUriujZT5ehDarF5IBSFeF8Tt8M%2fujGp1fGfzmHYgIt0u48F9qJlbHWaaqt9T12%2bD3Fh;path=/ipa;httponly;secure;']'
2023-08-03T21:03:32Z DEBUG storing cookie 'ipa_session=MagBearerToken=watAInwXIdcmWPtLSOOW2XqShlaIfxGksJgnAZfRFRG00r7oRcq5VDj4vwZOXu6VDtaVRV613A6oX12BVCIm07llKYItEzGHGpLeFlx7RW%2faUwWrGup6%2bdE3l%2bioJUriujZT5ehDarF5IBSFeF8Tt8M%2fujGp1fGfzmHYgIt0u48F9qJlbHWaaqt9T12%2bD3Fh;' for principal admin@OLD_DOMAIN_WE_ARE_RESUSING
2023-08-03T21:03:32Z INFO Check connection from master to remote replica 'IDM2':
2023-08-03T21:03:32Z INFO ERROR: Port check failed! Unable to resolve host name 'IDM2'

[sdurig1]

So here is what seems to be happening: running the ansible changes our hostname to another host that exists on our system. Specifically it populates the /etc/ipa/default.conf file with the wrong hostname.

My suspicion is it is somehow getting the wrong name in the ipareplica role where it runs ipareplica_test and sets hostname as:
hostname: "{{ ipareplica_hostname | default(ansible_facts['fqdn']) }}"

I'm not 100% that this is the correct location of the problem ,but I am 100% sure that our hostname is getting populated with a value that represents our old cluster and doesn't exist in our inventory file.

[rjeffman]

Hostname is obtained through DNS PTR records, check which hostname you get for the failing node.

AWS is known to bring some reverse DNS issues when installing IPA.

[sdurig1]

Yeah we discovered the problem was that since our domain was the same, we had a DNS SRV record that was pulling the old host name. We set some things such as "ipaserver_hostname", "ipareplica_hostname", "ipaclient_hostname" in our vars file, as well as put our host names in /etc/hosts on the boxes to get around this for now. That seemed to work for us.