Cannot access host after deploying seed service

[rmadrid24]

Hello,

I am trying to deploy OpenStack (Wallaby) on a CentOS 8.4 VM hosted on JetStream. After running the command kayobe seed service deploy, I get kicked out of my ssh session and I cannot access the host again. Even if I restart the host, I am not able to ssh to it anymore. I have to delete the instance a create a new one.

Any ideas of what could be the issue?

Log:
(kayobe-venv) [rmadridr@js-156-237 kayobe]$ kayobe seed service deploy
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Found both group and host with same name: seed
[WARNING]: Found both group and host with same name: seed-hypervisor

PLAY [Ensure defined container images are deployed on Seed node] ***************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************
ok: [seed]

TASK [deploy-containers : Deploy containers (loop)] ****************************************************************************************************************************

PLAY RECAP *********************************************************************************************************************************************************************
seed : ok=1 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Found both group and host with same name: seed
[WARNING]: Found both group and host with same name: seed-hypervisor

PLAY [Gather facts for localhost] **********************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************
[DEPRECATION WARNING]: Distribution centos 8.4.2105 on host localhost should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings
can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]

PLAY [Validate configuration options for kolla-ansible] ************************************************************************************************************************

PLAY [Ensure Kolla Ansible is configured] **************************************************************************************************************************************

TASK [Look for environment file in Kolla configuration path] *******************************************************************************************************************
ok: [localhost]

TASK [Flag that the Kolla configuration path has been used by another environment] *********************************************************************************************
skipping: [localhost]

TASK [Check whether a Kolla extra globals configuration file exists] ***********************************************************************************************************
ok: [localhost]

TASK [Read the Kolla extra globals configuration file] *************************************************************************************************************************
ok: [localhost]

TASK [Validate Kolla Ansible API address configuration] ************************************************************************************************************************
skipping: [localhost] => (item={'var_name': 'kolla_internal_vip_address', 'description': 'Internal API VIP address', 'required': True})
skipping: [localhost] => (item={'var_name': 'kolla_internal_fqdn', 'description': 'Internal API Fully Qualified Domain Name (FQDN)', 'required': True})
skipping: [localhost] => (item={'var_name': 'kolla_external_vip_address', 'description': 'external API VIP address', 'required': True})
skipping: [localhost] => (item={'var_name': 'kolla_external_fqdn', 'description': 'External API Fully Qualified Domain Name (FQDN)', 'required': True})

TASK [kolla-ansible : Check whether the legacy Kolla overcloud inventory files exist] ******************************************************************************************
ok: [localhost] => (item=seed)
ok: [localhost] => (item=overcloud)

TASK [kolla-ansible : Ensure the legacy Kolla overcloud inventory file is absent] **********************************************************************************************
skipping: [localhost] => (item=seed)
skipping: [localhost] => (item=overcloud)

TASK [kolla-ansible : Ensure the Kolla Ansible configuration directories exist] ************************************************************************************************
ok: [localhost] => (item=/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla)
ok: [localhost] => (item=/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/inventory/seed)
ok: [localhost] => (item=/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/inventory/overcloud/group_vars)
ok: [localhost] => (item=/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/config)
[WARNING]: The value "1000" (type int) was converted to "'1000'" (type string). If this does not look like what you expect, quote the entire value to ensure it does not
change.

TASK [kolla-ansible : Write environment file into Kolla configuration path] ****************************************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure the Kolla global configuration file exists] *******************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Ensure the Kolla seed inventory file exists] *************************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Ensure the Kolla overcloud inventory file exists] ********************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Look for custom Kolla overcloud group vars] **************************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Copy over custom Kolla overcloud group vars] *************************************************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure the Kolla passwords file exists] ******************************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Ensure the Kolla passwords file is copied into place] ****************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Ensure external HAProxy TLS directory exists] ************************************************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure the external HAProxy TLS certificate bundle is copied into place] *********************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure internal HAProxy TLS directory exists] ************************************************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure the internal HAProxy TLS certificate bundle is copied into place] *********************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Find certificates] ***************************************************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Find previously copied certificates] *********************************************************************************************************************
ok: [localhost]

TASK [kolla-ansible : Ensure certificates exist] *******************************************************************************************************************************
skipping: [localhost]

TASK [kolla-ansible : Ensure unnecessary certificates are absent] **************************************************************************************************************

PLAY [Generate Kolla Ansible host vars for the seed host] **********************************************************************************************************************

TASK [Set Kolla Ansible host variables] ****************************************************************************************************************************************
ok: [seed]

TASK [kolla-ansible-host-vars : Ensure the Kolla Ansible host vars directory exists] *******************************************************************************************
ok: [seed]

TASK [kolla-ansible-host-vars : Ensure the Kolla Ansible host vars file exists] ************************************************************************************************
ok: [seed]

PLAY [Generate Kolla Ansible host vars for overcloud hosts] ********************************************************************************************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************************************************************************************************************
localhost : ok=14 changed=0 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
seed : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Found both group and host with same name: seed
[WARNING]: Found both group and host with same name: seed-hypervisor

PLAY [Ensure Kolla Bifrost is configured] **************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************
[DEPRECATION WARNING]: Distribution centos 8.4.2105 on host localhost should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings
can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]

TASK [Check whether a Kolla Bifrost extra globals configuration file exists] ***************************************************************************************************
ok: [localhost]

TASK [Read the Kolla Bifrost extra globals configuration file] *****************************************************************************************************************
ok: [localhost]

TASK [kolla-bifrost : Ensure the Kolla Bifrost configuration directories exist] ************************************************************************************************
changed: [localhost]

TASK [kolla-bifrost : Ensure the Kolla Bifrost configuration files exist] ******************************************************************************************************
changed: [localhost] => (item={'src': 'bifrost.yml.j2', 'dest': 'bifrost.yml'})
changed: [localhost] => (item={'src': 'dib.yml.j2', 'dest': 'dib.yml'})
changed: [localhost] => (item={'src': 'servers.yml.j2', 'dest': 'servers.yml'})

PLAY RECAP *********************************************************************************************************************************************************************
localhost : ok=5 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Deploying Bifrost : ansible-playbook -i /home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/inventory/seed -e @/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/globals.yml -e @/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla/passwords.yml -e CONFIG_DIR=/home/rmadridr/kayobe/config/src/kayobe-config/etc/kolla -e kolla_action=deploy /home/rmadridr/kayobe/config/venvs/kolla-ansible/share/kolla-ansible/ansible/bifrost.yml
[WARNING]: Found both group and host with same name: seed

PLAY [Apply role bifrost] ******************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************
ok: [seed]

TASK [bifrost : include_tasks] *************************************************************************************************************************************************
included: /home/rmadridr/kayobe/config/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/bifrost/tasks/deploy.yml for seed

TASK [bifrost : Ensuring config directories exist] *****************************************************************************************************************************
changed: [seed] => (item=bifrost)

TASK [bifrost : Generate bifrost configs] **************************************************************************************************************************************
changed: [seed] => (item=bifrost)
changed: [seed] => (item=dib)
changed: [seed] => (item=servers)

TASK [bifrost : Template ssh keys] *********************************************************************************************************************************************
changed: [seed] => (item={'src': 'id_rsa', 'dest': 'id_rsa'})
changed: [seed] => (item={'src': 'id_rsa.pub', 'dest': 'id_rsa.pub'})
changed: [seed] => (item={'src': 'ssh_config', 'dest': 'ssh_config'})

TASK [bifrost : Starting bifrost deploy container] *****************************************************************************************************************************
changed: [seed]

TASK [bifrost : Ensure log directories exist] **********************************************************************************************************************************
changed: [seed]

TASK [bifrost : Bootstrap bifrost (this may take several minutes)] *************************************************************************************************************
client_loop: send disconnect: Broken pipe

[markgoddard]

Hi @rmadrid24. Are you using the stable/wallaby branch of a-universe-from-nothing? Strictly Wallaby only supports CentOS Stream, although CentOS Linux is still likely to work. 8.4 is a very recent release though, and probably untested.

My instinct says it sounds like a firewall issue, although I haven't come across anything like it.

[markgoddard]

Which host do you get locked out of? The seed VM, or the instance on Jetstream?

[rmadrid24]

Hi @markgoddard, yes, I am using the stable/wallaby branch for both repos, kayobe and a-universe-from-nothing. I get locked out of the instance on Jetstream.

I also think it is a firewall issue, but I am not sure which configuration could be causing this.

[markgoddard]

Another option could be port security. Are you able to disable port security on your Jetstream instance?