Docker registry not enabled by default, results in impedement when following tutorial

[adammccartney]

While running the pull-retag-push-images.sh script, the /pull-retag-push.yml/
playbook fails on the Push container images (may take a long time) task.

Inspection of the docker logs on the seed vm shows that it fails to connect to the
registry:

May 16 11:21:39 seed dockerd[29263]: time="2024-05-16T11:21:39.784093624Z" level=info msg="Attempting next endpoint for push after error: Get \"https://192.168.33.5:4000/v2/\": dial tcp 192.168.33.5:4000: connect: connection refused" spanID=56f2c85d5a1eb66e traceID=1d69a163f4ea809d25bbc3043b651a88
May 16 11:21:39 seed dockerd[29263]: time="2024-05-16T11:21:39.784305527Z" level=info msg="Attempting next endpoint for push after error: Get \"http://192.168.33.5:4000/v2/\": dial tcp 192.168.33.5:4000: connect: connection refused" spanID=56f2c85d5a1eb66e traceID=1d69a163f4ea809d25bbc3043b651a88

Inspection of docker in the seed vm shows that the registry container is not running.

To debug this, rerun the previous step with some verbose output kayobe seed host configure -vvv

Line 8491 in the following output shows 'enabled': False. This is due to the

...
8467
8468 TASK [docker-registry : Ensure Docker registry container is running] ***********
8469 task path: /home/rocky/kayobe/ansible/roles/docker-registry/tasks/deploy.yml:4
8470 redirecting (type: modules) ansible.builtin.docker_container to community.docker.docker_container
8471 <192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
8472 <192.168.33.5> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stac»
8473 <192.168.33.5> (0, b'/home/stack\n', b'')
8474 <192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
8475 <192.168.33.5> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stac»
8476 <192.168.33.5> (0, b'ansible-tmp-1715859604.244917-107649-217300013138574=/home/stack/.ansible/tmp/ansible-tmp-1715859604.244917-107649-217300013138574\n', b'')
8477 redirecting (type: modules) ansible.builtin.docker_container to community.docker.docker_container
8478 Using module file /home/rocky/kayobe-venv/lib64/python3.9/site-packages/ansible_collections/community/docker/plugins/modules/docker_container.py
8479 <192.168.33.5> PUT /home/rocky/.ansible/tmp/ansible-local-106305qo27bpff/tmpc7a4w7y2 TO /home/stack/.ansible/tmp/ansible-tmp-1715859604.244917-107649-217300013138574/AnsiballZ_docker_container.py
8480 <192.168.33.5> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User»
8481 <192.168.33.5> (0, b'sftp> put /home/rocky/.ansible/tmp/ansible-local-106305qo27bpff/tmpc7a4w7y2 /home/stack/.ansible/tmp/ansible-tmp-1715859604.244917-107649-217300013138574/AnsiballZ_docker_container.py\n', b'')
8482 <192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
8483 <192.168.33.5> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stac»
8484 <192.168.33.5> (0, b'', b'')
8485 <192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
8486 <192.168.33.5> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stac»
8487 <192.168.33.5> (0, b'\r\n{"changed": false, "invocation": {"module_args": {"env": {"REGISTRY_HTTP_ADDR": "0.0.0.0:4000"}, "image": "registry:latest", "name": "docker_registry", "network_mode": "host", "ports": [], "restart_policy":»
8488 <192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
8489 <192.168.33.5> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stac»
8490 <192.168.33.5> (0, b'', b'')
8491 ok: [seed] => (item={'key': 'docker_registry', 'value': {'container_name': 'docker_registry', 'env': {'REGISTRY_HTTP_ADDR': '0.0.0.0:4000'}, 'enabled': False, 'image': 'registry:latest', 'network_mode': 'host', 'ports': [], 'volume»
8492     "ansible_loop_var": "item",
8493     "changed": false,
8494     "invocation": {
8495         "module_args": {
8496             "api_version": "auto",
8497             "auto_remove": null,
...

Recursive grepping for the docker_registry_enabled variable shows that it is
being set in a couple of places. Importantly, the default (False) is set in
ansible/inventory/group_vars/all/docker-registry.

This value does not get overwritten by the role default due to the precedence
rules of ansible variables.¹ So the behaviour that the variable defaults to
False is expected, but nevertheless surprising when you follow the tutorial! The following output shows what files touch the variable:

[kayobe-venv] rocky@ad-univ-mu ~/kayobe  (stable/2023.1)
> rg registry_enabled
etc/kayobe/docker.yml
28:# Default is false, unless docker_registry_enabled is true and

etc/kayobe/docker-registry.yml
6:#docker_registry_enabled:

doc/source/configuration/reference/docker-registry.rst
18:``docker_registry_enabled``

ansible/roles/docker-registry/defaults/main.yml
9:docker_registry_enabled: true
48:    enabled: "{{ docker_registry_enabled }}"

ansible/roles/docker-registry/README.md
18:``docker_registry_enabled``: Whether the Docker registry is enabled. Defaults

ansible/inventory/group_vars/all/docker-registry
6:docker_registry_enabled: False

ansible/inventory/group_vars/all/docker
28:# Default is false, unless docker_registry_enabled is true and
30:docker_registry_insecure: "{{ docker_registry_enabled | bool and not docker_registry_enable_tls | bool }}"

config/src/kayobe-config/etc/kayobe/kolla.yml
89:# images. Default is false, unless docker_registry_enabled is true and

config/src/kayobe-config/etc/kayobe/docker.yml
29:# Default is false, unless docker_registry_enabled is true and

config/src/kayobe-config/etc/kayobe/docker-registry.yml
6:#docker_registry_enabled:

Workaround

Set the following in config/src/kayobe-config/etc/kayobe/docker-registry.yml

docker_registry_enabled: true

Possible solution

Maybe just mention this in the top level readme?

Footnotes

1. https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence ↩