TLS certificates stored in ephemeral storage do not persist across pod restarts

[shayancanonical]

Steps to reproduce

1. juju deploy -n 1 mysql-router-k8s
2. juju deploy -n 1 mysql-k8s
3. juju deploy -n 1 mysql-test-app
4. juju deploy -n 1 tls-certificates-operator
5. juju relate mysql-k8s mysql-router-k8s
6. juju relate mysql-router-k8s mysql-test-app
7. juju relate mysql-router-k8s tls-certificates-operator
8. kubectl -n delete pod mysql-router-k8s

Expected behavior

The unit should be able to persist and function across pod deletions/reschedule.

Actual behavior

The tls certificates stored in ephemeral storage are lost upon pod restart.

As a bonus, I also run into the following error trace upon the start hook when I run kubectl -n <namespace> delete pod mysql-router-k8s-0

unit-mysql-router-k8s-0: 19:17:56 ERROR unit.mysql-router-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/kubernetes_charm.py", line 183, in <module>
    ops.main.main(KubernetesRouterCharm)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/abstract_charm.py", line 189, in reconcile
    if not self._upgrade.is_compatible:
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/upgrade.py", line 70, in is_compatible
    previous_versions: dict[str, poetry_version.Version] = {
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/upgrade.py", line 71, in <dictcomp>
    key: poetry_version.Version.parse(value)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/poetry/core/version/pep440/version.py", line 186, in parse
    return parse_pep440(value, cls)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/poetry/core/version/pep440/parser.py", line 84, in parse_pep440
    return PEP440Parser.parse(value, version_class)  # type: ignore[arg-type]
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/poetry/core/version/pep440/parser.py", line 70, in parse
    raise InvalidVersion(f"Invalid PEP 440 version: '{value}'")
poetry.core.version.exceptions.InvalidVersion: Invalid PEP 440 version: '1+86e8516-dirty+86e8516-dirty+86e8516-dirty+26d0368-dirty'

Versions

Operating system: Ubuntu 22.04.2 LTS

Juju CLI: 2.9.45-ubuntu-amd64

Juju agent: 2.9.45

mysql-k8s charm revision: 99
mysql-router-k8s charm revision: 1 (locally built)

microk8s: MicroK8s v1.25.14 revision 5976

Log output

Juju debug log:

Additional context

[github-actions]

https://warthogs.atlassian.net/browse/DPE-2857

[carlcsaposs-canonical]

@shayancanonical I believe 1+86e8516-dirty+86e8516-dirty+86e8516-dirty+26d0368-dirty is caused by CTRL-C-ing tox run -e build—see tox-dev/tox#2377 —that prevents the cleanup from running (so for now you need to manually revert charm_version if you CTRL-C the build). We might want to switch from tox to something like make to resolve this issue

[carlcsaposs-canonical]

@shayancanonical Which TLS certificates do you see getting stored in the container storage? I thought they were stored in a peer databag, but not sure if I'm thinking of the same certs

[shayancanonical]

@carlcsaposs-canonical It is true that the certs are stored in the peer databag. However, to be used by mysqlrouter, the custom certs are also written to /etc/mysqlrouter (which is ephemeral storage).

The issue is that we need these certs when COS integration is formed, which could be anytime after the relation with tls-certificates-operator is formed

[carlcsaposs-canonical]

Are the certs in /etc/mysqlrouter identical to the certs in the peer databag or are they different certs?

[shayancanonical]

The certs retrieved from the tls-certificates-operator are stored in the peer databag and also written to /etc/mysqlrouter/custom-<key/certificate>.pem

[carlcsaposs-canonical]

Ohh thank you! I see the issue now