Enabling model-on-mesh blocks traffic from kubernetes to the model's operator

[ca-scribner]

Bug Description

@wood-push-melon reported that, after turning on the model-on-mesh config, he saw the following ztunnel logs:

2024-11-28T02:45:11.030584Z    error    access    connection complete    src.addr=172.31.20.48:3348 src.workload="kubernetes" src.namespace="default" dst.addr=10.1.48.89:17071 dst.service="modeloperator.testing.svc.cluster.local" dst.workload="modeloperator-797f7c5889-9nn7g" dst.namespace="testing" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection"

Specifically, we see "connection closed due to policy rejection" so I think that means reproduction requires us to have a policy bound at ztunnel to the model operator (maybe there was a global allow-nothing policy in place in @wood-push-melon's env?)

Not sure what communication that would be or whether it is important. Could it be a health check? We'd need to investigate further to triage.

To Reproduce

• deploy beacon to a model
• enable model-on-mesh
• add an authorization policy
• watch the logs in the ztunnel pod to see if anything gets blocked

Environment

Relevant log output

-

Additional context

No response

[wood-push-melon]

Provide more details here.

To reproduce:

• Deploy any application charms in a juju model
• Deploy beacon charm in the same juju model, and set model-on-mesh to True. This will put everything in this model into the mesh

Watch the logs of the ztunnel pod. There will be inbound traffic from other juju models (outside of the mesh) blocked by ztunnel. Here are some examples:

2024-12-12T00:59:26.629426Z	error	access	connection complete	src.addr=172.31.20.48:61196 dst.addr=10.1.48.86:17071 dst.service="modeloperator.testing.svc.cluster.local" dst.workload="modeloperator-54cf4669cc-wcxg7" dst.namespace="testing" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: allow policies exist, but none allowed"
2024-12-12T00:59:26.638658Z	error	access	connection complete	src.addr=172.31.20.48:62652 dst.addr=10.1.48.86:17071 dst.service="modeloperator.testing.svc.cluster.local" dst.workload="modeloperator-54cf4669cc-wcxg7" dst.namespace="testing" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: allow policies exist, but none allowed"
2024-12-12T00:59:26.647341Z	error	access	connection complete	src.addr=172.31.20.48:43358 dst.addr=10.1.48.86:17071 dst.service="modeloperator.testing.svc.cluster.local" dst.workload="modeloperator-54cf4669cc-wcxg7" dst.namespace="testing" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: allow policies exist, but none allowed"
2024-12-12T00:59:26.883653Z	error	access	connection complete	src.addr=172.31.20.48:25666 dst.addr=10.1.48.86:17071 dst.service="modeloperator.testing.svc.cluster.local" dst.workload="modeloperator-54cf4669cc-wcxg7" dst.namespace="testing" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: allow policies exist, but none allowed"

There are not any AuthorizationPolicy resources exist in all namespaces:

$ k get authorizationpolicies.security.istio.io -A
No resources found

The ztunnel policies:

$ istioctl ztunnel-config policies ztunnel-fl58b.istio-system -o json
[
    {
        "name": "istio_allow_waypoint_testing_istio-beacon-k8s-testing-waypoint",
        "namespace": "testing",
        "scope": "WorkloadSelector",
        "action": "Allow",
        "rules": [
            [
                [
                    {
                        "principals": [
                            {
                                "Exact": "cluster.local/ns/testing/sa/istio-beacon-k8s-testing-waypoint"
                            }
                        ]
                    }
                ]
            ]
        ]
    },
    {
        "name": "istio_allow_waypoint_testing_public-ingress",
        "namespace": "testing",
        "scope": "WorkloadSelector",
        "action": "Allow",
        "rules": [
            [
                [
                    {
                        "principals": [
                            {
                                "Exact": "cluster.local/ns/testing/sa/public-ingress-istio"
                            }
                        ]
                    }
                ]
            ]
        ]
    }
]

The naive resolution is to apply an "allow all" Authorization Policy, and everything goes OK:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all
  namespace: testing
spec:
  rules:
  - {}

However, this resolution is not acceptable for real-world cases.