Istio fails to push traces to tempo instance as its unknown for the cluster

[michaeldmitry]

Bug Description

When enabling tracing in istio to push traces to a Tempo instance on the mesh.

meshConfig:
    defaultProviders:
      tracing:
      - otel-tracing
    enablePrometheusMerge: true
    enableTracing: true
    extensionProviders:
    - name: otel-tracing
      opentelemetry:
        port: 4317
        service: tempo-0.tempo-endpoints.mesh.svc.cluster.local

istiod gets flooded with warn logs
2024-11-22T10:21:32.348122Z warn Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:"tempo-0.tempo-endpoints.mesh.svc.cluster.local" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry

Only after creating a ServiceEntry, things work:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: tempo
spec:
  hosts:
  - tempo-0.tempo-endpoints.mesh.svc.cluster.local
  ports:
  - number: 4317
    name: http-otel
    protocol: GRPC
  resolution: DNS

To Reproduce

Deploy istio
Deploy tempo on a mesh model
Integrate istio and tempo on workload-tracing
check logs of deployment/istiod

Environment

microk8s 1.28
juju 3.6/candidate

Relevant log output

2024-11-22T10:21:32.347119Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347222Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347310Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347392Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347467Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347593Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347672Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347756Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347842Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.347943Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.348038Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.348122Z     warn    Not able to configure requested tracing provider "otel-tracing": could not configure tracing provider "envoy.tracers.opentelemetry": could not find cluster for tracing provider "service:\"tempo-0.tempo-endpoints.mesh.svc.cluster.local\" port:4317": could not find service tempo-0.tempo-endpoints.mesh.svc.cluster.local in Istio service registry
2024-11-22T10:21:32.348813Z     info    delta   LDS: PUSH for node:istio-beacon-k8s-mesh-waypoint-6b886fbf5-jvfvs.mesh resources:3 removed:0 size:92.6kB
2024-11-22T10:21:32.349969Z     info    delta   WDS: PUSH for node:istio-beacon-k8s-mesh-waypoint-6b886fbf5-jvfvs.mesh resources:21 removed:0 size:5.2kB

Additional context

No response

[michaeldmitry]

Its not just with a headless service fqdn, but also with IPs (e.g: loadbalancer-provided IP)

[ca-scribner]

The docs here say service is "is a fully qualified host name of a service defined by the Kubernetes service or ServiceEntry", so I guess headless services don't count as services? I think the reason the ServiceEntry fixes this is that it adds this host to Istio's service registry, and resolution: DNS tells Istio to ask k8s to resolve the DNS