Research how to encrypt traces collected and sent by istio to tempo

[ca-scribner]

Enhancement Proposal

Our set of Istio charms generate traces at a few points (the charms all generate traces, and Istio as a system collects traces for traffic on the service mesh). When we forward these to tempo, we can have a few situations:

• tempo is on the mesh
• tempo is off the mesh

For both these cases, we need a way to encrypt the traces sent to tempo.

When tempo is on the mesh, we (should¹) get this for free as mesh-to-mesh traffic gets mTLS automatically.

When tempo is off the mesh, possible solutions include:

• using tempo's https endpoint, which is self-signed. An issue here is that istio the workload sends some traces directly to tempo and, because tempo's certs are self-signed, it rejects that communication because it cannot verify the certs. So we need to teach istio to verify that cert (discussed more here)
• can istio automate this somehow for us? via an egress, or something else that teaches things on the mesh how to talk with this particular endpoint?

Footnotes

1. Need to confirm this is working as we think ↩