WithOtlpExporter() causes Persistent Containers to be recreated every execution

[afscrome]

If you have a persistent container and try to enable open telemetry on it, the container persistence stops workign, with the container being recreated every time.

builder.AddRedis("redis")
       .WithLifetime(ContainerLifetime.Persistent)
       .WithOtlpExporter()
       ;

I believe this is because the instance id in the OTEL_RESOURCE_ATTRIBUTES environment variable and the api key in OTEL_EXPORTER_OTLP_HEADERS change on each app host restart, and so DCP believes it needs to re-create the container

[DamianEdwards]

FYI @karolz-ms @danegsta @eerhardt

This seems like a pretty fundamental issue with persistent containers in that anything with launch-time randomization will result in the definition changing, e.g. random password on start (like we used to do by default) would cause this too.

A possible tactical fix could be to preserve the OTLP API key in user secrets, like we do resource passwords for container-based integrations.

[karolz-ms]

The default logic that decides if a container instance can be reused takes environment variables into account. So yes, changing one of them per execution will effectively disable the persistence.

That is why LifecycleKey property exists for Containers. We should probably have a design discussion about the default "policy" for container reuse in Aspire and how it could be exposed via app model to allow developers to customize it.

@afscrome @mitchdenny any thoughts on this?