Attention: ADOT Collector upcoming breaking changes - third party exporter deprecation

[vasireddy99]

What is changing?

Last year we shared out intention to unify egress using the OpenTelemetry Protocol (OTLP) exporter. Now, we’re starting to implement this: the AWS Distro for OpenTelemetry (ADOT) collector will start deprecating third party exporters starting from v0.39.0 . The OTLP Exporter helps standardize how data can be exported from ADOT, ensuring minimal security risks and providing the reliable support that you expect from ADOT.

What do I need to do if I’m using third party components?

You are affected if you are using one of the following exporters present in the ADOT collector:

• datadog
• logzio
• sapm
• signalfx

You can enable the functionality behind the feature gate, you can pass the --feature-gates=-adot.exporter.<component-name>.deprecation in the command line parameter.

In ADOT Collector v0.39.0, the feature gate is disabled by default. In a future ADOT Collector release this feature gate will be enabled by default.

[eduardoasantana]

Is the ADOT team aware of any breaking changes in v0.39.0 that would affect third party exporters not listed here?

We are instrumenting our ECS services with the ADOT collector sidecar and using the Datadog exporter to send APM metrics and traces to Datadog.

When we updated some of our services from v0.38.2 to v0.39.0, the collector stopped sending APM metrics to Datadog entirely. Traces were still being sent successfully however.

We didn't get any errors logged from the aws-otel-collector sidecar container. The sidecar started up without issue as well

[Aneurysm9]

When we updated some of our services from v0.38.2 to v0.39.0, the collector stopped sending APM metrics to Datadog entirely. Traces were still being sent successfully however.

We didn't get any errors logged from the aws-otel-collector sidecar container. The sidecar started up without issue as well

Can you please create a support case so that you can confidentially share configuration and log information with us.

[eduardoasantana]

@Aneurysm9 Where should a support case be opened? I don't see an ADOT team in the AWS Support console

[mhausenblas]

Hi @eduardoasantana concerning your question "Where should a support case be opened?". Please work with your TAM to route it to the appropriate team.

[schmidt-galen-heb]

Is the ADOT team aware of any breaking changes in v0.39.0 that would affect third party exporters not listed here?

When we updated some of our services from v0.38.2 to v0.39.0, the collector stopped sending APM metrics to Datadog entirely.

@eduardoasantana we ran into the same issue, and I saw this in the startup logs:

{
   "level": "info",
    "caller": "[email protected]/factory.go:386",
    "msg": "Trace metrics are now disabled in the Datadog Exporter by default. To continue receiving Trace Metrics, configure the Datadog Connector or disable the feature gate.",
    "documentation": "https://docs.datadoghq.com/opentelemetry/guide/migration/",
    "feature gate ID": "exporter.datadogexporter.DisableAPMStats"
}

It looks like ADOT v0.39.0 upgraded the datadogexporter to > 0.95.0, which is what caused the change.

Since ADOT doesn't include the datadogconnector, the only option is to revert the feature gate (disable the DisableAPMStats gate, enabling calculation of APM stats):

--feature-gates=-exporter.datadogexporter.DisableAPMStats

After adding that to the arguments for the ADOT collector, we're seeing APM metrics again.

[eduardoasantana]

DisableAPMStats

@schmidt-galen-heb does that not effectively break APM traces and metrics when using the datadog exporter?

Another thing to note, while v0.39.0 still includes the datadogexporter, the datadogexporter was removed entirely and deprecated starting in v0.41.0 (released in April). So it looks like this is effectively deprecated with no path forward for support.

My understanding is that Datadog customers using otel can no longer use recent versions of the ADOT Collector anymore without also using the Datadog agent, as Datadog doesn't have an OTLP endpoint, but instead expects you to use either the datadogexporter otel exporter or the Datadog agent to send trace metrics to Datadog.

[mhausenblas]

My understanding is that Datadog customers using otel can no longer use recent versions of the ADOT Collector anymore without also using the Datadog agent, as Datadog doesn't have an OTLP endpoint, but instead expects you to use either the datadogexporter otel exporter or the Datadog agent to send trace metrics to Datadog.

ADOT PM here. That is incorrect. Datadog's custom exporter continues to be supported, we're working with them to enable native OTLP ingestion on their end. Until that happens, the Datadog exporter will still be supported.

[schmidt-galen-heb]

@schmidt-galen-heb does that not effectively break APM traces and metrics when using the datadog exporter?

@eduardoasantana DisableAPMStats is enabled by default in v0.95.0+. Setting --feature-gates=-exporter.datadogexporter.DisableAPMStats disables the feature gate, i.e. enables calculation of APM stats (note the hyphen before the feature gate name).

According to the migration guide, disabling the feature gate should simply restore the pre-0.95.0 behavior:

If you’re running a vendor-specific OpenTelemetry distribution that does not include the Datadog Connector, revert to the previous Trace Connector behavior by disabling the exporter.datadogexporter.DisableAPMStats feature gate.

I haven't done exhaustive testing after making the change, but I'm still seeing APM traces, and the APM Trace Metrics started working again.

---

Given that this was an intentional change on the Datadog side not related to the ADOT third party exporter deprecation, if disabling the feature gate doesn't fix the issue for you, I'd suggest opening a new issue

[schmidt-galen-heb]

we're working with them to enable native OTLP ingestion on their end.

@mhausenblas that's great news, I'd definitely prefer native OTLP ingest in Datadog.

This is probably worthy of its own issue, but it would be fantastic if as part of that work the ADOT collector could either add:

• The Schema Transformer Processor, or at least,
• The Transform Processor (previously requested in Support for the transform processor #2300)

Datadog is currently using Semantic Conventions v1.6.1, which is painful when our instrumentation is using a much newer version. Some attributes can be renamed using the Attributes/Resource processors, but it would be much easier with the other processors.

[eduardoasantana]

ADOT PM here. That is incorrect. Datadog's custom exporter continues to be supported, we're working with them to enable native OTLP ingestion on their end. Until that happens, the Datadog exporter will still be supported.

That's fantastic to hear! How does that work when using the ADOT Lambda extension since that doesn't include the Datadog exporter (to my knowledge)?

[luisamador]

I'm running ADOT on ECS Fargate and passing the custom collector config via a parameter in the AWS Parameter store as explained here

I need to be able to disable the feature gate from the ADOT collector config. Is it possible to disable the feature gate from the ADOT collector config? (I can't disable the feature gate via a parameter because ADOT reads the config from a parameter stored in the AWS parameter store)