feat(dotnet): export opentelemetry metrics

[rhamzeh]

Description

Adds initial OpenTelemetry metrics to the .NET SDK

References

Generates: openfga/dotnet-sdk#69

Review Checklist

• I have clicked on "allow edits by maintainers".
• I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
• The correct base branch is being used, if not main
• I have added tests to validate that the change in functionality is working as expected

[Hawxy]

Worth mentioning that these shouldn't be tags. Custom metrics ingest for most SaaS distributed tracing vendors charge based on the unique combinations of the metrics tags, so a latency distribution of 50-150ms would be result in "100 custom metrics" being charged, times the number of unique combinations of the other tags.

[rhamzeh]

Thanks for heads up @Hawxy!

We removed these tags for now, but we're thinking of converting them to a bucket, something like:

0-5ms
5-15ms
15-50ms
50-150ms
150ms-500ms
500ms+

But your question raises a few other problems for the other attributes/tags

• fga-client.user will be a problem for anyone with more than a couple of users
• Model ID, Store ID, URL (Full) will be a problem for anyone who is working across many stores/models

We are thinking of making them Opt-in, do you have feedback on how this is normally approached?
We can add config on our side to Opt-in to them. But we were wondering if OTEL has standard tooling/config to opt-in/out of specific tags.

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/customizing-the-sdk/README.md#select-specific-tags seems to indicate that end users can choose to drop specific tags they don't need

[Hawxy]

but we're thinking of converting them to a bucket, something like:

The question to ask is "are users going to slice metrics by these tags", which is the whole point of the tags to begin with. Keep in mind that these aren't tracing tags where you'd typically go ham with a pile of metadata to bake into individual traces. Does it make sense for a user to filter a metric graph of fga-client.request.duration by http.client.request.duration? I don't really think so, thus it's not a valuable tag to include. Is slicing by http.request.resend_count something people are going to do?

We are thinking of making them Opt-in, do you have feedback on how this is normally approached?

Some projects add a configuration option added to enable verbose tagging/metrics. Delegating to OTEL filtering is fine, albeit a bit clunky.

[rhamzeh]

Thinking a bit more about this, we're thinking of adding "fine-grained" config for the attributes/tags.

Something along the lines of:

var configuration = new ClientConfiguration() {
    ApiUrl = "http://localhost:8080",
    StoreId = "...",
    Credentials = new Credentials() { ... },
    Telemetry = new OpenFgaTelemetryConfig {
        Metrics: {
            [TelemetryHistograms.RequestDuration] = {
                Attributes: [Attributes.AttributeRequestMethod, Attributes.AttributeRequestStoreId]
            },
            [TelemetryCounters.TokenExchangeCountKey] = {
                Attributes: [Attributes.AttributeRequestModelId, Attributes.AttributeRequestClientId]
            },
        }
    }
};
var fgaClient = new OpenFgaClient(configuration);

If not set, we would enable a base set of metrics with minimal attributes, if configured, we follow whatever is configured. We will couple that with warnings in the OTEL config documentation around which attributes could be cost-prohibitive.

This allows folks to not enable this by accident (they'd have to manually opt-in), while giving them the ability to be able to have visibility on things like:

• Whether a client id is sending a disproportionate amount of calls or request tokens (could be an indication that it was misconfigured - eg.g they are initializing the SDK multiple times causing a credential request per call)
• Whether their new model is causing significantly more latency than the old one
• Whether slow requests are due to retries (tracing helps here, but usually traces are sampled and people might miss this)
• The ratio of success vs. bad requests vs rate limits by model id, store id or client id so folks can understand whether a particular client is being called incorrectly or a particular model is problematic
• Understanding whether they have still old clients running that they need to upgrade and how that goes with the errors they are getting (through the user agent)

How does that sound to you?

[Hawxy]

Yep, that sounds pretty good.

[rhamzeh]

@Hawxy FYI - this has just been released in v0.5.1 - we're backporting those to the SDK Generator soon

Docs: https://github.com/openfga/dotnet-sdk/blob/main/OpenTelemetry.md

[evansims]

LGTM