Adding an error label to the function.calls metric?

[emschwartz]

If you're debugging an alert based on an SLO and you go to the SLO dashboard to see which functions are erroring, the first question you're going to want to ask for the functions that have had high error rates recently is what types of errors they were.

Should we add an error label to the function.calls metric that would tell you the type of error the function is returning/throwing?

The intention would be for the implementations to take only the error type (not the full string of an error), or use error enum variant names in languages that have those. The values themselves wouldn't be machine-interpretable but the idea would be to provide human debuggers some additional information when they're looking at the metrics for a function.

I think the main downside to this would be the implications for the cardinality of the metrics. If you had a function returning a lot of different errors, you might increase the metrics cardinality more than we really want to.

Thoughts?

[IvanMerrill]

Hmm, when I first read this I thought that sounds very useful, but the more I think about it the less useful it becomes. Some questions I have:

• How automatic would the implementation be? Anything that would require a non-trivial amount of work to configure moves away from the core promise of autometrics IMO.
• How many different types of failure would any one function have? I'm wondering if there would ever be more than one or two failure types with perhaps the exception of functions with external dependencies.
• This then brings me onto the question of whether this would be able to be applied in specific functions as opposed to the whole codebase? If we're concerned about cardinality then maybe limiting to a few functions doing a lot of heavy-lifting might ease this concern?

It appears to me that the value in this proposition would exist in the cases where saying 'this function is producing a lot of errors' still leaves you with little idea as to what could actually be happening. I guess my assumption was that in most cases this wouldn't be the case because you know what the function is doing, therefore how it could fail.

[emschwartz]

How automatic would the implementation be? Anything that would require a non-trivial amount of work to configure moves away from the core promise of autometrics IMO.

Automatic. In dynamic languages, we'd be able to get it without the user doing anything. In Rust, you'd need to derive a trait for your error enum (which is a single line #[derive(AutometricsError)] per error type).

How many different types of failure would any one function have? I'm wondering if there would ever be more than one or two failure types with perhaps the exception of functions with external dependencies.

Often quite a few. I'd say between 4 and 12 for most functions.

This then brings me onto the question of whether this would be able to be applied in specific functions as opposed to the whole codebase?

Sure, it could be opt-in or opt-out

I guess my assumption was that in most cases this wouldn't be the case because you know what the function is doing, therefore how it could fail.

I think that's not quite right. If your function only does one single thing, it's probably too small a function. A small handful of tasks is generally better.

[keturiosakys]

I'm torn on this. On the one hand I agree this would probably be helpful for engineers debugging an issue to get more details from the on-set about the nature of it.

On the other hand I'm really not sure the drawbacks outweight the benefits here. In Rust, enums are helpful indications of an error value that inform you quite well about the feature. On the other hand e.g.: in JavaScript there are 7 native error types (Error, RangeError, TypeError...) but in practice they're hardly helpful without the full stack trace. I find myself looking more into the error message and the line numbers on the stack trace rather than trying to infer from the error type. Python has a lot more error types and maybe they're more helpful indicators but as far as I remember stack trace is often the better starting point there as well.

[emschwartz]

I agree this wouldn't be that useful just going off of built-in error types. For TypeScript, I was imagining more the situation where you create custom error types for specific cases. I don't know what you'd normally do in Python though

[emschwartz]

We could also add the error label to the spec and say it's optional. If it provides useful information for certain programming languages and not others, those implementations could produce it and the shared visualization tools would show it when it's present and ignore it when it's not.

[keturiosakys]

Yeah I think it could work as an optional item. Then the languages that can make use of it - would

[IvanMerrill]

This seems like a good compromise to me as well.