feat: new promslog and promslog/flag packages to wrap log/slog

[tjhop]

Prereq for prometheus/prometheus#14355

This adds a new promslog package to create an opinionated slog logger
for use within the prometheus ecosystem. By default, logs are written at
info level and formatted to add a kv pair for the source caller. If
backwards compatibility for the literal output format of the logging is
required, logs can be styled in a format that is similar to how the
promlog package formatted it's go-kit/log output [1]. The promslog
package also makes use of an slog.LevelVar to dynamically adjust the
level of an existing logger. Similar to the old promlog package, we
provide means to log in logfmt and json output formats.

Creating a logger is similar to the old promlog package -- optionally
populate a Config struct, and then call New() with the config.

In order to dynamically adjust the logger's level, retain the Config
struct as it's own variable to access the AllowedLevel.Set() method,
which internally updates the AllowedLevel's slog.LevelVar to the
desired log level. Ex:

config := &promslog.Config{} // Retain as variable if you need to dynamically adjust log level
logger := promslog.New(config)
config.Level.Set("debug")
logger.Debug("your message here", "hello", "world")

To use go-kit style log output:

config := &promslog.Config{Style: promslog.GoKitStyle} // Retain as variable if you need to dynamically adjust log level
logger := promslog.New(config)
config.Level.Set("debug")
logger.Debug("your message here", "hello", "world")

1. When using the go-kit log style, the following changes are made to the
   default slog output:

• adjusts slog default timestamp key from timestamp -> ts
• adjusts the timestamp value to use the same time format string
  ("2006-01-02T15:04:05.000Z07:00")
• adjusts slog default sourcecode key from source -> caller
• adjusts the formatting of the sourcecode values to trim paths with
  filepath.Base(). The formatting of the sourcecode value is similar
  to the go-kit/log usage, with the addition of the source function in
  parenthesis when debug logging is enabled.

Signed-off-by: TJ Hoplock [email protected]

[tjhop]

@SuperQ @beorn7:

This should be ready for review now. I believe I was able to set things up in a way where flag interactions can remain consistent (by also creating a new promslog/flag package to pair with the logger). Formatting for the literal log lines themselves should be pretty close old formatting of the logger from promlog, with the exception of the value of the caller source key -- I didn't look into trying to tap into runtime.Caller() myself for slog, as is used behind the scenes when we call log.Caller() here.

Feedback welcome!

[SuperQ]

Nice!

My only question so far is how much we care about keeping the same log line formatting as before. Since this is intended for Prometheus 3.0, we can make breaking changes.

Do we want to deviate from some of the minor slog defaults for formatting? Or use this as an opportunity to reset.

[tjhop]

We can certainly use it as an opportunity to reset -- my primary thought was that by maintaining log formatting, we would cause the least downstream disruption for anything that is parsing the resulting log lines. We're ok with making the breaking changes to formatting for prometheus because of the 3.0 release, but any other downstream projects would also inherit those formatting changes.

I'd like to not be the cause of someone getting paged in the middle of the night because an upgrade to $someService was done and their loki recording/alerting rules broke from unexpected formatting changes, etc 🙃

[SuperQ]

So let's break this down into individual decisions:

1. adjusts slog default timestamp key from timestamp -> ts
2. adjusts the timestamp value to use the same time format string
   ("2006-01-02T15:04:05.000Z07:00")
3. adjusts slog default sourcecode key from source -> caller
4. adjusts the formatting of the sourcecode values to trim paths with
   filepath.Base(). The formatting of the sourcecode value is similar
   to the go-kit/log usage, with the addition of the source function in
   parenthesis when debug logging is enabled.

My vote:

1. Yes
2. No
3. No
4. Maybe

The other option is we add an additional formatting config/flag. --log.style=(slog|go-kit).

[tjhop]

There's also this (it made it in during a final refactor but I forgot to update commit description/PR summary):

5. The normalization of the level key to use lowercase values, since slog's default is uppercase (DEBUG vs debug)

That said, I'll cast my vote:

1. I can go either way for timestamp key name
2. No
3. No
4. Yes for filepath trimming, maybe for adding in source func on debug level
5. Yes (purely personal preference)

[ArthurSens]

I think I'm a bit late with the discussions, but my main concern here is why would a Prometheus or prometheus exporter user decide the logging library used underneath 😬

Was there any related discussions that I've missed?

[tjhop]

I believe I've addressed all PR feedback, the only remaining item should be determining the approach we want to take on the style initialization, whether we take the constructor approach or advocate consumers use the existing methods when prepping the logger config

[ArthurSens]

You could use go's type safety to prevent library users from setting weird log styles:

type LogStyle string

const (
	SlogStyle  LogStyle = "slog"
	GoKitStyle LogStyle = "go-kit"
)

// Config is a struct containing configurable settings for the logger
type Config struct {
	Level    *AllowedLevel
	Format   *AllowedFormat
	Style    LogStyle
	ioWriter io.Writer
}

But while quickly trying this out I realized it would require to refactor the whole implementation 😵 . I'm fine with returning error from the constructor

[ArthurSens]

Hey, I'm creating a new release to unblock the UTF8 work, but I can create another release tomorrow with this PR as well :)

[ArthurSens]

Do you also need a new release ASAP to get things done in Prometheus?

[tjhop]

@ArthurSens @SuperQ the last commit adjusts the log style to be a custom type, it's no longer settable 👍

[tjhop]

Do you also need a new release ASAP to get things done in Prometheus?

A release would be helpful whenever y'all are comfortable with the final change set and have time. For now, I'm using go workspaces to start hacking on the prometheus 3.0 conversion 🙃

[ArthurSens]

Perfect!

[SuperQ]

@tjhop Do you want to squash up commits and finalize the commit/PR descriptions?

[tjhop]

@tjhop Do you want to squash up commits and finalize the commit/PR descriptions?

no worries, can do 👍

[tjhop]

Rebased on current upstream main, squashed to a single commit, and updated the commit description/PR summary with some more accurate docs. Lemme know if you have any further thoughts 👍

[ArthurSens]

Thanks! Merging and doing the release :)

[dswarbrick]

The non-UTC timestamp format here differs from the previous promlog (go-kit) timestamp format, which used .UTC().

	timestampFormat = log.TimestampFormat(
		func() time.Time { return time.Now().UTC() },
		"2006-01-02T15:04:05.000Z07:00",
	)