Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"error encoding and sending metric family" on HTTP/2 HEAD request #611

Open
es-korneev opened this issue Mar 23, 2024 · 0 comments
Open

Comments

@es-korneev
Copy link

Hi!

I have encountered probably undesirable behaviour of node_exporter when running HTTP/2. node_exporter complained about HTTP/2 write errors when HEAD request method was used. Steps to reproduce:

  1. Run node_exporter with TLS:
$ ./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

# node_exp.yml
---
tls_server_config:
  cert_file: node_exp.crt
  key_file: node_exp.key
  1. Send a request with curl:
$ curl https://localhost:9101 -Ik

curl will display 200 OK and exit cleanly, though node_exporter will write 299 lines like this:

ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"
  1. The error will show up in promhttp_metric_handler_errors_total metric:
$ curl https://localhost:9101/metrics -sk | grep encoding
promhttp_metric_handler_errors_total{cause="encoding"} 299

I have tracked errors down to prometheus/[email protected]/expfmt/text_create.go:96, http2 connection complained about invalid writes len(metric_families) times. HTTP/1.1 deals OK with such requests.

Not sure whether it is a problem related to HTTP/2 implementation in Go or error handling issue in promhttp, but I feel like some sort of error handling can be added to the library.

Thank you for any comments!

Some info about environment:

Host operating system: output of uname -a

Linux host 6.2.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct  6 10:23:26 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.7.0 (branch: master, revision: 6425f079d162ebd22d4c6c4e4d7e4a36ebbe2239)
  build user:       mint@mint-Vostro-14-5410
  build date:       20240323-17:52:29
  go version:       go1.22.1
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

# node_exp.yml
---
tls_server_config:
  cert_file: node_exp.crt
  key_file: node_exp.key

node_exporter log output

# repeated 299 times
ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

curl https://localhost:9101 -Ik

curl --version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant