Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki "deduplicates" multiple logs sent using otlp with the same body and timestamp #15425

Open
alexgenon opened this issue Dec 16, 2024 · 0 comments
Labels
OTEL/OLTP type/bug Somehing is not working as expected

Comments

@alexgenon
Copy link

alexgenon commented Dec 16, 2024

Describe the bug
We have Java applications where multiple threads are emitting exactly the same log body at exactly the same time (time granularity is ms). The OpenTelemetry resource attribute thread.name is set and used to distinguish between the 2 logs. However, it looks like, since they have the same timestamp and log body, Loki only takes 1 of the 2 logs entries and drop the other one (kind of a de-duplication which is not taking into account the structured metadata).

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (3.2.0)
  2. Push 1 log using the otlp endpoint. I have attached the logs I used for testing.
  3. Push another log using the otlp endpoint with the same body, attributes and timestamp, just change the thread.name resource attribute.
  4. Query: {service_name="duplicates.tests"} term. I tried this query with both Grafana and logcli
  5. Only 1 of the 2 logs is returned

Expected behavior
2 logs lines returned with different values for thread_name structured metadata.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output
log.json

@JStickler JStickler added type/bug Somehing is not working as expected OTEL/OLTP labels Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OTEL/OLTP type/bug Somehing is not working as expected
Projects
None yet
Development

No branches or pull requests

2 participants