Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] InfluxDB storage backend falls behind and loses data #65

Open
robagar opened this issue Feb 15, 2024 · 1 comment
Open

[Bug] InfluxDB storage backend falls behind and loses data #65

robagar opened this issue Feb 15, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@robagar
Copy link

robagar commented Feb 15, 2024

Describe the bug

Under moderately heavy load, the InfluxDB v1 backend falls behind and eventually starts dropping samples.

See https://discordapp.com/channels/914168414178779197/1207326913186762794 for the details

To reproduce

  1. Configure router to store in InfluxDB
  2. Start publisher for any data at a high-ish frequency, say 100Hz
  3. Start publisher at 1Hz for events with a counter
  4. Wait...
  5. Make a get request using a time range for the events

Once the storage falls behind by ~1m 40s, it starts dropping samples in batches, not making PUT requests to InfluxDB for them

System info

zenohd v0.10.1-rc-1-g15b36a0f built with rustc 1.72.0 (5680fa18f 2023-08-23)
InfluxDB 1.8.10

@robagar robagar added the bug Something isn't working label Feb 15, 2024
@robagar
Copy link
Author

robagar commented Mar 12, 2024

The InfluxDB docs recommend sending data in batches for performance, so here's our solution

https://github.com/engineeredarts/zenoh-backend-influxdb/tree/TR-6326-zenoh-influxdb

  • Collect PUT queries and send in single request if put_batch_size is set in the storage volume config
  • Send when batch is full, or put_batch_timeout_ms milliseconds after the first one, default 1000ms

Let me know if you want a PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant