Streaming device logs may loose log entries

[jellyfish-bot]

[fisehara] [fisehara] [fisehara] [fisehara] Device logs on dashboard and via balena cli are lacking entries when streamed with --tail option on CLI.
All entries are existing when querying logs without --tail option on balena cli.

Most probably here the investigation need to start:

open-balena-api/src/features/device-logs/lib/read.ts

Line 33 in d32efff

const ctx = await getReadContext(req);

See: https://balena.zulipchat.com/#narrow/stream/346007-balena-io.2FbalenaCloud/topic/Random.20missing.20logs

[jellyfish-bot]

[fisehara] This has attached https://jel.ly.fish/4bfc9691-ae87-4101-b110-9d7ce0efa6dc

[jellyfish-bot]

[fisehara] Re-opening as the issue is occurring again.

[fisehara]

Redis PubSub Delivery semantics:
https://redis.io/docs/interact/pubsub/#:~:text=Ctrl%2DC.-,Delivery%20semantics,-Redis%27%20Pub/Sub

The underlying redis PubSub structue only guarantees at-most-once so it's `expected`` to loose messages.
redis-backend needs to implement redis streams: https://redis.io/docs/data-types/streams/