How is 500 error handled by Serilog.Sinks.Seq?

[MaceWindu]

Could you please clarify Serilog.Sinks.Seq behavior for this error:

Client side: Received failed HTTP shipping result InternalServerError
Server side: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.

Questions are:

• what happens with failed batch: is it discarded or retried?
• If batch is discarded - what could be done on client to avoid data loss?

[nblumhardt]

Hello! The batch will be retried by Serilog in this scenario.

Hope this helps,
Nick

[MaceWindu]

@nblumhardt, have anything changed in recent releases regarding resource management to trigger such errors more often? Yesterday we updated to 2024.3.12021 (not remember from which version exactly, but it was 2024.3.x ) and now I see them from time to time when we load Seq with heavy queries.

Just curious wether it is change of Seq side or update was not very successful

[nblumhardt]

Hi @MaceWindu,

Nothing has changed that I can think of, but I guess because the minimum request body data rate etc. are configured using limits, small changes in the performance or latency of different paths might push the system over the limit more often if it's already close 🤔 (thinking of the .NET thread pool/task scheduling in particular).

You can configure the limits behind these values using the api.minRequestBodyDataRateBytesPerSecond and api.minRequestBodyDataRateGracePeriodMilliseconds configuration options on the Seq server, perhaps you might consider halving the minimum data rate and doubling the grace period to see whether that helps?

Docs on those settings are in: https://docs.datalust.co/docs/json-config#apiminrequestbodydataratebytespersecond

There's also a chance that load has increased on your server, e.g. from new applications coming online, or additional logging statements/traces being added. It might also be worth you taking a brief look at your Data > Ingestion and Data > Storage screens to see where data is coming from, and check that your retention policies are having the intended effect.

Hope this helps,
Nick

[MaceWindu]

Could it be caused by this error?

[2024-07-11T21:01:36.2991866+00:00 DBG] Exception raised when writing to event storage

Seq.Engine.SeqCapacityException: The stream is temporarily read-only.

   at Seq.Engine.Storage.StorageEngine.Add(FlatEvent[] events)

   at Seq.Server.Web.Api.IngestionController.DoIngest(BodyParser parseBody, ResponseFormatter formatResponse)

[nblumhardt]

Yes, that's definitely a possibility. That exception is thrown when incoming requests need to wait more than five seconds to acquire a write lock (essentially). The kinds of things triggering it will usually be I/O related - write IOPS or throughput limits being hit, for example, but any other long stall, such as long GC pauses, could also cause it.

If your machine is running light on I/O, improving that would be the first thing to try.

Hope this helps,
Nick