Batch writes in the file sink

[jszwedko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The file sink performance, especially when compressing, seems to be particular poor as evidenced by the investigation in #20739. One culprit of this seems likely to be that the file sink writes (and compresses) one event at a time. I think the throughput is likely to improve through batching writes.

Attempted Solutions

No response

Proposal

Add batch configuration to the file sink and batch writes.

References

• vector only sinks a tiny amount of data coming from a dnstap source, except if you sink to blackhole! #20739

Version

v0.39.0

[johnhtodd]

It would seem that one event at a time with compression would lower the symbol set massively, leading to very poor compression values. Isn't this what the "buffer" keyword implies? My expectation is that the number of events indicated by "buffer" would be put into a queue, and then compressed/written when that number was exceeded. (This begs the question of "why not a time- or bytes-based buffer flush setting as well?")

[jszwedko]

It would seem that one event at a time with compression would lower the symbol set massively, leading to very poor compression values. Isn't this what the "buffer" keyword implies? My expectation is that the number of events indicated by "buffer" would be put into a queue, and then compressed/written when that number was exceeded. (This begs the question of "why not a time- or bytes-based buffer flush setting as well?")

The buffer actually controls the buffer of input events to the sink. I think what you are referring to would commonly be controlled by the batch settings on the sink, but the file sink has no batch settings. There is a diagram that tries to distinguish between how the buffer is independent of sink batching here: https://vector.dev/docs/reference/configuration/sinks/http/#buffers-and-batches. It's admittedly a bit confusing.

[johnhtodd]

Thanks for the clarification @jszwedko - I didn't quite understand those differences and that is useful for other things I'm working on.

So what you're suggesting is actually adding "batch" parameters to the "file" sink? Does compression happen after buffering, or after batching, and is that consistent in all the sinks?

[jszwedko]

So what you're suggesting is actually adding "batch" parameters to the "file" sink? Does compression happen after buffering, or after batching, and is that consistent in all the sinks?

Yeah, that's correct, I'd like to add batch parameters to the file sink to batch the writes. I believe compression happens after batching. I'm pretty sure that's consistent across sinks. It definitely doesn't happen as part of buffering.