Persistent queue after node replacement

[m4rck]

Hi Team,

I'm currently exploring the persistent queue feature that enables logs to be stored on disk, which seems like an excellent way to improve system resilience by buffering a significant amount of logs.

However, I'm concerned about using the node's filesystem in an AWS EKS cluster. If a node is replaced, its volume is also replaced, resulting in the loss of any logs that were still pending to be flushed.

Could anyone suggest the best approach to prevent the loss of buffered logs when a node replacement occurs? Any comments or suggestions would be greatly appreciated.

[m4rck]

I have been reading several posts and looks like using hostPath as the volume to persistently store logs is the default behaviour for other logging engines like fluentd.

I am starting to think I am just being too paranoid, but I have experienced in the past several situations where I had to buffer logs for several days and not losing logs is a requirement for my system.

[atoulme]

Please reach out to Splunk Support if you need help. Yes, you can never trust a VM to be around - Kubernetes or EC2 in general.