High Memory Usage when reading from Disk Buffer

[ShahroZafar]

Is there a way to know / control the memory usage while reading from buffer?
We have observed that when vector for any reason stores the logs in disk for buffering and starts to read back from it, we see OOMKilled on pods even when the memory limit is configured to be 1024Mi as recommended.
We want to know how buffer uses memory and if there is a way using which we can limit that this much memory can be used while processing buffered events

Our setup
We have kubernetes_logs source which reads the logs of pods and then push it to kafka via kafka sink.

Config file

acknowledgements:
  enabled: true
api:
  address: 0.0.0.0:8686
  enabled: true
  playground: false
data_dir: /vector-data-dir
expire_metrics_secs: 900
sinks:
  kafka:
    batch:
      max_bytes: 1000000
      max_events: 10000
      timeout_secs: 3
    bootstrap_servers: kafka:9092
    buffer:
      max_size: 5000000000
      type: disk
      when_full: block
    compression: zstd
    encoding:
      codec: json
    inputs:
    - dedot_keys
    librdkafka_options:
      client.id: vector
      request.required.acks: "1"
    message_timeout_ms: 0
    topic: vector
    type: kafka
  prometheus_exporter:
    address: 0.0.0.0:9090
    buffer:
      max_size: 5000000000
      type: disk
      when_full: block
    flush_period_secs: 60
    inputs:
    - internal_metrics
    type: prometheus_exporter
sources:
  internal_metrics:
    type: internal_metrics
  kubernetes_logs:
    glob_minimum_cooldown_ms: 2000
    ingestion_timestamp_field: ingest_timestamp
    type: kubernetes_logs
transforms:
  dedot_keys:
    inputs:
    - kubernetes_logs
    source: |
      . = map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
    type: remap

It can be seen here

[jszwedko]

This might be fixed by #20467

[ShahroZafar]

The change seems to be related to kafka source. However in my configuration we have kubernetes_logs source and kafka sink.

[jszwedko]

Ah, you are right, that issue is regarding the kafka source, my mistake.

[ShahroZafar]

So is there any way we can control memory usage here or anything else that can be done?

[jszwedko]

If it is indeed the buffer reading that is using that much memory, I don't think there is anything to be done about that, but I also don't think it is expected that that uses a significant amount of memory. My guess is that the memory is instead being buffered in the kafka sink. I'd try fiddling with some of the librdkafka options. You could also try replacing the kafka sink with a blackhole sink to try to isolate the behavior.