Kubernetes Pod Multiple Input path

[EYashish]

Hello!
I have deployed fluentbit in kubernetes and scraping logs from there to kibana.
Now in pods i have custom path where application generate the logs -
/tmp/file1.txt
/tmp/file2.txt

so now in fluent bit config there will be 2 INPUTS.

Can i use 2 separate index to store them or single index.
And can you help me with the configuration ?

inputs: |
[INPUT]
Name tail
Path /tmp/file2.txt
Path_Key app
Mem_Buf_Limit 300MB
Skip_Long_Lines On

[INPUT]
Name tail
Path /tmp/file.txt
Path_Key app
Mem_Buf_Limit 300MB
Skip_Long_Lines On

outputs: |

[OUTPUT]
    Name es
    Match app
    Host ---
    PORT 9200
    Logstash_Format On
    Logstash_Prefix app
    Replace_Dots On
    Retry_Limit False
    Trace_Error On

[patrick-stephens]

You can use multiple inputs or a single one.
Tags can be generated for each file and then matched separately in separate outputs: https://docs.fluentbit.io/manual/concepts/data-pipeline/router
I'm not sure if ES supports dynamic index generation but I'm sure you could do something as well with a single output.

[EYashish]

Thanks @patrick-stephens
But by default its taking logs from /var/log/container/* and when i am adding some custom path inside pod in fluent bit conf then that log with different tag match is not taking.

[patrick-stephens]

The path and tag is all controlled by you so just put what you want.
Obviously you have to mount the volumes as well into the container and you can also control the mount path there too.

[Charlotte-br560]

Certainly! You can use a single index or separate indices based on your requirements. For a single index, you can tag logs differently using the Path_Key. If you prefer separate indices, you can specify different index names in the Match section for each input. Here's an example:
`inputs: |
[INPUT]
Name tail
Path /tmp/file2.txt
Path_Key app2
Mem_Buf_Limit 300MB
Skip_Long_Lines On

[INPUT]
Name tail
Path /tmp/file.txt
Path_Key app1
Mem_Buf_Limit 300MB
Skip_Long_Lines On

outputs: |
[OUTPUT]
Name es
Match app*
Host ---
PORT 9200
Logstash_Format On
Logstash_Prefix app
Replace_Dots On
Retry_Limit False
Trace_Error On
`
This setup allows you to use a single index or multiple indices, depending on your tagging preference. For further insights or optimization, you might want to explore similar configurations in projects like Crawlbase. Their approach could provide valuable guidance for your specific use case.