INFO: Collecting Logs on Bottlerocket AMI

[PettitWesley]

Bottlerocket log collection

No host logs on Bottlerocket

UPDATE: please see new workaround from community: #136 (comment)

The Bottlerocket AMIs are meant to be a very stripped down container OS. Consequently, there are fewer log types to collect.

I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket.

On Bottlerocket, the /aws/containerinsights/Cluster_Name/host will not be populated because /var/log/dmesg, /var/log/secure, and /var/log/messages files do not exist on bottlerocket.

dmesg logs can be obtained with journalctl -k or journalctl --dmesg:

bash-5.1# journalctl -k
May 19 19:21:55 localhost kernel: Linux version 5.15.108 (builder@buildkitsandbox) (x86_64-bottlerocket-linux-gnu-gcc (Buildroot 2022.11.1) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP Tue May 9 23:54:36 UTC 2023
May 19 19:21:55 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/vmlinuz console=tty0 console=ttyS0,115200n8 net.ifnames=0 netdog.default-interface=eth0:dhcp4,dhcp6? quiet bootconfig root=/dev/dm-0 rootwait ro raid=noautodetect random.trust_cpu=on selinux=1 enforcing=1 "dm-mod.create=root,,,ro,0 1884160 verity 1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=2 4096 4096 235520 1 sha256 f12f74a243a23635effccde072228bc55a9f06c4c6e587001619bff01f6f8a16 80248401eae3104a1d94e19d04da0bc2b90289d599845063bdf6b330c422e713 2 restart_on_corruption ignore_zero_blocks" -- systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
May 19 19:21:55 localhost kernel: KASLR enabled
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE

However, Fluent Bit appears to only be able to collect systemd unit journald logs. The docs say that it can filter by systemd unit file only: https://docs.fluentbit.io/manual/pipeline/inputs/systemd

And when I tried an input with no filters, no logs were collected:

[INPUT]
        Name                systemd
        Tag                 systemd.all.*
        DB                  /var/fluent-bit/state/systemd.db
        Read_From_Tail      Off

Here's the contents of /var/log on my node:

[ec2-user@admin]$ sudo sheltie
bash-5.1# ls /var/log
aws-routed-eni    containers  dmesg  journal  kdump  pods  private  support

The logs in aws-routed-eni might be interesting to some users and could be collected with a Tail input: https://docs.fluentbit.io/manual/pipeline/inputs/tail

bash-5.1# cd aws-routed-eni/
bash-5.1# ls
egress-v4-plugin.log  ipamd.log  plugin.log

Notes:

• There is a dmesg directory on my node in /var/log, but its empty. I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket. As noted above, you can use journalctl to obtain them.
• Make sure you use the admin container if you choose to poke around the filesystem. I found that it seemed some directories were not visible without admin.

Pod Logs will still be collected

Pod logs can still be collected without any change in experience.

Dataplane logs will still be collected

Kubelet and Containerd logs can still be collected without any change in experience.

[INPUT]
        Name                systemd
        Tag                 dataplane.systemd.*
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        DB                  /var/fluent-bit/state/systemd.db
        Path                /var/log/journal
        Read_From_Tail      ${READ_FROM_TAIL}

[arianvp]

FYI Systemd_Filter can take any journald filter! not just _SYSTEMD_UNIT

If you look at the manpage of journalctl you'll see:

       -k, --dmesg
           Show only kernel messages. This implies -b and adds the match
           "_TRANSPORT=kernel".

           Added in version 205.

AKA you can use the following to get dmesg logs:

[INPUT]
        Name                systemd
        Tag                 host.dmesg.*
        Systemd_Filter      _TRANSPORT=kernel

I hope this helps solve your problem

[joebowbeer]

Thanks for the info!

The docs need to include this:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/kubernetes-eks-logging.html