[forward plugin] tls.verify breaks in v3.0.7 (ubuntu22, ca-certificates: c20230311ubuntu0.22.04.1)

[mattmooree]

Bug Report

Describe the bug
In fluent-bit v3.0.7, when enabling tls and setting tls.verify to true in the "forward" plugin the TLS connection fails due to "[tls] error: unexpected EOF with reason: certificate verify failed"

To Reproduce
Create an OUTPUT section which forwards logs to another instance of fluent-bit running on a remote host.

[SERVICE]
Log_Level debug
Parsers_File /etc/fluent-bit/parsers.conf

[INPUT]
Name tail
Path /opt/tomcat/apache-tomcat-9.0.87/logs/catalina.out
Skip_Long_Lines On
Tag catalina

[OUTPUT]
Name forward
Match *
Compress gzip
Host fluent-bit-aggregator.uswe2.devtools.aws.cwan.io
tls On
Port 443

Expected behavior
Fluent-bit connects to the upstream service successfully with TLS certification validation enabled.

Screenshots

v3.0.7 Not working

v3.0.6 Working (on the same host)

Your Environment

• Version used: v3.0.7
• Configuration: (See above)
• Environment name and version: Amazon EC2
• Operating System and version: Ubuntu22 (Jammy Jellyfish)
• uname -a (Linux 6.5.0-1018-aws HTTP Input #18~22.04.1-Ubuntu SMP Fri Apr 5 17:44:33 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux)
• Filters and plugins: forward

apt show fluent-bit

Package: fluent-bit
Version: 3.0.7
Priority: optional
Section: devel
Maintainer: Eduardo Silva [email protected]
Installed-Size: 96.1 MB
Depends: libc6 (>= 2.34), libgcc-s1 (>= 4.2), libpq5 (>= 9.0~), libsasl2-2 (>= 2.1.27+dfsg2), libssl3 (>= 3.0.0~~alpha1), libsystemd0 (>= 221), libyaml-0-2, zlib1g (>= 1:1.2.0)
Download-Size: 41.6 MB
APT-Manual-Installed: yes
APT-Sources: https://packages.fluentbit.io/ubuntu/jammy jammy/main amd64 Packages
Description: Fast data collector for Linux
Fluent Bit is a high performance and multi platform Log Forwarder.

apt show ca-certificates

Package: ca-certificates
Version: 20230311ubuntu0.22.04.1
Status: install ok installed
Priority: important
Section: misc
Maintainer: Ubuntu Developers [email protected]
Original-Maintainer: Julien Cristau [email protected]
Installed-Size: 399 kB
Depends: openssl (>= 1.1.1), debconf (>= 0.5) | debconf-2.0
Breaks: ca-certificates-java (<< 20121112+nmu1)
Enhances: openssl
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Common CA certificates
Contains the certificate authorities shipped with Mozilla's browser to allow
SSL-based applications to check for the authenticity of SSL connections.
.
Please note that Debian can neither confirm nor deny whether the
certificate authorities whose certificates are included in this package
have in any way been audited for trustworthiness or RFC 3647 compliance.
Full responsibility to assess them belongs to the local system
administrator.

Additional context
Key additional context here is that we are using an AWS Certificate Manager certificate for the remote fluent-bit-aggregator hosts. ca-certificates and the EC2 instances should have all of the root certificates required to connect using TLS. Also - the exact same config on the exact same host works with fluent-bit v3.0.6, so this is not an OS breaking change, but a fluent-bit version incompatibility.

[mattmooree]

Potentially related to #8959

[mabrarov]

Hi @mattmooree,

My team is working on upgrade of Fluent Bit from 3.0.6 to 3.0.7 and we faced similar issue with TLS. In our case the root cause is invalid server TLS certificate provided by Fluentd (we connect Fluent Bit with Forward output plugin to Fluentd) - the CN and SAN of certificate don't match hostname / IP address used in configuration of Forward output plugin - i.e. it is an issue owned by my team (if we use correct server certificate then issue is resolved and both 3.0.6 and 3.0.7 versions work as expected).

It looks like #8934 (which is about #8072) introduced hostname verification into Fluent Bit TLS support. Unfortunately, release notes published on GitHub (https://github.com/fluent/fluent-bit/releases/tag/v3.0.7) don't cover that (don't reference #8072), but they reference https://fluentbit.io/announcements/v3.0.7/ which has:

What’s new ?

• Core
  • tls: fix virtual host verification when tls.verify is enabled

Thank you.