rcl_logging_syslog 🚢🚀🚂

rcl_logging_syslog is alternative logging backend implementation that can be used for ROS 2 application via rcl_logging_interface.

rcl_logging_syslog uses SYSLOG(3) to send the log data to rsyslog a.k.a rocket-fast system for log processing 🚀.

The main objective is that Enabling ROS 2 logging system with Cloud-Native Log Management and Observability.

see overview slide deck for more information.

Motivation

The logging data is critical especially for entire system observability and status, so that application can alert the administrator or even give the feedback to the system with adjusting the parameter. This importance rises once it comes to robotics and robot application, especially distributed system such as ROS 2 or edge computing because we must be able to specify what went wrong in the 1st place with these logging data.

rsyslog is available in default Ubuntu distribution managed by system service, performative, and many configuration supported including log data pipeline. So that user can choose the logging configuration depending on the application requirement and use case, sometimes file system sink, sometimes forwarding to remote rsyslogd, or even FluentBit.

FluentBit is a Fast Log Processor and Forwarder part of Graduated Fluentd Ecosystem and a CNCF sub-project.

Demonstration

See how it works 🔥

• rsyslog / FluebtBit

rcl_logging_syslog_to_fluent-bit.mp4

• rsyslog / Fluentd / Loki / Grafana

output.mp4

Tutorials

• How to integrate with Fluentd / Loki / Grafana

Supported ROS Distribution

Distribution	Supported	Branch
Rolling Ridley	✅	rolling (Development)
Jazzy Jalisco	✅	jazzy
Iron Irwini	✅	iron
Humble Hawksbill	✅	humble

Installation

Prerequisites

• rsyslog installation

rcl_logging_syslog requires rsyslog package, which Ubuntu should have already in default. But if you are using container, the situation is bit different from host system since there is no system services or rsyslogd is running by default. In the case of container, we need to install rsyslog packages in the container root file system.

Note

We can enable the container with host system privileges but that is NOT recommended, especially for security.

### Install rsyslog package
apt install rsyslog

• rsyslog service check

If you are running the application on host system, check the rsyslog system service.

systemctl status rsyslog

If you are running the application on container environment, start the rsyslogd daemon process.

/usr/sbin/rsyslogd -n -iNONE

• fluent-bit installation

if you want to use fluent-bit to forward the logging data from rsyslogd, please follow Fluent Bit Installation Manual to install it.

### Check fluent-bit package is installed
dpkg -l fluent-bit
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  fluent-bit     3.1.6        amd64        Fast data collector for Linux

Build

Currently we need to build the source code if we want to use alternative logging backend with ROS 2. See more details for ros2/rcl#1178.

Please follow ROS 2 Official Development / Installation to build the rcl_logging_syslog package below.

### Clone the repository under the workspace
cd <YOUR_WORKSPACE>/src
git clone https://github.com/fujitatomoya/rcl_logging_syslog.git

### Build rcl_logging_syslog
export RCL_LOGGING_IMPLEMENTATION=rcl_logging_syslog
colcon build --symlink-install --cmake-clean-cache --packages-select rcl_logging_syslog rcl

Test

Following to the previous build procedure, we need to copy ros2-test.conf to /etc/rsyslog.d/, and restart the rsyslog.

sudo cp ./config/ros2-test.conf /etc/rsyslog.d/

Then we can start the colcon test:

This test creates and uses /var/log/rcl_logging_syslog directory in temporary for the test, and it checks the file existence and contents using rcl_logging_syslog.

colcon test --event-handlers console_direct+ --packages-select rcl_logging_syslog

Configuration

SYSLOG(3) is really simple that does not have much interfaces to control on application side, it just writes the log data on rsyslog Unix Domain Socket. So we need to configure rsyslog how it manages the log message with /etc/rsyslog.conf, for example file system sink and forward the message to fluent-bit.

environmental variable	default	Note
RCL_LOGGING_SYSLOG_FACILITY	LOG_LOCAL1	syslog facility: either of LOG_CRON, LOG_DAEMON, LOG_SYSLOG, LOG_USER, and LOG_LOCAL0-7. (/etc/rsyslog.conf must be configured accordingly.)

Examples:

export RCL_LOGGING_SYSLOG_FACILITY="LOG_LOCAL2"

See more details for https://www.rsyslog.com/doc/index.html.

rsyslog

Copy ros2-logging.conf to /etc/rsyslog.d/, and replace <FLUENTBIT_IP> with your own IP address where fluent-bit listens on.

sudo cp ./config/ros2-logging.conf /etc/rsyslog.d/

Check the configuration file to see if there is no error,

rsyslogd -N1
rsyslogd: version 8.2312.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

then, restart the rsyslog system service, or simply restart rsyslog daemon when using container.

### via systemd (not container environment)
sudo systemctl restart rsyslog
systemctl status rsyslog

### container environment
/usr/sbin/rsyslogd -n -iNONE

fluent-bit

With above configuration, rsyslogd also sends log data to TCP port 5140 that fluent-bit listens with the following configuration. Replace <FLUENTBIT_IP> with your own IP address in fluent-bit.conf.

### Create working directory
mkdir fluentbit_ws; cd fluentbit_ws

### Copy the original files to the local directory
cp -rf /etc/fluent-bit/fluent-bit.conf fluent-bit.conf
cp -rf /etc/fluent-bit/parsers.conf parsers.conf

### Modify the configuration as below
cat fluent-bit.conf
[SERVICE]
    Flush        1
    Parsers_File parsers.conf

[INPUT]
    Name     syslog
    Parser   syslog-rfc3164
    Listen   <FLUENTBIT_IP>
    Port     5140
    Mode     tcp

[OUTPUT]
    Name     stdout
    Match    *

# Start fluent-bit
/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.conf

Usage

Logging messages from ROS 2 application will be sent to rsyslogd and forwarded to fluent-bit if that is configured.

Examples

The followings are example output for file system and fluent-bit.

Start ROS 2 demo application:

ros2 run demo_nodes_cpp talker
[INFO] [1724453270.445092186] [talker]: Publishing: 'Hello World: 1'
[INFO] [1724453271.444981999] [talker]: Publishing: 'Hello World: 2'
[INFO] [1724453272.445056310] [talker]: Publishing: 'Hello World: 3'
...

Logging files under file system:

With example configuration and template above, file name is <PROGRAMNAME>_<PID>_<YEAR>-<MONTH>-<DATE>.log. And log data is prefixed with hostname, facility name, log level and program name as well.

root@tomoyafujita:/var/log# find ros
ros
ros/tomoyafujita
ros/tomoyafujita/talker_31348_2024-08-28.log

root@tomoyafujita:/var/log# cat ros/tomoyafujita/talker_31348_2024-08-28.log
Aug 28 11:30:18 tomoyafujita local1.info: talker[31348]: [INFO] [1724869818.620827927] [talker]: Publishing: 'Hello World: 1'
Aug 28 11:30:19 tomoyafujita local1.info: talker[31348]: [INFO] [1724869819.620758249] [talker]: Publishing: 'Hello World: 2'
Aug 28 11:30:20 tomoyafujita local1.info: talker[31348]: [INFO] [1724869820.620777277] [talker]: Publishing: 'Hello World: 3'

FluentBit output:

/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.conf
Fluent Bit v3.1.6
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __
|  ___| |                | |   | ___ (_) |         |____ |/  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/08/23 16:02:00] [ info] [fluent bit] version=3.1.6, commit=, pid=1686882
[2024/08/23 16:02:00] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/23 16:02:00] [ info] [cmetrics] version=0.9.4
[2024/08/23 16:02:00] [ info] [ctraces ] version=0.5.5
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] initializing
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2024/08/23 16:02:00] [ info] [in_syslog] TCP server binding 43.135.146.89:5140
[2024/08/23 16:02:00] [ info] [sp] stream processor started
[2024/08/23 16:02:00] [ info] [output:stdout:stdout.0] worker #0 started
...
[0] syslog.0: [[1724844661.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:01", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869861.620769614] [talker]: Publishing: 'Hello World: 44'"}]
[0] syslog.0: [[1724844662.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:02", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869862.620773450] [talker]: Publishing: 'Hello World: 45'"}]
[0] syslog.0: [[1724844663.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:03", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869863.620773763] [talker]: Publishing: 'Hello World: 46'"}]
...

Reference

• https://www.rsyslog.com/doc/index.html
• https://docs.fluentbit.io/manual