Ouster-ROS ROS1-ROS2 bag files conversion scripts #2

Samahu · 2024-08-15T18:37:00Z

Summary of Changes

Ouster-ROS ROS1-ROS2 bag files conversion utility scripts

TODO(s)

Merge into a single python script

Validation

python3 src/ouster-ros-extras/convert-ros1-to-ros2.py <ROS1-BAG-FILE-PATH> --output <ROS2-BAG-FILE-PATH>

OR

python3 src/ouster-ros-extras/convert-ros2-to-ros1.py <ROS2-BAG-FILE> --output <ROS1-BAG-FILE-PATH>

Rotoslider · 2024-09-25T01:49:59Z

Seem --output is not used:
`python3 /home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py tunnel21b.bag --output tunnel21br2.bag
/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py:39: DeprecationWarning: Global type registration has been replaced with explicit typestores.

Perform all type registration and subsequent serialization and
deserialization on typestore instances:

from rosbags.typesys import Stores, get_typestore

typestore = get_typestore(Stores.ROS2_FOXY)
typestore.register(types)
register_types(add_types)
usage: convert-ros1-to-ros2.py [-h] input output
convert-ros1-to-ros2.py: error: unrecognized arguments: --outputwithout it i get the followingpython3 /home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py tunnel21b.bag tunnel21br2.bag
/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py:39: DeprecationWarning: Global type registration has been replaced with explicit typestores.

Perform all type registration and subsequent serialization and
deserialization on typestore instances:

from rosbags.typesys import Stores, get_typestore

typestore = get_typestore(Stores.ROS2_FOXY)
typestore.register(types)
register_types(add_types)
/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py:56: DeprecationWarning: Writer should be called with an explicit version number (8 or 9).
with Writer(file_out) as writer:
/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py:62: DeprecationWarning: Writer.add_connection should be called with typestore or msgdef/rihs01 pair.
conn_dict[in_topic] = writer.add_connection(topic_name, topic_type)
Traceback (most recent call last):
File "/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py", line 82, in
convert(file_in=args.input, file_out=args.output)
File "/home/asus/ros2_ws/src/ouster-ros-extras/convert-ros1-to-ros2.py", line 62, in convert
conn_dict[in_topic] = writer.add_connection(topic_name, topic_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/asus/miniconda3/lib/python3.12/site-packages/rosbags/rosbag2/writer.py", line 233, in add_connection
msgdef, _ = typestore.generate_msgdef(msgtype, ros_version=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/asus/miniconda3/lib/python3.12/site-packages/rosbags/typesys/store.py", line 448, in generate_msgdef
msgdef, md5sum = self.gendefhash(typename, subdefs, ros_version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/asus/miniconda3/lib/python3.12/site-packages/rosbags/typesys/store.py", line 371, in gendefhash
raise TypesysError(msg)
rosbags.typesys.base.TypesysError: Type 'ouster_sensor_msgs/msg/PacketMsg' is unknown.
`
it creates a folder, a yaml and a database file but nothing else.

Rotoslider · 2024-09-25T01:50:55Z

Yaml:
`rosbag2_bagfile_information:
compression_format: ''
compression_mode: ''
custom_data: {}
duration:
nanoseconds: null
files:

duration:
nanoseconds: null
message_count: 0
path: tunnel21br2.bag.db3
starting_time:
nanoseconds_since_epoch: null
message_count: 0
relative_file_paths:
tunnel21br2.bag.db3
ros_distro: rosbags
starting_time:
nanoseconds_since_epoch: null
storage_identifier: sqlite3
topics_with_message_count: []
version: 8`

Rotoslider · 2024-09-26T01:37:00Z

I update the code for the rosbag1 to 2 convertor. It now saves a db3 file and a yaml file. Whic appear correct.
here is the updated python code:
`from pathlib import Path
from rosbags.rosbag1 import Reader
from rosbags.rosbag2 import Writer
from rosbags.typesys import get_typestore, Stores, get_types_from_idl, get_types_from_msg

Define the message structure for Ouster

idl_text = """
module ouster_sensor_msgs {
module msg {
struct PacketMsg {
sequence buf;
};
};
};
"""

msg_text = """
uint8[] buf
"""

Create a Typestore for ROS2 Humble and register custom message types

typestore = get_typestore(Stores.ROS2_HUMBLE)

Register message types for Ouster

typestore.register(get_types_from_idl(idl_text))
typestore.register(get_types_from_msg(msg_text, 'ouster_sensor_msgs/msg/PacketMsg'))

Register message types for ROS1

ros1_msg_text = msg_text
typestore.register(get_types_from_msg(ros1_msg_text, "ouster_ros/msg/PacketMsg"))

def convert(file_in, file_out):
# Use the correct topics based on your ROS1 bag file
in_topics = ["/ouster/imu_packets", "/ouster/lidar_packets"]
out_topics = {
"/ouster/imu_packets": "ouster_sensor_msgs/msg/PacketMsg",
"/ouster/lidar_packets": "ouster_sensor_msgs/msg/PacketMsg"
}

# Mapping between the input (ROS1) and output (ROS2) topics
out_2_in_dict = {
    "/ouster/imu_packets": "/ouster/imu_packets",
    "/ouster/lidar_packets": "/ouster/lidar_packets"
}
in_2_out_dict = {value: key for key, value in out_2_in_dict.items()}

# Create a reader instance and open the ROS1 bag file
with Reader(file_in) as reader:
    # List available topics in the ROS1 bag
    available_topics = [conn.topic for conn in reader.connections]
    print(f"Available topics in ROS1 bag: {available_topics}")
    
    # Check if the specified topics are available in the ROS1 bag
    connections = [x for x in reader.connections if x.topic in in_topics]
    if not connections:
        print("No matching topics found in the bag. Please check your topic names.")
        return

    # Open the ROS2 bag file for writing
    with Writer(file_out, version=8) as writer:
        conn_dict = {}

        # Add connections to the writer for the ROS2 output topics
        for topic_name, topic_type in out_topics.items():
            in_topic = out_2_in_dict[topic_name]
            conn_dict[in_topic] = writer.add_connection(topic_name, topic_type, typestore=typestore)

        # Iterate over the messages in the ROS1 bag file
        for connection, timestamp, rawdata in reader.messages(connections=connections):
            conn = conn_dict.get(connection.topic)
            if conn is None:
                print(f"Unidentified connection topic {connection.topic}")
                continue

            try:
                # Deserialize the message from ROS1 format using the typestore
                msg = typestore.deserialize_ros1(rawdata, connection.msgtype)
                
                # Map the input topic to the correct ROS2 output topic
                out_topic = in_2_out_dict[connection.topic]
                
                # Serialize the message to ROS2 format using the typestore
                smsg = typestore.serialize_cdr(msg, out_topics[out_topic])
                
                # Write the message to the ROS2 bag file
                writer.write(conn, timestamp, smsg)
            except Exception as e:
                print(f"Error processing message on topic {connection.topic}: {e}")

if name == "main":
import argparse
parser = argparse.ArgumentParser(
description="Convert ROS1 bag format to ROS2 bag format"
)
parser.add_argument("input", help="ROS1 bag file")
parser.add_argument("output", help="ROS2 bag file output name")
args = parser.parse_args()
convert(file_in=args.input, file_out=args.output)
`

I now get the following when trying to run the replay.launch.xml

ros2 launch ouster_ros replay.launch.xml bag_file:=/home/asus/pointclouds/tunnel21br2.bag/tunnel21br2.bag.db3 metadata:=/home/asus/pointclouds/tunnel21br2.bag/metadata.yaml [INFO] [launch]: All log files can be found below /home/asus/.ros/log/2024-09-25-18-31-23-007879-aserver-19865 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [os_replay-1]: process started with pid [19878] [INFO] [component_container_mt-2]: process started with pid [19880] [INFO] [bash-3]: process started with pid [19882] [INFO] [bash-4]: process started with pid [19884] [INFO] [bash-5]: process started with pid [19887] [component_container_mt-2] [INFO] [1727314283.647582776] [ouster.os_container]: Load Library: /home/asus/ros2_ws/install/ouster_ros/lib/libos_cloud_component.so [component_container_mt-2] [INFO] [1727314283.670112057] [ouster.os_container]: Found class: rclcpp_components::NodeFactoryTemplate<ouster_ros::OusterCloud> [component_container_mt-2] [INFO] [1727314283.670148924] [ouster.os_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<ouster_ros::OusterCloud> [component_container_mt-2] [INFO] [1727314283.682840397] [ouster.os_cloud]: OusterCloud: node initialized! [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/ouster/os_cloud' in container '/ouster/os_container' [component_container_mt-2] [INFO] [1727314283.685891281] [ouster.os_container]: Load Library: /home/asus/ros2_ws/install/ouster_ros/lib/libos_image_component.so [component_container_mt-2] [INFO] [1727314283.697060668] [ouster.os_container]: Found class: rclcpp_components::NodeFactoryTemplate<ouster_ros::OusterImage> [component_container_mt-2] [INFO] [1727314283.697105180] [ouster.os_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<ouster_ros::OusterImage> [component_container_mt-2] [INFO] [1727314283.708287496] [ouster.os_image]: OusterImage: node initialized! [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/ouster/os_image' in container '/ouster/os_container' [os_replay-1] [INFO] [1727314285.099313803] [ouster.os_replay]: on_configure() is called. [os_replay-1] [ERROR] [1727314285.102369397] [ouster.os_replay]: Error when running in replay mode: Errors parsing metadata string: * Line 1, Column 1 [os_replay-1] Syntax error: value, object or array expected. [os_replay-1] [component_container_mt-2] [INFO] [1727314285.102804909] [ouster.os_image]: OusterImage: retrieved new sensor metadata! [component_container_mt-2] [INFO] [1727314285.102852342] [ouster.os_cloud]: OusterCloud: retrieved new sensor metadata! [component_container_mt-2] [2024-09-25 18:31:25.103] [ouster::sensor] [info] parsing legacy metadata format [component_container_mt-2] [2024-09-25 18:31:25.104] [ouster::sensor] [info] parsing legacy metadata format [component_container_mt-2] terminate called after throwing an instance of 'std::runtime_error' [component_container_mt-2] what(): Metadata must contain: lidar_mode [component_container_mt-2] terminate called recursively [os_replay-1] [INFO] [1727314285.105249940] [ouster.os_replay]: get_metadata service created [os_replay-1] [INFO] [1727314285.105320444] [ouster.os_replay]: Running in replay mode [bash-3] Transitioning successful [INFO] [bash-3]: process has finished cleanly [pid 19882] [bash-4] Transitioning successful [INFO] [bash-4]: process has finished cleanly [pid 19884] [bash-5] 2024-09-25 18:31:28.698 [RTPS_TRANSPORT_SHM Error] Failed init_port fastrtps_port9163: open_and_lock_file failed -> Function open_port_internal [bash-5] [INFO] [1727314289.150943001] [rviz2]: Stereo is NOT SUPPORTED [bash-5] [INFO] [1727314289.151052112] [rviz2]: OpenGl version: 4.5 (GLSL 4.5) [bash-5] [INFO] [1727314289.269606134] [rviz2]: Stereo is NOT SUPPORTED [bash-5] [INFO] [1727314289.846480149] [rviz2]: Stereo is NOT SUPPORTED [ERROR] [component_container_mt-2]: process has died [pid 19880, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=os_container -r __ns:=/ouster -p use_sim_time:=True'].

Rotoslider · 2024-09-26T02:01:58Z

The rosbag seems valid but does not play:
Files: tunnel21br2.bag
Bag size: 1.3 GiB
Storage id: sqlite3
Duration: 265.726518784s
Start: Sep 16 2024 10:16:42.558373288 (1726507002.558373288)
End: Sep 16 2024 10:21:08.284892072 (1726507268.284892072)
Messages: 196644
Topic information: Topic: /ouster/imu_packets | Type: ouster_sensor_msgs/msg/PacketMsg | Count: 26570 | Serialization Format: cdr
Topic: /ouster/lidar_packets | Type: ouster_sensor_msgs/msg/PacketMsg | Count: 170074 | Serialization Format: cdr

and here is the yaml:

`rosbag2_bagfile_information:
compression_format: ''
compression_mode: ''
custom_data: {}
duration:
nanoseconds: 265726518784
files:

duration:
nanoseconds: 265726518784
message_count: 196644
path: tunnel21br2.bag.db3
starting_time:
nanoseconds_since_epoch: 1726507002558373288
message_count: 196644
relative_file_paths:
tunnel21br2.bag.db3
ros_distro: rosbags
starting_time:
nanoseconds_since_epoch: 1726507002558373288
storage_identifier: sqlite3
topics_with_message_count:
message_count: 26570
topic_metadata:
name: /ouster/imu_packets
offered_qos_profiles: ''
serialization_format: cdr
type: ouster_sensor_msgs/msg/PacketMsg
type_description_hash: RIHS01_c21822af66817068b0850f02afb1a8bba77ec79cd412a2a3aaecddc108022032
message_count: 170074
topic_metadata:
name: /ouster/lidar_packets
offered_qos_profiles: ''
serialization_format: cdr
type: ouster_sensor_msgs/msg/PacketMsg
type_description_hash: RIHS01_c21822af66817068b0850f02afb1a8bba77ec79cd412a2a3aaecddc108022032
version: 8`

Rotoslider · 2024-09-26T02:15:59Z

My bad i should have used the json file not the metadata.yaml for the metadata.
I have no errors or warnings in terminal or rviz but I do not have any points in rviz displayed.

ros2 topic list
/clicked_point
/clock
/goal_pose
/initialpose
/ouster/imu
/ouster/imu_packets
/ouster/lidar_packets
/ouster/metadata
/ouster/nearir_image
/ouster/os_replay/transition_event
/ouster/points
/ouster/range_image
/ouster/reflec_image
/ouster/scan
/ouster/signal_image
/parameter_events
/rosout
/tf
/tf_static

Publish conversion ROS1-ROS2 bag files conversion scripts

b51013e

Samahu self-assigned this Aug 15, 2024

Samahu mentioned this pull request Aug 15, 2024

No Topic information of Ros2 ouster-lidar/ouster-ros#48

Closed

Samahu mentioned this pull request Sep 23, 2024

How to replay lidar data recorded under ros1 with ros2 ouster-lidar/ouster-ros#270

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ouster-ROS ROS1-ROS2 bag files conversion scripts #2

Ouster-ROS ROS1-ROS2 bag files conversion scripts #2

Samahu commented Aug 15, 2024

Rotoslider commented Sep 25, 2024

Rotoslider commented Sep 25, 2024

Rotoslider commented Sep 26, 2024

Rotoslider commented Sep 26, 2024

Rotoslider commented Sep 26, 2024

Ouster-ROS ROS1-ROS2 bag files conversion scripts #2

Are you sure you want to change the base?

Ouster-ROS ROS1-ROS2 bag files conversion scripts #2

Conversation

Samahu commented Aug 15, 2024

Summary of Changes

TODO(s)

Validation

Rotoslider commented Sep 25, 2024

Rotoslider commented Sep 25, 2024

Rotoslider commented Sep 26, 2024

Define the message structure for Ouster

Create a Typestore for ROS2 Humble and register custom message types

Register message types for Ouster

Register message types for ROS1

Rotoslider commented Sep 26, 2024

Rotoslider commented Sep 26, 2024