Lambkin got stuck during a benchmark run

[glpuga]

Bug description

While running a large benchmark run testing beluga, lambkin got stuck during a case and never recovered.

How to reproduce

No idea.

Expected behavior

Continue to run until the final case.

Actual behavior

About two days into the run, it stopped moving forward. ROS nodes where up, but nothing relevant was logged, and output bagfile was empty.

Additional context

No resources were obviously missing in the computer, there was enough disk space, and the computer (beefy) was otherwise idle.

These are the logs of the final few cases/iterations leading to the stop. I removed the bagfiles due to their size, but all but the last one were of the expected size. The one of the iteration that got stuck was empty, like nothing had been recorded since the iteration started.

tor_wic_slam_error.tar.gz

[glpuga]

Qualitatively this became much more noticeable after instrumenting measurements with the timememory. While a previous run of the run without timememory got stuck once during the four (effective) days run, so far I've had to restart it five times and I'm only halfway through the same bagfiles set.

A limited set of logs I observed seem to have these in common:

• the logs never start, the amcl nodes never start processing data.
• one node in the set (in one case nav2_amcl, in another the rosbag recorder), generates a log like this "failed to send response to /rosbag2_recorder/list_parameters (timeout)".

[glpuga]

I'll try this: ros2/rmw_fastrtps#704

[hidmic]

I'll try this: ros2/rmw_fastrtps#704

Did it make a difference? Service discovery in FastRTPS (or FastDDS) isn't great, some ros2cli verb can hang forever, and LAMBKIN doesn't consistently assign timeouts to all the processes it manages.

[glpuga]

I'll try this: ros2/rmw_fastrtps#704

Did it make a difference? Service discovery in FastRTPS (or FastDDS) isn't great, some ros2cli verb can hang forever, and LAMBKIN doesn't consistently assign timeouts to all the processes it manages.

All the lambkin runs I did after I created this issue used the proposal in ros2/rmw_fastrtps#704 and I never saw this issue again, so I guess it did.

[glpuga]

I permamently added this to #93 with fc23f02