microstrain_inertial_driver]: Unable to set device to idle

[devvaibhav455]

Describe the bug
I boot the Jetson and sometimes the IMU driver is not working. After a reboot, it works.

To Reproduce
Steps to reproduce the behavior:

1. Power ON the jetson. Get below error sometimes (not always)

Fail Log

Jan 01 00:00:53 echo-jetson-orin-nx clearpath-sensors-start[3429]: [INFO] [microstrain_inertial_driver_node-6]: process started with pid [3542]
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [INFO] [0000000054.226971680] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Running microstrain_inertial_driver version: unknown
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [INFO] [0000000054.227219488] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Using MIP SDK version: v0.0.0
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [WARN] [0000000054.227242240] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: No relative position configured. We will not publish relative odometry or transforms.
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [WARN] [0000000054.227253120] [a200_0524.sensors.imu_0.microstrain_inertial_driver]:   Please configure relative position to publish relative position data
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [INFO] [0000000054.227768096] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Attempting to open serial port </dev/microstrain_main> at <115200>
Jan 01 00:00:54 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [INFO] [0000000054.231347136] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting device to idle in order to configure
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [ERROR] [0000000060.236806784] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Unable to set device to idle
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [ERROR] [0000000060.236954336] [a200_0524.sensors.imu_0.microstrain_inertial_driver]:   Error(-4): Timed out
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [ERROR] [0000000060.237021312] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Failed to read configuration for node
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [ERROR] [0000000060.237048608] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Failed to configure node base
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [microstrain_inertial_driver_node-6] [FATAL] [0000000060.237064960] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Failed to configure node
Jan 01 00:01:00 echo-jetson-orin-nx clearpath-sensors-start[3429]: [ERROR] [microstrain_inertial_driver_node-6]: process has died [pid 3542, exit code 1, cmd '/home/nvidia/ros2_ws/install/microstrain_inertial_driver/lib/microstrain_inertial_driver/microstrain_inertial_driver_node --ros-args -r __node:=microstrain_inertial_driver -r __ns:=/a200_0524/sensors/imu_0 --params-file /tmp/launch_params_ztka9fm4 --params-file /etc/clearpath/sensors/config/imu_0.yaml --params-file /tmp/launch_params_xd0ycmr3 -r imu/data:=data -r /moving_ang:=moving_ang -r /tf:=tf -r /tf_static:=/a200_0524/tf_static'].

Pass Log

Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [INFO] [microstrain_inertial_driver_node-6]: process started with pid [3598]
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.603135936] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Running microstrain_inertial_driver version: unknown
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.603405376] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Using MIP SDK version: v0.0.0
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [WARN] [0000004535.603429536] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: No relative position configured. We will not publish relative odometry or transforms.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [WARN] [0000004535.603460992] [a200_0524.sensors.imu_0.microstrain_inertial_driver]:   Please configure relative position to publish relative position data
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.606022080] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Attempting to open serial port </dev/microstrain_main> at <115200>
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.609187232] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting device to idle in order to configure
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.625217632] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Main Connection Info:
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6]     #######################
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6]     Model Name:       3DMGX5-AHRS
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6]     Serial Number:    <Hidden on purpose>
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6]     Firmware Version: 1.1.76
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6]     #######################
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637202720] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic ekf/dual_antenna_heading
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637301536] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic ekf/llh_position
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637371456] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_1/llh_position
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637405024] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_1/odometry_earth
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637417152] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_1/time
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637426144] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_1/velocity
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637434944] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_1/velocity_ecef
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637442848] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_2/llh_position
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637452160] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_2/odometry_earth
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637460448] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_2/time
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637472576] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_2/velocity
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637481248] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss_2/velocity_ecef
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637508896] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic imu/wheel_speed
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637521696] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/ekf/aiding_measurement_summary
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637532032] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/ekf/gnss_dual_antenna_status
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637541600] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/ekf/gnss_position_aiding_status
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637551040] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/ekf/multi_antenna_offset_correction
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637563904] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_1/fix_info
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637577408] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_1/rf_error_detection
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637682432] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_1/sbas_info
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637697312] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_2/fix_info
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637706560] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_2/rf_error_detection
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637714912] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/gnss_2/sbas_info
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.637724384] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support publishing the topic mip/sensor/overrange_status
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [WARN] [0000004535.646427392] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Data rates of -1.0 are no longer supported. Disabling topic ekf/imu/data
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [WARN] [0000004535.666950912] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Data rates of -1.0 are no longer supported. Disabling topic imu/mag
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.705239840] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the GPIO config command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.705346560] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the PPS source command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.705364032] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the odometer settings command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.705385664] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the SBAS settings command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.705396928] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the nmea message format command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.707076032] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: Device does not support the RTK dongle config command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.707151936] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: Device does not support Siangle Config command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.708609376] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting Declination Source to 2 0.230000
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.714751776] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: Device does not support GNSS antenna offsets
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.714981344] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the vehicle dynamics mode command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.715054592] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting heading source to 1
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.720721024] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: Not setting initial heading because heading source is not 0
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.720844192] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting autoinitialization to 1
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726746560] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the filter adaptive settings command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726859392] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the filter aiding command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726877856] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the filter speed lever arm command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726889216] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the wheeled vehicle constraint command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726901024] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the vertical gyro constraint command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726910784] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the GNSS antenna calibration command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.726919840] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the next-gen filter initialization command.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.727018240] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting sensor to vehicle rotation euler to [-0.000000, -0.000000, -0.000000]
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.732721664] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The device does not support the reference point lever arm command
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.732826304] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Note: The settings were not saved as startup settings. Power cycling will remove changes from device
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.732848000] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Resetting the filter after the configuration is complete.
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.784411840] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Setting spin rate to <200.000000> hz
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.784710912] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Resuming the device data streams
Jan 01 01:15:35 echo-jetson-orin-nx clearpath-sensors-start[3472]: [microstrain_inertial_driver_node-6] [INFO] [0000004535.787088352] [a200_0524.sensors.imu_0.microstrain_inertial_driver]: Node activated

Expected behavior
The driver should work and publish the messages on imu's relevant topics

Environment (please complete the following information):

• OS: Ubuntu 22.04/ Jetson JP 6.0
• Architecture: arm64
• ROS Version: humble
• Version: [e.g. 2.0.0]
• Sensor(s): 3DMGX5-AHRS

Modifications
No

Launch Parameters
If you have an updated params.yml file, please attach it to this issue.

Additional context
I have the ROS2 branch of the driver. It runs through clearpath's systemd sensor bringup service. I didn't make any changes to the config files myself between these two runs.