Realtime control loop uses too much time when loading controllers (ros ticket #3715)

[ahendrix]

During startup, the realtime control loop uses too much time, typically on the EC part. We should disregard overruns that happen on startup, since we're still contacting the boards. This can either happen in pr2_etherCAT, or in the controllers-to-diagnostics script.

These errors are really annoying. During qualification, we see this sometimes, and it causes the "robot checkout" test to abort (it aborts for all warnings/errors).

This warning should only happen for 10 seconds, not 30. We can scroll back with the robot monitor if we want to.

trac data:

• Owner: wim
• Reporter: watts
• Reported at: Sun Jan 24 17:41:02 2010
• URL: https://code.ros.org/trac/wg-ros-pkg/ticket/3715

[ahendrix]

[wim] This problem actually occurs every time you load controllers into the controller manager, not only at startup. This seems to lock up the realtime loop for a while. And since ethercat is using most of the time in the realtime loop, ethercat is also most likely to be running when the realtime loop locks up. This is why it looks like the ehtercat communication is taking too much time, while the (dynamic) loading is actually the problem. This is a problem we need to address, but wasn't in the scope of M3. There might already be a ticket associated with this (rob?). To make the robot checkout test pass, the right way is to make the test expect this problem with the realtime loop. As for the timeout of the warning, there is no right answer of what the exact timeout duration or timeout mechanism should be. I would prefer to see a permanently latched error that you need to manually confirm every time an overrun happens, but it became a warning for 30 seconds.

[ahendrix]

[wim] Update: The overruns are caused by the construction of a pluginlib object. This means that every controller that uses a plugin (e.g. a filter) will cause overruns. The laser tilt controllers is one example. Most controllers only take about 0.1 ms to construct, but the laser tilt controller takes 150 ms because it constructs a filter plugin loader. The laser tile controller causes exactly 6 overruns every time it is constructed.

[ahendrix]

[wheeler] Further update: this appears to be happening when the pluginlib object calls rospack to get its list of plugins (ros::package::getPlugins). I've reproduced the problem in a much simpler test case, and I'm debugging further.