YarpLoggerDevice cannot log data at 1kHz #877
Comments
|
Hi @LoreMoretti, thank you for posting this :) You are right. According to the plots you show, the fact that when you collect at 1kHz, you get a maximum delta t of 0.0025s, is related to an outlier. I believe this outlier can be removed. Regardless, it seems that when we set a specific frequency, we consistently get a maximum error of 0.0006s (0.6ms). This error is negligible if we collect data at 100Hz, but it becomes significant when collecting data at 1kHz. A possibility is:
|
|
cc also @traversaro @S-Dafarra |
|
By the way, more in general if you want to reduce the jitter in the logging time, you need to reduce the scheduling jitter of thread. This can be done by the classical methods used for real time computing, that is:
An introduction to this topics I like a lot is the "Real-time programming with Linux" series of blog posts by Shuhao Wu: https://shuhaowu.com/blogseries.html#rt-linux-programming . The https://github.com/cactusdynamics/cactus-rt repo from the same author is good example of those techniques, even if to use them in our codebase I would prefer to have some kind of abstraction that is portable also to macOS on Windows (even using non-realtime features in those cases). |
|
After reading the resources shared by @traversaro and others (like https://www.youtube.com/watch?v=NrjXEaTSyrw), we could also investigate the benefits brought by changing the current logic of Now, if I got it right, the thread it is set to sleep for an amount of time given by the difference We could change it to something like sleep for The drawback could be that the CPU is kept busied with more not useful operations (i.e. the Without testing it, I am not sure if this could lead to improvements and also the entity of the improvements, and also how this code change would affect all other software we use. |
|
Another idea that came to my mind (reading for example https://unix.stackexchange.com/questions/326579/how-to-ensure-exclusive-cpu-availability-for-a-running-process) is to try to run the logger on a separate laptop reserving one CPU core just for the YarpLoggerDevice application. Also here, I guess that it is difficult to determine upfront the entity of the improvements (also compared to the alternative solution of just increasing the priority of the |
|
I have tried to implement what I explained in this comment through this commit. I have hard-coded some numeric values in the code, but this is just inteded for testing at this stage. Then, I have run the
With mean values being:
So the modification seems to improve a bit the I could run more tests to check if the statistics are confirmed. Still, perhaps these tiny improvements aren't worth changing that piece of YARP. |
|
The variance seems well reduced anyhow! FYI the link to the commit appears to be broken |
Thanks for the catch! I should have fixed it |
In any way, it is something that can be done inside the |
I have modified the Then to test it, I have run the
With mean values being:
In both cases, I have being using yarp3.9, the only difference is within the More tests should be run to confirm the statistics, but the preliminary results seem good. |
|
Can we have an option to disable busy waiting? There are many context in which you do not want to do busy waiting at all.
Ahh, my bad, this sentence was meant to write "increse the priority of the thread". If you did not tried that, I would try, it is a simple linux-specific code you can wrap in a #ifdef block, and I guess it can be really helpful even without a realtime kernel (that we may want to explore the use anyhow if we are interested in this, the early wakeup is the last of the strategies listed in https://shuhaowu.com/blogseries.html#rt-linux-programming, not the first :D ). |
|
Even without changing the priority, at least changing the scheduler used to use one of the realtime schedulers |
See https://stackoverflow.com/questions/51100861/stdthread-with-pthread-setschedparam-with-c-11 and the last suggestion of https://chatgpt.com/share/9e47ab2c-1302-433f-aa5a-5caa25b296ef . |
I have implemented it at this commit. Then to test it, I have run the
Here the thread priority used: A part from something happening at the beginning, it seems that increasing the thread priority and set a FIFO policy, helped to reduce the number of spikes. It could be interesting to combine it with the The Note that, the first time I run the YarpLoggerDevice, I was getting this error. Therefore, with the help of @S-Dafarra, we had to |
I have implemented the needed code at this commit. Regarding:
I tried different strategies to avoid to
But, I could not make it work. This is because after I did
But, checking the application and Though, if I trace the application with strace logI am not sure why. I have built the robotology-superbuild from source (with conda dependencies), maybe it works if one installs directly the conda binary packages of the superbuild.
|
|
We should understand if the created matfile has the right reading access |
I have run the
The Qualitatively, it seems that the orange curve exhibits less spikes (therefore improving the performances). Of course, I should run more tests to confirm the statics. Also, at this commit I can open a PR to add the changes of the YarpRobotLoggerDevice that allowed to reach these perfomances. Instead, regarding:
I was able to load the created matfile on So it seems that also others from root can read the file. |
|
I am preparing a PR: #883 |
We have run some data collection campaign using the YarpLoggerDevice to record and save data at different sample rates.
But, we have realized that YarpLoggerDevice was not able to log data at
1 kHz.Therefore, we have tested YarpLoggerDevice with 4 different sample rates, and we got:
max(dt) = 0.0025 [s](dtshould be0.001 [s]to respect real time)max(dt) = 0.0025 [s](dtshould be0.002 [s]to respect real time)max(dt) = 0.0056 [s](dtshould be0.005 [s]to respect real time)max(dt) = 0.0105 [s](dtshould be0.010 [s]to respect real time)with
dtbeing the time between one sample and the following one.It follows also the
dtplots for the 4 different datasets:Also, we checked that, within each dataset, the logged vectors (such as joint position, joint velocity, joint acceleration) have the same size (i.e. number of samples).
Lastly, the blf version we have been using is isorrentino@3d37188.
cc @GiulioRomualdi @isorrentino
The text was updated successfully, but these errors were encountered: