Add workflow to run tests with pixi #214

xela-95 · 2024-10-01T08:46:34Z

Closes #213

traversaro · 2024-10-01T09:08:06Z

.github/workflows/test-pixi.yml

+  schedule:
+  # * is a special character in YAML so you have to quote this string
+  # Execute a "nightly" build at 2 AM UTC
+  - cron:  '0 2 * * *'


If you want to run a schedule job, could it make sense to delete the lock file for those? Otherwise we would be testing just GitHub Actions-images related regressions, that even if possible are not really frequent in my experience. See https://github.com/robotology/whole-body-estimators/blob/80d45bf79c5e4be3c7f83e0bec8f2cdfef48caab/.github/workflows/pixi-ci.yml#L30-L35 for example.

Ok understood. Meanwhile, I'm thinking about just running these actions on pull requests, that is the original scope for this kind of action.

traversaro · 2024-10-01T09:10:09Z

I see that we have a failure:

CMake Error at .pixi/envs/default/lib/cmake/Qt5Gui/Qt5GuiConfigExtras.cmake:9 (message):
  Failed to find "GL/gl.h" in
  "/home/runner/work/gz-sim-yarp-plugins/gz-sim-yarp-plugins/.pixi/envs/default/include".
Call Stack (most recent call first):
  .pixi/envs/default/lib/cmake/Qt5Gui/Qt5GuiConfig.cmake:233 (include)
  .pixi/envs/default/lib/cmake/Qt5Quick/Qt5QuickConfig.cmake:100 (find_package)
  .pixi/envs/default/lib/cmake/Qt5/Qt5Config.cmake:28 (find_package)
  .pixi/envs/default/lib/cmake/gz-gui8/gz-gui8-config.cmake:101 (find_package)
  .pixi/envs/default/lib/cmake/gz-sim8/gz-sim8-config.cmake:98 (find_package)
  .pixi/envs/default/share/cmake/gz-cmake3/cmake3/GzFindPackage.cmake:243 (find_package)
  CMakeLists.txt:14 (gz_find_package)

We can fix this by adding a dependency on libgl-devel on Linux, conda-forge/staged-recipes#27107 .

traversaro · 2024-10-01T09:11:22Z

The Windows failure is tracked in #205, probably we can just skip the test on Windows at the action level?

See #214 (comment)

- Exclude Windows tests temporarily - Run this workflow only manually and on PRs

Make `libgl-devel` a dependency only for Linux targets

xela-95 · 2024-10-01T10:08:39Z

@traversaro, what do you think is the most appropriate thing to do between running the action against the pixi.lock already committed in the repository or removing it and creating a fresh one with the latest available dependencies from the action?

traversaro · 2024-10-01T10:33:45Z

@traversaro, what do you think is the most appropriate thing to do between running the action against the pixi.lock already committed in the repository or removing it and creating a fresh one with the latest available dependencies from the action?

It depends. If the action is trigged by a pull request, we should keep the pixi.lock as we want understand test failures related to the PR, not related to the package update. I just suggesting removing the pixi.lock in the context of scheduled jobs.

xela-95 · 2024-10-01T15:42:15Z

I see that we have a failure:

CMake Error at .pixi/envs/default/lib/cmake/Qt5Gui/Qt5GuiConfigExtras.cmake:9 (message):
  Failed to find "GL/gl.h" in
  "/home/runner/work/gz-sim-yarp-plugins/gz-sim-yarp-plugins/.pixi/envs/default/include".
Call Stack (most recent call first):
  .pixi/envs/default/lib/cmake/Qt5Gui/Qt5GuiConfig.cmake:233 (include)
  .pixi/envs/default/lib/cmake/Qt5Quick/Qt5QuickConfig.cmake:100 (find_package)
  .pixi/envs/default/lib/cmake/Qt5/Qt5Config.cmake:28 (find_package)
  .pixi/envs/default/lib/cmake/gz-gui8/gz-gui8-config.cmake:101 (find_package)
  .pixi/envs/default/lib/cmake/gz-sim8/gz-sim8-config.cmake:98 (find_package)
  .pixi/envs/default/share/cmake/gz-cmake3/cmake3/GzFindPackage.cmake:243 (find_package)
  CMakeLists.txt:14 (gz_find_package)

We can fix this by adding a dependency on libgl-devel on Linux, conda-forge/staged-recipes#27107 .

Now the pixi test on ubuntu is failing with a different error always related to OpenGL:

Error while loading the library [/home/runner/work/gz-sim-yarp-plugins/gz-sim-yarp-plugins/.pixi/envs/default/lib/gz-rendering-8/engine-plugins/libgz-rendering-ogre2.so]: libOpenGL.so.0: cannot open shared object file: No such file or directory
Error while loading the library [/home/runner/work/gz-sim-yarp-plugins/gz-sim-yarp-plugins/.pixi/envs/default/lib/gz-rendering-8/engine-plugins/libgz-rendering-ogre2.so]: libOpenGL.so.0: cannot open shared object file: No such file or directory

traversaro · 2024-10-01T15:51:54Z

We need also libopengl-devel : https://github.com/conda-forge/libglvnd-feedstock/blob/main/recipe/meta.yaml#L66 . This will not be needed once we migrate gz-* conda packages to use these packages instead of CDTs packages.

traversaro · 2024-10-01T15:53:01Z

Probably also libegl-devel will be needed.

xela-95 · 2024-10-01T16:06:53Z

Now we have this 😅

[Wrn] [Ogre2RenderEngine.cc:548] Unable to open display: . Trying to run in headless mode.
[Err] [Ogre2RenderEngine.cc:749] Unable to load Ogre Plugin[/home/runner/work/gz-sim-yarp-plugins/gz-sim-yarp-plugins/.pixi/envs/default/lib/OGRE-Next/RenderSystem_GL3Plus]. Rendering will not be possible.Make sure you have installed OGRE properly.
[Err] [Ogre2RenderEngine.cc:800] unable to find OpenGL 3+ Rendering Subsystem. OGRE is probably installed incorrectly. Double check the OGRE cmake output, and make sure OpenGL is enabled.

traversaro · 2024-10-01T19:48:41Z

I think that is ok. CI needs to have at least an EGL driver (even software only) installed, and that is supposed to be installed via apt, as it is a driver of the system.

traversaro · 2024-10-01T21:14:29Z

Ok, installing all the packages that we installed in the CI of conda worked fine, but let's try to be a bit more methodical. The conda-forge package libegl contains the library that is linked by the conda program, and that looks in the system for EGL drivers (that are instead installed by the system). How does libegl finds the system driver? Looking for some json files called icd, see https://github.com/NVIDIA/libglvnd/blob/master/src/EGL/icd_enumeration.md#icd-discovery .

In particular, to run the mesa software implementation of OpenGL, it needs to find the /usr/share/glvnd/egl_vendor.d/50_mesa.json file. This file belongs (as we can see from the apt-file output) to the libegl-mesa0 package:

traversaro@IITBMP014LW012:~$ apt-file search /usr/share/glvnd/egl_vendor.d/50_mesa.json
libegl-mesa0: /usr/share/glvnd/egl_vendor.d/50_mesa.json

The .icd file refers to the /usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0 file, that is also installed by libegl-mesa0:

traversaro@IITBMP014LW012:~$ cat /usr/share/glvnd/egl_vendor.d/50_mesa.json
{
    "file_format_version" : "1.0.0",
    "ICD" : {
        "library_path" : "libEGL_mesa.so.0"
    }
}
traversaro@IITBMP014LW012:~$ apt-file search /usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0
libegl-mesa0: /usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0

So, for sure we can try to first install libegl-mesa0 . By installing only that package, the test fail with error:

libEGL warning: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

libEGL warning: MESA-LOADER: failed to open zink: /usr/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

libEGL warning: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

libEGL warning: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

libEGL warning: MESA-LOADER: failed to open zink: /usr/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

libEGL warning: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

so libEGL_mesa.so.0 is loaded correctly, but then it fails as it can't find dri/swrast_dri.so and similar libraries. And which packages contains tese files?

traversaro@IITBMP014LW012:~$ apt-file search /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
libgl1-mesa-dri: /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so`

So indeed by installing libegl-mesa0 and libgl1-mesa-dri the test pass.

traversaro · 2024-10-02T07:58:45Z

After:

[v9] Remove CDTs and use regular conda packages for OpenGL support conda-forge/gz-rendering-feedstock#31
[v8] Remove CDTs and use regular conda packages for OpenGL support conda-forge/gz-rendering-feedstock#34

if we update the packages to use this latest builds, it should be not necessary anymore to manually add the libegl-devel packages.

xela-95 · 2024-10-02T08:19:36Z

Ok, installing all the packages that we installed in the CI of conda worked fine, but let's try to be a bit more methodical.

Thanks a lot, @traversaro for you time and effort. This, apart from solving this particular issue, taught me how to solve the next ones!

traversaro · 2024-10-02T08:23:06Z

Yes, unfortunately the related section in the conda-forge docs (https://conda-forge.org/docs/maintainer/knowledge_base/#when-should-cdts-be-used) is outdated, we need to eventually fix that.

xela-95 · 2024-10-02T08:24:56Z

if we update the packages to use this latest builds, it should be not necessary anymore to manually add the libegl-devel packages.

Ok so when both these PR will be merged we could eventually update our pixi.toml and the conda feedstock?

traversaro · 2024-10-02T08:43:48Z

if we update the packages to use this latest builds, it should be not necessary anymore to manually add the libegl-devel packages.

Ok so when both these PR will be merged we could eventually update our pixi.toml and the conda feedstock?

Yes, we need to make sure that those build are the one actually installed and not older versions for some reason. Anyhow, for some time we can just keep the lib*gl packages there just to be sure.

xela-95 · 2024-10-02T10:07:55Z

Squashed some minor commits. Merging 🚀

Add workflow to run tests with pixi

04ea117

xela-95 added the CI Issues regarding CI workflows label Oct 1, 2024

xela-95 self-assigned this Oct 1, 2024

traversaro reviewed Oct 1, 2024

View reviewed changes

xela-95 added 3 commits October 1, 2024 11:50

Add libgl-devel dependency to pixi.toml

f7fcdf7

See #214 (comment)

Update pixi workflow

5b614e0

- Exclude Windows tests temporarily - Run this workflow only manually and on PRs

Update pixi.toml

ab2d102

Make `libgl-devel` a dependency only for Linux targets

Update pixi workflow to disable caching of environment

c66297d

traversaro force-pushed the ci/pixi-test-action branch from 70c1e78 to 1933ae0 Compare October 1, 2024 21:17

traversaro mentioned this pull request Oct 1, 2024

Add support for both Gazebo Harmonic (gz-sim8) and Gazebo Ionic (gz-sim9) #208

Merged

traversaro approved these changes Oct 2, 2024

View reviewed changes

traversaro mentioned this pull request Oct 2, 2024

Manually apply "Rebuild for libabseil 20240722, libgrp 1.65 & libprotobuf 5.27.5" migration and add yum dependencies to run EGL-based tests conda-forge/gz-sensors-feedstock#51

Merged

5 tasks

xela-95 and others added 2 commits October 2, 2024 12:06

Update pixi.lock with new libgl-devel dependency

b404fd8

Install mesa OpenGL driver on Ubuntu

65e9ca1

xela-95 force-pushed the ci/pixi-test-action branch from 1933ae0 to 65e9ca1 Compare October 2, 2024 10:07

xela-95 merged commit 8ce978c into main Oct 2, 2024
7 checks passed

xela-95 deleted the ci/pixi-test-action branch October 2, 2024 10:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add workflow to run tests with pixi #214

Add workflow to run tests with pixi #214

xela-95 commented Oct 1, 2024

traversaro Oct 1, 2024

xela-95 Oct 1, 2024

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024 •

edited

Loading

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024 •

edited

Loading

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024

Add workflow to run tests with pixi #214

Add workflow to run tests with pixi #214

Conversation

xela-95 commented Oct 1, 2024

traversaro Oct 1, 2024

Choose a reason for hiding this comment

xela-95 Oct 1, 2024

Choose a reason for hiding this comment

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024

xela-95 commented Oct 1, 2024

traversaro commented Oct 1, 2024

traversaro commented Oct 1, 2024 • edited Loading

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024 • edited Loading

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024

traversaro commented Oct 2, 2024

xela-95 commented Oct 2, 2024

traversaro commented Oct 1, 2024 •

edited

Loading

xela-95 commented Oct 2, 2024 •

edited

Loading