Image dockerfile_images/basic/superbuild-icubhead-robometry fails to complete robotology-superbuild recompilation

[MSECode]

Task description 📝

When children image dockerfile_images/basic/superbuild-icubhead-robometry is triggered by dockerfile_images/basic/superbuild-icubhead-withuser it fails to complete the step where the robotology-superbuild is re-built. I think that the error is due to the cmake config files in: {CMAKE_INSTALL_PREFIX}/share/cmake/YCM, since if I delete that folder I do not get any error and the rebuild ends fine.
Therefore, the build fails if it found YCM already installed (might be that errors generate while trying to override some YCM config files?). If YCM it is not found the build ends clean.

The stack trace is the following:

0.771 -- YCM found in /home/icub/install_dir/share/YCM.
0.878 CMake Error: File /home/icub/install_dir/share/YCM/modules/YCMEPHelper/gitsafeclone.txt.in does not exist.
0.878 CMake Error at /home/icub/install_dir/share/YCM/modules/YCMEPHelper.cmake:883 (configure_file):
0.878   configure_file Problem configuring file
0.878 Call Stack (most recent call first):
0.878   /home/icub/install_dir/share/YCM/modules/YCMEPHelper.cmake:1242 (_ycm_ep_write_gitclone_script)
0.878   cmake/BuildYCM.cmake:8 (ycm_ep_helper)
0.878   /home/icub/install_dir/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
0.878   cmake/BuildYARP.cmake:8 (find_or_build_package)
0.878   /home/icub/install_dir/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
0.878   cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
0.878   CMakeLists.txt:67 (include)
0.878 
0.878 
0.880 CMake Error: File /home/icub/install_dir/share/YCM/modules/YCMEPHelper/RepositoryInfo.txt.in does not exist.
0.880 CMake Error at /home/icub/install_dir/share/YCM/modules/YCMEPHelper.cmake:1272 (configure_file):
0.880   configure_file Problem configuring file
0.880 Call Stack (most recent call first):
0.880   cmake/BuildYCM.cmake:8 (ycm_ep_helper)
0.880   /home/icub/install_dir/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
0.880   cmake/BuildYARP.cmake:8 (find_or_build_package)
0.880   /home/icub/install_dir/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
0.880   cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
0.880   CMakeLists.txt:67 (include)

Moreover, I can see that in the first error line of the call stack we are looking for a file that actually does not exist. Should we update the file path the build is looking for??

icub@7db77e752374:/$ ll /home/icub/install_dir/share/YCM/modules/
total 188
drwxr-xr-x 2 icub icub  4096 Dec 12 17:16 ./
drwxr-xr-x 8 icub icub  4096 Dec 12 17:16 ../
-rw-r--r-- 1 icub icub  8714 Dec 12 17:15 AddInstallRPATHSupport.cmake
-rw-r--r-- 1 icub icub  3324 Dec 12 17:15 AddUninstallTarget.cmake
-rw-r--r-- 1 icub icub  2491 Dec 12 17:15 ExtractVersion.cmake
-rw-r--r-- 1 icub icub  9512 Dec 12 17:15 FindOrBuildPackage.cmake
-rw-r--r-- 1 icub icub  1944 Dec 12 17:15 GetAllCMakeProperties.cmake
-rw-r--r-- 1 icub icub 15470 Dec 12 17:15 GitInfo.cmake
-rw-r--r-- 1 icub icub 14180 Dec 12 17:15 IncludeUrl.cmake
-rw-r--r-- 1 icub icub 28252 Dec 12 17:15 InstallBasicPackageFiles.cmake
-rw-r--r-- 1 icub icub  3371 Dec 12 17:15 ReplaceImportedTargets.cmake
-rw-r--r-- 1 icub icub 11578 Dec 12 17:15 StandardFindModule.cmake
-rw-r--r-- 1 icub icub 67173 Dec 12 17:15 YCMEPHelper.cmake
icub@7db77e752374:/$ 

Definition of Done ✅

Issue solved and build completes without removing {CMAKE_INSTALL_PREFIX}/share/cmake/YCM folder.

cc: @valegagge

[MSECode]

Hi @Nicogene and @traversaro,
I'm trying to understand the source of the bug I'm describing in this issue and I was wondering if you have any idea regarding that.
Basically what is happening is that when I'm updating the build of the superbuild adding the option ROBOTOLOGY_ENABLE_DYNAMICS I get those configuration error described in the body of the issue.
Specifically I'm building on Ubuntu.
Anyway, I have the same behavior on both my local environment and when building on dockers, as in this issue.

• Note 1: If I start from a clean environment, meaning that I do not have anything installed related to the superbuild and I add that option from the start everything ends fine.
• Note2 : Another solution I found is to delete the folders: {CMAKE_INSTALL_PREFIX}/share/cmake/YCM and {CMAKE_INSTALL_PREFIX}/share/YCM. If I do that both configuration and build go fine. Thus, it seems to me that the problem generates if YCM files already exist. However, this is not the solution I'd like to use because it is just a workaround and those folders should not be deleted to fix the problem.
• Note3: I'm interested in not starting from a clean environment because in this case I can generate a docker image where I have robometry by just starting from a base image with the superbuild already installed. If this is not possible or not a good practice I'll do differently and I'll generate the image with the ROBOTOLOGY_ENABLE_DYNAMICS option enabled from the beginning.
• Note 4: Still, I'd like to understand why during the configuration phase cmake is looking for those files. Moreover, the problem is that those files should exists or that the system should not look for them? (this regardless the issue)
  Let me know if you can give some insights about this, thanks.

cc: @valegagge

[traversaro]

I did not look into this in detail, but it seems that the docker files here use the YCM_EP_INSTALL_DIR variable, that is quite an advanced superbuild variable, and can be used only under certain limitations, see http://robotology.github.io/ycm-cmake-modules/gh-pages/latest/manual/ycm-superbuild.7.html?highlight=ycm_ep_install_dir#directories :

Each project will be installed in ${YCM_EP_INSTALL_DIR} (by default ${PROJECT_BINARY_DIR}/install). You should not change the YCM_EP_INSTALL_DIR variable, unless you wish to build the superbuild only once, and discard the build directory. In this case you should change this variable to the final destination of the build since, for many projects, the build is not relocatable. Please also note that, if you change it to a folder that is not writable by current user, you will have to run the whole build as superuser.

In a nutshell, you can't set the YCM_EP_INSTALL_DIR variable and then rebuild the same superbuild by changing some settings.

[traversaro]

Note3: I'm interested in not starting from a clean environment because in this case I can generate a docker image where I have robometry by just starting from a base image with the superbuild already installed. If this is not possible or not a good practice I'll do differently and I'll generate the image with the ROBOTOLOGY_ENABLE_DYNAMICS option enabled from the beginning.

I think that is possible and a good practice, as long as you do not set the YCM_EP_INSTALL_DIR to a non-default value.

[traversaro]

Having said that, indeed the failure is strange. It seems to be related to some kind of failure in the robotology-superbuild once YCM is installed and not bootstrapped?

[traversaro]

Related issues:

• Document YCM_EP_INSTALL_DIR robotology/robotology-superbuild#609
• Configure setup scripts directly in YCM_EP_INSTALL_DIR instead of hardcoding ${CMAKE_BINARY_DIR}/install robotology/robotology-superbuild#337
• User install prefix robotology/robotology-superbuild#316

[traversaro]

Indeed, I reproduced the error in a configuration that should be supported:

conda create -n testrobsub cmake pkg-config cxx-compiler make ninja ycm-cmake-modules
conda activate testrobsub
git clone https://github.com/robotology/robotology-superbuild/
cd robotology-superbuild/
cmake -Bbuild -S.

it fails with:

(testrobsub2) traversaro@IITBMP014LW012:~/testrobsub2/robotology-superbuild$ cmake -Bbuild -S.
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/traversaro/miniforge3/envs/testrobsub2/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/traversaro/miniforge3/envs/testrobsub2/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- YCM found in /home/traversaro/miniforge3/envs/testrobsub2/share/YCM.
CMake Error: File /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper/gitsafeclone.txt.in does not exist.
CMake Error at /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:883 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:1242 (_ycm_ep_write_gitclone_script)
  cmake/BuildYCM.cmake:8 (ycm_ep_helper)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/BuildYARP.cmake:8 (find_or_build_package)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
  CMakeLists.txt:67 (include)


CMake Error: File /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper/RepositoryInfo.txt.in does not exist.
CMake Error at /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:1272 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  cmake/BuildYCM.cmake:8 (ycm_ep_helper)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/BuildYARP.cmake:8 (find_or_build_package)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
  CMakeLists.txt:67 (include)

[traversaro]

Indeed, I reproduced the error in a configuration that should be supported:

conda create -n testrobsub cmake pkg-config cxx-compiler make ninja ycm-cmake-modules
conda activate testrobsub
git clone https://github.com/robotology/robotology-superbuild/
cd robotology-superbuild/
cmake -Bbuild -S.

it fails with:

(testrobsub2) traversaro@IITBMP014LW012:~/testrobsub2/robotology-superbuild$ cmake -Bbuild -S.
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/traversaro/miniforge3/envs/testrobsub2/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/traversaro/miniforge3/envs/testrobsub2/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- YCM found in /home/traversaro/miniforge3/envs/testrobsub2/share/YCM.
CMake Error: File /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper/gitsafeclone.txt.in does not exist.
CMake Error at /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:883 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:1242 (_ycm_ep_write_gitclone_script)
  cmake/BuildYCM.cmake:8 (ycm_ep_helper)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/BuildYARP.cmake:8 (find_or_build_package)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
  CMakeLists.txt:67 (include)


CMake Error: File /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper/RepositoryInfo.txt.in does not exist.
CMake Error at /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/YCMEPHelper.cmake:1272 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  cmake/BuildYCM.cmake:8 (ycm_ep_helper)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/BuildYARP.cmake:8 (find_or_build_package)
  /home/traversaro/miniforge3/envs/testrobsub2/share/YCM/modules/FindOrBuildPackage.cmake:205 (include)
  cmake/RobotologySuperbuildLogic.cmake:68 (find_or_build_package)
  CMakeLists.txt:67 (include)

A fix is provided in robotology/ycm-cmake-modules#468, can you check it @MSECode ? If it works, we can merge, release a YCM 0.18.1, and bump YCM to 0.18.1 in releases/2024.11 branch, so that it is ready for an eventual robotology distro 2024.11.2 (I already had a few fixes that it could make sense to release).

[MSECode]

Thanks @traversaro for the support.
I just tested yesterday fast and it worked.
I'll double check by doing all steps I had described in the issue to see if it is working fine.

[MSECode]

So,
I've tested it on a local docker image.
As you can see from here, if building using the standard "Unstable" configuration, thus on the devel branches we have the problem (as already described) on the configuration of YCM. I've done this just to check I was starting from a "broken" condition.

Starting from that, I re-built YCM by switching to the fix branch and by doing so it was then possible to compile all the robotology-superbuild using ROBOTOLOGY_ENABLE_DYNAMICS ON and with a custom install path.

So, I can consider that the fix worked and thus we can close this issue solved by: robotology/ycm-cmake-modules#468

Thanks @traversaro