miniconda, compiler toolchains, libstdc++ versions

[climbfuji]

This issue is not something that needs fixing urgently, but I want to capture the insight I got over the last days.

There is a complex interplay (or should I say a mess) between the miniconda environment and the compiler toolchains when it comes to libstdc++.

Miniconda is required on some systems (for unknown reasons not on all) to avoid the issues with poetry (see spack/spack#29308). Some other Python installations have problems with static vs. shared Python libraries (link to an issue?). The easy solution is to install miniconda-3.9.7 with only the basic packages (poetry and libpython-static as per instructions here). One side effect of this is that miniconda comes with its own libstdc++, which - for miniconda-3.9.7 - is as of gcc-9.3.0.

Compiler toolchains are needed for Intel compilers that rely on GNU's libstdc++ for the implementation of the various C++ standards. For systems where the default GNU gcc compiler is recent enough, nothing needs to be done. For systems that have an old version of GNU's gcc as default (e.g. 4.8.5 on most of the HPCs that we are working on), one needs to set up a toolchain to point the Intel compiler to a different version. In spack-stack, this is achieved by configuring PATH, LD_LIBRARY_PATH and CPATH accordingly in the spack compiler config.

A problem occurs when one package uses C++ and Python, because it will link against the miniconda libstdc++ version (equivalent to gcc-9.3.0). If this package also links against another package that uses C++ but doesn't use Python, then that packages points to the toolchain libstdc++. If the two libstdc++ are different (enough), the result is errors due to missing symbols etc. This is the case for fckit using eckit and miniconda when the toolchain is set up for gcc-11. When using gcc-9 or gcc-10 for the toolchain, there is no problem, because the libstdc++ are close enough.

I suspect that a similar problem would occur if no toolchain was used in case the miniconda libstdc++ and the OS default libstdc++ are incompatible.

The workaround for now is to make sure that the toolchain uses gcc-9 or gcc-10 to remain compatible with miniconda-3.9.7. While this is a working solution for now, the best option imo is to install gcc and g++ inside miniconda and use those for the toolchain as well. If gnu-9.3.0 isn't sufficient, newer versions can be installed from the conda-forge channel (up to 11.0.3 as of 03/23/2022).