Conda installation

[oscarxblanco]

Dear all,
recently I tried to install pyat including MPI on a conda virtual environment, but, I get an error with respect to a missing compiler that seems to be deprecated.
'''
... x86_64-conda_cos6-linux-gnu-c++ command not found
'''
Does any of you have tried to use pyat inside conda with OpenMPI and mpi4py from the conda forge repositories ?

Best regards,
o

[swhite2401]

Hello Oscar,
I haven't tried myself but It is seems the build is looking for a compiler it cannot find...
Did you try conda update?
You are still using pip to install the AT package in your conda env right?

[TeresiaOlsson]

Hi Oscar,
I tried it and had the same problem. I think I managed to fix it but then ran into another problem instead that I also had to solve. I will check what I did and let you know.
Best regards,
Teresia

[oscarxblanco]

Thank you both,
I managed to move one step forward by installing the openmpi compilers in conda

conda install -c conda-forge openmpi openmpi-mpicc openmpi-mpicxx  openmpi-mpifort mpi4py

It takes long to resolve the environment, but, the compiler error disappears.
However, I now get the following error :
/home/oblanco/miniconda3test4/lib/libz.so.1: undefined reference to `memcpy@GLIBC_2.14'

[oscarxblanco]

Dear all,
after installing glib from conda-forge the pyat compilation finishes w/o errors :

conda install -c conda-forge glib

I am not sure right now what would be the performance, but, the installation errors are solved.

Thank you again,
o

P.S. : I use conda 23.9.0.

[oscarxblanco]

Dear all,
I will summarize here the steps I followed in conda

conda update conda
conda update --all

Check the version

conda --version

In my case it is conda 23.9.0. Then do

conda install -c conda-forge openmpi openmpi-mpicc openmpi-mpicxx  openmpi-mpifort mpi4py glib

This step will take a while because it looks for compatibility between multiple packages.
That should be enough to compile AT with openmpi from sources. I leave here the line I use from the AT documentation https://atcollab.github.io/at/p/howto/multiprocessing.html

cd <at>
rm -rf build
pip install --config-settings mpi=1 ".[mpi]"

[swhite2401]

Maybe the documentation can be updated with your recipe?

[oscarxblanco]

Let me first check on other architecture. It works on a cluster running with CentOS 7. And I'm checking on Debian 12 and Ubuntu 23 at the moment.

This is a workaround when it is not possible to get sudo privileges to solve the mpi4py installation and the required pyat compilers, but, the automatic enviroment solution in conda slows a lot the installation (to give you and idea its been already several hours on Ubuntu 23 and haven't converged, it might be better to install one or few packages at the time instead of the summary line I put ...). I would not yet advise to follow strictly this recipe if there are other simpler solutions.

Maybe @TeresiaOlsson has another working recipe ? If so, and she is willing to share it here, I could compare and see what works better.

[TeresiaOlsson]

Sorry for my slow response. My best recipe at the moment is to build pyAT inside a container. Then I don't need to use conda or environments and I can build the code locally where I have all permissions. The drawback is that apptainer has to be installed where you want to run the code and there might be performance reductions when running MPI on the host together with the MPI inside the container depending on how you optimize it. That is something I want to investigate further.

For our lab, the aim is to run all software using containers to that's why I haven't investigated how to get it to work together with conda in more detail, but I just instead moved on to using a container when I discovered issues.

[oscarxblanco]

I have checked again my recipe as it is written. I'm able to compile pyat with parallel resources on

• Cent OS 7, architecture x86_64
• Debian 12, architecture x86_64
  Unfortunately in Ubuntu 23.04, the recipe has been running for half-a-day and conda has not solved the environment. I'm unable to confirm if it is the network, the build, the channel, or some other problem.

I leave here below the package versions installed in Debian 12

# Name                    Version                   Build  Channel
conda 23.9.0
mpi                       1.0                     openmpi    conda-forge
openmpi                   4.0.5                h9b22176_4    conda-forge
openmpi-mpicc             4.0.5                h7f98852_4    conda-forge
openmpi-mpicxx            4.0.5                h4bd325d_4    conda-forge
openmpi-mpifort           4.0.5                h1990efc_4    conda-forge
mpi4py                    3.1.4           py311h3e5f7c9_0
glib                      2.68.4               h9c3ff4c_1    conda-forge
glib-tools                2.68.4               h9c3ff4c_1    conda-forge
accelerator-toolbox       0.4.1.dev34+g52298a90          pypi_0    pypi

@swhite2401 after these tests I think the recipe could be included somewhere in the documentation, for example in a trouble-shooting section, or as alternatives when parallelization packages on the system are not compatible with pyat requirements.

[oscarxblanco]

my recipe also works on Ubuntu 23.10, with conda 23.11.0 and python 3.9.18.