-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using containers for payu conda environment #14
Comments
If you're using conda pack already you can get it to output in squashfs format, which can easily be added to a base container https://apptainer.org/docs/user/latest/persistent_overlays.html#overlay-embedded-in-sif |
Thanks @ScottWales- that is great to know! Using singularity containers is all very new to me |
Ok, so I have had some time to learn about using singularity containers. It took a little while to get the squashfs output of I was creating the containers using Apptainer (through a lima VM on my mac), and was running into issues with executing files inside the container from squashfs file generated locally. While creating conda-pack squashfs on gadi, adding the persistent overlay to a container and then executing commands inside the container worked fine. I think the errors likely came down to differences in architecture. I'm yet to test whether running After scrolling through the HIVE forum for singularity issues, @ScottWales I saw you mentioned this repo as an example for packaging environments in containers: https://github.com/ScottWales/singularity-conda-template/blob/master/build.sh. That has been really useful, and I've used it as a reference for setting up modulefiles and wrapper scripts for commands that runs the container. It also uses I am running into an issue that's probably an issue within payu. In
Comparing with
It still has So maybe if the python path, pointed to the wrapper python script which loads the container and runs payu-run inside the container, it would work. Gadi is currently down to maintenance so I am not able to test if modifying the qsub command would work. In another un-tested issue, I've also noticed that just running |
The I do what you're suggesting - we have the |
Ok, will probably switch to doing the squashfs manually then.. Ok thanks, that is good to know about MPI and |
Currently, payu is being deployed using
conda-pack
to compress a micromamba environment, then moving that file down to gadi and unpacking it in thevk83
project. Then, using modulefiles to activate the conda environment as described by @aidanheerdegen here.The unpacked environment currently is taking up a bit of space, for example for the
1.0.29
tag:As the conda-pack creates a standalone environment, it contains a subset of conda so it can re-create the environment when unpacked.
As conda enviroments have a lot of small files which have a performance impact on Gadi's Lustre filesystem, an improvement would be to use containers as discussed in this issue #4. Also, note this potential issue #2 with using containers with payu.
The text was updated successfully, but these errors were encountered: