You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I might say some naive and incorrect things. Feel free to correct me or close this :)
The memory issues in the mulled build seems to be caused by two factors:
the solver's memory usage
the package installation memory usage
The recent PRs to fix problem 1 (galaxy/14770 and #810) seem like a great step, but mamba would eventually run into the same issues as repositories grow. Furthermore, they do not address problem 2.
To solve problem 1, why not solve the environment outside of the mulled build container and then pass the solved environment to the container? If there are bigger containers available, problem 2 can be solved by estimating installation memory usage beforehand and assign an appropriate container size. Alternatively, this value could be used to skip doomed mulled builds. Assuming that the size of all dependency tarballs correlates to the memory usage during installation, we can obtain this fairly easily.
Here is a proof of principle, based on a package requiring only bioconda-utils as dependency:
# remove all downloaded packages to get the total download size
conda clean -afyq
# solve the environment
mamba create -n dryrun -d bioconda-utils > dryrun.txt
# obtain the solved_enironment.yamlecho"dependencies:"> solved_enironment.yaml
grep + dryrun.txt | \
awk -F ' +''{print " - " $6 "::" $3 "==" $4 "=" $5;}' \
>> solved_enironment.yaml
# obtain the download size
grep "Total download" dryrun.txt | \
awk -F ': ''{print $2;}'> size.txt
# clean up
rm dryrun.txt
I might say some naive and incorrect things. Feel free to correct me or close this :)
The memory issues in the mulled build seems to be caused by two factors:
The recent PRs to fix problem 1 (galaxy/14770 and #810) seem like a great step, but mamba would eventually run into the same issues as repositories grow. Furthermore, they do not address problem 2.
To solve problem 1, why not solve the environment outside of the mulled build container and then pass the solved environment to the container? If there are bigger containers available, problem 2 can be solved by estimating installation memory usage beforehand and assign an appropriate container size. Alternatively, this value could be used to skip doomed mulled builds. Assuming that the size of all dependency tarballs correlates to the memory usage during installation, we can obtain this fairly easily.
Here is a proof of principle, based on a package requiring only
bioconda-utilsas dependency:The results
size.txt:
solved_enironment.yaml:
The text was updated successfully, but these errors were encountered: