Add support for parallel universes #905
Conversation
| deploy_bin_repo = deploy_jll_repo = "JuliaBinaryWrappers/$(src_name)_jll.jl" | ||
| end | ||
|
|
||
| # Find the universe directory by ascending from pwd() until we find a "universes" directory |
There was a problem hiding this comment.
Hmmm, I'm not certain about this. Perhaps we should store these in a Scratch space?
There was a problem hiding this comment.
I thought about it, but I do kinda want it explicitly available in Yggdrasil in case you need to go in and edit something. Putting it in Scratch makes it really hard to find.
There was a problem hiding this comment.
It's also the same thing that git does to find its .git folders.
There was a problem hiding this comment.
I'd like to have a way for these universes to get GC'ed. I think Fredrik and I are going to add optional user-defined time limits to scratch spaces so that you can e.g. @get_scratch("Universe-$(universe)"; lifetime=Days(3)) and if you don't use it again in three days, it gets auto-cleaned up.
That's the main reason I want to use scratch spaces. Perhaps we can add a symlink in build, so that if you do julia build_tarballs.jl --universe=SS_rebuild --deploy=universe, it then creates not only build/SuiteSparse*.tar.gz, but also build/universes/SS_rebuild which points to the universe where that was built?
There was a problem hiding this comment.
Why should universes get explicitly gc'ed? Seems like something that should be very much up to the user.
There was a problem hiding this comment.
I think these should be auto-GC'ed because the usecase for universes is for experimentation, not for users to use this as their main source of JLLs. I want to use these within Yggdrasil to allow us to open a PR that builds multiple, interdependent packages, and use the builds from the bottom of the stack as dependencies for the top of the stack, without having to register in-between. If we're creating a universe per-PR, I want to be sure that they will be removed after a while.
Keno
force-pushed
the
kf/universes
branch
3 times, most recently
from
f2f0ef2
to
f30914b
Compare
This adds the concept of a "universe" for autobuilds. A universe is essentially a depot that has its own copy of the General registry. When building packages using `--deploy=universe` the built jll gets registered into this copy of the registry and subsequent builds can pick it up from there as if it had been registered with the real general registry. This allows successive builds of dependent packages without first having to go register the dependency. The universe to use is specified using `--universe=<name>`. If a universe by that name does not exist, it is created. Otherwise the previously created universe is re-used.
| if deploy_bin || deploy_jll | ||
| code_dir = joinpath(Pkg.devdir(), "$(src_name)_jll") | ||
| if universe | ||
| code_dir = joinpath(universe_path, "clones", string(jll_uuid("$(src_name)_jll"))) |
There was a problem hiding this comment.
Makes the commit available to the regular logic to just check it out from clones
There was a problem hiding this comment.
Okay. Could alternatively put in the devdir path as the url in the registry, right? I guess this avoids one layer of indirection since Pkg would cache it here anyway.
There was a problem hiding this comment.
Yes, this seemed easier.
There was a problem hiding this comment.
I'm confused; the clones directory in my normal depot is a bunch of bare repositories, and the names are all numeric, not UUIDs.
There was a problem hiding this comment.
Yea, cold be stored wherever. I don't think Pkg will pick this up.
Co-authored-by: Fredrik Ekre <[email protected]>
Co-authored-by: Fredrik Ekre <[email protected]>
This adds the concept of a "universe" for autobuilds. A universe
is essentially a depot that has its own copy of the General registry.
When building packages using
--deploy=universethe built jll getsregistered into this copy of the registry and subsequent builds can
pick it up from there as if it had been registered with the real
general registry. This allows successive builds of dependent packages
without first having to go register the dependency. The universe
to use is specified using
--universe=<name>. If a universe bythat name does not exist, it is created. Otherwise the previously
created universe is re-used.
cc @KristofferC @fredrikekre