You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This has been a bit of a thorn in our sides for a while. Essentially, spacks git fetch operation is silently failing during spack install for certain package repositories and not updating the local refs, leading to spack.VersionLookupErrors.
For a specific example, see https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/10104028532/job/27942359486#step:8:429 in which it states:
VersionLookupError: ea4d313d6731fe6621cab26495685a77963f7cdc is not a valid git ref for mom5
When it clearly is on the remote.
Background
Originally, this issue reared it's ugly head when I had implemented a 'Rolling Tag' logic for Prereleases. This would add and force push (NOTE: bad practice!) the tag in spack.specs[0] (for example, the 2024.08.0 in [email protected]) for each commit in a pull request. This was because the root SBD in the spack.yaml (such as access-om2) would expect the tag in the pull request to exist during the spack install. This led to the following scenario, whose end state is eerily similar to the one we have now:
Initially, push 2024.08.0 to HEAD of pull request
Deploy on Gadi, spack will git fetch --tags on the deployment repo and successfully pull the remote 2024.08.0.
Make a modification in the PR, CI will force push 2024.08.0 to the new HEAD (NOTE: still bad practice!)
Deploy on Gadi, spack will try to git fetch --tags on the deployment repo and silently fail, leading eventually to a spack.VersionLookupError.
I later removed this moving tag functionality as it was the cause of this iteration of silent failures.
It was debugged by modifying
to redirect output and error to str rather than os.devnull (why are they piping output and error to devnull?), which allowed me to see the git output which was failing to update the local tags. Unfortunately this irrevocably broke Prerelease and I had to remake the Prerelease environment, so don't simply redirect the output to str as it polluted a bunch of metadata in spack.
This fixed the issue so I thought I was done with it...
The Current Issue
Linking ACCESS-NRI/ACCESS-OM2#76 - A lot of the later failing runs are a similar issue where we have a spack.VersionLookupError where the ref can't be found locally when it is definitely on the remote. I suspect it's a similar issue to the above where there is an issue running git fetch, but I don't know what. Doing the workaround below solved it, and verified that the branches were not being updated by spack:
From https://github.com/ACCESS-NRI/mom5
= [up to date] 3-build-ci -> 3-build-ci
= [up to date] 9-mkmf-escape-fix -> 9-mkmf-escape-fix
= [up to date] access-esm1.5 -> access-esm1.5
* [new branch] delete -> delete
baaf7ed..d8ece40 development -> development
* [new branch] dougiesquire/issue388-accessom-gtracer -> dougiesquire/issue388-accessom-gtracer
* [new branch] dougiesquire/master-generic-tracers -> dougiesquire/master-generic-tracers
= [up to date] master -> master
= [up to date] upstream-master -> upstream-master
Things TODO
Next time we get a spack.VersionLookupError, copy the entire instance to a local location and see if we can find out what is preventing it from git fetching by either interrogating the git_repos folder, or redirecting spacks own git operations to a file.
Fix the issue!
Maybe notify spack developers that this is happening...
Workaround
The current workaround is to do the following:
cd$SPACK_ROOT/../git_repos
git -C ./<each hash> ls-tree HEAD --names-only # compare against the package repos contents because these repos are bare# Once you have found the hash that corresponds to the correct silently failing repo...
git -C $SPACK_ROOT/../git_repos/shcnhqk fetch -v -u origin +refs/heads/*:refs/heads/*# which will pull down the correct refs
The text was updated successfully, but these errors were encountered:
This has been a bit of a thorn in our sides for a while. Essentially,
spack
sgit fetch
operation is silently failing duringspack install
for certain package repositories and not updating the local refs, leading tospack.VersionLookupError
s.For a specific example, see https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/10104028532/job/27942359486#step:8:429 in which it states:
When it clearly is on the remote.
Background
Originally, this issue reared it's ugly head when I had implemented a 'Rolling Tag' logic for Prereleases. This would add and force push (NOTE: bad practice!) the tag in
spack.specs[0]
(for example, the2024.08.0
in[email protected]
) for each commit in a pull request. This was because the root SBD in thespack.yaml
(such asaccess-om2
) would expect the tag in the pull request to exist during thespack install
. This led to the following scenario, whose end state is eerily similar to the one we have now:2024.08.0
toHEAD
of pull requestspack
willgit fetch --tags
on the deployment repo and successfully pull the remote2024.08.0
.2024.08.0
to the newHEAD
(NOTE: still bad practice!)spack
will try togit fetch --tags
on the deployment repo and silently fail, leading eventually to aspack.VersionLookupError
.I later removed this moving tag functionality as it was the cause of this iteration of silent failures.
It was debugged by modifying
spack/lib/spack/spack/version/git_ref_lookup.py
Lines 145 to 149 in 21da7d7
output
anderror
tostr
rather thanos.devnull
(why are they piping output and error to devnull?), which allowed me to see the git output which was failing to update the local tags. Unfortunately this irrevocably broke Prerelease and I had to remake the Prerelease environment, so don't simply redirect the output tostr
as it polluted a bunch of metadata inspack
.This fixed the issue so I thought I was done with it...
The Current Issue
Linking ACCESS-NRI/ACCESS-OM2#76 - A lot of the later failing runs are a similar issue where we have a
spack.VersionLookupError
where the ref can't be found locally when it is definitely on the remote. I suspect it's a similar issue to the above where there is an issue runninggit fetch
, but I don't know what. Doing the workaround below solved it, and verified that the branches were not being updated by spack:Things TODO
spack.VersionLookupError
, copy the entire instance to a local location and see if we can find out what is preventing it fromgit fetch
ing by either interrogating thegit_repos
folder, or redirectingspack
s own git operations to a file.spack
developers that this is happening...Workaround
The current workaround is to do the following:
The text was updated successfully, but these errors were encountered: