Is it wise to reduce the cadence? Should we switch to monthly cadence?

[edwardhartnett]

Reducing the spack-stack cadence is a mistake - it's moving in the wrong direction.

Instead, any team who cannot keep up with a quarterly update needs to automate testing. (They can't test even every three months? Why not? What are they doing that takes three months to test?)

Industry has moved to dev-ops, which is a cadence increased to infinity. What has been achieved elsewhere may reasonably be aspired to at NOAA.

A quarterly cadence is already too slow, because it results in so many special updates. A better answer would be to automate testing and double the cadence.

Imagine a new release every month. Now we would not need many chained environments or special installs, just upgrade to the newest monthly release...

[climbfuji]

It is extremely UNWISE to reduce the cadence. We are finding ourselves in exactly the same situation as previous stack maintainers: we are maintaining an unwieldy mess of post-release install modifications/addon-environments that are unmanageable. I am absolutely annoyed and disappointed that we have allowed ourselves to go down this rabbit hole. We should be installing a new version of spack-stack every three months (for now, see below) and make ideally NO post-release install modifications/amendments in the official installations. If a user needs a test environment with a different/newer version of, let's say, esmf and mapl, then we guide them to doing that (as chained or standalone env) for TESTING PURPOSES ONLY. There shouldn't really be any other envs in the official installations than one per each compiler (maybe there's an addon such as gsi-addon is for the moment for each compiler, but installed at the time the release was made).

Look at this mess on Hercules - I have NO idea what all these different envs are, especiallly the ones created in April:

[role-epic@Hercules:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0]# ls -lart envs/
total 216
drwxr-sr-x+  6 role-epic epic 16384 Apr  4 23:12 ufswm-env/
drwxr-sr-x+  6 role-epic epic 16384 Apr 16 18:21 gsi-addon-env/
-rw-r--r--   1 role-epic epic 40960 Apr 16 18:22 ue-esmf861.tar
drwxr-sr-x+  6 role-epic epic 16384 Apr 16 20:34 ue-esmf861/
drwxr-sr-x+  6 role-epic epic 16384 Apr 17 23:33 unified-env-861/
drwxr-sr-x+  7 role-epic epic 16384 Apr 24 12:17 upp-addon-env/
drwxr-sr-x+  6 role-epic epic 16384 Jul  3 19:44 unified-env-intel-2023.2.4/
drwxr-sr-x+ 11 role-epic epic 16384 Jul  9 11:55 ./
drwxr-sr-x+  6 role-epic epic 16384 Jul  9 12:50 unified-env/
drwxr-sr-x+  6 role-epic epic 16384 Jul  9 12:52 ue-gcc12-openmpi416/
drwxr-sr-x+ 10 role-epic epic 16384 Jul  9 22:09 ../
drwxr-sr-x+  6 role-epic epic 16384 Jul  9 22:10 ue-esmf-8.6.1-mapl-2.46.2/

The latest ue-esmf-8.6.1-mapl-2.46.2 is a chained environment on top of unified-env. The new mapl in there doesn't build with mvapich2, which is needed for gcc in that particular release. We have moved away (with spack-stack-1.7.0 and later) to using gcc with openmpi on that system, but because we are piggybacking everything off of an old environment/configuration with KNOWN problems, that's what we've got. I need to patch the mapl package in that 1.6.0 checkout for just THAT reason.

And what about all these addon envs?

Do we need to add esmf 8.6.1 and mapl 2.46.2 for each of them as well? And also to ue-gcc12-openmpi416 - which is a unified environment with gcc@12 and openmpi instead of mvapich2 for testing (which it was used for as intended by JCSDA - during the three months between 1.6.0 and 1.7.0, we tested if openmpi performed better than mvapich2/solved known problems, found it to be better and then made the switch in 1.7.0). I am certainly NOT going to do this.

I would love to be on an even shorter cadence of monthly installs, with monthly updates from spack develop as well. But I would argue that our current installation steps are still too manual to do that. Let alone some of our customers. We should go back to a release every quarter and no modifications to existing releases. I'd go as far and say that if we REALLY need to make a modification to a spack-stack release BEFORE THE NEXT RELEASE is out the door, then it gets its own patch version increment (e.g. 1.6.1 instead of 1.6.0), we don't chain off of the official quarterly release but build from scratch (or a binary cache). Once the NEXT quarterly release is out the door, NO MODIFICATIONS are made to previous releases UNLESS it is an extremely important bug/security issue.

[edwardhartnett]

Can we reverse the decision to reduce cadence, and stick to quarterly?

We are providing a stable release on NOAA R&D machines. So why does a quarterly release have any effect on NOAA work? Why cannot NOAA simply ignore the quarterly releases and use the stable release we are providing?

Are we trying to slow all other teams to the speed of our slowest team? That doesn't seem like a good idea. Instead the slow teams should speed up. It's hard to see how testing, even 1980's style manual testing, slow and inefficient as it is, can take three months. If each spack-stack release causes many problems, then slowing the cadence will simply cause all those problems to be discovered in a group, later than they otherwise would have been.

Slowing the cadence does not stop even one bug. It's simply procrastination. How does it help?

We say this in the Project Charter:

Releases
Any stakeholder can perform a release at any time.

This is important, because any stakeholder might have an immediate need for a release.

It also specifies:

Regular, full releases of spack-stack are made every three months around the end of February, May, August, and November.

So, in this sense, we cannot prevent the project from doing a quarterly release. Nor should we.

[climbfuji]

I am all in favor. I suggest that we pick up the quarterly cadence again. The next release 1.8.0 is scheduled for end of August 2024, the one after that would then be 1.9.0 at the end of November.

[christopherwharrop-noaa]

Releases should never be altered after-the-fact. Developers rely on behavior of a given release, and if that behavior changes, it can result in unexpected and unanticipated problems that are hard to debug. It also potentially destroys the provenance of products generated from releases that have changed. Patch releases can easily solve that problem.

I agree quarterly releases should be the minimum cadence, and for my purposes, it's already too slow. I build spack-stack containers for my project, and sometimes I need to augment them by adding packages or making small modifications. Having outdated git submodule for Spack is a huge problem for me, and I often have to patch package.py files because of this, even with quarterly releases. Keeping up-to-date with authoritative Spack is really important.

[christopherwharrop-noaa]

My observation from the model development side is with UFS SRW App. The general approach seems to be that changes to the stack or submodules can't happen until they happen for ufs-weather-model first. Changes to things like ESMF seem to have a major ripple across the whole UFS ecosystem and, for some reason, it takes forever for updates to propagate. The SRW has, so far, been 1-3 releases behind. There is a PR there to upgrade to spack-stack 1.7.0 (from 1.5.1) that has been open (in draft form) since Apr 10. It should not take 3 months (an entire release cycle) to update to a new version of spack-stack.

[shlyaeva]

I completely agree with the sentiment on special updates. Having multiple post-release updates is not the best use of time for people involved in this development at JCSDA, and I would like to see this resolved. This amount of updates is unsustainable with our resources, and an obvious solution for us will be to not have people at JCSDA work on the requests that are not relevant for the JEDI work. It would be better to see this resolved in a way that would be more satisfactory to everyone involved. Both watching from the side and reading through this thread it looks like this can be resolved by generally pushing back on in-between release requests and possibly more frequent release cadence (which would allow for that push back to be easier).

[edwardhartnett]

Since NOAA has their own stable release, can we consider monthly releases of spack-stack?

That would eliminate a lot of special updating. I think, like any good agile project, it's cheap to do a release.

[christopherwharrop-noaa]

Dom are you talking about insufficient testing and deployment of the UFS/JEDI Apps? Or spack-stack? Having more frequent releases, without any back porting to previous ones, might also provide a carrot for motivating groups to keep up. To get the carrot they have to make their code ready for the release that has the carrot in it. Part of the problem seems to be that updating the stack in the UFS Apps is a huge undertaking when, really, it should be changing a few lines and pushing a button. The most work it should entail is updating modules (versions of packages) to load for the various platforms when needed. But instead we have 3+ months of work to update from one quarterly release to the next.

[climbfuji]

It's all of the above. I am not against it, and I say we should mov in that direction, but I am saying that I cannot do this with the current system we have. If EMC wants to throw more people at this effort to make it happen right now with what we have - be my guest!

Otherwise, it's like committing to sending astronauts to mars if your rocket only makes it to the moon (or doesn't leave orbit if it's made by Boeing). First you have to build the rocket that CAN get you there, then you put people in it and send it off.

[climbfuji]

FWIW, this is exactly what the spack-stack automation task is for that we are starting next week. Once that's in place, we should be in good shape. On the NRL side, I am planning on nightly builds and deployments of spack-stack on at least one if not all systems we use, and subsequent testing of NEPTUNE / NEPTUNE+JEDI on at least one system. I will work on this after my vacation.

[christopherwharrop-noaa]

Has anyone checked whether E4S has anything to offer us that may make life easier? Tom Robinson at GFDL has set up a good collaboration with them and they seem motivated to help NOAA. If nothing else, I think they have pretty huge set of build caches.

[climbfuji]

Yes, I talked with them several times. Our use cases don't match perfectly, therefore they have been a bit hesitant working on this (and the prolonged procedure getting accounts on the various system didn't help). Essentially what we asked them is to create and maintain build caches for us using our site configs and environment definitions. That's about 3/4 of the way we need to go. The thing is, it's not super hard for us to do, just takes some time. I actually had this running on the previous version of the JCSDA AWS ParallelCluster (similar to what I described above as my plans for NRL). I figure that it will take only a couple of days to get this set up initially on any given system, and then maybe a period of a few days to babysit it. Of course, you need to put alerting in place and monitor it, which is where similar, previous efforts from EPIC using Jenkins fell short of (they tried on Orion and then let it slip).