Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hdf5 bug avoidance for BGC 2024.06 #14

Closed
wants to merge 6 commits into from
Closed

Conversation

CodeGat
Copy link
Contributor

@CodeGat CodeGat commented Jul 3, 2024

In this PR:

  • Update spack-config to 2024.07.02 to avoid the buggy hdf5 1.14.3.
  • Update spack-packages so it is in line with newer versions.
  • Add comments!

References original PR #12
References #13

@CodeGat CodeGat self-assigned this Jul 3, 2024
Copy link

github-actions bot commented Jul 3, 2024

🚀 Deploying access-om2-bgc 2024.06.1 as prerelease pr14-2

Details and usage instructions

This access-om2-bgc model will be deployed as:

  • 2024.06.1 as a Release (when merged).
  • pr14-2 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-2

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-bgc-pr14-2 environment.

🛠️ Using: spack-packages `2024.07.03`, spack-config `2024.07.02`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

@dougiesquire
Copy link
Collaborator

Thanks @CodeGat!

Some very basic testing indicates that the prerelease runs successfully but there is maybe an impact on performance from these updates 😞

In the table below:

  • Exp 1 uses release-1deg_jra55_ryf_bgc modified to run for 1 year at a time;
  • Exp 2 uses the same configuration but the executable paths are changed to those built in prerelease pr14-2.

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 1543.25 s 1574.72 s
output001 1560.17 s 1578.38 s
output002 1538.42 s 1568.62 s
average 1547.28 s 1573.91

Below is the (selective) output of a diff between the two experiments:

$ git diff -- om2_1deg_jra55_ryf_bgc_pr14-orig om2_1deg_jra55_ryf_bgc_pr14-2
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/config.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/config.yaml
index a971d90..6d2bcae 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/config.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/config.yaml
@@ -31,7 +31,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
           - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
           - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -39,7 +39,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/fms_ACCESS-OM-BGC.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/bgc_param.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/co2_iaf.nc
@@ -64,7 +64,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/i2o.nc
           - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/o2i.nc
@@ -81,7 +81,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/mppnccombine.spack

 manifest:
   reproduce:
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/exe.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/exe.yaml
index fec7972..0952928 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/exe.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 6f048474c92f2b2cc4ce14d159d9c3ad
+    md5: f74eb0ac0ca781c62d45c3a5e3f2fde7
 work/ocean/fms_ACCESS-OM-BGC.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/fms_ACCESS-OM-BGC.x
   hashes:
-    binhash: 45352e33876da49ca042014a9f6686e5
-    md5: a909552e85690be692ad3ec94016181b
+    binhash: 6fce98ebeb5991cdb82d9ad399c6953d
+    md5: d5fd242ea24aca79e070b47a652b2b62
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/input.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/input.yaml
index 6f6eab3..81471ce 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/input.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/input.yaml
@@ -155,7 +155,7 @@ work/ocean/INPUT/csiro_bgc.res.nc:
 work/ocean/INPUT/csiro_bgc_sediment.res.nc:
   fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/csiro_bgc_sediment.res.nc
   hashes:
-    binhash: 5dcf7a15477f488cd7d2f85a1d39a176
+    binhash: fff3375ad2ae36b7d0521dd67912a9fd
     md5: 12488f7cf8f9c2f714c0ff2dbbb33a9b
 work/ocean/INPUT/dust.nc:
   fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/dust.nc

Copy link

github-actions bot commented Jul 7, 2024

🚀 Deploying access-om2-bgc 2024.06.1 as prerelease pr14-3

Details and usage instructions

This access-om2-bgc model will be deployed as:

  • 2024.06.1 as a Release (when merged).
  • pr14-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-3

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-bgc-pr14-3 environment.

🛠️ Using: spack-packages `2024.07.03`, spack-config `2024.07.02`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

@CodeGat CodeGat linked an issue Jul 7, 2024 that may be closed by this pull request
Copy link

github-actions bot commented Jul 8, 2024

🚀 Deploying access-om2-bgc 2024.06.1 as prerelease pr14-4

Details and usage instructions

This access-om2-bgc model will be deployed as:

  • 2024.06.1 as a Release (when merged).
  • pr14-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-4

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-bgc-pr14-4 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

Copy link

github-actions bot commented Jul 8, 2024

🚀 Deploying access-om2-bgc 2024.06.1 as prerelease pr14-5

Details and usage instructions

This access-om2-bgc model will be deployed as:

  • 2024.06.1 as a Release (when merged).
  • pr14-5 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-5

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-bgc-pr14-5 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

Copy link

github-actions bot commented Jul 8, 2024

🚀 Deploying access-om2-bgc 2024.06.1 as prerelease pr14-6

Details and usage instructions

This access-om2-bgc model will be deployed as:

  • 2024.06.1 as a Release (when merged).
  • pr14-6 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-6

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-bgc-pr14-6 environment.

🛠️ Using: spack-packages `2024.07.08`, spack-config `2024.07.05`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

@dougiesquire
Copy link
Collaborator

In the table below:

  • Exp 1 uses release-1deg_jra55_ryf_bgc modified to run for 1 year at a time;
  • Exp 2 uses the same configuration but the executable paths are changed to those built in prerelease pr14-6.

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 1532.09 s 1549.07 s
output001 1541.98 s 1554.08 s
output002 1540.03 s 1554.38 s
average 1538.03 s 1552.51 s

So the prerelease runs (~ 1% slower).

Below is the (selective) output of a diff between the two experiments:

$ git diff -- orig pr14-6
diff --git a/orig/config.yaml b/pr14-6/config.yaml
index a971d90..3230015 100644
--- a/orig/config.yaml
+++ b/pr14-6/config.yaml
@@ -31,7 +31,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
       input:
           - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
           - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -39,7 +39,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/fms_ACCESS-OM-BGC.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/bgc_param.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/co2_iaf.nc
@@ -64,7 +64,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/i2o.nc
           - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/o2i.nc
@@ -81,7 +81,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/mppnccombine.spack

 manifest:
   reproduce:
diff --git a/orig/manifests/exe.yaml b/pr14-6/manifests/exe.yaml
index fec7972..e9c5dbb 100644
--- a/orig/manifests/exe.yaml
+++ b/pr14-6/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: d14806ad4b0b890d4c2f78a7e34258ab
+    md5: 651ea586afa441f3d94376f28f26b54d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 000959a21e84ba4baa66c32e61c03d39
+    md5: 237c0c1838fd072e43f7b632315cc2bb
 work/ocean/fms_ACCESS-OM-BGC.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/fms_ACCESS-OM-BGC.x
   hashes:
-    binhash: 45352e33876da49ca042014a9f6686e5
-    md5: a909552e85690be692ad3ec94016181b
+    binhash: ba7f3b0ad1f0381786dda70c8fcb609f
+    md5: 9f989f3e7a06b11f91f2967b7c1e262a
diff --git a/orig/manifests/input.yaml b/pr14-6/manifests/input.yaml
index 6f6eab3..81471ce 100644
--- a/orig/manifests/input.yaml
+++ b/pr14-6/manifests/input.yaml
@@ -155,7 +155,7 @@ work/ocean/INPUT/csiro_bgc.res.nc:
 work/ocean/INPUT/csiro_bgc_sediment.res.nc:
   fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/csiro_bgc_sediment.res.nc
   hashes:
-    binhash: 5dcf7a15477f488cd7d2f85a1d39a176
+    binhash: fff3375ad2ae36b7d0521dd67912a9fd
     md5: 12488f7cf8f9c2f714c0ff2dbbb33a9b
 work/ocean/INPUT/dust.nc:
   fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/dust.nc

@dougiesquire
Copy link
Collaborator

We decided not to merge this yet, given that there still seems to be a performance hit and further investigation is needed to work out what is causing this.

@CodeGat
Copy link
Contributor Author

CodeGat commented Sep 13, 2024

Looking to close this PR on Monday since we no longer need this build.

@CodeGat CodeGat closed this Sep 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New release with WOMBAT coupling bug fix
2 participants