-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hdf5
bug avoidance for BGC 2024.06
#14
Conversation
🚀 Deploying access-om2-bgc Details and usage instructionsThis
This Prerelease is accessible on Gadi using: module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-2 where the binaries shall be on your DetailsIt will be deployed using:
If this is not what was expected, commit changes to |
Thanks @CodeGat! Some very basic testing indicates that the prerelease runs successfully but there is maybe an impact on performance from these updates 😞 In the table below:
I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the
Below is the (selective) output of a diff between the two experiments: $ git diff -- om2_1deg_jra55_ryf_bgc_pr14-orig om2_1deg_jra55_ryf_bgc_pr14-2
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/config.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/config.yaml
index a971d90..6d2bcae 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/config.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/config.yaml
@@ -31,7 +31,7 @@ input:
submodels:
- name: atmosphere
model: yatm
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+ exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
input:
- /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
- /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -39,7 +39,7 @@ submodels:
- name: ocean
model: mom
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+ exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/fms_ACCESS-OM-BGC.x
input:
- /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/bgc_param.nc
- /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/co2_iaf.nc
@@ -64,7 +64,7 @@ submodels:
- name: ice
model: cice5
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+ exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
input:
- /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/i2o.nc
- /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/o2i.nc
@@ -81,7 +81,7 @@ collate:
mem: 30GB
ncpus: 4
queue: normal
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/mppnccombine.spack
+ exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/mppnccombine.spack
manifest:
reproduce:
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/exe.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/exe.yaml
index fec7972..0952928 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/exe.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
version: 1.0
---
work/atmosphere/yatm.exe:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
hashes:
- binhash: 4e8b4ef76e971c4af3b26cfac632e160
- md5: 5baa1d417fe6708fc30cbeaa57d82f96
+ binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+ md5: f434bd5b517481ae27b4c8f47f95b94d
work/ice/cice_auscom_360x300_24x1_24p.exe:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
hashes:
- binhash: 3a65f67d21152e77034da28c22a94c66
- md5: 37866455b057c85c3ea50c0ef0ea840b
+ binhash: 6f048474c92f2b2cc4ce14d159d9c3ad
+ md5: f74eb0ac0ca781c62d45c3a5e3f2fde7
work/ocean/fms_ACCESS-OM-BGC.x:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-rwhd5nunuux7np2uxswxvujh2qvz63db/bin/fms_ACCESS-OM-BGC.x
hashes:
- binhash: 45352e33876da49ca042014a9f6686e5
- md5: a909552e85690be692ad3ec94016181b
+ binhash: 6fce98ebeb5991cdb82d9ad399c6953d
+ md5: d5fd242ea24aca79e070b47a652b2b62
diff --git a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/input.yaml b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/input.yaml
index 6f6eab3..81471ce 100644
--- a/om2_1deg_jra55_ryf_bgc_pr14-orig/manifests/input.yaml
+++ b/om2_1deg_jra55_ryf_bgc_pr14-2/manifests/input.yaml
@@ -155,7 +155,7 @@ work/ocean/INPUT/csiro_bgc.res.nc:
work/ocean/INPUT/csiro_bgc_sediment.res.nc:
fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/csiro_bgc_sediment.res.nc
hashes:
- binhash: 5dcf7a15477f488cd7d2f85a1d39a176
+ binhash: fff3375ad2ae36b7d0521dd67912a9fd
md5: 12488f7cf8f9c2f714c0ff2dbbb33a9b
work/ocean/INPUT/dust.nc:
fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/dust.nc |
🚀 Deploying access-om2-bgc Details and usage instructionsThis
This Prerelease is accessible on Gadi using: module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-3 where the binaries shall be on your DetailsIt will be deployed using:
If this is not what was expected, commit changes to |
…flags arg from openmpi
🚀 Deploying access-om2-bgc Details and usage instructionsThis
This Prerelease is accessible on Gadi using: module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-4 where the binaries shall be on your DetailsIt will be deployed using:
If this is not what was expected, commit changes to |
🚀 Deploying access-om2-bgc Details and usage instructionsThis
This Prerelease is accessible on Gadi using: module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-5 where the binaries shall be on your DetailsIt will be deployed using:
If this is not what was expected, commit changes to |
🚀 Deploying access-om2-bgc Details and usage instructionsThis
This Prerelease is accessible on Gadi using: module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2-bgc/pr14-6 where the binaries shall be on your DetailsIt will be deployed using:
If this is not what was expected, commit changes to |
In the table below:
I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the
So the prerelease runs (~ 1% slower). Below is the (selective) output of a diff between the two experiments: $ git diff -- orig pr14-6
diff --git a/orig/config.yaml b/pr14-6/config.yaml
index a971d90..3230015 100644
--- a/orig/config.yaml
+++ b/pr14-6/config.yaml
@@ -31,7 +31,7 @@ input:
submodels:
- name: atmosphere
model: yatm
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+ exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
input:
- /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
- /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -39,7 +39,7 @@ submodels:
- name: ocean
model: mom
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+ exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/fms_ACCESS-OM-BGC.x
input:
- /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/bgc_param.nc
- /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/co2_iaf.nc
@@ -64,7 +64,7 @@ submodels:
- name: ice
model: cice5
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+ exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
input:
- /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/i2o.nc
- /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/o2i.nc
@@ -81,7 +81,7 @@ collate:
mem: 30GB
ncpus: 4
queue: normal
- exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/mppnccombine.spack
+ exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/mppnccombine.spack
manifest:
reproduce:
diff --git a/orig/manifests/exe.yaml b/pr14-6/manifests/exe.yaml
index fec7972..e9c5dbb 100644
--- a/orig/manifests/exe.yaml
+++ b/pr14-6/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
version: 1.0
---
work/atmosphere/yatm.exe:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
hashes:
- binhash: 4e8b4ef76e971c4af3b26cfac632e160
- md5: 5baa1d417fe6708fc30cbeaa57d82f96
+ binhash: d14806ad4b0b890d4c2f78a7e34258ab
+ md5: 651ea586afa441f3d94376f28f26b54d
work/ice/cice_auscom_360x300_24x1_24p.exe:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
hashes:
- binhash: 3a65f67d21152e77034da28c22a94c66
- md5: 37866455b057c85c3ea50c0ef0ea840b
+ binhash: 000959a21e84ba4baa66c32e61c03d39
+ md5: 237c0c1838fd072e43f7b632315cc2bb
work/ocean/fms_ACCESS-OM-BGC.x:
- fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33/bin/fms_ACCESS-OM-BGC.x
+ fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2024.06.27_2024.06.27-ki64orvnb3qnwxgsbssj7pw4uvc34oe6/bin/fms_ACCESS-OM-BGC.x
hashes:
- binhash: 45352e33876da49ca042014a9f6686e5
- md5: a909552e85690be692ad3ec94016181b
+ binhash: ba7f3b0ad1f0381786dda70c8fcb609f
+ md5: 9f989f3e7a06b11f91f2967b7c1e262a
diff --git a/orig/manifests/input.yaml b/pr14-6/manifests/input.yaml
index 6f6eab3..81471ce 100644
--- a/orig/manifests/input.yaml
+++ b/pr14-6/manifests/input.yaml
@@ -155,7 +155,7 @@ work/ocean/INPUT/csiro_bgc.res.nc:
work/ocean/INPUT/csiro_bgc_sediment.res.nc:
fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/csiro_bgc_sediment.res.nc
hashes:
- binhash: 5dcf7a15477f488cd7d2f85a1d39a176
+ binhash: fff3375ad2ae36b7d0521dd67912a9fd
md5: 12488f7cf8f9c2f714c0ff2dbbb33a9b
work/ocean/INPUT/dust.nc:
fullpath: /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/dust.nc |
We decided not to merge this yet, given that there still seems to be a performance hit and further investigation is needed to work out what is causing this. |
Looking to close this PR on Monday since we no longer need this build. |
In this PR:
2024.07.02
to avoid the buggyhdf5 1.14.3
.spack-packages
so it is in line with newer versions.References original PR #12
References #13