Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly High Resolution CI Tests Thursday Sep 21, 2023 #1874

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Weekly High Resolution CI Tests Thursday Sep 21, 2023 #1874

wants to merge 1 commit into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Sep 21, 2023

[DO NOT MERGE] Weekly High Resolution CI Tests Thursday Sep 21, 2023

@emcbot emcbot marked this pull request as draft September 21, 2023 15:47
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Sep 21, 2023
@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Sep 21 15:48:21 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 15:52:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 16:16:08 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:11 UTC 2023 for experiment C384C192_hybatmda_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:14 UTC 2023 for experiment C384_atm3DVar_2889c56c

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 21, 2023
@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Sep 21 10:48:23 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 10:50:40 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 11:18:29 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 11:18:36 CDT 2023 for experiment C384_atm3DVar_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 11:18:41 CDT 2023 for experiment C384C192_hybatmda_2889c56c

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Sep 21, 2023 •
edited
Loading

Orion needs the "dump" stuff too for these C384C192 runs as is on Hera ...

Terry.McGuinness (hfe12) 1874 $ ls /scratch1/NCEPDEV/global/glopara/dump/gdas.20230402/
00  06	12  18

mterry (Orion-login-3) ~ $ ls /work/noaa/rstprod/dump/gdas.20230402/
ls: cannot access /work/noaa/rstprod/dump/gdas.20230402/: No such file or directory

@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Sep 21 15:48:21 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 15:52:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 16:16:08 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:11 UTC 2023 for experiment C384C192_hybatmda_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:14 UTC 2023 for experiment C384_atm3DVar_2889c56c
Experiment C384C192_hybatmda_2889c56c completed: *SUCCESS*
Experiment C384C192_hybatmda_2889c56c Completed at Thu Sep 21 19:00:21 UTC 2023
with 114 successfully completed jobs

@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Sep 21 15:48:21 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 15:52:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 16:16:08 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:11 UTC 2023 for experiment C384C192_hybatmda_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:14 UTC 2023 for experiment C384_atm3DVar_2889c56c
Experiment C384C192_hybatmda_2889c56c completed: *SUCCESS*
Experiment C384C192_hybatmda_2889c56c Completed at Thu Sep 21 19:00:21 UTC 2023
with 114 successfully completed jobs
Experiment C384_atm3DVar_2889c56c completed: *SUCCESS*
Experiment C384_atm3DVar_2889c56c Completed at Thu Sep 21 19:12:09 UTC 2023
with 75 successfully completed jobs

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 21, 2023
@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Sep 21 15:48:21 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 15:52:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 16:16:08 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:11 UTC 2023 for experiment C384C192_hybatmda_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 16:16:14 UTC 2023 for experiment C384_atm3DVar_2889c56c
Experiment C384C192_hybatmda_2889c56c completed: *SUCCESS*
Experiment C384C192_hybatmda_2889c56c Completed at Thu Sep 21 19:00:21 UTC 2023
with 114 successfully completed jobs
Experiment C384_atm3DVar_2889c56c completed: *SUCCESS*
Experiment C384_atm3DVar_2889c56c Completed at Thu Sep 21 19:12:09 UTC 2023
with 75 successfully completed jobs

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 21, 2023
@emcbot
Copy link
Author

emcbot commented Sep 21, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Sep 21 10:48:23 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 10:50:40 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 11:18:29 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 11:18:36 CDT 2023 for experiment C384_atm3DVar_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 11:18:41 CDT 2023 for experiment C384C192_hybatmda_2889c56c
Experiment C384_atm3DVar_2889c56c Terminated: *FAILED*
Experiment C384_atm3DVar_2889c56c Terminated with 1 tasks failed at Thu Sep 21 14:36:10 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1874/RUNTESTS/COMROT/C384_atm3DVar_2889c56c/logs/2023040200/gdasprep.log

@aerorahul
Copy link
Contributor

@KateFriedman-NOAA @WalterKolczynski-NOAA Please take note:

❯❯❯ ls /work/noaa/rstprod/dump/gdas.20230401/
/bin/ls: cannot access /work/noaa/rstprod/dump/gdas.20230401/: No such file or directory

is the cause of the failure in gdasprep in /work2/noaa/stmp/GFS_CI_ROOT/PR/1874/RUNTESTS/COMROT/C384_atm3DVar_2889c56c/logs/2023040200/gdasprep.log

The starting observation dataset is from gdas.20230402. I wonder how @CatherineThomas-NOAA is running an experiment on Orion with these ICs.

@CatherineThomas-NOAA
Copy link
Contributor

@aerorahul I don't think we ever tried running with these ICs on Orion. We mostly run our DA exps on Hera.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Sep 22, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Sep 22, 2023
@WalterKolczynski-NOAA
Copy link
Contributor

Necessary dump has been copied to Orion. Restarting test.

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Sep 21 21:00:56 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 21:03:04 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 21:35:44 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 21:35:53 CDT 2023 for experiment C384_atm3DVar_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 21:35:58 CDT 2023 for experiment C384C192_hybatmda_2889c56c

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Sep 21 21:00:56 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Sep 21 21:03:04 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Sep 21 21:35:44 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 21:35:53 CDT 2023 for experiment C384_atm3DVar_2889c56c
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Sep 21 21:35:58 CDT 2023 for experiment C384C192_hybatmda_2889c56c
Experiment C384_atm3DVar_2889c56c Terminated: *FAILED*
Experiment C384_atm3DVar_2889c56c Terminated with 2 tasks failed at Thu Sep 21 22:10:56 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1874/RUNTESTS/COMROT/C384_atm3DVar_2889c56c/logs/2023040200/gdasprep.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1874/RUNTESTS/COMROT/C384_atm3DVar_2889c56c/logs/2023040200/gfsprep.log

@WalterKolczynski-NOAA
Copy link
Contributor

Is this back to the old collision problem, since the hash hasn't changed?

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

So we just need to run the CI script again to get a new hash?

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA deleted the weekly_ci_20230921 branch September 22, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA will be requested when the pull request is marked ready for review TerrenceMcGuinness-NOAA is a code owner

@WalterKolczynski-NOAA WalterKolczynski-NOAA Awaiting requested review from WalterKolczynski-NOAA WalterKolczynski-NOAA will be requested when the pull request is marked ready for review WalterKolczynski-NOAA is a code owner

Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@emcbot @TerrenceMcGuinness-NOAA @aerorahul @CatherineThomas-NOAA @WalterKolczynski-NOAA @TerryMcGuinness-NOAA