Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly High Resolution CI Tests Friday Sep 22, 2023 #1877

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Weekly High Resolution CI Tests Friday Sep 22, 2023 #1877

wants to merge 1 commit into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Sep 22, 2023

[DO NOT MERGE] Weekly High Resolution CI Tests Friday Sep 22, 2023

@emcbot emcbot marked this pull request as draft September 22, 2023 14:58
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Sep 22 15:01:11 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 15:04:28 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 15:28:26 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:29 UTC 2023 for experiment C384C192_hybatmda_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:32 UTC 2023 for experiment C384_atm3DVar_6fa491f8

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 10:00:36 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 10:03:24 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 10:32:12 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 10:32:22 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 10:32:29 CDT 2023 for experiment C384C192_hybatmda_6fa491f8

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 10:00:36 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 10:03:24 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 10:32:12 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 10:32:22 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 10:32:29 CDT 2023 for experiment C384C192_hybatmda_6fa491f8
Experiment C384_atm3DVar_6fa491f8 Terminated: *FAILED*
Experiment C384_atm3DVar_6fa491f8 Terminated with 2 tasks failed at Fri Sep 22 11:06:09 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1877/RUNTESTS/COMROT/C384_atm3DVar_6fa491f8/logs/2023040200/gdasprep.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1877/RUNTESTS/COMROT/C384_atm3DVar_6fa491f8/logs/2023040200/gfsprep.log

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Walter just sync more data for this SDATE, we will restart in a couple of hours

@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Sep 22 15:01:11 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 15:04:28 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 15:28:26 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:29 UTC 2023 for experiment C384C192_hybatmda_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:32 UTC 2023 for experiment C384_atm3DVar_6fa491f8
Experiment C384C192_hybatmda_6fa491f8 completed: *SUCCESS*
Experiment C384C192_hybatmda_6fa491f8 Completed at Fri Sep 22 18:27:07 UTC 2023
with 114 successfully completed jobs

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Sep 22, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Sep 22, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Sep 22, 2023 •
edited
Loading 

mterry (Orion-login-4) GFS_CI_ROOT $ ls /work/noaa/rstprod/dump/gdas.20230401/18/atmos | tail -3
gdas.t18z.trkob.tm00.bufr_d
gdas.t18z.updated.status.tm00.bufr_d
gdas.t18z.vadwnd.tm00.bufr_d
mterry (Orion-login-4) GFS_CI_ROOT $

sync for SDATE (aka idate) 20230401:18 on Orion looks updated, reverting back to CI-Orion-Ready Label so regular CI scripts will restart the high resolution CI cases on Orion.

@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Sep 22 15:01:11 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 15:04:28 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 15:28:26 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:29 UTC 2023 for experiment C384C192_hybatmda_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:32 UTC 2023 for experiment C384_atm3DVar_6fa491f8
Experiment C384C192_hybatmda_6fa491f8 completed: *SUCCESS*
Experiment C384C192_hybatmda_6fa491f8 Completed at Fri Sep 22 18:27:07 UTC 2023
with 114 successfully completed jobs
Experiment C384_atm3DVar_6fa491f8 completed: *SUCCESS*
Experiment C384_atm3DVar_6fa491f8 Completed at Fri Sep 22 18:51:09 UTC 2023
with 75 successfully completed jobs

@emcbot emcbot added the CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully label Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Sep 22 15:01:11 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 15:04:28 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 15:28:26 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:29 UTC 2023 for experiment C384C192_hybatmda_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 15:28:32 UTC 2023 for experiment C384_atm3DVar_6fa491f8
Experiment C384C192_hybatmda_6fa491f8 completed: *SUCCESS*
Experiment C384C192_hybatmda_6fa491f8 Completed at Fri Sep 22 18:27:07 UTC 2023
with 114 successfully completed jobs
Experiment C384_atm3DVar_6fa491f8 completed: *SUCCESS*
Experiment C384_atm3DVar_6fa491f8 Completed at Fri Sep 22 18:51:09 UTC 2023
with 75 successfully completed jobs

@emcbot emcbot removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 22, 2023
@emcbot emcbot added the CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress label Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 13:41:09 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 13:43:09 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 14:13:29 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 14:13:36 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 14:13:43 CDT 2023 for experiment C384C192_hybatmda_6fa491f8

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 13:41:09 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 13:43:09 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 14:13:29 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 14:13:36 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 14:13:43 CDT 2023 for experiment C384C192_hybatmda_6fa491f8
Experiment C384_atm3DVar_6fa491f8 Terminated: *FAILED*
Experiment C384_atm3DVar_6fa491f8 Terminated with 1 tasks failed at Fri Sep 22 15:50:37 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1877/RUNTESTS/COMROT/C384_atm3DVar_6fa491f8/logs/2023040200/gfsfcst.log

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Sep 22, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 22, 2023
@emcbot
Copy link
Author

emcbot commented Sep 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 17:16:40 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 17:18:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 17:46:42 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:51 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:57 CDT 2023 for experiment C384C192_hybatmda_6fa491f8

@emcbot
Copy link
Author

emcbot commented Sep 23, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 17:16:40 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 17:18:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 17:46:42 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:51 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:57 CDT 2023 for experiment C384C192_hybatmda_6fa491f8
Experiment C384_atm3DVar_6fa491f8 completed: *SUCCESS*
Experiment C384_atm3DVar_6fa491f8 Completed at Fri Sep 22 20:26:08 CDT 2023
with 75 successfully completed jobs

@emcbot
Copy link
Author

emcbot commented Sep 23, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 17:16:40 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 17:18:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 17:46:42 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:51 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:57 CDT 2023 for experiment C384C192_hybatmda_6fa491f8
Experiment C384_atm3DVar_6fa491f8 completed: *SUCCESS*
Experiment C384_atm3DVar_6fa491f8 Completed at Fri Sep 22 20:26:08 CDT 2023
with 75 successfully completed jobs
Experiment C384C192_hybatmda_6fa491f8 completed: *SUCCESS*
Experiment C384C192_hybatmda_6fa491f8 Completed at Fri Sep 22 20:26:21 CDT 2023
with 114 successfully completed jobs

1 similar comment
@emcbot
Copy link
Author

emcbot commented Sep 23, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Sep 22 17:16:40 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Sep 22 17:18:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Sep 22 17:46:42 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:51 CDT 2023 for experiment C384_atm3DVar_6fa491f8
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Sep 22 17:46:57 CDT 2023 for experiment C384C192_hybatmda_6fa491f8
Experiment C384_atm3DVar_6fa491f8 completed: *SUCCESS*
Experiment C384_atm3DVar_6fa491f8 Completed at Fri Sep 22 20:26:08 CDT 2023
with 75 successfully completed jobs
Experiment C384C192_hybatmda_6fa491f8 completed: *SUCCESS*
Experiment C384C192_hybatmda_6fa491f8 Completed at Fri Sep 22 20:26:21 CDT 2023
with 114 successfully completed jobs

@emcbot emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 23, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Sep 25, 2023 •
edited
Loading

Validated that Git SHA hash for the pslot tag in DATA where insufficient for when a "redo" is issued. Walter suggested a bug fix task should be to rm -Rf the pslot_${git SHA} in the DATA directory by the CI driver.sh script just before creating the experiment.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA deleted the weekly_ci_20230922 branch September 29, 2023 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@emcbot @TerrenceMcGuinness-NOAA @TerryMcGuinness-NOAA