Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly CI Tests Friday Nov 03, 2023 #2032

Closed
wants to merge 2 commits into from
Closed

[DO NOT MERGE] Weekly CI Tests Friday Nov 03, 2023 #2032

wants to merge 2 commits into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Nov 3, 2023

[DO NOT MERGE] Weekly CI Tests Friday Nov 03, 2023

@emcbot emcbot marked this pull request as draft November 3, 2023 21:44
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 3, 2023
@emcbot
Copy link
Author

emcbot commented Nov 3, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Nov  3 16:45:24 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Fri Nov  3 16:46:29 CDT 2023
Build: Completed at Fri Nov  3 17:24:46 CDT 2023
*** Failed *** to create experiment: C384_atm3DVar_c2ef73a3

2023-11-03 17:24:53,043 - INFO     - root        : BEGIN: __main__.input_args
2023-11-03 17:24:53,044 - DEBUG    - root        : (  )

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 3, 2023
@emcbot
Copy link
Author

emcbot commented Nov 3, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Nov  3 21:48:28 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Nov  3 21:50:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Nov  3 22:32:14 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:19 UTC 2023 for experiment C384C192_hybatmda_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:28 UTC 2023 for experiment C384_S2SWA_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:42 UTC 2023 for experiment C384_atm3DVar_c2ef73a3

@emcbot
Copy link
Author

emcbot commented Nov 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Nov  3 21:48:28 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Nov  3 21:50:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Nov  3 22:32:14 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:19 UTC 2023 for experiment C384C192_hybatmda_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:28 UTC 2023 for experiment C384_S2SWA_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:42 UTC 2023 for experiment C384_atm3DVar_c2ef73a3
Experiment C384C192_hybatmda_c2ef73a3 completed: *SUCCESS*
Experiment C384C192_hybatmda_c2ef73a3 Completed at Sat Nov  4 01:06:18 UTC 2023
with 100 successfully completed jobs

@emcbot
Copy link
Author

emcbot commented Nov 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Nov  3 21:48:28 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Nov  3 21:50:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Nov  3 22:32:14 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:19 UTC 2023 for experiment C384C192_hybatmda_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:28 UTC 2023 for experiment C384_S2SWA_c2ef73a3
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Nov  3 22:32:42 UTC 2023 for experiment C384_atm3DVar_c2ef73a3
Experiment C384C192_hybatmda_c2ef73a3 completed: *SUCCESS*
Experiment C384C192_hybatmda_c2ef73a3 Completed at Sat Nov  4 01:06:18 UTC 2023
with 100 successfully completed jobs
Experiment C384_atm3DVar_c2ef73a3 completed: *SUCCESS*
Experiment C384_atm3DVar_c2ef73a3 Completed at Sat Nov  4 01:15:26 UTC 2023
with 60 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Nov 6, 2023 •
edited
Loading

Hera still stalls on gfsstage_ic for C384_S2SWA with:

/scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/GEFSwave20210528v2_refactored/2016070100/wave/20160701.000000.restart.glo_025 does not exist

@WalterKolczynski-NOAA is aware of this issue and CI code is being developed this week to automate the detections of such stalls of this type.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Nov 6, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 6, 2023
@emcbot
Copy link
Author

emcbot commented Nov 6, 2023 •
edited by TerrenceMcGuinness-NOAA
Loading

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Nov  6 08:54:49 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Mon Nov  6 08:55:38 CST 2023
Build: Completed at Mon Nov  6 09:41:21 CST 2023
*** Failed *** to create experiment: C384_atm3DVar_c2ef73a3

2023-11-06 09:41:26,802 - INFO     - root        : BEGIN: __main__.input_args
2023-11-06 09:41:26,802 - DEBUG    - root        : (  )

Waiting on latest CI PR for adding stdout/sterr to this messaging in CI bash

@emcbot emcbot added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Nov 6, 2023
@emcbot emcbot added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Nov 6, 2023
@emcbot
Copy link
Author

emcbot commented Nov 6, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Nov  6 12:20:26 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Mon Nov  6 12:21:22 CST 2023
Build: Completed at Mon Nov  6 13:05:53 CST 2023
*** Failed *** to create experiment: C384_atm3DVar_f140002e

2023-11-06 13:05:58,707 - INFO     - root        : BEGIN: __main__.input_args
2023-11-06 13:05:58,707 - DEBUG    - root        : (  )

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Nov 6, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 6, 2023
@emcbot
Copy link
Author

emcbot commented Nov 6, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Nov  6 15:40:27 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Mon Nov  6 15:41:20 CST 2023
Build: Completed at Mon Nov  6 16:26:51 CST 2023
Case setup: Completed for experiment C384_atm3DVar_f140002e
Case setup: Completed for experiment C384C192_hybatmda_f140002e
Case setup: Completed for experiment C384_S2SWA_f140002e

@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 6, 2023
@emcbot
Copy link
Author

emcbot commented Nov 6, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Nov  6 16:45:28 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Mon Nov  6 16:46:31 CST 2023
Build: Completed at Mon Nov  6 17:23:53 CST 2023
Case setup: Completed for experiment C384_atm3DVar_f140002e
Case setup: Completed for experiment C384C192_hybatmda_f140002e
Case setup: Completed for experiment C384_S2SWA_f140002e

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Nov 7, 2023
@emcbot
Copy link
Author

emcbot commented Nov 7, 2023 

Experiment C384_atm3DVar_f140002e Terminated: *** FAILED ***
Experiment C384_atm3DVar_f140002e Terminated with 1 tasks failed at Mon Nov  6 19:20:14 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2032/RUNTESTS/COMROT/C384_atm3DVar_f140002e/logs/2023040200/gfsanal.log

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Team working diligently on this issue: #1996

@WalterKolczynski-NOAA
Copy link
Contributor

A newer weekly test has taken the mantle.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress label Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emcbot @TerrenceMcGuinness-NOAA @WalterKolczynski-NOAA @TerryMcGuinness-NOAA