Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly CI Tests Friday Dec 01, 2023 #2111

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Weekly CI Tests Friday Dec 01, 2023 #2111

wants to merge 1 commit into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Dec 1, 2023

[DO NOT MERGE] Weekly CI Tests Friday Dec 01, 2023

@emcbot emcbot marked this pull request as draft December 1, 2023 22:44
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Dec 1, 2023
@emcbot
Copy link
Author

emcbot commented Dec 1, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Dec  1 22:49:02 UTC 2023 on hfe05
---------------------------------------------------
Checkout: Completed at Fri Dec  1 22:51:44 UTC 2023
Build: Completed at Fri Dec  1 23:42:58 UTC 2023
Case setup: Completed for experiment C384C192_hybatmda_8346a87b
Case setup: Completed for experiment C384_S2SWA_8346a87b
Case setup: Completed for experiment C384_atm3DVar_8346a87b

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 1, 2023
@emcbot
Copy link
Author

emcbot commented Dec 1, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Dec  1 16:48:20 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at Fri Dec  1 16:49:30 CST 2023
Build: Completed at Fri Dec  1 17:49:27 CST 2023
Case setup: Completed for experiment C384_atm3DVar_8346a87b
Case setup: Completed for experiment C384C192_hybatmda_8346a87b
Case setup: Completed for experiment C384_S2SWA_8346a87b

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 2, 2023
@emcbot
Copy link
Author

emcbot commented Dec 2, 2023 

Experiment C384C192_hybatmda_8346a87b Terminated: *** FAILED *** on Orion
Experiment C384C192_hybatmda_8346a87b Terminated with 1 tasks failed at Fri Dec  1 19:10:30 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384C192_hybatmda_8346a87b/logs/2023040200/enkfgdaseupd.log

@emcbot
Copy link
Author

emcbot commented Dec 2, 2023

Experiment C384C192_hybatmda_8346a87b SUCCESS Sat Dec 2 02:36:20 UTC 2023 on Hera

@emcbot
Copy link
Author

emcbot commented Dec 2, 2023

Experiment C384_atm3DVar_8346a87b SUCCESS Sat Dec 2 02:36:27 UTC 2023 on Hera

@emcbot
Copy link
Author

emcbot commented Dec 2, 2023

Experiment C384_S2SWA_8346a87b SUCCESS Sat Dec 2 04:32:17 UTC 2023 on Hera

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 2, 2023
@emcbot
Copy link
Author

emcbot commented Dec 2, 2023

All CI Test Cases Passed:

Experiment C384C192_hybatmda_8346a87b **SUCCESS** at Sat Dec  2 02:36:20 UTC 2023 on Hera
Experiment C384_atm3DVar_8346a87b **SUCCESS** at Sat Dec  2 02:36:27 UTC 2023 on Hera
Experiment C384_S2SWA_8346a87b **SUCCESS** at Sat Dec  2 04:32:17 UTC 2023 on Hera

@aerorahul
Copy link
Contributor

aerorahul commented Dec 4, 2023 •
edited by TerrenceMcGuinness-NOAA
Loading

/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384C192_hybatmda_8346a87b/logs/2023040200/enkfgdaseupd.log

stdout from the failed job shows something went wrong on Orion. @TerryMcGuinness-NOAA could you please rerun this job and let the CI continue?

less /work/noaa/stmp/mterry/RUNDIRS/C384C192_hybatmda_8346a87b/eupd.285253/stdout
240: [1701479237.248894] [Orion-25-40:391835:0]         select.c:438  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy
241: [1701479237.248900] [Orion-25-40:391836:0]         select.c:438  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy
242: [1701479237.248899] [Orion-25-40:391837:0]         select.c:438  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy
243: [1701479237.248896] [Orion-25-40:391838:0]         select.c:438  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy
244: [1701479237.248899] [Orion-25-40:391839:0]         select.c:438  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy

Done. ~T.McG

(gfs_workflow) mterry (Orion-login-4) C384C192_hybatmda_8346a87b $ rocotorewind -w C384C192_hybatmda_8346a87b.xml -d C384C192_hybatmda_8346a87b.db -t enkfgdaseupd -c 202304020000
202304020000: Rewind tasks for 202304020000 in state "activated" since 2023-12-01 23:50:24
202304020000: enkfgdaseupd: rewinding dead job.
202304020000: enkfgdaseupd: deleting all records of this job.
(gfs_workflow) mterry (Orion-login-4) C384C192_hybatmda_8346a87b $ rocotorun -v 10 -w C384C192_hybatmda_8346a87b.xml -d C384C192_hybatmda_8346a87b.db
12/04/23 09:09:50 CST :: C384C192_hybatmda_8346a87b.xml :: Submitting gdaspostanl using sbatch < /tmp/sbatch.in20231204-438079-47bmeg with input

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Dec 4, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator 

(gfs_workflow) mterry (Orion-login-4) C384C192_hybatmda_8346a87b $ rocotostat -w C384C192_hybatmda_8346a87b.xml -d C384C192_hybatmda_8346a87b.db -t enkfgdaseupd -c 202304020000
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202304020000            enkfgdaseupd                    15885160           SUCCEEDED                   0         1         387.0
(gfs_workflow) mterry (Orion-login-4) C384C192_hybatmda_8346a87b $

Rewind on enkfgdasepd on C384C192_hybatmd passed and CI is now proceeding in the Running State on Orion.

@emcbot
Copy link
Author

emcbot commented Dec 4, 2023

Experiment C384C192_hybatmda_8346a87b SUCCESS at Mon Dec 4 09:56:19 CST 2023 on Orion

@emcbot
Copy link
Author

emcbot commented Dec 4, 2023

Experiment C384_atm3DVar_8346a87b SUCCESS at Mon Dec 4 10:00:51 CST 2023 on Orion

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 4, 2023
@emcbot
Copy link
Author

emcbot commented Dec 4, 2023

Experiment C384C192_hybatmda_8346a87b Terminated: *** FAILED *** on Orion
Experiment C384C192_hybatmda_8346a87b Terminated with 1 tasks failed at Fri Dec 1 19:10:30 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384C192_hybatmda_8346a87b/logs/2023040200/enkfgdaseupd.log
Experiment C384C192_hybatmda_8346a87b *** SUCCESS *** at Mon Dec 4 09:56:19 CST 2023 on Orion
Experiment C384_atm3DVar_8346a87b *** SUCCESS *** at Mon Dec 4 10:00:51 CST 2023 on Orion
Experiment C384_S2SWA_8346a87b Terminated: *** FAILED *** on Orion
Experiment C384_S2SWA_8346a87b Terminated with 1 tasks failed at Mon Dec 4 10:30:21 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384_S2SWA_8346a87b/logs/2016070100/gfsocnpost_f054-f072.log

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Dec 4, 2023 •
edited
Loading

Note: This output has improved in pending PR 2039

In this "old" output format we can see that C384C192_hybatmda_8346a87b has indeed subsequently PASSED from a previous failure.

The FAIL on this Weekly in now here in C384_S2SWA_8346a87b in a fsocnpost job f054-f072 due to its time limit where all other fconpost jobs didn't.

Restarting it one more time.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Dec 4, 2023
@emcbot
Copy link
Author

emcbot commented Dec 4, 2023

Experiment C384_S2SWA_8346a87b SUCCESS at Mon Dec 4 12:06:07 CST 2023 on Orion

@emcbot emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 4, 2023
@emcbot
Copy link
Author

emcbot commented Dec 4, 2023

All CI Test Cases Passed:

Experiment C384C192_hybatmda_8346a87b Terminated: *** FAILED *** on Orion
Experiment C384C192_hybatmda_8346a87b Terminated with 1 tasks failed at Fri Dec  1 19:10:30 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384C192_hybatmda_8346a87b/logs/2023040200/enkfgdaseupd.log
Experiment C384C192_hybatmda_8346a87b *** SUCCESS *** at Mon Dec  4 09:56:19 CST 2023 on Orion
Experiment C384_atm3DVar_8346a87b *** SUCCESS *** at Mon Dec  4 10:00:51 CST 2023 on Orion
Experiment C384_S2SWA_8346a87b Terminated: *** FAILED *** on Orion
Experiment C384_S2SWA_8346a87b Terminated with 1 tasks failed at Mon Dec  4 10:30:21 CST 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/2111/RUNTESTS/COMROT/C384_S2SWA_8346a87b/logs/2016070100/gfsocnpost_f054-f072.log
Experiment C384_S2SWA_8346a87b *** SUCCESS *** at Mon Dec  4 12:06:07 CST 2023 on Orion

@emcbot emcbot closed this Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emcbot @aerorahul @TerrenceMcGuinness-NOAA @TerryMcGuinness-NOAA