Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly CI Tests Friday Dec 22, 2023 #2188

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Weekly CI Tests Friday Dec 22, 2023 #2188

wants to merge 1 commit into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Dec 22, 2023

[DO NOT MERGE] Weekly CI Tests Friday Dec 22, 2023

@emcbot emcbot marked this pull request as draft December 22, 2023 22:46
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 22, 2023
@emcbot
Copy link
Author

emcbot commented Dec 22, 2023 

CI Update on Hera at 12/22/23 10:48:08 PM
============================================
Cloning and Building global-workflow PR: 2188
with PID: 186957 on host: hfe05

@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Dec 22, 2023
@emcbot
Copy link
Author

emcbot commented Dec 22, 2023 

CI Update on Orion at 12/22/23 04:48:21 PM
============================================
Cloning and Building global-workflow PR: 2188
with PID: 73894 on host: Orion-login-1

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Dec 22, 2023
@emcbot
Copy link
Author

emcbot commented Dec 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Dec 22 22:53:01 UTC 2023 on hfe05
---------------------------------------------------
Build: Completed at 12/22/23 11:45:03 PM
Case setup: Completed for experiment C384C192_hybatmda_42568dc9
Case setup: Completed for experiment C384_S2SWA_42568dc9
Case setup: Completed for experiment C384_atm3DVar_42568dc9

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 22, 2023
@emcbot
Copy link
Author

emcbot commented Dec 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Dec 22 16:51:29 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build: Completed at 12/22/23 05:45:06 PM
Case setup: Completed for experiment C384_atm3DVar_42568dc9
Case setup: Completed for experiment C384C192_hybatmda_42568dc9
Case setup: Completed for experiment C384_S2SWA_42568dc9

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 23, 2023
@emcbot
Copy link
Author

emcbot commented Dec 23, 2023 

Experiment C384_atm3DVar_42568dc9  *** FAILED *** on Orion
Experiment C384_atm3DVar_42568dc9  with 1 tasks failed at 12/22/23 06:00:22 PM
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/ORION/PR/2188/RUNTESTS/COMROT/C384_atm3DVar_42568dc9/logs/2023040118/gdasfcst.log

@emcbot
Copy link
Author

emcbot commented Dec 23, 2023

Experiment C384C192_hybatmda_42568dc9 SUCCESS on Hera at 12/23/23 02:12:10 AM

@emcbot
Copy link
Author

emcbot commented Dec 23, 2023

Experiment C384_atm3DVar_42568dc9 SUCCESS on Hera at 12/23/23 02:28:20 AM

@emcbot
Copy link
Author

emcbot commented Dec 23, 2023

Experiment C384_S2SWA_42568dc9 SUCCESS on Hera at 12/23/23 04:20:09 AM

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 23, 2023
@emcbot
Copy link
Author

emcbot commented Dec 23, 2023

All CI Test Cases Passed on Hera:

Experiment C384C192_hybatmda_42568dc9 *** SUCCESS *** at 12/23/23 02:12:10 AM
Experiment C384_atm3DVar_42568dc9 *** SUCCESS *** at 12/23/23 02:28:20 AM
Experiment C384_S2SWA_42568dc9 *** SUCCESS *** at 12/23/23 04:20:09 AM

@aerorahul
Copy link
Contributor

/work2/noaa/stmp/GFS_CI_ROOT/ORION/PR/2188/RUNTESTS/COMROT/C384_atm3DVar_42568dc9/logs/2023040118/gdasfcst.log

Looks like Orion having issues again. A snippet from the log file:

+ exglobal_forecast.sh[149]: /bin/cp -p /work2/noaa/stmp/GFS_CI_ROOT/ORION/PR/2188/global-workflow/exec/ufs_mo
del.x /work/noaa/stmp/mterry/RUNDIRS/C384_atm3DVar_42568dc9/fcst.67812/
+ exglobal_forecast.sh[150]: srun -l --export=ALL -n 1280 /work/noaa/stmp/mterry/RUNDIRS/C384_atm3DVar_42568dc
9/fcst.67812/ufs_model.x
 400: slurmstepd: error: couldn't chdir to `/work/noaa/stmp/mterry/RUNDIRS/C384_atm3DVar_42568dc9/fcst.67812':
 No such file or directory: going to /tmp instead

However, directory /work/noaa/stmp/mterry/RUNDIRS/C384_atm3DVar_42568dc9/fcst.67812 does exist and with the executable in it.

❯ ls /work/noaa/stmp/mterry/RUNDIRS/C384_atm3DVar_42568dc9/fcst.314463/

@aerorahul
Copy link
Contributor

A ticket has been opened with Orion helpdesk: Ticket Number 2023122654000044

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emcbot @aerorahul @WalterKolczynski-NOAA @TerryMcGuinness-NOAA