Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace separate JEDI radiance bias correction files with tarball #2875

Merged

Conversation

RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Aug 28, 2024

Description

This PR updates g-w components which handle JEDI radiance bias correction files. Currently, JEDI radiance bias correction files are processed at the satellite_sensor. This PR replaces these multiple files with a single radiance bias correction tarball.

Resolves #2862

Refs:

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES

How has this been tested?

  • Clone, build, and cycled tests on WCOSS2 (Cactus)
  • Clone, build, and cycled tests on Hera
  • Clone, build, and cycled tests on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • New and existing tests pass with my changes

@RussTreadon-NOAA
Copy link
Contributor Author

This PR is opened as a draft to allow others to comment on the changes while we wait for g-w issue #2869 and GDASApp PR #1267 to be closed.

@RussTreadon-NOAA
Copy link
Contributor Author

Hera tests

Install GDASApp PR #1267 with feature/radbcor at 554c55a inside g-w feature/radbcor at 5985e9f.

All 48 test_gdasapp tests Passed (see comment here). All jobs in g-w C96C48_ufs_hybatmDA CI successfully run to completion.

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prtest$ rocotostat -d prtest.db -w prtest.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Sep 07 2024 13:35:14    Sep 07 2024 13:55:14
202402240000        Done    Sep 07 2024 13:35:14    Sep 09 2024 17:25:13

The changes in this PR are ready for review.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review September 9, 2024 17:29
@RussTreadon-NOAA
Copy link
Contributor Author

g-w CI tests

g-w feature/radbcor at 386313b along with GDASApp feature/radbcor at 554c55a have been installed on Hera, Hercules, and WCOSS2 (Dogwood).

The following g-w CI is being run on each machine.

  • Hera: C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA, C48mx500_3DVarAOWCDA
  • Hercules: C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA
  • WCOSS2 (Dogwood): C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA

The Hera and Hercules tests are complete. The indicated g-w CI configurations successfully ran to completion on these machines.

The Dogwood g-w CI configurations are still running.

@RussTreadon-NOAA
Copy link
Contributor Author

g-w CI tests on WCOSS2 (Dogwood)

The C96C48_hybatmDA, C96C48_ufs_hybatmDA, and C96C48_hybatmaerosnowDA g-w CI tests successfully ran to completion on Dogwood.

@emcbot
Copy link

emcbot commented Sep 29, 2024

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_5f7ceebe *** SUCCESS *** at 09/28/24 06:42:13 PM
Experiment C48_S2SW_5f7ceebe *** SUCCESS *** at 09/28/24 06:56:14 PM
Experiment C96C48_hybatmDA_5f7ceebe *** SUCCESS *** at 09/28/24 07:42:21 PM
Experiment C96C48_hybatmaerosnowDA_5f7ceebe *** SUCCESS *** at 09/28/24 08:35:20 PM
Experiment C96C48_ufs_hybatmDA_5f7ceebe *** SUCCESS *** at 09/28/24 09:28:16 PM
Experiment C96_atm3DVar_extended_5f7ceebe *** SUCCESS *** at 09/29/24 04:42:38 AM

@RussTreadon-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA , a rerun of C96C48_hybatmDA on Hercules successfully ran to completion for me. Any more tests you want me to run to move this PR forward?

@aerorahul aerorahul added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Sep 30, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_ATM FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_ATM_5f7ceebe/logs/2021032312/gfsmetpg2o1.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_ATM FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C48_ATM_5f7ceebe

@RussTreadon-NOAA
Copy link
Contributor Author

I do not run C48_ATM because this test does not exercise DA. Log file gfsmetpg2o1.log contains the message

09/30 10:50:15.619 metplus (met_util.py:247) INFO: Check the log file for more information: /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48_ATM_5f7ceebe/gfs.2021032312/met\
pg2o1.716680/grid2obs_step1/metplus_output/logs/C48_ATM_5f7ceebe/master_metplus_grid2obs_step1_upper_air_gatherbyVSDB_for20210323_runon20240930105013.log
srun: error: hercules-01-01: task 0: Exited with exit code 1

The indicated log file has

DEBUG 1: Default Config File: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.9.0/met-9.1.3-76sujnp/share/met/co\
nfig/STATAnalysisConfig_default
DEBUG 1: User Config File: /work2/noaa/stmp/CI/HERCULES/2875/gfs/sorc/verif-global.fd/parm/metplus_config/metplus_use_cases/METplusV3.1/grid2obs/met_config/METV9.1/STA\
TAnalysisConfig
ERROR  :
ERROR  : process_search_dirs() -> no STAT files found in the directories specified!
ERROR  :
ERROR  :
ERROR  : main() -> encountered an error value of 1.  Calling clean_up() and usage() before exiting.
ERROR  :

We need someone from VPPGB to examine the log files to figure out what went wrong.

@aerorahul
Copy link
Contributor

I do not run C48_ATM because this test does not exercise DA. Log file gfsmetpg2o1.log contains the message

09/30 10:50:15.619 metplus (met_util.py:247) INFO: Check the log file for more information: /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48_ATM_5f7ceebe/gfs.2021032312/met\
pg2o1.716680/grid2obs_step1/metplus_output/logs/C48_ATM_5f7ceebe/master_metplus_grid2obs_step1_upper_air_gatherbyVSDB_for20210323_runon20240930105013.log
srun: error: hercules-01-01: task 0: Exited with exit code 1

The indicated log file has

DEBUG 1: Default Config File: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.9.0/met-9.1.3-76sujnp/share/met/co\
nfig/STATAnalysisConfig_default
DEBUG 1: User Config File: /work2/noaa/stmp/CI/HERCULES/2875/gfs/sorc/verif-global.fd/parm/metplus_config/metplus_use_cases/METplusV3.1/grid2obs/met_config/METV9.1/STA\
TAnalysisConfig
ERROR  :
ERROR  : process_search_dirs() -> no STAT files found in the directories specified!
ERROR  :
ERROR  :
ERROR  : main() -> encountered an error value of 1.  Calling clean_up() and usage() before exiting.
ERROR  :

We need someone from VPPGB to examine the log files to figure out what went wrong.

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

@WalterKolczynski-NOAA
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

@aerorahul
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

@WalterKolczynski-NOAA
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

Agreed, since the failure is unrelated to this PR, I say we just merge.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit e2f56f5 into NOAA-EMC:develop Sep 30, 2024
8 of 11 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/radbcor branch September 30, 2024 17:02
@malloryprow
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

It does appear to be related

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96C48_hybatmDA_5f7ceebe/logs/2021122106/gdasanaldiag.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96_atm3DVar FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gdasanaldiag.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C96C48_hybatmDA_5f7ceebe

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96_atm3DVar FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C96_atm3DVar_5f7ceebe

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_S2SWA_gefs_5f7ceebe/logs/2021032312/wave_post_pnt_mem000.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C48_S2SWA_gefs_5f7ceebe

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

CI Failed on Hercules in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2875


Experiment C48_ATM_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 10:54:04 CDT 2024
Experiment C48_ATM_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_ATM_5f7ceebe/logs/2021032312/gfsmetpg2o1.log
Experiment C96_atm3DVar_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 14 dead at Mon Sep 30 12:18:41 CDT 2024
Experiment C96_atm3DVar_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f084.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f087.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f090.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f093.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f096.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f099.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f105.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f117.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f120.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f084.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f087.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f090.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f093.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f096.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f099.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f105.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f117.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f120.log
Experiment C96C48_hybatmDA_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 12:18:48 CDT 2024
Experiment C96C48_hybatmDA_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96C48_hybatmDA_5f7ceebe/logs/2021122106/gdasanaldiag.log
Experiment C48_S2SWA_gefs_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 12:25:28 CDT 2024
Experiment C48_S2SWA_gefs_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_S2SWA_gefs_5f7ceebe/logs/2021032312/wave_post_pnt_mem000.log
Experiment C48_S2SW_5f7ceebe Completed 1 Cycles: *SUCCESS* at Mon Sep 30 12:48:57 CDT 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Process JEDI radiance files as a single tarball