Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ens_design_RRFS] Modifications for user-specified MET INPUT/OUTPUT directories and breaking out MET's ensemble-stat tool #749

Conversation

michelleharrold
Copy link
Collaborator

DESCRIPTION OF CHANGES:

In METv10.1, the ensemble-stat tool has been split into two distinct tools: gen-ens-prod (new tool; creates ensemble products) and ensemble-stat (calculates ensemble stats). These changes break out the current functionality (solely using ensemble-stat) to use both gen-ens-prod and ensemble-stat.

In addition, functionality was added for the user to provide input and output locations for METplus verification via the config.sh.

Note, this PR is for the ens_design_RRFS branch.

TESTS CONDUCTED:

Ran test case (2019061500) with GFSv15.2 physics suite for a 3 member (SPPT) ensemble and compared baseline/current capability to new capability to show results are the same using the gen-ens-prod tool.

Ran a test case (2021053000) with 9-member RRFS ensemble data staged in a different directory than where the MET output was written to test MET INPUT/OUTPUT directory functionality. @willmayfield also ran a test with the 9-member RRFS ensemble data as a sanity check.

DEPENDENCIES:

None.

DOCUMENTATION:

When gen-ens-prod capability is included in develop, the documentation will be updated accordingly.

ISSUE (optional):

Related to issue #709 (eventually this will be merged into develop).

CONTRIBUTORS (optional):

@jwolff-ncar

ush/setup.sh Outdated Show resolved Hide resolved
ush/config_defaults.sh Outdated Show resolved Hide resolved
ush/setup.sh Outdated Show resolved Hide resolved
gsketefian and others added 4 commits April 29, 2022 11:13
…LSE (i.e. inline post is deactivated). Also, edits to WRITE_DOPOST description and informational messages.
…f the run_fcst task or run_post metatask as dependencies in the verification tasks when the run_fcst task is turned off.
Copy link
Collaborator

@willmayfield willmayfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

…nsemble) that run verification tasks only (i.e. without first running the forecast model and UPP or tasks upstream of those). Also, change the names of the two existing MET verification WE2E tests to match the two new ones.
LOG_SUFFIX=pointstat_${CDATE}
elif [[ ${DO_ENSEMBLE} == "TRUE" ]]; then
INPUT_BASE=${EXPTDIR}/${CDATE}/${SLASH_ENSMEM_SUBDIR}/postprd
OUTPUT_BASE=${EXPTDIR}/${CDATE}/${SLASH_ENSMEM_SUBDIR}
INPUT_BASE=${MET_INPUT_BASE}/${SLASH_ENSMEM_SUBDIR}/postprd
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SLASH_ENSMEM_SUBDIR already includes the leading "/" character, so toj avoid double-slashes (although it doesn't make a difference in functionality), you can change to:

INPUT_BASE=${MET_INPUT_BASE}${SLASH_ENSMEM_SUBDIR}/postprd

Same for OUTPUT_BASE.

# Create INPUT_BASE and OUTPUT_BASE to read into METplus conf files.
#
#-----------------------------------------------------------------------
#
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like for all (many of?) the vx tasks, MET_INPUT_BASE is set in the xml to MET_INPUT_DIR, then INPUT_BASE is set to MET_INPUT_BASE here. Why not just remove MET_INPUT_BASE from the xml and here and just set INPUT_BASE here to MET_INPUT_DIR (which is defined in var_defns.sh and is available here because var_defns.sh gets sourced at the beginning of this file (by the line . ${GLOBAL_VAR_DEFNS_FP})? MET_INPUT_BASE seems like an unnecessary middleman (much like CIRES is between me and NOAA ;) ). Or I may have missed something.

Same for the ...OUT... variables and all (most of?) the other vx tasks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can definitely make the change to remove INPUT/OUTPUT dir from the XML. I had some initial concerns in my head about how to implement this functionality when I first got started on this, but I think what you lay out makes sense.

ush/setup.sh Outdated Show resolved Hide resolved
# Create INPUT_BASE to read into METplus conf files.
#
#-----------------------------------------------------------------------
#
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line caused an error during my testing because MET_INPUT_BASE is not defined for this task in the xml. I commented out this line in my changes (that I'm going to push to your branch), but I'm not sure if instead MET_INPUT_BASE should be defined in the xml (depends on if INPUT_BASE is used in this task). You may have to undo my changes...

Copy link
Collaborator

@gsketefian gsketefian Apr 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that below, you added INPUT_BASE to the list of variables that get exported, so I assume it is needed. I won't comment out this line and leave it up to you to define it in the xml.

@gsketefian
Copy link
Collaborator

@michelleharrold Are you planning on merging this into the develop branch in time for the release? I have an issue here (#751) where I'm tracking which PRs that are/have been going into develop also need to be merged into the release branch (which has already been created). I can add this PR to the list if you like.

@michelleharrold
Copy link
Collaborator Author

@gsketefian -- this PR is not intended for develop before the release. Eventually, we will get these features into develop, but for the time being they are for the ens_design_RRFS branch we are using for the DTC ensemble task.

@gsketefian
Copy link
Collaborator

@gsketefian -- this PR is not intended for develop before the release. Eventually, we will get these features into develop, but for the time being they are for the ens_design_RRFS branch we are using for the DTC ensemble task.

Ok, no worries then. Thanks.

@michelleharrold michelleharrold merged commit 1b085c1 into ufs-community:ens_design_RRFS May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants