-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ens_design_RRFS] Modifications for user-specified MET INPUT/OUTPUT directories and breaking out MET's ensemble-stat tool #749
Conversation
…for ensgrid jobs.
…LSE (i.e. inline post is deactivated). Also, edits to WRITE_DOPOST description and informational messages.
…f the run_fcst task or run_post metatask as dependencies in the verification tasks when the run_fcst task is turned off.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
… feature/breakout_io
…nsemble) that run verification tasks only (i.e. without first running the forecast model and UPP or tasks upstream of those). Also, change the names of the two existing MET verification WE2E tests to match the two new ones.
LOG_SUFFIX=pointstat_${CDATE} | ||
elif [[ ${DO_ENSEMBLE} == "TRUE" ]]; then | ||
INPUT_BASE=${EXPTDIR}/${CDATE}/${SLASH_ENSMEM_SUBDIR}/postprd | ||
OUTPUT_BASE=${EXPTDIR}/${CDATE}/${SLASH_ENSMEM_SUBDIR} | ||
INPUT_BASE=${MET_INPUT_BASE}/${SLASH_ENSMEM_SUBDIR}/postprd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SLASH_ENSMEM_SUBDIR
already includes the leading "/" character, so toj avoid double-slashes (although it doesn't make a difference in functionality), you can change to:
INPUT_BASE=${MET_INPUT_BASE}${SLASH_ENSMEM_SUBDIR}/postprd
Same for OUTPUT_BASE
.
# Create INPUT_BASE and OUTPUT_BASE to read into METplus conf files. | ||
# | ||
#----------------------------------------------------------------------- | ||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like for all (many of?) the vx tasks, MET_INPUT_BASE
is set in the xml to MET_INPUT_DIR
, then INPUT_BASE
is set to MET_INPUT_BASE
here. Why not just remove MET_INPUT_BASE
from the xml and here and just set INPUT_BASE
here to MET_INPUT_DIR
(which is defined in var_defns.sh
and is available here because var_defns.sh
gets sourced at the beginning of this file (by the line . ${GLOBAL_VAR_DEFNS_FP}
)? MET_INPUT_BASE
seems like an unnecessary middleman (much like CIRES is between me and NOAA ;) ). Or I may have missed something.
Same for the ...OUT...
variables and all (most of?) the other vx tasks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can definitely make the change to remove INPUT/OUTPUT dir from the XML. I had some initial concerns in my head about how to implement this functionality when I first got started on this, but I think what you lay out makes sense.
# Create INPUT_BASE to read into METplus conf files. | ||
# | ||
#----------------------------------------------------------------------- | ||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line caused an error during my testing because MET_INPUT_BASE
is not defined for this task in the xml. I commented out this line in my changes (that I'm going to push to your branch), but I'm not sure if instead MET_INPUT_BASE
should be defined in the xml (depends on if INPUT_BASE
is used in this task). You may have to undo my changes...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just noticed that below, you added INPUT_BASE
to the list of variables that get exported, so I assume it is needed. I won't comment out this line and leave it up to you to define it in the xml.
@michelleharrold Are you planning on merging this into the |
@gsketefian -- this PR is not intended for develop before the release. Eventually, we will get these features into develop, but for the time being they are for the ens_design_RRFS branch we are using for the DTC ensemble task. |
Ok, no worries then. Thanks. |
DESCRIPTION OF CHANGES:
In METv10.1, the ensemble-stat tool has been split into two distinct tools: gen-ens-prod (new tool; creates ensemble products) and ensemble-stat (calculates ensemble stats). These changes break out the current functionality (solely using ensemble-stat) to use both gen-ens-prod and ensemble-stat.
In addition, functionality was added for the user to provide input and output locations for METplus verification via the config.sh.
Note, this PR is for the ens_design_RRFS branch.
TESTS CONDUCTED:
Ran test case (2019061500) with GFSv15.2 physics suite for a 3 member (SPPT) ensemble and compared baseline/current capability to new capability to show results are the same using the gen-ens-prod tool.
Ran a test case (2021053000) with 9-member RRFS ensemble data staged in a different directory than where the MET output was written to test MET INPUT/OUTPUT directory functionality. @willmayfield also ran a test with the 9-member RRFS ensemble data as a sanity check.
DEPENDENCIES:
None.
DOCUMENTATION:
When gen-ens-prod capability is included in develop, the documentation will be updated accordingly.
ISSUE (optional):
Related to issue #709 (eventually this will be merged into develop).
CONTRIBUTORS (optional):
@jwolff-ncar