diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index ec01b3d0b8..3e8ad837b1 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -43,19 +43,21 @@ Describe the steps to reproduce the behavior: - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** -### Projects and Milestone ### -- [ ] Select **Organization** level **Project** for support of the current coordinated release -- [ ] Select **Repository** level **Project** for development toward the next official release or add **alert: NEED CYCLE ASSIGNMENT** label +### Milestone and Projects ### - [ ] Select **Milestone** as the next bugfix version +- [ ] Select **Coordinated METplus-X.Y Support** project for support of the current coordinated release +- [ ] Select **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release ## Define Related Issue(s) ## Consider the impact to the other METplus components. - [ ] [METplus](https://github.com/dtcenter/METplus/issues/new/choose), [MET](https://github.com/dtcenter/MET/issues/new/choose), [METdataio](https://github.com/dtcenter/METdataio/issues/new/choose), [METviewer](https://github.com/dtcenter/METviewer/issues/new/choose), [METexpress](https://github.com/dtcenter/METexpress/issues/new/choose), [METcalcpy](https://github.com/dtcenter/METcalcpy/issues/new/choose), [METplotpy](https://github.com/dtcenter/METplotpy/issues/new/choose) + ## Bugfix Checklist ## See the [METplus Workflow](https://metplus.readthedocs.io/en/latest/Contributors_Guide/github_workflow.html) for details. - [ ] Complete the issue definition above, including the **Time Estimate** and **Funding Source**. @@ -65,20 +67,21 @@ Branch name: `bugfix__main__` - [ ] Add/update log messages for easier debugging. - [ ] Add/update unit tests. - [ ] Add/update documentation. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/overview.html#metplus-components-python-requirements) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. - [ ] Push local changes to GitHub. - [ ] Submit a pull request to merge into **main_\**. Pull request: `bugfix main_ ` - [ ] Define the pull request metadata, as permissions allow. -Select: **Reviewer(s)** and **Development** issues -Select: **Organization** level software support **Project** for the current coordinated release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next bugfix version +Select: Coordinated METplus-X.Y Support project for support of the current coordinated release - [ ] Iterate until the reviewer(s) accept and merge your changes. - [ ] Delete your fork or branch. - [ ] Complete the steps above to fix the bug on the **develop** branch. Branch name: `bugfix__develop_` Pull request: `bugfix develop ` -Select: **Reviewer(s)** and **Development** issues -Select: **Repository** level development cycle **Project** for the next official release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next official version +Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release + - [ ] Close this issue. diff --git a/.github/ISSUE_TEMPLATE/enhancement_request.md b/.github/ISSUE_TEMPLATE/enhancement_request.md index 1b21d5828a..1e7119061c 100644 --- a/.github/ISSUE_TEMPLATE/enhancement_request.md +++ b/.github/ISSUE_TEMPLATE/enhancement_request.md @@ -33,13 +33,14 @@ Consider breaking the enhancement down into sub-issues. - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version, **Consider for Next Release**, or **Backlog of Development Ideas** +- [ ] For the next official version, select the **METplus-Wrappers-X.Y.Z Development** project ## Define Related Issue(s) ## Consider the impact to the other METplus components. @@ -54,14 +55,15 @@ Branch name: `feature__` - [ ] Add/update log messages for easier debugging. - [ ] Add/update unit tests. - [ ] Add/update documentation. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/overview.html#metplus-components-python-requirements) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. +- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html). - [ ] Push local changes to GitHub. - [ ] Submit a pull request to merge into **develop**. Pull request: `feature ` - [ ] Define the pull request metadata, as permissions allow. -Select: **Reviewer(s)** and **Development** issues -Select: **Repository** level development cycle **Project** for the next official release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next official version +Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release - [ ] Iterate until the reviewer(s) accept and merge your changes. - [ ] Delete your fork or branch. - [ ] Close this issue. diff --git a/.github/ISSUE_TEMPLATE/new_feature_request.md b/.github/ISSUE_TEMPLATE/new_feature_request.md index de7fe1325c..855613a96e 100644 --- a/.github/ISSUE_TEMPLATE/new_feature_request.md +++ b/.github/ISSUE_TEMPLATE/new_feature_request.md @@ -37,13 +37,14 @@ Consider breaking the new feature down into sub-issues. - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version, **Consider for Next Release**, or **Backlog of Development Ideas** +- [ ] For the next official version, select the **METplus-Wrappers-X.Y.Z Development** project ## Define Related Issue(s) ## Consider the impact to the other METplus components. @@ -58,14 +59,15 @@ Branch name: `feature__` - [ ] Add/update log messages for easier debugging. - [ ] Add/update unit tests. - [ ] Add/update documentation. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/overview.html#metplus-components-python-requirements) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. +- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html). - [ ] Push local changes to GitHub. - [ ] Submit a pull request to merge into **develop**. Pull request: `feature ` - [ ] Define the pull request metadata, as permissions allow. -Select: **Reviewer(s)** and **Development** issues -Select: **Repository** level development cycle **Project** for the next official release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next official version +Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release - [ ] Iterate until the reviewer(s) accept and merge your changes. - [ ] Delete your fork or branch. - [ ] Close this issue. diff --git a/.github/ISSUE_TEMPLATE/new_use_case.md b/.github/ISSUE_TEMPLATE/new_use_case.md index 645f8ab3ba..66f287669a 100644 --- a/.github/ISSUE_TEMPLATE/new_use_case.md +++ b/.github/ISSUE_TEMPLATE/new_use_case.md @@ -45,14 +45,15 @@ Consider breaking the new feature down into sub-issues. - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** - [ ] Select **privacy** -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version, **Consider for Next Release**, or **Backlog of Development Ideas** +- [ ] For the next official version, select the **METplus-Wrappers-X.Y.Z Development** project ## Define Related Issue(s) ## Consider the impact to the other METplus components. @@ -67,14 +68,15 @@ Branch name: `feature__` - [ ] Add/update log messages for easier debugging. - [ ] Add/update unit tests. - [ ] Add/update documentation. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/overview.html#metplus-components-python-requirements) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. +- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html). - [ ] Push local changes to GitHub. - [ ] Submit a pull request to merge into **develop**. Pull request: `feature ` - [ ] Define the pull request metadata, as permissions allow. -Select: **Reviewer(s)** and **Development** issues -Select: **Repository** level development cycle **Project** for the next official release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next official version +Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release - [ ] Iterate until the reviewer(s) accept your changes. Merge branch into **develop**. - [ ] Create a second pull request to merge **develop** into **develop-ref**, following the same steps for the first pull request. - [ ] Delete your fork or branch. diff --git a/.github/ISSUE_TEMPLATE/sub-issue.md b/.github/ISSUE_TEMPLATE/sub-issue.md index 855e27d83d..d2aa26d7b2 100644 --- a/.github/ISSUE_TEMPLATE/sub-issue.md +++ b/.github/ISSUE_TEMPLATE/sub-issue.md @@ -23,10 +23,11 @@ This is a sub-issue of #*List the parent issue number here*. - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version, **Consider for Next Release**, or **Backlog of Development Ideas** +- [ ] For the next official version, select the **METplus-Wrappers-X.Y.Z Development** project diff --git a/.github/ISSUE_TEMPLATE/task.md b/.github/ISSUE_TEMPLATE/task.md index 2c43db7e65..4996485bc4 100644 --- a/.github/ISSUE_TEMPLATE/task.md +++ b/.github/ISSUE_TEMPLATE/task.md @@ -33,13 +33,14 @@ Consider breaking the task down into sub-issues. - [ ] Select **scientist(s)** or **no scientist** required ### Labels ### +- [ ] Review default **alert** labels - [ ] Select **component(s)** - [ ] Select **priority** - [ ] Select **requestor(s)** -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version, **Consider for Next Release**, or **Backlog of Development Ideas** +- [ ] For the next official version, select the **METplus-Wrappers-X.Y.Z Development** project ## Define Related Issue(s) ## Consider the impact to the other METplus components. @@ -54,14 +55,15 @@ Branch name: `feature__` - [ ] Add/update log messages for easier debugging. - [ ] Add/update unit tests. - [ ] Add/update documentation. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/overview.html#metplus-components-python-requirements) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. +- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html). - [ ] Push local changes to GitHub. - [ ] Submit a pull request to merge into **develop**. Pull request: `feature ` - [ ] Define the pull request metadata, as permissions allow. -Select: **Reviewer(s)** and **Development** issues -Select: **Repository** level development cycle **Project** for the next official release +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the next official version +Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release - [ ] Iterate until the reviewer(s) accept and merge your changes. - [ ] Delete your fork or branch. - [ ] Close this issue. diff --git a/.github/ISSUE_TEMPLATE/update_truth.md b/.github/ISSUE_TEMPLATE/update_truth.md index 5f0be681c1..23c22a3553 100644 --- a/.github/ISSUE_TEMPLATE/update_truth.md +++ b/.github/ISSUE_TEMPLATE/update_truth.md @@ -26,9 +26,10 @@ assignees: '' - [ ] Select **engineer(s)** or **no engineer** required - [ ] Select **scientist(s)** or **no scientist** required -### Projects and Milestone ### -- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label -- [ ] Select **Milestone** as the next official version or **Future Versions** +### Milestone and Projects ### +- [ ] Select **Milestone** as the next official version if updating truth data for the `develop` branch **OR** select next **METplus-Wrappers-X.Y.Z bugfix** version if updating truth data for a `main_vX.Y` branch. +- [ ] If updating truth data for the `develop` branch, select the **METplus-Wrappers-X.Y.Z Development** project **OR** if updating truth for a `main_vX.Y` branch, select the **Coordinated METplus-X.Y Support** project + ## Update Truth Checklist ### - [ ] Review the GitHub Actions workflow that was triggered by the PR merge diff --git a/.github/jobs/configure_sonarqube.sh b/.github/jobs/configure_sonarqube.sh index c60e73485b..aa6827c8a6 100755 --- a/.github/jobs/configure_sonarqube.sh +++ b/.github/jobs/configure_sonarqube.sh @@ -47,9 +47,7 @@ fi # Configure the sonar-project.properties [ -e $SONAR_PROPERTIES ] && rm $SONAR_PROPERTIES -sed -e "s|SONAR_PROJECT_KEY|METplus-GHA|" \ - -e "s|SONAR_PROJECT_NAME|METplus GHA|" \ - -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ +sed -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ -e "s|SONAR_HOST_URL|$SONAR_HOST_URL|" \ -e "s|SONAR_TOKEN|$SONAR_TOKEN|" \ -e "s|SONAR_BRANCH_NAME|$SOURCE_BRANCH|" \ diff --git a/.github/jobs/get_metviewer.sh b/.github/jobs/get_metviewer.sh index 5cf1058454..305fd27763 100755 --- a/.github/jobs/get_metviewer.sh +++ b/.github/jobs/get_metviewer.sh @@ -10,7 +10,7 @@ export METVIEWER_DOCKER_IMAGE=dtcenter/metviewer:develop apk add docker-cli-compose # download docker-compose.yml file from METviewer develop branch -wget https://raw.githubusercontent.com/dtcenter/METviewer/develop/docker/docker-compose.yml +wget https://raw.githubusercontent.com/dtcenter/METviewer/develop/internal/scripts/docker/docker-compose.yml # Run docker-compose to create the containers docker compose up -d diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json index ea51000c83..e53d706aad 100644 --- a/.github/parm/use_case_groups.json +++ b/.github/parm/use_case_groups.json @@ -1,7 +1,7 @@ [ { "category": "met_tool_wrapper", - "index_list": "0-29,59-63", + "index_list": "0-29,59-64", "run": false }, { @@ -57,6 +57,7 @@ { "category": "land_surface", "index_list": "0", + "disabled": true, "run": false }, { diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 7db8bd407f..689cc125a1 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -18,14 +18,15 @@ If **yes**, please describe: ## Pull Request Checklist ## See the [METplus Workflow](https://metplus.readthedocs.io/en/latest/Contributors_Guide/github_workflow.html) for details. -- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html) table. +- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table. +- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html). - [ ] Review the source issue metadata (required labels, projects, and milestone). - [ ] Complete the PR definition above. - [ ] Ensure the PR title matches the feature or bugfix branch name. - [ ] Define the PR metadata, as permissions allow. -Select: **Reviewer(s)** -Select: **Organization** level software support **Project** or **Repository** level development cycle **Project** +Select: **Reviewer(s)** and **Development** issue Select: **Milestone** as the version that will include these changes +Select: **Coordinated METplus-X.Y Support** project for bugfix releases or **METplus-Wrappers-X.Y.Z Development** project for official releases - [ ] After submitting the PR, select the :gear: icon in the **Development** section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected. - [ ] After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge. - [ ] Close the linked issue and delete your feature or bugfix branch from GitHub. diff --git a/.github/workflows/sonarqube.yml b/.github/workflows/sonarqube.yml index b204179029..694697b94c 100644 --- a/.github/workflows/sonarqube.yml +++ b/.github/workflows/sonarqube.yml @@ -54,6 +54,29 @@ jobs: # Disable shallow clones for better analysis fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: '3.10' + cache: 'pip' + + - name: Install Python Test Dependencies + run: | + python3 -m pip install --upgrade pip + python3 -m pip install -r internal/tests/pytests/requirements.txt + + - name: Run Pytests + run: coverage run -m pytest internal/tests/pytests + env: + METPLUS_TEST_OUTPUT_BASE: ${{ runner.workspace }}/pytest_output + + - name: Output coverage report + run: coverage report -m + if: always() + + - name: Generate XML coverage report + run: coverage xml + if: always() + - name: Get branch name id: get_branch_name run: echo branch_name=${GITHUB_REF#refs/heads/} >> $GITHUB_OUTPUT diff --git a/.gitignore b/.gitignore index 4c1083c0d4..ee3f35d9c1 100644 --- a/.gitignore +++ b/.gitignore @@ -89,6 +89,7 @@ ENV/ *.pytest_cache* .idea/workspace.xml +.idea/sonarlint* # tilda files generated by emacs *~ diff --git a/docs/Contributors_Guide/basic_components.rst b/docs/Contributors_Guide/basic_components.rst index 0b24209d85..d73dc7c98d 100644 --- a/docs/Contributors_Guide/basic_components.rst +++ b/docs/Contributors_Guide/basic_components.rst @@ -86,10 +86,15 @@ WRAPPER_ENV_VAR_KEYS This class variable lists all of the environment variables that are set by the wrapper. These variables are typically referenced in the wrapped MET config file for the tool and are named with a *METPLUS\_* prefix. -All of the variables that are referenced in the wrapped MET config file must -be listed here so that they will always be set to prevent an error when MET -reads the config file. An empty string will be set if they are not set to -another value by the wrapper. +An empty string will be set if they are not set to another value by the wrapper. +Almost all of the variables that are referenced in the wrapped MET config file +must be listed here so that they will always be set to prevent an error when MET +reads the config file. The exceptions are **METPLUS_MET_CONFIG_OVERRIDES** and +**METPLUS_TIME_OFFSET_WARNING** which are automatically set and included in the +CommandBuilder initialization step. + +See :ref:`bc_add_met_config` for more information on how to determine the +values to add to this list. DEPRECATED_WRAPPER_ENV_VAR_KEYS -------------------------------- @@ -217,13 +222,78 @@ it is not necessary to set *self.isOK = False* if this function is called. if something_else_goes_wrong: self.isOK = False +.. _bc_find_input_files: + +find_input_files function +========================= + +The **find_input_files** class function should be implemented to handle the +reading of input files that were found for a given run. This function should +add files to the **self.infiles** instance variable. It should return False +if any required input files were not able to be set. It should return True +if everything was read properly. + +.. _bc_set_command_line_arguments: + +set_command_line_arguments function +----------------------------------- + +The **set_command_line_arguments** class function should be implemented to +set any arguments to pass to the command. This function should add to the +**self.args** instance variable for each command line argument. +The *time_info* dictionary containing values to substitute into filename +template tags is passed to this function. Call the **do_string_sub** function +to substitute any values. Note: Eventually the wrappers will be refactored +to handle filename template tag substitution uniformly so substitution in +this function will no longer be necessary. + +.. _bc_set_environment_variables: + +set_environment_variables function +================================== + +The setting of environment variables to be read into the wrapped MET config +file is now handled automatically if using the *add_met_config* functions. +See :ref:`bc_add_met_config`. + +However, if additional environment variables need to be set for a specific +wrapper, the **set_environment_variables** function can be overridden. +**If it is overridden, make sure to call the parent class's implementation of +the function!** Include the following in the function:: + + super().set_environment_variables(time_info) + +Uses add_env_var function (CommandBuilder) to set any shell environment +variables that MET or other METplus wrappers +need to be set. + +.. _bc_get_command: + +get_command function +==================== + +**get_command** assembles the command that will be run. + +Some wrappers will need to define the **get_command** function to format the +command and its arguments. This function reads instance variables and returns +a string of the command to run. Note: Eventually the wrappers will be refactored +to handle this uniformly, so it will not be necessary to override this function +for each wrapper. .. _bc_run_at_time_once: run_at_time_once function ========================= -**run_at_time_once** runs a process for one specific time. The time depends +**run_at_time_once** runs a process for one specific time. + +Previously a **run_at_time_once** function needed to be implemented for each +wrapper. Now this logic is handled in the **RuntimeFreqWrapper**. +It is typically not necessary to override this function in a new wrapper +unless the wrapper requires special logic that isn't covered in the generic +version of the function in *RuntimeFreqWrapper*. + +The time depends on the value of {APP_NAME}_RUNTIME_FREQ. Most wrappers run once per each init or valid and forecast lead time. This function is often defined in each wrapper to handle command setup specific to the wrapper. There is a generic @@ -270,25 +340,6 @@ If a wrapper is not inheriting from RuntimeFreqWrapper or one of its child classes, then the **run_all_times** function can be implemented in the wrapper. This function is called when the wrapper is called. -get_command function -==================== - -**get_command** assembles the command that will be run. -It is defined in CommandBuilder but is overridden in most wrappers because -the command line arguments differ for each MET tool. - -set_environment_variables function -================================== - -Uses add_env_var function (CommandBuilder) to set any shell environment -variables that MET or other METplus wrappers -need to be set. This allows a wrapper to pass information into a MET -configuration file. The MET config file refers to the environment variables. -This is currently only set in wrappers that use MET config files, but the -other wrappers will also need to set environment variables -that are needed to be set in the environment when running, such as -MET_TMP_DIR and MET_PYTHON_EXE. - find_data/find_model/find_obs functions (in CommandBuilder) =========================================================== @@ -362,6 +413,8 @@ lead, lead_seconds, lead_minutes, lead_hours, offset, offset_hours, da_init pcp_combine uses a variety of time_util functions like ti_calculate and ti_get_lead_string +.. _bc_add_met_config: + Adding Support for MET Configuration Variables ============================================== diff --git a/docs/Contributors_Guide/create_wrapper.rst b/docs/Contributors_Guide/create_wrapper.rst index 287ffa0ea7..5edd7b8350 100644 --- a/docs/Contributors_Guide/create_wrapper.rst +++ b/docs/Contributors_Guide/create_wrapper.rst @@ -11,7 +11,8 @@ File Name Create the new wrapper in the *metplus/wrappers* directory and name it to reflect the wrapper's function, e.g.: new_tool_wrapper.py is a wrapper around an application named "new_tool." -Copy the **example_wrapper.py** to start the process. +Identify an existing wrapper that is similar to the new wrapper and +copy it to start the process. Class Name ---------- @@ -57,16 +58,119 @@ MTD or accidentally write PointToGrid instead of Point2Grid:: More than one entry is rarely needed, but they will not hurt anything as long as they do not cause any conflicts. +.. _cw_wrapped_met_config: + +Wrapped MET Config File +======================= + +If the new wrapper corresponds to a MET tool that supports a configuration file, +then a wrapped version of the MET configuration file should be created. +If unsure if the MET tool supports a configuration file, check the +MET User's Guide section for the tool. Alternatively, look for a +file ending with *\_default* that corresponds to the MET tool in the +`data/config `_ +directory of the MET repository. +If this file exists, then the tool supports a configuration file. + +If the new wrapper is for a MET tool that does not support a configuration file +or the wrapper does not correspond to a MET tool at all, then this section can +be skipped. + +Copy Default Config File +------------------------ + +Copy the appropriate default configuration file found in the +`data/config `_ +directory of the MET repository into the parm/met_config directory of the +METplus repository and rename it to end with *_wrapped* instead of *_default*. +For example, if creating the GridStat wrapper, +copy MET/data/config/GridStatConfig\_**default** to +METplus/parm/met_config/GridStatConfig\_**wrapped**. + +**MAKE SURE TO COPY THE DEFAULT CONFIG FILE FROM THE DEVELOP BRANCH TO GET +THE LATEST UPDATES** + +Remove Apostrophe +----------------- + +**REMOVE THE \' CHARACTER FROM THE HEADER COMMENTS TO PREVENT ERRORS RENDERING +THE DOCUMENTATION THAT INCLUDES THE WRAPPED CONFIG FILE** + +Change this line (typically the 5th line):: + + // For additional information, please see the MET User's Guide. + +to:: + + // For additional information, please see the MET Users Guide. + +Changes for All Wrappers +------------------------ + +There are a few changes to make to the wrapped MET config file for all tools. +These changes are found at the very end of the config file. + +tmp_dir +^^^^^^^ + +Set the value of *tmp_dir* to the ${MET_TMP_DIR} environment variable. + +Change:: + + tmp_dir = "/tmp"; + +to:: + + tmp_dir = "${MET_TMP_DIR}"; + + +Remove Version +^^^^^^^^^^^^^^ + +The **version** variable should not be set, as it will conflict with newer +versions of MET. However, it can be nice to keep the variable commented-out +in the wrapped file to easily see roughly when the new wrapper was added. + +Change:: + + version = "V12.0.0"; + +to:: + + //version = "V12.0.0"; + +Common Environment Variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All wrappers automatically support setting *time_offset_warning* and +:ref:`met-config-overrides`. + +Add the following to the very end of the wrapped MET config file:: + + ${METPLUS_TIME_OFFSET_WARNING} + ${METPLUS_MET_CONFIG_OVERRIDES} + +If these variables are not added, these values will not be read by the MET tool. + + Wrapper Components ================== Open the wrapper file for editing the new class. -Naming ------- +Naming/Parent Class +------------------- Rename the class to match the wrapper's class from the above sections. -Most wrappers should be a subclass of the RuntimeFreqWrapper:: +If the new tool falls under one of the existing tool categories, +then make the tool a subclass of one of the existing classes. +This should only be done if the functions in the parent class are needed +by the new wrapper. When in doubt, use the **RuntimeFreqWrapper**. + +See :ref:`bc_class_hierarchy` for more information on existing classes to +determine which class to use as the parent class. + +Example:: class NewToolWrapper(RuntimeFreqWrapper) @@ -80,24 +184,18 @@ To create EnsembleStat wrapper from GridStat, replace **grid_stat** with **ensemble_stat** and **GridStat** with **EnsembleStat**. -Parent Class ------------- - -If the new tool falls under one of the existing tool categories, -then make the tool a subclass of one of the existing classes. -This should only be done if the functions in the parent class are needed -by the new wrapper. When in doubt, use the **RuntimeFreqWrapper**. - -See :ref:`bc_class_hierarchy` for more information on existing classes to -determine which class to use as the parent class. -Class Variables for Runtime Frequency -------------------------------------- +Class Variables +--------------- **RUNTIME_FREQ_DEFAULT** and **RUNTIME_FREQ_SUPPORTED** should be set for all -wrappers that inherit from **RuntimeFreqWrapper**. +wrappers that inherit from **RuntimeFreqWrapper** or one of its sub-classes. -See :ref:`bc_class_vars` for more information. +If the tool can read a config file, then **WRAPPER_ENV_VAR_KEYS** should be +defined to include a list of the environment variables that will be read +by the wrapped config file. + +See :ref:`bc_class_vars` for more information on how to set these variables. Init Function ------------- @@ -132,11 +230,6 @@ The function should also always return the c_dict variable:: return c_dict -File Input/Output -^^^^^^^^^^^^^^^^^ - -METplus configuration variables that end with _DIR and _TEMPLATE are used -to define the criteria to search for input files. Allow Multiple Files ^^^^^^^^^^^^^^^^^^^^ @@ -151,95 +244,166 @@ If it is set to False and a list of files are found for an input (using wildcards or a list of files in the METplus config template variable) then the wrapper will produce an error and not build the command. -Run Functions -------------- +Input Files +^^^^^^^^^^^ + +METplus configuration variables that end with **\_INPUT_DIR** and +**\_INPUT_TEMPLATE** are used to search for input files. + +The **get_input_templates** function can be used to easily set up the wrapper +to read the appropriate METplus config variables for inputs. +The first argument is the c_dict variable, which will be modified by the +function. +The 2nd argument is a dictionary that defines the inputs. The key is the name +of the input type, e.g. *FCST* or *OBS*. The value is a dictionary that must +include at least the *prefix* key which defines the prefix of the METplus +configuration variables to read, +e.g. **{prefix}_INPUT_DIR** and **{prefix}_INPUT_TEMPLATE**. + +The *required* key can be set to specify if the input must be defined in the +METplus config file or not. +If set to True, an error is reported if the *{prefix}_INPUT_TEMPLATE* variable +is not set. If the *required* key is not defined, the default value is True. + +Example 1 (single observation input):: + + self.get_input_templates(c_dict, { + 'OBS': {'prefix': 'MADIS2NC', 'required': True}, + }) + +This will read the METplus config variables **MADIS2NC_INPUT_DIR** and +**MADIS2NC_INPUT_TEMPLATE** and set the c_dict items **OBS_INPUT_DIR** and +**OBS_INPUT_TEMPLATE**. +An error will be reported if **MADIS2NC_INPUT_TEMPLATE** is not set. + +Example 2 (forecast and obs input):: + + self.get_input_templates(c_dict, { + 'FCST': {'prefix': 'FCST_GRID_STAT', 'required': True}, + 'OBS': {'prefix': 'OBS_GRID_STAT', 'required': True}, + }) + +This will read the METplus config variables **FCST_GRID_STAT_INPUT_DIR**, +**FCST_GRID_STAT_INPUT_TEMPLATE**, **OBS_GRID_STAT_INPUT_DIR**, and +**OBS_GRID_STAT_INPUT_TEMPLATE** and set the c_dict items **FCST_INPUT_DIR**, +**FCST_INPUT_TEMPLATE**, **OBS_INPUT_DIR**, and **OBS_INPUT_TEMPLATE**. +An error will be reported if **FCST_GRID_STAT_INPUT_TEMPLATE** or +**OBS_GRID_STAT_INPUT_TEMPLATE** is not set. + +Output Files +^^^^^^^^^^^^ + +METplus configuration variables that end with **\_OUTPUT_DIR** and +**\_OUTPUT_TEMPLATE** are used to write output files. + +Add the following and change **APP_NAME** to the name of the new wrapper:: + + c_dict['OUTPUT_DIR'] = self.config.getdir('APP_NAME_OUTPUT_DIR', '') + c_dict['OUTPUT_TEMPLATE'] = ( + self.config.getraw('config', 'APP_NAME_OUTPUT_TEMPLATE') + ) + +The *OUTPUT_DIR* and *OUTPUT_TEMPLATE* will be concatenated to form the path +to write output. + +Some MET tools write a single output file and some write multiple output files +into a directory. + +If the tool writes multiple output files, then the +*OUTPUT_TEMPLATE* is optional, but can be used to create sub-directories that +include information specific to a given run, like timestamps. + +If the tool writes a single output file, the *OUTPUT_TEMPLATE* is required. +In this case, add a check to report an error if the value is unset:: + + if not c_dict['OUTPUT_TEMPLATE']: + self.log_error('APP_NAME_OUTPUT_TEMPLATE must be set') -* The **run_at_time_once** function or some the functions that it calls will - need to be overridden in the wrapper. - See :ref:`bc_run_at_time_once` for more information. - -* It is recommended to divide up the logic into small functions to make - the code more readable and easier to test. - -* The function self.set_environment_variables should be called by all - wrappers even if the MET tool does not have a config file. - This function is typically called from the run_at_time_once function. - This is done to set environment variables that MET expects to be set when - running, such as MET_TMP_DIR and MET_PYTHON_EXE. - If no environment variables need to be set specific to the wrapper, then no - implementation of the function in the wrapper needs to be written. Call the - implementation of the function from CommandBuilder, which sets the - environment variables defined in the [user_env_vars] section of the - configuration file and outputs DEBUG logs for each environment variable - that has been set in the wrapper. MET_TMP_DIR is automatically set for - each wrapper. - -* Once all the necessary information has been provided to create the MET - command, call self.build(). This calls self.get_command() - to assemble the command and verify that the command wrapper generated - contains all of the required arguments. The get_command() in the wrapper - may need to be overridden if the MET application is different from - the example. - For instance, some MET tools require flags such as -f to - precede the input filename. The get_command function in the wrapper can be - overwritten to prepend the required flag to the filename in the - constructed MET command. - -* Call self.clear() at the beginning of each loop iteration that tries to - build/run a MET command to prevent inadvertently reusing/re-running - commands that were previously created. This is called in the RuntimeFreq - wrapper before each call to run_at_time_once, but an additional call may be - needed if multiple commands are built and run in this function. - -* To allow the use case to use the specific wrapper, assign the wrapper name to - PROCESS_LIST:: - - [config] - PROCESS_LIST = NewExample - -.. note:: - - Do not include the text "Wrapper" at the end of the wrapper name. - - Each value must match an existing wrapper name without the ‘Wrapper' - suffix. The PROCESS_LIST :numref:`Process_list` is located under the - [config] section header in the - use case and/or example configuration file. - -* Add a section to the Python Wrappers page of the documentation with - information about the new tool including a list of all METplus - configuration variables that can be used. - -* Add an entry for each METplus configuration variable added to the wrapper - to the METplus Configuration Glossary. Each configuration variable should - be the MET tool name in all caps i.e. GRID_STAT followed by the variable - name. MET tool names generally have underscores between words unless there - is a number in the name. Examples below:: - - GRID_STAT_PROB_THRESH - REGRID_DATA_PLANE_METHOD - POINT2GRID_QC_FLAGS - -* Create a directory named after the new wrapper to hold the use case - configuration files in the met_tool_wrapper directory that users can run - to try out the new wrapper. In the corresponding directory under - docs/use_cases, be sure to include a .py file that contains the - documentation for that use case and a README file to create a header for - the documentation page. - -This new use case/example configuration file is located in a directory structure -like the following:: - - parm/use_cases/met_tool_wrapper/NewTool/NewTool.conf - docs/use_cases/met_tool_wrapper/NewTool/NewTool.py - docs/use_cases/met_tool_wrapper/NewTool/README.rst - -Note the documentation file is in METplus/docs while the use case conf file -is in METplus/parm. - -Refer to the :ref:`basic_components_of_wrappers` section of the Contributor's +Config File +^^^^^^^^^^^ + +If the wrapper corresponds to a MET tool that supports a MET configuration file, +then add a call to the **get_config_file** function to handle the METplus +configuration settings. Pass the name of the wrapped MET config file that you +have added to *parm/met_config* to the function + +Example for MADIS2NC wrapper:: + + c_dict['CONFIG_FILE'] = self.get_config_file('Madis2NcConfig_wrapped') + +See :ref:`cw_wrapped_met_config` for more information. + +Add calls to **self.add_met_config** or **self.add_met_config_dict** functions +to handle the reading of METplus configuration variables to set wrapped MET +config file variables. See :ref:`bc_add_met_config` for examples and +instructions to use a helper script to determine what to set to add support +for setting a MET config variable through METplus. +If a MET config variable is already supported in another wrapper, refer to +the *create_c_dict* function for that wrapper, copying and modifying function +calls as appropriate. + +Command Line Arguments +^^^^^^^^^^^^^^^^^^^^^^ + +Add calls to *self.config.get* functions to read values from the METplus +config to set *c_dict* items that can be used to set command line arguments. +The METplus configuration variables should match the format +{APP_NAME}_{ARG_NAME} where {APP_NAME} is the name of the wrapper and {ARG_NAME} +is the name of the command line argument. Use the appropriate get function that +corresponds to the argument data type, e.g. *getraw* for strings and +*getint* for integers. + +Example:: + + c_dict['TYPE'] = self.config.getraw('config', 'MADIS2NC_TYPE') + +If the command line argument is required to run, then add a check to report an +error if the value is unset:: + + if not c_dict['TYPE']: + self.log_error('Must set MADIS2NC_TYPE') + +Make sure to remember to add logic to read the c_dict item and add the value +to the command to generate. This can be done in the *set_command_line_arguments* +class function. + +Implement Class Functions +------------------------- + +The following functions should be implemented in the new wrapper: + +* find_input_files +* set_command_line_arguments + +Some wrappers will also need to implement: + +* set_environment_variables +* get_command + +See the :ref:`basic_components_of_wrappers` chapter for more information on +how to define these functions. + +Basic Use Case Example +====================== + +The new wrapper should include a basic use case under the +*parm/use_cases/met_tool_wrapper* directory to demonstrate how to configure it. + +Following the instructions in :ref:`adding-use-cases` and refer to an existing +use case for a similar wrapper. + +Refer to the :ref:`basic_components_of_wrappers` chapter of the Contributor's Guide for more information on what should be added. +Unit Tests +========== + +Unit tests for each wrapper should be defined under +*internal/tests/pytests/wrappers*. +Create a new directory for the new wrapper. +Copy an existing test script for a similar wrapper and modify as needed to +match the new wrapper. + Documentation ============= diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst index 671e6cc037..d268fdb793 100644 --- a/docs/Users_Guide/glossary.rst +++ b/docs/Users_Guide/glossary.rst @@ -1329,14 +1329,14 @@ METplus Configuration Glossary | *Used by:* MODE FCST_MTD_CONV_RADIUS - Comma separated list of convolution radius values used by mode-TD for forecast files. A corresponding variable exists for observation data called :term:`OBS_MTD_CONV_RADIUS`. + A single integer to specify the convolution radius used by mode-TD for forecast files. A corresponding variable exists for observation data called :term:`OBS_MTD_CONV_RADIUS`. - | *Used by:* + | *Used by:* MTD FCST_MTD_CONV_THRESH - Comma separated list of convolution threshold values used by mode-TD for forecast files. A corresponding variable exists for observation data called :term:`OBS_MTD_CONV_THRESH`. + A single convolution threshold used by mode-TD for forecast files. A corresponding variable exists for observation data called :term:`OBS_MTD_CONV_THRESH`. - | *Used by:* + | *Used by:* MTD FCST_MTD_FILE_WINDOW_BEGIN See :term:`OBS_MTD_FILE_WINDOW_BEGIN` @@ -2587,17 +2587,12 @@ METplus Configuration Glossary | *Used by:* MTD MTD_CONV_RADIUS - Comma separated list of convolution radius values used by mode-TD for both forecast and observation files. Has the same behavior as setting :term:`FCST_MTD_CONV_RADIUS` and :term:`OBS_MTD_CONV_RADIUS` to the same value. + A single integer to specify the convolution radius used by mode-TD for both forecast and observation files. Has the same behavior as setting :term:`FCST_MTD_CONV_RADIUS` and :term:`OBS_MTD_CONV_RADIUS` to the same value. | *Used by:* MTD MTD_CONV_THRESH - Comma separated list of convolution threshold values used by mode-TD for both forecast and observation files. Has the same behavior as setting :term:`FCST_MTD_CONV_THRESH` and :term:`OBS_MTD_CONV_THRESH` to the same value. - - | *Used by:* MTD - - MTD_FCST_CONV_RADIUS - Comma separated list of convolution radius values used by mode-TD for forecast files. + A single convolution threshold used by mode-TD for both forecast and observation files. Has the same behavior as setting :term:`FCST_MTD_CONV_THRESH` and :term:`OBS_MTD_CONV_THRESH` to the same value. | *Used by:* MTD @@ -2619,21 +2614,6 @@ METplus Configuration Glossary | *Used by:* MTD - MTD_FCST_CONV_THRESH - Comma separated list of convolution threshold values used by mode-TD for forecast files. - - | *Used by:* MTD - - MTD_OBS_CONV_RADIUS - Comma separated list of convolution radius values used by mode-TD for observation files. - - | *Used by:* MTD - - MTD_OBS_CONV_THRESH - Comma separated list of convolution threshold values used by mode-TD for observation files. - - | *Used by:* MTD - MTD_OUT_DIR .. warning:: **DEPRECATED:** Please use :term:`MTD_OUTPUT_DIR`. @@ -11522,6 +11502,48 @@ METplus Configuration Glossary | *Used by:* WaveletStat + GRID_STAT_UGRID_DATASET + Specify the value for 'ugrid_dataset' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_UGRID_MAX_DISTANCE_KM + Specify the value for 'ugrid_max_distance_km' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_UGRID_COORDINATES_FILE + Specify the value for 'ugrid_coordinates_file' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_UGRID_CONFIG_FILE + Specify the path to the optional ugrid configuration file for GridStat. + This is added to the command using the -config argument. + + | *Used by:* GridStat + + POINT_STAT_UGRID_DATASET + Specify the value for 'ugrid_dataset' in the MET configuration file for PointStat. + + | *Used by:* PointStat + + POINT_STAT_UGRID_MAX_DISTANCE_KM + Specify the value for 'ugrid_max_distance_km' in the MET configuration file for PointStat. + + | *Used by:* PointStat + + POINT_STAT_UGRID_COORDINATES_FILE + Specify the value for 'ugrid_coordinates_file' in the MET configuration file for PointStat. + + | *Used by:* PointStat + + POINT_STAT_UGRID_CONFIG_FILE + Specify the path to the optional ugrid configuration file for PointStat. + This is added to the command using the -config argument. + + | *Used by:* PointStat + POINT_STAT_LAND_MASK_FLAG Specify the value for 'land_mask.flag' in the MET configuration file for PointStat. @@ -11611,3 +11633,291 @@ METplus Configuration Glossary Specify the value for 'obs_perc_value' in the MET configuration file for PointStat. | *Used by:* PointStat + + MADIS2NC_CUSTOM_LOOP_LIST + Sets custom string loop list for a specific wrapper. See :term:`CUSTOM_LOOP_LIST`. + + | *Used by:* MADIS2NC + + LOG_MADIS2NC_VERBOSITY + Overrides the log verbosity for MADIS2NC only. If not set, the verbosity level is controlled by :term:`LOG_MET_VERBOSITY`. + + | *Used by:* MADIS2NC + + MADIS2NC_CONFIG_FILE + Path to configuration file read by madis2nc. + If unset, parm/met_config/Madis2NcConfig_wrapped will be used. + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_IF_OUTPUT_EXISTS + If True, do not run MADIS2NC if output file already exists. Set to False to overwrite files. + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_GRID + Named grid or a data file defining the grid for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_POLY + A polyline file, the output of gen_vx_mask, or a gridded data file with field information for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_SID + A station ID masking file or a comma-separated list of station ID's for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_INPUT_DIR + Directory containing input data to MADIS2NC. This variable is optional because you can specify the full path to the input files using :term:`MADIS2NC_INPUT_TEMPLATE`. + + | *Used by:* MADIS2NC + + MADIS2NC_INPUT_TEMPLATE + Filename template of the input file used by MADIS2NC. See also :term:`MADIS2NC_INPUT_DIR`. + + | *Used by:* MADIS2NC + + MADIS2NC_OUTPUT_DIR + Directory to write output data generated by MADIS2NC. This variable is optional because you can specify the full path to the output files using :term:`MADIS2NC_OUTPUT_TEMPLATE`. + + | *Used by:* MADIS2NC + + MADIS2NC_OUTPUT_TEMPLATE + Filename template of the output file generated by MADIS2NC. See also :term:`MADIS2NC_OUTPUT_DIR`. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_FLAG + Specify the value for 'time_summary.flag' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_RAW_DATA + Specify the value for 'time_summary.raw_data' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_BEG + Specify the value for 'time_summary.beg' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_END + Specify the value for 'time_summary.end' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_STEP + Specify the value for 'time_summary.step' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_WIDTH + Specify the value for 'time_summary.width' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_GRIB_CODE + Specify the value for 'time_summary.grib_code' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_OBS_VAR + Specify the value for 'time_summary.obs_var' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_TYPE + Specify the value for 'time_summary.type' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_VLD_FREQ + Specify the value for 'time_summary.vld_freq' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_VLD_THRESH + Specify the value for 'time_summary.vld_thresh' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_FILE_WINDOW_BEGIN + Used to control the lower bound of the window around the valid time to determine if an MADIS2NC input file should be used for processing. Overrides :term:`OBS_FILE_WINDOW_BEGIN`. See 'Use Windows to Find Valid Files' section for more information. + + | *Used by:* MADIS2NC + + MADIS2NC_FILE_WINDOW_END + Used to control the upper bound of the window around the valid time to determine if an MADIS2NC input file should be used for processing. Overrides :term:`OBS_FILE_WINDOW_END`. See 'Use Windows to Find Valid Files' section for more information. + + | *Used by:* MADIS2NC + + MADIS2NC_MET_CONFIG_OVERRIDES + Override any variables in the MET configuration file that are not + supported by the wrapper. This should be set to the full variable name + and value that you want to override, including the equal sign and the + ending semi-colon. The value is directly appended to the end of the + wrapped MET config file. + + Example: + MADIS2NC_MET_CONFIG_OVERRIDES = desc = "override_desc"; model = "override_model"; + + See :ref:`Overriding Unsupported MET config file settings` for more information + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_VALID_TIMES + List of valid times to skip for MADIS2NC only. + If set, values set in :term:`SKIP_VALID_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_INC_VALID_TIMES + List of valid times to include for MADIS2NC only. + If set, values set in :term:`INC_VALID_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_INIT_TIMES + List of initialization times to skip for MADIS2NC only. + If set, values set in :term:`SKIP_INIT_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_INC_INIT_TIMES + List of initialization times to include for MADIS2NC only. + If set, values set in :term:`INC_INIT_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_TYPE + Specify the value for the '-type' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_QC_DD + Specify the value for the '-qc_dd' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_LVL_DIM + Specify the value for the '-lvl_dim' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_REC_BEG + Specify the value for the '-rec_beg' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_REC_END + Specify the value for the '-rec_end' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + ASCII2NC_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for ASCII2NC. + + | *Used by:* ASCII2NC + + ENSEMBLE_STAT_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for EnsembleStat. + + | *Used by:* EnsembleStat + + GEN_ENS_PROD_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for GenEnsProd. + + | *Used by:* GenEnsProd + + GRID_DIAG_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for GridDiag. + + | *Used by:* GridDiag + + GRID_STAT_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + IODA2NC_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for IODA2NC. + + | *Used by:* IODA2NC + + MODE_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for MODE. + + | *Used by:* MODE + + MTD_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for MTD. + + | *Used by:* MTD + + PB2NC_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for PB2NC. + + | *Used by:* PB2NC + + PLOT_POINT_OBS_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for PlotPointObs. + + | *Used by:* PlotPointObs + + POINT_STAT_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for PointStat. + + | *Used by:* PointStat + + STAT_ANALYSIS_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for StatAnalysis. + + | *Used by:* StatAnalysis + + SERIES_ANALYSIS_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for SeriesAnalysis. + + | *Used by:* SeriesAnalysis + + TC_DIAG_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for TCDiag. + + | *Used by:* TCDiag + + TC_GEN_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for TCGen. + + | *Used by:* TCGen + + TC_PAIRS_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for TCPairs. + + | *Used by:* TCPairs + + TC_RMW_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for TCRMW. + + | *Used by:* TCRMW + + TC_STAT_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for TCStat. + + | *Used by:* TCStat + + WAVELET_STAT_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for WaveletStat. + + | *Used by:* WaveletStat diff --git a/docs/Users_Guide/quicksearch.rst b/docs/Users_Guide/quicksearch.rst index a78f687ba0..30aaca532e 100644 --- a/docs/Users_Guide/quicksearch.rst +++ b/docs/Users_Guide/quicksearch.rst @@ -22,6 +22,7 @@ Use Cases by MET Tool: | `GridStat <../search.html?q=GridStatToolUseCase&check_keywords=yes&area=default>`_ | `GridDiag <../search.html?q=GridDiagToolUseCase&check_keywords=yes&area=default>`_ | `IODA2NC <../search.html?q=IODA2NCToolUseCase&check_keywords=yes&area=default>`_ + | `MADIS2NC <../search.html?q=MADIS2NCToolUseCase&check_keywords=yes&area=default>`_ | `MODE <../search.html?q=MODEToolUseCase&check_keywords=yes&area=default>`_ | `MTD <../search.html?q=MTDToolUseCase&check_keywords=yes&area=default>`_ | `PB2NC <../search.html?q=PB2NCToolUseCase&check_keywords=yes&area=default>`_ @@ -49,6 +50,7 @@ Use Cases by MET Tool: | **GridStat**: *GridStatToolUseCase* | **GridDiag**: *GridDiagToolUseCase* | **IODA2NC**: *IODA2NCToolUseCase* + | **MADIS2NC**: *MADIS2NCToolUseCase* | **MODE**: *MODEToolUseCase* | **MTD**: *MTDToolUseCase* | **PB2NC**: *PB2NCToolUseCase* diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst index 5d33b75b0b..57f28f6a0e 100644 --- a/docs/Users_Guide/release-notes.rst +++ b/docs/Users_Guide/release-notes.rst @@ -17,7 +17,7 @@ is broken down into the following development cycles for each component: 1. **Beta1** releases for the METplus components occurred around 2023-09-15. 2. **Beta2** releases for the METplus components occurred around 2023-11-14. 3. **Beta3** releases for the METplus components occurred around 2024-02-08. -4. **Beta4** releases are tentatively scheduled for 2024-04-17. +4. **Beta4** releases for the METplus components occurred around 2024-04-17. 5. **Beta5** releases are tentatively scheduled for 2024-06-26. 6. **Release Candidate 1** releases have not yet been scheduled. 7. **Official Release** releases have not yet been scheduled. @@ -43,6 +43,43 @@ When applicable, release notes are followed by the `GitHub issue `__ number which describes the bugfix, enhancement, or new feature. + +METplus Version 6.0.0 Beta 4 Release Notes (2024-04-17) +------------------------------------------------------- + + .. dropdown:: Enhancements + + NONE + + .. dropdown:: Bugfix + + * ASCII2NC file window bad default value and redundant initialization of wrappers + (`#2520 `_) + + .. dropdown:: New Wrappers + + NONE + + .. dropdown:: New Use Cases + + NONE + + .. dropdown:: Documentation + + * Create pull-down menus within User's Guide for configuration files + (`#2372 `_) + + * Modify the format of the METplus Components Python Requirements Table + (`#2463 `_) + + .. dropdown:: Internal + + * Automate the MET version number update + (`#2509 `_) + * Add GitHub action to run SonarQube for METplus pull requests and feature branches + (`#2537 `_) + + METplus Version 6.0.0 Beta 3 Release Notes (2024-02-08) ------------------------------------------------------- diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index 321ca160c6..7f06973a12 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -64,6 +64,7 @@ METplus Configuration | :term:`ASCII2NC_INC_VALID_TIMES` | :term:`ASCII2NC_SKIP_INIT_TIMES` | :term:`ASCII2NC_INC_INIT_TIMES` +| :term:`ASCII2NC_TIME_OFFSET_WARNING` | .. _ascii2nc-met-conf: @@ -123,6 +124,18 @@ ${METPLUS_TIME_SUMMARY_DICT} * - :term:`ASCII2NC_TIME_SUMMARY_VALID_THRESH` - time_summary.vld_thresh +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`ASCII2NC_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -330,6 +343,7 @@ METplus Configuration | :term:`ENSEMBLE_STAT_INC_VALID_TIMES` | :term:`ENSEMBLE_STAT_SKIP_INIT_TIMES` | :term:`ENSEMBLE_STAT_INC_INIT_TIMES` +| :term:`ENSEMBLE_STAT_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -942,6 +956,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`ENSEMBLE_STAT_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`ENSEMBLE_STAT_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -1165,6 +1191,8 @@ METplus Configuration | :term:`GEN_ENS_PROD_INC_VALID_TIMES` | :term:`GEN_ENS_PROD_SKIP_INIT_TIMES` | :term:`GEN_ENS_PROD_INC_INIT_TIMES` +| :term:`GEN_ENS_PROD_TIME_OFFSET_WARNING` + .. _gen-ens-prod-met-conf: @@ -1517,6 +1545,18 @@ ${METPLUS_ENSEMBLE_FLAG_DICT} * - :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP` - ensemble_flag.climo_cdp +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GEN_ENS_PROD_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -2866,6 +2906,7 @@ METplus Configuration | :term:`GRID_DIAG_INC_VALID_TIMES` | :term:`GRID_DIAG_SKIP_INIT_TIMES` | :term:`GRID_DIAG_INC_INIT_TIMES` +| :term:`GRID_DIAG_TIME_OFFSET_WARNING` | .. _grid-diag-met-conf: @@ -2990,6 +3031,18 @@ ${METPLUS_MASK_DICT} .. note:: Since the default value in the MET config file for 'grid' is grid = [ "FULL" ];, setting GRID_DIAG_MASK_GRID to an empty string will result in a value of grid = []; in the MET config file. +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_DIAG_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -3158,6 +3211,11 @@ METplus Configuration | :term:`GRID_STAT_INC_VALID_TIMES` | :term:`GRID_STAT_SKIP_INIT_TIMES` | :term:`GRID_STAT_INC_INIT_TIMES` +| :term:`GRID_STAT_UGRID_DATASET` +| :term:`GRID_STAT_UGRID_MAX_DISTANCE_KM` +| :term:`GRID_STAT_UGRID_COORDINATES_FILE` +| :term:`GRID_STAT_UGRID_CONFIG_FILE` +| :term:`GRID_STAT_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED** @@ -3571,6 +3629,18 @@ ${METPLUS_DISTANCE_MAP_DICT} * - :term:`GRID_STAT_DISTANCE_MAP_BETA_VALUE_N` - distance_map.beta_value(n) +${METPLUS_SEEPS_P1_THRESH} +"""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_STAT_SEEPS_P1_THRESH` + - seeps_p1_thresh + ${METPLUS_OUTPUT_FLAG_DICT} """"""""""""""""""""""""""" @@ -3659,8 +3729,8 @@ ${METPLUS_NC_PAIRS_FLAG_DICT} * - :term:`GRID_STAT_NC_PAIRS_FLAG_SEEPS` - nc_pairs_flag.seeps -${METPLUS_SEEPS_P1_THRESH} -"""""""""""""""""""""""""" +${METPLUS_UGRID_DATASET} +^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: :widths: 5 5 @@ -3668,8 +3738,32 @@ ${METPLUS_SEEPS_P1_THRESH} * - METplus Config(s) - MET Config File - * - :term:`GRID_STAT_SEEPS_P1_THRESH` - - seeps_p1_thresh + * - :term:`GRID_STAT_UGRID_DATASET` + - ugrid_dataset + +${METPLUS_UGRID_MAX_DISTANCE_KM} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_STAT_UGRID_MAX_DISTANCE_KM` + - ugrid_max_distance_km + +${METPLUS_UGRID_COORDINATES_FILE} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_STAT_UGRID_COORDINATES_FILE` + - ugrid_coordinates_file ${METPLUS_GRID_WEIGHT_FLAG} """"""""""""""""""""""""""" @@ -3695,6 +3789,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`GRID_STAT_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_STAT_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -3767,6 +3873,7 @@ METplus Configuration | :term:`IODA2NC_INC_VALID_TIMES` | :term:`IODA2NC_SKIP_INIT_TIMES` | :term:`IODA2NC_INC_INIT_TIMES` +| :term:`IODA2NC_TIME_OFFSET_WARNING` .. _ioda2nc-met-conf: @@ -3989,6 +4096,18 @@ ${METPLUS_TIME_SUMMARY_DICT} * - :term:`IODA2NC_TIME_SUMMARY_VLD_THRESH` - time_summary.vld_thresh +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`IODA2NC_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -4002,6 +4121,138 @@ ${METPLUS_MET_CONFIG_OVERRIDES} - n/a +.. _madis2nc_wrapper: + +MADIS2NC +======== + +Description +----------- + +Used to configure the MET tool madis2nc + +METplus Configuration +--------------------- + +| :term:`MADIS2NC_INPUT_DIR` +| :term:`MADIS2NC_OUTPUT_DIR` +| :term:`MADIS2NC_INPUT_TEMPLATE` +| :term:`MADIS2NC_OUTPUT_TEMPLATE` +| :term:`LOG_MADIS2NC_VERBOSITY` +| :term:`MADIS2NC_SKIP_IF_OUTPUT_EXISTS` +| :term:`MADIS2NC_CONFIG_FILE` +| :term:`MADIS2NC_FILE_WINDOW_BEGIN` +| :term:`MADIS2NC_FILE_WINDOW_END` +| :term:`MADIS2NC_TYPE` +| :term:`MADIS2NC_QC_DD` +| :term:`MADIS2NC_LVL_DIM` +| :term:`MADIS2NC_REC_BEG` +| :term:`MADIS2NC_REC_END` +| :term:`MADIS2NC_MASK_GRID` +| :term:`MADIS2NC_MASK_POLY` +| :term:`MADIS2NC_MASK_SID` +| :term:`MADIS2NC_TIME_SUMMARY_FLAG` +| :term:`MADIS2NC_TIME_SUMMARY_RAW_DATA` +| :term:`MADIS2NC_TIME_SUMMARY_BEG` +| :term:`MADIS2NC_TIME_SUMMARY_END` +| :term:`MADIS2NC_TIME_SUMMARY_STEP` +| :term:`MADIS2NC_TIME_SUMMARY_WIDTH` +| :term:`MADIS2NC_TIME_SUMMARY_GRIB_CODE` +| :term:`MADIS2NC_TIME_SUMMARY_OBS_VAR` +| :term:`MADIS2NC_TIME_SUMMARY_TYPE` +| :term:`MADIS2NC_TIME_SUMMARY_VLD_FREQ` +| :term:`MADIS2NC_TIME_SUMMARY_VLD_THRESH` +| :term:`MADIS2NC_CUSTOM_LOOP_LIST` +| :term:`MADIS2NC_MET_CONFIG_OVERRIDES` +| :term:`MADIS2NC_SKIP_VALID_TIMES` +| :term:`MADIS2NC_INC_VALID_TIMES` +| :term:`MADIS2NC_SKIP_INIT_TIMES` +| :term:`MADIS2NC_INC_INIT_TIMES` +| :term:`MADIS2NC_TIME_OFFSET_WARNING` +| + +.. _madis2nc-met-conf: + +MET Configuration +----------------- + +Below is the wrapped MET configuration file used for this wrapper. +Environment variables are used to control entries in this configuration file. +The default value for each environment variable is obtained from +(except where noted below): + +`MET_INSTALL_DIR/share/met/config/Madis2NcConfig_default `_ + +Below the file contents are descriptions of each environment variable +referenced in this file and the corresponding METplus configuration item used +to set the value of the environment variable. For detailed examples showing +how METplus sets the values of these environment variables, +see :ref:`How METplus controls MET config file settings`. + +.. dropdown:: Click to view parm/met_config/Madis2NcConfig_wrapped + + .. literalinclude:: ../../parm/met_config/Madis2NcConfig_wrapped + +Environment variables in wrapped MET config +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +${METPLUS_TIME_SUMMARY_DICT} +"""""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_TIME_SUMMARY_FLAG` + - time_summary.flag + * - :term:`MADIS2NC_TIME_SUMMARY_RAW_DATA` + - time_summary.raw_data + * - :term:`MADIS2NC_TIME_SUMMARY_BEG` + - time_summary.beg + * - :term:`MADIS2NC_TIME_SUMMARY_END` + - time_summary.end + * - :term:`MADIS2NC_TIME_SUMMARY_STEP` + - time_summary.step + * - :term:`MADIS2NC_TIME_SUMMARY_WIDTH` + - time_summary.width + * - :term:`MADIS2NC_TIME_SUMMARY_GRIB_CODE` + - time_summary.grib_code + * - :term:`MADIS2NC_TIME_SUMMARY_OBS_VAR` + - time_summary.obs_var + * - :term:`MADIS2NC_TIME_SUMMARY_TYPE` + - time_summary.type + * - :term:`MADIS2NC_TIME_SUMMARY_VLD_FREQ` + - time_summary.vld_freq + * - :term:`MADIS2NC_TIME_SUMMARY_VLD_THRESH` + - time_summary.vld_thresh + +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_TIME_OFFSET_WARNING` + - time_offset_warning + +${METPLUS_MET_CONFIG_OVERRIDES} +""""""""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_MET_CONFIG_OVERRIDES` + - n/a + + .. _met_db_load_wrapper: METdbLoad @@ -4355,6 +4606,7 @@ METplus Configuration | :term:`MODE_INC_VALID_TIMES` | :term:`MODE_SKIP_INIT_TIMES` | :term:`MODE_INC_INIT_TIMES` +| :term:`MODE_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -4914,7 +5166,6 @@ ${METPLUS_WEIGHT_DICT} * - :term:`MODE_WEIGHT_INTEN_PERC_VALUE` - weight.inten_perc_value - ${METPLUS_INTEREST_FUNCTION_CENTROID_DIST} """""""""""""""""""""""""""""""""""""""""" @@ -5021,6 +5272,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`MODE_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MODE_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -5064,6 +5327,8 @@ METplus Configuration | :term:`FCST_MTD_CONV_THRESH` | :term:`OBS_MTD_CONV_RADIUS` | :term:`OBS_MTD_CONV_THRESH` +| :term:`MTD_CONV_RADIUS` +| :term:`MTD_CONV_THRESH` | :term:`MTD_CUSTOM_LOOP_LIST` | :term:`MTD_SKIP_IF_OUTPUT_EXISTS` | :term:`MTD_DESC` @@ -5090,6 +5355,7 @@ METplus Configuration | :term:`MTD_INC_VALID_TIMES` | :term:`MTD_SKIP_INIT_TIMES` | :term:`MTD_INC_INIT_TIMES` +| :term:`MTD_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -5227,7 +5493,7 @@ ${METPLUS_FCST_CONV_RADIUS} * - METplus Config(s) - MET Config File - * - :term:`MTD_FCST_CONV_RADIUS` + * - :term:`FCST_MTD_CONV_RADIUS` - fcst.conv_radius ${METPLUS_FCST_CONV_THRESH} @@ -5239,7 +5505,7 @@ ${METPLUS_FCST_CONV_THRESH} * - METplus Config(s) - MET Config File - * - :term:`MTD_FCST_CONV_THRESH` + * - :term:`FCST_MTD_CONV_THRESH` - fcst.conv_thresh ${METPLUS_OBS_FILE_TYPE} @@ -5283,7 +5549,7 @@ ${METPLUS_OBS_CONV_RADIUS} * - METplus Config(s) - MET Config File - * - :term:`MTD_OBS_CONV_RADIUS` + * - :term:`OBS_MTD_CONV_RADIUS` - obs.conv_radius ${METPLUS_OBS_CONV_THRESH} @@ -5295,7 +5561,7 @@ ${METPLUS_OBS_CONV_THRESH} * - METplus Config(s) - MET Config File - * - :term:`MTD_OBS_CONV_THRESH` + * - :term:`OBS_MTD_CONV_THRESH` - obs.conv_thresh ${METPLUS_MIN_VOLUME} @@ -5322,6 +5588,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`MTD_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MTD_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -5382,6 +5660,7 @@ METplus Configuration | :term:`PB2NC_INC_VALID_TIMES` | :term:`PB2NC_SKIP_INIT_TIMES` | :term:`PB2NC_INC_INIT_TIMES` +| :term:`PB2NC_TIME_OFFSET_WARNING` .. warning:: **DEPRECATED:** @@ -5586,6 +5865,18 @@ ${METPLUS_TIME_SUMMARY_DICT} * - :term:`PB2NC_TIME_SUMMARY_VALID_THRESH` - time_summary.vld_thresh +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`PB2NC_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -5854,6 +6145,7 @@ Configuration | :term:`PLOT_POINT_OBS_INC_VALID_TIMES` | :term:`PLOT_POINT_OBS_SKIP_INIT_TIMES` | :term:`PLOT_POINT_OBS_INC_INIT_TIMES` +| :term:`PLOT_POINT_OBS_TIME_OFFSET_WARNING` .. _plot-point-obs-met-conf: @@ -6183,6 +6475,18 @@ ${METPLUS_POINT_DATA} * - :term:`PLOT_POINT_OBS_POINT_DATA` - point_data +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`PLOT_POINT_OBS_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -6393,6 +6697,11 @@ Configuration | :term:`POINT_STAT_INC_VALID_TIMES` | :term:`POINT_STAT_SKIP_INIT_TIMES` | :term:`POINT_STAT_INC_INIT_TIMES` +| :term:`POINT_STAT_UGRID_DATASET` +| :term:`POINT_STAT_UGRID_MAX_DISTANCE_KM` +| :term:`POINT_STAT_UGRID_COORDINATES_FILE` +| :term:`POINT_STAT_UGRID_CONFIG_FILE` +| :term:`POINT_STAT_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -6855,6 +7164,18 @@ ${METPLUS_HIRA_DICT} * - :term:`POINT_STAT_HIRA_PROB_CAT_THRESH` - hira.prob_cat_thresh +${METPLUS_SEEPS_P1_THRESH} +"""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`POINT_STAT_SEEPS_P1_THRESH` + - seeps_p1_thresh + ${METPLUS_OUTPUT_FLAG_DICT} """"""""""""""""""""""""""" @@ -6909,8 +7230,8 @@ ${METPLUS_OUTPUT_FLAG_DICT} * - :term:`POINT_STAT_OUTPUT_FLAG_SEEPS_MPR` - output_flag.seeps_mpr -${METPLUS_SEEPS_P1_THRESH} -"""""""""""""""""""""""""" +${METPLUS_UGRID_DATASET} +^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: :widths: 5 5 @@ -6918,9 +7239,33 @@ ${METPLUS_SEEPS_P1_THRESH} * - METplus Config(s) - MET Config File - * - :term:`POINT_STAT_SEEPS_P1_THRESH` - - seeps_p1_thresh - + * - :term:`POINT_STAT_UGRID_DATASET` + - ugrid_dataset + +${METPLUS_UGRID_MAX_DISTANCE_KM} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`POINT_STAT_UGRID_MAX_DISTANCE_KM` + - ugrid_max_distance_km + +${METPLUS_UGRID_COORDINATES_FILE} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`POINT_STAT_UGRID_COORDINATES_FILE` + - ugrid_coordinates_file + ${METPLUS_OUTPUT_PREFIX} """""""""""""""""""""""" @@ -6933,6 +7278,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`POINT_STAT_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`POINT_STAT_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -7150,6 +7507,7 @@ METplus Configuration | :term:`SERIES_ANALYSIS_INC_VALID_TIMES` | :term:`SERIES_ANALYSIS_SKIP_INIT_TIMES` | :term:`SERIES_ANALYSIS_INC_INIT_TIMES` +| :term:`SERIES_ANALYSIS_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -7512,6 +7870,18 @@ ${METPLUS_HSS_EC_VALUE} * - :term:`SERIES_ANALYSIS_HSS_EC_VALUE` - hss_ec_value +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`SERIES_ANALYSIS_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -7817,6 +8187,7 @@ The following values are optional in the METplus configuration file: | :term:`STAT_ANALYSIS_INC_VALID_TIMES` | :term:`STAT_ANALYSIS_SKIP_INIT_TIMES` | :term:`STAT_ANALYSIS_INC_INIT_TIMES` +| :term:`STAT_ANALYSIS_TIME_OFFSET_WARNING` .. warning:: **DEPRECATED:** @@ -8279,6 +8650,18 @@ ${METPLUS_HSS_EC_VALUE} * - :term:`STAT_ANALYSIS_HSS_EC_VALUE` - hss_ec_value +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`STAT_ANALYSIS_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -8359,6 +8742,7 @@ METplus Configuration | :term:`TC_DIAG_INC_VALID_TIMES` | :term:`TC_DIAG_SKIP_INIT_TIMES` | :term:`TC_DIAG_INC_INIT_TIMES` +| :term:`TC_DIAG_TIME_OFFSET_WARNING` | .. _tc-diag-met-conf: @@ -8786,6 +9170,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`TC_DIAG_OUTPUT_BASE_FORMAT` - output_base_format +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`TC_DIAG_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -8891,6 +9287,7 @@ METplus Configuration | :term:`TC_GEN_DLAND_FILE` | :term:`TC_GEN_BASIN_FILE` | :term:`TC_GEN_NC_PAIRS_GRID` +| :term:`TC_GEN_TIME_OFFSET_WARNING` .. warning:: **DEPRECATED:** @@ -9457,6 +9854,18 @@ ${METPLUS_NC_PAIRS_GRID} * - :term:`TC_GEN_NC_PAIRS_GRID` - nc_pairs_grid +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`TC_GEN_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -9552,6 +9961,7 @@ METplus Configuration | :term:`TC_PAIRS_INC_VALID_TIMES` | :term:`TC_PAIRS_SKIP_INIT_TIMES` | :term:`TC_PAIRS_INC_INIT_TIMES` +| :term:`TC_PAIRS_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -9890,6 +10300,18 @@ ${METPLUS_DIAG_CONVERT_MAP_LIST} * - :term:`TC_PAIRS_DIAG_CONVERT_MAP_CONVERT` - diag_convert_map.convert +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`TC_PAIRS_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -9956,6 +10378,7 @@ METplus Configuration | :term:`TC_RMW_INC_VALID_TIMES` | :term:`TC_RMW_SKIP_INIT_TIMES` | :term:`TC_RMW_INC_INIT_TIMES` +| :term:`TC_RMW_TIME_OFFSET_WARNING` | .. _tc-rmw-met-conf: @@ -10229,6 +10652,18 @@ ${METPLUS_RMW_SCALE} * - :term:`TC_RMW_SCALE` - rmw_scale +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`TC_RMW_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -10313,6 +10748,7 @@ METplus Configuration | :term:`TC_STAT_INC_VALID_TIMES` | :term:`TC_STAT_SKIP_INIT_TIMES` | :term:`TC_STAT_INC_INIT_TIMES` +| :term:`TC_STAT_TIME_OFFSET_WARNING` | .. warning:: **DEPRECATED:** @@ -10936,6 +11372,18 @@ ${METPLUS_JOBS} * - :term:`TC_STAT_JOBS_LIST` - jobs +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`TC_STAT_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" @@ -11076,6 +11524,7 @@ METplus Configuration | :term:`WAVELET_STAT_WVLT_PLOT_PLOT_MIN` | :term:`WAVELET_STAT_WVLT_PLOT_PLOT_MAX` | :term:`WAVELET_STAT_OUTPUT_PREFIX` +| :term:`WAVELET_STAT_TIME_OFFSET_WARNING` .. _wavelet-stat-met-conf: @@ -11334,6 +11783,18 @@ ${METPLUS_OUTPUT_PREFIX} * - :term:`WAVELET_STAT_OUTPUT_PREFIX` - output_prefix +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`WAVELET_STAT_TIME_OFFSET_WARNING` + - time_offset_warning + ${METPLUS_MET_CONFIG_OVERRIDES} """"""""""""""""""""""""""""""" diff --git a/docs/Verification_Datasets/datasets.rst b/docs/Verification_Datasets/datasets.rst index 45b78ec5fc..4e320db648 100644 --- a/docs/Verification_Datasets/datasets.rst +++ b/docs/Verification_Datasets/datasets.rst @@ -16,6 +16,7 @@ Datasets datasets/mrms_composite_reflectivity datasets/nexrad_level_2 datasets/nexrad_level_3 + datasets/omi_aura_toms_level_3.rst datasets/opera_eumetnet datasets/gdas_ice datasets/ims_ice diff --git a/docs/Verification_Datasets/datasets/gdas_prepbufr.rst b/docs/Verification_Datasets/datasets/gdas_prepbufr.rst index 0fa11c195b..a11768fcb9 100644 --- a/docs/Verification_Datasets/datasets/gdas_prepbufr.rst +++ b/docs/Verification_Datasets/datasets/gdas_prepbufr.rst @@ -11,6 +11,8 @@ Description Note: GDAS prepbufr data is also available via other sources (e.g., NOMADS, NOAA HPSS, and NCAR/Research Data Archive); however, NOAA/NCEI does not require an account and provides a long archive period. + .. note:: Surface winds are by default encoded with a quality mark of 9, to avoid being used in data assimilation. To use surface winds from GDAS prepbufr, be sure to include quality mark values of 9 when running PB2NC, which can be controlled via the PB2NC configuration file. + Recommended use GDAS prepbufr data are commonly used for verification of standard meteorological variables, such as temperatures, dew point temperature, wind speed, visibility, and precipitation, and precipitation type. diff --git a/docs/Verification_Datasets/datasets/images/OMTO3e_003.png b/docs/Verification_Datasets/datasets/images/OMTO3e_003.png new file mode 100644 index 0000000000..37a39c5235 Binary files /dev/null and b/docs/Verification_Datasets/datasets/images/OMTO3e_003.png differ diff --git a/docs/Verification_Datasets/datasets/omi_aura_toms_level_3.rst b/docs/Verification_Datasets/datasets/omi_aura_toms_level_3.rst new file mode 100644 index 0000000000..21bb5affc4 --- /dev/null +++ b/docs/Verification_Datasets/datasets/omi_aura_toms_level_3.rst @@ -0,0 +1,48 @@ +.. _vx-data-omi_aura_toms_level_3.rst: + +OMI Aura Level 3 Total Ozone +---------------------------- + +Description + NASA Level-3 Aura/OMI Global Total Ozone Mapping Spectrometer-Like (TOMS-Like) Total Column Ozone gridded product OMTO3e (0.25deg Lat/Lon grid) + + https://disc.gsfc.nasa.gov/datasets/OMTO3e_003/summary + +Sample image + .. image:: images/OMTO3e_003.png + :width: 600 + +Recommended use + Evaluating total column ozone with remotely-sensed data + +File format + HDF-EOS5 + +Location of data + https://acdisc.gesdisc.eosdis.nasa.gov/data/Aura_OMI_Level3/OMTO3e.003/ + +Access restrictions + Access to GES DISC data requires all users to be registered with the Earthdata Login system. + +Spatial resolution, grid, or coverage + Full global coverage on at 0.25 degree grid + +Temporal resolution + 1 day + +Starting and/or ending dates + 2004-10-01 to 2024-05-12 + +Data latency + ~1 day + +Variables available + Total Column Ozone + +METplus Use Cases + Link to `METplus Use Cases `_ for this dataset. + +Keywords + .. note:: **Current Dataset:** VxDataOMIAURATOMSLEV3 + + .. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationMediumRange diff --git a/docs/_static/Estes_Park_Sky_3_March_2017.png b/docs/_static/Estes_Park_Sky_3_March_2017.png deleted file mode 100755 index f646d06581..0000000000 Binary files a/docs/_static/Estes_Park_Sky_3_March_2017.png and /dev/null differ diff --git a/docs/_static/METplus_banner_photo.png b/docs/_static/METplus_banner_photo.png deleted file mode 100755 index a33b858c87..0000000000 Binary files a/docs/_static/METplus_banner_photo.png and /dev/null differ diff --git a/docs/_static/METplus_banner_photo_web.png b/docs/_static/METplus_banner_photo_web.png old mode 100755 new mode 100644 diff --git a/docs/_static/METplus_logo.png b/docs/_static/METplus_logo.png old mode 100755 new mode 100644 diff --git a/docs/_static/data_assimilation-StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.png b/docs/_static/data_assimilation-StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.png old mode 100755 new mode 100644 diff --git a/docs/_static/download_metplus.png b/docs/_static/download_metplus.png old mode 100755 new mode 100644 diff --git a/docs/_static/marine_and_cryosphere_GridStat_MODE_fcstIMS_obsNCEP_Sea_Ice.png b/docs/_static/marine_and_cryosphere_GridStat_MODE_fcstIMS_obsNCEP_Sea_Ice.png old mode 100755 new mode 100644 diff --git a/docs/_static/met_tool_wrapper-MADIS2NC.png b/docs/_static/met_tool_wrapper-MADIS2NC.png new file mode 100644 index 0000000000..cab2ce69cc Binary files /dev/null and b/docs/_static/met_tool_wrapper-MADIS2NC.png differ diff --git a/docs/_static/met_tool_wrapper-PlotDataPlane.png b/docs/_static/met_tool_wrapper-PlotDataPlane.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo.png b/docs/_static/metplus_repo.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo_release.png b/docs/_static/metplus_repo_release.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo_releases_page.png b/docs/_static/metplus_repo_releases_page.png old mode 100755 new mode 100644 diff --git a/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py b/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py new file mode 100644 index 0000000000..709edb3fcc --- /dev/null +++ b/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py @@ -0,0 +1,105 @@ +""" +MADIS2NC: Basic Use Case +======================== + +met_tool_wrapper/MADIS2NC/MADIS2NC.conf + +""" +############################################################################## +# Scientific Objective +# -------------------- +# +# Converting file formats so point observations can be read by the MET tools. + +############################################################################## +# Datasets +# -------- +# +# | **Observations:** METAR observations in MADIS NetCDF files +# +# | **Location:** All of the input data required for this use case can be found in the met_test sample data tarball. Click here to the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases +# | This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See `Running METplus`_ section for more information. +# +# | **Data Source:** MADIS +# | + +############################################################################## +# METplus Components +# ------------------ +# +# This use case utilizes the METplus MADIS2NC wrapper to generate a command to run the MET tool madis2nc if all required files are found. + +############################################################################## +# METplus Workflow +# ---------------- +# +# MADIS2NC is the only tool called in this example. It processes the following +# run time: +# +# | **Valid:** 2012-04-09_12Z +# | + +############################################################################## +# METplus Configuration +# --------------------- +# +# METplus first loads the default configuration file found in parm/metplus_config, +# then it loads any configuration files passed to METplus via the command line. +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf + +############################################################################## +# MET Configuration +# --------------------- +# +# METplus sets environment variables based on user settings in the METplus configuration file. +# See :ref:`How METplus controls MET config file settings` for more details. +# +# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!** +# +# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to: +# :ref:`Overriding Unsupported MET config file settings` +# +# .. note:: See the :ref:`MADIS2NC MET Configuration` section of the User's Guide for more information on the environment variables used in the file below: +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/met_config/Madis2NcConfig_wrapped + +############################################################################## +# Running METplus +# --------------- +# +# Pass the path to MADIS2NC.conf as an argument to run_metplus.py:: +# +# run_metplus.py /path/to/METplus/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf +# + +############################################################################## +# Expected Output +# --------------- +# +# A successful run will output the following both to the screen and to the logfile:: +# +# INFO: METplus has successfully finished running. +# +# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated. +# Output for this use case will be found in madis2nc (relative to **OUTPUT_BASE**) +# and will contain the following file: +# +# * met_metar_2012040912_F000.nc + +############################################################################## +# Keywords +# -------- +# +# .. note:: +# +# * MADIS2NCToolUseCase +# +# Navigate to the :ref:`quick-search` page to discover other similar use cases. +# +# +# +# sphinx_gallery_thumbnail_path = '_static/met_tool_wrapper-MADIS2NC.png' +# diff --git a/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst b/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst new file mode 100644 index 0000000000..767da397da --- /dev/null +++ b/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst @@ -0,0 +1,2 @@ +MADIS2NC +-------- diff --git a/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py b/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py index 521f84f209..71a828ee1d 100644 --- a/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py +++ b/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py @@ -127,6 +127,7 @@ # * PythonEmbeddingFileUseCase # * RegriddingInToolUseCase # * NOAAEMCOrgUseCase +# * VxDataOMIAURATOMSLEV3 # # Navigate to the :ref:`quick-search` page to discover other similar use cases. # diff --git a/internal/scripts/installation/modulefiles/5.1.0_cheyenne b/internal/scripts/installation/modulefiles/5.1.0_cheyenne deleted file mode 100644 index 4c204e2708..0000000000 --- a/internal/scripts/installation/modulefiles/5.1.0_cheyenne +++ /dev/null @@ -1,17 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.1.0 - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module use /glade/p/ral/jntp/MET/MET_releases/modulefiles -module load met/11.1.0 -module load nco -module load grib-bins/1.3 -module load R - -setenv METPLUS_PATH /glade/p/ral/jntp/MET/METplus/METplus-5.1.0 -prepend-path PATH /glade/p/ral/jntp/MET/METplus/METplus-5.1.0/ush:/glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin diff --git a/internal/scripts/installation/modulefiles/6.0.0_casper b/internal/scripts/installation/modulefiles/6.0.0_casper new file mode 100644 index 0000000000..481e7297b3 --- /dev/null +++ b/internal/scripts/installation/modulefiles/6.0.0_casper @@ -0,0 +1,15 @@ +#%Module###################################################################### +## +## METplus +## +proc ModulesHelp { } { + puts stderr "Sets up the paths and environment variables to use the METplus-6.0.0 + *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" +} + +module use /glade/work/dtcrt/METplus/casper/components/MET/installations/modulefiles +module load met/12.0.0 +module load nco + +setenv METPLUS_PATH /glade/work/dtcrt/METplus/casper/components/METplus/installations/METplus-6.0.0 +prepend-path PATH /glade/work/dtcrt/METplus/casper/components/METplus/installations/METplus-6.0.0/ush:/glade/work/dtcrt/METplus/casper/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin \ No newline at end of file diff --git a/internal/scripts/installation/modulefiles/6.0.0_derecho b/internal/scripts/installation/modulefiles/6.0.0_derecho new file mode 100644 index 0000000000..9e9c2c7ae9 --- /dev/null +++ b/internal/scripts/installation/modulefiles/6.0.0_derecho @@ -0,0 +1,17 @@ +#%Module###################################################################### +## +## METplus +## +proc ModulesHelp { } { + puts stderr "Sets up the paths and environment variables to use the METplus-6.0.0 + *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" +} + +module use /glade/work/dtcrt/METplus/derecho/components/MET/installations/modulefiles +module load met/12.0.0 +module load nco + +setenv METPLUS_PATH /glade/work/dtcrt/METplus/derecho/components/METplus/installations/METplus-6.0.0 +prepend-path PATH /glade/work/dtcrt/METplus/derecho/components/METplus/installations/METplus-6.0.0/ush:/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin + + diff --git a/internal/scripts/installation/modulefiles/6.0.0_hercules b/internal/scripts/installation/modulefiles/6.0.0_hercules new file mode 100644 index 0000000000..5cba910f0f --- /dev/null +++ b/internal/scripts/installation/modulefiles/6.0.0_hercules @@ -0,0 +1,17 @@ +#%Module###################################################################### +## +## METplus +## +proc ModulesHelp { } { + puts stderr "Sets up the paths and environment variables to use the METplus-6.0.0 + *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" +} +module load contrib +module load intel-oneapi-compilers/2022.2.1 +module load met/12.0.0 +module load nco/5.1.6 +module load wgrib2/3.1.1 + +prepend-path PATH /work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin +setenv METPLUS_PATH /apps/contrib/MET/METplus/METplus-6.0.0 +prepend-path PATH /apps/contrib/MET/METplus/METplus-6.0.0/ush diff --git a/internal/scripts/installation/modulefiles/6.0.0_jet b/internal/scripts/installation/modulefiles/6.0.0_jet index f52b066e8a..94c9a3c0ae 100644 --- a/internal/scripts/installation/modulefiles/6.0.0_jet +++ b/internal/scripts/installation/modulefiles/6.0.0_jet @@ -12,7 +12,7 @@ prereq nco/4.9.1 prereq wgrib/1.8.1.0b prereq wgrib2/2.0.8 prereq R/4.0.2 -prereq met/12.0.0-beta1 +prereq met/12.0.0 -setenv METPLUS_PATH /contrib/met/METplus/METplus-12.0.0-beta1 -prepend-path PATH /contrib/met/METplus/METplus-12.0.0-beta1/ush:/mnt/lfs1/HFIP/dtc-hurr/METplus/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin +setenv METPLUS_PATH /contrib/met/METplus/METplus-6.0.0 +prepend-path PATH /contrib/met/METplus/METplus-6.0.0/ush:/mnt/lfs1/HFIP/dtc-hurr/METplus/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin diff --git a/internal/scripts/installation/modulefiles/5.1.0_orion b/internal/scripts/installation/modulefiles/6.0.0_orion similarity index 79% rename from internal/scripts/installation/modulefiles/5.1.0_orion rename to internal/scripts/installation/modulefiles/6.0.0_orion index 58ea4803fa..dc6238fa6a 100644 --- a/internal/scripts/installation/modulefiles/5.1.0_orion +++ b/internal/scripts/installation/modulefiles/6.0.0_orion @@ -3,7 +3,7 @@ ## METplus ## proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.1.0 + puts stderr "Sets up the paths and environment variables to use the METplus-6.0.0 *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" } #%Module###################################################################### @@ -11,15 +11,15 @@ proc ModulesHelp { } { ## METplus ## proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.1.0 + puts stderr "Sets up the paths and environment variables to use the METplus-6.0.0 *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" } module load contrib module load intel/2020.2 -module load met/11.1.0 +module load met/12.0.0 module load nco/4.8.1 module load wgrib/2.0.8 prepend-path PATH /work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin -setenv METPLUS_PATH /apps/contrib/MET/METplus/METplus-5.1.0 -prepend-path PATH /apps/contrib/MET/METplus/METplus-5.1.0/ush +setenv METPLUS_PATH /apps/contrib/MET/METplus/METplus-6.0.0 +prepend-path PATH /apps/contrib/MET/METplus/METplus-6.0.0/ush diff --git a/internal/scripts/sonarqube/run_sonarqube.sh b/internal/scripts/sonarqube/run_sonarqube.sh index 3a3669c37a..14708e23f2 100755 --- a/internal/scripts/sonarqube/run_sonarqube.sh +++ b/internal/scripts/sonarqube/run_sonarqube.sh @@ -118,9 +118,7 @@ SONAR_PROPERTIES=sonar-project.properties # Configure the sonar-project.properties [ -e $SONAR_PROPERTIES ] && rm $SONAR_PROPERTIES -sed -e "s|SONAR_PROJECT_KEY|METplus_NB|" \ - -e "s|SONAR_PROJECT_NAME|METplus Nightly Build|" \ - -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ +sed -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ -e "s|SONAR_HOST_URL|$SONAR_HOST_URL|" \ -e "s|SONAR_TOKEN|$SONAR_TOKEN|" \ -e "s|SONAR_BRANCH_NAME|${1}|" \ diff --git a/internal/scripts/sonarqube/sonar-project.properties b/internal/scripts/sonarqube/sonar-project.properties index d1ed593a85..1504a3f161 100644 --- a/internal/scripts/sonarqube/sonar-project.properties +++ b/internal/scripts/sonarqube/sonar-project.properties @@ -1,10 +1,11 @@ # Project and source code settings -sonar.projectKey=SONAR_PROJECT_KEY -sonar.projectName=SONAR_PROJECT_NAME +sonar.projectKey=METplus +sonar.projectName=METplus sonar.projectVersion=SONAR_PROJECT_VERSION sonar.branch.name=SONAR_BRANCH_NAME sonar.sources=docs,internal,manage_externals,metplus,parm,produtil,ush -sonar.coverage.exclusions=internal/tests/** +sonar.coverage.exclusions=internal/tests/**,parm/**,internal/scripts/**,manage_externals/**,docs/**,produtil/**,ush/**,metplus/wrappers/cyclone_plotter_wrapper.py +sonar.python.coverage.reportPaths=coverage.xml sonar.sourceEncoding=UTF-8 # SonarQube server diff --git a/internal/tests/data/madis/metar_2019040912_F000.nc b/internal/tests/data/madis/metar_2019040912_F000.nc new file mode 100644 index 0000000000..e69de29bb2 diff --git a/internal/tests/data/madis/metar_2019040918_F000.nc b/internal/tests/data/madis/metar_2019040918_F000.nc new file mode 100644 index 0000000000..e69de29bb2 diff --git a/internal/tests/pytests/conftest.py b/internal/tests/pytests/conftest.py index 9853f78fb4..de6a6f4efa 100644 --- a/internal/tests/pytests/conftest.py +++ b/internal/tests/pytests/conftest.py @@ -206,3 +206,35 @@ def get_test_data_path(subdir): return os.path.join(internal_tests_dir, 'data', subdir) return get_test_data_path + + +@pytest.fixture(scope="function") +def compare_command_and_env_vars(): + def do_comparison(all_commands, expected_cmds, env_var_values, wrapper, + special_values=None): + print(f"ALL COMMANDS: {all_commands}") + assert len(all_commands) == len(expected_cmds) + + missing_env = [item for item in env_var_values + if item not in wrapper.WRAPPER_ENV_VAR_KEYS + and item != 'DIAG_ARG'] + env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env + + for (cmd, env_vars), expected_cmd in zip(all_commands, expected_cmds): + # ensure commands are generated as expected + assert cmd == expected_cmd + + # check that environment variables were set properly + # including deprecated env vars (not in wrapper env var keys) + for env_var_key in env_var_keys: + print(f"ENV VAR: {env_var_key}") + match = next((item for item in env_vars if + item.startswith(env_var_key)), None) + assert match is not None + value = match.split('=', 1)[1] + if special_values is not None and env_var_key in special_values: + assert value == special_values[env_var_key] + else: + assert env_var_values.get(env_var_key, '') == value + + return do_comparison diff --git a/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py b/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py index 341f93112c..4209c2e047 100644 --- a/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py @@ -183,11 +183,30 @@ def test_ascii2nc_missing_inputs(metplus_config, get_test_data_dir, 'type = ["min", "max", "range", "mean", "stdev", "median", "p80"];' 'vld_freq = 0;vld_thresh = 0.5;}')}), + ({'TIME_OFFSET_WARNING': '3'}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;', + 'METPLUS_TIME_SUMMARY_DICT': + ('time_summary = {flag = FALSE;raw_data = FALSE;beg = "000000";' + 'end = "235959";step = 300;width = 600;' + 'grib_code = [11, 204, 211];obs_var = [];' + 'type = ["min", "max", "range", "mean", "stdev", "median", "p80"];' + 'vld_freq = 0;vld_thresh = 0.0;}') + }), + + ({'TIME_OFFSET_WARNING': '3', 'ASCII2NC_TIME_OFFSET_WARNING': '4'}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;', + 'METPLUS_TIME_SUMMARY_DICT': + ('time_summary = {flag = FALSE;raw_data = FALSE;beg = "000000";' + 'end = "235959";step = 300;width = 600;' + 'grib_code = [11, 204, 211];obs_var = [];' + 'type = ["min", "max", "range", "mean", "stdev", "median", "p80"];' + 'vld_freq = 0;vld_thresh = 0.0;}') + }), ] ) @pytest.mark.wrapper -def test_ascii2nc_wrapper(metplus_config, config_overrides, - env_var_values): +def test_ascii2nc_wrapper(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): wrapper = ascii2nc_wrapper(metplus_config, config_overrides) assert wrapper.isOK @@ -216,30 +235,7 @@ def test_ascii2nc_wrapper(metplus_config, config_overrides, f"-config {config_file} {verbosity}"), ] - assert len(all_commands) == len(expected_cmds) - for (cmd, _), expected_cmd in zip(all_commands, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - env_vars = all_commands[0][1] - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - - assert env_var_values.get(env_var_key, '') == value - - output_base = wrapper.config.getdir('OUTPUT_BASE') - if output_base: - shutil.rmtree(output_base) + compare_command_and_env_vars(all_commands, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py b/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py index d767391ba0..1f6e9bc7c8 100644 --- a/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py @@ -637,11 +637,18 @@ def test_ensemble_stat_field_info(metplus_config, config_overrides, 'OBS_ENSEMBLE_STAT_POINT_INPUT_TEMPLATE': obs_point_template}, {}), + ({'ENSEMBLE_STAT_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'ENSEMBLE_STAT_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), + ] ) @pytest.mark.wrapper_c def test_ensemble_stat_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -680,31 +687,12 @@ def test_ensemble_stat_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_c diff --git a/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py b/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py index 3c771f499f..50a09b569a 100644 --- a/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py +++ b/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py @@ -446,12 +446,20 @@ def test_gen_ens_prod_missing_inputs(metplus_config, get_test_data_dir, allow_mi # 68 ({'GEN_ENS_PROD_NORMALIZE': 'CLIMO_STD_ANOM', }, {'METPLUS_NORMALIZE': 'normalize = CLIMO_STD_ANOM;'}), - + # 69 + ({'GEN_ENS_PROD_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + # 70 + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + # 71 + ({'TIME_OFFSET_WARNING': 2, 'GEN_ENS_PROD_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper def test_gen_ens_prod_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -500,28 +508,11 @@ def test_gen_ens_prod_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert(cmd == expected_cmd) - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert(match is not None) - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_ENS_FIELD': - assert (actual_value == ens_fmt) - else: - assert(env_var_values.get(env_var_key, '') == actual_value) + special_values = { + 'METPLUS_ENS_FIELD': ens_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py b/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py index 9971071229..7179109ddc 100644 --- a/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py +++ b/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py @@ -344,10 +344,18 @@ def test_get_config_file(metplus_config): 'censor_thresh = [>12000, <5000];' 'censor_val = [12000, 5000];}' )}), + + ({'GRID_DIAG_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'GRID_DIAG_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper -def test_grid_diag(metplus_config, config_overrides, env_var_values): +def test_grid_diag(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -378,25 +386,8 @@ def test_grid_diag(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_DICT': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_DICT': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) diff --git a/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py b/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py index 40413a3db9..8f18f30384 100644 --- a/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py @@ -21,6 +21,8 @@ time_fmt = '%Y%m%d%H' run_times = ['2005080700', '2005080712'] +ugrid_config_file = '/some/path/UgridConfig_fake' + def set_minimum_config_settings(config): # set config variables to prevent command from running and bypass check @@ -708,12 +710,25 @@ def test_grid_stat_is_prob(metplus_config, config_overrides, expected_values): {'METPLUS_SEEPS_P1_THRESH': 'seeps_p1_thresh = ge0.1&&le0.85;'}), ({'GRID_STAT_CAT_THRESH': 'NA', }, {'METPLUS_CAT_THRESH': 'cat_thresh = [NA];'}), - + ({'GRID_STAT_UGRID_DATASET': 'mpas', }, + {'METPLUS_UGRID_DATASET': 'ugrid_dataset = "mpas";'}), + ({'GRID_STAT_UGRID_MAX_DISTANCE_KM': '30', }, + {'METPLUS_UGRID_MAX_DISTANCE_KM': 'ugrid_max_distance_km = 30;'}), + ({'GRID_STAT_UGRID_COORDINATES_FILE': '/met/test/input/ugrid_data/mpas/static.40962_reduced.nc', }, + {'METPLUS_UGRID_COORDINATES_FILE': 'ugrid_coordinates_file = "/met/test/input/ugrid_data/mpas/static.40962_reduced.nc";'}), + ({'GRID_STAT_UGRID_CONFIG_FILE': ugrid_config_file, }, {}), + + ({'GRID_STAT_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'GRID_STAT_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_b def test_grid_stat_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -726,6 +741,13 @@ def test_grid_stat_single_field(metplus_config, config_overrides, wrapper = GridStatWrapper(config) assert wrapper.isOK + # add extra command line arguments + extra_args = [' '] * len(run_times) + + if 'GRID_STAT_UGRID_CONFIG_FILE' in config_overrides: + for index in range(0, len(run_times)): + extra_args[index] += f'-ugrid_config {ugrid_config_file} ' + app_path = os.path.join(config.getdir('MET_BIN_DIR'), wrapper.app_name) verbosity = f"-v {wrapper.c_dict['VERBOSITY']}" config_file = wrapper.c_dict.get('CONFIG_FILE') @@ -733,39 +755,20 @@ def test_grid_stat_single_field(metplus_config, config_overrides, expected_cmds = [(f"{app_path} {verbosity} " f"{fcst_dir}/2005080700/fcst_file_F012 " f"{obs_dir}/2005080712/obs_file " - f"{config_file} -outdir {out_dir}/2005080712"), + f"{config_file}{extra_args[0]}-outdir {out_dir}/2005080712"), (f"{app_path} {verbosity} " f"{fcst_dir}/2005080712/fcst_file_F012 " f"{obs_dir}/2005080800/obs_file " - f"{config_file} -outdir {out_dir}/2005080800"), + f"{config_file}{extra_args[1]}-outdir {out_dir}/2005080800"), ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_b diff --git a/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py b/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py index 0a114624af..cc49602b90 100644 --- a/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py @@ -230,11 +230,20 @@ def test_ioda2nc_missing_inputs(metplus_config, get_test_data_dir, missing, 'IODA2NC_NMSG': '10', }, {}, ' -iodafile *INPUT_DIR*/other/file.nc -valid_beg 20200309_12 -valid_end 20200310_12 -nmsg 10'), + # 40 + ({'IODA2NC_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}, ''), + # 41 + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}, ''), + # 42 + ({'TIME_OFFSET_WARNING': 2, 'IODA2NC_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}, ''), ] ) @pytest.mark.wrapper def test_ioda2nc_wrapper(metplus_config, config_overrides, - env_var_values, extra_args): + env_var_values, extra_args, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -264,25 +273,7 @@ def test_ioda2nc_wrapper(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py b/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py new file mode 100644 index 0000000000..3f3c2622bd --- /dev/null +++ b/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 + +import pytest + +import os +import shutil + +from metplus.wrappers.madis2nc_wrapper import MADIS2NCWrapper + + +def madis2nc_wrapper(metplus_config, config_overrides=None): + config = metplus_config + overrides = { + 'DO_NOT_RUN_EXE': True, + 'INPUT_MUST_EXIST': False, + 'PROCESS_LIST': 'MADIS2NC', + 'LOOP_BY': 'VALID', + 'VALID_TIME_FMT': '%Y%m%d%H', + 'VALID_BEG': '2019040912', + 'VALID_END': '2019040918', + 'VALID_INCREMENT': '6H', + 'MADIS2NC_INPUT_TEMPLATE': '{INPUT_BASE}/met_test/data/sample_obs/madis/metar_{valid?fmt=%Y%m%d%H}_F000.nc', + 'MADIS2NC_OUTPUT_TEMPLATE': '{OUTPUT_BASE}/madis2nc/metar_{valid?fmt=%Y%m%d%H}.nc', + 'MADIS2NC_CONFIG_FILE': '{PARM_BASE}/met_config/Madis2NcConfig_wrapped', + 'MADIS2NC_TYPE': 'metar', + } + if config_overrides: + for key, value in config_overrides.items(): + overrides[key] = value + + instance = 'overrides' + if not config.has_section(instance): + config.add_section(instance) + for key, value in overrides.items(): + config.set(instance, key, value) + + return MADIS2NCWrapper(config, instance=instance) + + +@pytest.mark.parametrize( + 'missing, run, thresh, errors, allow_missing', [ + (1, 3, 0.5, 0, True), + (1, 3, 0.8, 1, True), + (1, 3, 0.5, 1, False), + ] +) +@pytest.mark.wrapper +def test_madis2nc_missing_inputs(metplus_config, get_test_data_dir, + missing, run, thresh, errors, allow_missing): + config_overrides = { + 'INPUT_MUST_EXIST': True, + 'MADIS2NC_ALLOW_MISSING_INPUTS': allow_missing, + 'MADIS2NC_INPUT_THRESH': thresh, + 'MADIS2NC_INPUT_TEMPLATE': os.path.join(get_test_data_dir('madis'), 'metar_{valid?fmt=%Y%m%d%H}_F000.nc'), + 'VALID_END': '2019041000', + } + wrapper = madis2nc_wrapper(metplus_config, config_overrides) + assert wrapper.isOK + + all_cmds = wrapper.run_all_times() + for cmd, _ in all_cmds: + print(cmd) + + print(f'missing: {wrapper.missing_input_count} / {wrapper.run_count}, errors: {wrapper.errors}') + assert wrapper.missing_input_count == missing + assert wrapper.run_count == run + assert wrapper.errors == errors + + +@pytest.mark.parametrize( + 'config_overrides, env_var_values', [ + ({}, {}), + + ({'MADIS2NC_TIME_SUMMARY_FLAG': 'True'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {flag = TRUE;}'}), + + ({'MADIS2NC_TIME_SUMMARY_RAW_DATA': 'true'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {raw_data = TRUE;}'}), + + ({'MADIS2NC_TIME_SUMMARY_BEG': '123456'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {beg = "123456";}'}), + + ({'MADIS2NC_TIME_SUMMARY_END': '123456'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {end = "123456";}'}), + + ({'MADIS2NC_TIME_SUMMARY_STEP': '500'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {step = 500;}'}), + + ({'MADIS2NC_TIME_SUMMARY_WIDTH': '900'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {width = 900;}'}), + # width as dictionary + ({'MADIS2NC_TIME_SUMMARY_WIDTH': '{ beg = -21600; end = 0; }'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {width = { beg = -21600; end = 0; };}'}), + + ({'MADIS2NC_TIME_SUMMARY_GRIB_CODE': '12, 203, 212'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {grib_code = [12, 203, 212];}'}), + + ({'MADIS2NC_TIME_SUMMARY_OBS_VAR': 'TMP, HGT, PRES'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {obs_var = ["TMP", "HGT", "PRES"];}'}), + + ({'MADIS2NC_TIME_SUMMARY_TYPE': 'min, range, max'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {type = ["min", "range", "max"];}'}), + + ({'MADIS2NC_TIME_SUMMARY_VLD_FREQ': '2'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {vld_freq = 2;}'}), + + ({'MADIS2NC_TIME_SUMMARY_VLD_THRESH': '0.5'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {vld_thresh = 0.5;}'}), + + ({'MADIS2NC_TIME_SUMMARY_FLAG': 'false', + 'MADIS2NC_TIME_SUMMARY_RAW_DATA': 'false', + 'MADIS2NC_TIME_SUMMARY_BEG': '123456', + 'MADIS2NC_TIME_SUMMARY_END': '125634', + 'MADIS2NC_TIME_SUMMARY_STEP': '500', + 'MADIS2NC_TIME_SUMMARY_WIDTH': '900', + 'MADIS2NC_TIME_SUMMARY_GRIB_CODE': '12, 203, 212', + 'MADIS2NC_TIME_SUMMARY_OBS_VAR': 'TMP, HGT, PRES', + 'MADIS2NC_TIME_SUMMARY_TYPE': 'min, range, max', + 'MADIS2NC_TIME_SUMMARY_VLD_FREQ': '2', + 'MADIS2NC_TIME_SUMMARY_VLD_THRESH': '0.5', + }, + {'METPLUS_TIME_SUMMARY_DICT': + ('time_summary = {flag = FALSE;raw_data = FALSE;beg = "123456";' + 'end = "125634";step = 500;width = 900;' + 'grib_code = [12, 203, 212];obs_var = ["TMP", "HGT", "PRES"];' + 'type = ["min", "range", "max"];' + 'vld_freq = 2;vld_thresh = 0.5;}')}), + ({'MADIS2NC_QC_DD': '4,5,6'}, {}), + ({'MADIS2NC_LVL_DIM': 'P500,P750'}, {}), + ({'MADIS2NC_REC_BEG': '2'}, {}), + ({'MADIS2NC_REC_END': '3'}, {}), + ({'MADIS2NC_MASK_GRID': 'mask_grid'}, {}), + ({'MADIS2NC_MASK_POLY': '/some/path/to/mask/poly'}, {}), + ({'MADIS2NC_MASK_SID': 'mask_sid,/some/path/to/mask/sid'}, {}), + ({'MADIS2NC_QC_DD': '4,5,6', + 'MADIS2NC_LVL_DIM': 'P500,P750', + 'MADIS2NC_REC_BEG': '2', + 'MADIS2NC_REC_END': '3', + 'MADIS2NC_MASK_GRID': 'mask_grid', + 'MADIS2NC_MASK_POLY': '/some/path/to/mask/poly', + 'MADIS2NC_MASK_SID': 'mask_sid,/some/path/to/mask/sid'}, {}), + ({'MADIS2NC_TIME_OFFSET_WARNING': '4', }, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), + + ] +) +@pytest.mark.wrapper +def test_madis2nc_wrapper(metplus_config, config_overrides, + env_var_values, compare_command_and_env_vars): + wrapper = madis2nc_wrapper(metplus_config, config_overrides) + assert wrapper.isOK + + input_dir = os.path.dirname(wrapper.config.getraw('config', 'MADIS2NC_INPUT_TEMPLATE')) + input_file1 = 'metar_2019040912_F000.nc' + input_file2 = 'metar_2019040918_F000.nc' + + output_dir = os.path.dirname(wrapper.config.getraw('config', 'MADIS2NC_OUTPUT_TEMPLATE')) + output_file1 = 'metar_2019040912.nc' + output_file2 = 'metar_2019040918.nc' + + all_commands = wrapper.run_all_times() + print(f"ALL COMMANDS: {all_commands}") + + app_path = os.path.join(wrapper.config.getdir('MET_BIN_DIR'), + wrapper.app_name) + verbosity = f"-v {wrapper.c_dict['VERBOSITY']}" + config_file = wrapper.c_dict.get('CONFIG_FILE') + + in_type = config_overrides['MADIS2NC_TYPE'] if 'MADIS2NC_TYPE' in config_overrides else 'metar' + extra_args = '' + for optional_arg in ('qc_dd', 'lvl_dim', 'rec_beg', 'rec_end', 'mask_grid', 'mask_poly', 'mask_sid'): + if f'MADIS2NC_{optional_arg.upper()}' in config_overrides: + extra_args += f" -{optional_arg} {config_overrides[f'MADIS2NC_{optional_arg.upper()}']}" + + expected_cmds = [ + (f"{app_path} {verbosity} {input_dir}/{input_file1} {output_dir}/{output_file1} " + f"-type {in_type} -config {config_file}{extra_args}"), + (f"{app_path} {verbosity} {input_dir}/{input_file2} {output_dir}/{output_file2} " + f"-type {in_type} -config {config_file}{extra_args}"), + ] + + compare_command_and_env_vars(all_commands, expected_cmds, env_var_values, wrapper) + + +@pytest.mark.wrapper +def test_get_config_file(metplus_config): + fake_config_name = '/my/config/file' + config = metplus_config + default_config_file = os.path.join(config.getdir('PARM_BASE'), + 'met_config', + 'Madis2NcConfig_wrapped') + + wrapper = MADIS2NCWrapper(config) + assert wrapper.c_dict['CONFIG_FILE'] == default_config_file + + config.set('config', 'MADIS2NC_CONFIG_FILE', fake_config_name) + wrapper = MADIS2NCWrapper(config) + assert wrapper.c_dict['CONFIG_FILE'] == fake_config_name diff --git a/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py b/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py index f510d524f2..f64953e87a 100644 --- a/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py +++ b/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py @@ -381,10 +381,17 @@ def test_mode_missing_inputs(metplus_config, get_test_data_dir, ({'MODE_OBS_MULTIVAR_LEVEL': 'Z10', }, {'METPLUS_OBS_MULTIVAR_LEVEL': 'multivar_level = "Z10";'}), + ({'MODE_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'MODE_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_a -def test_mode_single_field(metplus_config, config_overrides, env_var_values): +def test_mode_single_field(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config # set config variables needed to run @@ -412,7 +419,6 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") # set default values in expected output list # only if they are not set and if MODE_GRID_RES is set @@ -436,28 +442,12 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): f'{met_name} = {default_val};' ) - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py b/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py index c9267afc76..97c9b6bd6c 100644 --- a/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py +++ b/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py @@ -199,10 +199,18 @@ def test_mtd_missing_inputs(metplus_config, get_test_data_dir, ({'MTD_OUTPUT_PREFIX': 'my_output_prefix'}, {'METPLUS_OUTPUT_PREFIX': 'output_prefix = "my_output_prefix";'}), + + ({'MTD_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'MTD_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper -def test_mode_single_field(metplus_config, config_overrides, env_var_values): +def test_mode_single_field(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config # set config variables needed to run @@ -230,29 +238,12 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py index 4c596f8761..1622bb6c9a 100644 --- a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py @@ -282,10 +282,17 @@ def test_find_input_files(metplus_config, offsets, offset_to_find): ({'PB2NC_VALID_END': valid_end}, {}), ({'PB2NC_VALID_BEGIN': valid_beg, 'PB2NC_VALID_END': valid_end}, {}), + ({'PB2NC_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'PB2NC_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper -def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values): +def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): input_dir = '/some/input/dir' config = metplus_config @@ -342,24 +349,7 @@ def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/plot_point_obs/test_plot_point_obs_wrapper.py b/internal/tests/pytests/wrappers/plot_point_obs/test_plot_point_obs_wrapper.py index e69a324418..337d2d55ed 100644 --- a/internal/tests/pytests/wrappers/plot_point_obs/test_plot_point_obs_wrapper.py +++ b/internal/tests/pytests/wrappers/plot_point_obs/test_plot_point_obs_wrapper.py @@ -248,6 +248,12 @@ def test_plot_point_obs_missing_inputs(metplus_config, get_test_data_dir, ({'PLOT_POINT_OBS_POINT_DATA': point_data_input, }, {'METPLUS_POINT_DATA': f'point_data = {point_data_format}'}), + ({'PLOT_POINT_OBS_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'PLOT_POINT_OBS_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_c diff --git a/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py b/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py index 0d337cb22c..ec11aeaad4 100755 --- a/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py @@ -25,6 +25,8 @@ valid = valid.strftime(time_fmt) valids.append(valid) +ugrid_config_file = '/some/path/UgridConfig_fake' + def set_minimum_config_settings(config): # set config variables to prevent command from running and bypass check @@ -602,6 +604,12 @@ def test_met_dictionary_in_var_options(metplus_config): 'sid = ["one", "two"];' 'llpnt = [{ name = "LAT30TO40"; lat_thresh = >=30&&<=40; lon_thresh = NA; }, { name = "BOX"; lat_thresh = >=20&&<=40; lon_thresh = >=-110&&<=-90; }];}' )}), + ({'POINT_STAT_UGRID_DATASET': 'mpas', }, + {'METPLUS_UGRID_DATASET': 'ugrid_dataset = "mpas";'}), + ({'POINT_STAT_UGRID_MAX_DISTANCE_KM': '30', }, + {'METPLUS_UGRID_MAX_DISTANCE_KM': 'ugrid_max_distance_km = 30;'}), + ({'POINT_STAT_UGRID_COORDINATES_FILE': '/met/test/input/ugrid_data/mpas/static.40962_reduced.nc', }, + {'METPLUS_UGRID_COORDINATES_FILE': 'ugrid_coordinates_file = "/met/test/input/ugrid_data/mpas/static.40962_reduced.nc";'}), # land_mask dictionary ({'POINT_STAT_LAND_MASK_FLAG': 'false', }, {'METPLUS_LAND_MASK_DICT': 'land_mask = {flag = FALSE;}'}), @@ -659,12 +667,21 @@ def test_met_dictionary_in_var_options(metplus_config): {'METPLUS_OBS_SUMMARY': 'obs_summary = NONE;'}), ({'POINT_STAT_OBS_PERC_VALUE': '50', }, {'METPLUS_OBS_PERC_VALUE': 'obs_perc_value = 50;'}), - + ({'POINT_STAT_UGRID_CONFIG_FILE': ugrid_config_file, }, {}), + ({'OBS_POINT_STAT_INPUT_TEMPLATE': '{valid?fmt=%Y%m%d%H}/obs_file,{valid?fmt=%Y%m%d%H}/obs_file2', }, {}), + ({'OBS_POINT_STAT_INPUT_TEMPLATE': '{valid?fmt=%Y%m%d%H}/obs_file,{valid?fmt=%Y%m%d%H}/obs_file2,{valid?fmt=%Y%m%d%H}/obs_file3', }, {}), + + ({'POINT_STAT_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'POINT_STAT_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_a def test_point_stat_all_fields(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): level_no_quotes = '(*,*)' level_with_quotes = f'"{level_no_quotes}"' @@ -733,7 +750,21 @@ def test_point_stat_all_fields(metplus_config, config_overrides, verbosity = f"-v {wrapper.c_dict['VERBOSITY']}" config_file = wrapper.c_dict.get('CONFIG_FILE') out_dir = wrapper.c_dict.get('OUTPUT_DIR') + + # add extra command line arguments extra_args = [' '] * len(inits) + + if 'OBS_POINT_STAT_INPUT_TEMPLATE' in config_overrides: + for index in range(0, len(inits)): + extra_args[index] += f'-point_obs {obs_dir}/{valids[index]}/obs_file2 ' + # if obs_file3 is set, an additional point observation file is added + if 'obs_file3' in config_overrides['OBS_POINT_STAT_INPUT_TEMPLATE']: + extra_args[index] += f'-point_obs {obs_dir}/{valids[index]}/obs_file3 ' + + if 'POINT_STAT_UGRID_CONFIG_FILE' in config_overrides: + for index in range(0, len(inits)): + extra_args[index] += f'-ugrid_config {ugrid_config_file} ' + for beg_end in ('BEG', 'END'): if f'POINT_STAT_OBS_VALID_{beg_end}' in config_overrides: for index in range(0, len(inits)): @@ -748,41 +779,22 @@ def test_point_stat_all_fields(metplus_config, config_overrides, expected_cmds = [] for index in range(0, len(inits)): expected_cmds.append( - f"{app_path} {verbosity}{extra_args[index]}" + f"{app_path} {verbosity} " f"{fcst_dir}/{inits[index]}/fcst_file_F{lead_hour_str} " f"{obs_dir}/{valids[index]}/obs_file " - f"{config_file} -outdir {out_dir}/{valids[index]}" + f"{config_file}{extra_args[index]}-outdir {out_dir}/{valids[index]}" ) - all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - fcst_fmt = f"field = [{','.join(fcst_fmts)}];" obs_fmt = f"field = [{','.join(obs_fmts)}];" - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + all_cmds = wrapper.run_all_times() + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_a diff --git a/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py b/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py index 5589b054a1..dd69b8e4eb 100644 --- a/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py +++ b/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py @@ -424,7 +424,7 @@ def test_series_analysis_missing_inputs(metplus_config, get_test_data_dir, ) @pytest.mark.wrapper_a def test_series_analysis_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -464,7 +464,6 @@ def test_series_analysis_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") expected_len = len(expected_cmds) if 'SERIES_ANALYSIS_GENERATE_PLOTS' in config_overrides: @@ -473,29 +472,15 @@ def test_series_analysis_single_field(metplus_config, config_overrides, expected_len += 4 assert len(all_cmds) == expected_len - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - elif env_var_key == 'METPLUS_OUTPUT_STATS_DICT' and 'METPLUS_OUTPUT_STATS_DICT' not in env_var_values: - assert actual_value == stat_list_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + if 'METPLUS_OUTPUT_STATS_DICT' not in env_var_values: + special_values['METPLUS_OUTPUT_STATS_DICT'] = stat_list_fmt + # only compare first command since the rest are not series_analysis + compare_command_and_env_vars(all_cmds[0:1], expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_a diff --git a/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py b/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py index 9da87d0faa..6d0f619ec7 100644 --- a/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py +++ b/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py @@ -18,6 +18,7 @@ JOB_ARGS = '-job filter' + def stat_analysis_wrapper(metplus_config): """! Returns a default StatAnalysisWrapper with /path/to entries in the metplus_system.conf and metplus_runtime.conf configuration @@ -172,6 +173,15 @@ def set_minimum_config_settings(config): 'MODEL1': '{custom}', 'MODEL_LIST': '{custom}'}, {'METPLUS_MODEL': 'model = ["CUSTOM_MODEL"];'}), + # 17 time_offset_warning app + ({'STAT_ANALYSIS_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + # 18 time_offset_warning generic + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + # 19 time_offset_warning both + ({'TIME_OFFSET_WARNING': 2, 'STAT_ANALYSIS_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_d diff --git a/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py b/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py index a2c2635aec..88b9ef1eb5 100644 --- a/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py @@ -61,7 +61,6 @@ def set_minimum_config_settings(config): config.set('config', 'BOTH_VAR2_LEVELS', 'P1000, P900, P800, P700, P500, P100') - @pytest.mark.parametrize( 'missing, run, thresh, errors, allow_missing', [ (1, 3, 0.5, 0, True), @@ -239,11 +238,18 @@ def test_tc_diag_missing_inputs(metplus_config, get_test_data_dir, ({'TC_DIAG_OUTPUT_BASE_FORMAT': 's{storm_id}_{technique}_doper_{init_ymdh}', }, {'METPLUS_OUTPUT_BASE_FORMAT': 'output_base_format = "s{storm_id}_{technique}_doper_{init_ymdh}";'}), ({'TC_DIAG_ONE_TIME_PER_FILE_FLAG': 'false', }, {'METPLUS_ONE_TIME_PER_FILE_FLAG': 'one_time_per_file_flag = FALSE;'}), + + ({'TC_DIAG_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'TC_DIAG_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper def test_tc_diag_run(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -277,28 +283,11 @@ def test_tc_diag_run(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_FIELD': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_FIELD': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py b/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py index c6fe70f2a7..c9beca4fde 100644 --- a/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py @@ -339,10 +339,17 @@ def test_tc_gen_missing_inputs(metplus_config, get_test_data_dir, allow_missing, }, {'METPLUS_OPS_HIT_WINDOW_DICT': 'ops_hit_window = {beg = 1;end = 47;}'}), + ({'TC_GEN_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'TC_GEN_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_a -def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, env_var_values): +def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, + env_var_values, compare_command_and_env_vars): # expected number of 2016 files (including file_list line) expected_genesis_count = 7 expected_track_count = expected_genesis_count @@ -434,25 +441,7 @@ def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, env_var_val ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) # verify file count of genesis, edeck, shape, and track file list files with open(genesis_path, 'r') as file_handle: diff --git a/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py b/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py index 354394da65..70adbd62d5 100644 --- a/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py @@ -606,11 +606,18 @@ def test_tc_pairs_storm_id_lists(metplus_config, get_test_data_dir, config_overr 'TC_PAIRS_DIAG_SOURCE1': 'TCDIAG', }, {'DIAG_ARG': '-diag TCDIAG /bmlq2014123118.gfso.0104',}), + + ('VALID', {'TC_PAIRS_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ('VALID', {'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ('VALID', {'TIME_OFFSET_WARNING': 2, 'TC_PAIRS_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper def test_tc_pairs_run(metplus_config, get_test_data_dir, loop_by, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config remove_beg = remove_end = remove_match_points = False @@ -677,26 +684,7 @@ def test_tc_pairs_run(metplus_config, get_test_data_dir, loop_by, config_overrid ) all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS - and item != 'DIAG_ARG'] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) if remove_beg: del env_var_values[f'METPLUS_{loop_by}_BEG'] diff --git a/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py b/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py index 12bcc44db5..0f52648e6b 100644 --- a/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py @@ -242,10 +242,17 @@ def test_tc_stat_handle_jobs(metplus_config, config_overrides, expected_dirs, ({'TC_STAT_OUTPUT_TEMPLATE': 'tc_stat.out.nc', }, {}), + ({'TC_STAT_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'TC_STAT_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper -def test_tc_stat_run(metplus_config, config_overrides, env_var_values): +def test_tc_stat_run(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = get_config(metplus_config) # set config variable overrides @@ -286,25 +293,7 @@ def test_tc_stat_run(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py b/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py index 3556a709ed..9961b78277 100644 --- a/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py +++ b/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py @@ -125,11 +125,17 @@ def set_minimum_config_settings(config): ({'TC_RMW_RMW_SCALE': '15', }, {'METPLUS_RMW_SCALE': 'rmw_scale = 15.0;'}), + ({'TC_RMW_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'TC_RMW_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper def test_tc_rmw_run(metplus_config, get_test_data_dir, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -165,28 +171,11 @@ def test_tc_rmw_run(metplus_config, get_test_data_dir, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_FIELD': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_FIELD': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py b/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py index 5b98680136..52e4528ad2 100644 --- a/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py +++ b/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py @@ -313,10 +313,17 @@ def test_wavelet_stat_is_prob(metplus_config, config_overrides, expected_values) ({'OBS_WAVELET_STAT_FILE_TYPE': 'NETCDF_NCCF', }, {'METPLUS_OBS_FILE_TYPE': 'file_type = NETCDF_NCCF;'}), + ({'WAVELET_STAT_TIME_OFFSET_WARNING': 3}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 3;'}), + ({'TIME_OFFSET_WARNING': 2}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 2;'}), + ({'TIME_OFFSET_WARNING': 2, 'WAVELET_STAT_TIME_OFFSET_WARNING': 4}, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), ] ) @pytest.mark.wrapper_b -def test_wavelet_stat_single_field(metplus_config, config_overrides, env_var_values): +def test_wavelet_stat_single_field(metplus_config, config_overrides, + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -343,31 +350,12 @@ def test_wavelet_stat_single_field(metplus_config, config_overrides, env_var_val ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_b diff --git a/internal/tests/use_cases/all_use_cases.txt b/internal/tests/use_cases/all_use_cases.txt index daef1a124e..38c743911a 100644 --- a/internal/tests/use_cases/all_use_cases.txt +++ b/internal/tests/use_cases/all_use_cases.txt @@ -63,6 +63,7 @@ Category: met_tool_wrapper 61::PlotPointObs:: met_tool_wrapper/PlotPointObs/PlotPointObs.conf 62::TCDiag:: met_tool_wrapper/TCDiag/TCDiag.conf 63::WaveletStat:: met_tool_wrapper/WaveletStat/WaveletStat.conf +64::MADIS2NC:: met_tool_wrapper/MADIS2NC/MADIS2NC.conf Category: air_quality_and_comp 0::EnsembleStat_fcstICAP_obsMODIS_aod::model_applications/air_quality_and_comp/EnsembleStat_fcstICAP_obsMODIS_aod.conf diff --git a/metplus/VERSION b/metplus/VERSION index c1929f62be..a2514f36ac 100644 --- a/metplus/VERSION +++ b/metplus/VERSION @@ -1 +1 @@ -6.0.0-beta4-dev +6.0.0-beta5-dev diff --git a/metplus/util/constants.py b/metplus/util/constants.py index 72a316281f..3208aa2a26 100644 --- a/metplus/util/constants.py +++ b/metplus/util/constants.py @@ -20,7 +20,7 @@ 'griddiag': 'GridDiag', 'gridstat': 'GridStat', 'ioda2nc': 'IODA2NC', - 'makeplots': 'MakePlots', + 'madis2nc': 'MADIS2NC', 'metdbload': 'METDbLoad', 'mode': 'MODE', 'mtd': 'MTD', diff --git a/metplus/wrappers/command_builder.py b/metplus/wrappers/command_builder.py index 8886408857..cc5d6aedc9 100755 --- a/metplus/wrappers/command_builder.py +++ b/metplus/wrappers/command_builder.py @@ -103,6 +103,16 @@ def __init__(self, config, instance=None): # add key to list of env vars to set self.env_var_keys.append(self.MET_OVERRIDES_KEY) + # add time_offset_warning env var + self.env_var_keys.append('METPLUS_TIME_OFFSET_WARNING') + time_offset_warning = self.get_wrapper_or_generic_config( + 'TIME_OFFSET_WARNING', var_type='int' + ) + if time_offset_warning != '': + self.env_var_dict['METPLUS_TIME_OFFSET_WARNING'] = ( + f'time_offset_warning = {time_offset_warning};' + ) + # warn if any environment variables set by the wrapper are not # being utilized in the user's config file self.check_for_unused_env_vars() @@ -1566,15 +1576,20 @@ def handle_time_summary_dict(self): 'step': 'int', 'width': ('string', 'remove_quotes'), 'grib_code': ('list', 'remove_quotes,allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_GRIB_CODES']), + [f'{app_upper}_TIME_SUMMARY_GRIB_CODE', + f'{app_upper}_TIME_SUMMARY_GRIB_CODES']), 'obs_var': ('list', 'allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_VAR_NAMES']), + [f'{app_upper}_TIME_SUMMARY_OBS_VAR', + f'{app_upper}_TIME_SUMMARY_VAR_NAMES']), 'type': ('list', 'allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_TYPES']), + [f'{app_upper}_TIME_SUMMARY_TYPE', + f'{app_upper}_TIME_SUMMARY_TYPES']), 'vld_freq': ('int', None, None, - [f'{app_upper}_TIME_SUMMARY_VALID_FREQ']), + [f'{app_upper}_TIME_SUMMARY_VLD_FREQ', + f'{app_upper}_TIME_SUMMARY_VALID_FREQ']), 'vld_thresh': ('float', None, None, - [f'{app_upper}_TIME_SUMMARY_VALID_THRESH']), + [f'{app_upper}_TIME_SUMMARY_VLD_THRESH', + f'{app_upper}_TIME_SUMMARY_VALID_THRESH']), }) def handle_mask(self, single_value=False, get_flags=False, get_point=False): diff --git a/metplus/wrappers/compare_gridded_wrapper.py b/metplus/wrappers/compare_gridded_wrapper.py index 82a7a2f7e1..e606be4bb3 100755 --- a/metplus/wrappers/compare_gridded_wrapper.py +++ b/metplus/wrappers/compare_gridded_wrapper.py @@ -267,7 +267,11 @@ def set_command_line_arguments(self, time_info): @param time_info dictionary with time information """ - return None + # add ugrid config file if requested + if self.c_dict.get('UGRID_CONFIG_FILE'): + ugrid_config = self.c_dict['UGRID_CONFIG_FILE'] + ugrid_config = do_string_sub(ugrid_config, **time_info) + self.args.append(f'-ugrid_config {ugrid_config}') def get_command(self): """! Builds the command to run the MET application @@ -279,8 +283,6 @@ def get_command(self): return None cmd = '{} -v {} '.format(self.app_path, self.c_dict['VERBOSITY']) - for arg in self.args: - cmd += arg + " " if len(self.infiles) == 0: self.log_error("No input filenames specified") @@ -304,6 +306,9 @@ def get_command(self): cmd += self.param + ' ' + for arg in self.args: + cmd += arg + " " + if self.outdir == "": self.log_error("No output directory specified") return None diff --git a/metplus/wrappers/grid_stat_wrapper.py b/metplus/wrappers/grid_stat_wrapper.py index afc32ccbbd..11b5e7dbae 100755 --- a/metplus/wrappers/grid_stat_wrapper.py +++ b/metplus/wrappers/grid_stat_wrapper.py @@ -56,6 +56,9 @@ class GridStatWrapper(CompareGriddedWrapper): 'METPLUS_CENSOR_VAL', 'METPLUS_SEEPS_P1_THRESH', 'METPLUS_CAT_THRESH', + 'METPLUS_UGRID_DATASET', + 'METPLUS_UGRID_MAX_DISTANCE_KM', + 'METPLUS_UGRID_COORDINATES_FILE', ] # deprecated env vars that are no longer supported in the wrapped MET conf @@ -131,6 +134,11 @@ def create_c_dict(self): # get the MET config file path or use default c_dict['CONFIG_FILE'] = self.get_config_file('GridStatConfig_wrapped') + # get optional ugrid config file if requested + c_dict['UGRID_CONFIG_FILE'] = ( + self.config.getraw('config', 'GRID_STAT_UGRID_CONFIG_FILE') + ) + self.get_input_templates(c_dict, { 'FCST': {'prefix': 'FCST_GRID_STAT', 'required': True}, 'OBS': {'prefix': 'OBS_GRID_STAT', 'required': True}, @@ -212,6 +220,10 @@ def create_c_dict(self): self.add_met_config(name='nc_pairs_var_name', data_type='string', metplus_configs=['GRID_STAT_NC_PAIRS_VAR_NAME']) + self.add_met_config(name='ugrid_dataset', data_type='string') + self.add_met_config(name='ugrid_max_distance_km', data_type='int') + self.add_met_config(name='ugrid_coordinates_file', data_type='string') + self.add_met_config(name='grid_weight_flag', data_type='string', metplus_configs=['GRID_STAT_GRID_WEIGHT_FLAG'], extra_args={'remove_quotes': True, diff --git a/metplus/wrappers/madis2nc_wrapper.py b/metplus/wrappers/madis2nc_wrapper.py new file mode 100755 index 0000000000..420a5d4f22 --- /dev/null +++ b/metplus/wrappers/madis2nc_wrapper.py @@ -0,0 +1,127 @@ +""" +Program Name: madis2nc_wrapper.py +Contact(s): George McCabe +Abstract: Builds command for and runs madis2nc +History Log: Initial version +Usage: +Parameters: None +Input Files: MADIS files +Output Files: nc files +Condition codes: 0 for success, 1 for failure +""" + +import os + +from ..util import do_string_sub, MISSING_DATA_VALUE +from . import RuntimeFreqWrapper + +'''!@namespace MADIS2NCWrapper +@brief Wraps the madis2nc tool to reformat MADIS format to NetCDF +@endcode +''' + + +class MADIS2NCWrapper(RuntimeFreqWrapper): + + RUNTIME_FREQ_DEFAULT = 'RUN_ONCE_FOR_EACH' + RUNTIME_FREQ_SUPPORTED = 'ALL' + + WRAPPER_ENV_VAR_KEYS = [ + 'METPLUS_TIME_SUMMARY_DICT', + ] + + def __init__(self, config, instance=None): + self.app_name = "madis2nc" + self.app_path = os.path.join(config.getdir('MET_BIN_DIR', ''), + self.app_name) + super().__init__(config, instance=instance) + + def create_c_dict(self): + c_dict = super().create_c_dict() + c_dict['VERBOSITY'] = self.config.getstr('config', + 'LOG_MADIS2NC_VERBOSITY', + c_dict['VERBOSITY']) + + # file I/O + c_dict['ALLOW_MULTIPLE_FILES'] = True + self.get_input_templates(c_dict, { + 'OBS': {'prefix': 'MADIS2NC', 'required': True}, + }) + + c_dict['OUTPUT_DIR'] = self.config.getdir('MADIS2NC_OUTPUT_DIR', '') + c_dict['OUTPUT_TEMPLATE'] = ( + self.config.getraw('config', 'MADIS2NC_OUTPUT_TEMPLATE') + ) + if not c_dict['OUTPUT_TEMPLATE']: + self.log_error('MADIS2NC_OUTPUT_TEMPLATE must be set') + + # config file settings + c_dict['CONFIG_FILE'] = self.get_config_file('Madis2NcConfig_wrapped') + self.handle_time_summary_dict() + + # command line settings + c_dict['TYPE'] = self.config.getraw('config', 'MADIS2NC_TYPE') + if not c_dict['TYPE']: + self.log_error('Must set MADIS2NC_TYPE') + + c_dict['QC_DD'] = self.config.getraw('config', 'MADIS2NC_QC_DD') + c_dict['LVL_DIM'] = self.config.getraw('config', 'MADIS2NC_LVL_DIM') + c_dict['REC_BEG'] = self.config.getint('config', 'MADIS2NC_REC_BEG') + if c_dict['REC_BEG'] == MISSING_DATA_VALUE: + c_dict['REC_BEG'] = '' + c_dict['REC_END'] = self.config.getint('config', 'MADIS2NC_REC_END') + if c_dict['REC_END'] == MISSING_DATA_VALUE: + c_dict['REC_END'] = '' + c_dict['MASK_GRID'] = self.config.getraw('config', 'MADIS2NC_MASK_GRID') + c_dict['MASK_POLY'] = self.config.getraw('config', 'MADIS2NC_MASK_POLY') + c_dict['MASK_SID'] = self.config.getraw('config', 'MADIS2NC_MASK_SID') + + return c_dict + + def get_command(self): + """!Build command to run madis2nc + + @returns str: madis2nc command + """ + return (f"{self.app_path} -v {self.c_dict['VERBOSITY']}" + f" {' '.join(self.infiles)} {self.get_output_path()}" + f" {' '.join(self.args)}") + + def find_input_files(self, time_info): + """!Get list of input files to pass to command. Sets self.infiles. + + @param time_info dictionary containing time information + @returns bool: True if files were found, False otherwise + """ + if not self.c_dict.get('ALL_FILES'): + return False + + input_files = self.c_dict['ALL_FILES'][0].get('OBS', []) + if not input_files: + return False + + self.logger.debug(f"Adding input: {' and '.join(input_files)}") + self.infiles.extend(input_files) + return True + + def set_command_line_arguments(self, time_info): + """!Read self.c_dict and set command line arguments in self.args. + + @param time_info dictionary containing time information + """ + # set required command line arguments + val = do_string_sub(self.c_dict['TYPE'], **time_info) + self.args.append(f"-type {val}") + + config_file = do_string_sub(self.c_dict['CONFIG_FILE'], **time_info) + self.args.append(f"-config {config_file}") + + # set optional command line arguments if specified + optional_args = ('qc_dd', 'lvl_dim', 'rec_beg', 'rec_end', 'mask_grid', + 'mask_poly', 'mask_sid') + for arg in optional_args: + if self.c_dict[arg.upper()]: + val = self.c_dict[arg.upper()] + if isinstance(val, str): + val = do_string_sub(val, **time_info) + self.args.append(f"-{arg} {val}") diff --git a/metplus/wrappers/point_stat_wrapper.py b/metplus/wrappers/point_stat_wrapper.py index e88dd80d9b..2ff73c601d 100755 --- a/metplus/wrappers/point_stat_wrapper.py +++ b/metplus/wrappers/point_stat_wrapper.py @@ -51,6 +51,9 @@ class PointStatWrapper(CompareGriddedWrapper): 'METPLUS_FCST_FILE_TYPE', 'METPLUS_OBS_FILE_TYPE', 'METPLUS_SEEPS_P1_THRESH', + 'METPLUS_UGRID_DATASET', + 'METPLUS_UGRID_MAX_DISTANCE_KM', + 'METPLUS_UGRID_COORDINATES_FILE', ] # deprecated env vars that are no longer supported in the wrapped MET conf @@ -123,13 +126,10 @@ def create_c_dict(self): c_dict['OFFSETS'] = getlistint( self.config.getstr('config', 'POINT_STAT_OFFSETS', '0') ) - c_dict['FCST_INPUT_TEMPLATE'] = ( - self.config.getraw('config', 'FCST_POINT_STAT_INPUT_TEMPLATE', '') - ) - - c_dict['OBS_INPUT_TEMPLATE'] = ( - self.config.getraw('config', 'OBS_POINT_STAT_INPUT_TEMPLATE', '') - ) + self.get_input_templates(c_dict, { + 'FCST': {'prefix': 'FCST_POINT_STAT', 'required': True}, + 'OBS': {'prefix': 'OBS_POINT_STAT', 'required': True}, + }) c_dict['FCST_INPUT_DATATYPE'] = ( self.config.getstr('config', 'FCST_POINT_STAT_INPUT_DATATYPE', '') @@ -138,14 +138,6 @@ def create_c_dict(self): self.config.getstr('config', 'OBS_POINT_STAT_INPUT_DATATYPE', '') ) - c_dict['FCST_INPUT_DIR'] = ( - self.config.getdir('FCST_POINT_STAT_INPUT_DIR', '') - ) - - c_dict['OBS_INPUT_DIR'] = ( - self.config.getdir('OBS_POINT_STAT_INPUT_DIR', '') - ) - c_dict['OUTPUT_DIR'] = ( self.config.getdir('POINT_STAT_OUTPUT_DIR', '') ) @@ -160,6 +152,11 @@ def create_c_dict(self): # get the MET config file path or use default c_dict['CONFIG_FILE'] = self.get_config_file('PointStatConfig_wrapped') + # get optional ugrid config file if requested + c_dict['UGRID_CONFIG_FILE'] = ( + self.config.getraw('config', 'POINT_STAT_UGRID_CONFIG_FILE') + ) + self.add_met_config_window('obs_window') self.handle_mask(get_point=True) @@ -281,6 +278,10 @@ def create_c_dict(self): self.add_met_config(name='seeps_p1_thresh', data_type='string', extra_args={'remove_quotes': True}) + self.add_met_config(name='ugrid_dataset', data_type='string') + self.add_met_config(name='ugrid_max_distance_km', data_type='int') + self.add_met_config(name='ugrid_coordinates_file', data_type='string') + if not c_dict['FCST_INPUT_TEMPLATE']: self.log_error('Must set FCST_POINT_STAT_INPUT_TEMPLATE ' 'in config file') @@ -301,9 +302,64 @@ def set_command_line_arguments(self, time_info): @param time_info dictionary with time information """ + # call CompareGridded function + super().set_command_line_arguments(time_info) + # set optional obs_valid_beg and obs_valid_end arguments for ext in ['BEG', 'END']: if self.c_dict[f'OBS_VALID_{ext}']: obs_valid = do_string_sub(self.c_dict[f'OBS_VALID_{ext}'], **time_info) self.args.append(f"-obs_valid_{ext.lower()} {obs_valid}") + + def find_input_files(self, time_info): + # get model from first var to compare + model_path = self.find_model(time_info, + mandatory=True, + return_list=True) + if not model_path: + return False + + # if there is more than 1 file, create file list file + if len(model_path) > 1: + self.logger.warning('Multiple forecast files found.' + 'Using the first one') + + self.infiles.append(model_path[0]) + + # get observation to from first var compare + obs_path, time_info = self.find_obs_offset(time_info, + mandatory=True, + return_list=True) + if obs_path is None: + return False + + # add observation files found individually to use -point_obs argument + self.infiles.extend(obs_path) + + return True + + def get_command(self): + """! Builds the command to run point_stat + @rtype string + @return Returns a point_stat command with arguments that you can run + """ + fcst_file, *obs_files = self.infiles + if fcst_file.startswith('PYTHON'): + fcst_file = f"'{fcst_file}'" + + obs_file = obs_files[0] + if obs_file.startswith('PYTHON'): + obs_file = f"'{obs_file}'" + + cmd = (f"{self.app_path} -v {self.c_dict['VERBOSITY']} " + f"{fcst_file} {obs_file} {self.param}") + + if len(obs_files) > 1: + cmd += ' -point_obs ' + ' -point_obs '.join(obs_files[1:]) + + for arg in self.args: + cmd += f' {arg}' + + cmd += f' -outdir {self.outdir}' + return cmd diff --git a/metplus/wrappers/runtime_freq_wrapper.py b/metplus/wrappers/runtime_freq_wrapper.py index 03a6f1a31e..67f5f37638 100755 --- a/metplus/wrappers/runtime_freq_wrapper.py +++ b/metplus/wrappers/runtime_freq_wrapper.py @@ -131,9 +131,7 @@ def get_input_templates(self, c_dict, input_info=None): templates = getlist( self.config.getraw('config', f'{prefix}_INPUT_TEMPLATE') ) - template_list = [os.path.join(input_dir, template) - for template in templates] - template = ','.join(template_list) + template = ','.join(templates) c_dict[f'{label}_INPUT_TEMPLATE'] = template if not c_dict[f'{label}_INPUT_TEMPLATE']: if required: diff --git a/parm/met_config/Ascii2NcConfig_wrapped b/parm/met_config/Ascii2NcConfig_wrapped index 4233450615..e86116277b 100644 --- a/parm/met_config/Ascii2NcConfig_wrapped +++ b/parm/met_config/Ascii2NcConfig_wrapped @@ -38,4 +38,5 @@ message_type_map = [ tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/EnsembleStatConfig_wrapped b/parm/met_config/EnsembleStatConfig_wrapped index cbbbce57dd..c01f807c6b 100644 --- a/parm/met_config/EnsembleStatConfig_wrapped +++ b/parm/met_config/EnsembleStatConfig_wrapped @@ -237,4 +237,5 @@ ${METPLUS_OUTPUT_PREFIX} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/GenEnsProdConfig_wrapped b/parm/met_config/GenEnsProdConfig_wrapped index 9e8d21275c..223920e7c8 100644 --- a/parm/met_config/GenEnsProdConfig_wrapped +++ b/parm/met_config/GenEnsProdConfig_wrapped @@ -115,4 +115,5 @@ ${METPLUS_ENSEMBLE_FLAG_DICT} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/GridDiagConfig_wrapped b/parm/met_config/GridDiagConfig_wrapped index 691338acf2..8c5117ddbe 100644 --- a/parm/met_config/GridDiagConfig_wrapped +++ b/parm/met_config/GridDiagConfig_wrapped @@ -41,4 +41,5 @@ ${METPLUS_MASK_DICT} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/GridStatConfig_wrapped b/parm/met_config/GridStatConfig_wrapped index 4bb4636f17..11259f108d 100644 --- a/parm/met_config/GridStatConfig_wrapped +++ b/parm/met_config/GridStatConfig_wrapped @@ -160,6 +160,13 @@ gradient = { //distance_map = { ${METPLUS_DISTANCE_MAP_DICT} + +//////////////////////////////////////////////////////////////////////////////// +// Threshold for SEEPS p1 (Probability of being dry) + +//seeps_p1_thresh = +${METPLUS_SEEPS_P1_THRESH} + //////////////////////////////////////////////////////////////////////////////// // @@ -176,10 +183,15 @@ ${METPLUS_OUTPUT_FLAG_DICT} ${METPLUS_NC_PAIRS_FLAG_DICT} //////////////////////////////////////////////////////////////////////////////// -// Threshold for SEEPS p1 (Probability of being dry) -//seeps_p1_thresh = -${METPLUS_SEEPS_P1_THRESH} +//ugrid_dataset = +${METPLUS_UGRID_DATASET} + +//ugrid_max_distance_km = +${METPLUS_UGRID_MAX_DISTANCE_KM} + +//ugrid_coordinates_file = +${METPLUS_UGRID_COORDINATES_FILE} //////////////////////////////////////////////////////////////////////////////// @@ -193,4 +205,5 @@ ${METPLUS_OUTPUT_PREFIX} //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/IODA2NCConfig_wrapped b/parm/met_config/IODA2NCConfig_wrapped index ba1faa1695..7f25fb8cf1 100644 --- a/parm/met_config/IODA2NCConfig_wrapped +++ b/parm/met_config/IODA2NCConfig_wrapped @@ -114,4 +114,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/MODEConfig_wrapped b/parm/met_config/MODEConfig_wrapped index ff1ac480f9..54af9bcf4b 100644 --- a/parm/met_config/MODEConfig_wrapped +++ b/parm/met_config/MODEConfig_wrapped @@ -277,4 +277,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/MTDConfig_wrapped b/parm/met_config/MTDConfig_wrapped index d521670eac..f010982346 100644 --- a/parm/met_config/MTDConfig_wrapped +++ b/parm/met_config/MTDConfig_wrapped @@ -257,4 +257,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/Madis2NcConfig_wrapped b/parm/met_config/Madis2NcConfig_wrapped new file mode 100644 index 0000000000..282f6782f1 --- /dev/null +++ b/parm/met_config/Madis2NcConfig_wrapped @@ -0,0 +1,30 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// MADIS2NC configuration file. +// +// For additional information, please see the MET Users Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// The parameters listed below are used to summarize the MADIS data read in +// + +// +// Time periods for the summarization +// obs_var (string array) is added and works like grib_code (int array) +// when the obs name is given instead of grib_code +// + +//time_summary = { +${METPLUS_TIME_SUMMARY_DICT} + +// +// Indicate a version number for the contents of this configuration file. +// The value should generally not be modified. +// +//version = "V12.0.0"; + +${METPLUS_TIME_OFFSET_WARNING} + +${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/PB2NCConfig_wrapped b/parm/met_config/PB2NCConfig_wrapped index 8b9057923b..6174608bb9 100644 --- a/parm/met_config/PB2NCConfig_wrapped +++ b/parm/met_config/PB2NCConfig_wrapped @@ -129,4 +129,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/PlotPointObsConfig_wrapped b/parm/met_config/PlotPointObsConfig_wrapped index 2194f01fee..8fd113b86d 100644 --- a/parm/met_config/PlotPointObsConfig_wrapped +++ b/parm/met_config/PlotPointObsConfig_wrapped @@ -96,4 +96,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/PointStatConfig_wrapped b/parm/met_config/PointStatConfig_wrapped index b551a4f2fc..b7f1dc9334 100644 --- a/parm/met_config/PointStatConfig_wrapped +++ b/parm/met_config/PointStatConfig_wrapped @@ -177,6 +177,12 @@ ${METPLUS_INTERP_DICT} //hira = { ${METPLUS_HIRA_DICT} +//////////////////////////////////////////////////////////////////////////////// +// Threshold for SEEPS p1 (Probability of being dry) + +//seeps_p1_thresh = +${METPLUS_SEEPS_P1_THRESH} + //////////////////////////////////////////////////////////////////////////////// // @@ -186,10 +192,15 @@ ${METPLUS_HIRA_DICT} ${METPLUS_OUTPUT_FLAG_DICT} //////////////////////////////////////////////////////////////////////////////// -// Threshold for SEEPS p1 (Probability of being dry) -//seeps_p1_thresh = -${METPLUS_SEEPS_P1_THRESH} +//ugrid_dataset = +${METPLUS_UGRID_DATASET} + +//ugrid_max_distance_km = +${METPLUS_UGRID_MAX_DISTANCE_KM} + +//ugrid_coordinates_file = +${METPLUS_UGRID_COORDINATES_FILE} //////////////////////////////////////////////////////////////////////////////// @@ -201,4 +212,5 @@ ${METPLUS_OUTPUT_PREFIX} //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/STATAnalysisConfig_wrapped b/parm/met_config/STATAnalysisConfig_wrapped index d38c8def2d..e3e32deba4 100644 --- a/parm/met_config/STATAnalysisConfig_wrapped +++ b/parm/met_config/STATAnalysisConfig_wrapped @@ -164,4 +164,5 @@ tmp_dir = "${MET_TMP_DIR}"; //version = "V10.0"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/SeriesAnalysisConfig_wrapped b/parm/met_config/SeriesAnalysisConfig_wrapped index a7ccba88b0..4836623043 100644 --- a/parm/met_config/SeriesAnalysisConfig_wrapped +++ b/parm/met_config/SeriesAnalysisConfig_wrapped @@ -129,4 +129,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/TCDiagConfig_wrapped b/parm/met_config/TCDiagConfig_wrapped index 8dacb81b26..0407e982c5 100644 --- a/parm/met_config/TCDiagConfig_wrapped +++ b/parm/met_config/TCDiagConfig_wrapped @@ -180,4 +180,5 @@ tmp_dir = "${MET_TMP_DIR}"; //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/TCGenConfig_wrapped b/parm/met_config/TCGenConfig_wrapped index c18f895f8b..7d813aa105 100644 --- a/parm/met_config/TCGenConfig_wrapped +++ b/parm/met_config/TCGenConfig_wrapped @@ -294,4 +294,5 @@ ${METPLUS_NC_PAIRS_GRID} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/TCPairsConfig_wrapped b/parm/met_config/TCPairsConfig_wrapped index 93947c0c18..7b9aa04651 100644 --- a/parm/met_config/TCPairsConfig_wrapped +++ b/parm/met_config/TCPairsConfig_wrapped @@ -168,4 +168,5 @@ ${METPLUS_DIAG_CONVERT_MAP_LIST} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/TCRMWConfig_wrapped b/parm/met_config/TCRMWConfig_wrapped index 49a8b0bdab..0cd32cee3a 100644 --- a/parm/met_config/TCRMWConfig_wrapped +++ b/parm/met_config/TCRMWConfig_wrapped @@ -91,4 +91,5 @@ ${METPLUS_RMW_SCALE} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/TCStatConfig_wrapped b/parm/met_config/TCStatConfig_wrapped index dc4d5ec780..a6bd4afb23 100644 --- a/parm/met_config/TCStatConfig_wrapped +++ b/parm/met_config/TCStatConfig_wrapped @@ -233,4 +233,5 @@ ${METPLUS_JOBS} tmp_dir = "${MET_TMP_DIR}"; +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/met_config/WaveletStatConfig_wrapped b/parm/met_config/WaveletStatConfig_wrapped index 9f3f0e17f8..9114b6b810 100644 --- a/parm/met_config/WaveletStatConfig_wrapped +++ b/parm/met_config/WaveletStatConfig_wrapped @@ -125,4 +125,5 @@ ${METPLUS_OUTPUT_PREFIX} //////////////////////////////////////////////////////////////////////////////// +${METPLUS_TIME_OFFSET_WARNING} ${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC.conf b/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC.conf index bf7481144b..778ea8ae76 100644 --- a/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC.conf +++ b/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC.conf @@ -76,3 +76,5 @@ ASCII2NC_TIME_SUMMARY_VAR_NAMES = ASCII2NC_TIME_SUMMARY_TYPES = min, max, range, mean, stdev, median, p80 ASCII2NC_TIME_SUMMARY_VALID_FREQ = 0 ASCII2NC_TIME_SUMMARY_VALID_THRESH = 0.0 + +#ASCII2NC_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat.conf b/parm/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat.conf index da1c9d5e6d..92c9ef6a18 100644 --- a/parm/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat.conf +++ b/parm/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat.conf @@ -205,3 +205,5 @@ ENSEMBLE_STAT_NC_ORANK_FLAG_WEIGHT = FALSE #ENSEMBLE_STAT_CONTROL_ID = #ENSEMBLE_STAT_GRID_WEIGHT_FLAG = + +#ENSEMBLE_STAT_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf b/parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf index 55ff515571..2d2394d374 100644 --- a/parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf +++ b/parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf @@ -165,3 +165,5 @@ GEN_ENS_PROD_ENS_THRESH = 0.8 #GEN_ENS_PROD_ENS_MEMBER_IDS = #GEN_ENS_PROD_CONTROL_ID = + +#GEN_ENS_PROD_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/GridDiag/GridDiag.conf b/parm/use_cases/met_tool_wrapper/GridDiag/GridDiag.conf index 4104ee6016..23813198fa 100644 --- a/parm/use_cases/met_tool_wrapper/GridDiag/GridDiag.conf +++ b/parm/use_cases/met_tool_wrapper/GridDiag/GridDiag.conf @@ -83,3 +83,5 @@ GRID_DIAG_CONFIG_FILE = {PARM_BASE}/met_config/GridDiagConfig_wrapped #GRID_DIAG_REGRID_CENSOR_VAL = GRID_DIAG_MASK_POLY = MET_BASE/poly/SAO.poly + +#GRID_DIAG_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf b/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf index cf7e103c24..0780e9d0bb 100644 --- a/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf +++ b/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf @@ -209,3 +209,10 @@ GRID_STAT_NC_PAIRS_FLAG_APPLY_MASK = FALSE #GRID_STAT_CENSOR_THRESH = #GRID_STAT_CENSOR_VAL = #GRID_STAT_CAT_THRESH = + +#GRID_STAT_UGRID_DATASET = +#GRID_STAT_UGRID_MAX_DISTANCE_KM = +#GRID_STAT_UGRID_COORDINATES_FILE = +#GRID_STAT_UGRID_CONFIG_FILE = + +#GRID_STAT_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/IODA2NC/IODA2NC.conf b/parm/use_cases/met_tool_wrapper/IODA2NC/IODA2NC.conf index 0948944a55..a036901873 100644 --- a/parm/use_cases/met_tool_wrapper/IODA2NC/IODA2NC.conf +++ b/parm/use_cases/met_tool_wrapper/IODA2NC/IODA2NC.conf @@ -96,3 +96,5 @@ IODA2NC_TIME_SUMMARY_OBS_VAR = "WIND" IODA2NC_TIME_SUMMARY_TYPE = "min", "max", "range", "mean", "stdev", "median", "p80" IODA2NC_TIME_SUMMARY_VLD_FREQ = 0 IODA2NC_TIME_SUMMARY_VLD_THRESH = 0.0 + +#IODA2NC_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf b/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf new file mode 100644 index 0000000000..ac4067dd6c --- /dev/null +++ b/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf @@ -0,0 +1,88 @@ +[config] + +# Documentation for this use case can be found at +# https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/MADIS2NC/MADIS2NC.html + +# For additional information, please see the METplus Users Guide. +# https://metplus.readthedocs.io/en/latest/Users_Guide + +### +# Processes to run +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list +### + +PROCESS_LIST = MADIS2NC + + +### +# Time Info +# LOOP_BY options are INIT, VALID, RETRO, and REALTIME +# If set to INIT or RETRO: +# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set +# If set to VALID or REALTIME: +# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set +# LEAD_SEQ is the list of forecast leads to process +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control +### + +LOOP_BY = INIT +INIT_TIME_FMT = %Y%m%d%H +INIT_BEG = 2012040912 +INIT_END = 2012040912 +INIT_INCREMENT = 1H + +LEAD_SEQ = 0 + +MADIS2NC_RUNTIME_FREQ = RUN_ONCE_FOR_EACH + + +### +# File I/O +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info +### + +MADIS2NC_INPUT_DIR = +MADIS2NC_INPUT_TEMPLATE = {INPUT_BASE}/met_test/data/sample_obs/madis/metar/metar_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc + +MADIS2NC_OUTPUT_DIR = +MADIS2NC_OUTPUT_TEMPLATE = {OUTPUT_BASE}/madis2nc/met_metar_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc + +#MADIS2NC_SKIP_IF_OUTPUT_EXISTS = False + +#MADIS2NC_FILE_WINDOW_BEGIN = 0 +#MADIS2NC_FILE_WINDOW_END = 0 + +### +# MADIS2NC Settings +# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#madis2nc +### + +#LOG_MADIS2NC_VERBOSITY = 3 + +MADIS2NC_CONFIG_FILE = {PARM_BASE}/met_config/Madis2NcConfig_wrapped + +MADIS2NC_TYPE = metar + +#MADIS2NC_QC_DD = +#MADIS2NC_LVL_DIM = +#MADIS2NC_REC_BEG = +#MADIS2NC_REC_END = +#MADIS2NC_MASK_GRID = +#MADIS2NC_MASK_POLY = +#MADIS2NC_MASK_SID = + +#MADIS2NC_TIME_SUMMARY_FLAG = +#MADIS2NC_TIME_SUMMARY_RAW_DATA = +#MADIS2NC_TIME_SUMMARY_BEG = +#MADIS2NC_TIME_SUMMARY_END = +#MADIS2NC_TIME_SUMMARY_STEP = +#MADIS2NC_TIME_SUMMARY_WIDTH = +#MADIS2NC_TIME_SUMMARY_GRIB_CODE = +#MADIS2NC_TIME_SUMMARY_OBS_VAR = +#MADIS2NC_TIME_SUMMARY_TYPE = +#MADIS2NC_TIME_SUMMARY_VLD_FREQ = +#MADIS2NC_TIME_SUMMARY_VLD_THRESH = + +#MADIS2NC_TIME_OFFSET_WARNING = + +#MADIS2NC_MET_CONFIG_OVERRIDES = \ No newline at end of file diff --git a/parm/use_cases/met_tool_wrapper/MODE/MODE.conf b/parm/use_cases/met_tool_wrapper/MODE/MODE.conf index 0d1add4b8b..55fbba5a5f 100644 --- a/parm/use_cases/met_tool_wrapper/MODE/MODE.conf +++ b/parm/use_cases/met_tool_wrapper/MODE/MODE.conf @@ -169,3 +169,5 @@ MODE_QUILT = True #MODE_PS_PLOT_FLAG = #MODE_CT_STATS_FLAG = + +#MODE_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/MTD/MTD.conf b/parm/use_cases/met_tool_wrapper/MTD/MTD.conf index 6f615537e0..30d3c1403c 100644 --- a/parm/use_cases/met_tool_wrapper/MTD/MTD.conf +++ b/parm/use_cases/met_tool_wrapper/MTD/MTD.conf @@ -99,3 +99,5 @@ MTD_REGRID_TO_GRID = OBS MTD_MIN_VOLUME = 2000 MTD_OUTPUT_PREFIX = {MODEL}_{CURRENT_FCST_NAME}_vs_{OBTYPE}_{CURRENT_OBS_NAME}_{CURRENT_FCST_LEVEL} + +#MTD_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/PB2NC/PB2NC.conf b/parm/use_cases/met_tool_wrapper/PB2NC/PB2NC.conf index 93651289ea..9acd346c18 100644 --- a/parm/use_cases/met_tool_wrapper/PB2NC/PB2NC.conf +++ b/parm/use_cases/met_tool_wrapper/PB2NC/PB2NC.conf @@ -94,3 +94,5 @@ PB2NC_TIME_SUMMARY_VALID_FREQ = 0 PB2NC_TIME_SUMMARY_VALID_THRESH = 0.0 #PB2NC_OBS_BUFR_MAP = + +#PB2NC_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/PlotPointObs/PlotPointObs.conf b/parm/use_cases/met_tool_wrapper/PlotPointObs/PlotPointObs.conf index 304e1c5253..6289088f02 100644 --- a/parm/use_cases/met_tool_wrapper/PlotPointObs/PlotPointObs.conf +++ b/parm/use_cases/met_tool_wrapper/PlotPointObs/PlotPointObs.conf @@ -131,3 +131,5 @@ PLOT_POINT_OBS_POINT_DATA = colorbar_flag = FALSE; } } + +#PLOT_POINT_OBS_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/PointStat/PointStat.conf b/parm/use_cases/met_tool_wrapper/PointStat/PointStat.conf index b5a493b787..599d65620b 100644 --- a/parm/use_cases/met_tool_wrapper/PointStat/PointStat.conf +++ b/parm/use_cases/met_tool_wrapper/PointStat/PointStat.conf @@ -198,3 +198,10 @@ POINT_STAT_MESSAGE_TYPE = ADPUPA, ADPSFC #POINT_STAT_HIRA_PROB_CAT_THRESH = #POINT_STAT_MESSAGE_TYPE_GROUP_MAP = + +#POINT_STAT_UGRID_DATASET = +#POINT_STAT_UGRID_MAX_DISTANCE_KM = +#POINT_STAT_UGRID_COORDINATES_FILE = +#POINT_STAT_UGRID_CONFIG_FILE = + +#POINT_STAT_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis.conf b/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis.conf index 7b5fdb8edf..5dc6de3be6 100644 --- a/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis.conf +++ b/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis.conf @@ -172,3 +172,5 @@ SERIES_ANALYSIS_GENERATE_PLOTS = no PLOT_DATA_PLANE_TITLE = SERIES_ANALYSIS_GENERATE_ANIMATIONS = no + +#SERIES_ANALYSIS_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf b/parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf index 14eb180192..59378482fe 100644 --- a/parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf +++ b/parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf @@ -84,3 +84,5 @@ LINE_TYPE_LIST = GROUP_LIST_ITEMS = FCST_INIT_HOUR_LIST LOOP_LIST_ITEMS = FCST_VALID_HOUR_LIST, MODEL_LIST + +#STAT_ANALYSIS_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCDiag/TCDiag.conf b/parm/use_cases/met_tool_wrapper/TCDiag/TCDiag.conf index fc4d27a554..078e9c04ae 100644 --- a/parm/use_cases/met_tool_wrapper/TCDiag/TCDiag.conf +++ b/parm/use_cases/met_tool_wrapper/TCDiag/TCDiag.conf @@ -118,4 +118,4 @@ TC_DIAG_DOMAIN_INFO1_OVERRIDE_DIAGS = #TC_DIAG_CIRA_DIAG_FLAG = #TC_DIAG_OUTPUT_BASE_FORMAT = -#LOG_LEVEL=DEBUG \ No newline at end of file +#TC_DIAG_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCGen/TCGen.conf b/parm/use_cases/met_tool_wrapper/TCGen/TCGen.conf index ace36034eb..424c3376bb 100644 --- a/parm/use_cases/met_tool_wrapper/TCGen/TCGen.conf +++ b/parm/use_cases/met_tool_wrapper/TCGen/TCGen.conf @@ -177,3 +177,5 @@ TC_GEN_DLAND_FILE = MET_BASE/tc_data/dland_global_tenth_degree.nc TC_GEN_BASIN_FILE = MET_BASE/tc_data/basin_global_tenth_degree.nc TC_GEN_NC_PAIRS_GRID = G003 + +#TC_GEN_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_extra_tropical.conf b/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_extra_tropical.conf index e5a0bcf1eb..231138ef23 100644 --- a/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_extra_tropical.conf +++ b/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_extra_tropical.conf @@ -122,3 +122,5 @@ TC_PAIRS_MISSING_VAL = -9999 #TC_PAIRS_DIAG_CONVERT_MAP1_DIAG_SOURCE = #TC_PAIRS_DIAG_CONVERT_MAP1_KEY = #TC_PAIRS_DIAG_CONVERT_MAP1_CONVERT = + +#TC_PAIRS_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_tropical.conf b/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_tropical.conf index e68b5d45f3..9c3415b5ee 100644 --- a/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_tropical.conf +++ b/parm/use_cases/met_tool_wrapper/TCPairs/TCPairs_tropical.conf @@ -115,3 +115,5 @@ TC_PAIRS_DLAND_FILE = MET_BASE/tc_data/dland_global_tenth_degree.nc #TC_PAIRS_DIAG_CONVERT_MAP1_DIAG_SOURCE = #TC_PAIRS_DIAG_CONVERT_MAP1_KEY = #TC_PAIRS_DIAG_CONVERT_MAP1_CONVERT = + +#TC_PAIRS_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCRMW/TCRMW.conf b/parm/use_cases/met_tool_wrapper/TCRMW/TCRMW.conf index 91c4791912..252dca5dfd 100644 --- a/parm/use_cases/met_tool_wrapper/TCRMW/TCRMW.conf +++ b/parm/use_cases/met_tool_wrapper/TCRMW/TCRMW.conf @@ -96,3 +96,5 @@ TC_RMW_CYCLONE = 14 #TC_RMW_VALID_INCLUDE_LIST = #TC_RMW_VALID_EXCLUDE_LIST = #TC_RMW_VALID_HOUR_LIST = + +#TC_RMW_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/TCStat/TCStat.conf b/parm/use_cases/met_tool_wrapper/TCStat/TCStat.conf index ec1705915b..7f4be36833 100644 --- a/parm/use_cases/met_tool_wrapper/TCStat/TCStat.conf +++ b/parm/use_cases/met_tool_wrapper/TCStat/TCStat.conf @@ -117,3 +117,5 @@ TC_STAT_MATCH_POINTS = false #TC_STAT_EVENT_EQUAL_LEAD = #TC_STAT_OUT_INIT_MASK = #TC_STAT_OUT_VALID_MASK = + +#TC_STAT_TIME_OFFSET_WARNING = diff --git a/parm/use_cases/met_tool_wrapper/WaveletStat/WaveletStat.conf b/parm/use_cases/met_tool_wrapper/WaveletStat/WaveletStat.conf index 40bbd5cac0..1cf908fd50 100644 --- a/parm/use_cases/met_tool_wrapper/WaveletStat/WaveletStat.conf +++ b/parm/use_cases/met_tool_wrapper/WaveletStat/WaveletStat.conf @@ -131,3 +131,5 @@ WAVELET_STAT_OUTPUT_FLAG_ISC = STAT #WAVELET_STAT_WVLT_PLOT_PLOT_MAX = #WAVELET_STAT_OUTPUT_PREFIX = + +#WAVELET_STAT_TIME_OFFSET_WARNING =