Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance METdataio schema to handle new/modified MPR and ORANK columns introduced for MET 12.0.0 beta6 #320

Closed
11 of 26 tasks
JohnHalleyGotway opened this issue Aug 1, 2024 · 2 comments · Fixed by #322
Closed
11 of 26 tasks
Assignees
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing

Comments

@JohnHalleyGotway
Copy link
Contributor

JohnHalleyGotway commented Aug 1, 2024

Describe the Enhancement

Issue dtcenter/MET#2924 and corresponding pull requests dtcenter/MET#2939 and dtcenter/MET#2942 made changes to the MPR and ORANK line types. The METdataio schema needs to be updated to handle the diffs described below:

  1. In MPR, CLIMO_MEAN is renamed OBS_CLIMO_MEAN.
  2. In MPR, CLIMO_STDEV is renamed OBS_CLIMO_STDEV.
  3. In MPR, CLIMO_CDF is renamed OBS_CLIMO_CDF.
  4. In MPR, new FCST_CLIMO_MEAN and FCST_CLIMO_STDEV are added to the end of the line.
  5. In ORANK, CLIMO_MEAN is renamed OBS_CLIMO_MEAN.
  6. In ORANK CLIMO_STDEV is renamed OBS_CLIMO_STDEV.
  7. In ORANK, new FCST_CLIMO_MEAN and FCST_CLIMO_STDEV are added to the end of the line.

Please see updated MPR line type and ORANK line type tables.

@bikegeek, it is not clear to me if changes are needed in METplotpy, METcalcpy, and METviewer. I'm not sure if/how data from these columns are used in those components. Please let me know if you'd like me to write up additional issues for them.

METreformat code for the MPR linetype is impacted due to the addition of new columns:
#321

Time Estimate

1 day?

Sub-Issues

  • Retrieve ORANK and MPR sample data from the MET nightly regression tests
  • Make necessary changes to the METdbLoad/ush/constants.py module
  • Make necessary changes to the METdbLoad/sql/mv_mysql.sql file
  • Create and provide content for the update_for_6_0_beta6.sql file
  • Create a directory METdbLoad/tests/update_schema_6.0_beta6 and create a test to verify that the updated and new columns exist

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues

Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version

  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added type: enhancement Improve something that it is currently doing alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone labels Aug 1, 2024
@JohnHalleyGotway JohnHalleyGotway added this to the METdataio-3.0.0 milestone Aug 1, 2024
@bikegeek
Copy link
Collaborator

bikegeek commented Aug 16, 2024

Created issue #321 for the METreformat module to address additional columns added to the MPR linetype

@JohnHalleyGotway
Copy link
Contributor Author

@bikegeek FYI, the dtcenter/MET#2924 issue was completed today and the final PR merged. You can find updated MPR and ORANK data tomorrow in the output from the MET nightly build run on seneca.

bikegeek added a commit that referenced this issue Aug 24, 2024
bikegeek added a commit that referenced this issue Aug 24, 2024
bikegeek pushed a commit that referenced this issue Aug 27, 2024
@bikegeek bikegeek linked a pull request Aug 28, 2024 that will close this issue
15 tasks
bikegeek added a commit that referenced this issue Aug 28, 2024
* Fix pwd value

* Issue #320 updates for MPR and ORANK linetype changes

* Fix indentation

* Issue #320 updates to MPR and ORANK linetypes

* Issue #320 schema updates to reflect changes to MPR and ORANK linetypes.  Test, config, and test data.

* Skip testing MPR linetype in reformatter.  There were changes to the columns: names and additional columnsthat will break this test until the METreformat module is updated.

* Issue #320 fixed syntax error

* Add pip install of pytest

* Issue #320 tests to verify that the renamed column names were created
and the new columns are present.

* Issue #320 config file used to test that the new schema was loaded.

* Update update_for_6_0_beta6.sql

fixed the ADD COLUMN lines for ORANK

---------

Co-authored-by: Minna Win <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: load Tasks related to the database loading priority: blocker Blocker requestor: NOAA/EMC NOAA Environmental Modeling Center required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Projects
Status: 🏁 Done
Development

Successfully merging a pull request may close this issue.

2 participants