Feature #2769 tc_diag_driver_v0.11.0 #2812
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ag_driver_v0.11.0
…ag_driver_v0.11.0
…ag_driver_v0.11.0
…ag_driver_v0.11.0
…ag_driver_v0.11.0
…ag_driver_v0.11.0
to 7 characters but the units are enclosed in parantheses, increasing the field width to 9. If the strings are too long, truncate them and print warnings like this: WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic name "850TANGXXX" truncated to "850TANG"! WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic units string "(10^7C/MXXX)" truncated to "(10^7C/M)"! Pad the first 2 columns out to widths of 7 and 9 and set the inter-column spacing between columns 2 and 3 to 0. This could technically introduce parsing problems when the units are 7 characters long and the values have 5 digits, but this is the logic needed to exactly replicate the existing output.
…ag_driver_v0.11.0
…favor of the MET_KEEP_TEMP_FILE environment variable option that Howard is adding on his feature_2772_python_embedding_json branch. I did add documentation about that option here even though the functionality is coming from Howard's branch.
…ag_driver_v0.11.0
32 tasks
32 tasks
…ag_driver_v0.11.0
…e Python environment which is needed by TC-Diag.
…nce we didn't get it out on the 6th.
|
The latest GHA testing workflow run for this PR works as expected. I confirmed that it really does use METbaseimage v3.2 that includes the SciPy package, and I confirmed that the TC-Diag output file ( I will proceed with merging this PR to be included in the MET-12.0.0-beta3 development release. |
JohnHalleyGotway
added a commit
to dtcenter/METplus
that referenced
this pull request
Feb 8, 2024
* Update release guide to create the input test data directory for official MET releases. * Follow the METplus naming conventions for updating the the DTC web server data. * install library needed for geovista package, remove editable flag so geovista code will be added directly to python instead of reading it from the source code location, ci-skip-all * Feature 685 log updates (#1992) * add script to create conda environment with MetPy 1.4 for SWPC use case * Bugfix develop Fix broken documentation links (#2004) * fixed bug to set environment variables in the RegridDataPlane environment instead of the PyEmbedIngest environment * Feature #2015 modulefiles (#2024) * Adding modulefile for WCOSS2 * Updated references of 4.1.1 to 5.0.0 * Feature #2005 updates to create v5.1 conda environments that use debian10 and Python 3.10.4 (#2043) * update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <[email protected]> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: George McCabe <[email protected]> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies --------- Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Feb 8, 2024
* Store change of sonar.login to sonar.token. * Hotfix to develop, updating aclocal.m4 and config.h.in to what is created when running bootstrap inside the dtcenter/met-base:v3.1 Docker image. Updates autoconf 2.69 output to 2.71. * Adding -lnetcdf to configure_lib_args for NetCDF-CXX compilation * Added eckit and atlas loads and paths * Added atlas and eckit loads and paths * Changing -j to "-j 5" as the recommended value * Update ensemble-stat.rst Fixed incorrect path to obs error table * Updated values for GRIB2CLIB_NAME and BUFRLIB_NAME * Updated for proj, eckit, and atlas * Feature #2761 develop seneca (#2762) * Per #2761, define runtime python version for testing rather than using the default version which no longer exists in /usr/local * Per #2761, fix setting ci-skip-all * Per #2761, patching test_util.R to use the -C command line option for ncdiff. I did test running comp_dir.sh with this change and confirmed that it now runs to completion. * #2652 Added find_var_by_standard_name and separated common codes to find_xy_vars * #2757 Get the email list from the environment variable MET_CRON_EMAIL_LIST__MET (or MET_CRON_EMAIL_LIST_) * #2757 THe SonarQube token and URL are replaced by ujsing the environment variable SONAR_TOKEN_VALUE and SONAR_SERVER_URL * #2757 The SonarQube token and URL are replaced with the pre-defined strings, SONAR_TOKEN_VALUE and SONAR_SERVER_URL * Bugfix #2670 develop --enable-python (#2768) * #2755 Added a header count and checking header count instead of using header id (hid) * Update install_met_env.acorn_py3.10 * Update install_met_env.wcoss2_py3.10 * Feature #2776 cleanup (#2777) * Per #2776, delete stale prob_pair_info.h/.cc, wwmca_utils.h/.cc, and rmw_analysis_utils.h/.cc files that were not being compiled by their Makefiles. * Per #2776, update Makefile to compile the test_read_rmw utility. * Per #2776, add Makefile for the vx_python3_utils test utility directory. * add rpath for atlas and eckit lib dirs so dynamic libraries can be found when running on seneca * only set -L and -I arguments for atlas and eckit if the appropriate environment variable is set -- this matches how it is handled for other external library dependencies * Per #2776, just whitespace and capitalization --------- Co-authored-by: George McCabe <[email protected]> * Bugfix #2782 develop MASSDEN (#2783) * Per #2782, port over fixes from the bugfix_2782_main_v11.1_MASSDEN branch over to the bugfix branch for develop. Will also add new GRIB2 filtering options in this branch. * Per #2782, add support for 4 new GRIB2 filtering options for GRIB2_aerosol_type, GRIB2_aerosol_interval_type, GRIB2_aerosol_size_lower, and GRIB2_aerosol_size_upper. These are useful in filtering the MASSDEN records in the RRFS smoke output files. * Per #2782, fix aerosol_size_lower() and aersol_size_upper() inline definitions. * Per #2782, add a unit test for GRIB2 table 4.48. * Per #2782, switch from strict equality to using the is_eq() function when checking the GRIB2_aerosol_size_lower and upper values since they're doubles and not ints. * Feature #2701 ismn (#2758) * Add support for ISMN soil moisture data * Per #2701, update ascii2nc.cc with the ismn file option. * Per #2701, update ascii2nc docs * Per #2701, store the depth for precip as 0. * Per #2701, parse the ISMN observation value from the correct column. * Per #2701, every time I run ascii2nc to I see a log message for 'DEBUG 1: Number of NDBC skipped files due to no lookup 0'. This is printed by the NdbcHandler destructor. Update the logic to only print a debug level 3 log message when the number of missing locations is greater than 0, 'DEBUG 3: Skipped 5 NDBC files whose locations are not defined in 'MET_BASE/table_files/ndbc_stations.xml'. Set the MET_NDBC_STATIONS environment variable to override this file.' * Per #2701, as instructed by @anewman89, store the average of the depth values rather than the maximum value. * Per #2701, add a test of processing the ISMN data through ascii2nc. * Per #2701, fix parsing of year and month to subtract 1900 and 1, respectively. * Per #2701, set use_var_id to true for ISMN inputs * Per #2701, even though we're mapping obs data to GRIB code names and units, we want to encode them with use_var_id = true and so we need to keep track of those var_id values. * Per #2701, fix typo in the name of the ascii2nc netcdf output file. --------- Co-authored-by: MET Tools Test Account <[email protected]> * Feature 2753 comp script config (#2771) * set dynamic library file extension to .dylib if running on MacOS and .so otherwise * Added disabling of jasper documentation for compiliation on Hera * Updated * remove extra export of compiler env vars * include full path to log file so it is easier to file the log file to examine when a command fails * send cmake output to a log file * remove redundant semi-colon * use full path to log file so it is easier to examine on failure * use run_cmd to catch if rm command fails * Modifications for compilation on hera, gaea, and orion * Updating * fixed variable name * clean up if/else statements * set TIFF_LIBRARY_RELEASE argument to use full path to dynamic library file to prevent failure installing proj library * set LDFLAGS so that LDFLAGS value set in the user's environment will also be used * Updated based on gaea, orion, and hera installs * Updated * change extension of dynamic library files only if architecture is arm64 because older Macs still use .so * added netcdf library to args to prevent error installing NetCDF-CXX when PROJ has been installed in the same run of the script -- PATH is set in the COMPILE_PROJ if block that causes this flag from being added automatically * clean up how rpath and -L are added to LDFLAGS so that each entry is separate -- prevents errors installing on Mac arm64 because multiple rpath values aren't read using :. Also use MET_PROJLIB * Updated * removed -ltiff from MET libs * only add path to rpath and -L arguments if they are not already included in LDFLAGS * changed from using LIB_TIFF (full path to tiff lib file) to use TIFF_LIB_DIR (dir containing tiff lib file). Added TIFF_INCLUDE_DIR to proj compilation and -DJAS_ENABLE_DOC to jasper compliation taken from @jprestop branch * update comments * ensure all MET_* and MET_*LIB variables are added to the rpath for consistency * remove unnecessary if block and only export LDFLAGS at the end of setting locally * Updated * Added section for adding <VALUE>/lib64 and rearranged placement of ADDTL_DIR * Commenting out the running of the Jasper lib tests * Updating and/or removing files * Updating and/or removing files * Latest udpates which include the addition of the tiff library for proj * Remove commented out line. Co-authored-by: John Halley Gotway <[email protected]> * Make indentation consistent. Co-authored-by: John Halley Gotway <[email protected]> * Make indentation consistent. Co-authored-by: John Halley Gotway <[email protected]> * Make indentation consistent. Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * #2697 Share the temporary file for blocking * #2697 Reduced the temporary files for pb2nc * Moved typedef unixtime from vx_cal.h to time_array.h * #2697 Changed static const to constexpr for SonarQube (code smell) * Removing ${MAKE_ARGS} in some locations Removing ${MAKE_ARGS} from "make install" and "make test" for MET. Removing "met" prefix from met.configure.log because we really need config.log for any detail. It is confusing to have met.configure.log when that does not contain useful information. * #2673 Removed unused variable grid_x and grid_y * #2673 Renamed the shadowed variable n to nd * #2673 Use nullptr instead of literal 0 * #2673 Moved down namespace declarations. Removed unused variables * #2673 Reduced the scope of variables * #2673 Use nullptr instead of literal 0 * #2673 Removed unused variables * #2673 Renamed ex to ex2 which becomes shadowed variable * #2673 Removed always true condition * #2673 Moved down namespace declarations. Use nullptr instead of literal 0 * #2673 Use nullptr instead of literal 0. Removed unused variables * #2673 Moved down namespace declarations. * #2673 Added namespace std to string * #2673 Removed unsued variables. Check nullptr of gt * #2673 Moved down namespace declarations. * #2673 Added namespace std to string * #2772 Added quit_msg * Feature #2547 Read WRF output files natively (#2790) * Per #2547, add the file_type = NETCDF_WRF configuration option. * Per #2547, rename vx_data2d_nc_interp as vx_data2d_nc_wrf and vx_data2d_nccf as vx_data2d_nc_cf. * Per #2547, rename nccf files as nc_cf for consistency throughout * Per #2547, rename Pinterp classes as Wrf * only set -L and -I arguments for atlas and eckit if the appropriate environment variable is set -- this matches how it is handled for other external library dependencies * add rpath for atlas and eckit lib dirs so dynamic libraries can be found when running on seneca * ignore directory automatically created by CLion * rename pinterp variables and classes to Wrf * more rename and fix typos * per #2547, add support for reading WRF files by adding support for different X/Y/Z dimensions, read X/Y/Z dimension names when reading a field instead of storing them per file, track if data is staggered in X/Y/Z dimension (still need to add DataPlane function to destagger data), handle pressure field that also includes time dimension * per #2547, add DataPlane function to handle staggering of grid * per #2547, change file type returned from pinterp to wrf * per #2547, call function to stagger DataPlane if necessary * per #2547, keep store if variable is on pressure levels and only read pressure field if it is * ignore cmake directory * per #2547, support U_PL, U_ZL, U, and U<x> where <x> is an integer in is_u_wind and the same for is_v_wind * change regex to match zero or more instances of a number after U or V instead of 1 or more to include the fields named U and V * error and exit if var with unsupported subgrid dimension is processed * clean up logging * only destagger Z dimension for bottom_top_stag because other _stag Z dimensions don't need to be destaggered, clean up logic to read X/Y/Z dimension info * Per #2547, instead of checking dimensions of all vars in file, check dimensions for the var that is being read. Only read pressure var if the z dimension matches the var to read. Set class variable DimNames instead of local gDimNames so dump function will work as expected * per #2547, support reading variable that corresponds to dimension with different name as dimension and includes a time dimension, e.g. P_PL(Time, num_press_levels_stag) * per #2547, added unit tests to process WRF out data with plot_data_plane and grid_stat * merged develop and resolved conflicts * rename lib from pinterp to wrf * rename lib * fixed incorrect merge changes * fixed broken unit test format * Per #2547, restore the mention of NAK.poly to Appendix B. * Change model to WRF Co-authored-by: John Halley Gotway <[email protected]> * change obtype to WRF Co-authored-by: John Halley Gotway <[email protected]> * plot higher vertical level to get non-zero output Co-authored-by: John Halley Gotway <[email protected]> * indentation Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * #2772 Use JSON for attrubutes and numpy serialization dor 2D data instead of NetCDF * Hotfix to the develop branch after PR #2790 merged the feature_2547_wrf branch into develop. Rerunning bootstrap on the project machine seneca produces 2 small differences that should also be incorporated into develop. * Feature 2588 install rewrite (#2791) * updated first 3 sections * updated spacing * adding the Environment Variables to Run Script section * adding in the remainder of the google doc changes * trying to fix duplicate explicit target name issue * fixing required_external_libraries_to_build_met links * HDF4 testing double underscore * adding double underscores for HDF5 and HDF-EOS2 and changing library back * fixing HDF5 web link name * Added formatting, made minor edits, removed manual build instructions sections * Removed reference to sample-test-case which no longer exists * Resolving errors * Minor formatting change * Updated apptainer instructions, new version references * Formatting updates and removal of first person language * fixed code blocks * Resolving error * Removed first person references * Removed reference to met_directory_structure * updated bolding, removed end section * Changing X.Y.X references to X.Y.Z * updating thru Using the compile_MET_all.sh Script * fixing spacing for ATLAS web address * Fixed broken internal reference to installation due to typo * adding version numbers * fixing Recommended-Components MET * 3.4.2 section updates * fixing spacing * adding a period and more dropdown menus * fixing spacing * loose ends * removing space * removing bold * External Library handling in compile_MET_all.sh section updates * installation directory view * installation directory after * attempting to add figures via the web updating Executing the compile_MET_all.sh script section * fixing bolds, italics etc. * updating last half of doc * loose ends * loose ends * adding more dropdown menus * fixing dropdown menu and testing automatic version * testing automatic version * confirmed automatic version cannot be added to italics or code blocks * another round of updates * fixing indents * creating a table * fixing table * fixing table 2 * more loose end changes * more changes * hopefully final updates * trying a fancy new csv table * troubleshooting csv table * correcting table name * adding a grid table * fixing grid table * removing csv table * Per #2588, update to the environment variables table. * Delete docs/Users_Guide/environment_variables_table.csv * Per #2785, add new example installation files * updating small changes in the first half of the documentation * new dropdown menu and fixing some links * trying to fix line spacing in dropdown * take 2 fixing spacing * take 3 fix spacing * env var order * removing "s" * small changes * un capitalizing S * A few minor changes --------- Co-authored-by: j-opatz <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Minor hotfix to develop to fix a typo in the comments of the PB2NC config files. * Feature #2796 develop gha node20 (#2797) * Per #2796, update actions/checkout@v3 to actions/checkout@v4. * Per #2796, update to actions/upload-artifact@v4 and actions/setup-python@v5. * Per #2796, update to download-artifact@v4 and actions/github-script@v7 * Feature #2796 develop gha_node20, fix artifact names (#2799) * Per #2796, update actions/checkout@v3 to actions/checkout@v4. * Per #2796, update to actions/upload-artifact@v4 and actions/setup-python@v5. * Per #2796, update to download-artifact@v4 and actions/github-script@v7 * Per #2796, update testing workflow for update-artifact v4 version which requires unique artifact names. ci-run-unit * Per #2796, merge the logs into a single artifact. ci-run-unit * Per #2796, renaming rc_leads and rc as 1c and 2c, respectively for consistency purposes. Note that this commit should result in just a logs_compile artifact being created... and not doing the log merging step. * Per #2796, update the compilation workflow to make the log artifact names unique. * #2772 Change nan and inf to -9999 on reading ASCII input if failed to parse * Feature #2801 warnings (#2802) * Per #2801, add new time_offset_warning config option. * Per #2810, update MetConfig class to parse and check the time_offset_warning config option. * Per #2801, update Grid-Stat, MODE, PCP-Combine, Series-Analysis, Wavelet-Stat, and the WWMCA tool to call MetConfig::time_offset_warning() to decide whether to print a Warning or Debug(3) log message about initialization or valid time differences. * Per #2801, no change to content. Just removing unneeded whitespace. * Per #2801, remove extra empty line foromw m warning message. * Per #2801, remove extra empty line from MODE warning messages. --------- Co-authored-by: MET Tools Test Account <[email protected]> * #2772 Initial release, Separated from point.py * #2772 Added point_nc.py * #2772 Changed write_tmp_nc and read_tmp_nc to write_tmp_py and read_tmp_py * #2772 Removed python_key_point_data & python_key_point_data_list and replaced them to tmp_point_var_name and tmp_point_data * #2772 Renamed tmp_nc_base_name, tmp_nc_file_var_name & tmp_nc_point_var_name to tmp_py_base_name, tmp_file_var_name, & tmp_point_var_name * #2772 More log messages for error * #2772 Changed API (log_msg to log_message) * #2772 Use met_point_nc_tools instead of met_point_tools * #2772 Changed APIs * #2772 Changed API * #2772 Changed default temp output format to JSON ande numpoy serialization * #2772 Allow to keep the temporary files by using the environment variable * Added log message if the temprary fiule was not deleted * Feature #2745 mvmode enhancements (#2779) * Working mvmode with percentile thresholds and grid_type fixes, plus more unit tests * separated multivar and traditional mode frontend classes to simplify the mode executive class design * documentation changes to go with changes to percentiles and data_type * Cleanup and bugfix to doc * Bug fix to doc * Another bugfix attempt * synching up development.seneca with what is in the develop branch * added clone() method to all the vx_data2d var_info classes * Fix multivar_name/level definitions in UG * bug fix regarding accessing the config to create a verification grid * Update docs/Users_Guide/mode.rst * Update docs/Users_Guide/mode.rst * Per #2745, commit changes to Makefile.am/.in files after running bootstrap on this feature branch on seneca. * Per #2745, no real code changes. Just whitespace. --------- Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Per #2772, added MET_PYTHON_EXE to various test cases and removed what would have been a duplicate after adding MET_PYTHON_EXE * Per #2772, add documentation about 3 new environment variables. * Per #2772, tweak the wording. * Fixing typo * Feature #2772 python_embedding_tmp_file (#2807) Co-authored-by: Howard Soh <[email protected]> * Feature #2805 filter_set_hdr (#2806) * Per #2805, update STATAnalysisJob::dump_stat_line() to take an optional argument to indicate whether the set_hdr options should be applied. Update Stat-Analysis to only call dump_stat_line() with set_hdr for filter jobs. * Per #2805, update User's Guide with -set_hdr info. * Per #2805, the -set_hdr option doesn't apply to tc_stat * Per #2805, modify one Stat-Analysis filter job to demonstrate that the -set_hdr job command option is now applied to the -dump_row output. * Per #2805, enhance the R differencing logic to handle NA strings in the header columns. Previously, differences that include the NA string were ignored. This enhances the compareStatLty() function to handle NA strings properly. * Per #2805, no real code change. Just whitespace. --------- Co-authored-by: MET Tools Test Account <[email protected]> * Feature 2772 python embedding fill value (#2810) * #2772 Update masked value to -9999. Renamed error_messageg to error_message * #2772 Renamed error_messageg to error_message * #2772 Accept 4th argument as user defined fill value. * #2772 Added python_numpy_plot_data_plane_missing which processes user defined fill value --------- Co-authored-by: Howard Soh <[email protected]> * Feature #2809 nc_laea (#2811) * Per #2809 work in progress parsing LAEA grids from MET NetCDF files. * Per #2809, update the LaeaData struct to store the spheroid_name as a character array. I tried very hard to store this as a string instead but ran into lots of problems with segfaults when deleting the allocated structs. Using a fixed length character array while making calls to the m_strncpy() and m_strlen() utility functions enables the code to work properly while hopefully minimizing SonarQube findings. * Per #2809, call m_strncpy() to set LaeaData::spheroid_name * Per #2809, add tests to pcp_combine and plot_data_plane to demonstrate the reading the LAEA data from MET NetCDF files. * Per #2809, correct the expected PostScript file output name in unit_plot_data_plane.xml * Feature #2809 v12.0.0_beta3 (#2814) * Per #2809, update copyright to 2024 * Per #2809, add release v12.0.0-beta3 release notes. * Feature #2769 tc_diag_driver_v0.11.0 (#2812) * Per #2769, update to tc_diag_driver version 0.11.0. * Per #2769, add tmp_nc_diag_flag config option to retain temporary files. * Per #2769, change default regrdding method from NEAREST to BILIN. * Per #2769, add checks to make sure input files exist. * Fix the handling of diagnostic name and units strings. Both are limited to 7 characters but the units are enclosed in parantheses, increasing the field width to 9. If the strings are too long, truncate them and print warnings like this: WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic name "850TANGXXX" truncated to "850TANG"! WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic units string "(10^7C/MXXX)" truncated to "(10^7C/M)"! Pad the first 2 columns out to widths of 7 and 9 and set the inter-column spacing between columns 2 and 3 to 0. This could technically introduce parsing problems when the units are 7 characters long and the values have 5 digits, but this is the logic needed to exactly replicate the existing output. * Just whitespace * Per #2769, remove the tmp_nc_diag_flag TC-Diag config file option in favor of the MET_KEEP_TEMP_FILE environment variable option that Howard is adding on his feature_2772_python_embedding_json branch. I did add documentation about that option here even though the functionality is coming from Howard's branch. * Per #2769, remove tmp_nc_diag_flag that has been removed. * Per #2769, remove mention of tmp_nc_diag_flag option that has been removed. * Per #2769, upgrade to METbaseimage version 3.2 to provide SciPy in the Python environment which is needed by TC-Diag. * Per #2769, add SciPy to the list of required Python packages. * Per #2769, tweak AppendixF details. * Per #2769, update release date for MET-12.0.0-beta3 to be 20240207 since we didn't get it out on the 6th. --------- Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: root <root@83062d57c5dd> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: davidalbo <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Expected Differences
Note that this PR fixes several items mentioned in issue #2769, but does not complete the TC-Diag development work. While the formatting of the diagnostic output files is now correct, the computed diagnostic values do not match those computed by running the tc_diag_driver Python code directly. This may be due to differences in the transformation to cylindrical coordinates. And this discrepancy requires more investigating during the beta4 development cycle. I'll add a new issue for that.
Do these changes introduce new tools, command line arguments, or configuration file options? [No]
If yes, please describe:
Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [Yes]
If yes, please describe:
Modifies the spacing of the TC-Diag ascii output files to make it consistent with existing TC-Diag outputs.
Pull Request Testing
Describe testing already performed for these changes:
Compared TC-Diag output format to existing LS-Diag output files.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Most of the changes in this PR are updates to the tc_diag_driver python code. However, tc_diag.cc does contain updates for formatting the output.
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
None needed.
Do these changes include sufficient testing updates? [Yes]
Sufficient for the time being, but we do still need to add a better example of a nested grid in beta4 using data provided by @KathrynNewman.
Will this PR result in changes to the test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
Modifies the format and values of existing TC-Diag unit test.
Please complete this pull request review by [Tues 2/6/24].
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases