Feature 2647 use case mv mode for rrfs (#2706)

* establish documentation, need to fill out with provided info

* updated documentation, needs to have img path and output path fixed

* updated output strings, still need to update pic link

* removing pound sign to render correctly

* Revert "removing pound sign to render correctly"

This reverts commit ffe607a.

* targeted removal of pound sign with bullet points section.

* Per #2647, attempt to fix syntax error

* Per #2647, adding back contents with formatting changes to test if it works

* Per #2647, adding back contents in a different location to try to resolve syntax error

* Per #2647, testing pipe for horizontal scroll bar

* Per #2647, removed the pipe symbol and are attempting to fix syntax error again

* Per #2647, reverting to original failed attempt of a table of contents

* Per #2647, trying new formatting

* Per #2647, fixing new formatting

* Per #2647, attempt formatting changes

* Per #2647, reverting back to original state

* added to automated testing, updated docs, config file

* added docs back in

* added image, new quick search term, disabled auto tests

* testing use_case_groups.json to see if/how this fails

* Updated to disable use_case_groups.json run

* added use case to list of exceeding disk space

---------

Co-authored-by: lisagoodrich <[email protected]>
Co-authored-by: Lisa Goodrich <[email protected]>
Co-authored-by: Julie Prestopnik <[email protected]>

.github/parm/use_case_groups.json

@@ -276,6 +276,12 @@
     "disabled": true,
     "run": false
   },
+  {
+    "category": "short_range",
+    "index_list": "15",
+    "disabled": true,
+    "run":false
+  },
   {
     "category": "space_weather",
     "index_list": "0-1",

docs/Contributors_Guide/add_use_case.rst

@@ -169,6 +169,7 @@ Use Cases that Exceed GitHub Actions Disk Space
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 - *model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar*
 - *model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereQBO*
+- *model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning*
 
 .. _use_case_documentation:
 

docs/Users_Guide/quicksearch.rst

@@ -156,6 +156,7 @@ Use Cases by METplus Feature:
    | `METplotpy <../search.html?q=METplotpyUseCase&check_keywords=yes&area=default>`_
    | `MET_PYTHON_EXE Environment Variable  <../search.html?q=MET_PYTHON_EXEUseCase&check_keywords=yes&area=default>`_
    | `Multiple Conf File Use <../search.html?q=MultiConfUseCase&check_keywords=yes&area=default>`_
+   | `MultivariateMODE <../search.html?q=MvMODEUseCase&check_keywords=yes&area=default>`_
    | `Observation Time Summary <../search.html?q=ObsTimeSummaryUseCase&check_keywords=yes&area=default>`_
    | `Observation Uncertainty <../search.html?q=ObsUncertaintyUseCase&check_keywords=yes&area=default>`_
    | `Python Embedding Ingest <../search.html?q=PyEmbedIngestToolUseCase&check_keywords=yes&area=default>`_
@@ -187,6 +188,7 @@ Use Cases by METplus Feature:
    | **METplotpy**: *METplotpyUseCase*
    | **MET_PYTHON_EXE Environment Variable**: *MET_PYTHON_EXEUseCase*
    | **Multiple Conf File Use**: *MultiConfUseCase*
+   | **MultivariateMODE**: *MvMODEUseCase*
    | **Observation Time Summary**: *ObsTimeSummaryUseCase*
    | **Observation Uncertainty**: *ObsUncertaintyUseCase*
    | **Python Embedding Ingest**: *PyEmbedIngestToolUseCase*

docs/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.py

@@ -0,0 +1,179 @@
+"""
+MODEMultivar: Create objects of brightness temps and radar reflectivity  
+=======================================================================
+
+model_applications/
+short_range/
+MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf
+
+"""
+############################################################################## 
+# .. contents::
+#    :depth: 1
+#    :local:
+#    :backlinks: none
+
+##############################################################################
+# Scientific Objective
+# --------------------
+#
+# This use case identifies convective objects, which are defined by
+# the intersection of: 1) satellite infrared brightness temperature < 235 K and 
+# 2) radar reflectivity > 40 dBZ.
+# Satellite brightness temperatures are used in conjunction with radar reflectivity 
+# to capture both the cloud top (satellite) and in-cloud (radar) characteristics. 
+# Convective objects are also defined as lightning thresholds exceeding the 10th percentile. 
+# A percentile threshold is used for lightning data as RRFS lightning has units 
+# which are “non-dimensional” and therefore cannot be directly compared to the
+# Geostationary Lightning Mapper. 
+
+##############################################################################
+# Version Added
+# -------------
+#
+# METplus Version 6.0
+
+##############################################################################
+# Datasets
+# --------
+#
+# **Forecast:** Rapid Refresh Forecast System (RRFS) 3km resolution, 
+# channel 13 brightness temperature,
+# composite reflectivity, and lightning strike density
+#
+# **Observation:** Geostationary Operational Environmental Satellites (GOES) 3km resolution, 
+# channel 13 brightness temperature; 
+# Multi-radar Multi-sensor (MRMS) 3km resolution, composite reflectivity; 
+# GOES Global Lightning Mapper (GLM) 3km resolution, flash_extent_density
+# 
+# **Climatology:** None
+#
+# **Location:** All of the input data required for this use case can be found in a sample data 
+# tarball. Each use case category will have one or more sample data tarballs. It is only 
+# necessary to download the tarball with the use case’s dataset and not the entire 
+# collection of sample data. Click here to access the METplus releases page and download 
+# sample data for the appropriate release: https://github.com/dtcenter/METplus/releases
+# This tarball should be unpacked into the directory that you will set the value of 
+# INPUT_BASE. See :ref:`running-metplus` for more information.
+
+##############################################################################
+# METplus Components
+# ------------------
+#
+# The only tool this use case calls is MODE, which will identify super-objects
+# by intersection of the multiple variable fields.
+# 
+
+##############################################################################
+# METplus Workflow
+# ----------------
+#
+# | **Beginning Time (INIT_BEG):** 2024-01-09 05:00 UTC
+# | **End Time (INIT_END):** 2024-01-09 05:00 UTC
+# | **Increment between beginning and end times (VALID_INCREMENT):** 1 Hour
+# | **Sequence of forecast leads to process (LEAD_SEQ):** 9,10
+#
+# This use case runs twice, once for each forecast lead time provided. It 
+# creates objects valid at 14UTC and 15UTC from 09 January 2024 are compared to 
+# the 9h and 10h forecasts initialized at 05UTC on 9 January 2024.
+# Convective objects are identified with thresholds of
+# satellite brightness temperature < 235 K and radar reflectivity > 40 dBZ, 
+# or lightning > 10th percentile.
+# In this use case, MODE super-object intensity statistics are output for both 
+# radar reflectivity and lightning. Using the MODE_MULTIVAR_INTENSITY_FLAG, 
+# users can control for which variables super object intensity statistics will be output. 
+# If all are set to False, then no intensity information will be output
+# and only statistics relative to the super-object geometry will be available.
+
+
+
+##############################################################################
+# METplus Configuration
+# ---------------------
+#
+# METplus first loads all of the configuration files found in parm/metplus_config,
+# then it loads any configuration files passed to METplus via the command line, i.e.
+# parm/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf
+#
+# .. highlight:: bash
+# .. literalinclude:: ../../../../parm/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf
+
+##############################################################################
+# MET Configuration
+# ---------------------
+#
+# METplus sets environment variables based on user settings in the METplus configuration file. 
+# See :ref:`How METplus controls MET config file settings<metplus-control-met>` for more details. 
+#
+# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!**
+#
+# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to:
+# :ref:`Overriding Unsupported MET config file settings<met-config-overrides>`
+#
+# .. dropdown:: MODEConfig_wrapped
+#
+#   .. literalinclude:: ../../../../parm/met_config/MODEConfig_wrapped
+
+##############################################################################
+# Python Embedding
+# ----------------
+#
+# This use case does not use any Python Embedding.
+
+##############################################################################
+# Python Scripting
+# ----------------
+#
+# This use case does not use any Python Scripting.
+
+##############################################################################
+# Running METplus
+# ---------------
+#
+# Pass the use case configuration file to the run_metplus.py script
+# along with any user-specific system configuration files if desired::
+#
+#    run_metplus.py /path/to/METplus/parm/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf /path/to/user_system.conf
+#
+# See :ref:`running-metplus` for more information.
+
+##############################################################################
+# Expected Output
+# ---------------
+#
+# A successful run will output the following both to the screen and to the logfile::
+#
+#   INFO: METplus has successfully finished running.
+#
+# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated.
+# Output for this use case will be found in 
+# {OUTPUT_BASE}/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning/f??,
+# where the '??' characters will reflect the two forecast leads (09 and 10).
+# Each of these directories will contain the following files with their appropriate
+# verification times::
+#
+#  * mode_Fcst_LTNG_entireatmosphere_all_all_Obs_flash_extent_density_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_cts.txt
+#  * mode_Fcst_LTNG_entireatmosphere_all_all_Obs_flash_extent_density_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_obj.nc
+#  * mode_Fcst_LTNG_entireatmosphere_all_all_Obs_flash_extent_density_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_obj.txt
+#  * mode_Fcst_LTNG_entireatmosphere_all_all_Obs_flash_extent_density_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A.ps
+#  * mode_Fcst_REFC_entireatmosphere_consideredasinglelayer_all_all_Obs_MergedReflectivityQCComposite_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_cts.txt
+#  * mode_Fcst_REFC_entireatmosphere_consideredasinglelayer_all_all_Obs_MergedReflectivityQCComposite_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_obj.nc
+#  * mode_Fcst_REFC_entireatmosphere_consideredasinglelayer_all_all_Obs_MergedReflectivityQCComposite_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A_obj.txt
+#  * mode_Fcst_REFC_entireatmosphere_consideredasinglelayer_all_all_Obs_MergedReflectivityQCComposite_all_all_RRFS_or_ANALYSIS_090000L_20240109_140000V_000000A.ps
+
+##############################################################################
+# Keywords
+# --------
+#
+# .. note::
+#
+#   * MODEToolUseCase 
+#   * ShortRangeAppUseCase
+#   * NetCDFFileUseCase
+#   * MvMODEUseCase
+#
+#   Navigate to the :ref:`quick-search` page to discover other similar use cases.
+#
+#
+# sphinx_gallery_thumbnail_path = '_static/short_range-MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.png'
+#

internal/tests/use_cases/all_use_cases.txt

@@ -184,7 +184,7 @@ Category: short_range
 12::UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection::model_applications/short_range/UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection.conf:: metplotpy_env
 13::MODEMultivar_fcstHRRR_obsMRMS_HRRRanl::model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf
 14::UserScript_fcstRRFS_fcstOnly_Reformat_Aggregate_Plot::model_applications/short_range/UserScript_fcstRRFS_fcstOnly_Reformat_Aggregate_Plot.conf:: metdataio, metcalcpy, metplotpy,mp_analysis_env
-
+15::MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning::model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf
 
 Category: space_weather
 0::GridStat_fcstGloTEC_obsGloTEC_vx7:: model_applications/space_weather/GridStat_fcstGloTEC_obsGloTEC_vx7.conf

parm/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf

@@ -0,0 +1,132 @@
+[config]
+
+# Documentation for this use-case can be found at:
+# https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.html
+
+# Processes to run
+PROCESS_LIST = MODE
+
+# Time Info
+LOOP_BY = INIT
+INIT_TIME_FMT = %Y%m%d%H
+INIT_BEG=2024010905
+INIT_END=2024010905
+INIT_INCREMENT = 1H
+
+LEAD_SEQ = 9,10
+
+MODEL = RRFS
+OBTYPE = ANALYSIS
+
+##################################
+# Multivariate MODE Configurations
+##################################
+
+# Run MODE to output super objects
+MODE_MULTIVAR_LOGIC = #1 && #2 || #3
+MODE_MULTIVAR_INTENSITY_FLAG = FALSE,TRUE,TRUE
+
+FCST_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning
+FCST_MODE_INPUT_TEMPLATE= {init?fmt=%Y%m%d}.{init?fmt=%H}00.F{lead?fmt=%HH}00.nc,{init?fmt=%Y%m%d}.{init?fmt=%H}00.F{lead?fmt=%HH}00.nc,{init?fmt=%Y%m%d}.{init?fmt=%H}00.F{lead?fmt=%HH}00.nc
+
+OBS_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning
+OBS_MODE_INPUT_TEMPLATE = remap_GOES-16.{valid?fmt=%Y%m%d}.{valid?fmt=%H}0000.nc,{valid?fmt=%Y%m%d}-{valid?fmt=%H}0000_remap.nc,remap_GOES-16_lightning_{valid?fmt=%Y%m%d}-{valid?fmt=%H}00_75W.nc
+
+MODE_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning
+MODE_OUTPUT_TEMPLATE = f{lead?fmt=%2H}
+
+FCST_MODE_FILE_WINDOW_BEGIN = 0
+FCST_MODE_FILE_WINDOW_END = 0
+
+OBS_MODE_FILE_WINDOW_BEGIN = 0
+OBS_MODE_FILE_WINDOW_END = 0
+
+###
+# Field Info
+###
+
+
+FCST_VAR1_NAME = bt13_topofatmosphere_G16
+FCST_VAR1_LEVELS = (*,*)
+FCST_VAR1_OPTIONS = conv_radius = 5; conv_thresh = <=230; merge_thresh = <=230; merge_flag = NONE; file_type = NETCDF_MET; filter_attr_name = ["AREA"]; filter_attr_thresh= [ >=1 ]; censor_thresh = [<=0]; censor_val = [9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+
+OBS_VAR1_NAME = channel_13_brightness_temperature
+OBS_VAR1_LEVELS = L0
+OBS_VAR1_OPTIONS = conv_radius = 0; conv_thresh = <=230; merge_thresh = <=230; merge_flag = NONE; file_type = NETCDF_MET; filter_attr_name = ["AREA"]; filter_attr_thresh= [ >=1 ]; censor_thresh = [<=0]; censor_val = [9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+
+FCST_VAR2_NAME = REFC_entireatmosphere_consideredasinglelayer
+FCST_VAR2_LEVELS = (*,*)
+FCST_VAR2_OPTIONS = conv_radius = 5; conv_thresh = >=40; merge_thresh = >=40; merge_flag = NONE; file_type = NETCDF_MET; filter_attr_name = ["AREA"]; filter_attr_thresh= [ >=1 ]; censor_thresh = [< -30]; censor_val = [9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+OBS_VAR2_NAME = MergedReflectivityQCComposite
+OBS_VAR2_LEVELS = (*,*)
+OBS_VAR2_OPTIONS = conv_radius = 5; conv_thresh = >=40; merge_thresh = >=40; merge_flag = NONE; file_type = NETCDF_MET; filter_attr_name = ["AREA"]; filter_attr_thresh= [ >=1 ]; censor_thresh = [< -30]; censor_val = [9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+
+FCST_VAR3_NAME = LTNG_entireatmosphere
+FCST_VAR3_LEVELS = (*,*)
+FCST_VAR3_OPTIONS = conv_radius = 3; conv_thresh = >SFP10; merge_thresh = >SFP10; merge_flag = THRESH; file_type = NETCDF_MET; censor_thresh = [<= 0]; censor_val = [-9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+OBS_VAR3_NAME = flash_extent_density
+OBS_VAR3_LEVELS = (*,*)
+OBS_VAR3_OPTIONS = conv_radius = 3; conv_thresh = >SOP10; merge_thresh = >SOP10; merge_flag = THRESH; file_type = NETCDF_MET; censor_thresh = [<= 0]; censor_val = [-9999]; vld_thresh = 0.5; inten_perc_value  = 100; inten_perc_thresh = NA
+
+
+MODE_MATCH_FLAG = MERGE_BOTH
+
+MODE_REGRID_TO_GRID = NONE
+
+LOG_MODE_VERBOSITY = 4
+
+MODE_OUTPUT_PREFIX = {MODEL}_or_{OBTYPE}
+
+MODE_GRID_RES = 3
+
+MODE_INTEREST_FUNCTION_CENTROID_DIST = ((            0.0, 1.0 )(  60.0/grid_res, 1.0 )( 450.0/grid_res, 0.0 ))
+MODE_INTEREST_FUNCTION_BOUNDARY_DIST = ((            0.0, 1.0 )( 400.0/grid_res, 0.0 ))
+MODE_INTEREST_FUNCTION_CONVEX_HULL_DIST = ((            0.0, 1.0 )( 400.0/grid_res, 0.0 ))
+MODE_INTEREST_FUNCTION_ANGLE_DIFF = ((  0.0, 1.0 )( 30.0, 1.0 )( 90.0, 0.0 ))
+MODE_INTEREST_FUNCTION_ASPECT_DIFF = ((  0.00, 1.0 )(  0.10, 1.0 )(  0.75, 0.0 ))
+MODE_INTEREST_FUNCTION_AREA_RATIO = ((    0.0, 0.0 )(    0.8, 1.0 )(    1.0, 1.0 ))
+MODE_INTEREST_FUNCTION_INT_AREA_RATIO = (( 0.00, 0.00 )( 0.10, 0.50 )( 0.25, 1.00 )( 1.00, 1.00 ))
+MODE_INTEREST_FUNCTION_CURVATURE_RATIO = ((    0.0, 0.0 )(    0.8, 1.0 )(    1.0, 1.0 ))
+MODE_INTEREST_FUNCTION_COMPLEXITY_RATIO = ((    0.0, 0.0 )(    0.8, 1.0 )(    1.0, 1.0 ))
+MODE_INTEREST_FUNCTION_INTEN_PERC_RATIO = ((    0.0, 0.0 )(    0.8, 1.0 )(    1.0, 1.0 ))
+
+MODE_TOTAL_INTEREST_THRESH = 0.65
+
+MODE_MASK_GRID = ""
+MODE_MASK_GRID_FLAG = NONE
+MODE_MASK_POLY = ""
+MODE_MASK_POLY_FLAG = NONE
+
+MODE_MASK_MISSING_FLAG = BOTH
+MODE_MATCH_FLAG = MERGE_BOTH
+
+MODE_MAX_CENTROID_DIST = 600.0/grid_res
+
+MODE_WEIGHT_CENTROID_DIST = 4.0
+MODE_WEIGHT_BOUNDARY_DIST = 3.0
+MODE_WEIGHT_CONVEX_HULL_DIST = 1.0 
+MODE_WEIGHT_ANGLE_DIFF = 1.0
+MODE_WEIGHT_ASPECT_DIFF = 0.0
+MODE_WEIGHT_AREA_RATIO = 4.0
+MODE_WEIGHT_INT_AREA_RATIO = 3.0
+MODE_WEIGHT_CURVATURE_RATIO = 0.0
+MODE_WEIGHT_COMPLEXITY_RATIO = 0.0
+MODE_WEIGHT_INTEN_PERC_RATIO = 0.0
+MODE_WEIGHT_INTEN_PERC_VALUE = 50
+
+MODE_NC_PAIRS_FLAG_LATLON = TRUE
+MODE_NC_PAIRS_FLAG_RAW = TRUE
+MODE_NC_PAIRS_FLAG_OBJECT_RAW = TRUE
+MODE_NC_PAIRS_FLAG_OBJECT_ID = TRUE
+MODE_NC_PAIRS_FLAG_CLUSTER_ID = TRUE
+MODE_NC_PAIRS_FLAG_POLYLINES = TRUE
+
+MODE_QUILT = False
+
+MODE_PS_PLOT_FLAG = TRUE
+MODE_CT_STATS_FLAG = TRUE