Scorecard won't plot stat-analysis output due to event equalization

[briannen]

I have created a scorecard using output from point-stat that works correctly, but has too many lead times to be useful. I used stat-analysis to aggregate the lead times to 24 hours, but now the event equalization is throwing out all of the data and the scorecard won't plot. Why does event equalization affect the ability to plot the 24 hour averages using stat-analysis (vs. output from each forecast time)?

[JohnHalleyGotway]

Hi @briannen, I see you're having issues with scorecard generation. I read through your question and see there's no easy, simple answer to it. As is often the case, the answer is in the details.

Can you please provide me more information so that I'm able to replicate the behavior you're seeing?

• On what machine are you running?
• Please provide the scorecard XML and command that successfully generate the the scorecard with "too many lead times to be useful".
• Please provide the scorecard XML and command that discard all the data based based on event equalization.

Once I'm able to replicate the behavior you describe, hopefully I'll be able to offer advice on how to proceed.

[briannen]

I'm running on mohawk.

The working scorecard XML is /d2/projects/land_nysm/metv/scorecards/test_scorecard.xml.
The non-working scorecard XML is /d2/projects/land_nysm/metv/scorecards/scorecard_agg_leads.xml.

For both scorecards, I use the mv_scorecard.sh script from /d2/projects/land_nysm/metv. The command is ./mv_scorecard.sh scorecards/{file}.xml

Thanks!

[JohnHalleyGotway]

Thanks @briannen for pointing me to your work location on mohawk. I copied your XML files, modified test_scorecard.xml to direct the output to my working directory, and ran METviewer.

I DO NOT get a successful run. Instead, I'm seeing error messages about all the data being event-equalized out:

> cd /d2/personal/johnhg/METplus/discussions/2734
> ./mv_scorecard.sh test_scorecard.xml | tee test_scorecard.log
> egrep -i "error|equal" test_scorecard.log 
WARNING: event equalization removed 10 rows
INFO: Event equalization has produced no results.  Data frame is empty.
WARNING: event equalization removed 10 rows
INFO: Event equalization has produced no results.  Data frame is empty.
2024-10-18T19:33:02,003 ERROR [-1] [printFormattedTable] e.u.m.s.d.DatabaseManagerSql: 
  **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException in printFormattedTable(ResultSet res): result set contained no data
INFO: Event equalization has produced no results.  Data frame is empty.
...

I see that your data directory does contain output from Sep 30 and Oct 11. I wonder if perhaps that pre-existing data is giving you a false positive, making you think it's working when it isn't?

ls -lah /d2/projects/land_nysm/metv/scorecards/data/
-rw-r--r-- 1 briannen rap 2.3K Oct 11 18:50 test_scorecard_agg.agg_stat.info
-rw-r--r-- 1 briannen rap    1 Oct 11 18:50 test_scorecard_agg.data
-rw-r--r-- 1 briannen rap  130 Oct 11 18:50 test_scorecard_agg.data.agg_stat
-rw-r--r-- 1 briannen rap  14K Sep 30 14:24 test_scorecard.agg_stat.info
-rw-r--r-- 1 briannen rap 319K Sep 30 14:25 test_scorecard.data
-rw-r--r-- 1 briannen rap 325K Sep 30 14:24 test_scorecard.data.agg_stat

Please try re-testing test_scorecard.xml and either redirect the output to a new, empty directory or first empty out the subdirectories of /d2/projects/land_nysm/metv/scorecards.

We're assuming the problem lies in the difference between test_scorecard.xml and scorecard_agg_leads.xml but it might actually lie in test_scorecard.xml itself.

[briannen]

I realized that I didn't just add the stat-analysis output to the database, but I dropped and reloaded completely. I will have to reload the database with all of the stat-analysis and point-stat output first, but then I will re-test the scorecard xml.

[briannen]

Okay, I have re-tested the scorecard xmls. I am now running in /d2/projects/land_nysm/metv/scorecards_troubleshoot on mohawk. As you can see by the attached images, the non-aggregated scorecard (test_scorecard.xml) is working, but the aggregated scorecard (scorecard_agg_leads.xml) is not.

[JohnHalleyGotway]

@briannen thanks for reloading the database on mohawk. I reran using the same XML file you sent to me but am seeing a very different result.

Here's my commands on mohawk:

cd /d2/personal/johnhg/METplus/discussions/2734
 ./mv_scorecard.sh scorecard_agg_leads.xml | tee scorecard_agg_leads.log

I do note that there are 272 log messages about "discarding" data:

grep discarding scorecard_agg_leads.log | wc -l
272

But the resulting scorecard image does not match yours at all. I did remove all the "intermediate" files in the data, plots, and scripts directories before running this to avoid "old" data impacting the new results.

So it still isn't obvious to me yet exactly what needs to be debugged.

I'd recommend that you...

1. Compare /d2/personal/johnhg/METplus/discussions/2734/scorecard_agg_leads.xml to the version of the scorecard_agg_leads.xml to make sure there aren't obvious diffs (other than paths).
2. Delete the working contents of the data, plots, and scripts directories.
3. Rerun scorecard_agg_leads.xml to regenerate test_scorecard_agg.png.

I mean on the same machine (mohawk), we should be able run the same command with the same inputs to get exactly the same result. Are you still getting all blue non-statistically-relevant results or do you see the same image I'm generating?

[georgemccabe]

Hi @briannen , as far as I can tell, your config is set up correctly to loop over each value in the FCST_LEVEL_LIST. Could you rerun the config setting LOG_LEVEL=DEBUG and send me the path to the log output? You can set this on the command line call to run_metplus.py by adding config.LOG_LEVEL=DEBUG.

[briannen]

@georgemccabe I'm running on casper, and the path is /glade/work/briannen/metplus_land/workflow/output/logs/metplus.log.20241023160319

[JohnHalleyGotway]

@briannen and @georgemccabe, I found the problem.

Looking in metplus.log.20241023160319, I definitely see that METplus is setting those forecast levels separately:

grep fcst_lev metplus.log.20241023160319 | egrep -v export | cut -d')' -f2 | sort -u
 DEBUG: Building out_stat filename from template: out_stat/{fcst_var}_{fcst_level}_{fcst_lead}_out_stat_SL1L2.stat
 DEBUG: METPLUS_FCST_LEVEL=fcst_level = ["R5"];
 DEBUG: METPLUS_FCST_LEVEL=fcst_level = ["R7"];
 DEBUG: METPLUS_FCST_LEVEL=fcst_level = ["R9"];

The trouble is that the STAT-Analysis config options are named fcst_lev and obs_lev on this line of STATAnalysisConfig_default, not fcst_level and obs_level that METplus is setting.

So METplus is setting those variables, and STAT-Analysis is happily ignoring them and using the default empty lists for fcst_lev and obs_lev, which tells it to NOT FILTER by that column of data.

Seems like a METplus bug to me that requires a fix.

FWIW, I did some fancy grepping of that log file and found the following STAT-Analysis config options being exported:

fcst_lead 
fcst_level 
fcst_var 
jobs 
model 
obtype

To confirm that fcst_level is the only one that's not named as expected.

[georgemccabe]

Yikes! I created bugfix issue #2742 to address this. @briannen, will you need a METplus v5.1.1 bugfix release created to include this fix? Or is obtaining this fix via the main_v5.1 branch or the upcoming v6.0.0 release sufficient?

[briannen]

@georgemccabe It can wait until the v6.0.0 release.

[bikegeek]

Issue created in METviewer: dtcenter/METviewer#552

…

On Tue, Oct 22, 2024 at 1:51 PM John Halley Gotway ***@***.***> wrote: @briannen <https://github.com/briannen> I'm a bit surprised that METviewer does not already support grouping data by lead time directly, without first having to manually aggregate the data in Stat-Analysis. I went hunting around and found this comment <dtcenter/METviewer#16 (comment)> from @TatianaBurek <https://github.com/TatianaBurek> for issue dtcenter/METviewer#16 <dtcenter/METviewer#16> from back in 2019 about this feature. Based on that comment, I did try grouping lead times like this in the scorecard XML: <columns> <field name="vx_mask"> <val name="NY" /> <field equalize="true" name="fcst_lead"> <val name="120000:240000" label="Day 1" /> <val name="360000:480000" label="Day 2" /> </field> </field> </columns> But that produces the following type of error messages: NameError: Derived curve can't be calculated. Multiple values for one valid date/fcst_lead So that definitely does not work. I'm wondering if that feature DID exist in the java version of METviewer but didn't survive the transition to Python. Regardless, I certainly do see the ability to group the input data during scorecard creation as an incredibly useful feature. @bikegeek <https://github.com/bikegeek> I'd recommend writing this up as a feature request since asking users to aggregate the data themselves ahead of time is a very high bar and very error prone. — Reply to this email directly, view it on GitHub <#2734 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHTNBUYKKVGOM7XJM6TZ42UETAVCNFSM6AAAAABQGCS2SGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBSGE2DGMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>