METplus - verifying grid to grid with varying thresholds

[bmatilla]

Hello,

I'm not 100% sure if this is a feature that is already built into MET, so I wanted to ask beforehand. I have 5km flash flood guidance (FFG) data that I am looking to verify against some model QPF data. I understand that for the METplus workflow, I'd need to specify a categorical threshold to generate probabilities of exceeding that certain threshold and that has worked for the work I've done so far. But because FFG values vary from point to point on the grid, is there a way to bypass the categorical threshold parameter and just calculate statistics on exceeding those FFG values? Ideally the workflow would be to use gen_ens_prod to generate the NMEP probabilities of exceeding FFG and then calculating the statistics with grid_stat, but perhaps I'm missing something along the way. Thanks for your consideration!

[JohnHalleyGotway]

@bmatilla I see you have a question about if/how MET can be used to define the probability of exceeding a flash flood guidance, where the FFG value varies from grid-point to grid-point. MET does indeed have the ability to define a threshold that varies from grid-point to grid-point but it's done in the context of climatologies. However, I don't see any reason why you couldn't use that functionality to accomplish what you'd like to accomplish.

Please take a look at this MET User's Guide Gen-Ens-Prod section about Climatology Data. If you supply a climatology mean field, as well as a climatology standard deviation, you can define thresholds relative to that climo distribution. For example, with climo data defined in the config file, you could specify a threshold as >CDP50, which means "greater than the 50th percentile of the climo distribution". And that threshold changes from grid-point to grid-point.

The climo mean and standard deviation are used to define a normal distribution at each point... centered on the climo mean value. So if you were to just supply the mean and standard deviation as the SAME gridded field of FFG's and use the >CDP50 threshold, it'd work as you'd expect. Of course the standard deviation values would NOT be accurate, but the 50th percentile will always be the mean value.

In fact, I'll note that this unit test for Gen-Ens-Prod is already supplying climo data and using this config file setting to define the probability of exceeding the 75th percentile climatological percentile for 10-m V-wind. You'd be doing something pretty similar.

The only awkward part really is defining a standard deviation field basically with dummy values.

Please give that some thought and let me know if that approach works well for you. Also, would you recommend any changes? Perhaps we could add more simple threshold option that does NOT require a climo standard deviation field and could just indicates whether the value is above or below the climo mean?

I'll also add that, based on dtcenter/MET#2924, the >CDP50 threshold type is being renamed to >FCDP50 or >OCDP50 for MET version 12.0.0 at the end of this year. MET is being updated to support both forecast-based (FCDP) and observation-based (OCDP) climatological inputs. And those documentation updates have already been made in the develop version of the MET User's Guide.

[bmatilla]

Hi @JohnHalleyGotway,

Thanks for writing back to me and for providing the level of detail on the capability. I'm glad to see that there is an option to do the verification by way of MET's climatology features. My hope is this will be a good starting point to work from. I'll give your ideas some thought on how I will go about implementing it as I'll have limited time to explore that in the next couple of weeks, but will check back with you once I can circle back to this. Also, thanks for the heads up on the threshold type updates for MET 12.0.0. It'll be good to know once the new version is made available and we're able to upgrade.