Masking of input data with pcp_combine

[mpm-meto]

Our radar data processing capability requires us to take 12 5-min slices and sum them up to create an hourly accumulation. This forms the simplest/most fundamental building block for our radar-based verification. There are elements of QC that apply and must be accounted for. The question is can pcp_combine do it all and what is the best way?

1. At present, only one missing 5-min file in the hour is allowed to be missing. If this is the case (it is rare) a file with all missing data must be returned. An hourly accumulation file with real values is returned only when >=11 5-min files are found. Crucially, in all instances an hourly file is returned.
2. The individual 5-min slices may not have the same coverage from one 5-min data frame to the next as radar coverage and radar-specific QC removes grid squares for a multitude of reasons. The spatial masking needs to also ensure that even if 12 time slices are present, hourly accumulation values are only produced for the grid squares where >=11 5-min values are available to begin with.

The vld capability, we think, should be able to cope with this, i.e. setting it to 11/12 (though we don't know if that is > or >=) should suffice and remove any grid squares (and hours) where more than 1 in 12 is missing, but we want to check and ask for advice/tips.

[CPKalb]

Hi:

The vld_thresh is applied at every grid point, so, this is what is recommended for the filtering in number 2 above. The vld_thresh is filtered using >=. I would recommend setting the vld_thresh to 0.9 for your case, as that sits between 10/12 and 11/12.

For number 1, what does your pcp_combine command look like? Having that information would be helpful for me to give advice on what would work best in this case.

Christina

[JohnHalleyGotway]

@mpm-meto and @CPKalb, you raise some interesting questions.

One clear rule I'll note is that if you list a specific file name on the command line, and that file does not actually exist, pcp_combine will error out stating that fact:

ERROR  : grd_file_type() -> file does not exist "bad_path.grib2"

I'm thinking through the logic you describe, and see how its a reasonable compromise for handling missing data in an operational setting. I do not think that the existing pcp_combine MET tool and METplus wrapper can be configured as-is to achieve the desired result. Instead we'll want some combination of updates to the tool or wrapper to handle it.

Let me tag @georgemccabe to give him an opportunity to weigh in.

One option we could consider is setting up a METplus Python wrapper to always pass exactly 12 files to each run of pcp_combine with the add option. If one of the expected input file names is missing, we could make the wrapper smart enough to substitute in a dummy file filled with missing data values. And we could use the set_attr_init, set_attr_valid, set_attr_lead, and/or set_attr_accum config options to manually define the metadata for that missing time slice right on the command line. Then set -vld_thresh 0.9 so that we only get accums for grid points with at least 11 of the 12 inputs being containing valid data.

This approach would always produce an output file since we'd never pass it a missing input file.

So @georgemccabe, do you have any recommendations for how to implement this sort of user-specific handling of missing data via METplus wrappers? Should it be a custom user-script... or a variation of the existing pcp_combine wrapper?

[georgemccabe]

There are a lot of different ways to handle this. A brainstorming session may be helpful to figure out the best (or at least acceptable) solution. Here are some of my initial thoughts:

At first this issue made me think of EnsembleStat and GenEnsProd wrappers, which inserts dummy file paths that are marked as "MISSING" into the text files that contain the list of files to process. I'm not sure something like that would help here since pcp_combine doesn't allow you to pass it a file that doesn't exist nor does it accept a text file that contains a list of files to process.

Implementing logic to create dummy files that contain missing data could work. The PCPCombine wrapper has specific logic to find the files to process when using the -add or -derive methods, so this solution would have to be callable from that wrapper as opposed to running a UserScript instance first, as it would be difficult to determine which files are expected before running the PCPCombine wrapper.

Another option could be to modify the pcp_combine MET tool to include an option that allows missing files to be passed to it. The METplus wrapper would still have to modified because it will currently not build a command if it can't find all of the files it needs for the -add method. I'm not familiar with the logic behind the -sum method, but maybe that could be updated to allow missing files?

[JohnHalleyGotway]

Thanks @georgemccabe for adding your thoughts. @mpm-meto, I think this would be a great topic for brainstorming during the next MetOffice METplus developer catchup meeting. Or if you'd rather brainstorm in a smaller group, we could look for a time to discuss.

At the end of this, it'd be nice to have a METplus use-case that demonstrates this logic. To get there, we'll likely need changes in MET, METplus, or a combination of the two. I imagine we'll want to try out some sample data. So if you have an hour or day's worth to offer, that could help drive the discussion.

[mpm-meto]

Thanks @georgemccabe @JohnHalleyGotway and @CPKalb! We're going to schedule a catch-up next week so let's put this on the agenda.

[JohnHalleyGotway]

Thanks for sending over the sample data @mpm-meto. FYI, I copied it over from the RAL ftp site into seneca:/d1/projects/METplus/discussions/2409.

[georgemccabe]

The enhancements for this have been implemented and will be available to test in the beta5 releases.

MET enhancements: dtcenter/MET#2886
METplus wrapper enhancements: #2609