Two more issues in running custom+pcp_combine+grid_stat

[lesnow93]

Hello again! I am continuing to make progress in my test case running pcp_combine and grid_stat in a multi-model, custom configuration. I am now successful in processing the multi-model set up, but there are 2 remaining problems occurring:

1. I specify BUDGET for interpolating APCP to the OBS grid to occur within grid_stat (so as to not have to run regrid_data_plaane), but the MET OUTPUT files are defaulting to NEAREST, appearing to ignore the settings I have in my config file:

GRID_STAT_REGRID_TO_GRID = OBS
GRID_STAT_REGRID_METHOD = BUDGET
GRID_STAT_REGRID_WIDTH = 2
GRID_STAT_REGRID_SHAPE = SQUARE
GRID_STAT_REGRID_VLD_THRESH = 0.5

2. I have my config file set up to loop through A06, A12, and A24 accumulated precip intervals. It successfully runs through pcp_combine for all the intervals; however, the job errors out after processing the A06 in grid_stat.

OBS_VAR1_NAME = APCP
OBS_VAR1_LEVELS = A06, A12, A24
FCST_VAR1_NAME = APCP
FCST_VAR1_LEVELS = A06, A12, A24

I attached the latest config file and a log file for only 2 of the custom loop members (for expediency in testing)
hiwat_multiModel_GridStat_APCP_v3.conf.txt
metplus_201803291800_v3.log
.

Sincerely,
Jonathan

[georgemccabe]

Hi @lesnow93, I see the following message in your log file:

DEBUG: Skip writing output /raid1/sport/people/casejl/METplus/hiwat_output/gridstat/2018032918/grid_stat_HKH1_APCP_vs_imergf_APCP_A06*20180330_000000V* because it already exists. Remove file or change GRID_STAT_SKIP_IF_OUTPUT_EXISTS to False to process

This means that you have the wrappers configured to skip running grid_stat because the output file already exists. This would explain why you aren't seeing the updated regrid method used in the output. Setting the following in your config file should resolve this issue:

GRID_STAT_SKIP_IF_OUTPUT_EXISTS = False

It looks like the same is true for PCPCombine, so you may also need to set:

PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS = False

Also, it looks like your log file contains many runs of the METplus wrappers, so it is difficult to see what was run. Could you generate a fresh log file with the above changes and send it to me if you are still having issues?

[lesnow93]

I thought I uploaded a fresh log file, so I apologize if it had extra junk in there. I did do a clean METplus run where I deleted all the pcp_combine and grid_stat output, but I still get NEAREST instead of BUDGET for the interpolation in the grid_stat output. In fact, I never have gotten the BUDGET interpolation in the MET .stat and other output files. I'll try to upload a cleaner log file Thursday.

[georgemccabe]

Hi @lesnow93, I see what is happening now. The NEAREST setting is coming from smoothing, which is controlled by the interp dictionary. Here is the default value:
https://github.com/dtcenter/MET/blob/main_v10.1/met/data/config/GridStatConfig_default#L153

interp = {
   field      = BOTH;
   vld_thresh = 1.0;
   shape      = SQUARE;


   type = [
      {
         method = NEAREST;
         width  = 1;
      }
   ];
}

You can override this value by setting GRID_STAT_INTERP_TYPE_METHOD = BUDGET.

[georgemccabe]

Hi @lesnow93, thank you for that info. Do I understand correctly that you only need to compute the accumulations that are less than or equal the current forecast lead? For example, for a 6 hour lead, you want to compute the 6 hour accumulation; for a 12 hour lead, you want to compute 6 and 12 hour accumulations, etc.

If that is the case, then we could enhance the PCPCombine subtract method logic to warn and skip if a negative accumulation is encountered instead of attempting to substitute the negative value and error when the file is not found. This seems like a reasonable enhancement and would not take very long to implement. I created issue #1905 to track this work.

There is a config option called INPUT_MUST_EXIST that can be set to False to prevent errors if the input files are not found. This is primarily used internally for our unit tests and is not fully supported in every wrapper. This option is rarely used otherwise because it may also prevent errors that should not occur from being reported, therefore hiding bigger issues.

[lesnow93]

That is exactly right, George! At any given forecast time, if an accumulation interval length exceeds the forecast hour, then the pcp_combine subtraction should not take place and thus, processing should not occur. That is the logic I had in my crusty old version of a python wrapper for computing APCP with pcp_combine/subtract.

[georgemccabe]

OK, sounds like a good change to incorporate into the PCPCombine wrapper. Do you have any other on-going issues from this thread?

[lesnow93]

I commented on your new thread #1905 to clarify that the metplus logic is generating negative forecast hours when constructing the filename, not a negative accumulation interval.

Other than that, I think this thread is done. Regarding the regridding within grid_stat, I was getting confused with the INTERP method column in the .stat files (thinking that was the REGRID method). So I'm convinced that the regridding is going well within grid_stat. It's just that there isn't any output in the .stat columns to indicate that the regridding went as planned.

Best,
Jonathan

[georgemccabe]

Thank you for commenting on the new issue. Feel free to add any other important details you think are appropriate there. I created the issue to keep it on our radar but did not yet fill in all of the relevant context. The example you provided with the negative filename was exactly what I was planning on adding later, so I appreciate you doing that!

I'm going to lock this discussion to prevent future posts. You can track the development progress through the issue and feel free to create a new discussion if you have any other issues or questions.

[georgemccabe]

To summarize, issue #1905 was created to handle when the desired accumulation interval is less than or equal to the forecast lead in PCPCombine subtract method more appropriately.