Gaussian smoothing configuration option meanings

[Jeff-Duda]

I cannot figure out what the specific meanings are for the smoothing/interpolation settings for type GAUSSIAN (i.e., applying a Gaussian filter or Gaussian kernel density estimator to point or gridded data). I am familiar with the mathematical formula for a Gaussian KDE, and it only includes a single setting - the width of the window.

I'm using gen_ens_prod and wish to smooth the NMEP fields using a Gaussian smoother. The settings in the default configuration file show at least three settings for this operation:

nmep_smooth = { vld_thresh = 0.0; shape = CIRCLE; gaussian_dx = 81.27; gaussian_radius = 500; type = [ { method = GAUSSIAN; width = 1; } ]; }

So we have gaussian_dx, gaussian_radius, and nmep_smooth.type.width. The only documentation I can find for these options in the User's Guide is this:

The “gaussian_dx” entry specifies a delta distance for Gaussian smoothing. The default is 81.271. Ignored if not Gaussian method.
The “gaussian_radius” entry defines the radius of influence for Gaussian smoothing. The default is 120. Ignored if not Gaussian method.
The “gaussian_dx” and “gaussian_radius” settings must be in the same units, such as kilometers or degress. Their ratio (sigma = gaussian_radius / gaussian_dx) determines the Guassian weighting function.

But I don't know what any of this really means. Someone please spell this out for me.

[JohnHalleyGotway]

@Jeff-Duda I see you have questions about the Gaussian smoothing options in MET. I can speak to some aspects of this, but won't have much advise to offer about reasonable options or interpretation of results.

I can say that this was initially added to support the computation of surrogate severe forecasts based on storm reports from NOAA/SPC. More detail about that can be found in dtcenter/MET#1136. Burkley Twiest ([email protected]) advised on the initial functionality and @lindsayrblank is the NCAR scientist who worked on it at that time. But she no longer works for the DTC. Those default values you see:

   gaussian_dx     = 81.27;
   gaussian_radius = 120;

Are taken directly from the surrogate severe application.

As you've already found, here's the section of the User's Guide that discusses these config options. Your question is specifically about the nmep_smooth option in the gen_ens_prod tool. The maximum ensemble probability field is computed with these options... and then that resulting field is smoothed based on the the nmep_smooth settings. That smoothing is performed by the interp_gaussian_dp() function. It basically loops over all the grid points, and the value at each grid point is the weighted average of the nearby grid points where the weights are determined by the gaussian kernel parameters.

As for scientific advice on when to use these options, perhaps @CPKalb or @j-opatz could weigh in?

[Jeff-Duda]

Thanks for the response @JohnHalleyGotway. However, I don't think that really helps answer my question, which is more directed towards the practical meaning of those specific parameters in gen_ens_prod.

Just on my own I've tried playing with some of these settings, but the outputs are just not helping me understand how changing the configuration options should impact the output. For example, I got this error message when trying to run gen_ens_prod using gaussian_dx =120 and gaussian_radius = 500:

ERROR :
ERROR : The radius of influence (120) is less than the delta distance (500) for regridding method "GAUSSIAN".
ERROR :

I wasn't aware there were even such rules when computing a Gaussian smoother!

Unfortunately, since I can't attach images here I can't show a formula of what I'm talking about, but it is described textually in the panel on the left side of the top of this link: https://deepai.org/publication/gaussian-kernel-smoothing. My point is that the formula has a sig^2 in the denominator of the exponential, which is the only "width" or "spread" parameter I am expecting to have to specify to perform a Gaussian KDE in gen_ens_prod. But, as I indicated in my original post, there seem to be three separate parameters where I expect to see just the one. So I want to know how I should interpret all of those so I can have those three match the one that is expected from the formula.

[CPKalb]

I'm definitely not an expert on the Gaussian smoothing, but from my notes, I have the following information:

You can't set the Gaussian sigma value in MET. Instead, you set two variables that produce this sigma value. Specifically, the Gaussian sigma value (which is applied into Gaussian equation), is computed from the gaussian_radius and gaussian_dx set by the user. The gaussian_dx is specified as the dx of the grid we are going to. The number of points in the Gaussian filter is determined by the gaussian_radius. Specifically it tells MET how many neighboring points to use in the smoothing. The units of gaussian_dx and gaussian_radius need to be the same.

I've only ever used the surrogate severe Gaussian smoothing (which are the default values). But, from this information, if you know the dx of the grid you want to go to, and what sigma you want to specify, then the radius should be able to be derived, from the equation MET uses for computing sigma (sigma = gaussian_radius / gaussian_dx). MET has 1.47 as the default sigma which is 120 / 81.27

The width is not used for the Gaussian filtering process. Rather, it controls the neighborhood for for finding the max (as for NMEP). This should be computed by the dx of the from grid and the dx of the to grid. For example, the dx of the from grid is 3km and the dx of the to grid is 81, it should be 27 (=81/3). If the user then specifies a width less than 27, the NMEP values will be less than expected for many regridded cells. If the width is greater than 27, it returns higher values than what we would expect.

I think @hsoh-u did some of the work to implement this, so he may have more information as well.

Christina

[Jeff-Duda]

Thanks, @CPKalb, for the clarifications.

Regarding the "width" parameter, I am now looking for clarification on a unification on the radius of influence between the NEP and NMEP fields whose configuration settings are adjacent in the default control file (see below for an example):

//
// Neighborhood ensemble probabilities
//
nbrhd_prob = {
width = [ 7,11,15,21,31,51,101 ];
shape = CIRCLE;
vld_thresh = 0.01;
}

//
// NMEP smoothing methods
//
nmep_smooth = {
vld_thresh = 0.0;
shape = CIRCLE;
gaussian_dx = 50;
gaussian_radius = 1000;
type = [
{
method = GAUSSIAN;
width = 1;
}
];
}

I am under the impression that nbrhd_prob.width controls the radius of influence for both NEP and NMEP. Is that true? The outputs from running gen_ens_prod suggest it is. In that case, does nmep_smooth.type.width override nbrhd_prob.width if the former is anything other than 1? I do not understand your example about "from" grids and "to" grids with differing dx values. I am attempting to calculate NEP and NMEP fields from an ensemble that is all on the same grid, so the "from" and "to" grid are identical in my case. Please explain this further.

[CPKalb]

Hi Jeff:

I'm not sure exactly how nbrhd_prob.width and nmep_smooth.type.width are used in this case. I'm contacting another person over here to see if I can get further clarification on that, and I'll get back to you.

For the example of the from grid and the to grid. If I'm using the numbers given above (from grid dx is 3, and to grid dx is 81). If I select a width of less than 27 (81/3), then the neighborhood over which I'm computing the max (for nmep) is smaller than this ratio. So, some points will be excluded in the max. If I selected a width of greater than 27, then the max will be larger as some points will be included in multiple neighborhoods, which will cause the max to be larger. Does that help at all?

Christina

[Jeff-Duda]

From what I can gather, it seems like in the case that "from" and "to" grids are the same, then width=1 is fine and basically should never need to be changed unless for some reason you want to expand your neighborhoods to find NMEP beyond whatever nbrhd_prob.width is set to. Your discussion only seems to apply only to working with grids that are not the same. I don't think that will ever be the case for my usage of this tool.

Your previous discussion was mostly what I was looking for for an answer.

[hsoh-u]

The sigma (the weight of the center point being interpolated) is computed by radius / dx. If the sigma is less than 1, the weight can be increased as the distance is bigger.
The other issue is the Gaussian filter grid cells. This case is 1 by 1 filtering grid cell because the max_distance is 1 cell. The max_distance is not inclusive. If the max_distance (as grid cell count) is 5, the distance of 5 cells and more are excluded. The Gaussian filtering grid cells are 9 by 9 (horizontally/vertically -4 to 4 ) and sets the weight 0 if more than 5 grid cell distance (for example, cells in each corner: (-4,-4), (-4,4), (4,4 -4), (4,4)). The max distance is computed from gaussian_radius, gaussian_dx, and trunc_factor. The default trunc_factor is 3.5 (configurable at the Config file). The max_distance = the nearest integer of (radius / dx * trunc_factor) = nint(120 / 500 * 3.5) =nint(0.84) = 1.