Calculating Accuracy

[kozacy]

Calculate the proportion of absolute errors between two datasets that are less than a specified threshold.

[kozacy]

hello，Could I ask you a question?How to calculate the temperature check accuracy with a threshold of 1, if the error is less than 1 correct, else wrong

Cloud i use
FCST_VAR1_NAME = TMP
FCST_VAR1_LEVELS = L2
OBS_VAR1_NAME = TMP
OBS_VAR1_LEVELS = L2
COV_THRESH_LIST = [1]

i am not sure, can i get some help，thank you very much

[j-opatz]

Hello,

And thank you for your question. To better understand your question, I'm hoping to clarify what your intended output/outcome is.

Are you attempting to read in multiple METplus output files and see a listing of all individual CTS lines where Accuracy was 1 (or less than 1, if you'd like the inverse)?

Any additional details you can provide about the goal of your verification are greatly appreciated.

[kozacy]

Thank you very much for your reply

I would like to enter two columns of data (fcst and obs) and calculate the proportion of the absolute value of the error that is less than a threshold (samples like an absolute value of the error of T2m less than 2 degrees Celsius are recorded as correct and divided by the total number T), can met implement this calculation ？and can the ACC inside the CTS be used to calculate what I need?

Best regards

[j-opatz]

Thank you very much for the clarification. The good news is that, through a circuitous route, your goal can be accomplished through METplus with two tool calls.

This path does require that both of your datasets (forecast and observational) are gridded. You can still accomplish your intended goal with point observations, but it would require slightly different tools.

The first step is to acquire a difference field where the observation field has been subtracted from the forecast field (or vice versa). If you use Grid-Stat and request the netCDF output, there is a forecast minus observation difference field that you can obtain:

GRID_STAT_NC_PAIRS_FLAG_DIFF = TRUE

It is not necessary to create any other output (unless your verification process requires it).

The next step is to take the netCDF output file that was created in the first step and use it as both the forecast AND observation fields in another Grid-Stat call. Doing so will create a perfect 1-to-1 match between the forecast and observation fields; this will allow you to set thresholds on both the observation and forecast fields to find the proportion of those matched pairs that fall within the threshold. This would look something like:

FCST_VAR1_NAME = DIFF_TMP_L2_TMP_L2_FULL
FCST_VAR1_LEVELS = "(*,*)"
FCST_VAR1_THRESH=>=-2&&<=2
OBS_VAR1_NAME = DIFF_TMP_L2_TMP_L2_FULL
OBS_VAR1_LEVELS = "(*,*)"
OBS_VAR1_THRESH=>=-2&&<=2

Note that the thresholds have to include the negative and positive values since there is not a current METplus method for calculating the absolute difference field. In the example I listed, this would effectively restrict statistical calculations to gridpoints where the values are greater than or equal to -2 and less than or equal to 2 (or ABS(2)).

Finally, request the CTS line type for output:

GRID_STAT_OUTPUT_FLAG_CTS = STAT

Once this output is created, you'll look for the PODY column, for probability of detection. From its definition, this will be the fraction of events that were correctly forecasted to occur. So all of those mathced pairs that fall within the thresholds for both fields (>=-2&&<=2) divided by the matched pairs that fall within and without the thresholds (i.e. all of the matched pairs).

Hopefully this answer is what you're seeking. If you're interested in reading more on statistics, we just introduced a basics statistical review online, in conjunction with our online tutorial. This might be a good source for information on statistics during your verification journey.

[kozacy]

Oh my god. what a genius idea, I will verify it with this method, thank you very much for your patience and detailed answer.