Difference in plot values - summary versus aggregation

[briannen]

I am trying to create a lead series plot in METviewer and noticed that the median value is different when plotting just the median vs. including confidence intervals. Both plots are showing the PODY of CAPE.

Specifically, if you look at the 12 hour lead time, the point is at 0.7 on the median plot and 0.6 on the confidence interval plot. Is there supposed to be such a large difference between the two?

I've included the R data files from METviewer as well as the plots.

plot_cape_pody_median.txt

plot_cape_pody_cis_with_bootstrapping.txt

[TatianaBurek]

Each time you do bootstrapping you will get different results because of the random nature of the bootstrap sampling. To remove the randomness you need to introduce Bootstrapping seed. With the seed the sampling will be the same for all executions on your data.
If you use web app, put an integer in the 'Bootstrapping seed' field.
If you use the batch method, introduce the seed in the corresponding tag:
<boot_random_seed>1</boot_random_seed>

[JohnHalleyGotway]

@TatianaBurek I agree that bootstrapping is a random process. Each time you do it, the different replicates you get will yield somewhat different confidence intervals. In general, you want to select a large enough number of replicates so that the resulting intervals change very little from run to run. And setting the <boot_random_seed> tag is the way to make the results repeatable...

BUT, the bootstrapping process affects only the confidence intervals and not the basic statistics around which those CI's are defined. So I definitely agree with @briannen. At the 12-hour lead time, the median value of changing from 0.7 to 0.6 is a problem, as are the values for several other lead times. The median value of the original sample should remain the same whether you're plotting CI's or not.

@briannen can you please attach the XML files for these plots? We should inspect them to confirm that there isn't some difference in the configuration that would explain the difference in results.

[briannen]

Sure! These XMLs are before I tried setting the boot_random_seed tag, so I will be updating that later on.

plot_cape_pody_median.xml.txt

plot_cape_pody_cis_with_bootstrapping.xml.txt

[JohnHalleyGotway]

@briannen I did some hunting around on METviewer and think I found the XML's from 20210812 that demonstrate this difference:
plot_20210812_135632.xml.txt
plot_cape_pody_bootstrapping_only_1.xml.txt

Looking at these XML's, I am no longer concerned about there being a problem in METviewer. Instead, this is a great example of the difference between the computation of "summary" statistics and "aggregated" statistics. While both are very reasonable ways of processing/plotting the data, they are different and should be interpreted differently.

For summary, the dot on the plot is the median of the statistic computed separately for each case. If plotting 30 days of data, you'd have 30 PODY values and the dot is the median of those 30 PODY stats. This indicates the typical "daily" performance.

For aggregation, the dot on the plot is a single PODY value computed from an aggregated contingency table. If plotting 30 days of data, you sum up 30 contingency tables into one large one and then compute a single PODY value from it. This indicates the overall, aggregated performance.

When plotting the median, you can add CI's based on the "standard error". When plotting the aggregated value, you can add CI's using the bootstrapping method.

In general the median of the daily stats will NOT be the same as the aggregated statistic. Hope this helps clarify.

[briannen]

Thanks John! I was thinking it was probably a statistical thing that I was misunderstanding rather than a METviewer problem.