Stats changed from met/v11.0.2 meplus/v5.0.1 to beta3 testing (v12.0.0 and v6.0.0 beta3)

[Ho-ChunHuang-NOAA]

Perry has changed the module load to beta3 testing and applied the new environment for current EVSv1.0.8 configuration on the WCOSS2. I found the PointStat has been changed (both CTC and SL1L2) as the result of new environment setting, i.e., beta3. I would like to know whether it is normal due to some code/script changes in the beta3 environment.
beta3_evs.stats.aqm_bc.atmos.grid2obs_ozmax8.v20240329.stat.txt
evs.stats.aqm_bc.atmos.grid2obs_ozmax8.v20240329.stat.txt

[jprestop]

Hi @Ho-ChunHuang-NOAA. Thank you for your question. In a comparison run between MET v12.0.0-beta 3 and METplus v6.0.0-beta3 and MET v11.0.2 and METplus v5.0.1, can you please tell us if you see a decrease in matched pairs in the beta versions from the previous versions? If so, this is likely due to a bug that was discovered and fixed for METplus 5.0.2 (Bugfix: Define the order of the forecast variables numerically rather than alphabetically #2070). The beta version configuration is discarding "duplicate" observations while the 11.0.1 version is not configured to do so.

[Ho-ChunHuang-NOAA]

Hi, Julie: Not quite sure about what you said on "match pairs decrease". If looking at the first few row between two output, the CTC has 6 columns, 1st column has *the same total (like 1330, 1048, 425, .... for different regions)* between two outputs 2nd columns is FY-OY of 2x2 table It decreased from 11.0.2 to 12.0.0 beta3 (187 -> 134, 173->121, 65->37,...) 3rd column is FY_ON of 2x2 table It decreased from 11.0.2 to 12.0.0 beta3 (64 -> 37, 52 -> 30 24 -> 15,.....) 4th column is FN_OY of 2x2 table It increased from 11.0.2 to 12.0.0 beta3 ( 259->312, 247->299, 102->130,....) 5th column is FN_ON of 2x2 table It increased from 11.0.2 to 12.0.0 beta3 (820 -> 847, 576 -> 598, 234 -> 243,....) This is what I observed for the difference, both increase and decrease. Do you mean the changes observed are due to the fix of using duplicate obs? Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Tue, Apr 2, 2024 at 1:53 PM Julie Prestopnik ***@***.***> wrote: Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>. Thank you for your question. In a comparison run between MET v12.0.0-beta 3 and METplus v6.0.0-beta3 and MET v11.0.2 and METplus v5.0.1, can you please tell us if you see a decrease in matched pairs in the beta versions from the previous versions? If so, this is likely due to a bug that was discovered and fixed for METplus 5.0.2 (Bugfix: Define the order of the forecast variables numerically rather than alphabetically #2070 <#2070>). The beta version configuration is discarding "duplicate" observations while the 11.0.1 version is not configured to do so. — Reply to this email directly, view it on GitHub <#2534 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3BXMHZFCCIJWA4POPDY3LV77AVCNFSM6AAAAABFTUV5ZKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOBXGQ4DI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Ho-ChunHuang-NOAA]

Previous email uses the threshold > 50 as an example. Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958 On Tue, Apr 2, 2024 at 2:56 PM Ho-Chun Huang - NOAA Affiliate < ***@***.***> wrote:

Hi, Julie: Not quite sure about what you said on "match pairs decrease". If looking at the first few row between two output, the CTC has 6 columns, 1st column has *the same total (like 1330, 1048, 425, .... for different regions)* 2nd columns is FY-OY of 2x2 table It decreased from 11.0.2 to 12.0.0 beta3 (187 -> 134, 173->121, 65->37,...) 3rd column is FY_ON of 2x2 table It decreased from 11.0.2 to 12.0.0 beta3 (64 -> 37, 52 -> 30 24 -> 15,.....) 4th column is FN_OY of 2x2 table It increased from 11.0.2 to 12.0.0 beta3 ( 259->312, 247->299, 102->130,....) 5th column is FN_ON of 2x2 table It increased from 11.0.2 to 12.0.0 beta3 (820 -> 847, 576 -> 598, 234 -> 243,....) This is what I observed for the difference, both increase and decrease. Do you mean the changes observed are due to the fix of using duplicate obs? Ho-Chun -- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958 On Tue, Apr 2, 2024 at 1:53 PM Julie Prestopnik ***@***.***> wrote: > Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>. Thank you > for your question. In a comparison run between MET v12.0.0-beta 3 and > METplus v6.0.0-beta3 and MET v11.0.2 and METplus v5.0.1, can you please > tell us if you see a decrease in matched pairs in the beta versions from > the previous versions? If so, this is likely due to a bug that was > discovered and fixed for METplus 5.0.2 (Bugfix: Define the order of the > forecast variables numerically rather than alphabetically #2070 > <#2070>). The beta version > configuration is discarding "duplicate" observations while the 11.0.1 > version is not configured to do so. > > — > Reply to this email directly, view it on GitHub > <#2534 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALPHE3BXMHZFCCIJWA4POPDY3LV77AVCNFSM6AAAAABFTUV5ZKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOBXGQ4DI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[Ho-ChunHuang-NOAA]

HI, And for the SL1L2 column, Total, OABAR, OOABAR are the same FABAR, FOABAR, FFABAR changed between two output but for same regions. When OABAR and OOABAR are the same between two outputs, does it mean the number of obs used in v12.0.0 beta 3 and v11.0.2 are the same? i.e, no duplicate obs were used in v11.0.2. The difference here seems to be the number of forecast values used in two different version settings, yet we are using the same interpolation method for the two settings. Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958 On Tue, Apr 2, 2024 at 2:57 PM Ho-Chun Huang - NOAA Affiliate < ***@***.***> wrote:

Previous email uses the threshold > 50 as an example. Ho-Chun -- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958 On Tue, Apr 2, 2024 at 2:56 PM Ho-Chun Huang - NOAA Affiliate < ***@***.***> wrote: > Hi, Julie: > > Not quite sure about what you said on "match pairs decrease". > > If looking at the first few row between two output, the CTC has 6 > columns, 1st column has *the same total (like 1330, 1048, 425, .... for > different regions)* > > 2nd columns is FY-OY of 2x2 table > It decreased from 11.0.2 to 12.0.0 beta3 (187 -> 134, 173->121, > 65->37,...) > > 3rd column is FY_ON of 2x2 table > It decreased from 11.0.2 to 12.0.0 beta3 (64 -> 37, 52 -> 30 24 -> > 15,.....) > > 4th column is FN_OY of 2x2 table > It increased from 11.0.2 to 12.0.0 beta3 ( 259->312, 247->299, > 102->130,....) > > 5th column is FN_ON of 2x2 table > It increased from 11.0.2 to 12.0.0 beta3 (820 -> 847, 576 -> 598, 234 -> > 243,....) > > This is what I observed for the difference, both increase and decrease. > > Do you mean the changes observed are due to the fix of using duplicate > obs? > > Ho-Chun > > -- > > Ho-Chun Huang, Ph.D. > > Physical Scientist III, Contractor with Lynker in Support of > > NOAA/NWS/NCEP/EMC, U.S. Department of Commerce > > 5830 University Research Ct., Rm. 2792 > > College Park, MD 20740 > > ***@***.*** ***@***.***> > > 301-683-3958 > > > On Tue, Apr 2, 2024 at 1:53 PM Julie Prestopnik ***@***.***> > wrote: > >> Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>. Thank you >> for your question. In a comparison run between MET v12.0.0-beta 3 and >> METplus v6.0.0-beta3 and MET v11.0.2 and METplus v5.0.1, can you please >> tell us if you see a decrease in matched pairs in the beta versions from >> the previous versions? If so, this is likely due to a bug that was >> discovered and fixed for METplus 5.0.2 (Bugfix: Define the order of the >> forecast variables numerically rather than alphabetically #2070 >> <#2070>). The beta version >> configuration is discarding "duplicate" observations while the 11.0.1 >> version is not configured to do so. >> >> — >> Reply to this email directly, view it on GitHub >> <#2534 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ALPHE3BXMHZFCCIJWA4POPDY3LV77AVCNFSM6AAAAABFTUV5ZKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOBXGQ4DI> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

[JohnHalleyGotway]

@Ho-ChunHuang-NOAA I wanted to jump in on this discussion as well. I see that the TOTAL column values remain the same between the old and new runs... meaning we're using the same number of matched pairs for both versions. Julie had asked that question because last week a question from Steve Simon revealed a difference in how the duplicate_flag config option was being used. But with the same number of matched pairs, that's not what's going on here.

I took a closer look at the contingency table counts, both old and new, and observe the following...
The number of “observation yes/no counts” remains constant while the number of “forecast yes/no counts” do change!

COL: TOTAL FY_OY FY_ON FN_OY  FN_ON
OLD: 1330  187   64    259    820
NEW: 1330  134   37    312    847

Fcst yes count = FY_OY + FY_ON = 251 (OLD), 171 (NEW)
Fcst no count = FN_OY + FN_ON = 1079 (OLD), 1159 (NEW)
Obs yes count = FY_OY + FN_OY = 446 (OLD), 446 (NEW)
Obs no count = FY_ON + FN_ON = 884 (OLD), 884 (NEW)

This indicates to me that the point observation data remains unchanged since the obs yes/no counts remain the same in both versions. However since the forecast yes/no counts do differ between versions, that's where we should be paying more attnetion.

If possible, the next thing I'd do is rerun both versions for a single case with the output Matched Pair line type turned on (i.e. mpr = STAT;). That would enable us to look at the individual FCST and OBS values for each station. I expect the values in the FCST column to be different while I expect the OBS values to be the same. Once we confirm that, the next step is figuring out why the FCST values differ.

[Ho-ChunHuang-NOAA]

@JohnHalleyGotway Perry and I discussed on the possibility of different model fcst input to the PointStat. We decide to redo our EVS run on the AQM to assure our model fcst sources are the same. Please stay tune for the next update. Thank you for your help.

[Ho-ChunHuang-NOAA]

@jprestop @JohnHalleyGotway Perry found out that emc.vpppg runs and my NRT run used different source of AQM input that leads to the difference I reported in this discussion. After aligned our configuration, I repeated the comparison on both output on 04/05 and 04/07, and they have identical CTC and SL1L2 values. Thus, there will be no change of EVS AQM output switched to use met/v12.0.0 and metplus/v6.0.0 beta3. Please close this discussion.

[jprestop]

That's great news, @Ho-ChunHuang-NOAA. Thank you for letting us know. I'll mark this information as the answer and will lock this discussion. We appreciate Perry and you checking and following up with us.