failure to process new chemistry fields such as NO, NO2, NOX, NOY, and PM10

[edwardstrobach]

Good morning,

Last week I began creating netcdf files called prepbufr.${field}.${YYYYMMDD}.nc. These files are created automatically each day and look nearly identical to the netcdf files created during the actual pb2nc step. I went through each field that is written during pb2nc and modeled the approach of creating these files to resemble is closely as possible to the pb2nc tool. Using a routine I created a while back to create MET-friendly netcdf files, which has proven to work just fine for ozone and PM2.5, I then added forecast output to the netcdf files to be processed during the point_stat step.

Because I'm bringing down data from the amazon server and processing them into netcdf files that resemble the structure of the netcdf files created during pb2nc, I have removed the pb2nc step when processing stats for these particular fields. Thus, the processing really begins with point_stat. The exiting message indicates that everything was processed successfully. however, the point_stat files are not populated and thus the stat analysis files are, too, not populated. The log file (/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210830145146) indicates that the forecast field has been correctly identified, but that there appears to be a level mismatch with observations.

When surveying the files created by pb2nc more closely, I find that there is nothing that is specified about what the level should be. I looked closely at those files in hopes of determining the level so that I can add any level information required during the point_stat process. I assumed A1 for NO (and other new chemistry fields) given that the ozone and PM2.5 also use these level specifiers. However, from the document attached you can see that these were rejected. How do I go about figuring out what the level should be so that the observations from the files that I have created can be correctly selected and processed alongside the foreacst output?

Level_mismatch_new_chemistry_fields.docx

[jprestop]

Hi @edwardstrobach. Thank you for your question.

In order to look at the data in one of the prepbufr files more easily we converted it into a format that matches the ASCII2NC format by running pnt2ncascii.R:
Rscript /usr/local/met/share/met/Rscripts/pntnc2ascii.R prepbufr.no.20210826.nc > prepbufr.no.20210826.txt

Looking at the first line in the file:
PANDORA 000020301 20210826_020000 46.4783 -63.9869 NA NO 0 0 0 0.1
I can see that the level and height fields (after the variable name of "NO") are both equal to 0. Given that information, we used a level of Z0 for the vertical level.

That got rid of the level mismatch, but pushed all 1878 observations into being rejected for message type. You had:
export POINT_STAT_MESSAGE_TYPE="[\"\ONLYSF\"]"
when we know it should actually be the following based on the information in the prepbufr file:
export POINT_STAT_MESSAGE_TYPE="[\"\PANDORA\"]"

Unfortunately, the output log information was showing a message type of "NO" (String Value = "NO") instead of "PANDORA" even after this change was made. Looking in your PointStatConfig_NO configuration file, we saw that message_type was hard-coded to:
message_type = [ "NO" ];
so the value of POINT_STAT_MESSAGE_TYPE was not being used. We changed it to be:
message_type = ${POINT_STAT_MESSAGE_TYPE};
and then we saw some matched pairs.

I hope that helps!

[edwardstrobach]

Thank you for the excellent description! This helps a lot. I'll go ahead and follow your findings and apply them here shortly.

…

On Tue, Aug 31, 2021 at 3:36 PM jprestop ***@***.***> wrote: Hi @edwardstrobach <https://github.com/edwardstrobach>. Thank you for your question. In order to look at the data in one of the prepbufr files more easily we converted it into a format that matches the ASCII2NC format by running pnt2ncascii.R: Rscript /usr/local/met/share/met/Rscripts/pntnc2ascii.R prepbufr.no.20210826.nc > prepbufr.no.20210826.txt Looking at the first line in the file: PANDORA 000020301 20210826_020000 46.4783 -63.9869 NA NO 0 0 0 0.1 I can see that the level and height fields (after the variable name of "NO") are both equal to 0. Given that information, we used a level of Z0 for the vertical level. That got rid of the level mismatch, but pushed all 1878 observations into being rejected for message type. You had: export POINT_STAT_MESSAGE_TYPE="[\"\ONLYSF\"]" when we know it should actually be the following based on the information in the prepbufr file: export POINT_STAT_MESSAGE_TYPE="[\"\PANDORA\"]" Unfortunately, the output log information was showing a message type of "NO" (*String Value = "NO"*) instead of "PANDORA" even after this change was made. Looking in your PointStatConfig_NO configuration file, we saw that message_type was hard-coded to: message_type = [ "NO" ]; so the value of POINT_STAT_MESSAGE_TYPE was not being used. We changed it to be: message_type = ${POINT_STAT_MESSAGE_TYPE}; and then we saw some matched pairs. I hope that helps! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1120 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALO6JVDPCESTE3QJ2ZTJXJTT7UVLTANCNFSM5DCD3AAQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.