Input data

[Banias1305]

Hello,

My name is John , I am on a thesis on the WRF assessment for 2021 for my postgraduate studies as a Meteorologist.
My problem in running MET10.1.1 is that my WRFPRS data is given in a daily (96 hour forecast) format for each month of '21, so I have too many files and lists. I am trying to run the input data ( as you can see in the following example) as a directory but I can't. Can I have some help with this please?

Translated with www.DeepL.com/Translator (free version)
CODE:
[st13@node002 MSLP]$ point_stat /home/st13/WRFPRS/ mslp.nc confi -obs_valid_beg 20210101_000000 -obs_valid_end 20211231_240000
DEBUG 1: Default Config File: /opt/MET_10.1.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: confi
ERROR :
ERROR : check_grib() -> unable to read header from input file "/home/st13/WRFPRS/"
ERROR :
[st13@node002 MSLP]$
DATA FORMAT:
[st13@node002 WRFPRS]$ ls
APR21 AUG21 DEC21 FEB21 JAN21 JUL21 JUN21 MAR21 MAY21 NOV21 OCT21 SEP21
[st13@node002 WRFPRS]$ cd APR21
[st13@node002 APR21]$ ls
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
[st13@node002 APR21]$ cd 01
[st13@node002 01]$ ls
WRFPRS_d01.000 WRFPRS_d01.010 WRFPRS_d01.020 WRFPRS_d01.030 WRFPRS_d01.040 WRFPRS_d01.050 WRFPRS_d01.060 WRFPRS_d01.070 WRFPRS_d01.080 WRFPRS_d01.090
WRFPRS_d01.001 WRFPRS_d01.011 WRFPRS_d01.021 WRFPRS_d01.031 WRFPRS_d01.041 WRFPRS_d01.051 WRFPRS_d01.061 WRFPRS_d01.071 WRFPRS_d01.081 WRFPRS_d01.091
WRFPRS_d01.002 WRFPRS_d01.012 WRFPRS_d01.022 WRFPRS_d01.032 WRFPRS_d01.042 WRFPRS_d01.052 WRFPRS_d01.062 WRFPRS_d01.072 WRFPRS_d01.082 WRFPRS_d01.092
WRFPRS_d01.003 WRFPRS_d01.013 WRFPRS_d01.023 WRFPRS_d01.033 WRFPRS_d01.043 WRFPRS_d01.053 WRFPRS_d01.063 WRFPRS_d01.073 WRFPRS_d01.083 WRFPRS_d01.093
WRFPRS_d01.004 WRFPRS_d01.014 WRFPRS_d01.024 WRFPRS_d01.034 WRFPRS_d01.044 WRFPRS_d01.054 WRFPRS_d01.064 WRFPRS_d01.074 WRFPRS_d01.084 WRFPRS_d01.094
WRFPRS_d01.005 WRFPRS_d01.015 WRFPRS_d01.025 WRFPRS_d01.035 WRFPRS_d01.045 WRFPRS_d01.055 WRFPRS_d01.065 WRFPRS_d01.075 WRFPRS_d01.085 WRFPRS_d01.095
WRFPRS_d01.006 WRFPRS_d01.016 WRFPRS_d01.026 WRFPRS_d01.036 WRFPRS_d01.046 WRFPRS_d01.056 WRFPRS_d01.066 WRFPRS_d01.076 WRFPRS_d01.086 WRFPRS_d01.096
WRFPRS_d01.007 WRFPRS_d01.017 WRFPRS_d01.027 WRFPRS_d01.037 WRFPRS_d01.047 WRFPRS_d01.057 WRFPRS_d01.067 WRFPRS_d01.077 WRFPRS_d01.087
WRFPRS_d01.008 WRFPRS_d01.018 WRFPRS_d01.028 WRFPRS_d01.038 WRFPRS_d01.048 WRFPRS_d01.058 WRFPRS_d01.068 WRFPRS_d01.078 WRFPRS_d01.088
WRFPRS_d01.009 WRFPRS_d01.019 WRFPRS_d01.029 WRFPRS_d01.039 WRFPRS_d01.049 WRFPRS_d01.059 WRFPRS_d01.069 WRFPRS_d01.079 WRFPRS_d01.089
[st13@node002 01]$

[JohnHalleyGotway]

Hello @Banias1305, I see that you're new to METplus and are looking for some guidance on how to run to process multiple WRF output files. The Point-Stat tool is designed to be run once for each forecast time to be evaluated.

As you can see in the Point-Stat usage statement, on the command line, you tell it what forecast file, verifying observation, and ascii configuration file you'd like to use.

So you'll want to run it once for each of the WRFPRS_d01.### output files that exist.
And I'd recommend approach it this way...

1. Pick a single output file to start with and decide which observation data you'd like to use for verification.
2. If its a prepbufr file, you'll pre-process it using the pb2nc tool. For ascii point obs, you'll likely run ascii2nc.
3. Configure and run Point-Stat once manually, for a single output time. Reading through the Point-Stat Chapter of the User's Guide, you'll decide what variable/levels you'd like to verify, what observation message types should be used, what masking regions you want, what output line types you want.
4. Once you've decided how you like to configure Point-Stat, the next is switching over to METplus.
5. You'll setup a METplus use case to loop over all your model initializations and forecast lead times. In this way, METplus is a like a scripting layer to make many calls to the MET tools (perhaps PB2NC and Point-Stat in your case) for many output times.

So the first question to consider is what are your verifying observations?

[Banias1305]

First im thankful for your email.I will on surface like RH, T2M, MSLP and finally Prec . I chose to valuated through pointstatConfig mpr and cnt as output_flag.  And this is a script i created in the photo.What do you think sir? Στάλθηκε από το Ταχυδρομείο Yahoo σε Android Στις Τρί, 14 Φεβ, 2023 στις 20:23, ο χρήστηςJohn Halley ***@***.***> έγραψε:

[JohnHalleyGotway]

I don't see a photo attached. Perhaps that file attachment wasn't accepted by GitHub? If you rename an ascii file with a .txt extension, GitHub should let you add that as an attachment.

[Banias1305]

#!/bin/bash

Define the range of time steps

for i in $(seq -f "%03g" 6 96 3)
do

Create a directory for each time step

mkdir out/$i

Loop through each month of 2021

for m in $(seq -f "%02g" 1 12)
do
# Loop through each day of the month
for d in $(seq -f "%02g" 1 31)
do
# Skip days that don't exist in certain months
if [ $m == 2 ] && [ $d -gt 28 ]
then
continue
elif [ $m == 4 ] || [ $m == 6 ] || [ $m == 9 ] || [ $m == 11 ] && [ $d -gt 30 ]
then
continue
fi

  # Run the point_stat command and redirect output to the corresponding directory
  point_stat /home/st13/WRF_M/$m/$d/WRFPRS_d02.0$i /home/st13/MSLP/mslp.nc /home/st13/FILE_CONG/confi > out/$i/mslp_$i_$m-$d.out
done

done
done

[JohnHalleyGotway]

@Banias1305 you are of course welcome to run the Point-Stat tool directly via this shell script. The METplus python wrappers (website, repo, user's guide) are provided to meet this need, to prevent each user from needing to script up calls to the individual MET tools.

But since it looks like you're comfortable with bash scripting, that's a fine way to proceed. I don't really see any actual questions to answer yet. But perhaps some questions will come up along the way about the Point-Stat configuration file. If so, and reading the user's guide and searching through discussions doesn't answer it, just let me know and I'm happy to take a look.

Looking at your script so far I'd recommend:

1. Using the -outdir command line option to instruct Point-Stat where to write its output files.
2. Using the -log command line option to write log messages to a log file.

[Banias1305]

Im trying to run each forecast time step for each day of 2021 and finally to create a dir for each of fcst time step where it include point_stat_out for each day at the same step (f.e. at WRFPRS_d02.012).

[Banias1305]

I answered through github chat and not through email now. I copy paste the script. Στάλθηκε από το Ταχυδρομείο Yahoo σε Android Στις Τρί, 14 Φεβ, 2023 στις 21:09, ο χρήστηςJohn Halley ***@***.***> έγραψε: I don't see a photo attached. Perhaps that file attachment wasn't accepted by GitHub? If you rename an ascii file with a .txt extension, GitHub should let you add that as an attachment. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Banias1305]

I understand sir and im thankful for your time. May i ask about Prec, is there a way to let MET know through pointstatConfig file that at certain UTC i have 6-hourly and 12-hourly so it could calculate at the same time for FCST data?

[JohnHalleyGotway]

Yes, precipitation accumulation intervals are often a headache. I don't understand the exact details you're dealing with so I can't give you an easy, exact answer. But hopefully the following information will be helpful.

• Since your files are named WRFPRS_d01.f###, I assume that you're running them through the UPP post-processor software. So they are GRIB1 or GRIB2 files. If that's the case, then I believe that UPP can be configured to control when the precipitation accumulation bucket is "dumped". But I don't know the details.

  • You could configure it to dump the bucket every 6 hours, if that's the accumulation interval used for your precip observations.
  • Or you could configure it to dump a runtime accumulation... so that each file contains the total accumulation from 0 to the current forecast hour.

• The pcp_combine tool in MET is designed for this purpose. It processes precip from multiple input files and computes the desired output accumulation interval. For example, it can add up 4 6-hour accumulations into a 24-hour one. Or it can subtract the runtime accumulation from two files to compute how much fell in the time between.

All this logic should be based on the availability of your precip observations. If possible, configure UPP so that the forecast accumulations line up with the observation data. If not, run pcp_combine prior to running Point-Stat to adjust the forecast accumulation intervals as needed. If you do the latter, you'll need to run Point-Stat twice... once for the precip from the NetCDF output of pcp_combine and once for all the other variables in the GRIB file.

Hope that helps clarify.

[Banias1305]

My data (WRFPRS) had already been used through UPP , but my observations were not. My Obs Prec data is 6hour (00-12UTC) and 12hour (06-18UTC), so im trying to figure out how i can edit my data so i can run point_stat without a problem. Should I reformat my WRFPRS data to 6hour and 12hour in the same UTC (if that is possible) or first, via a fortran code convert my obs to 6hour and then the WRFPRS data to 6hour ?
Sorry to ask this, but this is my first use of MET or Linux.
MSLP: I managed to run point_stat for jan21 for each time step (006-096) for each day , now I am trying to use the stat_analysis tool to gather the output of point_stat for each time step.

[JohnHalleyGotway]

@Banias1305 wondering if any questions remain on this discussion or can we mark is as being "answered"?

[Banias1305]

Hi sir,       Just one last thing,if is possible. What kind of statistics you would suggest me to use through pointstatConfig_file for Prec data?      I would like to thank you for your time and for your valuable information. I am grateful to you . Ioannis 

[JohnHalleyGotway]

Typically, precipitation forecasts are verified using categorical statistics. So you'd defined one or categorical thresholds using the cat_thresh option. For example, setting cat_thresh = [ >0, >=2.54, >=12.7, >=25.4 ]; would define events of interest as any precip amount (>0), a trace amount (>=2.54 mm = 0.1 inches), a 1/2 inch (>=12.7 mm = 0.5 inches), or a full inch (>=25.4 mm = 1.0 inch). Of course, it's up to you define thresholds values that are consistent with the units of your data.

After defining cat_thresh, make sure that you request the CTC and CTS output be written (ctc = BOTH; and cts = BOTH;).

The categorical statistics in the CTS line type, like GSS for Gilbert Skill Score, are very commonly used in the evaluation of precip. And those statistics will be computed separately for each cat_thresh threshold entry you define.

[Banias1305]

Thank you a lot Sir for this discussion. I appreciate the time you spend. Thnx !