output file name does not reflect to the valid time format in stat_analysis

[weiwilliam]

I use stat analysis (METplus v5.1.0) to read data through a Python embedded script.
Here is the configs I set up to control my output folder and name template.

LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2024060100
VALID_END = 2024060118
VALID_INCREMENT = 6H
LEAD_SEQ = 6
...
STAT_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/../Data/output/stats_test
STAT_ANALYSIS_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H}.out

I want my output file to be named with a 10-digit valid time, however, it always names ALL.out at 00Z (06, 12, and 18Z are fine).
Does anyone know why the name file does not reflect the template at 00Z?

[JohnHalleyGotway]

@weiwilliam I see you're having issues getting the Stat-Analysis METplus wrapper to configure the output file name how you'd like.

I did adapt/run this StatAnalysis Wrapper Example to test. I updated the StatAnalysis.conf file to set:

STAT_ANALYSIS_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H}.out

And it did indeed write to the expected output file name:

(run_util.py) INFO: COMMAND: /Volumes/d1/projects/MET/MET_development/MET-develop/bin/stat_analysis -v 2 -lookin /Volumes/d1/projects/MET/MET_development/MET-develop/out/grid_stat -config /Volumes/d1/projects/METplus/METplus-develop/parm/met_config/STATAnalysisConfig_wrapped -out /Volumes/d1/projects/METplus/METplus-develop/METplus_2636/stat_analysis/2005080712.out

Inspecting the code in stat_analysis_wrapper.py, I note the ALL is used in general as a replacement for the wildcard (*) character, as seen on this line of code.

You're setting obviously doesn't include a * character, but when you run METplus with multiple input configuration files (i.e. multiple -c command line opts), then settings in later config files override those in earlier ones. So its possible that your intended setting is getting overridden by downstream config files.

I'd recommend re-running METplus, and look in the log output for the path to metplus_final.conf. That contains the final configuration that was applied. For example, in my run, I see that it really does contain:

STAT_ANALYSIS_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H}.out

Can you confirm it's present in your final config file as well? Or does the final have something like *.out instead... which would be modified by the wrapper to be ALL.out?

[weiwilliam]

Thank you, @JohnHalleyGotway
Yes, I confirm the TEMPLATE is still STAT_ANALYSIS_OUTPUT_TEMPLATE = f006/{valid?fmt=%Y%m%d%H}.out in final_conf.
I compared 00Z with other cycles and I could not find major differences.
It quite puzzles me.

Here are two sets of final conf and log files.
00Z cycle:
metplus.log.20240712204914.txt
metplus_final.conf.20240712204914.txt

12Z cycle:
metplus.log.20240712204935.txt
metplus_final.conf.20240712204935.txt

[JohnHalleyGotway]

@weiwilliam, the lead developer for the METplus wrappers, @georgemccabe is out of the office on paternity leave. I've played around a bit and have some information to share, but my apologies, that I doubt we'll settle on the solution as quickly as George would!

First, let me point you to this documentation about StatAnalysis Timing. It's often the case that users run Stat-Analysis to aggregate results through time. So by default, it only runs once using all of the input times defined in the configuration file.

This default behavior can be overridden using the STAT_ANALYSIS_RUNTIME_FREQ config option, whose options are defined here.

I tested using these settings:

LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG=2005080700
VALID_END=2005080718
VALID_INCREMENT = 6H

And found that METplus ran Stat-Analysis only once. After I added the following, it ran 4 times:

STAT_ANALYSIS_RUNTIME_FREQ=RUN_ONCE_PER_INIT_OR_VALID

Once each for 00, 06, 12, and 18.

I notice that the METplus-5.1 documentation DOES NOT list this as a valid option. However testing the v5.1.0 release, it does work as I describe. So hopefully its just a documentation issue.

Does using that option produce any different results?

[weiwilliam]

@JohnHalleyGotway Thank you, it is fine. I'm glad you jumped in. Now I just renamed the file after.
Yes, I did consider to run it once with all the cycles.
Do you know how to set up to skip any cycles?
The file I generate is unavailable at every 06Z, so it complains no file and exits when it calls my python embedded script.
That's why I run it cycle by cycle.

[JohnHalleyGotway]

The documentation for the develop branch of METplus (i.e. slated for the next METplus-6.0.0 release) does mention this SKIP_VALID_TIMES option. There's many uses of this. Once good example is to suppress running on Saturdays and Sundays. But you should be able to use it to skip the 6Z run.

In METplus 5.1, the config option was named simply SKIP_TIMES. So I think the logic has is being updated for the next release to differentiate between skipping valid times versus skipping initialization times.

Regardless, in my testing, I could not actually use either option to successfully skip the 06Z run:

STAT_ANALYSIS_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID
STAT_ANALYSIS_SKIP_TIMES = "%H:6"
STAT_ANALYSIS_SKIP_VALID_TIMES = "%H:6"

I tested both with the v5.1.0 release and the develop branch, along with several variations of what to skip. And even when using STAT_ANALYSIS_SKIP_VALID_TIMES for the develop branch, I still see this run in the log file:

(config_util.py) INFO: ****************************************
(config_util.py) INFO: * Running METplus StatAnalysisWrapper
(config_util.py) INFO: *  at valid time: 2005-08-07 06:00
(config_util.py) INFO: ****************************************

So the answer is that yes, we intend to provide robust, flexible support for skipping user-specified runs... but it doesn't seem to be working in Stat-Analysis as I'd expect.

[CPKalb]

Hi @weiwilliam:

I've done a lot of work with Stat-Analysis so am adding some information here that may help get these settings working as desired.

For the time labeling issue at 00Z, Stat-Analysis has inconsistent behavior when using the init and valid time templates. This is because it is designed to aggregate statistics over time, so often there isn't a single init or valid time for a Stat-Analysis run. To mitigate this, several additional tags can be used in the time templates when running Stat-Analysis. These additional time template tags are listed in the documentation.

I would suggest changing the time template in STAT_ANALYSIS_OUTPUT_TEMPLATE to use one of these special tags and see if that fixes the timing issue. So, you could change STAT_ANALYSIS_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H}.out to STAT_ANALYSIS_OUTPUT_TEMPLATE = {valid_end?fmt=%Y%m%d%H}.out. This template will tell Stat-Analysis to label the file using the the valid time at the end of the aggregation period for each Stat-Analysis run. There are many other tags we can use if this doesn't produce the desired labeling. Does this fix the issue or does it still label with ALL?

For timing controls, Stat-Analysis has two options, the STAT_ANALYSIS_RUNTIME_FREQ and the LOOP_LIST_ITEMS and GROUP_LIST_ITEMS variables. From the metplus_final configuration files, it looks like you have FCST_LEAD_LIST in the LOOP_LIST_ITEMS. Are you wanting to run stat analysis once for each lead time and for the 0, 12, and 18 UTC valid times (skipping 6Z)? If so, you could set the following:

FCST_VALID_HOUR_LIST = 1, 12, 18
LOOP_LIST_ITEMS = FCST_LEAD_LIST,FCST_VALID_HOUR_LIST

This should run stat analysis once fore each lead time in LEAD_SEQ and for the 1, 12, and 18 valid times. For this setup, you can omit the STAT_ANALYSIS_RUNTIME_FREQ variable. Is this how you want Stat-Analysis to run or should it be doing something different?

Christina

[CPKalb]

Hi @weiwilliam:

Getting a 3 digit lead hour from the Stat-Analysis wrapper is problematic at this time and something we need to address. If you are not processing any lead times that are 100 hours or more, you could set f0{fcst_lead_hour}, and that should give f006. If you are processing lead times of 100 hours or longer, there is another way to get around this limitation, but it's more complicated.

For missing data, it sounds like you won't always know what observation files will be missing before running Stat-Analysis? I'm checking on whether Stat-Analysis can be run with an empty object and I'll let you know once I have an answer.

Christina

[CPKalb]

Hi @weiwilliam:

It sounds like you just want to run Stat-Analysis once over the entire time period, combining the lead and valid times? If so, then setting the following should accomplish this:

LOOP_LIST_ITEMS =
GROUP_LIST_ITEMS = FCST_LEAD_LIST,FCST_VALID_HOUR_LIST

This will group together the lead and valid times rather than looping over them. Does that do what you need?

Christina

[weiwilliam]

I see. I can modify my script to create folders, like f06 and f120.
I guess {fcst_lead_hour} will give 2 digits below 100 and 3 digit above, right?

Yes or no, it could be random missing, but my script will generate the MPR data first.
So, I will know which cycle's MPR data is unavailable before I run Stat-Analysis.

[CPKalb]

Yes, {fcst_lead_hour} will give a 2 digit lead time below 100 and a 3 digit lead time above. You can do what you mentioned above and modify your script to create folders like f06, etc. The other option to handle this is to create a configuration section, so that you can have separate filename templates for the 2 digit lead times versus the 3 digit lead times. It would require 2 calls to Stat-Analysis in your process list, one for each of these sections. I have done this many times and can help set it up if you don't want to modify the folder names.

[CPKalb]

Also, you can give Stat-Analysis an empty object. If you set the following in your python embedding script:
mpr_data = []

This will send empty data to Stat-Analysis. You will get a warning in the log file which will look something like the following, but Stat-Analysis will still run. Also, you will get an empty output file from Stat-Analysis.
WARNING:
WARNING: do_job_aggr_stat() -> no matching STAT lines found for job: -job aggregate_stat -model WRF -fcst_valid_hour 120000 -obtype ADPUPA -line_type MPR -by FCST_VAR -out_stat /glade/derecho/scratch/kalb/METplus_testing/met_tool_wrapper/StatAnalysis_python_embedding/WRF_ADPUPA_valid20070331_fcstvalidhour120000Z_out_stat.stat -out_line_type SL1L2
WARNING:

Christina

[weiwilliam]

@CPKalb @JohnHalleyGotway
Thank you for all your help and suggestions.
I finally found the solution,

VALID_BEG=2024060100
VALID_END=2024060218
VALID_INCREMENT = 6H
LEAD_SEQ = 6

STAT_ANALYSIS_OUTPUT_TEMPLATE = f0{lead_hour}/{valid?fmt=%Y%m%d%H}.out

MODEL1_STAT_ANALYSIS_LOOKIN_DIR = python [embedded script] Data/output/hofx/f0{lead_hour}/hofx_obs.tempo_no2_tropo-wxaq.{valid?fmt=%Y%m%d%H}.nc4

FCST_LEAD_LIST = {LEAD_SEQ}
LOOP_LIST_ITEMS = FCST_LEAD_LIST

I also pass the empty list of mpr_data which can generate output file without any stat line inside.

It works great now.
Thank you both again.

[CPKalb]

Great! I'm going to mark this discussion as answered and close it to prevent future posts. If you have additional questions, feel free to start a new discussion.

Christina