STAT_ANALYSIS valid isn't work right

[weiz1995]

PROCESS_LIST = StatAnalysis
LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d
VALID_BEG = 20230407
VALID_END = 20230412
VALID_INCREMENT = 1H
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = PATH/{valid}*.stat

the valid in lookin dir is giving me all the days instead of the days between valid_beg and valid_end
Also, did some tests, the valid as variable just not be recognized at all. 
Any ideas why this is happening? 

[CPKalb]

Hi:

It looks like your {valid} doesn't have a format. These date and time templates in METplus need to be followed with a format specification, to tell METplus what format the date or time is in. For example, if your valid date is formatted as YYYYmmdd, you likely want something like
MODEL1_STAT_ANALYSIS_LOOKING_DIR = PATH/{valid?fmt=%Y%m%d}*.stat.

Depending on how your valid date or time is formatted, you may need something different after the ?fmt=. Some information on these templates is located here.

However, Stat-Analysis can be a bit tricky to get the dates correct. Depending on what you are doing, you may need to use a special key for your date. The special keys are available only to Stat-Analysis. These keys also need to be followed with a date format, in parentheses similar to the above.

How are you wanting to run Stat-Analysis?

Christina

[weiz1995]

I did try put in format into valid however it still not recognized the variable.

Where is the valid date set at? from valid_beg and valid_end?

I have over hundreds thousands stat files that I need to run though with Stat-Analysis, and it tooks long time to run. So I want to narrow down the files by giving config file a template to looking for the files that only between the valid_beg and valid_end days. And the stat files have day (yyyymmdd) on the filename
The version I am running is v4.1
Thanks

[CPKalb]

Hi:

Stat-Analysis had some major upgrades in METplus version 5.0. So, you may need to pull version 5.0 to get all the updates.

Depending on how your date is formatted, it may be tricky to narrow down by exact begin and ending dates. But, you can easily narrow down the files it's searching to simply use data for something like a month. When you are aggregating over a time period, there isn't a single valid time for the dataset. In this case, Stat-Analysis doesn't know what to use for valid, and so a special key is needed for the date. I'm guessing you would want one of the following: fcst_valid_beg, fcst_valid_end, obs_valid_beg, or obs_valid_end. Without seeing the rest of your configuration file, I don't know which one would work best.

Here is an example using special keys to narrow down searching for stat analysis. In this case, the script searches by model initialization time (rather than valid time which is what you want), using the year and month, with a * in place of the day. There are actually two directories specified here with the comma in between them. It looks for model initialization times from the previous month (shifted -2592000 seconds) and the current month.
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = {OUTPUT_BASE}/cosmo_stats/point_stat/{init_end?fmt=%Y%m?shift=-2592000}*, {OUTPUT_BASE}/cosmo_stats/point_stat/{init_end?fmt=%Y%m}

What are the date and time formats of your files or directories? There are multiple ways to handle filtering of dates as input to Stat-Analysis. But, it would be helpful to have more information on how your files and directories are structured to help get the filtering set up correctly. In the example above, are you wanting to run Stat-Analysis once and aggregating all data between 20230407 and 20230412, with Stat-Analysis only reading the input files between 20230407 and 20230412?

Christina

[weiz1995]

PROCESS_LIST = StatAnalysis
LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d
VALID_BEG = 20230407
VALID_END = 20230412
VALID_INCREMENT = 24H
FCST_LEAD_LIST = begin_end_incr(1,12,1) 
LEAD_SEQ = begin_end_incr(0,23,1)
FCST_VALID_HOUR_LIST = begin_end_incr(0,23,1)
STAT_ANALYSIS_CONFIG_FILE = PATH/STATAnalysisConfig_default
MODEL1 = CLOUD
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = PATH/YYYY/MM/*{fcst_lead?fmt=%H%M%S}*{valid?fmt=%Y%m%d}*{fcst_valid_hour?fmt=%H%M%S}*.stat
MODEL1_OBTYPE = ANALYS
STAT_ANALYSIS_OUTPUT_TEMPLATE = {MODEL1}_S{VALID_BEG}_E{VALID_ED}_{MODEL1_OBTYPE}_L{fcst_lead?fmt=%H}_OUT_STAT.stat 
STAT_ANALYSIS_OUTPUT_DIR = PATH
STAT_ANALYSIS_INPUT_TEMPLATE = grid_stat_CLOUD_{fcst_lead?fmt%H%M%S}L_{valid?fmt%Y%m%d}_{fcst_valid_hour?fmt%H%M%S}V.stat
STAT_ANALYSIS_JOB_NAME = summary
STAT_ANALYSIS_JOB_ARGS = -line_type CTS -by fcst_lead,fcst_thresh,fcst_valid_hour
LOOP_LIST_ITEMS = FCST_LEAD_LIST

Filename example: grid_stat_leadtime(HHMMSS)L_yyyymmdd_validhour(HHMMSS)V.stat

this is my current metplus config

I only want to run stat-analysis against all the data that is in the range valid_beg and valid_end so I can reduce the run-time of stat-analysis

[CPKalb]

Hi:

We made numerous updates to the Stat Analysis functionality so you may need METplus version 5 for this to work well.

As far as I know, there isn't a simple way to filter input files to an exact date range, without listing each date in the lookin dir separated by commas. I'm checking with a few other people over here to see if they have other solutions and I can get back to you on that.

In the meantime, you could filter the data by month, which should significantly cut down on the time it takes for Stat-Analysis to run. To do that, you would want to set:
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = PATH/YYYY/MM/*{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m}*{fcst_valid_hour?fmt=%H%M%S}*.stat
And then you will also need to add:
STAT_ANALYSIS_FCST_VALID_BEG = {fcst_valid_beg?fmt=%Y%m%d_%H%M%S}
STAT_ANALYSIS_FCST_VALID_END = {fcst_valid_end?fmt=%Y%m%d_%H%M%S}

If the data you want to filter is not in the same month and year, then there would need to be additions to account for that.

Christina

[CPKalb]

Hi:

At this time, the only way to filter the Stat-Analysis input to METplus between a valid_beg and valid_end time is to list each directory that you want to read in specifically in the lookin dir.

MODEL1_STAT_ANALYSIS_LOOKIN_DIR = PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d?shift=-5d}{fcst_valid_hour?fmt=%H%M%S}*.stat, PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d?shift=-4d}{fcst_valid_hour?fmt=%H%M%S}*.stat, PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d?shift=-3d}{fcst_valid_hour?fmt=%H%M%S}*.stat, PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d?shift=-2d}{fcst_valid_hour?fmt=%H%M%S}*.stat, PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d?shift=-1d}{fcst_valid_hour?fmt=%H%M%S}*.stat, PATH/YYYY/MM/{fcst_lead?fmt=%H%M%S}{fcstvalid_end?fmt=%Y%m%d}{fcst_valid_hour?fmt=%H%M%S}*.stat

If the number of days between your valid_beg and valid_end changes, you would need to add or remove the entries in the lookin directory accordingly.

Christina

[weiz1995]

Thank you, I think I have some kind ideas moving forward

[CPKalb]

This is something we would like to upgrade so it's easier to do moving forward.

Christina