Metplus command to control ascii2nc hourly and daily input file processing

[Ho-ChunHuang-NOAA]

I have 2 types of ascii2nc command for execution.

One is hourly input files (hour 00 to hour 23) and one is daily input file
export MET_AIRNOW_STATIONS=${idir}/Monitoring_Site_Locations_V2.dat
ascii2nc ${idir}/${ifile} ${odir}/${ofile} -format "airnowhourlyaqobs"
ascii2nc ${idir}/${ifile} ${odir}/${ofile} -format "airnowdaily_v2"

How to provide the METplus command on the time information to process hourly and daily file?

What I can find from user's guide are;
ASCII2NC_INPUT_DIR=${idir}
ASCII2NC_INPUT_TEMPLATE={valid?fmt=%Y%m%d}/HourlyAQObs_{valid?fmt=%Y%m%d%H}.dat
ASCII2NC_INPUT_TEMPLATE={valid?fmt=%Y%m%d}/daily_data_v2.dat
ASCII2NC_OUTPUT_DIR=${odir}
ASCII2NC_OUTPUT_TEMPLATE={valid?fmt=%Y%m%d}/airnow_hourly_{valid?fmt=%Y%m%d%H}.nc
ASCII2NC_OUTPUT_TEMPLATE={valid?fmt=%Y%m%d}/airnow_daily_{valid?fmt=%Y%m%d}.nc
ASCII2NC_SKIP_IF_OUTPUT_EXISTS=False
ASCII2NC_FILE_WINDOW_BEGIN= ????
ASCII2NC_FILE_WINDOW_END= ????
ASCII2NC_WINDOW_BEGIN= or ????
ASCII2NC_WINDOW_END= or ????
ASCII2NC_INPUT_FORMAT="airnowhourlyaqobs"
ASCII2NC_INPUT_FORMAT="airnowdaily_v2

Which one and how to specify hourly and daily? if DATE=YYYYMMDD
For hourly?
ASCII2NC_FILE_WINDOW_BEGIN=${DATE}00
ASCII2NC_FILE_WINDOW_END=${DATE}23
For daily?
ASCII2NC_FILE_WINDOW_BEGIN=${DATE}
ASCII2NC_FILE_WINDOW_END=${DATE}

Is there a similar ASCII2NC METplus command for processing frequency?
VALID_INCREMENT = 1H
VALID_INCREMENT = 1d

What is the corresponding command for "export MET_AIRNOW_STATIONS=${idir}/Monitoring_Site_Locations_V2.dat"

Thank you.

[georgemccabe]

To set an environment variable in a use case, you will add the variable and value to the [user_env_vars] section of the METplus config file. See User Environment Variables for more info.

[user_env_vars]
MET_AIRNOW_STATIONS={ENV[idir]}/Monitoring_Site_Locations_V2.dat

If the filenames match the times you are trying to process, then you don't need to set ASCII2NC_FILE_WINDOW_BEGIN/END. The *_FILE_WINDOW_* variables are used to find files that are within a range of time around the valid time you are processing. The ASCII2NC_WINDOW_BEGIN/END variables specify the number of seconds before and after the valid time to process, which is the obs_window variable in the MET config file. If the files contain all of the data you want to process, then you don't need to set these.

If you are trying to run 2 instances of ASCII2NC in the use case, then you will need to use instance identifiers to specify different settings for each call to ASCII2NC. See Instance Names in Process List for more information. Example:

[config]
PROCESS_LIST = ASCII2NC(daily), ASCII2NC(hourly)

ASCII2NC_INPUT_DIR={ENV[idir]}

[daily]
ASCII2NC_INPUT_TEMPLATE={valid?fmt=%Y%m%d}/daily_data_v2.dat
...

[hourly]
ASCII2NC_INPUT_TEMPLATE={valid?fmt=%Y%m%d}/HourlyAQObs_{valid?fmt=%Y%m%d%H}.dat
...

If you are using METplus v5.0.0, then you should be able to set the VALID_INCREMENT differently for each instance. If not, then you would either have to process each hour and set ASCII2NC_SKIP_IF_OUTPUT_EXISTS to prevent redundant runs of the hourly data OR you would have to make 2 separate calls to run_metplus.py with different config files.

[Ho-ChunHuang-NOAA]

George,
Thank you for the instruction.

Is my understanding correct that for METplus 4.x.x I need to run a loop each processes an hour of input data ? e.g., hourly input file are

HourlyAQObs_2023012200.dat
HourlyAQObs_2023012201.dat
HourlyAQObs_2023012202.dat
HourlyAQObs_2023012203.dat
HourlyAQObs_2023012204.dat
HourlyAQObs_2023012205.dat
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
HourlyAQObs_2023012221.dat
HourlyAQObs_2023012222.dat
HourlyAQObs_2023012223.dat

declare -a hourlist=( 00 01 02 .....23 )
export DATE=my_value
for hour in "${hourlist[@]}"; do
    [config]
        VALID_TIME_FMT = %Y%m%d%H

        VALID_BEG = ${DATE}${hour}
        VALID_END = ${DATE}${hour}
        '''''''
    [dir]
        ::::::::::::;;;;;
    [filename_templates]
        ASCII2NC_INPUT_TEMPLATE={valid?fmt=%Y%m%d}/HourlyAQObs_{valid?fmt=%Y%m%d%H}.dat
        :::::::::::::::::

    ${METPLUS_PATH}/ush/run_metplus.py

done

In users guide "5.10. Setting Config Variables with Environment Variables"
in script
export LOCAL_ENV_VAR=my_value

Then I should use METPLUS_MY_VAR = {ENV[LOCAL_ENV_VAR]} in my syntax.

Thus, in the example above, should I use
VALID_BEG = {ENV[DATE]}{ENV[hour]} ?

[georgemccabe]

Hi @Ho-ChunHuang-NOAA, you do not need to loop over the hours in your script. You can change the valid increment to 1 hour. If you are reading the date from the DATE environment variable, then you would set the following to process each hour of that day.

[config]
LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = {ENV[DATE]}00
VALID_END = {ENV[DATE]}23
VALID_INCREMENT = 1H

[Ho-ChunHuang-NOAA]

Hi George,

Thank you for clarification. You can close this discussion.