Config for GenEnsprod

[perthsb]

Hi,
I am looking into ways to generate statistics for Ensemble run (using GenEnsProd and Ensemble stat) and would like to know whether Ens_VAR need to be setup separately for Control and member runs ? I tried a run, which could read control file variables but not able to read fields from member runs (they are same file types and same variable name).
Is shows error like :
ERROR :
ERROR : process_command_line() -> trouble reading ensemble file
ERROR :
Logfile and configure file located in Cactus :
/lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/EP4_Full/Metplus/output_GenEnsProd/logs/metplus.log.20230829163625
/lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/EP4_Full/Metplus/GenEnsprod_EP4a_TAOD.conf

Any help is appreciated.
Thanks
Partha

[georgemccabe]

Hi @perthsb,

I see you are having trouble reading ensemble files using GenEnsProd. Thank you for sending your log, config, and input files. I see that you have a duplicate variable ENS_VAR1_OPTIONS in your config file:

ENS_VAR1_NAME = AOD
ENS_VAR1_OPTIONS = file_type = NETCDF_NCCF
#ENS_VAR1_LEVELS = "(0,,)"
ENS_VAR1_LEVELS = ({lead?fmt=%H},,)
ENS_VAR1_OPTIONS = ensemble_flag = TRUE

I was able to read one of the files listed in the log using plot_data_plane using the file_type setting:

plot_data_plane ep4a/20190102/chm/p01/gocart.inst_aod.20190102_0600z.nc4 out.nc 'name="AOD"; level="(00,*,*)"; file_type = NETCDF_NCCF;'

If you need to set both file_type = NETCDF_NCCF and ensemble_flag = TRUE then you can combine them separated by semi-colon:

ENS_VAR1_OPTIONS = file_type = NETCDF_NCCF; ensemble_flag = TRUE

I also noticed that the level value you have specified references the forecast lead time, but the input files only contain a single time per file. Each run besides the 0 forecast lead will likely fail to read the data. I suggest setting the following:

ENS_VAR1_LEVELS = (0,*,*)

The correct files should be found for each run because the input filename template contains init and valid. The valid time is calculated by adding init and lead for each run.

Let me know if that solves your issue or if you need additional assistance.

[perthsb]

That is fine @georgemccabe .
For your convenience I put the logfile along with this message.
metplus.log.20230830153932.txt

[JohnHalleyGotway]

@perthsb thanks for passing along the METplus log file. The following lines jump out at me:

08/30 15:39:32.276Z metplus INFO: METPLUS_ENS_FIELD=field = [{ name="AOD"; level="(0,*,*)"; file_type = NETCDF_NCCF; ensemble_flag = TRUE; }];
...
08/30 15:39:32.277Z metplus INFO: COMMAND: /apps/ops/prod/libs/intel/19.1.3.304/met/11.0.2/bin/gen_ens_prod -v 4 -ens /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/EP4_Full/Metplus/output_GenEnsProd/stage/file_lists.20230830153932/20190102060000_0_gen_ens_prod.txt -out /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/EP4_Full/Metplus/output_GenEnsProd/gen_ens_prod/20190102_0600V_ens.nc} -config /apps/ops/prod/libs/intel/19.1.3.304/metplus/5.0.1/parm/met_config/GenEnsProdConfig_wrapped -ctrl /lfs/h2/emc/gefstemp/Bing.Fu/ep4keep/ep4a/20190102/chm/c00/gocart.inst_aod.20190102_0600z.nc4
...
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
ERROR  : 
ERROR  : process_command_line() -> trouble reading ensemble file "/lfs/h2/emc/gefstemp/Bing.Fu/ep4keep/ep4a/20190102/chm/p01/gocart.inst_aod.20190102_0600z.nc4"
ERROR  : 

I see that the file specified using -ctrl command line option is the one triggering the runtime error. And I also see that you're using the file_type = NETCDF_NCCF; config option. So I wonder if perhaps that file_type option is not propagating correctly to the reading of the -ctrl file? I'll test that out using some sample data here to see if that's a bug.

[perthsb]

Thanks @JohnHalleyGotway . Let me know if you need me to transfer any file to Hera to test them.

[JohnHalleyGotway]

@perthsb OK, I figured out what's going on. It's not a bug, just a somewhat confusing detail about configuring the Gen-Ens-Prod tool. On this line of code, I see that the file_type option is parsed from the ens ensemble dictionary in the configuration file. It is NOT parsed separately for each entry in the ens.field array. So while you did include it in your configuration file, it is buried too far down in the config file context to take effect.

Putting it in the name = ...; level = ...; file_type = ...; field string for Plot-Data-Plane does indeed work for that tool, as @georgemccabe noted. But since Gen-Ens-Prod requires all ensemble member input files to be the same file format, its parsed from the ens dictionary. Letting it change for each ens.field array entry would imply that the tool supports a mixture input file formats, which it does not.

The fix to your METplus config file should pretty easy:

1. Remove the file_type = NETCDF_NCCF; setting from ENS_VAR1_OPTIONS.
2. Add the GEN_ENS_PROD_INPUT_DATATYPE

GEN_ENS_PROD_INPUT_DATATYPE = NETCDF_NCCF

I checked and see that this logic is the same in Ensemble-Stat. So while it's a bit confusing, at least it's consistent. Please let me know how it goes.

[perthsb]

@JohnHalleyGotway I have tried with the change you have suggested, but still getting the error.
Attached is config and logfile.
metplus.log.20230831163243.txt
metplus_final.conf.20230831163243.txt
Thanks

[georgemccabe]

Hi @perthsb, it looks like the glossary entry is incorrect for setting ens.file_type. According to the code, you should set GEN_ENS_PROD_ENS_FILE_TYPE instead:

GEN_ENS_PROD_ENS_FILE_TYPE = NETCDF_NCCF

[perthsb]

Thanks @georgemccabe . I tried that, but getting error. Attached is conf and log file.
metplus.log.20230905211723.txt
metplus_final.conf.20230905211723.txt

[georgemccabe]

Hi @perthsb , I see some incorrect variable names in your final conf, e.g. GEN_ENS_VAR1_NAME, GEN_ENS_VAR1_LEVELS, etc.

Remove the following variables:

GEN_ENS_VAR1_NAME = AOD
GEN_ENS_VAR1_LEVELS = (0,*,*)
GEN_ENS_PROD_ENS_FILE_TYPE = NETCDF_NCCF; ensemble_flag = TRUE

and set the following:

ENS_VAR1_NAME = AOD
ENS_VAR1_LEVELS = (0,*,*)
ENS_VAR1_OPTIONS = ensemble_flag = TRUE
GEN_ENS_PROD_ENS_FILE_TYPE = NETCDF_NCCF

[georgemccabe]

Hi @perthsb, were you able to resolve this issue? Please let me know if you need more assistance.