Read Multilocatonal temporal Netcdf File

[pugazenthi002]

Hi Help Community,

Iam having gridded rainfall "mm/day" dataset for longer time period in NetCDF format as single with following headers, How could i read the input file
like
OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}/rain.nc
OBS_GRID_STAT_INPUT_TEMPLATE =

Results of ncdump -h rain.nc
netcdf rain_tn {
dimensions:
time = 27028 ;
lat = 129 ;
lon = 135 ;
variables:
double rain(time, lat, lon) ;
rain:units = "mm/day" ;
rain:long_name = "Rainfall" ;
rain:_FillValue = NaN ;
int time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:units = "days since 1950-01-01" ;
time:calendar = "proleptic_gregorian" ;
double lon(lon) ;
lon:axis = "X" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:_FillValue = NaN ;
double lat(lat) ;
lat:axis = "Y" ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:_FillValue = NaN ;

// global attributes:
:Conventions = "CF-1.7" ;
:references = "" ;
:comment = "" ;
:crs = "epsg:4326" ;
}

Suggest me ways to read the file
and I want to Compare the Accumulated Precipation Output (RAINC and RAINNC) of the WRF output with this daily Precipitation Data.

Whether I want to segregate the input files or is it possible to read the single file.

[JohnHalleyGotway]

Hello, I see you have a question about comparing precipitation from a NetCDF analysis file to data from WRF. And based on the header you sent, the NetCDF file appears to follow the Climate-Forecast convention (:Conventions = "CF-1.7" ;). Yes, you certainly should be able to extract single time slices from the file. There's no need to break it up into separate files for each time step.

This section of the MET User's Guide contains some details about setting up the configuration to read data from NetCDF files. But whenever getting going with a new data set in MET, I always recommend starting by running the plot_data_plane tool (see usage here). That tool extracts a single 2D slice of data and plots it. It's great for figuring out what configuration string you need to read the data and also confirms that MET is placing the data at the correct place on earth. You call to plot_data_plane will look something like this:

plot_data_plane rain.nc plot_rain.ps 'name="rain"; level="(@20150807,*,*)";'

Where you can replace that date @20150807 with some timestamp that's actually valid for your data formatted as:
YYYYMMDD, YYYYMMDD_HH, or YYYYMMDD_HHMMSS

In general, when reading data from a NetCDF file, you'll specify a level string that contains 1 "element" for each dimension of the variable you're reading. The * character goes in the gridded dimension slots (i.e. lat/lon) and for the other dimensions, you specify an integer index for that dimension (i.e. 0, 1, 2,...) or a value for the corresponding coordinate variable, starting with an @ as shown for the time above.

Next, you need to figure out if these daily rain rates are directly comparable to the output from WRF. If your WRF output is also a daily rain rate, great. If it's an accumulation over a certain number of hours, then you'll need to make them comparable. There are 2 things you should be aware of:

1. You can use the convert(x) option in the configuration string to convert the input data in simple ways. For example, divide by 24 to convert mm/day to mm/hr:

plot_data_plane rain.nc plot_rain.ps 'name="rain"; level="(20150807,*,*)"; convert(x) = x/24;'

2. The pcp_combine tool in MET can be run to add or subtract data. For example, if you're WRF output contains a running accumulation of precip over the entire run, you could subtract the 72-hour forecast minus the 48-hour forecast to get the 24-hours in between.

Not saying that you'll necessarily need to do this, just pointing out that it's an option if needed.

[pugazenthi002]

Thanks for the suggestions @JohnHalleyGotway !!!
And plot_data_plane worked nicely.

[georgemccabe]

Hi @pugazenthi002, I'll add that you need to put at least part of the file path into the OBS_GRID_STAT_INPUT_TEMPLATE variable. You can either put the entire path in that variable like this:

OBS_GRID_STAT_INPUT_DIR =
OBS_GRID_STAT_INPUT_TEMPLATE = {INPUT_BASE}/rain.nc

Or you can split up the directory and template like this:

OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}
OBS_GRID_STAT_INPUT_TEMPLATE = rain.nc

Also, please note that any time information must be included in the _TEMPLATE variable. For example, if the file was named rain_20240507.nc, you could set:

OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}
OBS_GRID_STAT_INPUT_TEMPLATE = rain_{valid?fmt=%Y%m%d}.nc

or

OBS_GRID_STAT_INPUT_DIR =
OBS_GRID_STAT_INPUT_TEMPLATE = {INPUT_BASE}/rain_{valid?fmt=%Y%m%d}.nc

[pugazenthi002]

This seems good But Iam facing an error written followed as,
LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG=2012053118
INIT_END=2012093018
INIT_INCREMENT = 1m

FCST_GRID_STAT_INPUT_TEMPLATE = {INPUT_BASE}/2012/wrfout_d01_{init?fmt=%Y-%m-%d_%H}:00:00

with this, forecast file formatted are
wrfout_d02_2012-05-31_18:00:00
wrfout_d02_2012-06-30_18:00:00
wrfout_d02_2012-07-30_18:00:00
wrfout_d02_2012-08-30_18:00:00

But i need 31st in 07 and 08 th month

Iam would be thankful for any clarifications regarding.

And Iam using the Docker- Metplus-5.1
whether i need to run any environmental file to use ncdump, as it is currently missing shared library with error
"ncdump: error while loading shared libraries: libhdf5.so.200: cannot open shared object file: No such file or directory"

[georgemccabe]

Hi @pugazenthi002,

It is tricky to get the last day of each month. There are 2 approaches I can think of to get this result.

1. Specify an explicit list of initialization times using INIT_LIST. Instead of setting INIT_BEG, INIT_END, and INIT_INCREMENT, set:

INIT_LIST = 2012053118, 2012063018, 2012073118, 2012083118

2. Set the initialization times to the first of each month, then use the shift keyword to offset the initialization time in your filename templates by 1 day:

INIT_BEG=2012060118
INIT_END=2012100118
INIT_INCREMENT = 1m

FCST_GRID_STAT_INPUT_TEMPLATE = {INPUT_BASE}/2012/wrfout_d01_{init?fmt=%Y-%m-%d_%H?shift=-1d}:00:00

I have also reached out to a colleague who may have run into this issue before to see if there is a more elegant solution.

[georgemccabe]

Hi @pugazenthi002,

I discussed this with @CPKalb and we agreed that using the shift keyword to get the last day of the month (solution 2 above) is likely the easiest and cleanest solution to implement.

Another (more complicated) option is to loop over each day and skip all days except to last day using the Skipping Times setting to skip processing specific initialization times. This variable is called SKIP_INIT_TIMES for version v6.0.0-beta1 or higher. Using v5.1 or earlier, it is only possible to skip specific valid times. Since your begin and end times don't include February, you could do something like this:

INIT_BEG=2012060118
INIT_END=2012100118
INIT_INCREMENT = 1d

SKIP_INIT_TIMES = "%d:begin_end_incr(1,29,1)", "%m%d:0530,0730,0830"

This says to skip all days from 1-29 and also skip month-day combinations of 05-30, 07-30, and 08-30. If you were running over a time range that included February, you would need to also include the month-day skips for the other months that have 31 days and handle the leap day appropriately depending on the year.

[georgemccabe]

Regarding the ncdump dynamic library error, I am getting the same behavior you described using the dtcenter/metplus:5.1.0 Docker image. Thank you for bringing this to our attention. You should be able to run ncdump if you first set the LD_LIBRARY_PATH environment variable.

export LD_LIBRARY_PATH=/usr/local/lib

We will look into preventing this error in future versions so that setting LD_LIBRARY_PATH is not needed.

root@02aca85d1f8c:/metplus# ncdump
ncdump: error while loading shared libraries: libnetcdf.so.18: cannot open shared object file: No such file or directory
root@02aca85d1f8c:/metplus# export LD_LIBRARY_PATH=/usr/local/lib
root@02aca85d1f8c:/metplus# ncdump
ncdump [-c|-h] [-v ...] [[-b|-f] [c|f]] [-l len] [-n name] [-p n[,n]] [-k] [-x] [-s] [-t|-i] [-g ...] [-w] [-Ln] file
[-c] Coordinate variable data and header information
[-h] Header information only, no data
[-v var1[,...]] Data for variable(s) ,... only
[-b [c|f]] Brief annotations for C or Fortran indices in data
[-f [c|f]] Full annotations for C or Fortran indices in data
[-l len] Line length maximum in data section (default 80)
[-n name] Name for netCDF (default derived from file name)
[-p n[,n]] Display floating-point values with less precision
[-k] Output kind of netCDF file
[-s] Output special (virtual) attributes
[-t] Output time data as date-time strings
[-i] Output time data as date-time strings with ISO-8601 'T' separator
[-g grp1[,...]] Data and metadata for group(s) ,... only
[-w] With client-side caching of variables for DAP URLs
[-x] Output XML (NcML) instead of CDL
[-Xp] Unconditionally suppress output of the properties attribute
[-Ln] Set log level to n (>= 0); ignore if logging not enabled.
file Name of netCDF file (or URL if DAP access enabled)

[pugazenthi002]

And Iam having another query regarding the observed data also..
Iam having around 400 stations of observed rainfall
suggest me better way to compare these with wrfout files.

[georgemccabe]

Hi @pugazenthi002,

If you are using MET v12.0.0-beta3 or higher, then you can read native WRF data directly into MET tools. See this GitHub issue for more details: dtcenter/MET#2547

You should set the file type to specify that the input file is a wrfout file by setting:
file_type = NETCDF_WRF;

If you are using an earlier version of MET, then you will have to use Python Embedding to read the native WRF files into MET.

If you are able to, I would suggest obtaining and running the beta version to see if you are able to read your data directly into MET. This would help us ensure that different flavors of WRF files are supported as expected. If you are unable to use this development version of MET, please let me know and I will try to track down an example Python Embedding script that can be used to read WRF that way.

[pugazenthi002]

Thanks @georgemccabe ,
At last everything going fine now.

As iam using dockers(Latest version), Its not supporting the file_type = NETCDF_WRF; ,
Soon i will try this ,with MET v12.0.0-beta3 installing in my machine.
But upto that Iam going with cdo to extract needed variables and running MET which feels better now.

If possible suggest be simpler guides or tutorials for performing simple statisticts and to understand the results of .stat files in general

[pugazenthi002]

If possible anybody suggest simpler ways or guide links for @#2574 (reply in thread)

[georgemccabe]

Hi @pugazenthi002,

1. Regarding the beta release:

The latest METplus beta release is available on DockerHub for testing purposes.

docker pull dtcenter/metplus:6.0.0-beta4

This image includes METplus wrappers v6.0.0-beta4 and MET v12.0.0-beta4. See DockerHub dtcenter/metplus for more information.

2. Regarding guides or tutorials:

The METplus Online Tutorial provides many examples of how to use METplus. The latest tutorial was developed using METplus v5.0, so it does not contain some of the more recently added features, but it does contain a lot of useful resources to help your understanding.

The METplus User's Guide contains ample information on how to configure and use the METplus wrappers to run the MET tools.

The MET User's Guide also contains ample information, including specifics relating to the capabilities of each tool and format of the output files. Please navigate to a specific MET tool, e.g. Grid-Stat, to learn more.

[pugazenthi002]

Hi guys,
Iam facing another typical error mapping,
Iam having montly rainfall for India,
As you suggested i went with plotting before getting that into gridstat
But i faced the following confusions with conditions of plotting,

1. I have converted daily data to monthly with cdo monsum command which sums up -999 also to (-99931=-30969)
   and plotted with
   plot_data_plane rain_2011_05_1.nc plot_rain_obs.ps 'name="rain"; level="(@20110531,,*)";'

2. And then I have converted the -30969 to miss value with cdo monsum -setctomiss,-30969, which converts -30969 to "-".
   Again plotted which gives,
   
   In both the images Mostly outside the country boundaries where no data is available (-999) is shown as grey colour.
   But when i replaced -30969 with -999 using cdo the plot appears as follows,
   
   Actually this is the plot i exactly needed
   Iam so confused with this plots ,
   Irrespective of the plots, Which type of filled in missing value is needed for MET grid stat
   Kindly suggest me...

[JohnHalleyGotway]

Hello, I see you have a question about the handling of missing data in MET. I see that you've been using CDO to process some NetCDF files and then plotting the result using MET's plot_data_plane tool.

When MET reads data from a NetCDF file, it checks the variable attributes (either _FillValue or missing_value) to figure out what value should be interpreted as having bad data. For example, this is from a NetCDF file created by MET's pcp_combine tool:

	float APCP_12(lat, lon) ;
    ...
		APCP_12:_FillValue = -9999.f ;

This indicates that values of -9999 should be interpreted as being missing or bad data. And -9999 is the value that MET uses internally to store bad data. You don't need to use that yourself. You just need to make sure that the _FillValue variable attribute really does indicate the actually missing data value for your data.

Hope that helps clarify.

[pugazenthi002]

Thanks for the clarification @JohnHalleyGotway
Being a beginner, iam facing little constrains to understand easier, But i will,
I have to know whether, is it possible to obtain output for each matching grids, If so, suggest me how to configured as i tried with
GRID_STAT_ONCE_PER_FIELD = True and False also.

[JohnHalleyGotway]

The Grid-Stat tool can be configured to verify one or more fields each time it runs. This METplus GRID_STAT_ONCE_PER_FIELD option controls the logic of the METplus wrapper to decide if it should run a single instance of Grid-Stat to verify all the fields at once... or run it once for each field.

If you're only verifying a single field of 12-hour precip, I don't really think it matters how this is set. I think I'd just set it to false.

When getting started with METplus, it might help to look at some existing example of verifying Precipitation. Hopefully those examples will help inform the configuration options you choose.