Python Embedding for plot_data_plane and MTD

[anwe5]

Hi,

now that the issue with the python version is solved (see discussion), I'm finally trying to apply MTD on my data.

However, I encountered two issues with the Python embedding script that I created to use the data with MET:

1) plotting data with plot_data_plane:
When creating a plot using plot_data_plane with my data and the Python embedding, the variable field appears upside down. I attempted to adjust the grid description variables in the Embedding script, but this only changed the projected borders, not the variable field.

Below is an example with temperature data. On the left is the plot generated with plot_data_plane, and on the right is the output from ncview. The plotted area is Germany + the Alps, which are clearly visible in the temperature field. Unfortunately, in the plot from plot_data_plane, the Alps have decided to moove to Northern Germany! :D

The embedding script I used for plot_data_plane is embedding_T_2M.py, and the example file is called test.nc. I run plot_data_plane with the following command: plot_data_plane PYTHON_XARRAY output_tas/test_03.ps 'name="embedding_T_2M.py";' -v 5

2) use mtd on the data:
When I run MTD with the Python embedding, I receive the following error message:

DEBUG 2: regridding, if needed ...
DEBUG 3: Use the grid defined by string "rotlatlon Nx Ny lat_ll lon_ll delta_lat delta_lon true_lat_south_pole true_lon_south_pole aux_rotation".
DEBUG 3: Grid Definition: Projection: Rotated Lat/Lon Nx: 0 Ny: 0 rot_lat_ll: 0.000 rot_lon_ll: -0.000 delta_rot_lat: 0.000 delta_rot_lon: 0.000 wrapLon: false true_lat_south_pole: 0.000 true_lon_south_pole: -0.000 aux_rotation: 0.000
ERROR  :
ERROR  :
ERROR  :   DataHandle::set_size() -> bad size
ERROR  :

The embedding script is called embedding_MTD.pyand a sample file for the hourly data is hourly_data_00.nc. I've also attached the MTD log file and the configuration file.

I start MTD with the following command: mtd -single /scratch/b/b382378/test_data_MTD/hourly_data_* -config config_test -outdir -log mtd.log output_mtd/ -v 5

I've tried to look into what's causing the errors, but I haven't been able to find the reason. I experimented with different values for 'true_lat_south_pole' and 'true_lon_south_pole', and 'aux_rotation', but it didn't change temperature field.

Thanks in advance for your help! I really appreciate your time and support in helping me resolve these issues!

Best regards,
Annabell

files.zip

[j-opatz]

Hi Annabell,

And thank you for your questions (and providing the necessary input for testing). I'll reply to each of your questions in turn.

1) plotting data with plot_data_plane:

This flip is caused by how MET's Python Embedding reads in data. Unlike some data input systems where gridded data is read from the lower leftmost point to the upper rightmost point, MET's Python embedding will read in a gridded field from the upper leftmost point to the lower rightmost point. This can result in data being flipped from north to south, in a manner you saw in your plot-data-plane call.

To correct for this, add the following line to your script, after the data field is copied into met_data:

met_data = met_data[::-1]

This will flip the north-south orientation and produce the results you are expecting.

2) use mtd on the data:

This error was caused by your usage of the to_grid setting in the regrid dictionary. Looking at your MTD configuration file, I see:

regrid = {
   to_grid    = "rotlatlon Nx Ny lat_ll lon_ll delta_lat delta_lon true_lat_south_pole true_lon_south_pole aux_rotation";

When defining the grid that regrid will use using a grid specification string, you as the user need to provide the definition values of the verification grid. In your example, I think you want to use a Rotated Plate Carrée (i.e. Rotated Lat/Lon) grid given that your to_grid string begins with "rotlatlon" (which is correct). But instead of then providing those necessary values to define what Rotated Plate Carrée projection to use, you mistakenly list the character strings that are used as stand-ins for the actual values.

As you may have seen in the MET User's Guide section for Grid Specification Strings, you need to use the to_grid setting to assign the values of

• Nx
• Ny
• lat_ll
• lon_ll
• delta_lat
• delta_lon
• true_lat_sp
• true_lon_sp
• aux_rotation

after declaring the "rotlatlon" projection.

As an example of how to properly use the to_grid setting for a grid specification string, take a look at the first example in the User's Guide under Lambert Projection. Assume that a user wishes to use a Lambert projection. The documentation states they would need to use the following string:

lambert Nx Ny lat_ll lon_ll lon_orient D_km R_km standard_lat_1 [ standard_lat_2 ] N|S

A user might fill in the required information using the following values:

to_grid = "lambert 614 428 12.190 -133.459 -95.0 12.19058 6367.47 25.0 N";

As the documentation indicates,

As an example of specifying a Lambert grid, suppose you have a northern hemisphere Lambert grid with 614 points in the x direction and 428 points in the y direction. The lower left corner of the grid is at latitude 12.19 degrees north and longitude 133.459 west. The orientation longitude is 95 degrees west. The grid spacing is 12.19058 km. The radius of the Earth is the default value used in many grib files: 6367.47 km. Both standard parallels are at 25 degrees north.

Please let us know if you have any further issues with this verification process, or if any of the above information isn't clear.

[anwe5]

Thank you very much for your response!

The issue with plot_data_plane has been resolved, and the projection now looks correct.

However, I am still having issues with the MTD tracking:

I adjusted the to grid entry to match the grid in my data as follows:

to_grid    = "rotlatlon 302 308 -4.75 -4.075 0.025 0.025 -40.0 10.0 0.0";

When I run MTD with this adjustment, I encounter the following error:

DEBUG 2: regridding, if needed ...
DEBUG 3: Use the grid defined by string "rotlatlon 302 308 -4.75 -4.075 0.025 0.025 -40.0 10.0 0.0".
DEBUG 3: Grid Definition: Projection: Rotated Lat/Lon Nx: 302 Ny: 308 rot_lat_ll: -4.750 rot_lon_ll: 4.075 delta_rot_lat: 0.025 delta_rot_lon: 0.025 wrapLon: false true_lat_south_pole: -40.000 true_lon_south_pole: -10.000 aux_rotation: 0.000
DEBUG 5: Conv data range is -4.62534e-17 to 4.51089
DEBUG 5:
DEBUG 2: Splitting object field
DEBUG 2: Done splitting
DEBUG 2: Calculating 3D fcst single attributes
DEBUG 2: Calculating 2D fcst attributes
DEBUG 2: Creating 2D constant-time slice attributes file: "output_mtd//mtd_20090101_003000V_2d.txt"
DEBUG 2: Creating 3D single simple attributes file: "output_mtd//mtd_20090101_003000V_3d_single_simple.txt"
DEBUG 2: Creating NetCDF file: "output_mtd//mtd_20090101_003000V_obj.nc"
ERROR  :
ERROR  :
ERROR  :   bool write_nc_grid(NcFile &, const Grid &) -> unsupported projection type
ERROR  :

My understanding was that I needed to specify the to grid entry to the "rotlatlon"-projection, since my data is on that grid and I’m using Python embedding. But I don’t actually want to regrid the data.

When I try setting to grid = NONE, I encounter this error:

DEBUG 2: regridding, if needed ...
DEBUG 3: Grid Definition: Projection: Rotated Lat/Lon Nx: 302 Ny: 308 rot_lat_ll: -4.750 rot_lon_ll: 4.075 delta_rot_lat: 0.025 delta_rot_lon: 0.025 wrapLon: false true_lat_south_pole: -40.000 true_lon_south_pole: -10.000 aux_rotation: 0.000
DEBUG 5: Conv data range is -4.62534e-17 to 4.51089
DEBUG 5:
DEBUG 2: Splitting object field
DEBUG 2: Done splitting
DEBUG 2: Calculating 3D fcst single attributes
DEBUG 2: Calculating 2D fcst attributes
DEBUG 2: Creating 2D constant-time slice attributes file: "output_mtd//mtd_20090101_003000V_2d.txt"
DEBUG 2: Creating 3D single simple attributes file: "output_mtd//mtd_20090101_003000V_3d_single_simple.txt"
DEBUG 2: Creating NetCDF file: "output_mtd//mtd_20090101_003000V_obj.nc"
ERROR  :
ERROR  :
ERROR  :   bool write_nc_grid(NcFile &, const Grid &) -> unsupported projection type
ERROR  :

Should I be specifying the to_grid option in this case, or set it to NONE if I don’t want to regrid?

[j-opatz]

I'm glad to hear the first fix worked as intended.

As for the second fix (and your new question), the regrid dictionary provides the methods of regridding any input gridded files. The MET User's Guide has a very useful section dedicated to the methods available within the regrid dictionary.

Additionally, regridding is only necessary if any of your data's grids are on different projections from one another. So from your description, it is not necessary to use any regridding methods and you should use

to_grid = NONE;

to disable any regridding from taking place. It is concerning to see, however, that when you set to_grid = NONE you still received an error. Certain types of map projections are unsupported in METplus, so I want to make sure you're not running into that problem. As a note, rotated Plate Carrée gridded data IS supported.

[anwe5]

Hi @j-opatz ,
Thank you very much for your response. When I set the regridding variable to match the grid of our data (to_grid = "rotlatlon 302 308 -4.75 -4.075 0.025 0.025 -40.0 10.0 0.0";) or the grid from the example in the users guide, I don’t receive any errors, but the output files are empty. However, when I set to_grid= None I encounter the following error:

DEBUG 2: Creating 2D constant-time slice attributes file: "output_mtd//mtd_20090101_003000V_2d.txt"
DEBUG 2: Creating 3D single simple attributes file: "output_mtd//mtd_20090101_003000V_3d_single_simple.txt"
DEBUG 2: Creating NetCDF file: "output_mtd//mtd_20090101_003000V_obj.nc"
ERROR  :
ERROR  :
ERROR  :   bool write_nc_grid(NcFile &, const Grid &) -> unsupported projection type
ERROR  :

I’ve attached the log file and the config file for the run with Embedding and setting the grid to None.
files.zip

From the output while running MTD, I noticed the following warning appearing multiple times across all runs:

/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/pyproj/__init__.py:89: UserWarning: pyproj unable to set database path.

It seems that MTD runs and generates the output files, but they remain empty whether I use regridding or not. I’m struggling to find the cause of this. Could there be an issue with my config file? Since the Embedding is working for plot_data_plane, I assume that’s not the issue. Or could the error be related to the Warning?

Additionally, if I don't use Embedding, I get the following error immediately:

DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 2: mtd_read_data() -> processing file "/scratch/b/b382378/test_data_MTD/hourly_data_00.nc"
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "seconds since 2009-01-01T00:00:00Z"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2009-01-01T00:00:00Z"
DEBUG 4:                 as a reference time of 20090101_000000 and 1 second(s) per time step.
DEBUG 3: NcCfFile::open() -> read time from the bounds variable "time_bnds"
DEBUG 5: add_to_unixtime() -->20090101_000000 plus 3600 times 1 seconds = 20090101_010000
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
ERROR  :
ERROR  : NcCfFile::get_grid_mapping_rotated_latitude_longitude() -> Didn't find X dimension (degrees_east) in netCDF file.
ERROR  :

If Plate Carrée gridded data is supported, does this mean I can use MET on my data without embedding?

[j-opatz]

If Plate Carrée gridded data is supported, does this mean I can use MET on my data without embedding?

That's a good question, and the answer is yes! As long as the input netCDF file follows Climate and Forecast (CF) conventions, it should be usable by METplus without the need for Python Embedding. For more information on those conventions, please see the MET User's Guide.

To check if the file was netCDF-CF compliant, I first ran your embedding_MTD.py script, along with the sample file hourly_data_00.nc with Plot-Data-Plane. It's always a good idea to check a new file or Python Embedding setup with Plot-Data-Plane, as the resulting image and processing is quicker to the point if there are any errors/issues. The command I used below was a success:

plot_data_plane PYTHON_XARRAY test_01.ps 'name="embedding_MTD.py";' -v 5

I then reran Plot-Data-Plane, but this time I directly passed in the sample file hourly_data_00.nc. This command failed:

plot_data_plane hourly_data_00.nc test_without_script.ps 'name="TOT_PREC"; level="(0,*,*)";' -v 5

The error was proceeded by a warning; I've printed both of those here:

WARNING: 
WARNING: NcCfFile::get_grid_mapping_rotated_latitude_longitude() -> Didn't find X dimension (degrees_east) in netCDF file.
WARNING: 
DEBUG 5: NcCfFile::open() -> coordinate variables: x=N/A, y=N/A, z=rlat, t=time
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
ERROR  : 
ERROR  : NcCfFile::update_grid(Grid &) -> Grid is not ready
ERROR  : 

As you can see, the input file is NOT CF-compliant (aka not ready). To do so, some corrections would need to be made to the latitude and longitude variables. Take a look at this valid unit strings table and you'll notice that in order for MET to interpret a variable as the longitude, for example, it will need to have a unit of "degrees_east", "degree_east", or any of the listed acceptable units. As those issues are fixed, you may find additional attributes that need to be adjusted to bring the file into CF-compliance.

For the purposes of troubleshooting, I went ahead and continued to use your embedding_MTD.py script and the config_test configuration file in a MODE-TD call. I was met with the same error you found:

ERROR  : 
ERROR  :   bool write_nc_grid(NcFile &, const Grid &) -> unsupported projection type
ERROR  : 

After some internal discussion, it seems like this error originates from MTD's use of a code function that only provides support for lat/lon, lambert conformal, and polar stereographic projections. So if you provide MTD input data in any other gridded projection, even if those projections are supported by other MET tools, during the output writing step MET will error out.

As a result, I've created this issue documenting the bug and what we need to do to patch this. We hope to make this bugfix available in the upcoming METplus coordinated 6.0 release, as well as a MET 11.1 bugfix.

In the meantime, you should be able to sidestep this issue by first calling the Regrid-Data-Plane tool to change the projection from rotated lat/lon to one of the 3 supported projections, then calling MTD for verification. Of course, this does introduce regridding distortions so please be aware of that potential.

[anwe5]

Thank you very much for taking the time to look into my issue! I will regrid my data and try MTD on the regridded data. Would you recommend using MET's Regrid-Data-Plane tool, or can I use any regridding tool, like CDO for example?

[j-opatz]

You are welcome to use any regridding application you wish. I would suggest to use the Regrid-Data-Plane tool for two reasons:

1. The output from the Regrid-Data-Plane tool can be immediately used by any other METplus tool without the worry of compatibility that might be present from the output of a third-party application
2. You can chain multiple tools together into one configuration file using METplus wrappers, which reduces the number of calls to METplus to one while still utilizing multiple tools. Information regarding the wrappers usage can be found in the METplus wrappers User's Guide.