python embedding yyerror()

[JosemariePalero]

Hi, I would like to ask for help in configuring my python embedding for ECMWF forecast data.

Here is the log file:
metplus.log.20240828235254.txt

Here is the tmp file:
met_config_7220_0.txt

Here is my config file:
PointStattp_24h.conf.txt

Here is my python script:
fcsttp_24hPythonEmbedding.py.txt

My python script for embedding works well with WRF and GFS forecast data. But I can't seem to make it work for the ECMWF forecast data. Thank you very much.

[DanielAdriaansen]

Greetings! Thank you for sharing your files.

The error you are receiving is due to problems with the METplus wrapper substituting your value for FCST_VAR1_THRESH from your METplus Wrappers config file into the MET config file being used for PointStat.

I tried testing various combinations of Python arguments and FCST_VAR1_THRESH but was unable to reproduce your error.

Can you please try the following and report back:

1. Set LOG_LEVEL=DEBUG inside your PointStattp_24h.conf.txt config file and run again and share the log file
2. Double check that this file: /home/met/WRF/METplus-5.1.0/parm/met_config/PointStatConfig_wrapped has not been modified in any way from the version obtained when installing METplus-5.1.0. This section is relevant for your error, and that file should match this:

   METplus/parm/met_config/PointStatConfig_wrapped

   Lines 50 to 53 in a07b368

   	fcst = {
   	${METPLUS_FCST_FILE_TYPE}
   	${METPLUS_FCST_FIELD}
   	}

3. Share the final configuration used by METplus wrappers when you're running. In your log file, this is the file I'd like to see: Final Conf: /home/met/WRF/METplus-5.1.0/Output/metplus_final.conf.20240828235254

Hopefully with that additional information we can narrow down what might be happening. Thanks!

[JosemariePalero]

Hi, thank you for your response.

Here is the updated log file with LOG_LEVEL set to DEBUG:
metplus.log.20240829234246.txt

I have double-checked the PointStatConfig_wrapped and I have confirmed that nothing has been modified from the version you provided. But just in case you need it, here's my PointStatConfig_wrapped:
PointStatConfig_wrapped.c.txt

Lastly, here's my Final Conf.:
metplus_final.conf.20240829234246.txt

While trying to figure out if the data is the one causing this error, I have figured that the shape of the data when it is in GRIB1 format is (451, 601), which implies that the data is in 2D. But when I convert it to netCDF using the ECMWF's grib_to_netcdf tool, the data is now of shape (1, 451, 601). From being defined by just the lat and lon, it is now defined by tp(time, latitude, longitude). I don't know if this can help figure out what the problem is. I just find it weird, since when I try to use the PCPCombine tool, for example, on my WRF fcst data, the dimension of the netCDF file produced is still 2D.

I have provided a sample data in palero_data.

Again, thank you very much.

[DanielAdriaansen]

Hello. Apologies for the delay.

Can you please try the following command?
/home/met/WRF/MET-11.1.0/bin/plot_data_plane /share/mbs/DATA/ECMWF-HRES/raw/2024/04/202404_0000/A2D04010000040200001 test_grib1.ps 'name="TP";' -v 5

When I try that command to use plot_data_plane to read your GRIB1 file, I get this error:

DEBUG 1: Opening data file: A2D04010000040200001
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Gb1".
ERROR  : 
ERROR  : GribFile::read() -> file read error ... requested 16776952 bytes, got 3143732
ERROR  : 

MET recognizes that it is a GRIB1 file, but the file seems corrupt. Using wgrib to inspect the file also reveals some suspicious errors:

grib2 message ignored (use wgrib2)

I'm not sure why there are grib2 records in the grib1 file, but could you see if you're able to discover any problems with this file?

So far, I have been unable to reproduce the error you are receiving, but I have not been able to get past the errors with the GRIB1 file yet.

I realize you are passing the total precip via Python using pygrib, but you are telling MET to read the same file to obtain the grid information, which requires MET to be able to open the file. You could perhaps avoid this by specifying the grid information by hand in Python, but I realize it's easier to just use a gridded data file.

Either way, I think there is an error with the GRIB1 file that results in MET not being able to read it. Let me know what you're able to find out about the file. Thanks!

[JosemariePalero]

Hi. Thank you very much for your reply. I tried to run the command you provided using the same data I sent, and I got the same error. Interestingly, when I try the command with the other ECMWF-HRES data I have, I also get the same error. It seems like MET can't read any of my ECMWF-HRES files. I also tested and confirmed that there are indeed grib2 messages in my grib1 files, 5 of them, actually. Now we can see 2 ways to solve it. First is to extract variable specific message from the main file or second, define the grid manually, as you mentioned above. I'll try both solutions and update you on the results. Thank you very much.

[JosemariePalero]

Hi, I have tried the first possible solution and I got past the errors with the GRIB1 file. It also seems like I was able to get past the yyerror() too. However, I can't get past the level mismatch error.

Here's my log file:
metplus.log.20240906020053.txt

Here's my config file:
PointStattp_24h.conf.txt

Here's my script for fcst python embedding:
fcstPythonEmbedding.py.txt

Here's my script for obs python embedding:
obsPythonEmbedding.py.txt

And I have uploaded a sample data in palero_data.

I have tried to change the attrs['level'] to Z0, L0, A24 and Surface, but I still can't match the levels.

I don't get this problem on my other forecast files from different models like WRF and GFS.

Again, thank you very much for your help.

[JosemariePalero]

Hi, I have fixed my level mismatch error by setting the 'hgt' to 0. I think we're done here. You can now close the discussion. Again thank you very much.

[DanielAdriaansen]

Great news! Thanks for reporting back with your success. If you encounter any other issues don't hesitate to reach out.