python embedding error

[JosemariePalero]

Hi. I would like to request assistance in configuring my Python embedding.
Here is my log file:
log.txt

And I get this error message:

DEBUG 4: 	/home/met/miniconda3/envs/METplus/bin/python3 /home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py /home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0 /home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24
Python Script:	'/home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py'
User Command:	'/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24'
Temporary File:	'/home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0'
Traceback (most recent call last):
  File "/home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py", line 26, in <module>
    met_in = pyembed_tools.call_python(sys.argv)
  File "/home/met/WRF/MET-11.1.0/share/met/python/pyembed/python_embedding.py", line 82, in call_python
    spec.loader.exec_module(met_in)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE.py", line 23, in <module>
    d = int(matches[0].split(':')[0])
IndexError: list index out of range
ERROR  : 
ERROR  : tmp_nc_dataplane() -> command "/home/met/miniconda3/envs/METplus/bin/python3 /home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py /home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0 /home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24" failed ... status = 256
ERROR  : 
08/20 04:59:56.322Z metplus.bfcc8b32 INFO: Finished running /home/met/WRF/MET-11.1.0/bin/point_stat - took 0:00:03.003523
08/20 04:59:56.338Z metplus.bfcc8b32 (command_builder.py:250) ERROR: (command_builder.py:1304) Command returned a non-zero return code: /home/met/WRF/MET-11.1.0/bin/point_stat -v 6 'PYTHON_NUMPY' /home/met/ai4rp/data/obs/synop/processed/nc/tp_24h/2024/04/synoptp_24h202404020000.nc /home/met/WRF/METplus-5.1.0/parm/met_config/PointStatConfig_wrapped -outdir /home/met/ai4rp/output/pointStat/
08/20 04:59:56.338Z metplus.bfcc8b32 INFO: Check the logfile for more information on why it failed: /home/met/WRF/METplus-5.1.0/Output/logs/metplus.log.20240820045953

I only get that error message if wgrib2 is not installed in my environment. I have set the MET_PYTHON_EXE environment variable to /home/met/miniconda3/envs/METplus/bin/python3.

When I try to run the command in my conda environment with wgrib2 installed, it runs as expected:

(METplus) met@met:~/ai4rp/pythonTools$ /home/met/miniconda3/envs/METplus/bin/python3 /home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py /home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0 /home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24
Python Script:	'/home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py'
User Command:	'/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24'
Temporary File:	'/home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0'
['427:4914578:d=2024040100:APCP Total Precipitation [kg/m^2]:surface:0-1 day acc fcst:', '428:4944917:d=2024040100:APCP Total Precipitation [kg/m^2]:surface:23-24 hour acc fcst:', '429:4958460:d=2024040100:APCP Total Precipitation [kg/m^2]:surface:21-24 hour acc fcst:', '430:4978170:d=2024040100:APCP Total Precipitation [kg/m^2]:surface:18-24 hour acc fcst:', '431:5001954:d=2024040100:APCP Total Precipitation [kg/m^2]:surface:12-24 hour acc fcst:']

But when I run it in my base environment without wgrib2 installed, it throws me the same error in my log file:

met@met:~/ai4rp/pythonTools$ /home/met/miniconda3/envs/METplus/bin/python3 /home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py /home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0 /home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24
Python Script:	'/home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py'
User Command:	'/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24'
Temporary File:	'/home/met/WRF/METplus-5.1.0/Output/tmp/tmp_met_nc_604459_0'
[]
Traceback (most recent call last):
  File "/home/met/WRF/MET-11.1.0/share/met/python/pyembed/write_tmp_dataplane.py", line 26, in <module>
    met_in = pyembed_tools.call_python(sys.argv)
  File "/home/met/WRF/MET-11.1.0/share/met/python/pyembed/python_embedding.py", line 82, in call_python
    spec.loader.exec_module(met_in)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE.py", line 24, in <module>
    d = int(matches[0].split(':')[0])
IndexError: list index out of range

I think that even though I explicitly set the environment variable MET_PYTHON_EXE to the python executable with the wgrib2 installed, MET still uses my base environment's python. Is there a workaround for this without installing wgrib2 and pygrib in my base environment?

my echo $MET_PYTHON_EXE in base environment:
met@met:~/ai4rp/pythonTools$ echo $MET_PYTHON_EXE
/home/met/miniconda3/envs/METplus/bin/python3

my echo $MET_PYTHON_EXE in conda environment:
(METplus) met@met:~/ai4rp/pythonTools$ echo $MET_PYTHON_EXE
/home/met/miniconda3/envs/METplus/bin/python3

Here is my MET-11.1.0 config.log:
config.log

my met.make.log:
met.make.log

Thank you very much.

[DanielAdriaansen]

Greetings! Thanks for sending your log file and details.

I see this error in your first log message:

  File "/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE.py", line 23, in <module>
    d = int(matches[0].split(':')[0])
IndexError: list index out of range

This tells me that in the Python script WRF-12kmtp_24hPE.py there is an error on line 23.

Looking at your Python embedding command from the log file, I see this:

/home/met/WRF/MET-11.1.0/bin/point_stat -v 6 'PYTHON_NUMPY' /home/met/ai4rp/data/obs/synop/processed/nc/tp_24h/2024/04/synoptp_24h202404020000.nc /home/met/WRF/METplus-5.1.0/parm/met_config/PointStatConfig_wrapped -outdir /home/met/ai4rp/output/pointStat/

So your Python embedding script looks like it is for your forecast data.

I see your field entry in the fcst dictionary point_stat is using looks like this:

/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE.py /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2 APCP 20240402_0000 20240401_0000 24

So it loooks like you are passing 5 arguments to your Python script:

1. /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2
2. APCP
3. 20240402_0000
4. 20240401_0000
5. 24

Are you able to share the Python script as well? I think it would be helpful to understand what the Python script is doing. The write_tmp_dataplane.py script is not designed to be called directly outside of MET, and so I think we should focus on getting your Python script working with MET first if possible. Thanks!

[DanielAdriaansen]

Also, if you are using METplus wrappers, you must set MET_PYTHON_EXE in your METplus config file, not on the command line. You can do this with a section that looks like this:

[user_env_vars]
MET_PYTHON_EXE=/home/met/miniconda3/envs/METplus/bin/python3

[JosemariePalero]

Hi, here's my Python script:
WRF-12kmtp_24hPE.py.txt

I am now passing 6 arguments to my Python script:

1. /share/mbs/DATA/WRF-12km/raw/2024/04/202404_0000/postwrf_d01_20240401_0000_f02400.gr2
2. APCP
3. 20240402_0000
4. 20240401_0000
5. 24
6. WRF-12km

I am now passing the model since I would like my Python script to be versatile and should be able to read different input forecast models, depending on what I need and define in my config file.

Also, here's my config file:
PointStatWRF-12kmtp_24h.conf.txt

[JosemariePalero]

Hi, I have fixed the error in the log message:
File "/home/met/ai4rp/pythonTools/WRF-12kmtp_24hPE.py", line 23, in
d = int(matches[0].split(':')[0])
IndexError: list index out of range

I fixed it by not using wgrib2 at all. I am only using wgrib2 to match APCP with the grib messages since the data that I have has 'unknown' name and 'unknown' shortName for the Total precipitation. But then after analyzing the attributes of the grib message with unknown name and shortName, I realized that I can use parameterName and stepRange instead. Now, my Python script looks like this:
WRF-12kmtp_24hPE.py.txt

and I have set my MET_PYTHON_EXE in my Point Stat configuration file. Now, my config file looks like this:
PointStatWRF-12kmtp_24h.conf.txt

Now, I get the following error:
metplus.log.20240821003152.txt

It can't find the netcdf4 module.
Note that my usr/bin/python3 doesn't have netcdf4 installed but my /home/met/miniconda3/envs/METplus/bin/python3 do have netcdf4 installed.

Thank you very much.

[DanielAdriaansen]

Hi,

Thank you for the updates, it seems like you've made a lot of progress!

In order to use Python embedding with MET, even when using your own Python installation (by setting MET_PYTHON_EXE), MET must be compiled with a version of Python with the packages installed listed in the documentation here:
https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#compiling-met-for-python-embedding

Unfortunately, it looks like your MET installation will need to be recompiled against a version of Python with those minimum set of requirements. Here is more information about installing MET:
https://met.readthedocs.io/en/latest/Users_Guide/installation.html

The key is using the --enable-python flag when compiling, and setting appropriate environment variables to point to the Python installation you'd like to compile against:
https://met.readthedocs.io/en/latest/Users_Guide/installation.html#set-environment-variables'

It looks like you did compile against this Python:
DEBUG 3: Initializing MET compile time python instance: /usr/bin/python3

But perhaps netcdf4 or other required packages are not installed there. You can do a test by doing the following:

/usr/bin/python3
import netcdf4

If that does not work, then the Python at /usr/bin/python3 doesn't meet the requirements for a Python installation to use with Python embedding in MET.

If you need additional assistance with compiling MET against Python and what changes to make, please reply and let us know otherwise try and recompile and that should resolve this error.

[JosemariePalero]

Hi. Thank you so much for your reply, I now have installed the required packages in my usr/bin/python3 and recompiled MET. It recompiled successfully and the importation error are now gone, but I still receive the following error:
metplus.log.20240821021149.txt

Thank you very much.

[JosemariePalero]

Hi. I have fixed the error by updating the netcdf tarball in the tar_files to netcdf-4.9.2.tar.gz and recompiling MET. It now processes my forecast and observation files and outputs statistics. Thank you very much.