MvMODE with Python Embedding

[CPKalb]

Hi:

I am struggling to set up running Multivariate MODE with python embedding on two fields, relative humidity and wind speed. Both datasets are read in using python embedding. I have tested each of these fields and python embedding scripts individually in MODE and verified that they work and produce reasonable output. However, when I try to run MvMODE, I get the following error:

Traceback (most recent call last):
File "/glade/work/dtcrt/METplus/derecho/components/MET/installations/11.1.0/share/met/python/met/dataplane.py", line 3, in
import numpy as np
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/numpy/init.py", line 141, in
from . import core
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/numpy/core/init.py", line 23, in
from . import multiarray
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/glade/work/dtcrt/METplus/derecho/miniconda/miniconda3/envs/metplus_v5.1_py3.10/lib/python3.10/site-packages/numpy/core/overrides.py", line 6, in
from numpy.core._multiarray_umath import (
SystemError: /home/conda/feedstock_root/build_artifacts/python-split_1669104676633/work/Objects/structseq.c:481: bad argument to internal function
WARNING:
WARNING: straight_python_dataplane() -> an error occurred importing module "/glade/u/home/kalb/FireWx/read_wrfout_fire_rh"
WARNING:
ERROR :
ERROR : setup_fcst_obs_data() -> can't get forecast data "PYTHON" from file "PYTHON_NUMPY"
ERROR :

I'm especially confused since this same python embedding script works as expected when running MODE, but it seems to fail here when using MvMODE. Also, when I look in the log file, it seems that it has in fact read in the RH field at some point using the python embedding (I can see the output from the python script), but then it fails again later with an error about not reading RH (and it doesn't appear to have tried to read wind yet). I couldn't seem to find any examples online that would show how to properly configure MvMODE with python embedding, so maybe my configuration is not correct? Any idea what may be causing the error?

I'm attaching my configuration and log file here:
mv_mode_caldor.conf.txt
metplus.log.20240726200808.txt

Thanks,
Christina

[JohnHalleyGotway]

Hi @CPKalb, I tried running multivariate MODE this morning using Python embedding to provide 2 inputs for both the forecast and observation dictionaries.

However, I think I get less far on seneca than you did on derecho. MODE is erroring out when parsing file lists containing only PYTHON_NUMPY PYTHON_NUMPY:

ERROR  : multivar_consistency_checks() ->Want at least 2 input files for fcst or obs in multivar mode, neither had 2 or more

It would appear on derecho that you've made it past the file list parsing step and made it to the actual processing step.

I took a look on derecho to see the content of your file lists:

cat /glade/derecho/scratch/kalb/FireWx/MET_output/stage/file_lists.20240726200808/20210817171500_0_mode_fcst.txt

But that file doesn't exist anymore. In fact, I can't find any file_list files in your MET_output directory. Any idea where I could take a look at that?

[JohnHalleyGotway]

@CPKalb, as we discussed in the METplus engineering meeting today, I'm confident that code changes in MET will be required to get Python embedding working in multivariate MODE as expected. I wrote up this dtcenter/MET#2940 issue to describe the work and listed the two of us as assignees.

If this is easily fixed, we can include a fix in the upcoming 12.0.0 release. If it's more involved, we'll need to push the work to the next coordinated release.

For now, I'd recommend using a workaround. Prior to running multivariate MODE, run PCP-Combine for each of the Python embedding calls and have it write a NetCDF output file containing the data served up through Python. Then modify your multivariate MODE configuration to reads those PCP-Combine outputs instead.

You can use the PCP-Combine -add option to basically just pass the data through the output, as shown in the example below:

bin/pcp_combine -add PYTHON_NUMPY 'name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt fcst";' fcst.nc

I'll go ahead and mark this discussion as being answered. But please feel free to comment on dtcenter/MET#2940 to provide more detail as needed.