Singularity HDF5 Point Obs Readin #2671
Replies: 2 comments 23 replies
-
Hi Matt (@fearon10). I converted the METplus Issue to a METplus Discussion. We use Discussions to interact with our users about any questions they have and when work is identified for enhancements, bugfixes, new use cases, etc. we typically create Issues. Thank you for this information. You mentioned that you are not receiving any matched pairs, which I also see. Based on the information in your log file and in your post, I see that 297447 observations were processed:
However, all 297447 observations are being rejected for one of the three reasons below:
I can see that you're looking for the obs name of
Running the following command, I can see that
Looking at the obs_vid variable (index of variable names at var_name), I want to determine how many instances there are with a value equal to 0, which shows how many values there are for
To find a total of only the values with 0, and therefore the total number of
which gave me 83586. Going back to the log output, we see there were 297447 observations processed and 213861 were rejected due to the obs var name. 297447 observations processed minus 83586 Regarding the 82998 observations rejected due to level mismatch:
I see the following, along with other instances, in your log file:
It looks like the level you are requesting doesn't line up with current range of level values. Please revisit the level values that you are requesting vs. what is available. Regarding the 588 rejected for the message type:
I see in you are looking for a message type of
produces the following:
Please try modifying your configuration file and see if that helps. |
Beta Was this translation helpful? Give feedback.
-
Hi Daniel, Apologies for the slow response. Unfortunately, I can't share the NAVGEM reader script and data without going through public release. I did receive the same result, of no matched pairs, when changing the name and level for the NAVGEM data. I attached my script for converting the raob file to netcdf, maybe there's something in there? I also attached my point-stat Thanks |
Beta Was this translation helpful? Give feedback.
-
Discussed in #2391
Hi,
I'm following this previous issue closely as I'm attempting to do something very similar, to use python embedding to ingest a point text file (a raob text file) into point_stat. I've had similar success of getting the data into point_stat but receive no matched pairs.
DEBUG 2: Processing airtmp/pre_0850.0 versus t/P85000, for observation type RAOB, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 297447
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 213861
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 0
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 82998
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 588
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad fcst value = 0
DEBUG 2: Rejected: bad climo mean = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter = 0
DEBUG 2: Rejected: duplicates = 0
As a preliminary step, I have used ascii2nc with python embedding to create a netcdf file -- this works well. I then attempt to ingest the netcdf file into point_stat and receive the result above of no matched pairs. My netcdf and config files (within point.zip) are attached along with my log file. My point_stat call is as follows:
singularity exec met_v12.0.0-beta1.simg point_stat PYTHON_NUMPY test.nc
PointStatConfig_IODA_navgem_v2.1.1_2021030400_000000 -outdir outdir -v 5 -log log.point_stat
point.zip
log.point_stat.txt
Thanks,
Matt
Originally posted by justintsu October 25, 2023
Hello,
I am trying to get point observations (radiosondes for now) in an HDF5 file read into point_stat using embedding. I've written the script test_h5_embed.py which reads in the hdf5 file and rearranges the incoming raobs in a dataframe whose indices match what I read on the MET documentation
df_master = pd.DataFrame(columns=['typ','sid','vld','lat','lon','elv','var',
'lvl','hgt','qc','obs'])
Finally, the dataframe values are extracted and converted to a list and stored in a variable named met_point_obs (although I've also tried storing it in varialbes named point_obs_data and point_data, neither worked). The script is invoked with arguments specifying what particular observation to store in 'obs'. I've placed a temperature file along with this script attached to this discussion. The script is invoked like:
test_h5_embed.py NAF_raob_2023033000.h5 tmp
With the correct environmental variables set, I run:
singularity exec /app/projects/NEPTUNE/software/singularity/met_v11.0.1.simg point_stat 'PYTHON_NUMPY' 'PYTHON_NUMPY' PointStat2023033000NEST1a0121x0121 -v 5 -outdir ./out/point_stat/radiosonde
and get this non-descript error:
ERROR :
ERROR : process_obs_file() -> trouble parsing the python command PYTHON_NUMPY.
ERROR :
Code_Config.zip
(See below comment for the observation file)
Beta Was this translation helpful? Give feedback.
All reactions