implement ODIM BUFR readers #101

jfigui · 2023-03-22T10:15:53Z

At the last OPERA ET meeting it emerged that there is a real interest for xradar to have and ODIM BUFR reader, in particular to read the French radar data.
I am fully aware that that is not an easy task but considering that there are plans to open the OPERA data archive and that, particularly at the early stages of the OPERA program, many files are in BUFR format this would be very welcomed by the European radar community.

kmuehlbauer · 2023-03-22T10:56:32Z

@jfigui Thanks for bringing this to attention.

wradlib used to read BUFR back in the old days, but dropped support because of all kind of issues. The bufr related python ecosystem evolved much since then, so this might not be an issue any more.

If you can point to some resources on how to read french bufr data that would be great. Also docs on how to handle tables etc. would be nice to have.

jfigui · 2023-03-22T11:41:13Z

@kmuehlbauer ,

you probably know that but there are resources linked to OPERA Bufr here:
https://www.eumetnet.eu/activities/observations-programme/current-activities/opera/

In particular:
https://www.eumetnet.eu/wp-content/uploads/2017/01/OPERA_bufr_description_2012.pdf

https://www.eumetnet.eu/wp-content/uploads/2017/03/bufr_3.2.zip

I can probably send you some examples of polar volume bufr files as well if needed.

kmuehlbauer · 2023-03-22T11:53:46Z

Yeah, but good it's now linked here. I've thought there would be some low hanging fruit with new packages like pdbufr or even directly with eccodes. The code on that eumetnet-page is rather old, unfortunately.

It would be great if you could provide some bufr polar volumes. Do they need special tables or can they be read with the standard tables?

jfigui · 2023-03-22T12:02:52Z

I think French data need special tables. What I positively know is that eccodes cannot open the polar volume bufr data directly (I tried). I have always tried to stay away from BUFR files so I am definitely not an expert but I will ask for support at MF.

jfigui · 2023-03-22T12:19:38Z

I attach a copy of the tables that are used at Météo-France
tables_bufr_opera.zip

jfigui · 2023-03-22T12:28:19Z

and here there is an example of so called PAM file:

a file containing a single sweep (i.e. one elevation) of the fields uncorrected reflectivity, co-polar correlation coefficient, non-corrected differential phase and differential reflectivity in polar coordinates (240 m x 0.5°) plus the standard deviation of the reflectivity in Cartesian coordinates
PAMA59LFPW180755.gz

jfigui · 2023-03-22T12:30:55Z

here there is an example of so called PAG file, these files are sent to the OPERA data center:

a file containing a single sweep (i.e. one elevation) of the fields uncorrected reflectivity, de-aliased Doppler velocity and standard deviation of the reflectivity.
PAGA59EODC180755.gz

jfigui · 2023-03-22T12:36:42Z

One thing to keep in mind is that as far as I understood those files are not a single bufr file with multiple fields but a concatenation of bufr files. To be able to decode them properly you have to first cut the file and then read the bufr.
Another problematic issue is that the field data is compressed using z lib so even when you manage to decode the bufr you still have to decompress the data.

kmuehlbauer · 2023-03-22T12:46:08Z

I feel like I'm remembering now, why we've dropped bufr in wradlib 😬

jfigui · 2023-03-22T12:49:45Z

I can fully understand your decision!
But if we could give it another try ... :)

kmuehlbauer · 2023-03-22T13:03:45Z

If at first you don't succeed...

mgrover1 · 2023-03-22T13:09:31Z

I am wondering if this might be of use here
https://github.com/pytroll/trollbufr

jfigui · 2023-03-22T13:19:54Z

Last time I checked trollbufr was not able to decode the Météo France BUFR. Among other things it was not prepared to read BUFR version 2 or 3 but it can be starting point.

kmuehlbauer · 2023-03-22T13:47:25Z

There's also pybufrkit, which might be worth a test.

rzambranap · 2023-06-30T16:34:57Z

Hey everyone, have we had any success in this field ? I recently received a load of bufr radar files from météofrance, and I'm having little to no success in reading them with pybufrkit or even the opera bufr linux software listed in one of the comments up there. For me it is quite important and necessary to read these files and I use a python environment, if someone could give me any clues of how to move forward, I'll gladly be implementing a reader to use in an xarray environment. If someone advanced on this we could work together. I've attached a data sample and the tables they sent me. Thanks in advance!

202304090000.gz
tables_bufr_281.zip

kmuehlbauer · 2023-07-03T09:08:27Z

No, there haven't been any advances here. The major pain point is, and this is a show stopper, that the provided tables are in some ".csv" format. eccodes does not read these tables per default and they will have to be converted for that purpose to the eccodes table format.

Please ask your table provider to provide the tables in ECMWF eccodes format.

Francesco-Uboldi · 2023-12-13T08:49:33Z

I am having a similar problem with some sample BUFR files I had from Météo-France.
See here

kmuehlbauer · 2023-12-13T08:57:17Z

@Francesco-Uboldi Are your MF tables the same as linked above by @jfigui and @rzambranap?

And would you join a team to implement an eccodes based BUFR reader for xradar?

Francesco-Uboldi · 2023-12-13T09:01:58Z

I have version 279 instead of 281... I will try the 281 from @rzambranap .
I can try and help, but I am quite new to both radar data in general and BUFR format. In particular, I never used eccodes: I met this software recently, of course... but the table problem stopped me, and I went towards OPERA instead.
And I have limited knowledge of python! And of C. I come from Fortran and R...
By the way... wouldn't it be nice if some BUFR user from Météo-France agreed to be involved?!? Anyone out there?

kmuehlbauer · 2023-12-13T09:06:06Z

@Francesco-Uboldi As I see the situation it all depends on the tables. We would need those tables in the format accepted by eccodes. Or there would need to be some transformation tool to convert the tables. It's a pity that BUFR data is so unaccessible.

rzambranap · 2023-12-13T10:13:14Z

Hey everyone, I wasn't able to implement something on Python, but I was able to sort it out via Python using the subprocess library, calling the installed BUFR software locally. Two caveats: I couldn't launch it on VMs, and there was something about 32 vs 64-bit architecture that somehow worked on my PC but not on my office's servers. I can send you the tables Météo-France sent to me and the functions I did in Python calling subprocess and everything, but it's not up to the standards you might be accustomed to...

…

On Wed, Dec 13, 2023 at 10:06 AM Kai Mühlbauer ***@***.***> wrote: @Francesco-Uboldi <https://github.com/Francesco-Uboldi> As I see the situation it all depends on the tables. We would need those tables in the format accepted by eccodes. Or there would need to be some transformation tool to convert the tables. It's a pity that BUFR data is so unaccessible. — Reply to this email directly, view it on GitHub <#101 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO6FHGG2GQ5OLQG2IZX2B3LYJFVYTAVCNFSM6AAAAAAWDTIYCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJTGUZDCOBWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Rodrigo Zambrana

kmuehlbauer added the enhancement New feature or request label Mar 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement ODIM BUFR readers #101

implement ODIM BUFR readers #101

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

mgrover1 commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

rzambranap commented Jun 30, 2023 •

edited

Loading

kmuehlbauer commented Jul 3, 2023 •

edited

Loading

Francesco-Uboldi commented Dec 13, 2023 •

edited

Loading

kmuehlbauer commented Dec 13, 2023

Francesco-Uboldi commented Dec 13, 2023 •

edited

Loading

kmuehlbauer commented Dec 13, 2023

rzambranap commented Dec 13, 2023 via email

implement ODIM BUFR readers #101

implement ODIM BUFR readers #101

Comments

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

mgrover1 commented Mar 22, 2023

jfigui commented Mar 22, 2023

kmuehlbauer commented Mar 22, 2023

rzambranap commented Jun 30, 2023 • edited Loading

kmuehlbauer commented Jul 3, 2023 • edited Loading

Francesco-Uboldi commented Dec 13, 2023 • edited Loading

kmuehlbauer commented Dec 13, 2023

Francesco-Uboldi commented Dec 13, 2023 • edited Loading

kmuehlbauer commented Dec 13, 2023

rzambranap commented Dec 13, 2023 via email

rzambranap commented Jun 30, 2023 •

edited

Loading

kmuehlbauer commented Jul 3, 2023 •

edited

Loading

Francesco-Uboldi commented Dec 13, 2023 •

edited

Loading

Francesco-Uboldi commented Dec 13, 2023 •

edited

Loading