Xarray in PyART

[mgrover1]

Starting a discussion here to open the conversation about updating the Radar object to utilize the xarray data model!

We have started a design presentation, and open to feedback/discussion!

[kmuehlbauer]

That's great news.

current wradlib status

I'm not sure if PyArt devs have followed the developments within wradlib in the last 3 years. Currently wradlib is able to read ODIM_H5 (also GAMIC), CfRadial1/2, Rainbow5 and Iris/Sigmet into xarray-Datasets. Sweeps are handled separately and are wrapped in a shallow RadarVolume class. Native output to ODIM_H5 and CfRadial2 (trying to adhere to standard recommendations) is possible.
Examples here: https://docs.wradlib.org/en/stable/notebooks/fileio/wradlib_xarray_backends.html

It's currently designed as xarray-Backends ("odim", "gamic", "cfradial1", "cfradial2", "iris", "rainbow") so that you can get the volume with wradlib-function:

f = 'cfrad.20080604_002217_000_SPOL_v36_SUR.nc'
vol = wrl.io.open_cfradial1_dataset(f)

or just one sweep using plain xarray with engine and group-kwarg:

swp = xr.open_dataset(f, engine="cfradial1", group="sweep_9")

Even reading multiple timesteps into one dataset is as easy as using the available open_*_mfdataset functions. To handle this wradlib's current internal data layout differs slightly in terms of dimension-naming (time vs azimuth/elevation) for PPI/RHI, but that's just an implementation detail which should not block any interactions.

other resources

With the recently emerging https://github.com/TomNicholas/datatree and https://github.com/alexamici/xarray-datagroup the wradlib developments will surely follow those tracks with nested (group) structures.

FM301

The overall implementation should closely adhere or be based on the approaching FM301 WMO standard (https://community.wmo.int/wmo-jet-owr-seminar-series-weather-radar-data-exchange). So the openradarscience-community might join forces or at least cooperate closely on that topic.

Recommendations

For a PyArt Xradar object it would be great if a wradlib xarray RadarVolume or sweep Dataset could directly be imported into the new Xradar object.

Another idea would be to just use the already implemented wradlib xarray backends/reading functionality as loader/base for the Xradar object.

Anyway, great to see this coming to PyArt. I'm open to answer any questions regarding implementation details on the wradlib side, problems we had over time and how they have been solved.

[scollis]

Thanks @kmuehlbauer ! This is great info. FYI @mgrover1 just started and will be a key player here. @mgrover1 Py-ART does not sit alone and our key partners in this include wradlib, LROSE and Baltrad.

[mgrover1]

@kmuehlbauer thank you so much for the information! Let's set up a time to chat soon!

We have started to hold PyART dev meetings every other week on Tuesdays at 3 PM Central Time, but I understand this might be tough to attend given the time zone difference.

#1045

I am interested in hearing more about your xarray integration work...

[zssherman]

@kmuehlbauer Thanks for the response! Yeah some sort of integration or bridge between wradlib would be great!

[jthielen]

This is wonderful to hear, and I'm excited to follow along as this moves forward! I wouldn't be able to make the next dev call due to travel, but I'd love to be able to join in later on.

A few limited thoughts from a downstream "PyART consumer" project (OpenMosaic):

• Not sure if it was just for ease of visualization that it was presented this way, but from the design presentation it looked like latitude and longitude coordinates are "built-in" to this new approach. I'd definitely vote to make such coordinates optional or lazy-evaluated only, since they are slow to compute (prohibitively so for large mosaics) and less useful (at least for gridding-type workflows) than the gate_x, gate_y coordinates in azimuthal equidistant space.
• The current Radar object's arrays don't split/group by sweep, and rather have (ray, range_gate) dimensions, which is a natural structure for full volume, gate-by-gate applications like gridding...it's nice to have just one contiguous array per-radar-per-field. Would it make sense for the new xarray data model to maintain this structure (which should allow it to just be a Dataset rather than dict-like of Datasets), and then have sweep splitting be a method on a pyart accessor? Or are the sweep-based semantics paramount to array contiguous-ness (in which case sweep concatenation to get back to a ray-by-ray representation can be done after-the-fact)?
• Would performance benchmarking/ASV tests be within scope for this work? I haven't gotten around to doing so myself, but it would be awesome to see such results upstream, as subjectively I've noticed loading Level II data into memory and building the associated coordinates to be the slowest part of mosaic construction. I'd hate to see data model improvements lead to significant performance regressions.

[mgrover1]

Thanks for the feedback and suggestions!! We will definitely prototype multiple approaches to investigate the most suitable approach...

[jhamman]

With the recently emerging https://github.com/TomNicholas/datatree and https://github.com/alexamici/xarray-datagroup the wradlib developments will surely follow those tracks with nested (group) structures.

Just here to encourage you all to engage with the design conversations happening around the datatree/datagroup concept. Based on your slide that references a dict-like object that points to multiresolution dataarrays/datasets, I suspect there is a good deal of synergy in the development happening there right now.

[mgrover1]

Thanks @jhamman we definitely want to be in discussions with those groups!

[mgrover1]

After some conversations with @TomNicholas at SciPy last week, I put together a prototype of this functionality

• Link to example notebook

I think we should decide where we want to allow multiple volumes in a single object, or just a single volume. Open to discussion! The linked notebook combines multiple volumes with their associated sweeps, with the hierarchy being:

• Tree
  • Volume0
    • Sweep1
  • Volume1
    • Sweep1

[mgrover1]

Coming back to this - we are moving development to the xradar, a dedicated IO package that will used both here as well as wradlib!

From @kmuehlbauer 's discussion over in the wradlib repository:

"wradlib's xarray reading capability will be moved to xradar. If that process is finished, xradar becomes a dependency of wradlib. For the user only minor adaptions on the code are needed"

A similar path will be taken with Py-ART. Xradar will become a dependency of Py-ART, and will enable this xarray integration.