Open data from existing group?

[rabernat]

Thanks for providing this great plugin!

We have been exploring storing and reading OME-Zarr data in Arraylake. For our application, it's not possible to just pass an HTTP URL directly. We would need to open the Zarr data outside of Napari and then pass a Zarr group directly. Something like this

import napari
import arraylake as al

client = al.Client()
repo = client.get_repo("earthmover-demos/ome-zarr")
group = repo.root_group["path-to-OME-Zarr-group"]  # type: Zarr.group

viewer = napari.Viewer()
viewer.open(group, plugin="napari-ome-zarr")

Today this does not work (ValueError: Given reader 'napari' is not a compatible reader for ['2']. No compatible readers are available for ['2'].)

Supporting opening a group directly would also perhaps fix the compatibility issues with WebKnossos described in #66.

[joshmoore]

I imagine we might need some help from naparistas like @DragaDoncila or @jni. AFAIK, the current napari reader plugin API is path base (e.g., https://napari.org/dev/plugins/testing_workshop_docs/3-readers-and-fixtures.html). I very much assume there's a way to "just" create & inject the image layer that the plugin is creating bypassing, but it would take some playing around (for me at least).

[rabernat]

Thanks for the reply Josh. I'm just learning about this ecosystem.

It seems like OME-Zarr does not actually depend on zarr-python significantly? It looks like it implements its own reader for Zarr, rather than using Zarr python's i/o:

https://github.com/ome/ome-zarr-py/blob/f2160a7281dee18eb63579560021c977244d2038/ome_zarr/io.py#L21

This definitely complicates the way we imagined Arraylake would plug into this ecosystem. We hoped that the zarr.Group was the basic interface that would be used to explore hierarchies, rather than the Store.

Edit: for context, we assumed this because that's how Xarray does it: https://github.com/pydata/xarray/blob/24ad846b89faa4e549a0cc6382b4735c872c371d/xarray/backends/zarr.py#L368

[joshmoore]

I'm just learning about this ecosystem.

❤️

It seems like OME-Zarr does not actually depend on zarr-python significantly? It looks like it implements its own reader for Zarr, rather than using Zarr python's i/o

No. It actually is overly dependent on zarr-python (and specifically the fsspec implementation). ZarrLocation did pre-date FSStore and might could likely be ripped out these days.

And "Reader" is unfortunately overloaded. They exist in OME land for interpreting microscopy formats (ImageReader)

[jni]

Hi folks!

It is beyond exciting to see you trying out napari @rabernat. 😃 Please let me know if you want to pair on this at some point at http://meet.jni.codes 😃.

I need to think about this more, but to do a little thinking out loud:

I'm a little unsure why there can't be a specific http endpoint that would return the group? Basically, I wonder if "napari plugins" and Viewer.open are the wrong places for this abstraction. Why can't the above be expressed as https://data.earthmover.io/repos/earthmover-demos/ome-zarr/path-to-OME-Zarr-group?

On the Python side, to do what you were trying to do with viewer.open @rabernat: if napari-ome-zarr has a function that converts a zarr Group to a list of LayerDataTuples (which I presume it does, or anyway some lines that do that could be bundled up into a function), then you would do:

for dt in napari_ome_zarr.data_tuples_from_group(group):
    viewer._add_layer_from_data(*dt)

(Yes I know that's private and bad, but you get my drift. 😂 I think we've been playing with the idea of making something public for some time.)

The reader plugin API is kinda meant for use cases like napari path/to/file or napari https://earthmover.io/path/to/example.zarr. It would be a little weird to shoehorn other classes in there.

But, that's not to say that we can't think about a new kind of contribution, something like registering data types (zarr.Group, np.array, xarray.DataArray) to particular functions that return a list of layer data tuples when given such an array, and then one can do viewer.add(my_obj, plugin='foo') and it would find whether the plugin has a method registered for interpreting those objects? 🤷

And then maybe str is just a special case of that? 🤔

Anyway, my turn to apologise @rabernat — I'm just learning about this idea. 😂 But while I think about it, can you maybe explore my question above about why arraylake datasets can't be exposed as URLs? 😅