Multiscale

[pattonw]

Add basic support for multiscale.

array -> multiscale helper function

open array from ome-ngff compliant pyramid

[mzouink]

related (it was working before):

from funlib.persistence import open_ds
path = "/nrs/cellmap/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s2"
ds = open_ds(path, 'r')
ds.voxel_size

result: (1, 1, 1)
Is there an easy fix to it
@pattonw @yuriyzubov

[mzouink]

our old zarr file (no multiscale) - metadata:
cat .zattrs

{
    "offset": [
        16900,
        17156,
        21164
    ],
    "resolution": [
        2,
        2,
        2
    ]
}

new version of funlib.persistence can't detect the voxel_size from resolution

[pattonw]

You can set the default metadata in a couple ways.

1. python

from funlib.persistence.arrays.metadata import MetaDataFormat, set_default_metadata_format

set_default_metadata_format(MetaDataFormat(voxel_size_attr="resolution")

2. config files
   You can set the MetaDataFormat (offset_attr, voxel_size_attr, axis_names_attr, units_attr) attributes in any of the following places:

LOCAL_PATHS = [Path("pyproject.toml"), Path("funlib_persistence.toml")]
USER_PATHS = [
    Path.home() / ".config" / "funlib_persistence" / "funlib_persistence.toml"
]
GLOBAL_PATHS = [Path("/etc/funlib_persistence/funlib_persistence.toml")]

If set in the pyproject.toml you have to set it under "tool.funlib_persistence.{field}"

[mzouink]

thank you. i did fix the old data one resolution data. but still the multiscale bug

/nrs/cellmap/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/.zattrs :

{
    "multiscales": [
        {
            "axes": [
                {
                    "name": "z",
                    "type": "space",
                    "unit": "nanometer"
                },
                {
                    "name": "y",
                    "type": "space",
                    "unit": "nanometer"
                },
                {
                    "name": "x",
                    "type": "space",
                    "unit": "nanometer"
                }
            ],
            "coordinateTransformations": [
                {
                    "scale": [
                        1.0,
                        1.0,
                        1.0
                    ],
                    "type": "scale"
                }
            ],
            "datasets": [
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                4.0,
                                4.0,
                                4.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                0.0,
                                0.0,
                                0.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s0"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                8.0,
                                8.0,
                                8.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                2.0,
                                2.0,
                                2.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s1"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                16.0,
                                16.0,
                                16.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                6.0,
                                6.0,
                                6.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s2"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                32.0,
                                32.0,
                                32.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                14.0,
                                14.0,
                                14.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s3"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                64.0,
                                64.0,
                                64.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                30.0,
                                30.0,
                                30.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s4"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                128.0,
                                128.0,
                                128.0
                            ],
                            "type": "scale"
                        },
                        {
                            "translation": [
                                62.0,
                                62.0,
                                62.0
                            ],
                            "type": "translation"
                        }
                    ],
                    "path": "s5"
                }
            ],
            "name": "/recon-1/em/fibsem-uint8",
            "version": "0.4"
        }
    ],
    "name": "fibsem-uint16"

[pattonw]

For working with OME metadata I would recommend using an ome-ngff specific tool to read and write the metadata. open_ds lets you directly pass in the offset, voxel_size, axis_names, and units if they aren't directly available on the array you are trying to open.

Something like:

offset, voxel_size, axis_names, units = read_ome_metadata("/path/to/pyramid", scale_level=0)
open_ds("path/to/pyramid/0", offset=offset, voxel_size=voxel_size,axis_names=axis_names, units=units)

The OME-NGFF metadata spec is quite extensive so it is probably better to use a library dedicated to it rather than writing a custom implementation in funlib.persistence.

If you have a good library for reading/writing OME-NGFF metadata let me know, I plan on adding a open_ome_ds and prepare_ome_ds at some point

[mzouink]

is there anyway to have a strict mode if it is activated raise an error if no voxel_size is found instead of returning default values

funlib.persistence/funlib/persistence/arrays/metadata.py

Line 141 in 3c0760e

else Coordinate((1,) * self.physical_dims)

[pattonw]

Not at the moment but that would be a good feature to add

[yuriyzubov]

Hi @pattonw
So initially I modified _read_voxel_size_offset method so it could fetch voxel size and offset from metadata. I had my own issues with the way it was implemented, as I believe that fetching metadata for N5 and Zarr should be separated, but it worked ok.

did you encounter any issues when testing open_ds() method, before pushing code to the main branch?

If you have a good library for reading/writing OME-NGFF metadata let me know, I plan on adding a open_ome_ds and prepare_ome_ds at some point

I tried to look into ome-zarr-py repo. Perhaps it makes sense to utilize ome_zarr.reader.Reader when creating open_ome_ds?

[mzouink]

update, after funlib.persistence changes, DaCapo+funlib.show.neuroglancer are not working anymore with all cellmap data