Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZarrReader: Add support for OME-XML from bf2raw #31

Merged
merged 9 commits into from
May 16, 2022

Conversation

dgault
Copy link
Member

@dgault dgault commented Apr 28, 2022

This PR adds parsing for the METADATA.ome.xml file produced via bioformats2raw.
Currently the XML is parsed first and any conflicting data in the Zarr attributes will override the XML values.

To test this the best approach would be to take existing OME-TIFF samples, convert them using bf2raw and then compare the OME-XML output from the original to that from the ZarrReader. The ZarrReader is still currently dumping the ngff metadata as annotations, so you will see extra annotations in the converted version.

@dgault dgault changed the title Ome xml support ZarrReader: Add support for OME-XML from bf2raw Apr 28, 2022
@joshmoore
Copy link
Member

joshmoore commented Apr 29, 2022

A clarification:

  • Without this PR, you could only import each of the two directories "0" and "1" under out after bioformats2raw a&series=2.fake out (The OME/METADATA.ome.xml is ignored)
  • With this PR, we can finally use bioformats2raw a&series=2.fake a_real.ome.zarr and import the entire output and the OME/METADATA.ome.xml is taken into account, e.g. for image name, etc.

Background information is available in ome/ngff#112

cc: @pwalczysko

@will-moore
Copy link
Member

Testing with...

$ omero import --fetch-jars https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-build/lastSuccessfulBuild/artifact/src/target/OMERO.java-5.6.3-228-6e1847c-ice36-b1178.zip 11511419.zarr

@pwalczysko
Copy link
Member

A clarification:

* Without this PR, you could only import each of the two directories "0" and "1" under out after `bioformats2raw a&series=2.fake out` (The `OME/METADATA.ome.xml` is ignored)

* With this PR, we can finally use `bioformats2raw a&series=2.fake a_real.ome.zarr` and import the entire output and the `OME/METADATA.ome.xml` is taken into account, e.g. for image name, etc.

Background information is available in ome/ngff#112

cc: @pwalczysko

thank you @joshmoore . I think I undestand. The two directories "0" and "1" were the images in the two "Series" as defined by the bf2raw. But ZarrReader did not read the ome.xml file where the info which is BF usually using to decide whether this is a MIF or not and how to order it and name it. Thus, prior to this PR, the MIF OME-NGFFs were not imported as MIFs, you could only import single images ? And surely they were wrongly named, I saw that during testing.

@joshmoore
Copy link
Member

@pwalczysko : that all sounds correct, yes.

@sbesson
Copy link
Member

sbesson commented Apr 29, 2022

And surely they were wrongly named, I saw that during testing.

This is possibly a topic for a separate issue but I am raising here as I suspect this will become apparent with the testing of non-HCS multi-image OME-NGFF.

It would be good to write down our expectation for the naming of these images. The current historical logic for non-HCS imports to use the filename of the first file in the fileset (as detected by Bio-Formats), hence for OME-NGFF most images will be named 0 since this is the filename of the first chunk. With the OME metadata included, an alternative would be to start consuming the Image.Name metadata.

@pwalczysko
Copy link
Member

start consuming the Image.Name metadata.

This seems natural to me. I remember a discussion where this issue was raised even in pre-ngff times (i.e. this should be what BF is doing, but is not.. (was one voice of the discussion)). This seems to me now like the best solution.

@will-moore
Copy link
Member

Image converted to NGFF with bioformats2raw then imported at https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230717 (user-3) has same metadata as original image imported into same Dataset.
The only difference I can see is the name (known issue) and the thumbnail (seems to cover an area larger than the image itself, so that the image is in the top-left quarter of the thumbnail).
@sbesson I think the rendering settings (default Z and T) have been changed / saved in webclient since import.

@sbesson
Copy link
Member

sbesson commented May 2, 2022

@willmoore after resetting the rendering settings to Imported, the two images (original and converted) look identical although the thumbnail is not updated

Screenshot 2022-05-02 at 13 39 24

@will-moore
Copy link
Member

$ wget https://downloads.openmicroscopy.org/images/SVS/77928.svs
$ bioformats2raw 77928.svs 77928.zarr
$ find 77928.zarr/ -type f | wc
  14613   14613  387520
$ omero import -d 54570 --depth 10 77928.zarr
Using session for [email protected]:4064. Idle timeout: 10 min. Current group: read-only-1
2022-05-02 16:28:08,943 276        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.11-SNAPSHOT
2022-05-02 16:28:08,973 306        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0-SNAPSHOT revision: 0b26396c63282885de616d6f1c71aedd9024b3ef date: 29 April 2022
2022-05-02 16:28:09,305 638        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-05-02 16:28:09,868 1201       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-05-02 16:28:15,175 6508       [      main] INFO      ome.formats.importer.ImportCandidates - 14613 file(s) parsed into 1 group(s) with 1 call(s) to setId in 2116ms. (5307ms total) [0 unknowns]
2022-05-02 16:28:15,275 6608       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2022-05-02 16:28:15,821 7154       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2022-05-02 16:28:16,300 7633       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2022-05-02 16:28:16,313 7646       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2022-05-02 16:28:16,313 7646       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.5.11-SNAPSHOT
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_152-release
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.15.2.el7.x86_64
2022-05-02 16:28:16,468 7801       [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
2022-05-02 16:28:16,898 8231       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:54570
2022-05-02 16:28:16,928 8261       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:54570
2022-05-02 16:28:16,936 8269       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2022-05-02 16:31:24,159 195492     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2022-05-02 16:31:24,187 195520     [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /uod/idr-scratch/will/svs/77928.zarr/2/3/0/2/0/0/0...
...
2022-05-02 16:51:21,383 1392716    [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /uod/idr-scratch/will/svs/77928.zarr/.zgroup
2022-05-02 16:52:17,362 1448695    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2022-05-02 16:52:18,079 1449412    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1344084
2022-05-02 16:52:29,340 1460673    [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 1344084
2022-05-02 16:52:30,157 1461490    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 1344084
2022-05-02 16:52:37,304 1468637    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 1344084
2022-05-02 16:52:37,328 1468661    [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 1344084
2022-05-02 16:52:37,348 1468681    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 1344084
2022-05-02 16:52:37,950 1469283    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /uod/idr-scratch/will/svs/77928.zarr/2/3/0/2/0/0/0
Image:230866,230867,230868
Other imported objects:
Fileset:123608

https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230866

But, thumbnails are broken (for full-sized image 230866) or incorrect as above for the other images in the FileSet.
Also, the number of resolutions is more limited than the original (imported into same Dataset), meaning you can't zoom out to see the whole image in iviewer.

@sbesson
Copy link
Member

sbesson commented May 2, 2022

From the logs when trying to save a thumbnail with the file above

2022-05-02 16:09:19,009 INFO  [                 org.perf4j.TimingLogger] (.Server-25) start[1651507758938] time[70] tag[omero.call.exception]
2022-05-02 16:09:19,009 WARN  [        ome.services.util.ServiceHandler] (.Server-25) IllegalArgumentException thrown.

java.lang.IllegalArgumentException: 62425×60101 plane too large, cannot exceed 2^31 pixels
	at omeis.providers.re.RGBIntBuffer.<init>(RGBIntBuffer.java:63)
	at omeis.providers.re.RenderingStrategy.getIntBuffer(RenderingStrategy.java:186)
	at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:291)
	at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:558)
	at ome.services.ThumbnailBean.createScaledImage(ThumbnailBean.java:648)
	at ome.services.ThumbnailBean._createThumbnail(ThumbnailBean.java:845)
	at ome.services.ThumbnailBean.retrieveThumbnail(ThumbnailBean.java:1189)
	at ome.services.ThumbnailBean.retrieveThumbnailAndUpdateMetadata(ThumbnailBean.java:1140)
	at ome.services.ThumbnailBean.getThumbnailByLongestSide(ThumbnailBean.java:1320)

It almost feels like the server only sees the full resolution of the WSI - which would match the slow user experience in the viewer.

@sbesson
Copy link
Member

sbesson commented May 2, 2022

@dgault pointing showinf at the file generated by Will above, I get

Series count = 3
Series #0 :
	Resolutions = 9
		sizeX[0] = 62425
		sizeX[1] = 62425
		sizeX[2] = 62425
		sizeX[3] = 62425
		sizeX[4] = 62425
		sizeX[5] = 62425
		sizeX[6] = 62425
		sizeX[7] = 62425
		sizeX[8] = 62425
	Image count = 3
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 62425
	Height = 60101
	SizeZ = 1
	SizeT = 1
	SizeC = 3
	Tile size = 1024 x 1024
	Thumbnail size = 128 x 123
	Endianness = intel (little)
	Dimension order = XYZCT (uncertain)
	Pixel type = uint8
	Valid bits per pixel = 8
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0
	Plane #1 <=> Z 0, C 1, T 0
	Plane #2 <=> Z 0, C 2, T 0

Series #1 :
	Resolutions = 2
		sizeX[0] = 428
		sizeX[1] = 428
	Image count = 3
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 428
	Height = 424
	SizeZ = 1
	SizeT = 1
	SizeC = 3
	Tile size = 428 x 424
	Thumbnail size = 128 x 126
	Endianness = intel (little)
	Dimension order = XYZCT (uncertain)
	Pixel type = uint8
	Valid bits per pixel = 8
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0
	Plane #1 <=> Z 0, C 1, T 0
	Plane #2 <=> Z 0, C 2, T 0

Series #2 :
	Resolutions = 4
		sizeX[0] = 1280
		sizeX[1] = 1280
		sizeX[2] = 1280
		sizeX[3] = 1280
	Image count = 3
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1280
	Height = 431
	SizeZ = 1
	SizeT = 1
	SizeC = 3
	Tile size = 1024 x 431
	Thumbnail size = 128 x 43
	Endianness = intel (little)
	Dimension order = XYZCT (uncertain)
	Pixel type = uint8
	Valid bits per pixel = 8
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0
	Plane #1 <=> Z 0, C 1, T 0
	Plane #2 <=> Z 0, C 2, T 0

so it looks like an issue with the detection of sub-resolutions

@sbesson
Copy link
Member

sbesson commented May 3, 2022

This seems natural to me. I remember a discussion where this issue was raised even in pre-ngff times (i.e. this should be what BF is doing, but is not.. (was one voice of the discussion)). This seems to me now like the best solution.

Cross-linking to existing links

Overriding the target image name via the importer is always a possibility although this presents obvious usability and scalability issue.

It might be worth moving this discussion into a separate issue to limit the scope of the review for this particular PR. As a first question, what is the expectation for the default image name for these datasets under the various scenarios (single images, MIFs)?

@pwalczysko
Copy link
Member

Pasting here a first start on the testing of comparisons of ome.xml files between pre-conversion ome.tiffs and post-conversion ome.zarrs. Testing done with

BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml tubhiswt-2D/tubhiswt_C0.ome.tif >> tubhiswt-2D/tubhiswt_C0.ome.xml
bioformats2raw tubhiswt-2D/tubhiswt_C0.ome.tif  tubhiswt-2D/tubhiswt_C0.ome.zarr
BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs >> tubhiswt-2D/tubhiswt_C0.ome.zarr.xml
diff -u tubhiswt-2D/tubhiswt_C0.ome.xml tubhiswt-2D/tubhiswt_C0.ome.zarr.xml > difference.txt

The content of the difference.txt is

--- tubhiswt-2D/tubhiswt_C0.ome.xml	2022-05-04 16:27:43.000000000 +0100
+++ tubhiswt-2D/tubhiswt_C0.ome.zarr.xml	2022-05-04 19:01:35.000000000 +0100
@@ -1,21 +1,31 @@
-Checking file format [OME-TIFF]
+Checking file format [Zarr]
 Initializing reader
-OMETiffReader initializing tubhiswt-2D/tubhiswt_C0.ome.tif
-Reading IFDs
-Populating metadata
+ZarrReader initializing tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs
 Parsing schema path
 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd
 Validating OME-XML
 No validation errors found.
-Initialization took 0.34s
+Initialization took 0.538s
 
 Reading core metadata
-filename = /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.tif
+filename = tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs
 Used files:
-	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.tif
-	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C1.ome.tif
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/.zgroup
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/.zattrs
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/.zgroup
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/.zarray
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/0/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/0/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/.zarray
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/0/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/0/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/OME/METADATA.ome.xml
 Series count = 1
-Series #0 -- tubhiswt:
+Series #0 -- 0/0:
+	Resolutions = 2
+		sizeX[0] = 512
+		sizeX[1] = 256
 	Image count = 2
 	RGB = false (1) 
 	Interleaved = false
@@ -28,10 +38,10 @@
 	Tile size = 512 x 512
 	Thumbnail size = 128 x 128
 	Endianness = intel (little)
-	Dimension order = XYZTC (certain)
+	Dimension order = XYZCT (uncertain)
 	Pixel type = uint8
 	Valid bits per pixel = 8
-	Metadata complete = true
+	Metadata complete = false
 	Thumbnail series = false
 	-----
 	Plane #0 <=> Z 0, C 0, T 0
@@ -57,10 +67,10 @@
       <Detector ID="urn:lsid:loci.wisc.edu:Detector:OWS2" Manufacturer="Bio-Rad" Model="1024TLD" Type="Photodiode"/>
       <Objective CalibratedMagnification="100.0" Correction="PlanApo" ID="urn:lsid:loci.wisc.edu:Objective:OWS2" Immersion="Oil" LensNA="1.4" Manufacturer="Nikon" Model="S Fluor" NominalMagnification="100.0" SerialNumber="044989" WorkingDistance="0.13" WorkingDistanceUnit="µm"/>
    </Instrument>
-   <Image ID="Image:0" Name="tubhiswt">
+   <Image ID="Image:0" Name="0/0">
       <AcquisitionDate>2013-01-15T17:02:40</AcquisitionDate>
       <ExperimenterRef ID="urn:lsid:loci.wisc.edu:Experimenter:116"/>
-      <Pixels BigEndian="false" DimensionOrder="XYZTC" ID="Pixels:0" Interleaved="false" SignificantBits="8" SizeC="2" SizeT="1" SizeX="512" SizeY="512" SizeZ="1" Type="uint8">
+      <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:0" Interleaved="false" SignificantBits="8" SizeC="2" SizeT="1" SizeX="512" SizeY="512" SizeZ="1" Type="uint8">
          <Channel Color="-1" ID="Channel:0:0" SamplesPerPixel="1">
             <LightSourceSettings ID="urn:lsid:loci.wisc.edu:LightSource:OWS1"/>
             <DetectorSettings ID="urn:lsid:loci.wisc.edu:Detector:OWS1"/>
@@ -81,5 +91,28 @@
          <Plane TheC="1" TheT="0" TheZ="0"/>
       </Pixels>
    </Image>
+   <StructuredAnnotations>
+      <XMLAnnotation ID="Annotation:0">
+         <Value>{
+  "bioformats2raw.layout" : 3
+}</Value>
+      </XMLAnnotation>
+      <XMLAnnotation ID="Annotation:1">
+         <Value>{
+  "multiscales" : [ {
+    "metadata" : {
+      "method" : "loci.common.image.SimpleImageScaler",
+      "version" : "Bio-Formats 6.8.0"
+    },
+    "datasets" : [ {
+      "path" : "0"
+    }, {
+      "path" : "1"
+    } ],
+    "version" : "0.2"
+  } ]
+}</Value>
+      </XMLAnnotation>
+   </StructuredAnnotations>
 </OME>

This does not look to me like the zarr reader is missing something, all differences seem logical ? cc @sbesson @dgault .

@dgault
Copy link
Member Author

dgault commented May 5, 2022

That looks rather promising to me. The differences there all seem to be expected or explainable

  • DimensionOrder: As this is a 2D image the unused dimensions are uncertain and reverting to a default dimensionOrder
  • Extra structuredAnnotations: To be expected as its currently dumping the Zarr attributes here to preserve them
  • ImageName: The only real difference but this is one that we are already aware of
  • MetadataComplete: The correct value here is probably up for discussion

@pwalczysko
Copy link
Member

pwalczysko commented May 5, 2022

diff -u tubhiswt-3D/tubhiswt_C0.ome.xml tubhiswt-3D/tubhiswt_C0.ome.zarr.xml > tubhiswt-3D/C0.diff
which produces the C0.diff below

--- tubhiswt-3D/tubhiswt_C0.ome.xml	2022-05-04 16:45:37.000000000 +0100
+++ tubhiswt-3D/tubhiswt_C0.ome.zarr.xml	2022-05-05 14:26:30.000000000 +0100
@@ -1,21 +1,107 @@
-Checking file format [OME-TIFF]
+Checking file format [Zarr]
 Initializing reader
-OMETiffReader initializing tubhiswt-3D/tubhiswt_C0.ome.tif
-Reading IFDs
-Populating metadata
+ZarrReader initializing tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs
 Parsing schema path
 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd
 Validating OME-XML
 No validation errors found.
-Initialization took 0.357s
+Initialization took 0.819s
 
 Reading core metadata
-filename = /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.tif
+filename = tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs
 Used files:
-	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.tif
-	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C1.ome.tif
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/.zgroup
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/.zattrs
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/.zgroup
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/.zarray
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/18/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/18/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/9/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/9/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/0/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/0/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/11/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/11/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/7/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/7/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/16/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/16/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/6/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/6/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/17/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/17/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/1/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/1/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/10/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/10/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/19/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/19/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/8/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/8/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/4/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/4/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/15/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/15/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/3/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/3/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/12/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/12/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/2/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/2/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/13/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/13/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/5/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/5/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/14/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/14/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/.zarray
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/18/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/18/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/9/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/9/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/0/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/0/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/11/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/11/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/7/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/7/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/16/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/16/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/6/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/6/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/17/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/17/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/1/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/1/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/10/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/10/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/19/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/19/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/8/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/8/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/4/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/4/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/15/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/15/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/3/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/3/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/12/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/12/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/2/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/2/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/13/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/13/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/5/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/5/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/14/0/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/14/1/0/0/0
+	/Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/OME/METADATA.ome.xml
 Series count = 1
-Series #0 -- tubhiswt:
+Series #0 -- 0/0:
+	Resolutions = 2
+		sizeX[0] = 512
+		sizeX[1] = 256
 	Image count = 40
 	RGB = false (1) 
 	Interleaved = false
@@ -28,18 +114,18 @@
 	Tile size = 512 x 512
 	Thumbnail size = 128 x 128
 	Endianness = intel (little)
-	Dimension order = XYZTC (certain)
+	Dimension order = XYZCT (uncertain)
 	Pixel type = uint8
 	Valid bits per pixel = 8
-	Metadata complete = true
+	Metadata complete = false
 	Thumbnail series = false
 	-----
 	Plane #0 <=> Z 0, C 0, T 0
-	Plane #18 <=> Z 0, C 0, T 18
-	Plane #19 <=> Z 0, C 0, T 19
-	Plane #20 <=> Z 0, C 1, T 0
-	Plane #21 <=> Z 0, C 1, T 1
-	Plane #22 <=> Z 0, C 1, T 2
+	Plane #18 <=> Z 0, C 0, T 9
+	Plane #19 <=> Z 0, C 1, T 9
+	Plane #20 <=> Z 0, C 0, T 10
+	Plane #21 <=> Z 0, C 1, T 10
+	Plane #22 <=> Z 0, C 0, T 11
 	Plane #39 <=> Z 0, C 1, T 19
 
 
@@ -62,10 +148,10 @@
       <Detector ID="urn:lsid:loci.wisc.edu:Detector:OWS2" Manufacturer="Bio-Rad" Model="1024TLD" Type="Photodiode"/>
       <Objective CalibratedMagnification="100.0" Correction="PlanApo" ID="urn:lsid:loci.wisc.edu:Objective:OWS2" Immersion="Oil" LensNA="1.4" Manufacturer="Nikon" Model="S Fluor" NominalMagnification="100.0" SerialNumber="044989" WorkingDistance="0.13" WorkingDistanceUnit="µm"/>
    </Instrument>
-   <Image ID="Image:0" Name="tubhiswt">
+   <Image ID="Image:0" Name="0/0">
       <AcquisitionDate>2013-01-15T17:02:41</AcquisitionDate>
       <ExperimenterRef ID="urn:lsid:loci.wisc.edu:Experimenter:116"/>
-      <Pixels BigEndian="false" DimensionOrder="XYZTC" ID="Pixels:0" Interleaved="false" SignificantBits="8" SizeC="2" SizeT="20" SizeX="512" SizeY="512" SizeZ="1" Type="uint8">
+      <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:0" Interleaved="false" SignificantBits="8" SizeC="2" SizeT="20" SizeX="512" SizeY="512" SizeZ="1" Type="uint8">
          <Channel Color="-1" ID="Channel:0:0" SamplesPerPixel="1">
             <LightSourceSettings ID="urn:lsid:loci.wisc.edu:LightSource:OWS1"/>
             <DetectorSettings ID="urn:lsid:loci.wisc.edu:Detector:OWS1"/>
@@ -197,46 +283,69 @@
             <UUID FileName="tubhiswt_C1.ome.tif">urn:uuid:f801ea0a-e93e-4f4f-99b3-7ecc15048c12</UUID>
          </TiffData>
          <Plane TheC="0" TheT="0" TheZ="0"/>
-         <Plane TheC="0" TheT="1" TheZ="0"/>
-         <Plane TheC="0" TheT="2" TheZ="0"/>
-         <Plane TheC="0" TheT="3" TheZ="0"/>
-         <Plane TheC="0" TheT="4" TheZ="0"/>
-         <Plane TheC="0" TheT="5" TheZ="0"/>
-         <Plane TheC="0" TheT="6" TheZ="0"/>
-         <Plane TheC="0" TheT="7" TheZ="0"/>
-         <Plane TheC="0" TheT="8" TheZ="0"/>
-         <Plane TheC="0" TheT="9" TheZ="0"/>
-         <Plane TheC="0" TheT="10" TheZ="0"/>
-         <Plane TheC="0" TheT="11" TheZ="0"/>
-         <Plane TheC="0" TheT="12" TheZ="0"/>
-         <Plane TheC="0" TheT="13" TheZ="0"/>
-         <Plane TheC="0" TheT="14" TheZ="0"/>
-         <Plane TheC="0" TheT="15" TheZ="0"/>
-         <Plane TheC="0" TheT="16" TheZ="0"/>
-         <Plane TheC="0" TheT="17" TheZ="0"/>
-         <Plane TheC="0" TheT="18" TheZ="0"/>
-         <Plane TheC="0" TheT="19" TheZ="0"/>
          <Plane TheC="1" TheT="0" TheZ="0"/>
+         <Plane TheC="0" TheT="1" TheZ="0"/>
          <Plane TheC="1" TheT="1" TheZ="0"/>
+         <Plane TheC="0" TheT="2" TheZ="0"/>
          <Plane TheC="1" TheT="2" TheZ="0"/>
+         <Plane TheC="0" TheT="3" TheZ="0"/>
          <Plane TheC="1" TheT="3" TheZ="0"/>
+         <Plane TheC="0" TheT="4" TheZ="0"/>
          <Plane TheC="1" TheT="4" TheZ="0"/>
+         <Plane TheC="0" TheT="5" TheZ="0"/>
          <Plane TheC="1" TheT="5" TheZ="0"/>
+         <Plane TheC="0" TheT="6" TheZ="0"/>
          <Plane TheC="1" TheT="6" TheZ="0"/>
+         <Plane TheC="0" TheT="7" TheZ="0"/>
          <Plane TheC="1" TheT="7" TheZ="0"/>
+         <Plane TheC="0" TheT="8" TheZ="0"/>
          <Plane TheC="1" TheT="8" TheZ="0"/>
+         <Plane TheC="0" TheT="9" TheZ="0"/>
          <Plane TheC="1" TheT="9" TheZ="0"/>
+         <Plane TheC="0" TheT="10" TheZ="0"/>
          <Plane TheC="1" TheT="10" TheZ="0"/>
+         <Plane TheC="0" TheT="11" TheZ="0"/>
          <Plane TheC="1" TheT="11" TheZ="0"/>
+         <Plane TheC="0" TheT="12" TheZ="0"/>
          <Plane TheC="1" TheT="12" TheZ="0"/>
+         <Plane TheC="0" TheT="13" TheZ="0"/>
          <Plane TheC="1" TheT="13" TheZ="0"/>
+         <Plane TheC="0" TheT="14" TheZ="0"/>
          <Plane TheC="1" TheT="14" TheZ="0"/>
+         <Plane TheC="0" TheT="15" TheZ="0"/>
          <Plane TheC="1" TheT="15" TheZ="0"/>
+         <Plane TheC="0" TheT="16" TheZ="0"/>
          <Plane TheC="1" TheT="16" TheZ="0"/>
+         <Plane TheC="0" TheT="17" TheZ="0"/>
          <Plane TheC="1" TheT="17" TheZ="0"/>
+         <Plane TheC="0" TheT="18" TheZ="0"/>
          <Plane TheC="1" TheT="18" TheZ="0"/>
+         <Plane TheC="0" TheT="19" TheZ="0"/>
          <Plane TheC="1" TheT="19" TheZ="0"/>
       </Pixels>
    </Image>
+   <StructuredAnnotations>
+      <XMLAnnotation ID="Annotation:0">
+         <Value>{
+  "bioformats2raw.layout" : 3
+}</Value>
+      </XMLAnnotation>
+      <XMLAnnotation ID="Annotation:1">
+         <Value>{
+  "multiscales" : [ {
+    "metadata" : {
+      "method" : "loci.common.image.SimpleImageScaler",
+      "version" : "Bio-Formats 6.8.0"
+    },
+    "datasets" : [ {
+      "path" : "0"
+    }, {
+      "path" : "1"
+    } ],
+    "version" : "0.2"
+  } ]
+}</Value>
+      </XMLAnnotation>
+   </StructuredAnnotations>
 </OME>
 

In the above output, I do not understand the Plane... T 18 bit, which seems to have vanished in the ome.zarr ?

The tubhiswt-3D/..C1.. seems to have analogous output/problem. (Edit)

@pwalczysko
Copy link
Member

pwalczysko commented May 5, 2022

There is something I do not understand about the tubs.. samples. It seems that these are planes of a multi-z muti-t image, according to the names of the single ome.tiffs. But bf2raw does not accept folders as input (?) Hence I am converting the single planes. Is that a sound approach ? Am I missing some point here ? cc @sbesson @joshmoore

Maybe whatever ome.tiff inside the folder I pick, bf2raw will take all the other sections into account and produce the zarr out of all of them ? (eidt)
Indeed, when importing the whole folder into OMERO, I have just one image.

@pwalczysko
Copy link
Member

pwalczysko commented May 5, 2022

diff -u tubhiswt-4D/tubhiswt_C0_TP0.ome.xml tubhiswt-4D/tubhiswt_C0_TP0.ome.zarr.xml > tubhiswt-4D/C0.T0.diff
produces C0.T0.diff below. This starts to be quite big.

So big in fact I cannot paste it here. I think this workflow needs a revision.
@dgault I have put the C0.T0.diff on idr0-slot3:/tmp for you. We can study it in parallel

Studying the output more closely, I can see 4 questions

  1. Resolutions motif added in zarr
+Series #0 -- 0/0:
+	Resolutions = 2
+		sizeX[0] = 512
+		sizeX[1] = 256
  1. Similarly to the tubhiswt-3D/tubhiswt_C0.ome.xml described in the comment above, Planes seem to be taken away (from the xml of the original ome.tiff), such as some T 41 planes and they do not appear to me to be reported as being added in other places in the same numbers, e.g.
 	Plane #0 <=> Z 0, C 0, T 0
-	Plane #428 <=> Z 8, C 0, T 42
-	Plane #429 <=> Z 9, C 0, T 42
-	Plane #430 <=> Z 0, C 1, T 0
-	Plane #431 <=> Z 1, C 1, T 0
-	Plane #432 <=> Z 2, C 1, T 0
+	Plane #428 <=> Z 8, C 0, T 21
+	Plane #429 <=> Z 9, C 0, T 21
+	Plane #430 <=> Z 0, C 1, T 21
+	Plane #431 <=> Z 1, C 1, T 21
+	Plane #432 <=> Z 2, C 1, T 21
 	Plane #859 <=> Z 9, C 1, T 42
  1. The <Plane... motives
 <Plane TheC="1" TheT="27" TheZ="0"/>
+         <Plane TheC="1" TheT="27" TheZ="1"/>
+         <Plane TheC="1" TheT="27" TheZ="2"/>
+         <Plane TheC="1" TheT="27" TheZ="3"/>
+         <Plane TheC="1" TheT="27" TheZ="4"/>
+         <Plane TheC="1" TheT="27" TheZ="5"/>
+         <Plane TheC="1" TheT="27" TheZ="6"/>
+         <Plane TheC="1" TheT="27" TheZ="7"/>
+         <Plane TheC="1" TheT="27" TheZ="8"/>
+         <Plane TheC="1" TheT="27" TheZ="9"/>

Hard to judge, there are too many of them. Does that maybe bind with the point Ad 2. in this comment ?

  1. Extra metadata in zarr (possibly expected cf. @dgault 's comment re 2D case)
+      </Pixels>
+   </Image>
+   <StructuredAnnotations>
+      <XMLAnnotation ID="Annotation:0">
+         <Value>{
+  "bioformats2raw.layout" : 3
+}</Value>
+      </XMLAnnotation>
+      <XMLAnnotation ID="Annotation:1">
+         <Value>{
+  "multiscales" : [ {
+    "metadata" : {
+      "method" : "loci.common.image.SimpleImageScaler",
+      "version" : "Bio-Formats 6.8.0"
+    },
+    "datasets" : [ {
+      "path" : "0"
+    }, {
+      "path" : "1"
+    } ],
+    "version" : "0.2"
+  } ]
+}</Value>
+      </XMLAnnotation>
+   </StructuredAnnotations>
+</OME>

Imported the zarr I created during the workflow into merge-ci user-8 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231015 . It looks good, no difference spotted when comparing with the ome.tiff https://outreach.openmicroscopy.org/webclient/?show=image-95214

@pwalczysko
Copy link
Member

Bug: When trying to create an ome.xml output from the BBBC plate, I got an error.

Command and error see below

BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml BBBC/NIRHTa-001.ome.zarr/.zattrs >> BBBC/NIRHTa-001.ome.zarr.xml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/pwalczysko/ngff-testing/OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Exception in thread "main" java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.String (java.lang.Integer and java.lang.String are in module java.base of loader 'bootstrap')
	at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:553)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:281)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1035)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1121)

@joshmoore
Copy link
Member

Testing the zarr from spatial-image/multiscale-spatial-image#32:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1
	at loci.formats.in.ZarrReader.getOptimalTileHeight(ZarrReader.java:130)
	at loci.formats.ImageReader.getOptimalTileHeight(ImageReader.java:753)
	at loci.formats.ReaderWrapper.getOptimalTileHeight(ReaderWrapper.java:585)
	at loci.formats.tools.ImageInfo.readCoreMetadata(ImageInfo.java:577)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1048)

chunks are [64, 64, 64]. However, since this dataset follows ome/ngff#114 we might should handle this separately from this PR.

@sbesson
Copy link
Member

sbesson commented May 6, 2022

I was able to reproduce #31 (comment) independently by converting a test plate in the context of broadinstitute/lincs-cell-painting#54 (comment).

I believe the issue is related to the handling of the column_index/row_index keys. I can see either trying to fix this as part of this PR or focus on the non-HCS modalities first and looking into another series of HCS fixes as as separate PR to facilitate the review. Happy to discuss more this morning

@will-moore
Copy link
Member

Comparing the OME.XML from https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230717 (converted to NGFF and imported) with that of the Original Image (in the same Dataset) I noticed that the <Plane> elements are ordered differently in the NGFF image. Looking at the images in webclient, I can see that in the Original Image, the Exposure values are 0.8 for 1 channel and 0.5 for the other channel, whereas in the NGFF image, they alternate 0.5 and 0.8 for both channnels, and the T indices look like they are sorted strangely: 1, 21, 2, 22, 3 ...

Screenshot 2022-05-06 at 11 11 08

@pwalczysko
Copy link
Member

@dgault Is the strange sorting of T indices reported by @will-moore something which was indicated already in the diff of #31 (comment) maybe ? The T 42 in the diff seemed suspect to me.

@pwalczysko
Copy link
Member

pwalczysko commented May 6, 2022

Here is the slightly truncated diff of xml files of ome.tiff vs ome.zarr of the Leica-1.ome.tiff from /uod/idr/repos/curated/ome-tiff/public/2016-06/sub-resolutions/Brightfield/Leica-1/Leica-1.ome.tiff

I am not sure about the handling of the RGB true or false difference between ome.tiff and ome.zarr, this feels wrong, the number of images is different as well.

Further, the number of Resolutions differ between the tiff and zarr.

< Series #0 -- macro:
< 	Resolutions = 3
---
> Series #0 -- 0/0:
> 	Resolutions = 6
cat sub-resolutions/Leica-1.diff | grep -v "/uod/idr"
1d0
< Checking file format [OME-TIFF]
3,9c2,4
< Reading IFDs
< Populating metadata
< Reading IFDs
< Populating metadata
< Reading IFDs
< Populating metadata
---
> Checking file format [Zarr]
> Initializing reader
> ZarrReader initializing sub-resolutions/Leica-1.ome.zarr/.zattrs
14c9
< Initialization took 0.799s
---
> Initialization took 1.572s
17c12,5612
---
> filename = sub-resolutions/Leica-1.ome.zarr/.zattrs
> Used files:
19,22c5614,5615
< Reading IFDs
< Populating metadata
< Series #0 -- macro:
< 	Resolutions = 3
---
> Series #0 -- 0/0:
> 	Resolutions = 6
24,27c5617,5623
< 		sizeX[1] = 404
< 		sizeX[2] = 101
< 	Image count = 1
< 	RGB = true (3) 
---
> 		sizeX[1] = 808
> 		sizeX[2] = 404
> 		sizeX[3] = 202
> 		sizeX[4] = 101
> 		sizeX[5] = 50
> 	Image count = 3
> 	RGB = false (1) 
34,35c5630,5631
< 	SizeC = 3 (effectively 1)
< 	Tile size = 1616 x 216
---
> 	SizeC = 3
> 	Tile size = 1024 x 1024
38c5634
< 	Dimension order = XYCZT (certain)
---
> 	Dimension order = XYZCT (uncertain)
41c5637
< 	Metadata complete = true
---
> 	Metadata complete = false
44a5641,5642
> 	Plane #1 <=> Z 0, C 1, T 0
> 	Plane #2 <=> Z 0, C 2, T 0
46,49c5644,5645
< Reading IFDs
< Populating metadata
< Series #1 -- :
< 	Resolutions = 5
---
> Series #1 -- 1/0:
> 	Resolutions = 9
51,56c5647,5656
< 		sizeX[1] = 9208
< 		sizeX[2] = 2302
< 		sizeX[3] = 576
< 		sizeX[4] = 144
< 	Image count = 1
< 	RGB = true (3) 
---
> 		sizeX[1] = 18416
> 		sizeX[2] = 9208
> 		sizeX[3] = 4604
> 		sizeX[4] = 2302
> 		sizeX[5] = 1151
> 		sizeX[6] = 575
> 		sizeX[7] = 287
> 		sizeX[8] = 143
> 	Image count = 3
> 	RGB = false (1) 
63,64c5663,5664
< 	SizeC = 3 (effectively 1)
< 	Tile size = 1616 x 216
---
> 	SizeC = 3
> 	Tile size = 1024 x 1024
67c5667
< 	Dimension order = XYCZT (certain)
---
> 	Dimension order = XYZCT (uncertain)
70c5670
< 	Metadata complete = true
---
> 	Metadata complete = false
73a5674,5675
> 	Plane #1 <=> Z 0, C 1, T 0
> 	Plane #2 <=> Z 0, C 2, T 0
77,105d5678
< BitsPerSample: 8
< Compression: JPEG
< ImageLength: 4668
< ImageWidth: 1616
< MetaDataPhotometricInterpretation: RGB
< NumberOfChannels: 3
< PhotometricInterpretation: YCbCr
< PlanarConfiguration: Chunky
< ReferenceBlackWhite: 0
< ResolutionUnit: Centimeter
< SamplesPerPixel: 3
< TileLength: 512
< TileWidth: 512
< XResolution: 608.3301634770519
< YCbCrSubSampling: chroma image dimensions are half the luma image dimensions
< YResolution: 608.3301634770519
< macro collection.name: ImageCollection_0000000128
< macro collection.uuid: urn:uuid:d0fa25ee-baa8-48cd-944d-5598a36b3f7b
< macro creationDate: 2011-05-31T09:43:06.873Z
< macro device.model for image: Leica SCN400;Leica SCN
< macro device.version for image: 1.4.0.9691 2011/03/30 10:30:59;1.4.0.9708
< macro scanSettings.illuminationSettings.illuminationSource for image: brightfield
< macro scanSettings.illuminationSettings.numericalAperture for image: 0.4
< macro scanSettings.objectiveSettings.objective for image: 20
< macro view.offsetX for image: 5389341
< macro view.offsetY for image: 17548313
< macro view.sizeX for image: 18416000
< macro view.sizeY for image: 19216000
< macro view.spacingZ for image: 400
116c5689
<    <Image ID="Image:0" Name="macro">
---
>    <Image ID="Image:0" Name="0/0">
121,122c5694,5701
<       <Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:0" Interleaved="false" PhysicalSizeX="16.438446163366336" PhysicalSizeXUnit="µm" PhysicalSizeY="16.438446015424162" PhysicalSizeYUnit="µm" SignificantBits="8" SizeC="3" SizeT="1" SizeX="1616" SizeY="4668" SizeZ="1" Type="uint8">
<          <Channel ID="Channel:0:0" IlluminationType="Transmitted" SamplesPerPixel="3">
---
>       <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:0" Interleaved="false" PhysicalSizeX="16.438446163366336" PhysicalSizeXUnit="µm" PhysicalSizeY="16.438446015424162" PhysicalSizeYUnit="µm" SignificantBits="8" SizeC="3" SizeT="1" SizeX="1616" SizeY="4668" SizeZ="1" Type="uint8">
>          <Channel ID="Channel:0:0" IlluminationType="Transmitted" SamplesPerPixel="1">
>             <LightPath/>
>          </Channel>
>          <Channel ID="Channel:0:1" IlluminationType="Transmitted" SamplesPerPixel="1">
>             <LightPath/>
>          </Channel>
>          <Channel ID="Channel:0:2" IlluminationType="Transmitted" SamplesPerPixel="1">
128a5708,5709
>          <Plane TheC="1" TheT="0" TheZ="0"/>
>          <Plane TheC="2" TheT="0" TheZ="0"/>
131c5712
<    <Image ID="Image:1" Name="">
---
>    <Image ID="Image:1" Name="1/0">
136,137c5717,5724
<       <Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:1" Interleaved="false" PhysicalSizeX="0.5" PhysicalSizeXUnit="µm" PhysicalSizeY="0.5" PhysicalSizeYUnit="µm" PhysicalSizeZ="0.4" PhysicalSizeZUnit="µm" SignificantBits="8" SizeC="3" SizeT="1" SizeX="36832" SizeY="38432" SizeZ="1" Type="uint8">
<          <Channel ID="Channel:1:0" IlluminationType="Transmitted" SamplesPerPixel="3">
---
>       <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:1" Interleaved="false" PhysicalSizeX="0.5" PhysicalSizeXUnit="µm" PhysicalSizeY="0.5" PhysicalSizeYUnit="µm" PhysicalSizeZ="0.4" PhysicalSizeZUnit="µm" SignificantBits="8" SizeC="3" SizeT="1" SizeX="36832" SizeY="38432" SizeZ="1" Type="uint8">
>          <Channel ID="Channel:1:0" IlluminationType="Transmitted" SamplesPerPixel="1">
>             <LightPath/>
>          </Channel>
>          <Channel ID="Channel:1:1" IlluminationType="Transmitted" SamplesPerPixel="1">
>             <LightPath/>
>          </Channel>
>          <Channel ID="Channel:1:2" IlluminationType="Transmitted" SamplesPerPixel="1">
143a5731,5732
>          <Plane TheC="1" TheT="0" TheZ="0"/>
>          <Plane TheC="2" TheT="0" TheZ="0"/>
148,153c5737,5739
<          <Value>
<             <OriginalMetadata>
<                <Key>macro view.spacingZ for image</Key>
<                <Value>400</Value>
<             </OriginalMetadata>
<          </Value>
---
>          <Value>{
>   "bioformats2raw.layout" : 3
> }</Value>
156,161c5742,5763
<          <Value>
<             <OriginalMetadata>
<                <Key>macro scanSettings.objectiveSettings.objective for image</Key>
<                <Value>20</Value>
<             </OriginalMetadata>
<          </Value>
---
>          <Value>{
>   "multiscales" : [ {
>     "metadata" : {
>       "method" : "loci.common.image.SimpleImageScaler",
>       "version" : "Bio-Formats 6.8.0"
>     },
>     "datasets" : [ {
>       "path" : "0"
>     }, {
>       "path" : "1"
>     }, {
>       "path" : "2"
>     }, {
>       "path" : "3"
>     }, {
>       "path" : "4"
>     }, {
>       "path" : "5"
>     } ],
>     "version" : "0.2"
>   } ]
> }</Value>
164,169c5766,5793
<          <Value>
<             <OriginalMetadata>
<                <Key>macro scanSettings.illuminationSettings.illuminationSource for image</Key>
<                <Value>brightfield</Value>
<             </OriginalMetadata>
<          </Value>
---
>          <Value>{
>   "multiscales" : [ {
>     "metadata" : {
>       "method" : "loci.common.image.SimpleImageScaler",
>       "version" : "Bio-Formats 6.8.0"
>     },
>     "datasets" : [ {
>       "path" : "0"
>     }, {
>       "path" : "1"
>     }, {
>       "path" : "2"
>     }, {
>       "path" : "3"
>     }, {
>       "path" : "4"
>     }, {
>       "path" : "5"
>     }, {
>       "path" : "6"
>     }, {
>       "path" : "7"
>     }, {
>       "path" : "8"
>     } ],
>     "version" : "0.2"
>   } ]
> }</Value>

@pwalczysko
Copy link
Member

Another example of ordering of planes problem is /uod/idr/repos/curated/ome-tiff/public/2016-06/sub-resolutions/EM/BGal_000438_frames.ome.tiff vs its zarr counterpart. Both imported into https://merge-ci.openmicroscopy.org/web/webclient/?show=dataset-54970

the output of @will-moore 's script see below:

Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12895186d8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '22'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '4'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518728>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '31'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '1'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518778>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '9'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '6'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12895187c8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '2'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '36'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518818>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '37'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '27'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518868>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '21'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '24'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12895188b8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '17'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '2'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518908>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '24'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '34'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518958>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '32'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '15'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12895189a8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '10'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '16'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12895189f8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '3'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '20'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518a48>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '4'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '11'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518a98>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '23'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '14'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518ae8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '19'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '28'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518b38>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '5'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '33'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518b88>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '33'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '7'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518bd8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '18'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '8'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518c28>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '15'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '37'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518c78>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '14'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '29'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518cc8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '25'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '9'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518d18>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '27'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '0'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518d68>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '29'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '5'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518db8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '12'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '18'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518e08>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '35'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '30'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518e58>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '16'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '12'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518ea8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '11'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '25'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518ef8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '7'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '35'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518f48>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '26'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '13'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f1289518f98>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '36'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '3'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2048>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '34'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '10'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2098>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '1'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '17'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a20e8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '28'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '26'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2138>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '0'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '19'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2188>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '6'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '22'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a21d8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '13'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '31'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2228>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '8'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '21'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a2278>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '30'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '23'}
...----------------------
Checking element <Element '{http://www.openmicroscopy.org/Schemas/OME/2016-06}Plane' at 0x7f12894a22c8>
-------- ** difference... ** --------------
CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '20'}
CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '32'}
...----------------------

@sbesson
Copy link
Member

sbesson commented May 11, 2022

Initial thought while reading #31 (comment) is that the planes seem to be unordered in the XML output. This might be a feature of the exporter code which possible retrieves the plane metadata in no particular order.

The downside is that it makes the XML (at least the Plane section) not directly comparable unless the planes are sorted.

@dgault dgault mentioned this pull request May 12, 2022
@dgault
Copy link
Member Author

dgault commented May 12, 2022

Ok, a number of updates with the new commits, namely in the areas below:

HCS data
The reports of exceptions in #31 (comment) and on issue #33 should be resolved. The parsing has been updated to handle 3 possible scenarios:

  • parsing rowIndex/columnIndex for NGFF v0.4 data
  • parsing column_index/row_index as per BF2raw 0.3 and 0.4
  • fallback to parsing the data out of the path

Original metadata
The missing original metadata as reported on #34 should now be resolved. Note that the XML annotations were still present and correct beforehand, but now the original metadata will also be present in the UI.

Plane ordering
This accounts for ordering of T and Z as mentioned in #31 (comment), https://www.google.com/url?q=https://github.com/ome/ZarrReader/pull/31%23issuecomment-1118624516&sa=D&source=editors&ust=1652367499105103&usg=AOvVaw0yf2ZZWHAyQKCXodhL9I9C and #31 (comment)

The plane ordering issue is a bit a messy, the below are the steps that are occuring here:

  • original PFF is read in its original dimension order and planes are created in this order
  • the associated OME-XML for the format has a new dimensionOrder (note in the OME-XML the order of the planes differs from the dimension order)
  • The ZarrReader is initialized with the planes in the original order and the new dimensionOrder
  • ZarrReader attempts to update the Z, T, C indexes of the planes as per the data from the Zarr
  • The existing planes are in a different order which results in values being overwritten, so additional metadata fields such as exposure time are now incorrect

Although this is the most common scenario in which we would see problems, the list of planes could be in any given order and still be considered valid. This leads to a potentially messy attempt to reorder them or the hopefully simpler approach of trusting the original data. So with the last the commit the reader will now trust the original plane data to be correct and avoid any attempt to update those fields. As long as the associated OME-XML is valid for the dataset then this should be correct.

Number of resolutions
The issues with number of resolutions as reported on #31 (comment) and #31 (comment) are due to the bioformats2raw conversion process which has created extra resolutions. So the behaviour here looks to be correct

RGB
This one is probably open for discussion as it may not directly correlate to TIFF or other formats. For OME-TIFF for example we rely on samples per pixel or IFD tags for Photometric Interpretation.

Proposed 0.5 changes
From Josh's comment #31 (comment), I have only taken a quick look but this seem like a pretty major breaking change and certainly not for this PR. The metadata appears to have different values for shape and chunks. Currently we are reading the shape values via JZarr.

@pwalczysko
Copy link
Member

Original metadata
The missing original metadata as reported on #34 should now be resolved. Note that the XML annotations were still present and correct beforehand, but now the original metadata will also be present in the UI.

Retested and test passing. See user-8 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231365. Fresh reimport of the ome.zarr was necessary, but I guess this is expected. Good to go for my part of the tests.

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the HCS front, tested the latest state of this PR with three different HCS samples to try and cover the different codepaths mentioned in #31 (comment)

  • theidr0104 plate mentioned in https://www.openmicroscopy.org/2020/12/01/zarr-hcs.html which is stored according to the OME-NGFF 0.1 specification without column/row index keys
  • a fake test&plateRows=2&plateCols=2&fields=2.fake converted using bioformats2raw 0.4.0 (OME-NGFF 0.2 specification with column_index/row_index keys) and bioformats2raw 0.5.0-rc1(OME-NGFF 0.4 specification with columnIndex/rowIndex keys)

For the last 2 samples, two changes were required in the the acquisitions key of the top-level .zattrs

With these two changes, all three plates imported without issues with all images successfully loading. The OME-NGFF 0.4 plate in particular shows both original metadata as well as original metadata for each field of view confirming another feature introduced in this PR.

Screenshot 2022-05-13 at 11 42 48

From my perspective, this is ready to merge and possibly release immediately. I'll open a follow-up PR relaxing the requirement on maximumfieldcount.

Copy link
Member

@will-moore will-moore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing today:

$ omero import -d 54419 --depth=10 438CTR_01_4_R3D_D3D.zarr/
...
Bioformats version: 6.10.0-SNAPSHOT revision: 4abd23de11919a93aa5a2cc8e8b41f2b8f144127 date: 16 May 2022

Comparing OME.xml with script (sorting Planes by Z/C/T) finds no diffs and metadata looks correct in webclient. 👍 LGTM.

Screenshot 2022-05-16 at 09 47 35

@sbesson
Copy link
Member

sbesson commented May 16, 2022

Also retested 83843bc and confirmed that HCS OME-NGFF with plate.acquisitions without the maximumfieldcount can still import as expected.

@sbesson sbesson merged commit f2f7c39 into ome:main May 16, 2022
@sbesson
Copy link
Member

sbesson commented May 16, 2022

For the next steps, two questions:

  • are there any outstanding issues that needs to be fixed or minimally captured as issues ?
  • related, are we ready to cut a new release of the reader with the metadata changes included? If so would you call this rather a 0.1.6 or 0.2.0 @dgault ?

@imagesc-bot
Copy link

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/intermission-ome-ngff-0-4-1-bioformats2raw-0-5-0-et-al/72214/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants