Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZarrReader performance improvements #64

Merged
merged 37 commits into from
Dec 8, 2023
Merged

Conversation

dgault
Copy link
Member

@dgault dgault commented Sep 7, 2023

Initial performance improvements for ZarrReader related to #63

This PR is an initial PR for testing purposes, in its current state it looks to reduce the initFile time by about 50%, hopefully further improvements can be made.

Based on a subset of idr0036 the initial profiling looked as below,
Screenshot 2023-09-07 at 12 49 34

The updated profiling should look more like this (the Zarrutils.toJSON should also be gone). If we can manage to remove some of the remaining JZarr calls then this may be reduced further:
Screenshot 2023-09-07 at 16 46 32

The initial fixes here target the below areas:

  • Reduce the unnecessary parsing and storage of attribute files in XMLAnnotations. This should also reduce the size of the memo file. An option has been added to allow this to be configurable if needs be.
  • Reduce the number of calls JZarr to open the .zarray. These calls were taking a large portion of the initialization time, mostly occurring on setSeries. I have tried to reduce this down but it is atill required for retrieving the shape of each array to populate the dimension sizes.
  • Reduce the number of calls JZarr to fetch group attributes. These calls were slow and as such I have tried to reduce the level of duplication. Hopefully I can get this down further still.

@snoopycrimecop
Copy link
Member

Conflicting PR. Removed from build BIOFORMATS-push#94. See the console output for more details.
Possible conflicts:

--conflicts

@snoopycrimecop
Copy link
Member

snoopycrimecop commented Sep 8, 2023

Conflicting PR. Removed from build BIOFORMATS-push#22. See the console output for more details.
Possible conflicts:

--conflicts Conflict resolved in build BIOFORMATS-push#95. See the console output for more details.

@will-moore
Copy link
Member

Updated ZarrReader on idr-testing as described at IDR/idr-metadata#672 (comment) (after having used mkngff on 3 Plates from idr0025).
Then tried viewing images from Plate 2 for the first time.
Logs say it took 95 mins for setid, but images appear black - see IDR/idr-metadata#672 (comment)

@will-moore
Copy link
Member

Today's build took 4 mins for setId on idr0025 Plate 3: IDR/idr-metadata#672 (comment)

@snoopycrimecop
Copy link
Member

snoopycrimecop commented Sep 13, 2023

Conflicting PR. Removed from build BIOFORMATS-push#99. See the console output for more details.
Possible conflicts:

--conflicts Conflict resolved in build BIOFORMATS-push#100. See the console output for more details.

@will-moore
Copy link
Member

Testing "black" images OMEZarrReader using build from 11th Sept when I first noticed it (today's didn't include this PR)...

https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/534/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Testing on idr0125-pilot...

ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1040:localhost:80
sudo -u omero-server -s
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/534/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/

# as wmoore...
sudo service omero-server restart

Tried viewing a Plate P170 from idr0004 where I'd run mkngff and viewed the images using previous build of ZarrReader -
IDR/idr-metadata#637 (comment)
On this plate, images rendered fine - looks good.

Then I tried viewing a different Plate where I'd run mkngff at the same time but not viewed the images (no memo file created). Plate P144, e.g. http://localhost:1040/webclient/?show=image-696604

Memo file created just now...

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "08-34-48.864_mkngff"
2023-09-13 09:35:24,553 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/..zattrs.bfmemo (219313 bytes)
2023-09-13 09:35:24,553 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1694597666657] time[57896] tag[loci.formats.Memoizer.setId]
2023-09-13 09:35:24,553 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/.zattrs Series: 0

But this image is black, even with the rendering settings adjusted to be bright:

Screenshot 2023-09-13 at 10 45 40

Also, the black images are rendering very slowly - ~25 seconds, compared to Plate P170 above where the images are rendering correctly at "normal" speeds (less than 0.5 secs).

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

Discussed this morning - improved logging, comparison between memo file generation on import and after mkngff.

Update today's build of ZarrReader on idr-testing:omeoreadwrite...

https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/539/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Then tried viewing 3rd Plate from idr0025: http://localhost:1080/webclient/?show=image-3260835

3-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IContainer
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery(omero.api._IQueryTie@ab037ce6)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin(omero.api._IAdminTie@8040fd4d)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript(omero.api._IScriptTie@b8b56855)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig(omero.api._IConfigTie@a6cf999d)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig
2023-09-15 09:41:23,692 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:23,800 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/1
2023-09-15 09:41:23,995 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,106 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/2
2023-09-15 09:41:24,190 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,300 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/3
2023-09-15 09:41:24,517 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,743 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 5 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/1
2023-09-15 09:41:24,803 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,955 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 6 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/2
2023-09-15 09:41:25,212 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:25,368 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 7 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/3
2023-09-15 09:41:25,674 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:25,830 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 9 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/1
2023-09-15 09:41:25,899 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,006 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 10 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/2
2023-09-15 09:41:26,122 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,232 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 11 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/3
2023-09-15 09:41:26,544 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,794 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 13 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/1
2023-09-15 09:41:26,863 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,975 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 14 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/2
2023-09-15 09:41:27,038 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:27,113 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 15 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/3
2023-09-15 09:41:27,576 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,741 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 17 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/1
2023-09-15 09:41:27,814 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,925 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 18 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/2
2023-09-15 09:41:27,993 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:28,099 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 19 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/3
2023-09-15 09:41:28,508 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,811 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 21 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/1
2023-09-15 09:41:28,874 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,983 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 22 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/2
2023-09-15 09:41:29,119 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:29,229 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 23 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/3
2023-09-15 09:41:29,509 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,677 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 25 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/1
2023-09-15 09:41:29,753 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,868 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 26 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/2
2023-09-15 09:41:29,940 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:30,101 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 27 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/3
2023-09-15 09:41:30,410 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,572 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 29 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/1
2023-09-15 09:41:30,637 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,706 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 30 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/2
2023-09-15 09:41:30,764 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,971 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 31 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e



...
23-09-15 09:50:33,759 INFO  [                      omero.cmd.SessionI] (l.Server-8) Unregistered servant:fa0664ae-6693-4e84-ad2a-c48335260413/472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer(omero.api._IContainerTie@be549585)
2023-09-15 09:50:33,759 INFO  [                      omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: 472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo9156189850653740893
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694771435991] time[104] tag[loci.formats.Memoizer.saveMemo]
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
2023-09-15


@will-moore
Copy link
Member

First attempt to view the Image above gave this in webclient...

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1008, in getRenderingDefId
    return _M_omero.api.RenderingEngine._op_getRenderingDefId.invoke(self, ((), _ctx))
Ice.ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero

But on next attempt it worked and Image is NOT black!

Screenshot 2023-09-15 at 10 55 33

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "15-26-46.818_mkngff"
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694516674666] time[244964] tag[loci.formats.Memoizer.setId]
2023-09-12 11:08:39,631 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-13 23:00:16,310 DEBUG [                   loci.formats.Memoizer] (.Server-13) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2018-01/24/14-20-58.331_mkngff/8c168760-7bc0-4692-90b1-c774711e7dd8.zarr/OME/.METADATA.ome.xml.bfmemo (666953 bytes)
--
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0

712022 ms is 12 minutes.

Previous plates on idr0125 have taken 95 minutes for Plate 1 and 4 minutes also on Plate 3 on idr-testing server.

@will-moore
Copy link
Member

Comparing regular import (no chunks) and timing of memo file creation with after mkngff...

On idr0125-pilot (we already have NGFF data without chunks) - update ZarrReader as above...

cd /ngff/idr0025

for dir in *; do
  omero import --transfer=ln_s --depth=100 --name="${dir/.ome.zarr/}" --skip=all "$dir" --file "/tmp/$dir.log"  --errs "/tmp/$dir.err";
done

But err logs have:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-15 10:39:17,672 294        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-15 10:39:17,691 313        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 995067a5752310c81d89accc34337b4b1d0624df date: 20230915-0010
2023-09-15 10:39:17,753 375        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-15 10:39:17,754 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-15 10:39:17,758 380        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-15 10:39:18,155 777        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-09-15 10:40:44,670 87292      [      main] INFO      ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 78529ms. (86515ms total) [0 unknowns]
2023-09-15 10:40:44,722 87344      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-15 10:40:45,306 87928      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-15 10:40:45,960 88582      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

Reverted ZarrReader jar to OMEZarrReader-0.3.1-jar-with-dependencies.jar on idr0125-pilot and re-ran import of idr0025 plates above...

e.g.

2023-09-15 11:36:42,662 1321330    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287669

==> Summary
2509 files uploaded, 1 fileset, 1 plate created, 384 images imported, 0 errors in 0:21:56.370

Blank images (no chunks) are viewable.

I see several "saved memo" events in the logs. Times are 1.5 - 3.5 seconds.

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "11-14-46.693/10x"
2023-09-15 11:36:08,898 DEBUG [                   loci.formats.Memoizer] (2-thread-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503832 bytes)
2023-09-15 11:36:08,899 DEBUG [                   loci.formats.Memoizer] (2-thread-1) start[1694777765366] time[3532] tag[loci.formats.Memoizer.setId]
2023-09-15 11:36:09,000 INFO  [    o.s.blitz.repo.ManagedImportRequestI] (2-thread-1) File format: Zarr
--
2023-09-15 12:02:14,851 DEBUG [                   loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,851 DEBUG [                   loci.formats.Memoizer] (l.Server-6) start[1694779333111] time[1740] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,852 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 300
--
2023-09-15 12:02:14,899 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,899 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694779333111] time[1787] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,899 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 132

Rendering of image planes in webclient takes 200-300 ms.

Now update symlink as omero-server user... using IDR/idr-metadata#647 (comment)

cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
rm -rf "10x images plate 1.ome.zarr"
ln -s /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr "10x images plate 1.ome.zarr"

Image can be viewed as normal - "Save to All" generates thumbs for the Plate.

Then removed the memo file...

rm -rf "/data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo"

Trying to view the Image in webclient:

  serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

EDIT - Ignore this failure. See #64 (comment)

Will try mkngff on idr0025/Plate 1 (with deleted memo file) from above...

(mkngff) bash-4.2$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287669 "/bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr" > "5287669.sql"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15 // 11-14-46.693 for fileset 5287669
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr -> /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr

$ psql -U omero -d idr -h $DBHOST -f 5287669.sql
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        5287719
(1 row)
COMMIT

Viewing image...

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
}

Blitz.log

2023-09-15 13:31:52,866 INFO  [         ome.security.basic.EventHandler] (l.Server-3)  Auth:    user=52,group=3,event=326848895(User),sess=6abc8eea-4910-4f1b-b57d-cf8b244189d8
2023-09-15 13:31:53,391 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1694784712636] time[755] tag[loci.formats.Memoizer.setId]
2023-09-15 13:31:53,391 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
2023-09-15 13:31:53,391 ERROR [                ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor340.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
        at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
        at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
        at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
        at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)

Checked that we only have a single unique original file path/name:

idr=> select id from OriginalFile where path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/' and name = 'METADATA.ome.xml';
    id    
----------
 52318508
(1 row)

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

EDIT - Ignore this failure. See #64 (comment)

Try on Plate 2

omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287670 "/bia-integrator-data/S-BIAD846/52304cdf-4eba-4f0a-84b1-690e0d66add9/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr" > 5287670.sql 

(took 8 minutes)...

$ psql -U omero -d idr -h $DBHOST -f 5287670.sql 
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        5287720
(1 row)
COMMIT

Not working...

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-09/15/11-36-48.798_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml
}

@will-moore
Copy link
Member

will-moore commented Sep 15, 2023

EDIT - Ignore this failure. See #64 (comment)

mkngff hasn't been working today on NGFF Filesets.
Going to try again on original .pattern file data...

On idr0125-pilot... still trying with OMEZarrReader-0.3.1-jar-with-dependencies.jar
Try with some Plates from idr0011...

Try with first 4 Plates from first screen...

idr0011.csv

idr0011/Plate1-Blue-A.ome.zarr,S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f,21689
idr0011/Plate1-Blue-B.ome.zarr,S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1,21687
idr0011/Plate1-Green-A.ome.zarr,S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b,21688
idr0011/Plate1-Green-B.ome.zarr,S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b,21685
$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-56-03.328 for fileset 21689
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr -> /bia-integrator-data/S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-54-46.325 for fileset 21687
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr -> /bia-integrator-data/S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-55-31.533 for fileset 21688
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr -> /bia-integrator-data/S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-43-55.929 for fileset 21685
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr -> /bia-integrator-data/S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr
bash-4.2$ for r in $(cat $IDRID.csv); do
>   fsid=$(echo $r | cut -d',' -f3)
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
> done
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287721
(1 row)

COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287722
(1 row)

COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287723
(1 row)

COMMIT
UPDATE 46
BEGIN
 mkngff_fileset 
----------------
        5287724
(1 row)

COMMIT
    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/OME/METADATA.ome.xml
}

Something wrong with omero-mkngff?
All above today was testing with IDR/omero-mkngff#9 (which shouldn't behave any differently).
But lets revert to version I've been using mostly...

 pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'

Try with 2 more plates from idr0011...

idr0011/Plate1-Red-A.ome.zarr,S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d,21692
idr0011/Plate1-Red-B.ome.zarr,S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086,21686
$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3);   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"; done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 14-00-14.412 for fileset 21692
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr -> /bia-integrator-data/S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-48-45.381 for fileset 21686
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr -> /bia-integrator-data/S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr
$ for r in $(cat $IDRID.csv); do   fsid=$(echo $r | cut -d',' -f3);   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"; done
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287725
(1 row)
COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287726
(1 row)
COMMIT
    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr/OME/METADATA.ome.xml
}

@will-moore
Copy link
Member

will-moore commented Sep 18, 2023

EDIT - Ignore this failure. See #64 (comment)

Looking at the Blitz log for the last exception above (viewing http://localhost:1080/webclient/?show=well-1227572 from idr0011/Plate1-Red-A.ome.zarr I see...

Caused by: java.lang.IllegalArgumentException: Invalid index: -1
	at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
	at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
2023-09-18 07:58:56,503 INFO  [                 org.perf4j.TimingLogger] (l.Ser

Looking at the code...
https://github.com/ome/bioformats/blob/bc1fcddd5329d900853d55c6f8378e9235a26a53/components/formats-api/src/loci/formats/FormatReader.java#L960

Here, core.size() is returning 0 so that we get coreIndexToSeries(-1) being called, which throws exception above.

@will-moore
Copy link
Member

Discussed Friday's testing at IDR meeting today - I realise that the released OMEZarrReader-0.3.1-jar-with-dependencies.jar is likely failing for mkngff workflow because it is missing one of the currently open PRs:
This one (#64), #58, #53 or #20.

So I will return to testing mkngff workflow with the merge-ci build...

@will-moore
Copy link
Member

Using todays merge-ci build of ZarrReader: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/617/console

2023-09-18 00:06:16,745 [   scc.merge] INFO  Repository: ome/ZarrReader
2023-09-18 00:06:16,745 [   scc.merge] INFO  Excluded PRs:
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 58 dgault 'Add new option to list pixels in getUsedFiles' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO  Already up-to-date.
2023-09-18 00:06:16,746 [   scc.merge] INFO  
2023-09-18 00:06:16,746 [   scc.merge] INFO  Merged PRs:
2023-09-18 00:06:16,746 [   scc.merge] INFO    - PR 62 joshmoore 'Update README.md'
2023-09-18 00:06:16,746 [   scc.merge] INFO    - PR 64 dgault 'ZarrReader performance improvements'

From https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/default/lastSuccessfulBuild/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Testing memo file generation by viewing previously unviewed mkngff plate from idr0033:

@will-moore
Copy link
Member

will-moore commented Sep 18, 2023

Viewing image from 2nd plate: http://localhost:1080/webclient/?show=image-3230447
having checked the logs for memo file generation for that fileset to confirm that we haven't yet generated memo file..

grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.2 | grep -A 2 "2017-03/07/16-50-40.721_mkngff"

@will-moore
Copy link
Member

Updated the proxy idr-testing omeroreadonly-X servers with same OMEZarrReader_p665_b587.jar as tested above on idr-testing omeroreadwrite server.

@will-moore
Copy link
Member

Testing whether that last commit fixes issue with mismatching pixel values on multi-well plates (that contain OME/METADATA.ome.xml) as reported IDR/idr-metadata#675 (comment)
Updated ZarrReader on idr-testing to today's build jar named OMEZarrReader_p672_b594.jar

@will-moore
Copy link
Member

The Channels issue above seems to be fixed 👍

However, ManagedRepo paths issue remains (see #64 (comment))
Importing idr0125 Plates e.g. original data is at /ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr on idr0125-pilot, I get paths like this (which shouldn't contain ngff/idr0125/):

demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-11/07/09-32-28.422/ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr/C/10/8/.zgroup

@dgault
Copy link
Member Author

dgault commented Nov 16, 2023

@will-moore, with that last commit I believe the file path issue should be resolved. It seems that getRequiredDirectories is used to trim the paths down to remove the common path, see https://github.com/ome/omero-blitz/blob/8d18dfcecc63b002ad89e924e6f03763b16052de/src/main/java/ome/services/blitz/repo/ManagedRepositoryI.java#L1467. For Zarr files we should only need one level below the common parent, that being the .zarr folder itself.

@will-moore
Copy link
Member

will-moore commented Nov 16, 2023

Updated ZarrReader on idr-testing to today's build: OMEZarrReader_p681_b603.jar

omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/import_20231116.log --err /tmp/import_20231116.err

...
2023-11-16 13:28:59,517 807861     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-11-16 13:28:59,750 808094     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-11-16 13:29:05,157 813501     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-11-16 13:29:05,477 813821     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61676957
2023-11-16 13:29:10,368 818712     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61676957
2023-11-16 13:29:18,622 826966     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61676957
2023-11-16 13:29:22,497 830841     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61676957
2023-11-16 13:29:22,542 830886     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61676957
2023-11-16 13:29:22,576 830920     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61676957
2023-11-16 13:29:22,782 831126     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314846

==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:13:35.110

In webclient, checking managed repo paths I see eg:

demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-11/16/13-15-52.390/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/17/0/1/0

This removed the /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/ from the path as expected 👍 💯

@will-moore
Copy link
Member

@dgault @joshmoore @sbesson - What's the next steps towards getting this PR merged?
I don't have any outstanding issues with this now. It's working fine on idr-testing for the mkngff data and @dominikl has been using it for importing idr0138 data (although I don't know which build he's been using - probably not the last commit?).

If we are ready to get this merged and released then it would be great to get that ticked off the TODO list.

@dominikl
Copy link
Member

dominikl commented Dec 4, 2023

It was the build from 13th Nov, if that helps. But I'll import a few images with the current build again.

@sbesson
Copy link
Member

sbesson commented Dec 4, 2023

@dgault @joshmoore @sbesson - What's the next steps towards getting this PR merged?

Based on the latest state of the conversation, it looks like no more changes are planned, is that correct? If so, any reason not to follow the standard approach i.e. assign a set of reviewers, get the PR approved, merge and release. My understanding is that between yourself and @dominikl, there has been a lot of functional review (and you should probably be reviewers). Do we also want to run some code review? Proposing we add this discussion to the upcoming planning meeting as I would also consider a ZarrReader tag as essential if we decide to spin up idr-next /cc @francesw @jburel

@dgault
Copy link
Member Author

dgault commented Dec 4, 2023

Yeah I believe this should be ready for a final review now.

The one area that I can think of changing is the naming of the new options (https://github.com/ome/ZarrReader/pull/64/files#diff-71931eb54d3ee0b5a673044b8cdcdb8fa1bee03c583e3fc916e1b6b47a7dd7bbR93-R98). Currently as they have been added rather ad-hoc there is a mixture zarrreader and omezarr prefixes. So we probably want to devcide which we are keeping. Personally I would opt for OMEZarr as the preferred prefix.

@dgault
Copy link
Member Author

dgault commented Dec 5, 2023

Following the discussion today I have updated the options names to all use the omezarr prefix. I have also changed the default behaviour to disable the performance improvements.

@will-moore, this will mean you will be required to add the following the the bfoptions file:

omezarr.quick_read=true

@will-moore
Copy link
Member

Updated the mkngff bfoptions command at IDR/omero-mkngff#13 and used this to recreate the .bfoptions for all idr0004 NGFF data on idr0125-pilot.
Then manually removed the omezarr.quick_read=true for idr0004 plates P115 and P124 that have different sized images.

Then updated the ZarrReader to today's build to include last commits above. Regenerated memo files for regular plate and P115 and memo files were regenerated and images viewable.
IDR/omero-mkngff#13 (comment)

Looks good!

Copy link
Member

@will-moore will-moore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionally this is working for me. LGTM 👍

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performed a review of the latest state of the changes proposed here. I found no conceptual issue that would constitute a blocker but a few questions emerged as I was reading through the code:

  • the signature of several private utility methods (parseResolutionCount, parseOmeroMetadata, parseLabels, parseImageLabels) is updated to take the an attributes map as an extra input. For all these APIs, it looks like the first argument root is no longer used by the API and could be completed removed from the signature? Since these are internal methods, this can be handled as a follow-up cleanup / patch release if necessary
  • the parsePlate utility also takes an attr map as an extra argument but this is now the first argument rather than the last, is this done on purpose?
  • the extra openZarr boolean added to the new setSeries/setResolution APIs conceptually looks similar to isThisType(String id, boolean open) but I share some of the wider concerns in https://github.com/ome/ZarrReader/pull/64/files#r1350073451. Given the nature of the format, I would expect the reader to internalize the decision of whether to open attribute files or not to be able to fetch additional metadata while trying to keep the number of reads minimal. In other terms I think the omezarr.quick_read is a requirement in order to have this used at all for IDR but I wonder whether the mid-term goal is to have this option removed completely and just let the reader do the right thing
  • looking at the new options and their default value, my undertanding is that the two largest backwards-incompatible changes in the default behavior of the reader are that XML annotations for the JSON metadata are not created and label images are not treated as series anymore. Is that correct? I think both of these assumptions are completely acceptable to meet the IDR requirements but they should probably be communicated in the release notes to avoid surprises.

@dgault
Copy link
Member Author

dgault commented Dec 7, 2023

Performed a review of the latest state of the changes proposed here. I found no conceptual issue that would constitute a blocker but a few questions emerged as I was reading through the code:

  • the signature of several private utility methods (parseResolutionCount, parseOmeroMetadata, parseLabels, parseImageLabels) is updated to take the an attributes map as an extra input. For all these APIs, it looks like the first argument root is no longer used by the API and could be completed removed from the signature? Since these are internal methods, this can be handled as a follow-up cleanup / patch release if necessary

Yeah that parameter is no longer needed and can be safely removed from those method signatures

  • the parsePlate utility also takes an attr map as an extra argument but this is now the first argument rather than the last, is this done on purpose?

No, there is no specific reason for that, it can safely be moved to be the last parameter

  • the extra openZarr boolean added to the new setSeries/setResolution APIs conceptually looks similar to isThisType(String id, boolean open) but I share some of the wider concerns in https://github.com/ome/ZarrReader/pull/64/files#r1350073451. Given the nature of the format, I would expect the reader to internalize the decision of whether to open attribute files or not to be able to fetch additional metadata while trying to keep the number of reads minimal. In other terms I think the omezarr.quick_read is a requirement in order to have this used at all for IDR but I wonder whether the mid-term goal is to have this option removed completely and just let the reader do the right thing

I agree that the handling here is certainly not ideal and Im open to alternative suggestions. Unfortunately even in the quick read scenario we still need some of the calls to open the zarr and some to ignore it, so there has to be some logic handling that decision. The number of calls to setSeries is extremely high and keeping it as bare bones as possible was a big part of the performance improvements. If there was a way to get the same performance gains without the extra method signature then that would be ideal.

When you say have the option removed, do you mean omezarr.quick_read? That would likely be extremely difficult, as we see in instances such idr0004, if the data doesn't align with the assumptions that quick_read is making then the reader doesn't really have any way of verifying the correctness of the handling other than the original 'slow' read option.

  • looking at the new options and their default value, my undertanding is that the two largest backwards-incompatible changes in the default behavior of the reader are that XML annotations for the JSON metadata are not created and label images are not treated as series anymore. Is that correct? I think both of these assumptions are completely acceptable to meet the IDR requirements but they should probably be communicated in the release notes to avoid surprises.

Yes that is correct, the handling of the labels is probably the biggest functional change and should definitely be covered in the release notes

@sbesson
Copy link
Member

sbesson commented Dec 7, 2023

When you say have the option removed, do you mean omezarr.quick_read? That would likely be extremely difficult, as we see in instances such idr0004, if the data doesn't align with the assumptions that quick_read is making then the reader doesn't really have any way of verifying the correctness of the handling other than the original 'slow' read option.

Yes that's what I was referring to. Definitely not a blocker given the IDR timeline but I think that post-release, it might be useful to come back to this use case and understand what makes idr0004 different from other Zarr datasets in the sense that it cannot use the assumptions of the omezarr.quick_read option.

Should this alongside the API feedback be captured as issues and addressed in follow-up releases?

@dgault dgault mentioned this pull request Dec 7, 2023
4 tasks
@dgault
Copy link
Member Author

dgault commented Dec 7, 2023

Opened an Issue to capture the feedback here: #68

@dgault
Copy link
Member Author

dgault commented Dec 8, 2023

Getting this merged as it seems everyone is happy with the testing and handling the API feedback as part of the follow up issue.

@dgault dgault merged commit 8a02b4f into ome:main Dec 8, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants