Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Seems like the imageio uses more memory than needed #17

Open
hinerm opened this issue Jun 2, 2014 · 4 comments
Open

Seems like the imageio uses more memory than needed #17

hinerm opened this issue Jun 2, 2014 · 4 comments
Labels
bug

Comments

@hinerm
Copy link
Member

hinerm commented Jun 2, 2014

Needed over 3 gigs of memory to convert one plane of a 1.3 GB ome.tiff dataset.
@hinerm hinerm added the bug label Jun 2, 2014
@mabruce
Copy link
Contributor

mabruce commented Oct 6, 2014

Seconded. Writing three 150 MB files somehow needs 4-5 GB.

@phcerdan
Copy link

Seconded. An error sample going out of memory using SCIFIO-ITK bridge.

terminate called after throwing an instance of 'itk::ExceptionObject' what(): /home/phc/Software/ITK/build-devtoolset/Modules/Remote/SCIFIO/src/itkSCIFIOImageIO.cxx:906: itk::ERROR: SCIFIOImageIO(0xc4b1c0): SCIFIOImageIO: 'SCIFIOITKBridge read' exited abnormally. Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at loci.common.DataTools.allocate(DataTools.java:829) at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:183) at io.scif.itk.SCIFIOITKBridge.read(SCIFIOITKBridge.java:429) at io.scif.itk.SCIFIOITKBridge.executeCommand(SCIFIOITKBridge.java:153) at io.scif.itk.SCIFIOITKBridge.executeCommand(SCIFIOITKBridge.java:109) at io.scif.itk.SCIFIOITKBridge.waitForInput(SCIFIOITKBridge.java:84) at io.scif.itk.SCIFIOITKBridge.executeCommand(SCIFIOITKBridge.java:164) at io.scif.itk.SCIFIOITKBridge.main(SCIFIOITKBridge.java:696)

ctrueden added a commit to scifio/scifio-itk-bridge that referenced this issue Apr 13, 2015
@ctrueden
Reuse the byte buffer instead of reallocating
3b0966c 
For performance, Bio-Formats supports passing a preallocated byte
buffer. Let's definitely do this whenever possible, instead of getting
back a newly allocated buffer (of the exact same size) every time.

Might help some with scifio/scifio-imageio#17.
@ctrueden
Copy link
Member

I pushed a small performance improvement for reading. But it won't affect the writing performance. There is surely more going on here than just lack of buffer reuse. Best would be to profile the Java-side code. @hinerm has mad skills there—but finding the time is another issue...

@emmenlau
Copy link

emmenlau commented Nov 19, 2020 •
edited
Loading

I guess this is still relevant? Did somebody follow this up further?

In a similar benchmark with Java 1.6.0 and BioFormats 4.x I have found that the JVM can accommodate with significantly less memory if low limits are imposed already during startup. It seems that the JVM can make excessive use of memory when no tight limits are set, or at least it seems its not calling the garbage collector frequently enough.

Of course this policy of low initial limits has the flaw that if the limit is too low, the execution will eventually fail. The best policy I could come up with was to test with continuously growing memory limits until the image writing would work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ctrueden @emmenlau @hinerm @phcerdan @mabruce