pyramid creation on large image - out of memory

[jmoore]

I presume both Blitz and Pixeldata should get more?

I would start with just PixelData, if that's what's throwing the OOM.

However, when reading the Trello card I see a note from @davidgault1 that there may be a hard limit array size of 2^31. If that's true, I guess increasing memory won't help, right?

Beyond some point, it certainly won't help, correct.

One sneaky issue I noticed is: after the OOM failure, no new pyramid creation is even initiating until I restart the OMERO.server. I.e. nothing is being logged in PixelData-0.log. After a restart PixelData goes to work on the next pyramid. So that OOM does mess something up badly but silently.

Evil. Thanks for letting us know. More digging to be done.

All the best,
~Josh

[dsudar]

Just a quick update that with double the allocated memory:

JVM settings: Blitz-${index} -Xmx76800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx76800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

The exact same OOM error happens in the exact same stage in PixelData-0.log. I'll try another doubling for PixelData and will report.

This image is a plain tiff RGB image (8 bits/channel) of 115554 by 75720, tiled with 1360 by 1024 tiles.

Thanks,
- Damir

[jmoore]

Hi Damir,

I had a chance to run this with

Code: Select all

java -Xmx64000m

today and still saw the same failure:

Code: Select all

2018-02-26 15:45:25,055 ERROR [  ome.services.pixeldata.PixelDataThread] (2-thread-2) ExceptionException!
ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Requested array size exceeds VM limit
   at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_161]
   at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_161]
   at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_161]
   at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.8.0_161]
   at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.8.0_161]
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.7.3]
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.7.3]
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.7.3]
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.7.3]
   at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]


I fear that without a code change this isn't going to work.
~Josh

[dsudar]

Hi Josh,

Thanks for checking. So I'm looking at a couple of alternative ways to deal with these very large images. The trick appears to be to avoid OMERO having to generate a pyramid which means getting the image into a format that is already acceptably pyramid'ed before importing.
Roger Leigh has started working on an extension to OME.TIFF that encodes pyramids so that will eventually be the preferred way to go.
But for the interim I'm looking to convert my files into any other format that Bio-Formats already accepts as properly pyramid'ed. JPEG2000 appeared to be a good candidate so I am trying to generate proper JPEG2000 files using ImageMagick and/or the OpenJPEG tools from Louvain (http://www.openjpeg.org/). So far not much luck generating a jp2 file that doesn't end up going into the "pyramiding" pipeline inside OMERO (and thus failing eventually with the same OOM error as my large TIFF files). Would anyone have a recommendation how I can generate properly JPEG2000 encoded files that OMERO accepts as "no need to pyramid"?

Thanks,
- Damir

[wmoore]

Hi Josh,

What about uploading the base level tiles individually via the Python API as we do to create new tiled images in the Images_From_ROIs script?
https://github.com/ome/scripts/blob/dev ... OIs.py#L47

Or would this run into similar memory issues when OMERO tries to generate the other levels of the pyramid?

Will.

[emil.rozbicki]

Hi Damir,

The options are slightly limited at the moment at least until the support for the pyramids is added to OME-TIFF.

From the existing formats that already support pyramids and should be relatively easy to use, SVS and Imaris HDF are good starting points. They do work out of the box however we don’t have any writers for them so you would have to add them yourself. Also Imaris HDF does not support JPEG-2000 compression.

OMERO / Bio-Formats supports single channel plain TIFF with JPEG-2000 tiles. We’ve done some preliminary tests and in principle it could be extended to support multiple channels (Z & T unconfirmed at the moment). Since we’re bound by the Bio-Formats and OMERO release schedules it will take some time before it could be released and be ready to use with OMERO. It is possible that by this time OME-TIFF will support pyramids.

Cheers,
Emil

[sbesson]

Hi Damir,

Following up on Emil’s answer on the OME-TIFF pyramid, we have begun the work proposed in http://openmicroscopy.github.io/design/OME005/ to extend the OME-TIFF specification for pyramidal support and add reading and writing support for this extended format in Bio-Formats. We are currently targeting that effort towards Bio-Formats 5.9 with first versions appearing in the next several weeks.

Reading this thread, we understand that your code is handling the image generation and downsampling and you are mainly looking for a pyramid-aware container format as well as a writing API. Is that a correct assumption or would the plane downsampling also be a feature that you need immediately from the writing API?

Best,
Sebastien

[dsudar]

Hi Sebastien,

Great that a standard way to support pyramids in OME-TIFFs is on its way. Thanks!!

Reading this thread, we understand that your code is handling the image generation and downsampling and you are mainly looking for a pyramid-aware container format as well as a writing API. Is that a correct assumption or would the plane downsampling also be a feature that you need immediately from the writing API?

Currently my "code" mostly consists of a bash script that ties together a bunch of utility commands such as ImageMagick's convert, bfconvert, tiffcomment, some python code, and a Matlab script for image registration. So for this ad-hoc workflow I would be relying on a bfconvert that actually does the pyramid creation/downsampling etc.

But you are correct that for my (future) real implementation, my code would do the downsampling/pyramid_creation and I would just need the appropriate writing API that implements the writing into the proper pyramid-aware container format. My language of choice is C/C++ so I would be looking at the OME-Files library.

Thanks,
- Damir