Open Microscopy Environment

The OME community
https://www.openmicroscopy.org/community/

pyramid creation on large image - out of memory

https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=8380

Page 1 of 2

pyramid creation on large image - out of memory

PostPosted: Mon Oct 16, 2017 8:34 pm
by dsudar
Hi,

I'm trying to import very large images (over 100K by 100K pixels) generated by FEI Maps software in TIFF format. They initially import fine, the pyramid creation appears to complete fine, but then I get a set of out of memory errors in PixelData-0.log:
Code: Select all
2017-10-16 08:36:28,034 INFO  [             ome.io.nio.FilePathResolver] (2-thread-3) Metadata only file, resulting path: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif
2017-10-16 08:36:28,043 INFO  [             ome.io.nio.FilePathResolver] (2-thread-5) Metadata only file, resulting path: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif
2017-10-16 08:36:28,043 WARN  [ ome.services.pixeldata.PixelDataHandler] (2-thread-5) Pixels:812259 -- Already locked! /data/share/OMERO.data/Pixels/Dir-812/.812259_pyramid.pyr_lock
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Creating BfPixelBuffer: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif Series: 0
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Destination pyramid tile size: java.awt.Dimension[width=512,height=512]
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 1/140400 (0%).
2017-10-16 08:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.193700041194688E-4, m1=5.778884146644216E-9, m5=1.7016008277662782E-4, m15=4.121759882047415E-4, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 08:56:42,931 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 14041/140400 (10%).
2017-10-16 09:11:32,284 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 28081/140400 (20%).
2017-10-16 09:22:27,813 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 42121/140400 (30%).
2017-10-16 09:38:34,959 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 56161/140400 (40%).
2017-10-16 09:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.1883495468831373E-4, m1=5.060286122646805E-35, m5=1.0454996826395106E-9, m15=7.549266558565082E-6, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 09:58:58,758 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 70201/140400 (50%).
2017-10-16 10:10:32,823 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 84241/140400 (60%).
2017-10-16 10:22:34,409 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 98281/140400 (70%).
2017-10-16 10:45:26,516 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 112321/140400 (80%).
2017-10-16 10:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.1830250888146442E-4, m1=4.431044989528208E-61, m5=6.423772065474417E-15, m15=1.3826964016147306E-7, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 10:56:10,355 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 126361/140400 (90%).
2017-10-16 11:06:37,806 INFO  [                ome.io.nio.PixelsService] (2-thread-3) SUCCESS -- Pyramid created for pixels id:812259
2017-10-16 11:06:37,826 INFO  [       loci.formats.in.MinimalTiffReader] (2-thread-3) Reading IFDs
2017-10-16 11:06:37,843 INFO  [       loci.formats.in.MinimalTiffReader] (2-thread-3) Populating metadata
2017-10-16 11:06:38,237 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 749256
2017-10-16 11:06:38,237 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:38,584 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 2463322981
2017-10-16 11:06:38,585 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:38,595 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0xc7f4f6a1 at 2451788470
2017-10-16 11:06:38,605 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x25a3d892 at 2109537927
2017-10-16 11:06:38,614 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x2deb3e64 at 382680608
2017-10-16 11:06:38,667 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x6781989a at 214293002
2017-10-16 11:06:38,963 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 4925845350
2017-10-16 11:06:38,963 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:39,008 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x76e5b130 at 4914310839
2017-10-16 11:06:39,009 INFO  [              loci.formats.in.TiffReader] (2-thread-3) Checking comment style
2017-10-16 11:06:39,009 INFO  [          loci.formats.in.BaseTiffReader] (2-thread-3) Populating OME metadata
2017-10-16 11:10:11,346 ERROR [        ome.services.util.ServiceHandler] (2-thread-3) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
Requested array size exceeds VM limit
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
2017-10-16 11:10:11,353 WARN  [        ome.services.util.ServiceHandler] (2-thread-3) Unknown exception thrown.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
2017-10-16 11:10:11,354 ERROR [        ome.services.util.ServiceHandler] (2-thread-3) Method interface ome.services.util.Executor$Work.doWork invocation took 9223276
2017-10-16 11:10:11,354 ERROR [  ome.services.pixeldata.PixelDataThread] (2-thread-2) ExceptionException!
ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]


My current jvmcfg settings are:
Code: Select all
omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
indexer=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})
pixeldata=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
repository=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})


I thought these were pretty high. Where should I increase memory to get the import to fully complete?

Thanks,
- Damir

Re: pyramid creation on large image - out of memory

PostPosted: Tue Oct 17, 2017 11:02 am
by mtbc
Dear Damir,

It's indeed surprising that you're running into this issue. You are already doing the right thing in configuring the pixel data server process with plenty of memory; I don't know how much scope you have to try further increasing it nor if that extra would make the difference.

Is it always the same image of the set that triggers the problem? If so, would you be able to provide it to us so we can try some profiling at our end to see if there is some issue with TiffWriter or similar?

Cheers,
Mark

Re: pyramid creation on large image - out of memory

PostPosted: Tue Oct 17, 2017 12:36 pm
by dsudar
Hi Mark,
The image is quite large (5.5GB). Can you provide alternative upload instructions?
Thanks,
Damir

Re: pyramid creation on large image - out of memory

PostPosted: Tue Oct 17, 2017 4:21 pm
by mlinkert
Hi Damir,

I will send FTP upload information via private message.

Regards,
-Melissa

Re: pyramid creation on large image - out of memory

PostPosted: Wed Oct 18, 2017 2:12 pm
by mtbc
Damir,

Which version of OMERO are you running?

Cheers,
Mark

Re: pyramid creation on large image - out of memory

PostPosted: Wed Oct 18, 2017 3:09 pm
by dsudar
Hi Mark,
This is still on 5.3.5. I am planning upgrading to 5.4 in the next week or so.
Thanks,
Damir

Re: pyramid creation on large image - out of memory

PostPosted: Thu Oct 19, 2017 12:35 pm
by mtbc
Dear Damir,

I can at least reproduce your issue with OMERO 5.4.0: I get the same out of memory error in OmeroPixelsPyramidWriter.recompressSeries after building the pyramid for your 115,063 × 106,010 image; I had pixeldata=-Xmx30367m so just a little lower than yours. I'm afraid that I'd thus expect that you'll still see this error after upgrading but, thanks to your image submission, at least we can investigate at our end.

Cheers,
Mark

Re: pyramid creation on large image - out of memory

PostPosted: Thu Feb 22, 2018 5:56 am
by dsudar
Hi team,

I was wondering if there is any insight yet into the cause of this OOM issue. Meanwhile more folks at OHSU are creating files that have this same problem. Would it help to have a few more examples? I'll be happy to upload to the alternative upload site since these are large.

Cheers,
- Damir

Re: pyramid creation on large image - out of memory

PostPosted: Thu Feb 22, 2018 12:50 pm
by jmoore
Hi Damir,

Mark reproduced with a trivial example (see https://trello.com/c/fFBOR0rY/282-tiffsaver-oom-on-big-image-import#comment-59e9ac132ed089f6d553a1b8) so sending more files won't be necessary (though as always, thanks for the offer!) We don't however have a proposed fix yet.

Do you have the system resources and, if so, have you tried upping the memory on the process?
~Josh

Re: pyramid creation on large image - out of memory

PostPosted: Thu Feb 22, 2018 5:55 pm
by dsudar
Hi Josh,

Thanks for pointing me at the Trello card, I had missed that.

I already have pretty high memory allocation:
Code: Select all
omero_user@omero:~$ omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
indexer=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})
pixeldata=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
repository=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})

but I do have some headroom and will try to go even higher. I presume both Blitz and Pixeldata should get more?

However, when reading the Trello card I see a note from @davidgault1 that there may be a hard limit array size of 2^31. If that's true, I guess increasing memory won't help, right?

One sneaky issue I noticed is: after the OOM failure, no new pyramid creation is even initiating until I restart the OMERO.server. I.e. nothing is being logged in PixelData-0.log. After a restart PixelData goes to work on the next pyramid. So that OOM does mess something up badly but silently.

Thanks,
- Damir
All times are UTC
Page 1 of 2
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/