Java heap space pb with script ran on > 100 images

[tom_deschamps]

Dear all,

I have a script that is executed through the insight client, launched on a whole directory containing hundreds of images.
The following issue used to happen when I was reaching close to 100 images processed:

Code: Select all

WARNING:omero.gateway:InternalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <8ac1b385-e1c5-4fe1-89c6-c21bdb9e9200omero.api.RawPixelsStore> getPlane((1, 2, 0), {})
Traceback (most recent call last):
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3459, in __call__
    return self.f(*args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero_api_RawPixelsStore_ice.py", line 1099, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}
ERROR:omero.gateway:Failed to getPlane() or getTile() from rawPixelsStore
Traceback (most recent call last):
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 5442, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3462, in __call__
    return self.handle_exception(e, *args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3459, in __call__
    return self.f(*args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero_api_RawPixelsStore_ice.py", line 1099, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}
ERROR performFiltering: Caught exception exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}



For each dataset (roughly 1000 x 1000 X 7) I would get all the planes through getPlane and to perform a MIP on each of the 3 channel.
I changed the default values in the following file

Code: Select all

etc/grid/templates.xml

to get the following processes running

java -Xmx1024M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback.xml -Domero.logfile=var/log/${omero.name}.log -Domero.logbase=/OMERO/ManagedRepository/ -Domero.name=Blitz-0 -jar lib/server/blitz.jar --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/Blitz-0/config/config
java -Xmx512M -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -jar lib/server/blitz.jar ome.fulltext --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/Indexer-0/config/config
java -Xmx512M -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0 -jar lib/server/blitz.jar ome.pixeldata --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/PixelData-0/config/config

But still, the same java heap space issue happens after approximately 200 images.
Is there I can do in order to overcome this issue other than increasing these numbers in the configuration file for omero server?

My script is using getPlane a lot. Does compiling all these getPlane in a single getPlanes will make any difference?
Thanks!

[wmoore]

Hi,

The getPlane() in blitz gateway creates, initialises and closes a RawPixelsStore for each call, so it should clean up after itself each time. However, it's possible that this cleanup is not instantaneous, so that many rapid calls to getPlane() do cause memory problems.

Certainly worth trying to use getPlanes() instead, which will only use (and close) a single RawPixelsStore for all the planes. You might want to avoid reading too many planes into memory in one go if they are large. getPlanes() returns a generator, so try something like:

Code: Select all

        image = conn.getObject("Image", 1)
        sizeZ = image.getSizeZ()
        sizeC = image.getSizeC()
        sizeT = image.getSizeT()
        zctList = []
        for z in range(sizeZ):
            for c in range(sizeC):
                for t in range(sizeT):
                    zctList.append( (z,c,t) )
       
        planes = image.getPrimaryPixels().getPlanes(zctList)
        for p in planes:
            # perform some manipulation on each plane


If you still have memory issues, maybe you could send us a copy of your script (privately if you prefer) and we could see if there's anything else causing the problem.

Cheers,

Will.

[wmoore]

Hi Tom,

Looking at your script (private message)... Did you try to use getPlanes()?

E.g.

Code: Select all

      if MIP is True :
         for z in range(sizeZ) :
            plane = np.maximum( plane, pixels.getPlane(z, c, t) )


could be written like

Code: Select all

      if MIP is True :
         zctList = [(z,c,t) for z in range(sizeZ)]
         for p in pixels.getPlanes(zctList):
             plane = np.maximum( plane, p )


If you already tried that, then I can't see anything else obvious from your script. Writing the many output images shouldn't be too memory intensive (it's only a jpg and a tiff per image, right)? Do you still run out of memory if writingSpotImage & writingCellImage are False?

I guess you could try the same approach for writingResultTable, just so you know if the memory issue is due to reading / analysis rather than handling results.

Do you always get the OutOfMemoryError on RawPixelsStore.getPlane() or sometimes on another call?

If none of these ideas help, you could try looking at the server Blitz log at the time that the script fails & see if there's anything obvious. Or maybe send us a chunk of it at that point.

Sorry I couldn't be more help,

Will.

[tom_deschamps]

Thanks, I've increased the memory space reserved by the application in the config files and it got rid of the problem so far.