Open Microscopy Environment

The OME community
https://www.openmicroscopy.org/community/

Java heap space pb with script ran on > 100 images

https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7334

Page 1 of 1

Java heap space pb with script ran on > 100 images

PostPosted: Wed Oct 16, 2013 11:26 pm
by tom_deschamps
Dear all,

I have a script that is executed through the insight client, launched on a whole directory containing hundreds of images.
The following issue used to happen when I was reaching close to 100 images processed:
Code: Select all
WARNING:omero.gateway:InternalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <8ac1b385-e1c5-4fe1-89c6-c21bdb9e9200omero.api.RawPixelsStore> getPlane((1, 2, 0), {})
Traceback (most recent call last):
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3459, in __call__
    return self.f(*args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero_api_RawPixelsStore_ice.py", line 1099, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}
ERROR:omero.gateway:Failed to getPlane() or getTile() from rawPixelsStore
Traceback (most recent call last):
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 5442, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3462, in __call__
    return self.handle_exception(e, *args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3459, in __call__
    return self.f(*args, **kwargs)
  File "/magic2/OMERO/OMERO.server/lib/python/omero_api_RawPixelsStore_ice.py", line 1099, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}
ERROR performFiltering: Caught exception exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
   at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
   at ome.scifio.io.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:132)
   at ome.scifio.io.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:118)
   at ome.scifio.io.NIOFileHandle.buffer(NIOFileHandle.java:535)
   at ome.scifio.io.NIOFileHandle.seek(NIOFileHandle.java:257)
   at ome.scifio.io.RandomAccessInputStream.seek(RandomAccessInputStream.java:151)
   at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:109)
   at loci.poi.poifs.storage.DocumentBlock.read(DocumentBlock.java:200)
   at loci.poi.poifs.filesystem.POIFSDocument.read(POIFSDocument.java:284)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:350)
   at loci.poi.poifs.filesystem.DocumentInputStream.read(DocumentInputStream.java:274)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:121)
   at loci.formats.services.POIServiceImpl.getDocumentBytes(POIServiceImpl.java:111)
   at loci.formats.services.POIServiceImpl.getDocumentStream(POIServiceImpl.java:106)
   at loci.formats.in.ZeissZVIReader.openBytes(ZeissZVIReader.java:126)
   at loci.formats.ImageReader.openBytes(ImageReader.java:416)
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:150)
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:314)
   at ome.io.bioformats.BfPixelsWrapper.getWholePlane(BfPixelsWrapper.java:361)
   at ome.io.bioformats.BfPixelsWrapper.getPlane(BfPixelsWrapper.java:258)
   at ome.io.bioformats.BfPixelBuffer.getPlaneDirect(BfPixelBuffer.java:196)
   at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:393)
   at sun.reflect.GeneratedMethodAccessor658.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at ome.security.basic.EventHandler.invoke(EventHandler.java:154)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
}



For each dataset (roughly 1000 x 1000 X 7) I would get all the planes through getPlane and to perform a MIP on each of the 3 channel.
I changed the default values in the following file
Code: Select all
etc/grid/templates.xml


to get the following processes running

java -Xmx1024M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback.xml -Domero.logfile=var/log/${omero.name}.log -Domero.logbase=/OMERO/ManagedRepository/ -Domero.name=Blitz-0 -jar lib/server/blitz.jar --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/Blitz-0/config/config
java -Xmx512M -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -jar lib/server/blitz.jar ome.fulltext --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/Indexer-0/config/config
java -Xmx512M -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0 -jar lib/server/blitz.jar ome.pixeldata --Ice.Config=/magic2/OMERO/OMERO.server-5.0.0-beta1-ice34-b3470/var/master/servers/PixelData-0/config/config

But still, the same java heap space issue happens after approximately 200 images.
Is there I can do in order to overcome this issue other than increasing these numbers in the configuration file for omero server?

My script is using getPlane a lot. Does compiling all these getPlane in a single getPlanes will make any difference?
Thanks!

Re: Java heap space pb with script ran on > 100 images

PostPosted: Thu Oct 17, 2013 9:21 am
by wmoore
Hi,

The getPlane() in blitz gateway creates, initialises and closes a RawPixelsStore for each call, so it should clean up after itself each time. However, it's possible that this cleanup is not instantaneous, so that many rapid calls to getPlane() do cause memory problems.

Certainly worth trying to use getPlanes() instead, which will only use (and close) a single RawPixelsStore for all the planes. You might want to avoid reading too many planes into memory in one go if they are large. getPlanes() returns a generator, so try something like:

Code: Select all
        image = conn.getObject("Image", 1)
        sizeZ = image.getSizeZ()
        sizeC = image.getSizeC()
        sizeT = image.getSizeT()
        zctList = []
        for z in range(sizeZ):
            for c in range(sizeC):
                for t in range(sizeT):
                    zctList.append( (z,c,t) )
       
        planes = image.getPrimaryPixels().getPlanes(zctList)
        for p in planes:
            # perform some manipulation on each plane


If you still have memory issues, maybe you could send us a copy of your script (privately if you prefer) and we could see if there's anything else causing the problem.

Cheers,

Will.

Re: Java heap space pb with script ran on > 100 images

PostPosted: Fri Oct 18, 2013 10:02 pm
by wmoore
Hi Tom,

Looking at your script (private message)... Did you try to use getPlanes()?

E.g.
Code: Select all
      if MIP is True :
         for z in range(sizeZ) :
            plane = np.maximum( plane, pixels.getPlane(z, c, t) )


could be written like
Code: Select all
      if MIP is True :
         zctList = [(z,c,t) for z in range(sizeZ)]
         for p in pixels.getPlanes(zctList):
             plane = np.maximum( plane, p )


If you already tried that, then I can't see anything else obvious from your script. Writing the many output images shouldn't be too memory intensive (it's only a jpg and a tiff per image, right)? Do you still run out of memory if writingSpotImage & writingCellImage are False?

I guess you could try the same approach for writingResultTable, just so you know if the memory issue is due to reading / analysis rather than handling results.

Do you always get the OutOfMemoryError on RawPixelsStore.getPlane() or sometimes on another call?

If none of these ideas help, you could try looking at the server Blitz log at the time that the script fails & see if there's anything obvious. Or maybe send us a chunk of it at that point.

Sorry I couldn't be more help,

Will.

Re: Java heap space pb with script ran on > 100 images

PostPosted: Mon Oct 21, 2013 10:45 pm
by tom_deschamps
Thanks, I've increased the memory space reserved by the application in the config files and it got rid of the problem so far.
All times are UTC
Page 1 of 1
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/