OutOfMemoryError when importing svs

[mpetruc]

Hi again!
i finally got around to compile my own client. i used the subversion available at:
http://svn.openmicroscopy.org.uk/svn/om ... mero-4.1.1
- i tried to compile it on a X64 machine (64 bit python and java) without success (the 64 bit python does not include ctypes; was not able to install ctypes manually because of missing ml64.exe)
-on a 32 bit machine, i modified clients.ice and changed the MessageSizeMax to 2GB.
-i copied omero_client.jar to C:\OMERO.clients-Beta4.1.1.win\libs and renamed it omero_client-Beta-4.1.1-r5927-b91.jar.
the result?
no more "Out of memory error". instead i'm getting this one:
2010-02-08 11:07:01,065 283390 [2-thread-1] INFO ormats.importer.gui.LoggingImportMonitor - IMPORT_STEP Image: 1 Series: 0 Total Series: 4
2010-02-08 11:07:05,784 288109 [2-thread-1] INFO ormats.importer.gui.LoggingImportMonitor - INTERNAL_EXCEPTION: C:\images\S07-10268.svs
Ice.SocketException
error = 0
at IceInternal.TcpTransceiver.write(TcpTransceiver.java:146)
at Ice.ConnectionI.sendMessage(ConnectionI.java:1757)
at Ice.ConnectionI.sendRequest(ConnectionI.java:379)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1132)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
at ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:3047)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:581)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412)
at ome.formats.importer.gui.ImportHandler.importImages(ImportHandler.java:182)
at ome.formats.importer.gui.ImportHandler.access$100(ImportHandler.java:45)
at ome.formats.importer.gui.ImportHandler$2.run(ImportHandler.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(Unknown Source)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at IceInternal.TcpTransceiver.write(TcpTransceiver.java:99)
... 21 more

[....]
Ice.ConnectionLostException
error = 0
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._MetadataStoreDelM.createRoot(_MetadataStoreDelM.java:24)
at omero.api.MetadataStorePrxHelper.createRoot(MetadataStorePrxHelper.java:48)
at omero.api.MetadataStorePrxHelper.createRoot(MetadataStorePrxHelper.java:20)
at ome.formats.OMEROMetadataStoreClient.createRoot(OMEROMetadataStoreClient.java:500)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:523)
at ome.formats.importer.gui.ImportHandler.importImages(ImportHandler.java:182)
at ome.formats.importer.gui.ImportHandler.access$100(ImportHandler.java:45)
at ome.formats.importer.gui.ImportHandler$2.run(ImportHandler.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2010-02-08 11:07:11,784 294109 [4-thread-1] ERROR me.formats.importer.util.ClientKeepAlive - Exception while executing ping(), logging Connector out:
2010-02-08 11:07:11,784 294109 [4-thread-1] DEBUG ome.formats.OMEROMetadataStoreClient - Logout called, shutting keep alive down.
2010-02-08 11:07:11,784 294109 [4-thread-1] DEBUG ome.formats.OMEROMetadataStoreClient - keepalive shut down.
2010-02-08 11:07:11,784 294109 [4-thread-1] DEBUG ome.formats.OMEROMetadataStoreClient - closing client session.
2010-02-08 11:07:11,800 294125 [4-thread-1] DEBUG ome.formats.OMEROMetadataStoreClient - client closed.

the 32 bit version compiled by me does not work on a x64 machine with 64 bit java.
am i making progress here? any other suggestions?

thanks a lot

marius

[cxallan]

Hello Marius!

This implies that something rather evil happened on the server. Can you have a look in the server log to see if there was an exception?

-Chris

[tstokes]

Hello,

I've been following along on this thread to try to increase my image capacity. I'm working on version 4.2 now. My client and my server are both running 64-bit Java, but my Importer executable won't start if I increase -Xmx beyond 1546. It says the 'main' method threw an exception.

I have recompiled with the new setting in Constants.ice. After an import fails with the Socket closed exception:
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(Unknown Source)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at IceInternal.TcpTransceiver.write(TcpTransceiver.java:99)
at Ice.ConnectionI.send(ConnectionI.java:1679)
at Ice.ConnectionI.socketReady(ConnectionI.java:1106)
at Ice.ConnectionI$SocketReadyCallback.socketReady(ConnectionI.java:2299)
at IceInternal.SelectorThread.run(SelectorThread.java:203)
at IceInternal.SelectorThread$HelperThread.run(SelectorThread.java:273)

As an aside, this error causes my client to be logged out from the server.
I check the /var/log directory and the only one with interesting messages is Blitz-0.log

2010-07-20 19:27:12,517 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.LogicalChannel,2602
2010-07-20 19:27:12,517 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.Channel,2602
2010-07-20 19:27:12,518 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.acquisition.DetectorSettings,53
2010-07-20 19:27:12,518 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.LogicalChannel,2603
2010-07-20 19:27:12,519 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.Channel,2603
2010-07-20 19:27:12,520 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.PlaneInfo,51
2010-07-20 19:27:12,520 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.PlaneInfo,52
2010-07-20 19:27:12,521 INFO [ ome.security.basic.CurrentDetails] (l.Server-4) Adding log:INSERT,class ome.model.core.PlaneInfo,53
2010-07-20 19:27:12,524 INFO [ org.perf4j.TimingLogger] (l.Server-4) start[1279668432436] time[88] tag[omero.saveImportGraph]
2010-07-20 19:27:12,554 INFO [ org.perf4j.TimingLogger] (l.Server-4) start[1279668432429] time[125] tag[omero.call.success]
2010-07-20 19:27:12,554 INFO [ ome.services.util.ServiceHandler] (l.Server-4) Rslt: (ome.model.core.Pixels:Id_2051)
2010-07-20 19:27:12,608 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Meth: interface ome.api.RawPixelsStore.prepare
2010-07-20 19:27:12,608 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Args: [(2051)]
2010-07-20 19:27:12,612 INFO [ ome.security.basic.EventHandler] (l.Server-1) Auth: user=2,group=13,event=null(User),sess=1c71c703-9b5c-4e64-8e5f-f87881aade7b
2010-07-20 19:27:12,621 INFO [ org.perf4j.TimingLogger] (l.Server-1) start[1279668432608] time[13] tag[omero.call.success]
2010-07-20 19:27:12,621 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Rslt: null
2010-07-20 19:27:23,474 INFO [ ome.services.blitz.impl.ServiceFactoryI] (l.Server-6) Keep alive: 1c71c703-9b5c-4e64-8e5f-f87881aade7b
2010-07-20 19:27:26,088 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Meth: interface ome.api.RawPixelsStore.setPixelsId
2010-07-20 19:27:26,088 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Args: [2051, true]
2010-07-20 19:27:26,090 INFO [ ome.security.basic.EventHandler] (l.Server-7) Auth: user=2,group=13,event=null(User),sess=1c71c703-9b5c-4e64-8e5f-f87881aade7b
2010-07-20 19:27:26,090 INFO [ ome.io.nio.PixelsService] (l.Server-7) Pixel buffer file exists returning ROMIO pixel buffer.
2010-07-20 19:27:26,093 INFO [ org.perf4j.TimingLogger] (l.Server-7) start[1279668446088] time[5] tag[omero.call.success]
2010-07-20 19:27:26,093 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Rslt: null

Are there any new ideas on this problem?

Thanks!
Todd

[cxallan]

The unfortunate reality, even with 4.2.0, is that very large X-Y images will still give you a heap of problems. This is something we're looking to address as soon as we can.

[jrswedlow]

Dear All-

Just following up on this.

First, as of OMERO 4.3, SVS files should import properly, and should be viewable in OMERO.insight and OMERO.web. Let us know if this is not the case.

Second, we've added a section to our documentation describing how to adjust memory usage in OMERO Java clients. See:

http://www.openmicroscopy.org/site/supp ... quirements

Cheers,

Jason