Local OMERO server down?

[ferranda]

Hi,

We implemented our facility with OMERO a few months ago, using the OMERO virtual machine. We have a few test users using it now.

Everything is going smoothly, and we have good feedbach from users.

Unfortunately, today we are experiencing an issue on our server. We cannot retrieve data, or import new ones. But we can see the tree of data for each user. The user was working with a large amount of data yesterday and only today we cannot retrieve the data.

The cache might have run out of memory? Or we lost the connection somehow?
We have different error messages depending on the action we do: retrieving data or importing new data. See below for the error messages.

We are using OMERO 4.3.3.

Any tips to try to fix this problem, please?

Thanks!



The error log when trying to retrieve data:

java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot load hierarchy for class pojos.DatasetData.
at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:911)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadContainerHierarchy(OMEROGateway.java:2655)
at org.openmicroscopy.shoola.env.data.OmeroDataServiceImpl.loadContainerHierarchy(OmeroDataServiceImpl.java:203)
at org.openmicroscopy.shoola.env.data.views.calls.DMLoader$1.doCall(DMLoader.java:87)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._IContainerDelM.loadContainerHierarchy(_IContainerDelM.java:592)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:730)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:702)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadContainerHierarchy(OMEROGateway.java:2651)
... 9 more

at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:182)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:271)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:241)
at org.openmicroscopy.shoola.agents.treeviewer.DataBrowserLoader.handleException(DataBrowserLoader.java:141)
at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84)
at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:682)
at java.awt.EventQueue.access$000(EventQueue.java:85)
at java.awt.EventQueue$1.run(EventQueue.java:643)
at java.awt.EventQueue$1.run(EventQueue.java:641)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:652)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)


The error message when trying to import new data:

Ice.UnknownLocalException
unknown = "TcpTransceiver.cpp:100: Ice::ConnectionLostException:
connection lost: Connection reset by peer"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._RawPixelsStoreDelM.setTile(_RawPixelsStoreDelM.java:1767)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2435)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2407)
at ome.formats.OMEROMetadataStoreClient.setTile(OMEROMetadataStoreClient.java:2522)
at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:884)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:796)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:573)
at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6402)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1228)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:111)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)

at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6447)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1228)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:111)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source) Caused by: Ice.UnknownLocalException
unknown = "TcpTransceiver.cpp:100: Ice::ConnectionLostException:
connection lost: Connection reset by peer"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._RawPixelsStoreDelM.setTile(_RawPixelsStoreDelM.java:1767)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2435)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2407)
at ome.formats.OMEROMetadataStoreClient.setTile(OMEROMetadataStoreClient.java:2522)
at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:884)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:796)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:573)
at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6402)
... 11 more

[jmoore]

How much memory have you given to your virtual machine? Have you modified the settings for OMERO at all? Do you remember what action you were performing when it ran out of memory?

Cheers,
~Josh

[ferranda]

Hi Josh,

I am not sure how much memory was assigned in the end. When I first installed the virtual machine, I did not modify the settings at all. Then we had to reinstall it later on, and my colleague is now in charge of OMERO, and he probably changed some settings, as we wanted to have it to a bigger scale.
Sorry I cannot be more precise...

The user was adjusting the intensity levels of about 200 dv files and using the batch processing. The progress window appeared then he had the error message that the data is not accessible (see first post).
From there, he could see his data tree (folders), but no thumbnails or images in the folders.

My colleagues in charge of this side of OMERO is back on Monday, I'll check with him.

Cheers,
Alexia

[ehrenfeu]

Hey Josh,

the machine still runs as it was preconfigured in the VirtualBox image, which means it has 1 GB of RAM (we just converted it into a VMware/ESX image to deploy it on our infrastructure here).

Modifications in the OMERO settings were just made for the storage location and the LDAP/ActiveDirectory authentication, nothing else was changed. More precise, the java processes still run with the "-Xmx512M" flag if this is your question

I don't know for sure what the user did exactly when this error showed up, but I will try to reproduce it with him - yet this will take some time as he is rather busy preparing a talk... As Alexia has already mentioned, he probably tried to batch process some of his files, but we'll see.

Anything useful I can figure out? If it helps, I can attach the logfiles (or use gist if you prefer).
Cheers,
Niko

[cxallan]

At a minimum it would be good to check the disk space allocated to ensure that it hasn't filled up. Failing an issue with that the output of bin/omero admin diagnostics would be a useful first start.

[ehrenfeu]

The disk space definitely hasn't filled up. The repository storage has ~250GB free space, the root file system more than 25GB.

Here's the diag output:

Code: Select all

================================================================================
OMERO Diagnostics 4.3.3-1155-ac713a4-ice33-b2964
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to Beta-4.3.4 See http://trac.openmicroscopy.org.uk/omero for the latest version

Commands:   java -version                  1.6.0     (/usr/bin/java)
Commands:   python -V                      2.6.6     (/usr/bin/python)
Commands:   icegridnode --version          3.3.1     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.3.1     (/usr/bin/icegridadmin)
Commands:   psql --version                 8.4.8     (/usr/bin/psql -- 2 others)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 20799, enabled)
Server:     DropBox                        active (pid = 20818, enabled)
Server:     FileServer                     active (pid = 20819, enabled)
Server:     Indexer-0                      inactive (disabled)
Server:     MonitorServer                  active (pid = 20821, enabled)
Server:     OMERO.Glacier2                 active (pid = 20822, enabled)
Server:     OMERO.IceStorm                 active (pid = 20823, enabled)
Server:     PixelData-0                    inactive (disabled)
Server:     Processor-0                    active (pid = 20825, enabled)
Server:     Tables-0                       active (pid = 20826, enabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /home/omero/OMERO.server/var/log exists

Log files:  Blitz-0.log                    16.0 MB       errors=4    warnings=62
Log files:  Blitz-0.log.1                  512.0 MB      errors=479  warnings=711
Log files:  DropBox.log                    14.0 KB       errors=0    warnings=5
Log files:  FileServer.log                 2.0 KB
Log files:  Indexer-0.log                  208.0 KB      errors=7    warnings=0
Log files:  MonitorServer.log              9.0 KB        errors=0    warnings=4
Log files:  OMEROweb.log                   0.0 KB
Log files:  OMEROweb_request.log           0.0 KB
Log files:  PixelData-0.log                211.0 KB      errors=7    warnings=0
Log files:  Processor-0.log                2.0 MB        errors=1    warnings=1
Log files:  Tables-0.log                   9.0 KB        errors=0    warnings=2
Log files:  TestDropBox.log                n/a
Log files:  master.err                     351.0 KB
Log files:  master.out                     0.0 KB
Log files:  Total size                     531.97 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-6-sun/bin:/usr/lib/jvm/java-6-sun/bin:/usr/share/Ice-3.3.1:/usr/lib/postgresql/8.4/bin:/home/omero/OMERO.server/bin
Environment:ICE_HOME=/usr/share/Ice-3.3.1
Environment:LD_LIBRARY_PATH=/usr/share/java:/usr/lib:
Environment:DYLD_LIBRARY_PATH=/usr/share/java:/usr/lib:

OMERO data dir: '/export/OMERO.data/'   Exists? True    Is writable? True
OMERO.web status... [RUNNING] (PID 6643)


[cxallan]

All right that's one issue down then. Can you ZIP up your var/log/Blitz-0.log and attach it here please?

[ehrenfeu]

I'll strip it down to the relevant date and try to attach it here. What filename suffixes are allowed for attachments btw? (log and txt are not...)

[cxallan]

Just ZIP it up, those are fine.

[ehrenfeu]

Sorry, but I've been pretty busy the last days...

In addition, the board software allows just for file sizes of max. 256 KB, so eventually I placed the 7-zip compressed logfile in the download space of our xuvtools project:

http://data.xuvtools.org/tmp/Blitz-0_2012-05-17.log.7z

You can find the corresponding error message in line 495460 (or using the timestamp "2012-05-17 18:46:49,369")

Cheers
~Niko