Open Microscopy Environment

i just finished installing omero 4.1.1 and it works fine except that i can't import svs files. a more detailed description of the problem is here:
http://qa.openmicroscopy.org.uk/qa/feed ... 434a14fd29

basically i changed the templates like this: <option>-Xmx3072M</option>, and the server has 4Gb of RAM.
i have no problem importing jpg files, it's only svs files that give this error, even the small ones (17Mb). i don't think it's acompatibiulity issue, because i can use the bioformats.jar in ImageJ to open those files with no problem whatsover.

thanks a lot

Hello.

Apologies that no one got to your feedback sooner. We have a backlog which we are working through, but it's good you also found the forum.

The OutOfMemoryError which you are seeing is not occurring in the server, but in the client. Modifying the template is definitely the right thing to do if your server is having issues, but you'll also need to modify the value before starting up the importer.

Which client package on which OS are you using? I can provide more detailed instructions, but briefly:

• For Linux, you will need to modify the shell script.
• For Mac, you will need to right-click on the .app, choose "Show Package Contents", go into "Contents", and double click on Info.plist. There you can edit the Java > VMOptions property.
• For Windows, you will need to pass the -Xmx option to the .exe.Update: This information is wrong! See below for how to specify options to the .exe

Hope that helps.
~Josh.

Josh,
thanks a lot for your help. sorry for not mentioning that both the server and the client are in windows.

if i understand correctly, you suggest that i should pass the -Xmx to OMERO.importer.exe? i tried
OMERO.importer.exe -Xmx1024

but nothing happens, the appliocation doesn't even try to start. would you mind posting the exact command?

thanks a lot

Marius

i have attempted to import files directly on the omero server:

C:\omero_dist\bin>omero import -s localhost -u username -w pass -d 52 c:\images

it didn't work either. here's the error message:

2009-12-18 12:53:21,333 0 [ main] WARN ome.formats.importer.uti
l.IniFileLoader - Creating temporary ini file: C:\Documents and Settings\Admini
strator\omero\tmp\omero_Administrator\952@omero\.omero.importer71758378153981088
66ini
2009-12-18 12:53:21,349 16 [ main] INFO ome.formats.import
er.ImportConfig - OMERO Version: Beta-4.1.1-r5927-b91
2009-12-18 12:53:25,380 4047 [ main] INFO ome.formats.import
er.ImportConfig - Bioformats version: 4.1.1 revision: 5720 date: 25 November 20
09
2009-12-18 12:53:26,271 4938 [ main] INFO ome.formats.importer.I
mportCandidates - 2 file(s) parsed into 2 groups with 2 call(s) to setId in 656
ms. (672ms total) [0 unknowns]
2009-12-18 12:53:27,146 5813 [ main] INFO ome.formats.importe
r.ImportLibrary - Reader is not of HCS domain, use metafile: true
2009-12-18 12:53:27,583 6250 [ main] ERROR ome.formats.importer.c
li.ErrorHandler - INTERNAL_EXCEPTION: c:\images\S02-3-1.svs
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:520
)
at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.jav
a:185)
at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporte
r.java:128)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter
.java:336)
Caused by: java.lang.OutOfMemoryError: Java heap space
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:540)

at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412
)
... 3 more

java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:520
)
at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.jav
a:185)
at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporte
r.java:128)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter
.java:336)
Caused by: java.lang.OutOfMemoryError: Java heap space
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:540)

at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412
)
... 3 more
2009-12-18 12:53:27,599 6266 [ main] INFO ome.formats.importe
r.ImportLibrary - Exiting on error

is anybody able to import svs files into omero? is there a size limitation? what is the system memory recommendation?
thanks a lot

Hi Marius,

with the current version, you'll need to make sure that whatever client-side importer executable has an increased memory setting. For OMERO.importer.exe, I was wrong above. Instead, from http://www.ej-technologies.com/products/exe4j/features.html:

VM parameters file
For every executable, you can create a user editable VM parameters file. If your executable is called hello.exe, the VM parameters file is called hello.exe.vmoptions and each line in it is added as a single VM parameter.

So, create a file "OMERO.importer.exe.vmoptions" beside the existing "OMERO.importer.exe" file with the contents (for example):

Code: Select all

-Xmx2000M


Just be sure that notepad (or similar) doesn't add a ".txt" to the end of the file name.

To use "bin/omero import", you can set JAVA_OPTS:

Code: Select all

set JAVA_OPTS=-Xmx2000M
bin\omero import ...


For future versions, we'll work on reducing the memory required for importing SVS files. Keep us posted on your results, including how much memory you need for a file of a given size.

Cheers,
~J

your solution for addressing the java outofmemory seems to have worked (-Xmx1024m), however, now ice is complainig about not enough memory:
2009-12-30 17:10:54,149 199047 [2-thread-1] ERROR ome.formats.importer.gui.ImportHandler - Generic error while importing image.
2009-12-30 17:10:54,149 199047 [2-thread-1] DEBUG ome.formats.importer.gui.GuiImporter - Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeByteSeq(BasicStream.java:728)
at Ice.ByteSeqHelper.write(ByteSeqHelper.java:19)
at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1123)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
at ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:3047)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:581)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412)
at ome.formats.importer.gui.ImportHandler.importImages(ImportHandler.java:182)
at ome.formats.importer.gui.ImportHandler.access$100(ImportHandler.java:45)
at ome.formats.importer.gui.ImportHandler$2.run(ImportHandler.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeByteSeq(BasicStream.java:728)
at Ice.ByteSeqHelper.write(ByteSeqHelper.java:19)
at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1123)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
at ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:3047)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:581)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412)
at ome.formats.importer.gui.ImportHandler.importImages(ImportHandler.java:182)
at ome.formats.importer.gui.ImportHandler.access$100(ImportHandler.java:45)
at ome.formats.importer.gui.ImportHandler$2.run(ImportHandler.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

do you know what should i do to allocate more meory to ice?

thanks a lot.

here's an update with what i've done so far:
- i am running Omero Beta-4.1.1-r5927-b91 on a windows 2003 server with 4Gb of RAM.
- the file i am trying to import is only 17Mb
- created a OMERO.importer.exe.vmoptions file with -Xmx1524m; if i try -Xmx2048m, Omero.importer can't start.
- increased MESSAGESIZEMAX in Constants.ice to 5242800

the result? still unable to import the file:
Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeByteSeq(BasicStream.java:728)
at Ice.ByteSeqHelper.write(ByteSeqHelper.java:19)
at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1123)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
at ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:3047)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:581)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:412)
at ome.formats.importer.gui.ImportHandler.importImages(ImportHandler.java:182)
at ome.formats.importer.gui.ImportHandler.access$100(ImportHandler.java:45)
at ome.formats.importer.gui.ImportHandler$2.run(ImportHandler.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

at this point i will be greatful i somebody could tell me if they have been succesful in importing WSI files, and what was the largest file they were able to import. what is your system configuration?
thanks a lot and a Happy New Year!

To start I'd just like to preface by saying that OMERO is, at this point, not a system designed specifically for digital pathology. To date, we have not taken many digital pathology use cases into account and this is especially true of whole-slide scanning. Metadata support in particular is quite poor at present.

We are trying our best to begin to understand these use cases as best we can, and implement some solutions, in the run up to Beta 4.2.0.

For some technical background on why you're getting these errors, it's probably best to start with the file and a simple explanation on what the native format of an Aperio SVS file is and how OMERO treats image data. Firstly, while the image is 17MB it's also lossily compressed using JPEG2000. It is not uncommon to have whole-slide scans that are 20,000 x 10,000 or larger. Uncompressed this is many 100's of MB of data.

OMERO is a planar system and it uses a 2D plane as its unit of work due to the multi-dimensionality of digital microscopy images. These can often be 10's of GB in size. With many microscopists performing quantitative operations on images, lossy compression and invasive operations on the pixel data are avoided at all costs. Preserving the integrity of the data is something the OME project values very highly.

When OMERO.importer uses Bio-Formats to decompress an Aperio SVS file it does so on the entire image. Using 20,000 x 10,000 as an example, this requires approximately 600MB of memory. It also copies this entire 600MB of pixel data to the server as a single byte array. Furthermore, many operations within OMERO must operate on the entire image, this includes:

• Thumbnailing
• Visualization
• Projections

Removing this requirement is an ongoing project that we currently refer to internally as "big images" and it applies not only to digital pathology but also to large detector electron microscopy and stitched image high-content screening.

So, with all that said here's what you're going to need to do in order to support this in OMERO Beta 4.1.1:

• Run the OMERO.server on a 64-bit system with at least a Java virtual machine maximum heap size of 2GB
• Adjust the MessageSizeMax on the OMERO.server, OMERO.insight and OMERO.importer to at least 1GB

These adjustments may require re-compiling some of the OMERO support libraries and running debug builds of all client software. We are also actively trying to solicit help narrowing down use cases and file format support from the digital pathology community. You will see resources put up to this effect over the next few weeks.

Are you really sure you want to do this?

Is there a specific use case you're trying to solve?

Hi!
and thanks a lot for the clarifications! they are much appreciated!
like i said in my PM, i am very eager to help in any way i can, to test/debug/provide feedback, etc.

i've followed your recommendations and installed Omero server on a x64 box with 16Gb of ram.
i created an OMERO.importer.exe.vmoptions. what's interesting is that _xmx2000M does not work! the highest value i found (trial and error) to work is 1556. if i go higher that that i get an error message: "The JVM could not be started. The maim method may have thrown an exception"

like i said in my previous post, i've already adjust the MessageSizeMax on the OMERO.server. i edited the Constants.ice, is it correct? the value was adjusted to 5242800 (that's 5Gb, right?). i don't know how to adjust this parameter in OMERO.insight and OMERO.importer.

so far i'm still unable to import SVMs. i'm still getting that OMERO.insight and OMERO.importer i mentioned above.

please let me know if there's anything i can do at this point.
thanks a lot
marius

After playing with this a bit here the easiest way to achieve what you want is going to be building your own OMERO client library. For this you're going to need a Beta 4.1.1 checkout. The Subversion URLs are on the download page:

svn co http://svn.openmicroscopy.org.uk/svn/om ... mero-4.1.0

Once you've got this checked out you're going to want to modify Constants.ice as you've already found:

Code: Select all

...
35     /**
36      * Default Ice.MessageSizeMax (65536kb). Not strictly necessary, but helps to
37      * curb memory issues. Must be set before communicator initialization.
38      **/
39     const int MESSAGESIZEMAX = 65536;
...


Once you've done this you'll need to rebuild by running python build.py from the root of your OMERO checkout. Once you've got this built, you'll need to replace your existing new OMERO client JAR with the one you've built: dist/lib/client/omero_client.jar.

You'll need to be certain that you have a 64-bit JVM to increase the maximum heap size beyond about 1.5GB. You may have to download the 64-bit JDK in order to do this.

Let me know how this goes and then I'll get you started on updating the server with the client JAR you've built.