setup/configuration for big images

[achessel]

Hi all,

We are in the process of setting up an OMERO server, in particular to deal with big images (big stitched confocal stacks and SPIM in particular) and I thought I would seek advice on configuration and setup, both from devs and working setups. I have experience with omero but not (yet) with big images and initial testing indicates that things kinda have to be well thought from the start or one is quickly overwhelmed...
- what sort of machine, in particular for pyramid generation and viewing? Where would the bottleneck(s) be, memory, disk IO, CPU, bandwidth to client...? Can virtual machine be an viable option?
- what configuration to change on OMERO? There is bunch of relevant things in the 'Configuration properties glossary', but I am not sure what they affect/how to change them.

I did initial tests on an oldish test server (8Gb ram 4/8 cores with a network mounted drive) with vanilla OMERO and while it worked well for a 3.5kx3.5k image, pyramid generation for a 12k x 3k x 500 volume took 2-3 days and viewing is very (very) sluggish (say >30s to 1-2 minute when one changes plan or zoom level). I'd say bottleneck might be CPU and/or network speed to the network disks?...

More generally how are things expected to scale with image size? How big is too big for pyramid generation and/or comfortable viewing? Also would the performance change depending on the original file format (say OME-TIFF vs plain TIFF vs LIF vs Imaris)?

Lots of question sorry (is there a 'big image' doc page I missed?)
Many thanks

Anatole

[jmoore]

Hi all,

Hi Anatole,

First off, perhaps you can tell us what you mean by big in each of the regular dimensions. Also, what file format?

- what sort of machine, in particular for pyramid generation and viewing? Where would the bottleneck(s) be, memory, disk IO, CPU, bandwidth to client...? Can virtual machine be an viable option?

Disk IO and memory are probably the larger bottlenecks since the pyramid generation is not particularly multi-threaded. In a VM, I think disk IO will be the largest problem.

- what configuration to change on OMERO? There is bunch of relevant things in the 'Configuration properties glossary', but I am not sure what they affect/how to change them.

For pyramid generation, these properties are the most important: https://www.openmicroscopy.org/site/support/omero5.1/sysadmins/config.html#pixeldata

And if it doesn't but too much strain on the system, https://www.openmicroscopy.org/site/support/omero5.1/sysadmins/config.html#omero-pixeldata-threads is likely the one you'd like to increase the most.

I did initial tests on an oldish test server (8Gb ram 4/8 cores with a network mounted drive) with vanilla OMERO and while it worked well for a 3.5kx3.5k image, pyramid generation for a 12k x 3k x 500 volume took 2-3 days and viewing is very (very) sluggish (say >30s to 1-2 minute when one changes plan or zoom level). I'd say bottleneck might be CPU and/or network speed to the network disks?...

What configuration settings did you use? Most important is likely the output of `bin/omero admin jvmcfg`.

More generally how are things expected to scale with image size? How big is too big for pyramid generation and/or comfortable viewing?

Good question. Perhaps someone else has an answer for what image has simply been too large.

Also would the performance change depending on the original file format (say OME-TIFF vs plain TIFF vs LIF vs Imaris)?

Certainly. Optimal in terms of pyramid generation would be any format which includes pre-generated resolution levels, meaning OMERO spends no time re-generating.


Cheers,
~Josh

P.s. currently there's only the http://www.openmicroscopy.org/site/support/omero5.1/developers/Server/ScalingOmero.html page.

[achessel]

Hi Josh, thanks for answering

First off, perhaps you can tell us what you mean by big in each of the regular dimensions. Also, what file format?

For those tests 'big' is 12k*3k with 400 zslice and 3 colors. It's stored in plain tiff (output from fiji stiching). I wouldn’t think of that as 'too big', I am doing more tests do debug the slow viewing, it could be just due to server config here.

For pyramid generation, these properties are the most important: https://www.openmicroscopy.org/site/support/omero5.1/sysadmins/config.html#pixeldata

And if it doesn't but too much strain on the system, https://www.openmicroscopy.org/site/support/omero5.1/sysadmins/config.html#omero-pixeldata-threads is likely the one you'd like to increase the most.

omero.pixeldata.threads would help when generating several pyramid (ie importing several big images) at the same time, but not really speeding up one in particular, right?

Are omero.pixeldata.tile_height and omero.pixeldata.tile_width for the smallest tiles? So would increasing them decrease pyramid generatioin (less levels) but increase viewing time at the highest zoom (bigger tiles to load)?

What configuration settings did you use? Most important is likely the output of `bin/omero admin jvmcfg`.

It was all default values for 5.1.4 (so 2 threads and 256*256 tiles?) and

Code: Select all

omero@ubuntutest:~/OMERO.server$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx1253m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx835m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx1253m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx835m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


Certainly. Optimal in terms of pyramid generation would be any format which includes pre-generated resolution levels, meaning OMERO spends no time re-generating.

What format would that be (that have pyramids and that bioformat/omero recognises/uses as multiresolution)? (I am just getting into big images...). Imaris 5.5 (which I just see made it's format 'open')? Would it help for viewing to or just import?

Thanks again
A.

[jmoore]

Hi Anatole,

For those tests 'big' is 12k*3k with 400 zslice and 3 colors. It's stored in plain tiff (output from fiji stiching). I wouldn’t think of that as 'too big', I am doing more tests do debug the slow viewing, it could be just due to server config here.

Agreed, those don't sound completely horrible, though perhaps we've let a bug creep in with the z-dimension? We would have to catch the server doing something naughty to know for sure.

omero.pixeldata.threads would help when generating several pyramid (ie importing several big images) at the same time, but not really speeding up one in particular, right?

Correct.

Are omero.pixeldata.tile_height and omero.pixeldata.tile_width for the smallest tiles? So would increasing them decrease pyramid generatioin (less levels) but increase viewing time at the highest zoom (bigger tiles to load)?

This sounds correct, but I don't have any metrics to hand.

Code: Select all

blitz=-Xmx1253m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx835m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx1253m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx835m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


This is in general not a lot of memory for a production system. Blitz I would generally have at 8G or higher. For a big-image-heavy workflow, I'd likely make PixelData similarly large. Note: it could run on another system if the file system is visible to both and supports locking.

What format would that be (that have pyramids and that bioformat/omero recognises/uses as multiresolution)? (I am just getting into big images...). Imaris 5.5 (which I just see made it's format 'open')?

Imaris, and SVS also comes to mind.

Would it help for viewing to or just import?

Primarily import, but the tile-size etc. in a native format may be optimized for a particular image.

~J.

[achessel]

Thanks for those info.

I tested the same image in a server on a much beefier machine and the viewing was still fairly sluggish (though pyramid was now 'only' a few hours).
I then tried to extract just one plane (so 12k*3k*1*3 instead of 12k*3k*400*3, still plain tiff saved by fiji) and import it, and the viewing is now fairly smooth, including on the older machine.

Not sure what the hierarchical viewing algo is but since only one plane is viewed at a time, wouldn't one expect the speed to be the same?

[sbesson]

Thanks Anatole,

we will have a go at reproducing your test case by importing fake images of the same dimensions into our CI servers. Which client have you been using for the viewing?

Best,
Sebastien

[sbesson]

Hi Anatole,

I finally managed to test your scenario. For this, I created a fake file named image&sizeX=12000&sizeY=3000&sizeZ=400&sizeC=3.fake. Since this is using the FakeReader, there is no subresolution optimization.

The large image was imported into a 5.1.4 server with the following memory settings:

Code: Select all

[hudson@ome-c6100-3 OMERO-5.1-latest-deploy]$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


Post import, the pyramid generation took roughly an hour to complete:

Code: Select all

2015-10-21 20:52:12,855 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Creating BfPixelBuffer: /repositories/OMERO-5.1-latest-deploy/ManagedRepository/user-4_5/2015-10/21/20-52-06.105/image&sizeX=12000&sizeY=3000&sizeZ=400&sizeC=3.fake Series: 0
2015-10-21 20:52:12,855 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Destination pyramid tile size: java.awt.Dimension[width=256,height=256]
2015-10-21 20:52:12,857 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:20901 1/676800 (0%).
2015-10-21 21:00:34,411 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:20901 67681/676800 (10%).
...
2015-10-21 22:16:56,304 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-5) HANDLED EventLog:3106737(entityId=20901) [5084248 ms.]


Loading individual tiles using the OMERO.web client take between 1 and 7 seconds.

Best,
Sebastien

[achessel]

Hi Sebastien, thanks for looking into it...

Loading individual tiles using the OMERO.web client take between 1 and 7 seconds.

Do you mean the time it takes to open a web viewer? Or Once the web viewer is open, the time it takes when one do something (say change z or move to a completely different part of the image) until the image is completely loaded (say the control on the side are ungrayed)?

My observation was that that takes tens of seconds for a 400 z image stake but if I extract one z plane (same x, y, c) and load that as another image it would maybe take only one or two second...

Am I making sense? Is that expected? Could you reproduce it?

A.

[sbesson]

Hi Anatole,

yes I meant that once the Web viewer is opened, 1-7 seconds is the time it takes to fully refresh the window when either zooming, changing the Z slider or moving the image in XY.
Could you try importing the fake image I mentioned earlier on your server and let us know of the metrics you get (PixelData processing and image loading) as well as the JVM settings of your server?

Best,
Sebastien

[achessel]

Sure.
Hum, sorry I must have missed it but where the image?