Importing Performance

[30laur]

Hi,
were doing some simple performance testing of OMERO using the importer tool. We have about 19MB to 20MB of 145 images and we noticed that it takes about 1 minute to go through the following OMERO importer process (we see these on the Importer status bar)
1. 1/3
2. 2/3
3. 3/3
4. Archiving
5. Thumbnail
6. Prepping
7. Updating DB

The most CPU intensive process is 7. Updating DB
The most longest process is 7. Updating DB
The second most longest process is 5. Thumbnail

Can anyone explain what the status above are doing and how we could reduce the CPU and timeframes?
e.g. are there any OMERO config parameters we could tune?

[jrswedlow]

Hi 30laur-

To answer your question, we need a bit more detail.

What file format are you uploading? What are the x,y sizes of the images you are loading?

Cheers,

Jason

[30laur]

Hi Jason,
uploading NEF files which average around 19MB to 20MB in size.
Dimensions (XY) 4320 x 2868
Pixel Size (XY) 84.6677 x 84.667

We are currently "testing the performance of OMERO" as we wanted to see how it could handle a "big load".
Stats so far
NEF's-> 143/145 files uploaded, about 20 MB per file, 2.9 GB total files. 2 hours & 41 minutes
JPGs-> 1.13 hours, 482 JPG files of various sizes (300kb to 500kb) , 369.08 gb in total

Performance will be the biggest issue - i.e. retrieval of files etc...
Regards,
Raymond

[cxallan]

I assume these images are photographs?

[30laur]

Yes

[cxallan]

So first some background. We consider all images quantitative data and therefore are very careful to preserve the pixel intensities of those images. Thus in your case we completely decompress each of the JPEG and NEF files into their core 8-bit (or higher) colour components. This results in a huge explosion of image data volume. OMERO supports no compression internally at present.

Furthermore, OMERO is not a system designed with digital photography use cases in mind and is not going to give you good performance when working with these types of files. This is a result of our data storage mechanisms, outlined above, and our design decisions surrounding the use cases of digital microscopy.

Are their digital microscopy use cases you have in mind that you'd like to optimize for? What sort of hardware are you running the OMERO server and clients on?