slow retrieval of hypercube data from OMERO

[gyellen]

Hi all -
I've been wrestling with the slow retrieval of FLIM data (and more generally of all data) from an OMERO server.

I can test the actual transport time of data from the server by fetching the data of a large FileAnnotation. I can reliably read 1.2 MBytes in 0.15 - 0.32 sec (this is the time just for the store.read(size) operation).

In contrast, when these data are stored as 256 planes of 2 channels of 128x128 uint16 pixels (which is 1.68 MBytes uncompressed), it takes 2.25 - 2.5 seconds for the store.getHypercube(...) operation.

It even takes 0.6 - 0.7 sec to retrieve only 60 planes of data using getHypercube. I can't tell from reading the code whether the planes are assembled into a hypercube on the server side, or whether the planes are requested individually by the client. In any case, the performance is very slow.

I'm inclined at this point just to save my FLIM data as a FileAnnotation to the smaller amount of image data, but this defeats the interoperability of having data stored as OMERO images. (I would even consider storing ALL of my data as FileAnnotations!).

It would be better if the performance could be improved instead.

Best,
Gary

[gyellen]

I misstated one of the benchmarks. The correct values are (using average values):

1.2 MByte file, retrieved by rawFileStore.read(size): 0.23 sec (min of 0.14 s)

The same data in this 1.2MB file, moved as a hypercube:
512x(128x128) uint16, retrieved by rawPixelsStore.getHypercube(offset,size,step): 2.38 sec (min of 2.23 s)

I also realized that the main difference might be data compression. The uncompressed size of the data is actually 16.7 MByte. So if the getHypercube data are being transported in uncompressed form, the difference in time would be explained.

Is there any data compression used during server-client transport of pixels data?

Thanks - gary

[gyellen]

Aha! I added the following lines to the ice.config file:

Ice.Override.Compress=1
Ice.Compression.Level=8
# (doesn't improve with Level=9)

This reduces the speed of the already-compressed file transport (0.5 s, up from 0.23 s), but dramatically improves the speed of the getHypercube move (0.87 s, down from 2.23 s).

I still wonder if getHypercube performance can be improved, if the client is currently asking the server for individual planes (I can't tell). Would it be possible instead for the hypercube structure to be 'assembled' on the server side, and sent as a single message to the client?

Thanks - gary

[i.munro]

Hi again Gary

Sorry I can't help with your issue but I am interested in the performance increase you saw by modifying the config file. Do you think this approach might work with our uncompressed data?

Thanks

Ian

[sbesson]

Hi Gary,

thanks for investigating this performance issue. Are you also using an unencrypted session as described in http://www.openmicroscopy.org/site/supp ... ure-client? If not, this may be another way for you to speed up data transfer.

Best,
Sebastien

[gyellen]

Hi Ian -
Yes, I think it should help - the only time I saw a decrease in performance was when I moved data of an already-compressed OriginalFileI.

The trick is to include the lines I mentioned in the ice.config file, but also to make sure that the file is used in the loadOmero call. I know that including some/all arguments in the loadOmero call prevents ice.config from being loaded.

Sebastien - I'll have to check on the encryption selection - I'll get back to you.

Best,
Gary

[sbesson]

Regarding the connection via a configuration file, there are two ways to achieve it (assuming you have a valid configuration file)

- either pass no argument to loadOmero/connectOmero. In this case, the function will pick up the configuration file defined by the ICE_CONFIG environment variable or found in the MATLAB path

Code: Select all

[c,s] = loadOmero()

See https://github.com/openmicroscopy/openm ... mero.m#L49

- or pass the absolute path to the configuration file directly to loadOmero

Code: Select all

[c,s] = loadOmero('/path/to/ice.config')

See https://github.com/openmicroscopy/openm ... mero.m#L59

Best,
Sebastien

[i.munro]

Thanks both. I'll experiment as soon as I get a chance.
Regards

Ian