Large ModuloAlongT data - import is slow

[yuriy_alexandrov]

Hi All,

Our FLIMfit software can work with non-imaging ("single-pixel") data.
When importing such data to Omero, the lifetime axis is chosen to be not XY - the convention is ModuloAlongT.

For such "single-pixel" data, I noticed that placing lifetimes along non-XY dimension is very slow.

In particular, we have real-world high-resolution 31 channel TCSPC data from cuvette system (X=1, Y=1, Z=1, T=4096, C=31).
Every 4096-bin channel is imported in 30 seconds, so it takes 16 minutes to import the whole 1.3M file.

My laptop is slow but I think the problem is still there, - for the comparison, the 32M (4-channel 128x128 256-bin) sdt image is imported in 30 seconds.

The slow part of the code is this, and the problem is most likely in "setPlane" function.

Code: Select all

           
...
bytear = ConvertClientToServer(pixels, plane) ;
rawPixelsStore.setPlane(bytear, int32(z-1), int32(c-1), int32(t-1));               
...


Is it possible to speed it up, say if applying other "set" function?

Thanks,
Y.

[sbesson]

Hi Yuriy,

Looking at the mat2omeroImage_native function in your code, here are 2 thoughts:
1- the API is not optimized for the type of data you are trying to upload. Here, you need to invoke setPlane for every single pixel in your input matrix within a triple loop.
2- profiling your code, a lot of the overhead actually arises from the setChannelGlobalMinMax method that is being invoked on every channel.
Typically,

Code: Select all

mat2omeroImage_native(session, zeros(1,1,1,1000,'uint16'), 'uint16', 'test', 'desc','')

runs in 392s with the MATLAB Profiler turned on where 40s are spent in ConvertClientToServer and setPlane while 298s are spent in setChannelGlobalMinMax.

I opened a PR against your code (https://github.com/openmicroscopy/Imper ... it/pull/65) suggesting refactoring changes. With this PR in, the same test function as above runs in the same condition in only 111s with 37s spent on the IUpdate.saveAndReturn function updating the channels. Can you try out these changes? This should at least decrease the total amount of time. I also think your function can be further optimised for this type of data.

Let us know if this works,
Sebastien

[i.munro]

Thanks Sebastien. I'll merge that & get Yuriy to test when he gate back after Christmas.

Ian

[yuriy_alexandrov]

Thanks Sebastien,

I remember trying the similar simplified vesrion but didn't notice improvement.
I will look at it again in January.

Wish good Christmas and New Year to all!

Best regards,
Y.