IMPORT planes in parallel threads, ims format

[romainGuiet]

Dear OME community,

I’m currently testing teraSticher (https://github.com/abria/TeraStitcher/wiki) that proposes to export files as :
-ims
-hdf5
Unfortunately, I can’t use (yet?) the h5 file with Fiji/bio-formats, BUT I can open the ims file.

I wrote an ImageJ plugin that can process planes (Z, C, T, Series) on parallel threads, thanks to the Bioformats virtualStack capabilities. (the processing get so fast when you have 32-cores at your disposal!).

While it works fine (multiple planes in parallel simultaneously) if the file is an ICS file,
When it’s an IMS file, the task is done on multiple cores BUT sequentially! It seems that the processing of each plane is done on a different core but starts when the previous one is done!

Would you have any input on this issue?
Is that a limitation of the .ims file format ? or the way the reader access data in the .ims file ?

Thank you for your help,

Romain

[dgault]

Hi Romain,

Although we do not guarantee that Bio-Formats readers are fully thread safe my initial impression is that there should be no particular difference between the ICS and IMS image readers. However I have not been able to test the IMS reader specifically so there may indeed be a limitation. Would you have a sample macro or code snippet of the type of processing you are attempting?

David Gault

[romainGuiet]

Hi David,
Thank you for your reply.
You will find here a zip containing the plugin and the source.

The multithread part is based/copied of : “Multithreaded programming for ImageJ” is in the IMS_issue_4OME.java
The image initialization is also based on an example and can be found in the IMS_issue_4OMEParam.java

To test it, you can use this dataset, an image 20Zx3C in formats ids/ics and ims.

Using the ids, the planes are processed in parallel simultaneously ( in about 20s) while using the ims the planes are processed in parallel “sequentially” ( takes about 180s).

Thank you for your help and suggestions to solve this.
Romain

[dgault]

Thanks Romain, I have downloaded the dataset and source provided. Over the next day or two I will test and debug the processing to see if I can isolate any improvements or bugs. I shall keep this thread up to date with any progress I make.

[dgault]

To follow up on this, I have been able to take a look through the source code in more detail and in general the structure of the plugin looks good from a Bio-Formats perspective.

The only section in which Bio-Formats will synchronise the threads should be the getProcessor call on BFVirtualStack. This may be the cause of the apparent sequential execution, although I am still unsure as to why IMS images appear to be affected specifically. My suggestion would be to test retrieving the processors earlier outside of the threaded execution.

[romainGuiet]

>To follow up on this, I have been able to take a look through the source code in more detail and in general the structure of the plugin looks good from a Bio-Formats perspective.

I'm glad that "the structure of the plugin looks good"

>The only section in which Bio-Formats will synchronise the threads should be the getProcessor call on BFVirtualStack. This may be the cause of the apparent sequential execution, although I am still unsure as to why IMS images appear to be affected specifically.
That's my main concern here, why IMS is treated differentially than IDS?

> My suggestion would be to test retrieving the processors earlier outside of the threaded execution.
Could you please explain how it would help us ?
Correct me but then it won't be parallel anymore ? so we won’t see any difference?
OR
we'll have to pre-load the whole data set and there is no use for virtual stack and we can't process files bigger than the RAM.

Thanks for your help with this,
Romain

[dgault]

Hi Romain,

I have pulled out the core parts of the code from the plugin to a standalone Java program for testing with Imaris files. From what I am seeing the main threaded section is being executed in parallel as expected. Surrounding it with timing commands is also showing an improvement in execution time with a greater number of threads. This appears to confirm that the issue is not fundamentally linked to Imaris files.

As mentioned previously the getProcessor call on BFVirtualStack is a synchronised method, from debugging and checking execution times it does seem that some threads are being held up at this point waiting for the getProcessor call. The effect of this will vary depending on the particular image. For some smaller sample images this does not have much of a negative effect. From the sample file you provided however this getProcessor call is taking a significant portion of the execution time which causes it to hold up other threads execution giving the impression that the processing is happening sequentially.

David Gault

[Oli]

Dear David,

Thank you so much for your time and effort!

From what you are saying, I am interpreting it this way
This getProcessor() from BFVirtualStack call ends up being slower for .ims files than for the .ids/.ics, even though we call for the same pixel data.

It turns out it's so slow that it becomes the main bottleneck of this (very simple) processing, making it seem like it's not running in parallel…

Is this correct?

Do you have any thoughts on how we can change the structure of the code to try and circumvent this issue? Perhaps declare multiple readers to read the same file at different locations? Something more elegant?

As always, thank you very much for your time.

Oli

[dgault]

Hi Oli,

That is correct, it is the call to getProcessor that is taking much longer for the ims files and causing the bottleneck here. In terms of improving the performance my initial thoughts were the same as yours, to split the getProcessor over an array of BFVirtualStack objects.

I ran some initial tests for this and using an array of BFVirtualStack with the same reader allowed the threads to run in parallel although it looks to have thrown IllegalStateExceptions in the NetCDF library. Running with an array of BFVirtualStack each with a separate reader appears to have worked much better with a significant speed up in execution time. This will come at the cost of extra overhead of memory performance though, for this particular file it accounted for approx 20Mb for each extra BFVirtualStack and 20Mb per processor.

David Gault

[Oli]

Hi David,

Sorry for the late reply. We checked and indeed it does work better!

So we would initialize as many readers and BFVirtualStack as needed, then split the threads.

Thanks for checking the extra overhead, which is perfectly acceptable in our case.

Thanks again for all the great work!

Oli