pixeldata threads/repetitions for big TIFF pyramid workload

[dtrudg]

Sebastien,

Many thanks for looking into this for us, and spending the time to reproduce etc.

Short summary is that:

• there should be significant performance improvements for the pyramid generation of this type of data notably due to PRs like https://github.com/openmicroscopy/bioformats/pull/1643 on the 5.1.x line
• the multi-threaded PixelData could certainly benefit for further investigation/optimization but this work is unlikely to be done for 5.1.0 at this point.

Good to see that there's already something in the pipeline that improves the speed. For now we can cope with the 5.0.6 speed given initial import sizes, and when 5.1.x is available we'll definitely be glad of the speedup.

Thanks again,

DT

Edit - NB - just deleted subsequent post as it was nonsense. Need coffee before I write things!

[sbesson]

HI Dave,

as a follow-up, OMERO 5.1.0-m5 is available as a preview of the upcoming release and should include the pyramid generation improvements described in this thread.

Best regards,
Sebastien

[dtrudg]

Hi Sebastien,

Thanks for the pointer. I'll try and check the m5 preview with our files sometime soon (but will be on a much smaller test machine). We'll definitely be updating to 5.1.0 when fully released.

Thanks,

DT

[dtrudg]

Just wanted to say that the 5.1.0 changes work wonderfully. Pyramid creation on our large TIFFs is now running very smoothly, taking only ~19s per file, with 4 being created in parallel.

Thank you for looking into the problem, and for the fix!

[sbesson]

Hi Dave,

thanks for letting us know and glad the 5.1.0 fix is improving performance for your use case.
Following up on another open question in this forum thread, do you observe the average time per pyramid to be roughly independent of the number of simultaneous threads for the same type of files on your production system?

Best regards,
Sebastien

[dtrudg]

Following up on another open question in this forum thread, do you observe the average time per pyramid to be roughly independent of the number of simultaneous threads for the same type of files on your production system?

Yes - it's around 19s on these TIFFs, regardless of whether there is 1 pyramid creation thread running or 4.

Thanks,

DT

[sbesson]

Hi Dave,

thanks that's good news. As always, let us know if you encounter performance issues with specific file formats in the future.

Best,
Sebastien