Error loading a (big) image in the fiji plugin

[achessel]

Hi all,

I am having issues opening some images using the omero imagej plugin. It just ends with 'an error occurred while loading the image'. They are rather big plain tiff file saved in fiji (12kx3k for example).

It imported all right in omero, and can be opened fine in insight, the original file can be opened fine in imagej with bioformats and other image in omero can be opened fine in fiji. Additionally omero logs seems fine as well, with the relevant calls seemingly finishing with a successful call to getPlane. There is just a lot of 'File is a raw codestream not a JP2.' and 'Unknown JPEG 2000 box'.

Any Idea of what is happening or how I could debug? is there a log in the omero imagej plugin side for example?

Many thanks.

[jburel]

Hi Anatole

Will it possible to submit one of the files so we can debug,
When the image stored in OMERO is opened in imageJ
we use the Bio-formats plugin, since can independly open the image in Bio-formats
something else migth be generating a problem

Have the images been recently imported? i.e. post migration to OMERO 5

Cheers
Jmarie

[achessel]

The server and client are 5.1.4 (for import and reading).
It still fails if I take only one plane of the big stack, which I can upload more easily than the whole thing(240 Mb, 10k*4k). Hum; where do I upload again?

Many thanks for the help...

A.

[manics]

where do I upload again?

https://www.openmicroscopy.org/qa2/qa/upload/

[achessel]

image uploaded , thanks again

[mlinkert]

Hi Anatole,

I am having issues opening some images using the omero imagej plugin. It just ends with 'an error occurred while loading the image'. They are rather big plain tiff file saved in fiji (12kx3k for example).

It imported all right in omero, and can be opened fine in insight, the original file can be opened fine in imagej with bioformats and other image in omero can be opened fine in fiji. Additionally omero logs seems fine as well, with the relevant calls seemingly finishing with a successful call to getPlane.

Thank you for uploading the file - I can reproduce the same error. The Blitz-0.log is clear, but master.err should show an IceMemoryLimitException around the time when image is opened in the OMERO ImageJ plugin.

There is just a lot of 'File is a raw codestream not a JP2.' and 'Unknown JPEG 2000 box'.

This isn't a problem in and of itself - it's just a side effect of the file containing a big image.

Any Idea of what is happening or how I could debug? is there a log in the omero imagej plugin side for example?

The master.err server log file should confirm that an IceMemoryLimitException was thrown by getPlane. This appears to be due to the plane being larger than Ice.MessageSizeMax (see https://www.openmicroscopy.org/site/sup ... agesizemax). Unfortunately, I see the same error even when the "Crop on import" option is selected in the "Bio-Formats Import Options" window. We'll look into how best to fix that as noted here:

https://trello.com/c/2GKksxIS/80-omero- ... big-images

In the meantime, increasing the value of Ice.MessageSizeMax may help as a temporary work-around.

Regards,
-Melissa

[achessel]

thanks for looking into it.

I had those messages, and increased the Ice.MessageSizeMax, and stoped having them (the master.err log file is not updated when that error occurs). Just in case I increased it again to above 500mb, which is more than the size of the whole image (is there a way to check that it was taken into account? I could not see a way to get the CLI to spit it the actual value of that parameter out), and the error is still occurring, with no master.err log...

Did it solve the issue on your end to increase that parameter?

Thanks

[achessel]

On a related point: I tried to save the image as OME.tiff, in case it would change something (it does not).
But I also tried to save it as OME.btf (ome big tiff?), which is not needed for that one plane but would be for the whole 80Gb stack, and import failed at pyramid generation, with the following error in PixelData log:

Code: Select all

2016-01-15 15:34:57,717 INFO  [          loci.formats.in.BaseTiffReader] (2-thread-4) Populating OME metadata
2016-01-15 15:34:58,032 ERROR [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Failed to handle pixels 31051
java.lang.NullPointerException: null
   at loci.common.RandomAccessInputStream.getFilePointer(RandomAccessInputStream.java:198) ~[formats-common.jar:5.1.4]
   at loci.formats.tiff.OnDemandLongArray.toArray(OnDemandLongArray.java:79) ~[formats-bsd.jar:5.1.4]
   at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:858) ~[formats-bsd.jar:5.1.4]
   at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:785) ~[formats-bsd.jar:5.1.4]
   at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:314) ~[formats-bsd.jar:5.1.4]
   at loci.formats.ImageReader.openBytes(ImageReader.java:453) ~[formats-api.jar:5.1.4]
   at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156) ~[formats-bsd.jar:5.1.4]
   at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:225) ~[formats-bsd.jar:5.1.4]
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:349) ~[formats-api.jar:5.1.4]
   at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:349) ~[formats-api.jar:5.1.4]
   at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:269) ~[formats-bsd.jar:5.1.4]
   at ome.io.bioformats.BfPixelsWrapper.getTile(BfPixelsWrapper.java:345) ~[romio.jar:na]
   at ome.io.bioformats.BfPixelBuffer.getTile(BfPixelBuffer.java:475) ~[romio.jar:na]
   at ome.io.nio.PixelsService$1.run(PixelsService.java:418) ~[romio.jar:na]
   at ome.io.nio.Utils.forEachTile(Utils.java:88) ~[romio.jar:na]
   at ome.io.nio.Utils.forEachTile(Utils.java:39) ~[romio.jar:na]
   at ome.io.nio.PixelsService.performWrite(PixelsService.java:402) ~[romio.jar:na]
   at ome.io.nio.PixelsService.makePyramid(PixelsService.java:306) ~[romio.jar:na]
   at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:146) [server.jar:na]
   at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:113) [server.jar:na]
   at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) [server.jar:na]
   at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) ~[na:na]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
   at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:566) [server.jar:na]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) [server.jar:na]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) [org.springframework.orm.jar:3.0.1.RELEASE-A]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) [org.springframework.transaction.jar:3.0.1.RELEASE-A]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) [server.jar:na]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) [server.jar:na]
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [org.springframework.aop.jar:3.0.1.RELEASE-A]
   at com.sun.proxy.$Proxy73.doWork(Unknown Source) [na:na]
   at ome.services.util.Executor$Impl.execute(Executor.java:447) [server.jar:na]
   at ome.services.util.Executor$Impl.execute(Executor.java:391) [server.jar:na]
   at ome.services.pixeldata.PixelDataThread.go(PixelDataThread.java:255) [server.jar:na]
   at ome.services.pixeldata.PixelDataThread.access$000(PixelDataThread.java:52) [server.jar:na]
   at ome.services.pixeldata.PixelDataThread$1.call(PixelDataThread.java:203) [server.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]


AFAIK it's a standard ome.btf saved in fiji, but I can upload the image if that helps

[mlinkert]

I had those messages, and increased the Ice.MessageSizeMax, and stoped having them (the master.err log file is not updated when that error occurs). Just in case I increased it again to above 500mb, which is more than the size of the whole image (is there a way to check that it was taken into account? I could not see a way to get the CLI to spit it the actual value of that parameter out), and the error is still occurring, with no master.err log...

Did it solve the issue on your end to increase that parameter?

You're right; increasing to 256MB locally resolved the stack trace, but not the actual error. I would suspect that there is still a memory related issue due to the size of the image, but we're still investigating.

On a related point: I tried to save the image as OME.tiff, in case it would change something (it does not).
But I also tried to save it as OME.btf (ome big tiff?), which is not needed for that one plane but would be for the whole 80Gb stack, and import failed at pyramid generation, with the following error in PixelData log:

I can reproduce that as well; it seems to be specific to BigTIFF (*.btf) files. We are testing a fix now:

https://github.com/openmicroscopy/bioformats/pull/2196

[achessel]

Thanks for the info and looking into it.

(I just tried with a crop and a 5.8kx2.8k, 97MB crop seem to work fine...)