new Olympus Scan^R filesets error on import

[dsudar]

Hi all,

We just upgraded our Olympus Scan^R to software version 2.6.1 and file sets created with that version give an exception when importing into OMERO (v. 5.2.4). Below the command line output when importing a 96-well plate, 4 fields/well, 4 channels:

sudard@omero:/data/share/sudard$ omero import -r 1052 -n LINCS_BMPval_plate1 LINCS_BMPval_plate1_001
Previous session expired for dsudar on localhost:4064
Server: [localhost:4064]
Username: [dsudar]
Password:
Created session c43fb537-013e-4626-b99d-24bad5975d34 (dsudar@localhost:4064). Idle timeout: 10 min. Current group: MEP-LINCS
2016-07-19 16:58:13,565 323 [ main] INFO ome.formats.importer.ImportConfig - OMERO Version: 5.2.4-ice35-b23
2016-07-19 16:58:13,584 342 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 5.1.9 revision: c3 2016
2016-07-19 16:58:13,642 400 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO
2016-07-19 16:58:14,082 840 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2016-07-19 16:58:20,839 7597 [ main] INFO ome.formats.importer.ImportCandidates - 1284 file(s) parsed into 2 group(s) wiknowns]
2016-07-19 16:58:21,826 8584 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to l
2016-07-19 16:58:22,430 9188 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.2.4
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.2.4-ice35-b23
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_101
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2016-07-19 16:58:22,745 9503 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.13.0-88-generic
2016-07-19 16:58:23,426 10184 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Screen:1052
2016-07-19 16:58:23,444 10202 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2016-07-19 16:58:23,741 10499 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2016-07-19 16:58:23,753 10511 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/sudard/LINCS_
2016-07-19 16:58:23,771 10529 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/sudar
2016-07-19 16:58:23,794 10552 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/suda
2016-07-19 16:58:23,803 10561 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/sudard/LINCS_
2016-07-19 16:58:23,818 10576 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/sudar
2016-07-19 16:58:23,835 10593 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/suda
2016-07-19 16:58:23,843 10601 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/sudard/LINCS_
2016-07-19 16:58:23,858 10616 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/sudar
2016-07-19 16:58:23,873 10631 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/suda
2016-07-19 16:58:23,924 10682 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2016-07-19 16:58:24,009 10767 [ main] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1986801
2016-07-19 16:58:24,056 10814 [l.Client-0] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /data/share/sudard
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {message=, stacktrace=java.lang.NullPointerException
at loci.common.Location.getHandle(Location.java:306)
at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:131)
at loci.formats.in.MinimalTiffReader.initFile(MinimalTiffReader.java:432)
at loci.formats.FormatReader.setId(FormatReader.java:1426)
at loci.formats.in.ScanrReader.initFile(ScanrReader.java:548)
at loci.formats.FormatReader.setId(FormatReader.java:1426)
at loci.formats.ImageReader.setId(ImageReader.java:835)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:289)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.Memoizer.setId(Memoizer.java:678)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at ome.services.blitz.repo.CheckedPath.bfSetId(CheckedPath.java:404)
at ome.services.blitz.repo.ManagedImportRequestI.open(ManagedImportRequestI.java:731)
at ome.services.blitz.repo.ManagedImportRequestI.init(ManagedImportRequestI.java:229)
at omero.cmd.HandleI.steps(HandleI.java:428)
at omero.cmd.HandleI$1.doWork(HandleI.java:366)
at omero.cmd.HandleI$1.doWork(HandleI.java:362)
at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy70.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:443)
at omero.cmd.HandleI.run(HandleI.java:360)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at ome.services.util.Executor$Impl$1.call(Executor.java:484)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:666)
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
at IceInternal.Incoming.invoke(Incoming.java:222)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
at Ice.ConnectionI.message(ConnectionI.java:1213)
at IceInternal.ThreadPool.run(ThreadPool.java:321)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
at java.lang.Thread.run(Thread.java:745)

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {message=, stacktrace=java.lang.NullPointerException
at loci.common.Location.getHandle(Location.java:306)
at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:131)
at loci.formats.in.MinimalTiffReader.initFile(MinimalTiffReader.java:432)
at loci.formats.FormatReader.setId(FormatReader.java:1426)
at loci.formats.in.ScanrReader.initFile(ScanrReader.java:548)
at loci.formats.FormatReader.setId(FormatReader.java:1426)
at loci.formats.ImageReader.setId(ImageReader.java:835)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:289)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at loci.formats.Memoizer.setId(Memoizer.java:678)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
at ome.services.blitz.repo.CheckedPath.bfSetId(CheckedPath.java:404)
at ome.services.blitz.repo.ManagedImportRequestI.open(ManagedImportRequestI.java:731)
at ome.services.blitz.repo.ManagedImportRequestI.init(ManagedImportRequestI.java:229)
at omero.cmd.HandleI.steps(HandleI.java:428)
at omero.cmd.HandleI$1.doWork(HandleI.java:366)
at omero.cmd.HandleI$1.doWork(HandleI.java:362)
at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy70.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:443)
at omero.cmd.HandleI.run(HandleI.java:360)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at ome.services.util.Executor$Impl$1.call(Executor.java:484)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:666) ~[blitz.jar:na]
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334) [blitz.jar:na]
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118) [blitz.jar:na]
at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145) [blitz.jar:na]
at IceInternal.Incoming.invoke(Incoming.java:222) [ice.jar:na]
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482) [ice.jar:na]
at Ice.ConnectionI.dispatch(ConnectionI.java:1258) [ice.jar:na]
at Ice.ConnectionI.message(ConnectionI.java:1213) [ice.jar:na]
at IceInternal.ThreadPool.run(ThreadPool.java:321) [ice.jar:na]
at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na]
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693) [ice.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
2016-07-19 16:58:24,062 10820 [ main] ERROR ome.formats.importer.ImportLibrary - Error on import
java.lang.Exception: Import failure
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:561) ~[blitz.jar:na]
at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:290) ~[blitz.jar:na]
at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:266) [blitz.jar:na]
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:886) [blitz.jar:na]
2016-07-19 16:58:24,062 10820 [ main] INFO ome.formats.importer.ImportLibrary - Exiting on error

==> Summary
3 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:01.196

I tried to load the same file set with Fiji and that worked fine (Fiji has BioFormats 5.1.10) so was this maybe fixed 5.1.10? I uploaded just the experiment_descriptor.xml file to the QA system.

One interesting thing: I noticed that the new version of Scan^R software creates a file called "metadata.ome.xml" in the data subdirectory. Should that be picked up by the importer?

Thanks,
- Damir

[dgault]

Hi Damir,

An exception like that quite often indicates that it has been unable to locate or parse the name of one of the associated tiff files.

Would you be able to send on the output of the following directory listing commands to help us understand what could be going wrong:
ls -l
ls -l data/

David Gault

[dsudar]

Hi David,
Thanks!! I uploaded the listing (listing.txt) of the "experiment_descriptor" directory and the "data" directory to the QA site.
Cheers,
- Damir

[dsudar]

Quick follow-up: I forgot to mention that column 1 and 12 were not scanned since they were empty. So of the 96-well plate only wells A2 - H11 were scanned.
- Damir

[dgault]

Hi Damir,

I have been carrying out some more testing today on this issue and have so far been unable to reproduce the same behaviour. After comparing the directory listing you provided with the file list the reader expects when reading the xml, I did notice that there were a few files missing but these should simply have been ignored and not caused the error shown. In fact the exception appears to be caused due to there being no suitable files found at all.

I have attempted to mock up the fileset based on the directory listing but it appeared to read the directory structure as expected. The fact that it also uploads correctly through Fiji would suggest that the issue is not with reading the files but rather that it has for some reason been unable to locate the files in the data directory in this instance. There shouldn't be any difference for the ScanRReader between the 2 versions (5.1.9 and 5.1.10).

When searching for the associated tiff files the reader first looks for a data directory and if it cannot find it then searches the top level directory (where the xml and dat files are located). If the tiff files are moved up to this directory do you see the same error occur?

As an aside, with regards your previous question about the metadata.ome.xml that is generated, any additional xml or dat files in the directory will get picked up and added to the list of metadata files.

[dsudar]

Hi David,

Thanks for the detective work. It still appears to be a mystery. I've uploaded a smallish full fileset to the QA system called Scanr.tar.gz
It has 80 wells scanned from a 96-well plate, 4 fields/well, 4 channels. This fileset failed to import with that exact error that I reported. A similar fileset from an earlier version (2.5.1) of the ScanR software imports fine and I tried to look through the experiment_descrtiptor.xml file for differences. While there are differences, nothing stands out as a likely culprit.

I did try to move the .tifs from the data subdirectory into the main directory where the experiment_descriptor.xml file is located but that made no difference. Same exact error.

Thanks for any insights.
Cheers,
- Damir

[dgault]

Thanks for the upload Damir, I have tested the dataset you provided and I can certainly reproduce the same error you are seeing. I will be carrying out further testing this afternoon to try and root cause this issue and I will keep you updated once I have a better idea of the problem.

David

[dsudar]

Thanks David. I very much appreciate you looking at this and let me know if I can provide more info.
- Damir

[dgault]

Hi Damir,

I just wanted to follow up to let you know that as I have not found a solution yet I have opened a bug ticket to track this issue - https://trac.openmicroscopy.org/ome/ticket/13271#ticket

David

[dsudar]

Hi David,

Again thanks for looking into this. Since it is an important issue for me, I'd like to look a bit more in-depth at it. From what you've seen, do you have an idea where I should focus? It appears that it has nothing to do with the .tif files but instead that it's something in the .xdce file. Any pointers would be appreciated.

Thanks,
- Damir