Fail to Import:

[etienne.dumoulin]

Hi,

I try to import a 5-stack plate from an Incell microscope.
The folder structure seems fine. I have a 96 well plate with 6 fields, 5 stacks and 4 channels per well. I have only one DAPI channel image per field though.

On the terminal I run the following:

Code: Select all

$ nohup omero import --transfer=ln_s --checksum-algorithm=CRC-32 /my_path/PAI032_3D_MF0180_3D_1.xdce -u myuser -s localhost -p 4064 -w mypassword -r 302 &
$ tail /tmp/nohup.out
2018-11-19 21:05:50,848 2706286    [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 446416
2018-11-19 21:05:51,498 2706936    [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /mnt/disk/OMERO/raw_data/PAI032_2/ReviewScan_2018.11.14.14.30.56/PAI032_3D/PAI032_3D_MF0180_3D_1/PAI032_3D_MF0180_3D_1.xdce
Plate:508
Other imported objects:
Fileset:7932
2018-11-19 21:05:51,509 2706947    [l.Client-5] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0

==> Summary
7361 files uploaded, 1 fileset, 1 plate created, 460 images imported, 0 errors in 0:45:03.566
Previous session expired for phenomic on localhost:4064


The number of file uploaded is wrong. It should be 9216. When I check the var/log/Blitz-0.log
Here is the exception that occur at the same time. I have run this import several time and the error is consistent:

Code: Select all

.
.
2018-11-19 21:05:50,818 INFO  [       ome.security.basic.CurrentDetails] (erver-1040) Adding log:UPDATE,class ome.model.core.OriginalFile,446416
2018-11-19 21:05:50,820 INFO  [                 org.perf4j.TimingLogger] (erver-1040) start[1542661550810] time[9] tag[omero.call.success.ome.logic.UpdateImpl.saveObject]
2018-11-19 21:05:50,820 INFO  [        ome.services.util.ServiceHandler] (erver-1040)  Rslt:    null
2018-11-19 21:05:50,820 INFO  [                 org.perf4j.TimingLogger] (2-thread-2) start[1542661352472] time[198348] tag[omero.import.request.step]
2018-11-19 21:05:50,820 INFO  [                 org.perf4j.TimingLogger] (2-thread-2) start[1542661352472] time[198348] tag[omero.request.step.2]
2018-11-19 21:05:50,821 INFO  [ o.s.b.r.ManagedImportRequestI.@1d6de206] (2-thread-2) notify step 3 of 5
2018-11-19 21:05:50,821 INFO  [        ome.services.util.ServiceHandler] (erver-1035)  Executor.doWork -- ome.services.blitz.impl.MetadataStoreI.postProcess[]
2018-11-19 21:05:50,821 INFO  [        ome.services.util.ServiceHandler] (erver-1035)  Args:    [null, InternalSF@1034769247]
2018-11-19 21:05:50,821 INFO  [    ome.security.basic.BasicEventContext] (erver-1035)  cctx:    group=3
2018-11-19 21:05:50,821 INFO  [         ome.security.basic.EventHandler] (erver-1035)  Auth:    user=3,group=3,event=null(User),sess=5c1c9dad-f304-4b48-830c-298996af4a56
2018-11-19 21:05:50,824 INFO  [                 org.perf4j.TimingLogger] (erver-1035) start[1542661550821] time[2] tag[omero.call.exception]
2018-11-19 21:05:50,825 WARN  [        ome.services.util.ServiceHandler] (erver-1035) Unknown exception thrown.

java.lang.IllegalStateException: Already logged in. Use Executor.submit() and .get().
        at ome.services.util.Executor$Impl.execute(Executor.java:437) [server.jar:na]
        at ome.services.util.Executor$Impl.execute(Executor.java:396) [server.jar:na]
        at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) ~[blitz.jar:na]
        at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) ~[blitz.jar:na]
        at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) ~[blitz.jar:na]
        at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) ~[blitz.jar:na]
        at ome.services.blitz.impl.ServiceFactoryI.sharedResources(ServiceFactoryI.java:416) ~[blitz.jar:na]
        at ome.services.blitz.impl.MetadataStoreI$6.doWork(MetadataStoreI.java:301) ~[blitz.jar:na]
        at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:573) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy72.doWork(Unknown Source) [na:na]
        at ome.services.util.Executor$Impl.execute(Executor.java:452) [server.jar:na]
        at ome.services.util.Executor$Impl.execute(Executor.java:396) [server.jar:na]
        at ome.services.throttling.Adapter.run(Adapter.java:50) [blitz.jar:na]
        at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89) [blitz.jar:na]
        at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154) [blitz.jar:na]
        at ome.services.blitz.impl.MetadataStoreI.postProcess_async(MetadataStoreI.java:281) [blitz.jar:na]
        at sun.reflect.GeneratedMethodAccessor1960.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop.jar:3.0.1.RELEASE]
        at omero.cmd.CallContext.invoke(CallContext.java:85) [blitz.jar:na]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop.jar:3.0.1.RELEASE]
        at omero.cmd.CallContext.invoke(CallContext.java:85) [blitz.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy115.postProcess_async(Unknown Source) [na:na]
        at omero.api._MetadataStoreTie.postProcess_async(_MetadataStoreTie.java:78) [blitz.jar:na]
        at omero.api._MetadataStoreDisp.___postProcess(_MetadataStoreDisp.java:328) [blitz.jar:na]
        at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:412) [blitz.jar:na]
        at IceInternal.Incoming.invoke(Incoming.java:221) [ice.jar:na]
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) [ice.jar:na]
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145) [ice.jar:na]
        at Ice.ConnectionI.message(ConnectionI.java:1056) [ice.jar:na]
        at IceInternal.ThreadPool.run(ThreadPool.java:395) [ice.jar:na]
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na]
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) [ice.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2018-11-19 21:05:50,825 INFO  [        ome.services.util.ServiceHandler] (erver-1035)  Excp:    java.lang.IllegalStateException: Already logged in. Use Executor.submit() and .get().
.
.


Did anybody run into a similar issue?

Thanks for your time.

Regards,

Etienne

[jmoore]

Hi Etienne,

I don't think the server-side error is particularly related. If you try

Code: Select all

omero import -f /my_path/PAI032_3D_MF0180_3D_1.xdce


I assume you will also see only the 7361 files listed without the need to import. If that's the case, we'll need to look into what's going wrong with this dataset. (I assume none of the files have been renamed or similar?)

Cheers,
~Josh.

[etienne.dumoulin]

Hi Josh,

Thank you for your reply.

I have been playing with the -f option and, as you guessed, the list is not complete. I have done many tests but I couldn't make it work.

I think the problem is in the xml file. I have reproduced the issue with the following set up:
1- download the attached tar
2- untar it and go into the folder
3- run the script

Code: Select all

python file_creator.py

, the script creates 9216 symlinks from one image in the current folder
4- run

Code: Select all

omero import -f --debug DEBUG PAI032_3D_MF0180_3D_1.xdce

I cannot attach the tar file due to the attachment size restriction (it is 5.5Mb), let me know how I can send it to you.

I would take any suggestions at this stage. Thanks for your time.

Regards,

Etienne

[jmoore]

For MB sized files, the instructions under https://docs.openmicroscopy.org/bio-formats/5.9.2/about/bug-reporting.html#sending-a-bug-report should work fine. Alternatively, if you have the files available somewhere online, we're happy to just download them. If none of the above work, we'll send FTP instructions.

~Josh

[etienne.dumoulin]

Hi Josh,

I have uploaded the file in the QA system, its id is 27112. It should be a tar file with the name `troubleshooting_3D.tar.gz`.

Regards,

Etienne

[mlinkert]

Hi Etienne,

Thank you for uploading a test dataset. I can reproduce the same problem; the number of detected files, wells, and fields are all incorrect.

The issue has to do with the single DAPI image confusing the plate dimension arithmetic. I have opened a card on our issue tracker:

https://trello.com/c/jSvtOrBD/297-incel ... ng-z-sizes

We will update that card as we make progress on a fix.

-Melissa

[etienne.dumoulin]

Hi Melissa,

Thank you so much for the quick feedback.
For the time being, could I image again with a 5 stack DAPI channel?

I tried to modify the XML earlier to repeat the DAPI channel image per stack but I couldn't get the format right.

Regards,

Etienne

[jmoore]

Etienne,

from what I understand of the fix, I do think keeping consistent dimensions would work around the issue you've been seeing.

Cheers,
~Josh