PlaneID support for Columbus imports

[dgault]

This looks like a version mismatch with the other Bio-Formats jars, you will have to keep the core Bio-Formats jars to at least the same minor version. In this case if you have a 5.9 version of formats-gpl then upgrading formats-api and formats-bsd to a 5.9 version to match should prevent these errors.

[etienne.dumoulin]

Hi David,

The link you sent previously are for bio-formats 5.10 jars today. I don't think you keep previous build online on Jenkins. Do you have them somewhere else?

[dgault]

Sure, you can retrieve the 5.9 components at https://web-proxy.openmicroscopy.org/we ... omponents/

Notably the 3 core components are the ones you will need:
formats-api
formats-bsd
formats-gpl

[frankgu968]

Hi David,

I am currently working with Etienne and Oren (from the previous posts in this thread). I have tried swapping the library versions to the newer 5.9.2 and 6.0.0 but still can't seem to have the PlaneID of the Columbus data imported. The imports are still single z-plane. I noticed similarity between the Columbus data and the PE Operetta format. After some modification to the xml file, I was able to make the Operetta reader ingest the columbus data and correctly import the filesets. Unfortunately, depending on the version of Bio Formats (5.9.2 or 6.0.0), I get some funky errors in from the ImportLibrary saying that there is a "checksum mismatch".

What would you suggest I try to get the z-plane imported from a columbus dataset?

Thanks,
Frank Gu

[dgault]

Hi Frank,

Do you have the stack trace for the error you are seeing with Bio-Formats 6.0.0?

[frankgu968]

Hi David,

With the 5.9.3 library:

Code: Select all

2019-02-28 02:52:43,704 1707658    [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import: A checksum mismatch has occurred.
omero.ChecksumValidationException: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_191]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_191]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_191]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_191]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_191]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.grid.ImportProcessPrxHelper.end_verifyUpload(ImportProcessPrxHelper.java:1616) ~[blitz.jar:na]
   at omero.grid.ImportProcessPrxHelper.verifyUpload(ImportProcessPrxHelper.java:1495) ~[blitz.jar:na]
   at omero.grid.ImportProcessPrxHelper.verifyUpload(ImportProcessPrxHelper.java:1482) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:607) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:306) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:286) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-02-28 02:52:43,704 1707658    [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Exiting on error

With the 6.0.0 library:

Code: Select all

2019-02-28 16:27:50,596 168803     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2019-02-28 16:27:50,612 168819     [2-thread-1] DEBUG   ome.services.blitz.util.CurrentPlatform - recognized current operating system as being Linux
2019-02-28 16:27:56,743 174950     [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import: could not insert: [ome.model.core.OriginalFile]; SQL [insert into originalfile (atime, ctime, creation_id, external_id, group_id, owner_id, permissions, update_id, hash, hasher, mimetype, mtime, name, path, repo, "size", version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [originalfile_repo_path_index]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.core.OriginalFile]
omero.ValidationException: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_191]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_191]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_191]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_191]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_191]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.end_importFileset(ManagedRepositoryPrxHelper.java:175) ~[blitz.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:46) ~[blitz.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:33) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.createImport(ImportLibrary.java:420) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:563) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:306) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:286) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-02-28 16:27:56,743 174950     [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Exiting on error

Hope this helps!

Thanks,
Frank Gu

[dgault]

Thanks Frank,

Would it also be possible to upload the original unmodified Columbus dataset to https://www.openmicroscopy.org/qa2/qa/upload/

We received a partial dataset last time but having the full set would allow us to hopefully put in place a long term fix.

[frankgu968]

Hi David,

The dataset is 115G. Can the upload portal support this?

Frank

[mlinkert]

Hi Frank,

Datasets that large cannot be uploaded to the QA page, but we have an FTP server that can be used instead. I have sent instructions for uploading via FTP in a private message.

-Melissa