importing OME.tiffs in Omero 5

[yuriy_alexandrov]

Hi All,

This is in continuation to the OME.tiff managing activity - remind, I create them using OmeroMatlab bfsave-type methods, where the "Description" field is used to store additional metadata (encoded in UUIDFileNames) - and that was tested in Omero 4.4.10.

After updating to Omero 5 today, I found that when importing these files by 5-th version of Insight,
1) the description isn't set up and
2) for the larger file (360 planes, 1.44M), import took more than 40 minutes (that was found occasionally!).

There were no such problems in 4.4.10, - the description field was set up properly and importing the same large file took only 70 seconds.

I wonder if that is some sort of real Omero 5 glitch, or maybe I'm doing something wrong (maybe Bioformats version?).

Thanks,
Y.

[wmoore]

I don't have a solution, but this seems very similar to this bug: https://trac.openmicroscopy.org.uk/ome/ticket/11850
Although that was for OME-tiffs generated from OMERO export (and was fixed by OMERO 5.0.0), so I'm not sure if it's the same issue.

Could you send us a sample file - upload at
http://qa.openmicroscopy.org.uk/qa/upload/
and we can take a closer look.

Thanks,

Will.

[yuriy_alexandrov]

Many thanks Will for fast reply,

I believe I sent the 1.44M file (at least got an automatic confirmation).

The proc that saves these OME.tiffs is here:
https://github.com/openmicroscopy/Imper ... _OMEtiff.m

Best,
Y.

[wmoore]

Hi,

Thanks for sending the files and apologies for the delay.

I tried importing them and saw similar results as you. No description for either image and much slower import than expected for images of this size.

Command line import took about 8 minutes for the 1.4M OME.tiff:

Code: Select all

...
2014-03-07 10:12:27,883 32065      [l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0
2014-03-07 10:12:27,887 32069      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2014-03-07 10:12:27,898 32080      [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.1.0-demo-nottingham-DEV-ice35
2014-03-07 10:12:27,898 32080      [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.1.0-demo-nottingham-DEV-ice35 revision: faf44e1 date: 10 February 2014
2014-03-07 10:12:28,384 32566      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2014-03-07 10:12:28,387 32569      [      main] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff...
2014-03-07 10:12:28,436 32618      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff
2014-03-07 10:12:38,038 42220      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff
2014-03-07 10:12:39,501 43683      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2014-03-07 10:12:42,076 46258      [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 4207
2014-03-07 10:16:29,067 273249     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 4207
2014-03-07 10:20:13,191 497373     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 4207
2014-03-07 10:20:13,235 497417     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 4207
2014-03-07 10:20:13,266 497448     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 4207
2014-03-07 10:20:13,401 497583     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff



Inspecting the log via

Code: Select all

$ bin/omero download 4207 -


Showed a very large number of these lines:

Code: Select all

2014-03-07 10:13:00,424 DEBUG [    o.s.blitz.repo.ManagedImportRequestI] (3-thread-5) Plane:26 X:0 Y:37 TileWidth:430 TileHeight:1 arrayBuf.length:860


which I think is basically the same problem as the bug I mentioned above. The 'tiles' of pixels being read a quite small, so there's a very large number of tiles to read.
I'll re-open the ticket, but Melissa might re-assign if this is a different issue.


When I open the fluor.OME.tif in Fiji, the OMX-XML shows the Description as having an entire OME-XML block in it:

Code: Select all

<Description><?xml version="1.0" encoding="utf-8"?><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2013-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2013-06 http://www.openmicroscopy.org/Schemas/OME/2013-06/ome.xsd"><Image ID="Image:0"><Pixels BigEndian="true" DimensionOrder="XYZCT" ..... </Pixels></Image></OME></Description>


Is this what you expect?

Cheers,

Will.

[yuriy_alexandrov]

Hi Will,

Yes the Image Description contains whole XML block as I didn't find other way to retrieve metadata I need.
Please note that UUID FIlenames that are in that Description block (correct ones) are not reproduced by XML generated in the course of Insight import.

Best,
Y.

[wmoore]

I have created a ticket to address the loss of XML from OME.tiff description on import: https://trac.openmicroscopy.org.uk/ome/ticket/12056

If we can help with any work-around in the meantime, let us know.

Regards,

Will.

[wmoore]

Hi Yuriy,

A fix for the slow import of your ome-tiffs has just been merged. They seemed to import within a few secs when I tested it - see https://github.com/openmicroscopy/bioformats/pull/1009.

[yuriy_alexandrov]

Hi again,

Question is (likely to Bioformats team) about possible changes.
I updated Bioformats to the new version, - that needed replacing "loci_tools.jar" by the "bioformats_package.jar" archive.

When tried to run the OME-tiff-saving code, I've got an error on the line

Code: Select all

img_description = char(loci.formats.MetadataTools.getOMEXML(metadata));                                                     

the error message suggests that the function getOMEXML is missing

log4j:WARN No appenders could be found for logger (loci.common.services.ServiceFactory).
log4j:WARN Please initialize the log4j system properly.
The class loci.formats.MetadataTools has no property or method named 'getOMEXML'.

Also, the function "bfsave_with_description.m" is missing from the package, but I applied it before and found very useful.

When I switch back to the old version of Bioformats - everything works.
Well, not counting problems described above, but at least OME.tiffs are created..

Best,
Y.

[jmoore]

Hi Yuriy,

the major change between `loci_tools.jar` and `bioformats_package.jar` was the version of logging that the required (log4j --> logback). Some versions of MATLAB have a hard dependency on log4j. If that's the case, then you can stick with the newer version of the `loci_tools.jar` jar. See http://downloads.openmicroscopy.org/bio-formats/5.0.1/artifacts/loci_tools.jar which is under the Components section.

Cheers,
~Josh.

[yuriy_alexandrov]

Thanks Will, Josh for addressing these issues.

With the "loci_tools.jar" file that Josh suggested, the same error occurs - it looks like the function

loci.formats.MetadataTools.getOMEXML

missed or renamed (or moved to another archive?).

Best,
Y.