OMERO and data from Perkin Elmer Opera Phenix

[klh]

We have a Perkin Elmer Opera Phenix instrument, we are evaluating OMERO as a possible solution for data storage/management for this instrument. Our Opera Phenix instrument through its Harmony software is able to export the data as tiff files (several thousands in one single directory) and all the metadata inside one single Index.idx.xml file. I managed to upload a subset of our tiff images and the Index.idx.xml file into OMERO Demo Server, at upload time I selected "File of Type: PerkinElmer Operetta (*.tif, *.tiff, *.xml) in the OMERO.insight importer window. After import, the data shows up fine in OMERO and we are seeing the multiple fields (4) & channels(2, wave lenghts).

What we are uncertain about and need help to answer: When importing our Opera tiffs + Index.idx.xml as "PerkinElmer Operetta" in OMERO, are we missing some important metadata? Will all the metadata from the Opera formatted Index.idx.xml be available in OMERO? Would someone be able to inform us if there is any significant difference between Operetta vs. Opera exported data?

Thanks,
Kia

[sbesson]

Hi Kia,

Bio-Formats does not have direct support for the TIFF-based format generated by the Harmony software - see the list of supported formats.

Given the structure of the fileset, our assumption is that the PerkinElmer Operetta reader will correctly detect all of the files in the dataset. There might be variations in the metadata but answering this question precisely is complex in the absence of format specification and/or representative datasets

Would you have one that could be shared with us to be able to investigate this question? Datasets smaller than 2GB can be uploaded it directly at:

http://qa.openmicroscopy.org.uk/qa/feedback/

Best,
Sebastien

[klh]

Hi Sebastien,
Thank you for your response. Yes, I do have a couple of Perkin Elmer Opera Harmony exported data sets that I can share with you, but their sizes are around 60 GB (full data set). If you want you can contact me privately and we can figure out how to transfer these 60 GB to you some how.
If you think looking at the tiff files for a specific row-column-field (channel 1 & 2) and the full Index.idx.xml file would be helpful I can probably see if I can get this subset down to 2GB and upload it to http://qa.openmicroscopy.org.uk/qa/upload/
Let me know if you need full data set, or if you can do with just a specific row-column-field subset.

Thanks
Kia

[sbesson]

HI Kia,

a valuable first step might be to upload only the metadata file, i.e. Index.idx.xml and a listing of all the TIFF files in the directory to http://qa.openmicroscopy.org.uk/qa/upload/.

Best,
Sebastien

[klh]

Sebastien,
I uploaded this file: Kia-PE_Opera_Phenix_sample_list.tar.gz
to http://qa.openmicroscopy.org.uk/qa/upload/

Please let me know if you need additional information.
Thank you,
Kia

[sbesson]

Thanks Kia,

we received the archive, will take a look at it and get back to you on this thread if there are any questions.

Best,
Sebastien

[rleigh]

I have done some testing and investigation of the file format. Overall, it seems to be a new variant of the existing Operetta file format, and as such the existing OperettaReader should read these files correctly. All of the metadata seems to be structured in exactly the same way as the existing files we have in this format.

However, your example file metadata contains additional metadata not present in our existing samples, which may be new additions to the format. Currently these properties are unsupported by the OperettaReader and will be ignored. It should be possible to support several of these where they map onto existing OME model properties. https://trello.com/c/WrSJs57r/506-impro ... y-metadata contains some of the details from the investigation and possible improvements to the Operetta reader. You can certainly read your file with Bio-Formats and OMERO right now, with the caveat that these properties will be missing.

Regards,
Roger

[frankgu968]

Hi all,

We are trying to import a 3D image set into OMERO 5.4.9. The imaging source is a PE Phenix, and is a 3 channel, 81 plane image stack. To import the images, we followed OMERO's importation guide. We created a screen (ID=1) and used the following command for a parallel in-place import:

Code: Select all

bin/omero import /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11 \
--transfer=ln_s \
--parallel-fileset 16 --parallel-upload 32 \
--skip all \
-r 1

OMERO was able to correctly identify the files for import, and the log shows the importer traversing and importing the all the files:

Code: Select all

2018-12-11 20:04:01,254 14646200   [-thread-23] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p45-ch1sk1fk1fl1.tiff...
2018-12-11 20:04:01,259 14646205   [-thread-16] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p45-ch2sk1fk1fl1.tiff...
2018-12-11 20:04:01,261 14646207   [3-thread-4] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p45-ch3sk1fk1fl1.tiff...
2018-12-11 20:04:01,261 14646207   [-thread-32] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p33-ch3sk1fk1fl1.tiff
2018-12-11 20:04:01,270 14646216   [-thread-20] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p46-ch1sk1fk1fl1.tiff...
2018-12-11 20:04:01,271 14646217   [-thread-13] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p32-ch1sk1fk1fl1.tiff
2018-12-11 20:04:01,274 14646220   [-thread-17] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p46-ch2sk1fk1fl1.tiff...
2018-12-11 20:04:01,278 14646224   [-thread-18] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p29-ch3sk1fk1fl1.tiff
2018-12-11 20:04:01,285 14646231   [-thread-13] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c11f09p46-ch3sk1fk1fl1.tiff...
2018-12-11 20:04:01,286 14646232   [-thread-15] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /mnt/disk/OMERO/raw_data/


However, after the import, nothing shows up on OMERO. There are no orphaned images, and we can't seem to find the data under the created screen. Is there something that we are doing incorrectly?

Thanks,
Frank

[jmoore]

Can you also share the lines on the console after

Code: Select all

2018-12-11 20:04:01,286 14646232   [-thread-15] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /mnt/disk/OMERO/raw_data/


(especially the last few lines)? What does `bin/omero fs sets` show?

Cheers,
~Josh

[frankgu968]

Hi Josh,

Thanks for your prompt response!

Here is the abridged version of the full console output:

Code: Select all

2018-12-11 15:59:55,247 193        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.9-ice36-b101
2018-12-11 15:59:55,261 207        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.9.2 revision: 9fc607f85b8900be786813296f1eee75cc1ed883 date: 31 August 2018
2018-12-11 15:59:55,293 239        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting parallel fileset: 16
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting parallel upload: 32
2018-12-11 15:59:55,294 240        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2018-12-11 15:59:55,295 241        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-12-11 15:59:55,622 568        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-12-11 17:58:21,442 7106388    [      main] INFO      ome.formats.importer.ImportCandidates - 57351 file(s) parsed into 1 group(s) with 246 call(s) to setId in 1786722ms. (7105820ms total) [2 unknowns]
2018-12-11 17:58:21,498 7106444    [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2018-12-11 17:58:21,867 7106813    [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.9
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.9-ice36-b101
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_191
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2018-12-11 17:58:22,178 7107124    [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.4.0-1072-aws
2018-12-11 17:58:42,621 7127567    [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Screen:1
2018-12-11 17:58:42,635 7127581    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2018-12-11 18:25:14,612 8719558    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2018-12-11 18:25:14,629 8719575    [3-thread-4] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c08f01p01-ch3sk1fk1fl1.tiff...
2018-12-11 18:25:14,630 8719576    [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/Index.idx.xml...
2018-12-11 18:25:14,630 8719576    [3-thread-2] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c08f01p01-ch1sk1fk1fl1.tiff...
2018-12-11 18:25:14,630 8719576    [3-thread-8] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c08f01p03-ch1sk1fk1fl1.tiff...
2018-12-11 18:25:14,630 8719576    [-thread-10] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c08f01p03-ch3sk1fk1fl1.tiff...
2018-12-11 18:25:14,630 8719576    [3-thread-6] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r02c08f01p02-ch2sk1fk1fl1.tiff...


There isn't any further output from the log file after the FILE_UPLOAD_COMPLETE. The log is very long, but only shows the transfer starting, and completing; nothing else...

Below is the output of

Code: Select all

omero fs sets

Code: Select all

# | Id  | Prefix                              | Images | Files  | Transfer
---+-----+-------------------------------------+--------+--------+----------
0 | 401 | phenomic_3/2018-12/11/17-58-43.774/ | 0      | 57106  | ln_s
1 | 351 | phenomic_3/2018-12/10/07-23-47.127/ | 0      | 145072 | ln_s
2 | 251 | phenomic_3/2018-12/09/09-27-02.972/ | 0      | 145072 | ln_s
3 | 201 | phenomic_3/2018-12/09/06-08-41.170/ | 0      | 145072 | ln_s
4 | 153 | phenomic_3/2018-12/08/00-43-00.814/ | 0      | 145072 | ln_s
5 | 152 | phenomic_3/2018-12/08/00-28-31.924/ | 0      | 145072 | ln_s
6 | 151 | phenomic_3/2018-12/08/00-27-38.279/ | 0      | 145072 | ln_s
7 | 102 | phenomic_3/2018-12/06/22-15-46.936/ | 0      | 145072 | ln_s
8 | 101 | phenomic_3/2018-12/06/22-12-50.320/ | 0      | 145072 | ln_s
(9 rows, starting at 0 of approx. 9)

Regards,
Frank