OMERO and data from Perkin Elmer Opera Phenix

[frankgu968]

Hi Josh,

Just got back to working on the dataset again. The dataset in question is a 3D screen in the PE Operetta format I believe. The dataset itself is quite large, what subset would be useful to you? I'm guessing the metadata + several image data? Also, where shall I upload the sample data?

As an update, I've also observed some odd messages when the "--debug ALL" flag is set. Immediately upon calling the import, I see the following repeated log entires:

2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - Location(/some/location/Unnamed.xml)
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - Location is not a URL
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - getParent()
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - getName()
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - Location(/some/location/Unnamed.htm)
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - Location is not a URL
2019-02-19 18:34:42,426 842 [ main] TRACE loci.common.Location - exists()
......
2019-02-19 18:34:43,569 1985 [ main] TRACE loci.common.Location - Location(/some/location/r02c08f01p30-ch3sk1fk1fl1.tiff)
2019-02-19 18:34:43,569 1985 [ main] TRACE loci.common.Location - Location is not a URL
2019-02-19 18:34:43,569 1985 [ main] TRACE loci.common.Location - getAbsolutePath()

This is happening during the scanning phase; could it be that the scanner is not finding the proper files?

Thanks,
Frank

[mtbc]

Dear Frank,

How large is the full dataset? It could be that it'd still fit comfortably into what we're happy to provide FTP credentials for.

Cheers,
Mark

[frankgu968]

Hi Mark,

The smallest dataset is in the 100GB range. I"m playing around with some other options such as swapping in the newer BioFormats 6.0.0 library. Printing DEBUG, it appears to correctly recognize the Operetta format, but perhaps have trouble with figuring out the multiplane? (see logs below)

Code: Select all

2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 77
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 78
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 79
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 80
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 81
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 82
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 83
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 84
2019-02-20 19:13:34,959 91003      [      main] DEBUG                loci.formats.FormatHandler - skipping null plane series = 237, plane = 85


Thanks,
Frank

[dgault]

Hi Frank,

If you are able to send the fileset for us to test we would likely only need the metadata files, the directory listing and the pixel files for a single well. If that is under 2GB you can upload it to https://www.openmicroscopy.org/qa2/qa/upload/, otherwise send me a private message with a name and email address which we can use to arrange ftp details to be sent for the transfer.

David Gault

[frankgu968]

Hi David,

I've sent you a message. I'm continuing my tests and I've also made a subset of the data at 2.6G that contains 2 fields and 81 planes for 1 well. This imports correctly with Bio-Formats 6.0.0 and I can see the data in OMERO. Currently trying to import the single well (10 fields, 3 ch, 81 planes) subset. When I tried to import the whole plate (300G of data), the importation fails with an IceUnknownError. I have a feeling it may be that something ran out of resources or timed out? My connection time is currently set to 1 hour. Perhaps the "scanning" phase of the import doesn't initiate the keep-alive ping?

I also noticed that the metadata file that we have is missing several fields and planes for the plate. Does OMERO support variable number of fields per plate? Also, how does the importer handle situations where the tiff exists but is not referenced by any metadata files?

Best,
Frank

[frankgu968]

Hi all,

I managed to get the data imported with the following configuration:

54G of provisioned RAM
32 CPU cores (parallel fileset and upload 64)
OMERO.server 5.4.9 w/ BioFormats 6.0.0

The fileset import is not very resources intensive, but the metadata processing at the end of the import takes quite a bit of RAM (>30G). The whole process for importing 77 wells (3 ch, 11 fields, 81 planes) took about 34 hours in total for reference.

Best,
Frank

[sbesson]

Hi Frank,

very good news. Thanks for updating this thread.

Although your plate is 1K images, 34 hours is quite on high end of the spectrum. Depending on how fast vs complete/rich you expect your plate to be post-import, there should be ways to reduce import time. For High Content Screening data, typically those are:

• using in-place import and skipping the checksums
• using the parallel upload option introduced in OMERO 5.4.8 to upload all the files of fileset
• skipping the calculation of the minimum/maximum pixels values for each channel of each image
• skipping the thumbnail generation at import time. The thumbnail generation can be triggered asynchronously post-import using the omero-cli-render plugin.

Examples of import configurations combining these options can be found in the IDR metadata repositories, see for instance this configuration file used for a similar dataset.

In your case, it might be interesting to know where most of the time is spent, you should be able to use the fs importtime subcommand introduced in OMERO 5.4.8 to have a more detailed breakdown:

Code: Select all

bin/omero fs importtime <Fileset_ID>


Best,
Sebastien