OMERO handling of Micromanager produced stacks.

[PhilippeP]

Hi List,
Because of unexpected results in OMERO with stacks produced with MicroManager2, we did a small test mimicking HCS results imported in OMERO.
The test consists, using the MM HCS plugin, in acquiring 4 x-y positions (1t, 1z) in 6 wells of a 24 well plate, using the 4 possible combinations of options available in MM2 when acquiring images:
- Create metadatas.txt files with image stack files (yes or no),
- Save xy positions in separate image stack files (yes or no).
MM2 generates 1 folder for each of these 4 conditions, containing stack(s) and metadata.txt (or none), as expected from the selected options.

These 4 folders are then imported in OMERO via Insight.
Surprisingly, we always get 24 stacks being displayed in OMERO (while in 2 conditions, a single "big" one with 24 images was generated by MM2). More of a problem to us, the well identifications are never present in OMERO image matadata tab. They are only found within the OMERO generated stack names when "Save xy positions in separate image stack files" was selected in MM2.

All this is summarized here:

MM HCS to OMERO.png (32.83 KiB) Viewed 4128 times

Is that expected?

The main point we would like to know is:
Is there a way to have in OMERO the well identifications when not selecting "Save xy positions in separate image stack files" in MM2?

Thanks for reading!
Philippe

[bramalingam]

Hi Philippe,

Thank you for submitting your issue.

Yes you can read the well identities for the images in OMERO, and we recommend the following measures to achieve the same,

• Please select the "ome-tiff stack" as your master file (instead of the *metadata.txt file) while importing your images into an OMERO server.
• This will enable Bio-formats to use the OMETiffReader to read the file and also facilitate the reading of the Well/site information. You can view the same in the “image metadata” tab in the insight/web client (Screenshot for the same attached)

Screen Shot 2016-08-24 at 13.58.22.png (420.49 KiB) Viewed 4090 times

We have started documenting the ideal work-flows for the import of images acquired via Micro-manager in our documentation page,
http://www.openmicroscopy.org/site/supp ... index.html

and we will try to add another matrix to cover the import permutations of the screen-well-plate workflows as well,at the earliest.

Hope that helps.

Best
Balaji

[PhilippeP]

I am coming back to this topic, since I am still having problems keeping well addresses in stack names after import to OMERO from Micromanager2.
For example:
I made a simple test stack with MicroManager2: 2 channels, 4 positions. I asked MM2 to "create metadata.txt file with image stack files" (not "save x-y positiopns in separate image stack file"), and saved the acquisitions as "image stack file" (not individual images).
When importing to OMERO, no error is seen. But when browsing the result in OMERO:
1- when import uses the metadata file present in the folder to be imported, I get only one image (with 2 channels) but not a stack. i.e, all other imported images are not visible;
2- when importing the stack itself, not via the folder/metadata text file, I do get the four images in OMERO, but there is no way of knowing which image is which (all names are identical, despite the "override default naming" being selected in Insight.
What should I do to get the original image names displayed (like seen in MM2: 1-Pos_001_001, and so on)?

PS: the link in post above is unfortunately dead: http://www.openmicroscopy.org/site/supp ... index.html

PPS: in the screenshot above, all images have the same identical name: stack_1_MMStack.ome.tif[stack_1_MMStack], which is the problem I would like to solve.

[sbesson]

Hi Philippe

thanks for getting back to this. I think there are 2 separate issues: one at the file format level (Bio-Formats) and one at the import level (OMERO).

I made a simple test stack with MicroManager2: 2 channels, 4 positions. I asked MM2 to "create metadata.txt file with image stack files" (not "save x-y positiopns in separate image stack file"), and saved the
acquisitions as "image stack file" (not individual images).
When importing to OMERO, no error is seen. But when browsing the result in OMERO:
1- when import uses the metadata file present in the folder to be imported, I get only one image (with 2 channels) but not a stack. i.e, all other imported images are not visible;

First, note that Bio-Formats does not officially support any of the intermediate formats generated by Micro-Manager 2.0 (beta and/or gamma) although these datasets might be readable to some extent. We are following progress on the Micro-Manager 2 roadmap and awaiting the stabilization of the file format used by Micro-Manager 2 - see also this thread by Mark Tsuchida on the Micro-Manager mailing list.

Are you also seeing this behavior with datasets generated with Micro-Manager 1.4? If so, having representative samples uploaded to http://qa.openmicroscopy.org.uk/qa/upload/ might help us troubleshoot whether this is something we can fix.

2- when importing the stack itself, not via the folder/metadata text file, I do get the four images in OMERO, but there is no way of knowing which image is which (all names are identical, despite the "override default naming" being selected in Insight.
What should I do to get the original image names displayed (like seen in MM2: 1-Pos_001_001, and so on)?

This is a known OMERO limitation that the IDR team also faced when publishing this single cell study with multi-position datasets generated using Micro-Manager 1.4.16. Historically, the OMERO import has been relying on the filename rather than the Image/Plate name metadata read by Bio-Formats - see https://github.com/openmicroscopy/design/issues/57. We have started deviating from this paradigm notably in the High-Content Screening case but this has not been done in the general case of Images in Project/Dataset hierarchy.

At the moment, we do not have a generic solution. Users of the command-line importer can override the image name via an option or image names need to be updated post-import. I will certainly record your encountering of this limitation on the design issue mentioned above.

Best,
Sebastien

[PhilippeP]

Thanks Sebastien for your response.

I did some more tests with MM2 freshly installed (last nightly build), all default settings except when indicated.
6 well plate, 2 positions per well, 2 channels.
I asked in the acquisition interface (MDA) for saving images as "image stack file".
And I selected two different settings in the preferences: either "Save x-y positions in different stack files", or not.
I got 1 directory for each of the 2 tests. I imported the 2 directories to OMERO directly (i.e., selecting each main directory, not the inside stack(s).
What I got is attached (screen shot with directory structures, and OMERO GUI).

The well addresses are present only when asking in MM2 preferences for "Save x-y positions in different stack files". This is annoying, since when doing high-throughput, you do not want thousands of stacks, but only a big one (which is split in 4GB substack when acqhisition is >4GB). This also avoid the OMERO error message when over 2000 (if I remember correctly?) files are being imported.
What is weird is that the test where "Save x-y positions in different stack files" is not selected does give as expected a single directory, which when opened with MM2 displays all well and position information.

It thus seems like there indeed is something amiss when importing that second type of directory into OMERO.

PS: I quickly checked with MM1 (also last nightly build), that we are not using anymore, and it appears to be the same problem.

Thanks again for your help.
Philippe

[PhilippeP]

If it can help, I just uploaded via the interface http://qa.openmicroscopy.org.uk/qa/upload/ the results of the two tests (MM2-1 and MM2-2) discussed in the above post.
Philippe

PS: I just realized that the second "displaySettings.txt" file replaced the first one... (same name).
They are identical anyway, so it should not be a problem.

[rgozim]

Hi Philippe,

Thanks for sending through the files!

We currently have a Trello card to record a similar issue to the one you described, and I have updated it to include your findings. If you follow the card you will see from the latest comment by Sébastien that the MM2_2_MMStack.ome.tif file you supplied has the same xml Name attribute assigned to each <Image> element. As OMERO uses the image name within the xml to assign a name to each image, it results in what you have seen.

Running the following command through a terminal with the tiffcomment tool (included with the bioformats command line tools) :

Code: Select all

$ tiffcomment directory_to_your_file/MM2_2_MMStack.ome.tif | xmlindent

Screen-Shot-2018-09-25-at-11.18.47.jpg (162.61 KiB) Viewed 2358 times

looking at the screenshot, it can be seen from the <StageLabel> element that there is a unique Name attribute for each image within the stack file. One possible solution therefore would be to combine the <StageLabel Name=""> with the <Image Name=""> when creating the stack file.

Another solution, which is probably easier, would be to use the latest OMERO.insight client (found here) which has just been released. It will work with OMERO.server 5.4.x and removes the 2000 file limit that you mentioned previously!

Thanks,
Riad

[PhilippeP]

Hi Riad,

Thank you for your detailed explanation.
And the information on the new Insight version, which is clearly the solution I am going for!

Philippe

[PhilippeP]

We bumped in a problem using "Save x-y positions in different stack files" in Micro Manager MDA settings to allow for OMERO well and position information recording (see above). Everything goes smoothly for about 1000 images, and then MM crashes, because of memory problem.

Apparently, MM keep the images in RAM during the acquisition when selecting "Save x-y positions in different stack files", whereas when not selecting "Save x-y positions in different stack files", we have no problems even after more than 10000 images. Except for the fact that we are unable to know from what wells the images are coming from once imported in OMERO...

I am thus afraid we don't have any way of importing our images with well and position information in OMERO when doing high throughput acquisitions with Micro Manager.

[dgault]

If you have a small sample file saved with and without the "Save x-y positions in different stack files" option, then you can upload it to https://www.openmicroscopy.org/qa2/qa/upload/ and I would be happy to take a look at it and test it further.