putting acquisition metadata into OMERO after import

[dsudar]

Hi,

For many of the files we import from HCS experiments, the OME acquisition metadata is filled in only partially. This happens for nd2 files from a Nikon HCA instrument as well as image sets acquired using an Olympus ScanR and GE INCell 6000. Much of the missing info is in the proprietary files themselves and much of that does appear in the "Original Metadata" table. Assuming I have a way to extract and decode that stuff, is there an recommended method to populate the OME model inside OMERO with that missing acquisition metadata?

Thanks,
- Damir

[sbesson]

Hi Damir,

On the OMERO side, I can think of two approaches to populate OME metadata:

- using MapAnnotations, assuming you can generate CSV files, you should be able to use the populate metadata functionality to update existing data
- in case you want to update and/or create non Annotations elements e.g. Instrument, Detector, you liekly need to use the API and create specific scripts to achieve such operations. Ola had worked on a similar problem to update the physical sizes of pathology images. She is on leave this week but might have some pointers on return.

In the midterm, the canonical way to solve these issues would be to populate this acquisition metadata in the OME-XML metadata store directly in the corresponding Bio-Formats reader(s). That way, new filesets would be directly imported with the full metadata into an OMERO server.

Do you have a preliminary list or spreadsheet detailing the correspondence between OriginalMetadata values and OME metadata?

Best,
Sebastien

[wmoore]

Hi Damir,

Although this script is for reading instead of writing metadata, it might give you some idea of where to store various pieces of metadata.

https://github.com/openmicroscopy/openm ... etadata.py

It might also be useful as a starting point for writing metadata if you're mostly filling in missing attributes on existing objects.
However, it would need a bit more work to create all those objects from scratch.

Regards,

Will.

[dsudar]

Hi Sebastien and Will,

Thanks for the quick replies.

On Sebastien's first suggestion: yes, I had thought about just using the regular MapAnnotations but the significant downside is that those will then be separate and incompatible with the Instrument etc. acquisition metadata that is correctly ingested by the importer and used to populate the OME-XML data structures in OMERO.
Second suggestion: yes, that is what I'm really after and Will's pointer and example code is a good start. If Ola indeed has some good examples how to actually fill those structures, that would be extremely useful.

And indeed, ultimately the right way is to properly fill those structures during the import. Since the Nikon HCA-generated nd2 files are my main concern at this time, I'll upload an example one of those to the QA system with a table of suggested mappings that seem the most appropriate to me. I'll get to the Scan^R and INCell ones in the next few days.

Cheers,
- Damir

[wmoore]

Hi Damir,

Just to get you started using Map Annotations, here's a script that will pick specified pieces of Original Metadata and use them to create a new map annotation on the image.

https://gist.github.com/will-moore/03ef ... 38ba379b4c

To populate all the metadata in the correct place in the model in OMERO will be a fair bit more work and will be most efficient to implement this only in Bio-Formats (if the map-annotation solution works for you in the meantime)?
I've created a card for that: https://trello.com/c/5peIUoc4/133-nd2-metadata

Hope that helps,

Cheers,

Will.

PS. If Ola has anything that looks useful then we'll let you know.

[dsudar]

Hi Will,

Thanks much for the piece of code. Looks indeed very straightforward to add MapAnnotations that way.

On your point that the best place to populate the OME model is in BioFormats, I agree in principle. And thanks for adding a Trello card for that. I'll provide some more the tables I did for other formats (similar to what I did for ND2) that I use. I guess implementing that type of stuff is for the rainy afternoon when there's nothing more interesting to do?

However, in practice that will only provide a partial solution since many of the proprietary formats simply don't have all the relevant information in them. ND2 is a case in point where info about the filters used is simply missing. For that one would still need to be able to populate the model after the import ... or ... have a provision to provide a "companion file" with the image files that Bio-Formats can pick up and use to populate the model. Maybe worth a brief discussion on the Friday at the Users Meeting?

Thanks,
- Damir

[atarkowska]

Hi Damir,

Sorry for a slow response. Once I had to populate missing pixel data and reimport wasn't an option. The following example https://gist.github.com/aleksandra-tark ... 8a1c53406e may give you an idea. (that was written against 5.1).

Regarding MapAnnotations, python scripts are good starting points, although you may try similar to what we use in IDR, automation of post import processing? I've just backported new features to main line, so I could give you a custom 5.3 build including https://github.com/openmicroscopy/openm ... /pull/5241 and instruction how to prepare input data.

Here is test example how that works:

First you need csv and yml files (test files: https://github.com/aleksandra-tarkowska ... ontext.yml and https://github.com/aleksandra-tarkowska ... ext_ns.csv) providing input data and then just populate them across entire screen at once.

Let me know what you think

Ola

[sbesson]

Hi Damir,

in addition to Ola's links, a few answers more specific to the formats questions and thoughts.

On your point that the best place to populate the OME model is in BioFormats, I agree in principle. And thanks for adding a Trello card for that. I'll provide some more the tables I did for other formats (similar to what I did for ND2) that I use. I guess implementing that type of stuff is for the rainy afternoon when there's nothing more interesting to do?

That's definitely one way to put it. The Bio-Formats backlog board is periodically reviewed by the open-source team. The prioritization of these cards is usually a balance between the importance/impact versus the size of the task and the available time and resources.

On your initial proposal of metadata mappings, does it contain anything private or would it be acceptable for us to make the spreadsheet public e.g. via the card? Doing so would allow both the discussion to happen publicly but also anyone with resources who would be interested/concerned by the issue to look into this problem without necessarily waiting for the open-source team agenda.

However, in practice that will only provide a partial solution since many of the proprietary formats simply don't have all the relevant information in them. ND2 is a case in point where info about the filters used is simply missing. For that one would still need to be able to populate the model after the import ... or ... have a provision to provide a "companion file" with the image files that Bio-Formats can pick up and use to populate the model. Maybe worth a brief discussion on the Friday at the Users Meeting?

Agreed. The problem of expressing, storing in a flexible way and linking metadata that is not always captured during the scope of the acquisition is a generic problem that arises under various forms. Providing some robust solutions to address these problems is definitely on our radar for the upcoming years. It would be timely to organize this discussion at the Users Meeting and invite everyone interested by the issue so that we can capture all use cases.

Best,
Sebastien

[dsudar]

Hi Ola and Sebastien,

Thanks Ola, for the example and the news about the new annotation tools. Let me look carefully at all that and I'll be in touch soon.

On the question whether there is anything private in the my metadata mapping proposal, the answer is no so yes, it's perfectly fine to post the spreadsheet for others to comment on and add to.

I'll try to have a few slides to initiate a discussion at the Users Meeting on the "adding various metadata after the fact" issues.

Cheers,
- Damir