GUI support for Key-Value Pairs (MapAnnotation) metadata?

[Kouichi_C_Nakamura]

Hello,

One of the major drawbacks I found in OMERO in terms of sharing images within a lab/institute was that the difficulty in handling metadata. The metadata embedded within images are fine, but adding Key-Value Pairs (MapAnnotation objects) to images takes rather complicated Java or Python programming. I can't ask everybody in the lab to understand a dozen of JAVA objects just for this purpose.

I wrote a MALTAB function (https://github.com/kouichi-c-nakamura/o ... notation.m) to simplify this process. It takes a string array with 2 columns as an input argument and turns it into MapAnnotation object. But this still takes accessing OMERO via JAVA API from within MATLAB.

To extend this idea further, how about providing GUI support for Key-Value Pairs assignment? This can be implemented in OMERO.insight or as a plugin for OMERO.insight, or even in web client (this is probably the most user-friendly option).

1. A user is prompted to choose a .csv, .tsv, .xls, or .xlsx file that contains data in 2 columns.
2. The user is then prompted to choose an object or multiple objects (project, dataset, or image etc) to which he/she wants to link the new MapAnnotation data, by either clicking the objects in web client or typing the ID numbers.
3. If the input file contains more than two columns, issues an error.

Also, I can think of GUI support for exporting MapAnnotations into a .csv, .tsv, or Excel file.

I hope that you like the idea.

Best,
Kouichi

[Kouichi_C_Nakamura]

Given that OMERO.insight is now in maintenance mode, the most realistic implementation is to write an app for OMERO.web in Python?
https://docs.openmicroscopy.org/omero/5 ... teApp.html

Webclient Plugins may also be the one.
https://docs.openmicroscopy.org/omero/5 ... lugin.html

I'm not sure about the relationship between them, though.

[jmoore]

Good morning, Kouichi.

In general, yes!, we like the idea. There have been a number of different discussions regarding this (including the ones you triggered yesterday). Here's a quick summary with options:

• (MATLAB) Someone in the community had previously developed such a GUI in MATLAB. We didn't find it in a repository we could link you to, but perhaps he will see this and post himself.
• (Scripts) Chris Evenuis did something similar using OMERO.scripts. See https://github.com/evenhuis/omero-user-scripts#key_val_to_csvpy-key_value_from_csvpy Scripts are a fairly low-cost way to add functionality to both UIs at the same time. That may get you what you need for the moment. If so, perhaps something along this lines should be shipped with the base OMERO installation.
• (Web) Writing an app or an extension to an existing app is indeed likely the way forward. If that's something you'd like to get involved in, the team would be happy to discuss implementation details here.
• (Insight) Though "It's not dead yet" does seem to be the most appropriate slogan for insight, adding larger pieces of functionality does take some time. But of course, if anyone is interested in pursuing this, we're happy to help.

Let us know which way you are leaning and we can discuss further. Also, if anyone else in the community is pining for the same thing, let us know!
~Josh

[wmoore]

Petr and I discussed this a bit in the office yesterday.
In general we'd like to improve the batch adding of Key-Value pairs in the webclient.

Adding a new set of Key-Value pairs to a bunch of selected Objects (Images, Datasets etc) is pretty
straight-forward. We'd create a new Map Annotation for each object so that editing one of them doesn't update the values on all objects.
However, there are some issues with batch editing: If the user modifies the Key-Value pairs on each object,
so that each one has non-identical Map-Annotation objects, then this makes it tricky to allow batch editing.
At the moment, when multiple objects are selected, all the Map annotations are shown separately and non editable in the right panel. We *could* look for any Map annotations that have identical Key-Value pairs and combine them in the UI and make them editable, so that you'd be able to batch-edit them, as long as they remain identical.

This wouldn't allow applying different values for each object, so you'd still want to use CSV upload and script to do that.

If this becomes the way that most users want to update and edit Key-Value pairs, then probably the webclient could support that functionality natively (e.g. in the same way that iviewer allows you to export ROIs as a CSV).

Will.

[wmoore]

Dear Kouichi,

To answer your question above about the difference between OMERO.web "Apps" and "Plugins"...

We use "Plugin" to refer to an OMERO.web App that is further configured to display it's content within the webclient UI, instead of in a separate web page.
This makes it easier for users to find and use the additional functionality within webclient, and the app can work with the currently selected objects in webclient, instead of requiring users to open a separate page.
However, with the "Open with" functionality (see https://docs.openmicroscopy.org/omero/5 ... #open-with), it is possible to open selected objects in a separate app page (this is what we do with the OMERO.iviewer and OMERO.figure apps).

So it really depends whether your app can work in the centre or right panels of the webclient (a Plugin) or if it needs it's own page (use Open with).
Hope that helps,

Will.

[Kouichi_C_Nakamura]

Thanks, guys.

Chris Evenuis' work appears to be what I was looking for!
https://code.research.uts.edu.au/MIF/OM ... -csv-files

I really think something like this should be part of the main OMERO.web. Without GUI access, you can't really encourage people to use Key-Value Pairs.

[wmoore]

Trello card for this is at https://trello.com/c/VZqmJCyP/41-batch- ... nnotations

[evenhuis]

Hi all,

I just saw this thread and thought I would put my 2 cents in.

Also, if anyone else in the community is pining for the same thing, let us know!

As your aware, we would love more support for key-values across the board. Getting web plugin / app to replace the kludgy scripts I've written would be great. I happy to help out however I can, I don't have any web programming experience though.

Storage of key-values
Just some internal details of how the script handle key-value pairs. Most the complexity comes from OMERO allowing multiple copies of the same object and from MappAnnotations being immutable. I made the decision for the keys be unique and to allow each key to have multiple unique values.

To achieve this when the script reads in keys they are stored as an ordered set. This automatically removes redundant keys. The values or each key are gain stored as ordered sets.

We *could* look for any Map annotations that have identical Key-Value pairs and combine them in the UI and make them editable, so that you'd be able to batch-edit them, as long as they remain identical.

These types of operations are very easy with set operations.


Use of key-value pairs

Just a quick word on how our users are using key-value pairs. Kouichi, I don't know if you saw the other scripts in the repo but the one fore extracting kv pairs from the file names is probably our most used one.


This is workflow that most people use:

• Dataset uploaded
• Common Dataset / Project kv pairs added through the dataset description
• kv pairs extracted from the file names
• csv exported and kv pairs corrected for typos and more added
• new csv uploaded and kv pair updated.

This wouldn't allow applying different values for each object, so you'd still want to use CSV upload and script to do that.

Some thing like the autotagger plugin would be great for this.

If this becomes the way that most users want to update and edit Key-Value pairs, then probably the webclient could support

is it possible to make editable OMERO.table for this? Maybe a bit overkill as it could end up as being a full spreadsheet program embedded in OMERO.

Cheers,

Chris

[wmoore]

Hi Chris,

Thanks for your input. It's really useful to have some concrete workflows and your guides to understand what functionality is missing.

The "file_ann_id" at https://github.com/evenhuis/omero-user- ... sv.py#L122 allows the user to pick a particular file that is attached to the Dataset. It looks like your workflow is relying on the most recently added FileAnnotation to be listed last, but I don't think this is guaranteed.
If you click the "scripts" icon under the Attachments pane, you'll see a checkbox next to each FileAnnotation. The checked item (see screenshot) will be used to populate the File_Annotation input when you launch the script dialog.

I noticed that you have

Code: Select all

namespace = "openmicroscopy.org/mapr/gene"

in a few places in Key_Val_from_Description.py which I didn't understand?

Regards,
Will.

Screen Shot 2019-04-10 at 14.58.00.png (28.51 KiB) Viewed 1131 times

[evenhuis]

Hi Will,

If you click the "scripts" icon under the Attachments pane, you'll see a checkbox next to each FileAnnotation. The checked item (see screenshot) will be used to populate the File_Annotation input when you launch the script dialog.

thanks for that, I didn't know about that one.

in a few places in Key_Val_from_Description.py which I didn't understand?

Code: Select all

namespace = "openmicroscopy.org/mapr/gene"


This is left over code from seeing if namespaces and mapr would be help organise key value pairs (as you see from the code I'm not great with keeping the code clean and using branches for source control).

As the more of users add and use kv pairs we can see the need to search and filter kv pairs across experiments, between groups and across the institution. For this reason it we are keen for the update to the search tool to get proper searches (precedence of Boolean operations etc).

I never quite got my head around how namespaces worked for kv's but we didn't pursue it as it seemed that this approach would require a global set of of agreed upon kv's. I was keen to make it as easy as possible for users to start using it.

In past experience with metadata, having an XML architecture-astronaut say the first we need to agree on an ontologoly / schema is a road to nowhere. My plan for this was to set up some tools in something to like Jupyter notebooks to explore the kv's - things like clustering keys to detect groups and correct typos. Then when an analysis span datasets to remap the local keys / values as needed.

Cheers,

Chris