organizational hierarchy

[dsudar]

Hi team,

I frequently get the question why OMERO doesn't provide more levels of "organizational structure" such as sub-projects and/or sub-datasets etc. I typically babble my way out of that with statements such as: that's the way it is, can't you organize your data within that structure?, isn't this much clearer/cleaner than having many folders in folders in folders in folders ....?, how deep of a structure WOULD you need and will that always be enough?, etc. etc.

Is there a good write-up somewhere that properly explains and justifies the considerations that went into the structure and data model? It would be great to have such an authoritative reference rather than my bumbling babble.

Once someone has accepted that that's the way it is, I can try to explain to them the use of tags to impose an additional or orthogonal organizational ability and a very few actually have started down that path but it's hard to get folks to think of OMERO as more than just a filesystem.

Thanks,
- Damir

[jburel]

Dear Damir

Thanks for raising the issue. This is a regularly asked question.

The OME model is used as the basis of the Bio-Formats/OMERO stack and was originally designed in 2003 with only the levels Project/Dataset/Image/Pixels in mind. We then added support for the screening data model in 2007 [1].

Such a restricted approach has its advantages when, for example, retrieving the data since queries can be optimized and clients can know the objects to expect and display. It also offers a “rigid” organisation, useful when collaborating on a research project.

Nonetheless, the tree display of the data in both OMERO.web and OMERO.insight suggests that it “should” behave like a file system without offering the flexibility that scientists are familiar with from using a file system. There are organizational workflows that can used to extend the existing hierarchy. A folder’s name might correspond to a tag; other knowledge can be encapsulated in the file name itself.

In 2016 [2], we introduced the concept of Folders in the model in order to add support for multiple levels of hierarchies. A Folder specifies a possibly heterogeneous collection of data and may contain other Folders, Images and ROIs. Data may be in multiple Folders but a Folder may not be in more than one other Folder, giving a strict tree hierarchy.
The primary driver for the addition is to organize ROIs into hierarchies. This was described in a blog post [3]. At the client level, it was only implemented at the time in OMERO.insight and this is certainly a key feature we will need to add to the web client and apps.

In the long term, the extensibility of the Folders structure already introduced in the model and supported by the API could allow us to replace the actual tree display of the data. This could have an impact on the retrieval/display performance due to the heterogeneous nature of the structure and this impact will be to be accounted for.

A few points to consider were captured in some initial design issues [4]. Such major changes will require serious planning and will come at the cost of other important requirements.

Do you think that the limited hierarchy is a blocker to start using OMERO?

Thanks for your valuable comments and feedback

Cheers
Jmarie

[1] https://docs.openmicroscopy.org/latest/ome-model/schemas/june-2007.html
[2] https://docs.openmicroscopy.org/latest/ome-model/schemas/june-2016.html
[3] http://blog.openmicroscopy.org/data-model/future-plans/2016/05/23/folders-upcoming/
[4] https://github.com/openmicroscopy/design/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+Folders+in%3Atitle+

[evenhuis]

Hi Jmarie,

At the moment I am putting together some guidelines for transitioning scientists from filesystem storage to OMERO. I would love to hear how people approach this and I think it would be really helpful if there were examples in the documentation of how to manage and organise data in OMERO.

Here is my 2 cents:

Do you think that the limited hierarchy is a blocker to start using OMERO?

Definitely, as Damir points out, a deeply nested hierarchical folder system is a very natural and flexible way for people to organise their data. Squashing this down to 2 levels means that researchers are faced with exponentially more data at project and dataset level. This is a real problem for PI with decades worth of data and little time for philosophical discussions of metadata conventions.

Without some concrete guidelines of how to deal with this, and maybe some examples of the benefits, users will revert to filesystem storage on unmanaged USB drives.

can try to explain to them the use of tags to impose an additional or orthogonal organizational ability

I haven't found tags that useful, but key-value pairs are really powerful for searching and organising data. Unfortunately, I've had trouble finding tools for adding / managing kv pairs in OMERO and I've started to write some scripts for extracting metadata from filenames and directory structures. Excuse the terrible code quality:
https://github.com/evenhuis/omero-user-scripts

There are organizational workflows that can used to extend the existing hierarchy. A folder’s name might correspond to a tag; other knowledge can be encapsulated in the file name itself.

Could you share some on how to go about this? Sometimes I feel like other groups must have addressed this.

The web tagging tool looks really user friendly way to extracting metadata from filenames, but I'd rather use kv pairs.
http://help.openmicroscopy.org/web-tagging.html

mapr looks really good way to view kv pairs, I assume that is what is being used on IDR website. I haven't found how to manage them yet.
https://github.com/ome/omero-mapr/

My pitch is that the search on kv pairs gives the power to slice and dice and reorganise your data in ways that a fixed folder-based hierarchy can't.

Cheers,

Chris

[wmoore]

Hi Chris,

Thank you for your feedback.

As Jean-Marie said, we need to plan when to tackle the hierarchy/folders issue as it’s one of a number of big feature requests. We are also finding the limited hierarchy to be an issue in our own usage of OMERO for https://idr.openmicroscopy.org/. Your feedback will be part of that decision so it’s good to have.

In the short term, we’ll put together some documentation on how to use mapr to add a level or two of hierarchy via key-value pairs.

Thanks for the link to your scripts. We’ll have a look at them. We’ve noted the lack of tools for adding map annotations (https://trello.com/c/VZqmJCyP/41-batch- ... nnotations) so we’ll be trying to improve these workflows.

Regards,

Will.

[wmoore]

Hi Chris,

I've added more info to the MAPR docs at https://github.com/ome/omero-mapr
so hopefully there should be enough info there for you to get it working to search for user-added Map Annotations.
The only limitation is that the Keys that you search for have to be in the server-side configuration, since mapr was designed to work with a relatively small number of pre-defined Keys (as in IDR).
But let us know how you get on.

Regards,

Will.

[evenhuis]

Hi Will,

thanks for adding more documentation to github page - it helps a lot. We'll take it for a spin once we have chance.

One last question, I'm finding hard to information about what a namespace is and how to use them. There's a little bit here:

https://docs.openmicroscopy.org/omero/5.3.4/developers/Model/StructuredAnnotations.html#names-and-namespaces

From the mapr readme, is this step here adding a new namespace?

Code: Select all

$ bin/omero config append omero.web.mapr.config '{"menu": "antibody", "config":{"default":["Primary Antibody"], "all":["Primary Antibody", "Secondary Antibody"], "ns":["openmicroscopy.org/omero/client/mapAnnotation"], "label":"Antibody"}}'

Cheers,

Chris

Cheers,

Chris

[wmoore]

Hi Chris,

OMERO uses namespaces on annotations simply as a way of classifying annotations, E.g. Ones that serve a particular role or come from certain software.
E.g. OMERO.figure saves files as FileAnnotations with namespace ""omero.web.figure.json" so it can query for all saved figures.

It's just an optional string on all Annotations - there's no namespace table to list, add or keep track of them.
You can see some examples of setting namespace on annotation by searching for "namespace" in the Python docs
https://docs.openmicroscopy.org/omero/5 ... ython.html (also similar for Java).

The config in mapr is simply telling mapr to query existing MapAnnotations with a particular namespace. You have to create those MapAnnotations with matching namespaces, either in the client UI (which will create MapAnnotations with the 'client' namespace as discusses) or via the OMERO API.

Hope that helps?

Cheers,

Will.

[wmoore]

Hi - I've just been told that there is a Namespace table in the DB and it seems to auto-update itself, but I don't know what it's used for (and you don't have to know about it to use namespaces (as I didn't)!

Will.