Open Microscopy Environment

Hello,

I am reading through the documentation and find that I have come to a crossroads in terms of understanding the difference and use for the OMERO Python API and simply the OMERO API. Let me back up a step to walk you through my logic and see where I have gone wrong.

The first developers page makes reference to OMERO.web and OMERO Clients. https://www.openmicroscopy.org/omero/developers/.

If you follow the link to OMERO clients, https://docs.openmicroscopy.org/omero/5.4.0/developers/GettingStarted/AdvancedClientDevelopment.html you will notice the first `Note` that says if you are only interested in using and not developing the OMERO client you should follow the OMERO client overview https://docs.openmicroscopy.org/omero/5.4.0/users/clients-overview.html.

Here the series of client applications: OMERO.web, OMERO.insight and OMERO.importer, are mentioned. Which leads me to my first question.
1) Is OMERO.web simply a component of OMERO clients? What is their relationship and why does the first page you enter mention them as separate entities if going down the chain you can learn about OMERO.web anyway?

Flow Diagram 1:
Developing OMERO clients -> OMERO clients overview -> OMERO.web framework

The next inconsistency in my understanding is the difference between the OMERO API and the OMERO Python API. If you start reading from the 'Developing OMERO clients' page, there is mention that a Blitz client is any application which uses the OMERO API to talk to the OMERO.blitz server. Without an example I don't know what a Blitz client is . If we follow a separate track via the OMERO.web framework page https://docs.openmicroscopy.org/omero/5.4.0/developers/Web.html, there is mention that the OMERO.web framework is all based on the OMERO Python API. I couldn't find any "OMERO Python API", but I was able to locate a page for the "OMERO Python language bindings" which states that in addition to the auto-generated Python libraries of the core OMERO API, you have developed a more user-friendly Python module ‘Blitz Gateway’ that facilitates several aspects of working with the Python API.

This leads to my second question.
2) What is the difference between the OMERO API vs OMERO Python API vs OMERO Python Language Bindings

Flow Diagram 2:
Developing OMERO clients -> OMERO API
vs
OMERO.web framework -> OMERO Python API
vs
Using OMERO API -> OMERO Python Language Bindings (developed Blitz gateway instead of OMERO API)

And last question,
3) How does the Blitz client differ from the Blitz gateway.

This is my depiction of the system so far, which definitely has mistakes. I am attempting to understand the flow of data throughout the system.



I would most like to clear up how and data is transferred from OMERO.web to OMERO.server, the Blitz gateway has thrown off my understanding due to the comments on the omero.model being wrapped and then transferred to the OMERO API (if this is correct). My original thought was that we were passing JSON objects between the Django API app and OMERO-server but that can't be right. I know this is probably the longest post by the most confused individual you have seen in 12 years so I thank you in advance.




Personal Note (My 2 cents after working with OMERO for 6 weeks):
I truly believe in this project, but I also think the number of hyperlinks embedded within each document makes it hard to get through a page without first being told to read through 5-10 other pages. There should be an effort to consolidate the number of pages so that there is a natural flow to the way the docs are read because currently a new developer is thrown into the deep-end in trying to make sense of the docs.

I think lessons can be learned from the Docker tutorials (https://docs.docker.com/get-started/) and the Robot-Operating-System tutorials (http://wiki.ros.org/ROS/Tutorials). In the OMERO docs, there isn't a clear path to teach yourself what you're looking for. An example is on the OMERO.web framework page (https://docs.openmicroscopy.org/omero/5 ... s/Web.html), there is mention of an app that lets you browse data through map annotations linked to images called OMERO.mapr. It makes reference to a repository (https://github.com/ome/omero-mapr/) with the instructions on how to install and test the application. But nowhere is there mention of how a user actually is supposed to use it.

A second example can be found in the installations page for the OMERO.server (https://docs.openmicroscopy.org/omero/5 ... ation.html). You'll notice that the first sentence that reads, "Depending upon which platform you are using, you may find a more specific walk-through listed below but we recommend you read through this page first as it explains the entire process." The problem with this is that you yourself have to keep track of the flow of docs that should be read. Some developers may choose to skip straight to the document that is specific to their system and some may read through to the bottom of the page and forget to go back to the top to find their OS specific installation.

In my case, I moved onto the Ubuntu Installation where the very first line tells me that this page "should be read in conjunction with OMERO.web administration" (https://docs.openmicroscopy.org/omero/5 ... l-web.html). I find that is difficult to hold the readers attention in one place, to ask them to read 2 documents at the same time with no guidance in what order they should be read, makes things difficult. In the paragraph following I'm told "This guide does not describe how to install OMERO.web. To deploy OMERO.web separately from OMERO.server (recommended), please read OMERO.web installation separately from OMERO.server on Ubuntu 16.04 and IcePy 3.6 or to deploy with OMERO.server OMERO.web installation with OMERO.server on Ubuntu 16.04 and IcePy 3.6". Here's the first problem, I now have to keep track of more links to make sure I get this done correctly. There are often times where I had 15-20 tabs open in my browser to different OMERO pages. This quickly becomes a programmers, all too often, nightmare of keeping track of an unnecessarily complicated system. The second problem and more importantly, is that I'm not told WHY I should be choosing either scenario. Throughout the docs components of the system are given to the user but often times the reason behind some functional component is left out. A system that is extremely versatile and has 10,000 tools is not necessarily better than a system that has 100 tools. Especially if the user/developer doesn't know how to make use of 90% of the tools.

I have been able to learn complex architectures (with more functionality than OMERO) in under a week, but have been working on OMERO for nearly 6 weeks and am still unable to tell you the difference between OMERO.web and an OMERO client and how to go about editing the source code without breaking the OMERO data model. Infact, the only reason I was able to get the server/client up and running is because there are OMERO-web and OMERO-server Docker images in the OME repository.

I think if you insert into the docs some language that focuses on reiterating the WHY of each step, that would be hugely important. For example why the 'web-gateway' is used or the 'Blitz-gateway' or 'Ice', instead of just the pure functionality of how it works. Don't assume the developer has a deep understanding of the system the 5th layer down. A gentle constant reminder of why we are using OMERO Python API would be great. If you really want a product to become widespreadly used it needs to be EASY to learn, and EASY to use. You really can't create a standard if nobody knows how to use it. If the documentation is cleaned up, an added bonus is that there is less likely a need for the forum because ideas are clearly expressed in the documentation and walkthroughs have a natural flow in installing the system and extending the system.

I'd like to close by saying that although everything I just said may seem critical, it goes without saying that this project has done a great job blending years of knowledge in the field for scientists to consolidate their needs in using a system that allows them to store and share their data. I am using OMERO because it is the best option out there, and I would like the number of users to spread like wildfire, but I don't know if the documentation yet allows the functionality of the project to shine through.

Hi Max,

I am reading through the documentation and find that I have come to a crossroads in terms of understanding the difference and use for the OMERO Python API and simply the OMERO API. Let me back up a step to walk you through my logic and see where I have gone wrong.

Thanks for the outline of how you got to where you are. I’ll answer the more immediate questions inline, and deal with the larger issues at the bottom.

Here the series of client applications: OMERO.web, OMERO.insight and OMERO.importer, are mentioned. Which leads me to my first question.
1) Is OMERO.web simply a component of OMERO clients? What is their relationship and why does the first page you enter mention them as separate entities if going down the chain you can learn about OMERO.web anyway?

There are currently three separate clients to OMERO that we maintain. OMERO.web and OMERO.insight are the two main clients. There’s also an OMERO.cli which is more heavily used by power-users and sysadmins. OMERO.importer is now included in OMERO.insight and is as such no longer separately downloadable.

The next inconsistency in my understanding is the difference between the OMERO API and the OMERO Python API.

The Python API is one language binding for the Ice protocol that we use. The API is defined in Ice’s modeling language, slice. We also support Java and C++. In each language, there’s a layer on top of the code-generated bindings to make working with the language simpler.

If you start reading from the 'Developing OMERO clients' page, there is mention that a Blitz client is any application which uses the OMERO API to talk to the OMERO.blitz server.

Blitz is synonymous with OMERO.server.

This leads to my second question.
2) What is the difference between the OMERO API vs OMERO Python API vs OMERO Python Language Bindings

The latter two are synonymous and as above, represent a concrete binding of the former.

And last question,
3) How does the Blitz client differ from the Blitz gateway.

A “Blitz client” or synonymously an “OMERO client” can choose to make use of the BlitzGateway which is the layer on top of the code-generated language binding. (Note: there is no BlitzGateway in C++)

This is my depiction of the system so far, which definitely has mistakes. I am attempting to understand the flow of data throughout the system.

If I understand your arrows correctly, you probably have one or two data transmissions too many. With ------ being a process boundary requiring data serialization:

Code: Select all

  Client
  ------
  OMERO.web (Django server, but OMERO.client making use of Python BlitzGateway)
  ------
  OMERO.server (OMERO.server written in Java)
  ------
  PostgreSQL


You are correct, however, that internally OMERO.web and the apps written for it ("OMERO plugins" which are "Django applications") do make use of the JSON API which requires another serialization.

Code: Select all

I would most like to clear up how and data is transferred from OMERO.web to OMERO.server, the Blitz gateway has thrown off my understanding due to the comments on the omero.model being wrapped and then transferred to the OMERO API (if this is correct).

Unfortunately, no. Sorry for the confusion. BlitzGateway is a client-side helper, not a service itself.

My original thought was that we were passing JSON objects between the Django API app and OMERO-server but that can't be right.

In the case of the under-development JSON API, an extra Django app generates the JSON for any consumer (browser or other app).

I know this is probably the longest post by the most confused individual you have seen in 12 years so I thank you in advance.

I would have to check, but in any event, you’re welcome.

Personal Note (My 2 cents after working with OMERO for 6 weeks): ... snip ...

All of your suggestions are appreciated. It’s not in any way an excuse but perhaps an explanation: much of the content that you've been diligently reading was migrated from the much older trac wiki. Obviously, like everything, it should be updated and continually polished, a task we're aware of and very much would like to tackle, but of course, there are only so many things we can tackle at once. If it gives you some hope, we recently finished the upgrade of https://www-legacy.openmicroscopy.org to https://www.openmicroscopy.org. Here's to an equally remarkable process for the docs!

Cheers,
~Josh

Hi Josh,

Thank you so much for taking the time to clear things up. I want to complete my understanding of the system. You say BlitzGateway is a client side helper. It works well with the Python API because of its ability to store sessions and retrieve data objects from OMERO. I also see how the Blitz Gateway wraps omero.model objects into Pythonic objects to be used by the Python API.

What I don't see however is how the JSON API differs from the OMERO Python API/BlitzGateway. Can you help me fill in the blanks?

Code: Select all

Option 1:
User annotates image -> "webclient" django app -> "JSON api" django app ----JSON---> OMERO.server -> PostgreSQL

Option 2:
User annotates image -> MISSING -> BlitzGateway & OMERO Python API in a file ------ICE----> OMERO.server -> PostgreSQL


Oddly my data flow either uses the JSON API or Python API/BlitzGateway but not both. I read in the docs that the JSON API converts an OMERO object into JSON, this would seem to make sense as it can then be sent across the internet and retrieved by the OMERO.server. But know I've also looked at an example Python script involving including the BlitzGateway to retrieve data from the OMERO.server. Are both viable options to send data between OMERO.web and OMERO.server?

Max,

maxgitt wrote:Thank you so much for taking the time to clear things up.

Gladly.

What I don't see however is how the JSON API differs from the OMERO Python API/BlitzGateway. Can you help me fill in the blanks?

At the moment, the JSON API is implemented using OMERO Python API/BlitzGateway. Longer-term, the hope is that the JSON API will be the main interface.

Extending my previous notation:

Option 1:

Code: Select all

User annotates image  (Browser)
------ (HTTP POST)
OMERO.web view ("webclient" django app)
------ (HTTP GET)
"JSON api" django app - using BlitzGateway
------ (Ice)
OMERO.server
------ (JDBC)
PostgreSQL


Option 2:

Code: Select all

User annotates image  (Browser)
------ (HTTP POST)
OMERO.web view ("webclient" django app) - using BlitzGateway
------ (Ice)
OMERO.server
------ (JDBC)
PostgreSQL


So perhaps the confusion is that the "JSON API" is a Django App that provides a JSON endpoint to the client, but internally, it's just another OMERO.client using Ice. (Note: option 2 is the more likely one for this example. Each app will decide for itself though if it wants to consume the JSON API or not.)

Oddly my data flow either uses the JSON API or Python API/BlitzGateway but not both.

Though the JSON API internally uses the BlitzGateway, the either/or split makes sense. We'll continue migrating services from BlitzGateway to the JSON API.

I read in the docs that the JSON API converts an OMERO object into JSON, this would seem to make sense as it can then be sent across the internet and retrieved by the OMERO.server. But know I've also looked at an example Python script involving the BlitzGateway to retrieve data from the OMERO.server. Are both viable options to send data between OMERO.web and OMERO.server?

Definitely, or between any OMERO.client and OMERO.server, though the JSON API is not yet complete.

Cheers,
~Josh

Great. So to my understanding any Django app, that lies within the OMERO.web client, can use the BlitzGateway to communicate with the OMERO.server (aka Blitz server). I can also make use of the BlitzGateway through the OMERO.insight client to contact the Blitz server.

We have referred to the BlitzGateway and OMERO Python API rather synonymously, however. Which makes me question how they differ. I understand the BlitzGateway is a client-side library that wraps omero.model objects, but how does it make use of the Python API. And as depicted below, it seems the Python API plays a role in the allowing the gateway to contact the server.



So what is the relationship between the Gateway and Python API? Does the gateway essentially make calls to the functions within the Python API? In my effort to update the OMERO.web webclient app, will I only need to use methods from the BlitzGateway library or is there a chance I also require usage of the functions in the Python API?

What I also don't understand about the documentation is why in the BlitzGateway page, you mention, "we encourage its use for all access to the OMERO Python API." So why order the readings by having the developer read about the Python API first (https://docs.openmicroscopy.org/omero/5.4.0/developers/index.html#using-the-omero-api). My understanding is that the Python API is not directly used by me if I simply use the BlitzGateway.

Thanks ahead of time,
Max

maxgitt wrote:Great. So to my understanding any Django app, that lies within the OMERO.web client, can use the BlitzGateway to communicate with the OMERO.server (aka Blitz server). I can also make use of the BlitzGateway through the OMERO.insight client to contact the Blitz server.

Yes. In the case of insight, it will be the Java BlitzGateway which is much less pythonic.

We have referred to the BlitzGateway and OMERO Python API rather synonymously, however. Which makes me question how they differ. I understand the BlitzGateway is a client-side library that wraps omero.model objects, but how does it make use of the Python API. And as depicted below, it seems the Python API plays a role in the allowing the gateway to contact the server.

Staying with Python:

• omero.gateway.BlitzGateway can either create a connection to the server itself or be given an omero.clients.BaseClient object
• BaseClient.createSession creates an omero.api.ServiceFactoryPrx
• BlitzGateway methods will use ServiceFactoryPrx to create other services ending in Prx
• Anything ending in Prx is a code-generated stub for RPC directly to OMERO.server

So what is the relationship between the Gateway and Python API? Does the gateway essentially make calls to the functions within the Python API?

Yes.

In my effort to update the OMERO.web webclient app, will I only need to use methods from the BlitzGateway library or is there a chance I also require usage of the functions in the Python API?

It is likely that you will eventually reach a point where you want to use additional methods on the services themselves, but it makes sense to use the omero.api instances which have been wrapped by BlitzGateway (just as the omero.model instances are also wrapped).

What I also don't understand about the documentation is why in the BlitzGateway page, you mention, "we encourage its use for all access to the OMERO Python API." So why order the readings by having the developer read about the Python API first (https://docs.openmicroscopy.org/omero/5.4.0/developers/index.html#using-the-omero-api). My understanding is that the Python API is not directly used by me if I simply use the BlitzGateway.

This may come from a time when the BlitzGateways was being added, as the JSON API is being added now. As we've definitely already established, these pages need a review!


All the best,
~Josh

Hi Max,

Just to add a note: Even when using the Python BlitzGateway you may find that you need to use the underlying omero.model objects and api services since the BlitzGateway doesn't have complete API coverage. Certainly the OMERO.web framework does have significant usage of the plain OMERO Python API.

And the example I gave in your other post viewtopic.php?f=4&t=8397 is using the BltizGateway was the entry point but then using omero.model objects and the update service directly.

Thanks for your feedback.

Will.

Hi Will,

Thanks for the mention. Now that there is clarity on a solid chunk of architecture, I have begun looking at the OMERO.model and your code from the previous post. I have taken a look at the Schema (http://www.openmicroscopy.org/Schemas/D ... 6/ome.html) and the glossary of all OMERO model objects (http://docs.openmicroscopy.org/omero/5. ... bject.html) to get a better understanding for my use case. There are a lot of ModelObjects that map to the Schema Objects, so maybe it'd be easier if I was more specific.

Here is exactly what I'd like to do:
1) User opens OMERO.web and selects an image and clicks on the 'Acquisition' tab
2) User clicks 'show unset fields' and sees the following

3) User edits the 'Name' to "Will's healthy cells"
4) This triggers a python script that makes use of BlitzGateway to retrieve this OMERO.object from Postgres and updates the object

Your great example (https://gist.github.com/will-moore/b9ba ... b1d1b47858) definitely helps me get started with step 4. But I currently am faced with two issues:

1) I don't know where to place the python script that can make the API calls and update the webview

2) In your example, I don't know where your knowledge to call objects such as MicroscopeI(), ImageI(), and ProjectI() is coming from. The BlitzGateway page itself doesn't mention what these objects are.

Hi,

This page lists all the OMERO objects and their relationships (and which attributes are 'not null') etc.
http://docs.openmicroscopy.org/latest/o ... bject.html

You can also use tools to inspect the Postgres database which will show corresponding tables.
E.g. for looking up Enum types with psql:

Code: Select all

omero=# select * from immersion
;
id | permissions |    value     | external_id
----+-------------+--------------+-------------
  1 |         -52 | Oil          |           
  2 |         -52 | Water        |           
  3 |         -52 | WaterDipping |           
  4 |         -52 | Air          |           
  5 |         -52 | Multi        |           
  6 |         -52 | Glycerol     |           
  7 |         -52 | Other        |           
  8 |         -52 | Unknown      |           
(8 rows)


In your screenshot example, the Name is the name of the first Channel. You can edit channel names where Channels are listed under the General tab for an image.

You can't directly edit the metadata under the Acquisition tab, but I have written a server-side OMERO scripting service script (see http://docs.openmicroscopy.org/omero/5. ... index.html).

https://gist.github.com/will-moore/370e ... 5e85389272

Once uploaded, select the Image(s) you want to edit, then launch the script which looks like
this:



Once the script is run, the changes will be shown under the Acquisition Tab.
Hope that helps.

Will.

Hi Will,

What if I want to allow the user to make edits directly through the UI without running a script? Am I crazy? I just want to allow biologists to use the interface with as little programming knowledge as possible. If all goes well I will be scaling this project to a few dozen Universities around the country.

I think I may be missing something. What advantage do the scripts hold over conventionally using the UI?

Thank you for your time and patience,
Max