Importing MATLAB/NumPy arrays into OMERO

[clays]

Hello,

I would like to import raw sensor data and metadata acquired in MATLAB into OMERO. What is the best procedure for this? What I've gathered thus far:

Convert my raw data (2-d or 3-d vector) to a Java array and then use OMERO.tables for import. This is then associated with an image (how?). My metadata for the raw data would then just serve as the metadata for some dummy image. Am I on the right track?

[manics]

Hi Clays

The OMERO MATLAB package has some support for OMERO.tables, though you'd have to reformat it from a matrix into a columnar table. There's an example at https://www-legacy.openmicroscopy.org/s ... ero-tables

For the metadata key-value pairs could be an alternative: https://www-legacy.openmicroscopy.org/s ... nnotations

How are you planning to use the sensor and metadata once it's in OMERO?

Simon

[clays]

Thanks for the response Simon. Indeed, I've been looking at that example, and I've found myself a bit confused on how to use the approach there to be honest. I suppose it would be helpful if the example started out with a plain matlab matrix and then proceeded to go through the steps of how one would upload it into a table. For example, is using javaArray.m necessary? As for the metadata, that's what I meant I suppose. Does the table itself have the capacity to be associated with metadata? Does the table get associated with a "Dataset" (which, as far as i can tell, is strictly limited to images), and should my metadata then be the annotations for this dataset?

As for what I (we) would like to do with it, just use it for storage more or less. We are an imaging group, and so an image database is very useful of course. But our images are derived from sensor data that we acquire from an experiment. So ideally, it would be nice to upload the raw data after an experiment along with any metadata. Then we'd retrieve it later on, generate some images and more processed data, then send that back as a different dataset. And, while I'm aware that OMERO is primarily intended for microscopy images and not sensor data, if it smartly had the capacity for both, I think it would be even more powerful then it currently is. OCT, ultrasound, etc - all come into play.

So any tips would be much appreciated. We could of course store sensor data outside of OMERO, but having a unified platform is important I think to make things simpler.

Clay

[clays]

Also, it would be great if the same type of example could be done with NumPy arrays.

[i.munro]

Hi Clays

If it's just a matter of storage it is possible to store non-image(e'g. txt) files as attachments to images in OMERO.

Best Wishes

Ian

[clays]

It would be nice to be able to look at and process the data in the database later on. So using tables would be ideal.

[wmoore]

Hi,

OMERO.tables is really an API for creating and querying hdf files in OMERO. These tables are not part of the OMERO relational database, although that database is used to store a reference to the OMERO.tables files.

You can use annotations of various types to store any metadata that you have. These include Comment/text, File Annotation, Tag, Map Annotation (ordered list of Key/Value pairs), Long Annotation, etc. You can link annotations to Images or Datasets or even other Annotations.

So, you could create a Dataset and link all the raw table data to that Dataset.
Then subsequently retrieve this data and create images in this Dataset (or create a new one if desired).

Rows in an OMERO.table can be assigned to Images by using an "omero.grid.ImageColumn" instead of a LongColumn for Image IDs, and by convention we attach the OMERO.table as a File Annotation on the parent Dataset. However, there's no sql query that can tell you if there are any tables that have rows relating to a particular Image.

There's a python example at https://docs.openmicroscopy.org/omero/5 ... ero-tables which could be converted to using numpy arrays. Let us know if you need help with this.

Hope that helps,

Will.

PS: I'll refer your Matlab question to others' who have more experience to that...

[wmoore]

Another thought occurred to me:

If your raw sensor data is 2D or 3D arrays of data, this sounds like it should be stored in OMERO as an Image instead of in OMERO.tables?

E.g. an example of creating a new Image in OMERO from a series of numpy arrays can be found at
https://docs.openmicroscopy.org/omero/5 ... eate-image

You can read the Image pixel data back as a numpy array as shown https://docs.openmicroscopy.org/omero/5 ... ata-access

This could also be done in Java/Matlab.

Regards,

Will.

[clays]

Hey Will,

Thanks for your input. A couple things:

"Rows in an OMERO.table can be assigned to Images by using an 'omero.grid.ImageColumn' instead of a LongColumn for Image IDs, and by convention we attach the OMERO.table as a File Annotation on the parent Dataset. However, there's no sql query that can tell you if there are any tables that have rows relating to a particular Image."

I'm a bit confused by this. My notion was to store the raw sensor data for a frame in a table, where the row would correspond to one dimension and the column, the other. I'm guessing then, for a 3-D scan, I'd have to link multiple tables to a dataset with each one corresponding to a frame.

Regarding your second suggestion - just storing the raw data in image files - this is making more and more sense to me. Because the sensor data already has the dimensionality of an image, I suppose one could just consider it image data already. And although the images themselves may not be visually interesting on first glance, others have indicated that this approach can be useful in other ways. For example, I can see utility in merely hovering over a pixel with the mouse corresponding to some point in space and seeing the raw data value. Treating sensor data in the same way as image data also just seems like a simpler way to use the platform.

[manics]

Hi Clays

It sounds like storing your sensor data as a multi-dimensional image is the best option. OMERO.tables uses the underlying HDF5 table API (which is not the same as a HDF5 array). It's designed for "columnar" data along the lines of a typical CSV file, or R or Python Pandas dataframe.

Hope this helps, Simon