Immobile data and Omero

[jamesan]

This may be a feature request, but what is the best way to make immobile image data visible in Omero?

Is there a web interface I can provide on my server to make data visible (read-only is OK) on the Omero server without moving any data?

We have a 20TB and 60TB volume which is not practical to duplicate. I was hoping Omero.fs was the solution, but what I’ve read suggests the data may still be moved to the Omero servers binary repository:

"This means that much of the import process can now take place on the server once the original image files have been uploaded."

https://www.openmicroscopy.org/site/sup ... itory.html

My existing HTTP URL interface for image/voxel data is:

Code: Select all

http://{volume}/{channel}?Boundingbox=, Resolution=…

My hope is to add a mapping which matches Omero’s expectations. Then I would import using the volume URL and the data is available on the Omero server.

My motivation at the moment is to learn what Omero can do and hopefully use Omero's data sharing abilities and web viewers without writing my own. Thanks

[wmoore]

It sounds like you want to do something we call "in-place import". Have a look at the docs https://www.openmicroscopy.org/site/sup ... mport.html and let us know if this does what you want.

Once the data has been imported, you can browse data with the web client http://help.openmicroscopy.org/web-client.html or make various views available via the WebGateway web services: https://www.openmicroscopy.org/site/sup ... teway.html

Thanks,

Will.

[jamesan]

Unfortunately I don't feel this is practical in my case.

I may be able to link to tiles registered into the smaller 20TB volume but there are hundreds of millions of them. The volumes are spread across a few NAS boxes. I'd have to look into soft-linking over the file share. For newer volumes I do not calculate registered tiles until they are requested so I can't simply link to a file.

What I'm hoping to find, maybe in a later version, is a URL interface Omero uses between the binary store and server. I like the ideas behind Omero and I'm still figuring out where the modular breaks in the system are because I don't have the resources to go all-in and adopt the entire server stack. It has to be a gradual process where I keep adding compatible interfaces to obtain access to specific features.

[dpwrussell]

Hi,

I'm sympathetic to your need to import data without moving it and it sounds like you also have a strong requirement for retaining direct access through your current system to your data. I have a few questions to help me understand your setup better.

It sounds like an image from your perspective is very big and accessed by looking at one or many tiles, but from OMERO's perspective, it's still 1 logical image?

If you don't generate these tiles up-front, presumably there is some primary data which is then used to generate the correct tiles depending on requests? I assume the primary data is also huge?

Presumably you don't want to import the tiles into OMERO as if they were images in their own right? The tiles are just how you render the data? Is there anything really complicated about this tile generation which requires it be done in your system?

Cheers,

Douglas

[jamesan]

A bit of an aside, this is a long-term goal I don't expect to be addressed right away. I have three use cases for Omero. I am optimistic that I can use Omero for my first two goals.

It sounds like an image from your perspective is very big and accessed by looking at one or many tiles, but from OMERO's perspective, it's still 1 logical image?

From Omero's perspective it would be a single huge image or volume.

To clarify my web URL does not serve tiles. The client asks for a region from a channel at an arbitrary resolution and the requested image or volume is returned. On my end I plan to explore storing the volumes on disk without using image file formats so I intentionally avoided the 2D tile interface.

If you don't generate these tiles up-front, presumably there is some primary data which is then used to generate the correct tiles depending on requests? I assume the primary data is also huge?

The primary data for a single section is ~1000 4096x4096 pixel images. Images overlap 15%. Transforms are generated for each image to create a single mosaic. Another registration transform is generated between adjacent mosaics to create a mosaic to volume transform. Then I generate a final transform to warp each original image into volume space.

An overview of registration:
http://nornir.github.io/overview/overview.html#mosaic-registration

Presumably you don't want to import the tiles into OMERO as if they were images in their own right? The tiles are just how you render the data? Is there anything really complicated about this tile generation which requires it be done in your system?

The tile generation is not complicated. It is possible for me to create a bunch of tiled tiffs and import those. The volume creation is complicated. Even after I finalize the volume registration if I decide to pre-generate the 256x256 tiles it takes weeks or months and the majority of full-res tiles never read. I farm calculations out to a primitive cluster where possible to speed the process.

My impression is that everyone manages their big data differently and the Omero server can't support every data pipeline. Long-term I feel that complementing BIO-Formats with a BIO-URL's initiative is the simplest solution. Interfaces that have reached a critical user mass is the feature I want to add. Putting my data on the server is only a means to that end.

Thanks

[jmoore]

Hi James,

as Will's pointed out, there's not really an out-of-the-box solution, but there are probably several ways to accomplish what you want by extending OMERO and/or Bio-Formats. To suggest one along the lines of your "BIO-URLs" initiative, you could write a Bio-Formats reader for your own type of file which specifies where to do the data loading from. Imagine: test.urldata file (for lack of a better name) which lists:

Code: Select all

base=http://example.com/service
tiles=100
tile_x=256
tile_y=256
tile1=/tiles/tile1
...


There have been proprietary readers which read from commercial DB's and similar, so though you'd to write much of the logic yourself, we know that once done, OMERO5 should be able to import such a ".urldata" file since all IO is re-directed through Bio-Formats. Assuming that's done generically enough, there's no reason others couldn't make use of it to provide access to their URL-based data.

Cheers,
~Josh.

[jamesan]

That sounds like a reasonable conclusion for the current setup. This solution is also a good task for me to work in parallel with adding tiled tiff support to Pillow. I will disappear for a while to read more docs.

Thanks.