Using Omero with S3 File System

[pmann]

We are trying out Omero, but the images we want to host are quite large (~500 Mb each). We would like to host the images on S3, but Omero requires a file-system to store the images in. Would it be possible to use something like s3fs to link the folder where Omero stores the images to S3? If not, them I'm curious about what other people have done to get around this. Is the solution just to get more disc storage?

Thanks for the help.

[manics]

Hi

Officially OMERO only supports POSIX file systems. The Image Data Repository has around 90TB of files including several multi-gigabyte images so size isn't a problem as far as OMERO is concerned.

BioFormats 6.0.0 adds preliminary support for reading off S3. For instance if you download the BioFormats command line tools you can use the showinf command to open an S3 image:

Code: Select all

./bftools/showinf s3://s3.amazonaws.com/bucket-name/path/to/image.jpg


If you try this one obvious issue is the latency in opening and viewing images. There's an optional cache included in the S3 reader but that's not a full solution, for example it doesn't automatically refresh, nor is there a size limit.

Getting this into OMERO will require a bit of work as the current release 5.4.10 uses BioFormats 5.9.2.
In addition you'll need to compile and include this reader which is a wrapper to store the URL https://gitlab.com/openmicroscopy/incub ... urlreader/

In the current architecture OMERO always requires a local file on disk, which is provided by this wrapper that stores the URL and some options. The S3 file is then streamed remotely.

I've got an OMERO.server built with an older version of the S3 work running, so we know it's possible. It's definitely not suitable for production use. Supporting object stores in OMERO is in our long term plans, so if you're interested in trying it out and giving feedback on how it should work your input would be very welcome!

I haven't tried S3fs, but I suspect you'd run into the same problems with latency. If you're also writing to S3fs through OMERO it's possible you'll have problems with file locks. If you try it I'm sure others would be interested to know whether it works.

Simon

[pmann]

Hey Simon,

Thank you for responding to my question. It's great that S3 support is a long term goal for Omero.

If the S3FS idea doesn't work out, then another option we're thinking about is to archive images into S3 and then delete them from Omero after a regular amount of time or when our capacity gets filled up.

I'll post here again if the S3FS solution performs well but I'm expecting that it will cause a lot of problems.

Thanks again,
Philip