Large export

[evenhuis]

Hi,

I've been running into problems with export of multiple files hitting the Zip64 limit. https://trac.openmicroscopy.org/ome/ticket/12691

People have been downloading the original files individually by clicking on eacb file listed below the Zip file option.

Is there a way to either:
1. enable the Zip64
2. to disable Zip file create and just download the list of files uncompressed (preferred)

Thanks,

Chris

[wmoore]

Hi Chris,

We know a number of our users have been requesting better downloading of large data volumes.
However, most users seem to run into web server timeout issues while building the zip and don't reach the zip size limitation.

Are you generally trying to zip a small number of large files (which maybe reaches the size limit quite quickly, before the webserver times out - typically 1 minute)?
Any info on your typical usage that would help us reproduce or test this would be great. Size & number of files where you see this problem, what is the error you see, web-server timout etc.

We're going to look into this, with one potential option to use tar file that can be streamed directly instead of waiting for all the files to be assembled into a zip.
See https://trello.com/c/CkOEJucx/127-web-d ... bigger-zip

Thanks for your feedback,

Will.

[evenhuis]

Hi Will,

We have seen timeouts too, changing the settings nginx has helped with this but WiFi connections are still flakey. Also, big zip files can lock the web server up. Downloading through OMERO.insight is nicer as it avoids the web related restrictions and lets users save to specific locations rather than filling up their downloads folder.

As for our usage, our aim is get our users to upload their image data directly from the aquistion computers. By avoiding the current transfer to USB drives,
• imaging data will be catalogued, managed and backed up,
• we avoid possible USB malware.

Many of our users go on to process raw images in imaris or Huygens, or have exisiting workflows that are based around files sitting on a file system. So I’ve been looking for a way to easily get dataset-level data exported from OMERO.

Z-stacks and time series easily exceed 2GB so zipping seem to out of the question.
Loss of metadata from OME-tif is another problem we’re having.
I’ve had some success downloading original files and one-tif in the python scripts, then stuffing the missing metadata into the tifs with the bftools commandline tools. It’s rube-Goldberg solution that is not for normal users.

We have been viewing data flows into and out of OMERO. Should we be see omero as the final step rather than the first?

Thanks,

Chris

[wmoore]

Hi Chris,

Thanks for the feedback.
Certainly the issue of "locking-up" all the resources of the web server if there are multiple users downloading is a big problem that won't be solved by different zip or tar-streaming strategies.

We have started exploring the usage of micro-services for some features (rendering images/thumbnails etc) and this strategy might be extended to downloads, so they don't impact on the main OMERO.web. However this will not be a solution in the short term.

As you said, Insight does offer some improvements over the webclient but we'll look at improving webclient downloads if possible as mentioned above.

With respect to the Metadata export, it should be possible to improve the metadata exported in OME-TIFF Please let us know what specific metadata is missing.

If the access to this metadata can be done via ImageJ (as you were asking about at viewtopic.php?f=4&t=8451&p=19063) then this is likely to be a quicker solution. Please let us know if we can help more on that topic.

Regards,

Will.

[evenhuis]

Hi Will,

I've uploaded an .ndf2 file to OMERO then three versions of the file in FIJI there ways.
1. the FIJI-OMERO plugin
2. Exported the OME-TIF from OMERO
3. Opening the raw file Bioformats

The attachment compares the three metadata. The files coming from OMERO are the same, they both have 42 lines of metadata which seems to be the minimum metadata to define the tif (dimensions, channels, frames etc). The mapannotations I added in OMERO are not exported either.

The nd2 file has 926 lines of metadata. The entries I find use in scripts include the filters on the channels, the timestamps for time lapses, the stage coordinates etc.

I've had a look through the python bindings for OME-TIF. I can create the OME-TIF but I can't find out how to add metadata, it can find some java API for though.

Thanks,

Chris