Open Microscopy Environment

Hi,

some of our users recently wanted to download rather larger amounts of data in multiple files that together exceeded 2 GiB. This will currently fail in OMERO.web because of the 2 GiB file size limit for ZIP files. A solution would be to use 7z or another more modern ZIP format as a second option.

Cheers,

Tristan

Hi Tristan,

Thanks for the feedback and suggestion.
I don't know if you're using OMERO 5.1.2 (or later) but this added support for downloading individual files from web (see https://github.com/openmicroscopy/openm ... /pull/3785).

So, this would allow you to pick individual files, even if the total size of multiple zipped files was > 2 GiB.
This also removes the overhead of creating a zip in the web server, since the files are streamed directly from OMERO so there shouldn't be a file size limit.

I hope that helps?

Cheers,

Will.

Hi Tristan

The Python ZipFile module can be changed to support large files (see the allowZip64 flag https://docs.python.org/2/library/zipfi ... le-objects) but as Will pointed out it imposes a large overhead on the server.

Simon

Hi Will & Simon,

thanks for you quick response. We are using 5.1.3, and yes, downloading individual files from the dialog is certainly a way around the 2 GiB limit. I agree that implementing the option of having extremely large ZIP files generated on the server side would have the potential of creating unsustainable overhead, so the limit is probably a good idea. However, the user needs to be aware of this limitation. There cannot be an option of creating a zip file if the predicted final file size is over 2 GB. Instead, there should be an option of splitting the download into a set of ZIP files (<= 2GiB) and raw data (potentially > 2GiB). These zip files should only be generated the moment the user actually requests the download to limit the maximum overhead.

Also: Multiple file uploads are no problem with current HTML versions, so is there any progress on importing via OMERO.web? I would really like to tell our users that there is no reason any more to use the Java client. Currently, manual upload, ROIs and multiple downloads is the only reason to use Insight.

Cheers,

Tristan

Hi Tristan,

Zip files are currently generated 'on the fly' when the user requests the download, but clearly the user shouldn't be able to request this if the size is over 2 GB.
Thanks for your suggestions on improving the download.
Ticket created: https://trac.openmicroscopy.org/ome/ticket/12980

I'm afraid we are not currently working on web-based import, although we are discussing our next steps for supporting ROI creation/editing in web.

Regards,

Will.

A question that I guess is in the same vein:
We generate a lot of small files (segmented cells and associated data) following image processing in a collaboration with other labs. We get (tens of) thousands of files under multiple formats, and we would like to associate these files to our stacks on our OMERO server. Since the files are small but quite numerous, we zip them. Unfortunately, using OMERO Insight 5.2.0 ice35 b12, uploading for example a 4.2Go Zip always fail.
Is there an alternative way for us to upload this Zip "next" to our data so our colleagues can easily find them?
Thanks for your time.
Philippe

Hi Philippe,

just to clarify: It is not a zip file with images, it's a zip file with various data, and you're trying to upload it as file attachment (not via the importer), right?

Kind Regards,
Dominik

Oops...
Thank you Dominik for your enlightening response. I was indeed out to lunch, trying to upload the zip the "classical" way, via the importer...
Uploading zip files as an attachment does work perfectly.

Ok, good to hear. The importer tries to interpret everything as image file.

Regards,
Dominik