DropBox - what does "en masse" mean?

[Lachie]

Hi,

We are looking at setting up Drop Box, but notice the phrase "The DropBox system is designed for image files to be copied in at normal acquisition rates. Copying many files en masse may result in files failing to import."

(from: https://www.openmicroscopy.org/site/sup ... opbox.html)


What constitutes en masse?

Most of the instrument makers recommend you acquire to a local disk (or are set up that way) so you end up copying or moving your files to the network/SAN/whatever in batch mode anyway...

How many files/much data can we safely move without issues?

[cblackburn]

Hi,

What constitutes en masse?

We have in the past seen problems with missed files and failed import when file sets have been copied in bulk. However, it's very difficult to put a figure on this as it will depend on the hardware, network and copy semantics.

Most of the instrument makers recommend you acquire to a local disk (or are set up that way) so you end up copying or moving your files to the network/SAN/whatever in batch mode anyway...

How many files/much data can we safely move without issues?

We would recommend transferring each acquired file set as it is available if possible. Or transferring file set by file set later.

What sort of numbers of files/amount of data would you be expecting for your system?

Cheers,

Colin

[Lachie]

Hi Colin,

Thanks for the response. It's hard to put numbers on my estimates too. Basically the workflow of our users is to spend a few hours on a microscope, saving files locally as they go. Then at the end of the session they'll copy across our network to their user share or to a general purpose share which they'll then copy somewhere else.

As you can imagine, I like the idea that they just send it to their OMERO dropbox, but if they've acquired 100 files (which is probably on the upper end of how many they'd get) I was wondering if it would fail or not. Chances are when I get it configured I'll just try it and see but was wondering what your thoughts would be.

Thanks,
-Lachie

[cblackburn]

Hi Lachie,

We don't have any hard and fast figures due to the great variability in the criteria I mentioned. We know that we can create missed file events by copying large numbers of small files at the same time. We have some code that tries to mitigate this. In this thread:

viewtopic.php?f=4&t=3376&p=7672&hilit=+en+masse+#p7667

an OMERO admin cites an example of many files adding up to 50GB being copied in one go. In this case it was partially mitigated by user training.

If you do find specific situations cause problems with missed or broken imports than please do get back to us. Also, if you find a workflow that works for your users then any data would be very useful.

Cheers,

Colin