Omero 4.4.5 & DropBox

[ppouchin]

Hi,


I upgraded our server this morning to the latest OMERO version, and I was wondering if the DropBox still works the same way.
I thought the previous version did not reimport a file if it was moved inside a user's folder, but this new version does.

Did I have a wrong understanding of the process or has it really changed ?

[cblackburn]

Hi,

the process hasn't changed for some time and certainly not in version 4.4.5.

DropBox is fairly dumb in that it monitors for new image files and then imports them. This means that if you copy in image.dv, then after import remove it, and then later still copy the same image.dv in again it will be imported a second time and so you will have two identical imports. No attempt is made to check whether the file has already been imported.

Due to dependence on the underlying file system notifications, file modifications or file moves within a user's DropBox may also been seen as new files and so may trigger a new import of an existing file. As such a user's DropBox directory isn't currently intended to be an active user file system but a fairly limited post box for triggering imports.

Cheers,

Colin

[ppouchin]

Ok.

I knew that removing/re-copying the image triggered the import system, but I had the impression users could move their files inside their folder. It must have been a wrong impression.

Anyway, thank you for your answer !

[ppouchin]

In fact, I had an other question regarding the "DropBox" feature...

Sometimes, users make it crash, when one of them imports too many files at once.
Is there a way to regulate that ?
Or would there be a way to detect if the process has crashed to automatically restart the server ?

[cblackburn]

Hi,

I knew that removing/re-copying the image triggered the import system, but I had the impression users could move their files inside their folder. It must have been a wrong impression.

It is certainly possible that some moves are missed as being new files while others aren't. As I said before DropBox is a fairly dumb system and we hadn't really envisaged it as a system where users might reorganise their data once it was copied in.

Sometimes, users make it crash, when one of them imports too many files at once.
Is there a way to regulate that ?
Or would there be a way to detect if the process has crashed to automatically restart the server ?

DropBox is only designed to handle files being copied in at acquisition rates. Copying in large numbers of files en masse can certainly cause problems. While there's no quick solution to this, other than asking users to change their behaviour, we are always interested in looking at specific requirements and if possible meeting those requirements. Maybe you could outline your typical use case or your users' typical workflows? For instance, do they use the DropBox folder for files other than acquired image files, i.e. as more general user directories? If it is possible for you to make available a copy of your logs following such a crash along with the DropBox configuration in etc/grid/templates.xml might help us look at whether we could address the problem in our future work.

Colin

[ppouchin]

In fact, my aim is to have users put their files in the DropBox folder after each acquisition. To be more precise, the server runs Samba, and I let them access their home directory (located in the DropBox folder) through the network so they can access their files from anywhere on the network, especially from the microscopes. Theoretically, they should only put images, but it seems some have related files too (xls, imj, ...). I also warned them not to move their files once their are on the server, but sometimes, some want to reorganize their raw data...

The problem is that researchers started to use the server massively only recently... and 4 weeks ago (on the 25/10), a researcher put all his images at once on the server (~50 GB), despite my warnings during the training sessions. (On the bright side, it won't happen again.)

Anyway, I suspect this to be the cause of the DropBox crash that prevented data from being imported in the following days.
When I noticed it, I restarted the server and used a script to detect which files had not been imported and forced them through a generated bash script that moves the incriminated files out of the user's folder and then back in, to trigger the DropBox.
However, I have the same limitation and I have to import files by batch of 10-15, otherwise I make the server crash. (That's why I suspected it to be the root of my troubles...)

Otherwise, DropBox is very robust and I've only witnessed 2 possible sources of crash:

• The user running OMERO does not have the rights to read a file, solved through a modification in path.py - http://trac.openmicroscopy.org.uk/ome/ticket/9156.
• Too many files are being imported at once.

It also seems some MDB files (Zeiss) are not imported properly and one MIGHT have crashed the DropBox, but there was no hard evidence of it. Therefore, I advised users not to put those files on the server. But I should investigate this.

Anyway, if I could just force the DropBox to restart when it crashes, it would save me some time since re-importing the missing images is a long process...

If you want them, I put my logs here (with the templates.xml), but since it happened 4 weeks ago, some data might be missing.

[cblackburn]

Hi,

thanks for the reply and the logs. I've added a ticket to look at this issue:

http://trac.openmicroscopy.org.uk/ome/ticket/9936

It's also good to know that the system is otherwise fairly robust.

If you are having problems with, "MDB files (Zeiss) are not imported properly", it might be worth raising that separately in the forums as given some sample files it may be an issue that can be fixed.

Cheers,

Colin

[ppouchin]

Thanks.

I'll try to look in the MDB files when I have time, and if I find anything, I'll post it in a more appropriate thread.