Making the dropbox facility more useful?

[zebrax0r]

Hi all.

So, I've been thinking a little about the Omero dropbox. I run a highly multi-tenant'ed environment that is predominantly driven via confocal + super-resolution microscopy (STORM/GSD etc).

It's all Active Directory auth'ed.

As of today, I finally got my Omero server joined to LDAP/AD such that my users can all login to it on their own accord with their single sign on names.

We've been considering using the Omero Dropbox + OmeroFS as a means of getting big data from high throughput scopes and imaging equipment back into the Omero data store.

What we can't really conceptualise yet it:

1. How we'd automate such a thing? When a user logs into a workstation attached to an instrument, how do we signal to Omero that it should somehow mount a "share" from the Omero host that happens to be a Dropbox "watched" directory, so that the user can target this location?

2. How would we make it truly multi-tenanted? Every day - 5, 10, 20 different users might login to these devices. How will that play out?

I'd love to hear what others have done, or what they are thinking.

z

[jmoore]

We've been considering using the Omero Dropbox + OmeroFS as a means of getting big data from high throughput scopes and imaging equipment back into the Omero data store.

Using Dropbox via the current state of OMERO5 (i.e. today's beta1 or likely even the upcoming beta2) will still lead to data duplication. You would acquire to the Dropbox directory, Dropbox would upload (i.e. copy) to /OMERO/ManagedRepository and then you could delete from the Dropbox if you wanted.

Use of "in-place import" might make this more useful, but as Dropbox stands, modification of the files would still lead to another import.

1. How we'd automate such a thing? When a user logs into a workstation attached to an instrument, how do we signal to Omero that it should somehow mount a "share" from the Omero host that happens to be a Dropbox "watched" directory, so that the user can target this location?

There's nothing in OMERO for doing this specifically, but there are certainly options. Do you have a particular layout of user directories? (If you're looking to share over NFS be sure to read http://www.openmicroscopy.org/site/support/omero4/sysadmins/windows/server-binary-repository.html#locking-and-remote-shares). My best guess at how this work would be that each user would know what his/her directory is named and would import to that location.

2. How would we make it truly multi-tenanted? Every day - 5, 10, 20 different users might login to these devices. How will that play out?

At the moment, the primary enforced bottleneck will be how many simultaneous imports can run. More simultaneous uploads are possible since these are pretty strict "read-and-write-to-disk" actions, each of which limited to about 64MB. The metadata imports can't have quite large memory requirements. Depending on how much memory you can provide to your OMERO server, you could certainly try upping this thread constraint.

Cheers,
~Josh

[ppouchin]

1. How we'd automate such a thing? When a user logs into a workstation attached to an instrument, how do we signal to Omero that it should somehow mount a "share" from the Omero host that happens to be a Dropbox "watched" directory, so that the user can target this location?

Over here we don't have Active Directory, but we use LDAP and a Samba domain instead.
Practically no one uses Linux here, and Samba is good enough for us for now since the main workstations use Windows (although many scientists use Mac OS X, which also works with Samba).

Since our servers use Debian, users can be authenticated by the system directly through LDAP, but they can easily access their home folders on the servers through Samba (from Windows, Mac OS or even Linux).
These can even be mounted automatically as network drives when domain users log on a domain machine under Windows. Maybe it can be achieved with Mac OS too.

On the OMERO server, I configured the folder containing all the LDAP users' "homes" as the "OMERO DropBox" folder. This can be problematic if a user can connect through SSH and change the access rights, though, as files wouldn't be imported... But overall, it works.

My objective was to make the microscopes join the samba domain and force users to have a different session which would have allowed them to directly put their data in their home folder - or even make a script to automatically do that. At the same time, I would have made sure people couldn't delete other people files while using a common Windows session.

However, switching between session is too time consuming with old software (Leica SP5 or Zeiss LSM 510 programs are slow, as is Windows XP on these machines). Therefore, I've not gone there for now. Unfortunately.

So people still need to manually connect to a share and input their logins before copying files. Although there might be ways around that.

I'm not sure if it answers your questions though...