Minor documentation flaw for in-place import

[dsudar]

Hi all,

Just starting to see how to use in-place importing. I noticed one small flaw in the documentation on: https://www.openmicroscopy.org/site/sup ... mport.html

When following the recipe how you set things up at Dundee, it may be useful to specify also that the umask for the omero_system_user needs to be 002. Otherwise the directories in ManagedRepository are created without group write and thus the inplace_user cannot write into them.

Inplace import is blazing fast and no more duplication of storage! Thanks!

Now I'm trying to figure out how to prevent folks from deleting or modifying the files that are now hard-linked but still give them read access and ability to drop analysis results and other files in source directory. Would you recommend a cron script to make all image files a different owner and read-only while keeping the directories open for write? Or is there a better way?

Thanks,
- Damir

[jmoore]

Hi Damir,

Just starting to see how to use in-place importing. I noticed one small flaw in the documentation on: https://www.openmicroscopy.org/site/sup ... mport.html

When following the recipe how you set things up at Dundee, it may be useful to specify also that the umask for the omero_system_user needs to be 002. Otherwise the directories in ManagedRepository are created without group write and thus the inplace_user cannot write into them.

I assume that means that this line isn't enough?

Code: Select all

[inplace_user@ome-server ~]$ umask
0002


What value do you have and where would you have looked? In the text proceeding the code block?

Inplace import is blazing fast and no more duplication of storage! Thanks!

Very glad to hear it!

Now I'm trying to figure out how to prevent folks from deleting or modifying the files that are now hard-linked but still give them read access and ability to drop analysis results and other files in source directory.

Would you recommend a cron script to make all image files a different owner and read-only while keeping the directories open for write? Or is there a better way?

The permissions on the source files will match that of the target files, so I'd be most inclined to try to have the user/permissions set *before* importing. A cron job that protects either all your source data or all the files (matching some query perhaps?) in the ManagedRepository would certainly also work. Another option would be to extend one of the existing `FileTransfer` methods to include a `chmod` on the data. This was discussed for inclusion previously, but we didn't feel we had enough feedback with the community and experience on whether to include this in the mainline, but if it works for your site, feel free!

All the best,
~Josh.

[dsudar]

Hi Josh,

That umask line relates to the "inplace_user" and is indeed also needed but as far as I saw when running it, the directory structure in ManagedRepository is being created by "omero_system_user" so its umask also needs to be 0002. Is that not the case?

Ah yes, extending the FileTransfer method would be a good place where to do a more precise chmod on only the actual files that omero imported and leave the directories and other files alone. Thanks, I'll look how to do that. If you have an example laying around, even better ....

Thanks,
Damir

[jmoore]

That umask line relates to the "inplace_user" and is indeed also needed but as far as I saw when running it, the directory structure in ManagedRepository is being created by "omero_system_user" so its umask also needs to be 0002. Is that not the case?

Ah, fair enough. The umask for that user is 002. Thanks for the heads up.

Ah yes, extending the FileTransfer method would be a good place where to do a more precise chmod on only the actual files that omero imported and leave the directories and other files alone. Thanks, I'll look how to do that. If you have an example laying around, even better ....

https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/blitz/src/ome/formats/importer/transfers/MoveFileTransfer.java might be a good class to build on. Rather than deleting the source of the hard-link, you could chmod it. There's not currently an example of doing this. Just keep in mind that the new class needs to be available to the client classpath ($OMERO_CURRENT/lib/client) as opposed to the server lib.

Cheers,
~Josh.

[mtbc]

In terms of protecting data on UNIX-like systems, do note that permissions on the containing directory matter:

Code: Select all

$ ls -al
total 12
drwxr-xr-x 2 mtbc mtbc 4096 Apr 27 13:13 .
drwxr-xr-x 3 mtbc mtbc 4096 Apr 27 13:13 ..
-r-------- 1 mtbc mtbc    5 Apr 27 13:13 data
$ rm -f data
$ ls -al
total 8
drwxr-xr-x 2 mtbc mtbc 4096 Apr 27 13:14 .
drwxr-xr-x 3 mtbc mtbc 4096 Apr 27 13:13 ..
$ touch data
$ chmod 555 .
$ chmod 666 data
$ ls -al
total 8
dr-xr-xr-x 2 mtbc mtbc 4096 Apr 27 13:15 .
drwxr-xr-x 3 mtbc mtbc 4096 Apr 27 13:13 ..
-rw-rw-rw- 1 mtbc mtbc    0 Apr 27 13:15 data
$ rm -f data
rm: cannot remove ‘data’: Permission denied
$


Cheers,
Mark

[dsudar]

Hi Mark,

Yes, very good point. I'll have to be pretty careful at how the permissions on that whole directory structure are set.

Just to give you an idea what I'm trying to achieve:
- folks upload the entire datasets using Samba to the server in to /data/share. For a single plate this is a 2-level directory structure with a number of files required by ScanR software in specific places.
- OMERO uses the same /data/OMERO filesystem for ManagedRepository etc.
- I want to somehow (ideally automatically - yet to be figured out how) import that data into OMERO using hard links so there's no data copying and the folks don't see anything different from when they uploaded.
- folks can still use their Olympus ScanR software to do their analysis on the data in /data/share and drop their analysis results into that same directory structure the way the ScanR software likes to do things
- meanwhile OMERO has full access to all the image data but it needs to be protected from folks doing weird stuff

Thanks,
- Damir

[dsudar]

Hi all,

I just bumped into another small issue with "hard link" inplace imports mostly related to documentation about it. On CentOS 6 (and probably all derivatives of RHEL6), the "inplace_user" ONLY needs read access to the source files in question and the "ln" command will create the hard link with the original owner of that file as the owner. Very convenient but ..... on Ubuntu 14.04 and probably a bunch of other Linuces, "inplace_user" will need write access to the source files to make the hard link. The documentation says that only read access is needed so that should probably modified to explain differences between flavors of Linux/Unix.

Overall, inplace import is a great functionality.

Thanks,
- Damir

[manics]

Hi Damir

What umasks do you have for the in_place user on Centos and Ubuntu?

Simon

[dsudar]

Hi Simon,

Umask is 002 on both Ubuntu14.04 and CentOS6, both for the inplace_user and the omero_user (i.e. the system account that runs the omero server).

I think this issue with slightly different behavior of permissions for hard links has something to do with a relatively recent add to the kernel called "protected_hardlinks" (see: http://www.openwall.com/lists/kernel-ha ... 2/02/21/20) but this is way over my head.

Cheers,
- Damir

[manics]

Hi Damir

Thanks very much for looking into the problem. We'll look into it and update our docs.

Simon