In-place importing as sysadmin

[ratamero]

Hi there!

we're looking into making the many many TB of existing data available to users inside our new OMERO install. Duplication is not a possibility, so in-place importing seems like the right thing to do. If I understood it correctly, the documentation implies we should have an "in-place" user to which people can log in and in-place import for their OMERO account.

That's not feasible in our situation: we're talking about ~100 users, many of which are not computer-savvy and most of which have no idea what SSH is. In addition to that, the in-place user would need access to everyone's data, which means anyone would have access to everyone else's data, which is a major issue. Is there any way for a sysadmin to handle all the in-place importing? A solution where I can import data into anyone's OMERO account would be good enough.

Thanks in advance!

[jmoore]

The functionality you're looking for is generally called "import as". From the CLI, it's accessible via the `--sudo` flag. See https://docs.openmicroscopy.org/omero/5.4.3/users/cli/import.html?highlight=import#advanced-import-commands for a brief example. The operating system user (either the account running OMERO or a dedicated in-place user) will need to have the password for an OMERO admin account.

Cheers,
~Josh

[ratamero]

Thanks! I tested this and it seemed to have gone well - the images were imported correctly (as far as I could see), with correct metadata and everything. I had two issues, still:

1) I used a custom target for importing, something like "regex:+name:^.*accountname/(?<Container1>.*?)". That made every single image have a comment by the user with the text "regex:+name:^.*msrlbl/(?<Container1>.*?)". Is that supposed to be there?

2) After finishing uploading, reading metadata and pixel data, I had the error below. As I said, as far as I could see everything was correctly imported, but I might have missed something!

2018-02-12 11:28:37,396 3472918 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 5370
2018-02-12 11:30:05,887 3561409 [l.Client-2] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /data/omeroimport/msrlbl/160926_CGX261_ChTOGmut_Pericentrin_IF/160926_ChTOGmut_Pericentrin_IF.mvd2
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.RuntimeException: omero.OptimisticLockException
serverStackTrace = "ome.conditions.OptimisticLockException: You are not authorized to change the update event for ome.model.display.Thumbnail:Id_862 from ome.model.meta.Event:Id_60274 to ome.model.meta.Event:Id_60086

[jmoore]

Thanks!

No problem.

I had two issues, still:

1) I used a custom target for importing, something like "regex:+name:^.*accountname/(?<Container1>.*?)". That made every single image have a comment by the user with the text "regex:+name:^.*msrlbl/(?<Container1>.*?)". Is that supposed to be there?

The client code does add an annotation so that the server-process can expand all the variables. At the moment, these annotations are not being removed once processed. Would you consider this a bug?

2) After finishing uploading, reading metadata and pixel data, I had the error below. As I said, as far as I could see everything was correctly imported, but I might have missed something!

2018-02-12 11:28:37,396 3472918 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 5370
2018-02-12 11:30:05,887 3561409 [l.Client-2] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /data/omeroimport/msrlbl/160926_CGX261_ChTOGmut_Pericentrin_IF/160926_ChTOGmut_Pericentrin_IF.mvd2
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.RuntimeException: omero.OptimisticLockException
serverStackTrace = "ome.conditions.OptimisticLockException: You are not authorized to change the update event for ome.model.display.Thumbnail:Id_862 from ome.model.meta.Event:Id_60274 to ome.model.meta.Event:Id_60086

Hmm.... this is surprising. Can you show us the entire command-line you're using to import the images? While the import is ongoing, have you modified the image or dataset via the UI?

~Josh

[ratamero]

The client code does add an annotation so that the server-process can expand all the variables. At the moment, these annotations are not being removed once processed. Would you consider this a bug?

I'd say that, from a user perspective, it can be confusing to have a "weird" comment on your images - especially when you didn't put it there yourself. I wouldn't say it's a bug since it seems to be working as intended, it's just that the intended behaviour is not quite what I would want from it.

Hmm.... this is surprising. Can you show us the entire command-line you're using to import the images? While the import is ongoing, have you modified the image or dataset via the UI?

~Josh

So I created a session with

bin/omero login --sudo root --password <PASSWORD> -s localhost -u <USERNAME> -g <GROUPNAME>

and then created an import with

bin/omero import /data/omeroimport/<USERNAME> -u <USERNAME> -T "regex:+name:^.*<USERNAME>/(?<Container1>.*?)"

I didn't change anything, via UI or CLI, during the import. But now that you mention it, I did cancel a previous import of the same data because of a mistyped username. Could it be that the data that got uploaded in the cancelled import somehow got mixed up with the one from the correct import?

[jmoore]

I'd say that, from a user perspective, it can be confusing to have a "weird" comment on your images - especially when you didn't put it there yourself. I wouldn't say it's a bug since it seems to be working as intended, it's just that the intended behaviour is not quite what I would want from it.

Fair enough. Whether bug or enhancement, I've filed a card regardless: https://trello.com/c/x2QzRaUd/59-cli-remove-nstargettemplate-annotation

I didn't change anything, via UI or CLI, during the import. But now that you mention it, I did cancel a previous import of the same data because of a mistyped username. Could it be that the data that got uploaded in the cancelled import somehow got mixed up with the one from the correct import?

I wouldn't expect a cancelled import to cause such an issue, but perhaps conceivable. I've also recorded this issue but we'd appreciate any more info regarding under what conditions it occurs.

All the best,
~Josh

[ratamero]

Our current setup is two separate servers (CentOS 7) running OMERO.server and OMERO.web separately. The OMERO.server machine mounts an SMB share from our storage server and uses that as its data directory. This issue happened when trying to import data from a different storage server (that is also mounted as an SMB share). I'm not sure whether that helps with diagnosing it or whether there is any other info that could help - please let me know! I'll make sure to note if any similar issues happen again.

On a related note: as the topic name says, my intention is to do in-place importing in that setup. The normal import that gave me that error was only a test to make sure everything else was working fine. However, it turns out that symlinking between the two mount points (the one where the data is and the one where OMERO stores everything) is not possible, so the in-place import data would need to be stored locally on the OMERO.server machine. Is there any way of doing that, while making sure "normal" imports are still stored on one of the SMB shares?

Once more, thank you so much for your time!

[jmoore]

= I'm not sure whether that helps with diagnosing it or whether there is any other info that could help - please let me know! I'll make sure to note if any similar issues happen again.

We'll let you know if we think of anything and do feel free to do the same. Thanks.

However, it turns out that symlinking between the two mount points (the one where the data is and the one where OMERO stores everything) is not possible

How do you mean it's not possible? Can you show us?

so the in-place import data would need to be stored locally on the OMERO.server machine. Is there any way of doing that, while making sure "normal" imports are still stored on one of the SMB shares?

OMERO will always want to create some files under /OMERO. These will either be regular files, soft-links or hard-links depending on your arguments to `bin/omero import --transfer` (e.g. ln_s or ln). You can also symlink other locations into /OMERO and OMERO will continue to use them as if locally present. I imagine between all of that, we can find something that will work for you.

Once more, thank you so much for your time!

Gladly.
~Josh

[ratamero]

How do you mean it's not possible? Can you show us?

hardlinking gave me an "Invalid cross-device link" (which is fair, I guess), and softlinking game me an "failed to create symbolic link: Operation not supported" error. I talked to the storage team in our IT services department, and according to them "creating symlinks on the high capacity storage, across mount points isn't possible".

My very convoluted solution for that at the moment would be:

1) Making omero.managed.dir a local directory instead of a mount;
2) extending the repository, creating a new directory inside omero.managed.dir;
3) mounting the storage server share into the new directory.

Then I could play with omero.fs.repo.path to set whether files are created on the OMERO.server machine (for symlinks) or on the storage server (for normal imports). Does that sound feasible?

[jmoore]

hardlinking gave me an "Invalid cross-device link" (which is fair, I guess),

Yes, that's expected.

softlinking game me an "failed to create symbolic link: Operation not supported" error. I talked to the storage team in our IT services department, and according to them "creating symlinks on the high capacity storage, across mount points isn't possible".

Ah, well, that's fairly definitive. We haven't run into this situation tend to assume that with `unix extensions = yes` symlinks will work. Certainly one to keep in mind.

My very convoluted solution for that at the moment would be:

1) Making omero.managed.dir a local directory instead of a mount;
2) extending the repository, creating a new directory inside omero.managed.dir;
3) mounting the storage server share into the new directory.

We certainly use 1) for the IDR without any problems, but there all imports are in-place.

Then I could play with omero.fs.repo.path to set whether files are created on the OMERO.server machine (for symlinks) or on the storage server (for normal imports). Does that sound feasible?

I'm a bit unsure of what repo.path is going to get you this granularity since you'll need some criterium (group, user) for picking the correct directory. That doesn't mean it's not possible, just that I don't yet have a great suggestion. Hopefully we can find one though.

~Josh