OMERO Bulk import

[alexr]

Hi Josh,
thanks for the ticket opening. I can confirm, that import with other files is working. So it seams to be specific for these files.
I have two additional questions:
Is there a way how I can get the dataset ID back after the bulk import? I mean, if I start the import using the bulk.yaml and filelist.tsv the newly added dataset is given an dataset ID by OMERO server. I then want directly to add tags (and probably key /value pairs) without looking up the dataset ID first.

The next question is related. I figured out that I can add tags by this command:
tag_ann = omero.gateway.TagAnnotationWrapper(conn)
tag_ann.setValue("New Tag")
tag_ann.save()
dataset = conn.getObject("DatasetName", ID)
dataset.linkAnnotation(tag_ann)

But can I give a list of tags / multiple tags in one go?
I tried:
tag_ann.setValue(["New Tag 1","New Tag 2","New Tag 3"])
but get the error that omero.gateway expects a string:
_textValue = <invalid value - expected ::omero::RString>

Is there a direct way to solve that or do I simply implement a loop?

Thanks
Alex

[jmoore]

Hey Alex,

thanks for the ticket opening. I can confirm, that import with other files is working. So it seams to be specific for these files.

Good to hear that other files are working.

Is there a way how I can get the dataset ID back after the bulk import? I mean, if I start the import using the bulk.yaml and filelist.tsv the newly added dataset is given an dataset ID by OMERO server. I then want directly to add tags (and probably key /value pairs) without looking up the dataset ID first.

Sorry, no, the importer doesn't report the ID for the object that got created. (Card created). One option though would be to pre-create the dataset and then use that:

Code: Select all

DATASET=$(bin/omero -q obj new Dataset name:foo)
bin/omero import -T $DATASET my.file


Note, however, that this creates a dataset every time. To re-use a dataset, the command would need to be a bit more complex, first testing if a dataset exists.

But can I give a list of tags / multiple tags in one go?
I tried:
tag_ann.setValue(["New Tag 1","New Tag 2","New Tag 3"])
but get the error that omero.gateway expects a string:
_textValue = <invalid value - expected ::omero::RString>

Is there a direct way to solve that or do I simply implement a loop?

Here you want to create multiple tag_ann instances and link them:

Code: Select all

dataset = conn.getObject("DatasetName", ID)
for x in ("New Tag 1", "New Tag 2"):
    tag_ann = omero.gateway.TagAnnotationWrapper(conn)
    tag_ann.setValue(x)
    tag_ann.save()
    dataset.linkAnnotation(tag_ann)


cheers,
~Josh

[alexr]

Hi Josh,
thanks for the update.
when you suggest:

DATASET=$(bin/omero -q obj new Dataset name:foo)
bin/omero import -T $DATASET my.file

This only works with single file import, but in the bulk.yml import method I give the dataset name in the filelist.tsv, correct?

So to attach tags directly after import I would require to:
1st. import the files to a dataset (dataset name defined in the filelist.tsv)
2nd wait for the import to be finished
3rd get the ID for the dataset name
4th add the tags based on the dataset ID with a loop as suggested.

Now my question is for the 3rd point. How can I get the ID for a given Dataset name? And how can it be handled if multiple users have given identical dataset names, can I filter the datasets by user and then retrieve the ID?
Again I could not find an example for this problem on the omero website / git.
Thanks
Alex

[jmoore]

Hey,

This only works with single file import, but in the bulk.yml import method I give the dataset name in the filelist.tsv, correct?

True. You'd need to encode the ID into the tsv file.

So to attach tags directly after import I would require to:
1st. import the files to a dataset (dataset name defined in the filelist.tsv)
2nd wait for the import to be finished
3rd get the ID for the dataset name
4th add the tags based on the dataset ID with a loop as suggested.

For a dataset created during the import, yes.

Now my question is for the 3rd point. How can I get the ID for a given Dataset name? And how can it be handled if multiple users have given identical dataset names, can I filter the datasets by user and then retrieve the ID?

It's definitely possible to query for a dataset by name and user:

Code: Select all

$ bin/omero -q hql "select d.id from Dataset d where d.details.owner.id = 0 and d.name = 'josh' " --style=plain | cut -f2 -d,
24651


but you aren't sure that this is where your images are. It might be better to ask for the dataset that has one or all of your images:

Code: Select all

$ bin/omero -q hql "select d.id from Dataset d join d.imageLinks dil where dil.child.id = 132202"
# | Col1
---+-------
0 | 24651
(1 row)


Cheers,
~Josh

[alexr]

Thnaks that helped.
Alex

[alexr]

Dear forum members,
after I had my inplace bulk importer working for some time I face a new problem after moving to a freshly installed Omero server with dedicated hardware.
I have again the image files on a file server that is permanently mounted to the Omero server.

The smb share is mounted in /mnt/CCHL-User/ and I checked to have read and write access for both the user that runs the omero server (user:omero) as well as the inplace user (user:inplace).
Also these both users have full access to the managed repository /OMERO/ManagedRepository/...

But when I try to run a bulk import using the two files:

bulki-59892.yml

Code: Select all

---
continue: "true"
transfer: "ln_s"
checksum_algorithm: "File-Size-64"
logprefix: "logs/"
output: "yaml"
path: "/OMERO/ManagedRepository/ipimp-59892.tsv"
columns:
   - target
   - path


ipimp-59892.tsv

Code: Select all

Dataset:name:microtus   /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
Dataset:name:microtus   /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59417.zvi


using the following commands:

Code: Select all

/home/omero/OMERO.server/bin/omero --sudo inplace -w ##### -u "####" login -s 130.###.###.###


followed by

Code: Select all

/home/omero/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki-59892.yml --skip upgrade


I get the message:

Code: Select all

2019-05-07 13:25:05,939 391        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.10-ice36-b105
2019-05-07 13:25:05,962 414        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.9.2 revision: 9fc607f85b8900be786813296f1eee75cc1ed883 date: 31 August 2018
2019-05-07 13:25:06,038 490        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2019-05-07 13:25:06,039 491        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2019-05-07 13:25:06,045 497        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2019-05-07 13:25:06,045 497        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting output format: yaml
2019-05-07 13:25:06,046 498        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2019-05-07 13:25:06,651 1103       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2019-05-07 13:25:08,257 2709       [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 1593ms. (1606ms total) [0 unknowns]
2019-05-07 13:25:08,342 2794       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to 130.###.###.###:4064
2019-05-07 13:25:09,276 3728       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.10
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.10-ice36-b105
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_212
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2019-05-07 13:25:09,781 4233       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.9.0-8-amd64
2019-05-07 13:25:10,477 4929       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:name:microtus
2019-05-07 13:25:10,528 4980       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2019-05-07 13:25:11,541 5993       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2019-05-07 13:25:11,560 6012       [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi...
2019-05-07 13:25:11,678 6130       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
2019-05-07 13:25:11,835 6287       [3-thread-1] WARN   formats.importer.transfers.TransferState - Exception closing b534e6fe-bd64-446d-b239-523a6abfde71/a1f9e735-6c21-45f4-b0f8-c2a0cf2e4b60-RepoRawFileStoreI -t -e 1.0:tcp -h 130.83.98.66 -p 39395 -t 60000
omero.ResourceError: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_212]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_212]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_212]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_212]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_212]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.api.RawFileStorePrxHelper.end_close(RawFileStorePrxHelper.java:1846) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1733) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1720) ~[blitz.jar:na]
   at ome.formats.importer.transfers.TransferState.closeUploader(TransferState.java:252) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:72) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,837 6289       [3-thread-1] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
java.lang.RuntimeException: omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
The server could not check the size of the file:
-----------------------------------------------
/OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
-----------------------------------------------
Most likely the server process has no read access
and therefore in-place import cannot proceed. You
should delete this file manually if you are sure
that the original is safe.

   at ome.formats.importer.transfers.AbstractExecFileTransfer.checkTarget(AbstractExecFileTransfer.java:231) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:66) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,840 6292       [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.RuntimeException: omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
The server could not check the size of the file:
-----------------------------------------------
/OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
-----------------------------------------------
Most likely the server process has no read access
and therefore in-place import cannot proceed. You
should delete this file manually if you are sure
that the original is safe.

   at ome.formats.importer.transfers.AbstractExecFileTransfer.checkTarget(AbstractExecFileTransfer.java:231) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:66) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) ~[blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,842 6294       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Continuing after error

==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:01.907
1 failed imports


and the import fails. I see the softlinks generated in the respective folder, but no images are imported in Omero.
I think there is a problem with file access, but I can not figure out what. The error:
omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
indicates that the server has no permission on the ManagedRepository, or does this refer to the mounted share?
I used ACL to set permissions in the managed repositories and the files I get after import look like this:

Code: Select all

-rw-rw-rw-+  1 omero inplace  27K May  7 13:27 13-27-48.191.log
drwxrwsrwx+  2 omero inplace 4.0K May  7 13:27 13-27-48.191
-rw-rw-rw-+  1 omero inplace  27K May  7 13:27 13-27-47.110.log
drwxrwsrwx+  2 omero inplace 4.0K May  7 13:27 13-27-47.110


And the link file in the directories looks like this

Code: Select all

lrwxrwxrwx   1 inplace inplace  115 May  7 13:25 female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi -> /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi


Which user is in my case the one the requires access if I run the above lines for the import? Is it the inplace or the omero user?

I also checked the server status with
/bin admin diagnostics and get:
OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/home/inplace/omero/tmp' Exists? True Is writable? True (Size: 0)

Thanks for the help,
Alex

[manics]

This could be a simple permissions problem, try sudo-ing to the omero server OS user and checking you can access the symlinked file on the samba share.

However it also sounds very similar to viewtopic.php?f=5&t=8450&p=19691#p19691 If this is the problem then unfortunately there's no definitive solution. Did anything change on the samba server when you switched to the new OMERO server?

[alexr]

Dear mancis,
thanks for the replay. I checked with both
omero and inplace user.
They can access, read and write the smb share. But you were correct. The smb share was also updated. I am currently checking the smb versions and log files to see if there is anything similar to the link you posted.
Thanks
Alex

[alexr]

Dear manics,
I can support the problem described in this previous thread:
viewtopic.php?f=5&t=8450
After I changed my SMB server back to support smb version 1 and changed the etc/fstab on the omero server to mount the share with option vers=1.0 the inplace import is working again.
Thanks for your help. Best wishes
Alex

[manics]

Thanks, it's useful to know it's specific to the Samba protocol version.