OMERO Bulk import
Posted: Wed Jun 13, 2018 6:55 am
Dear omero users,
I am currently trying to setup an automated inplace import from a network share into omero server.
As a first step I try to implement the inplace import using the "bulk.yml" method.
The test scenario is like this:
omero-server:
mounted file-server in: /media/CCHL-User
Omero-ManagedRepository in /OMERO/ManagedRepository
Omero-binaries in /home/omerouser/OMERO.server/bin/omero
the following users are installed on the omero-server:
omero = admin
omerouser = account that runs omero server
inplace = inplace import user, with read/write access to ManagedRepository
Then I generated a bulk.yml file to import my test images. It looks like this:
Together with that I made the file "toImport.tvs" that looks like this:
I then tried to perform the import with the following command, logged in as omerouser:
/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulk.yml
the toImport.tsv is in the same location as the bulk.yml
I also tried the following command:
/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulk.yml
the toImport.tsv is in the same location as the bulk.yml -u 'Alex R'
Again no error message, but no upload either.
Here is the console output:
Only at the end I get the error message: " error: 29 communicators not destroyed during global destruction"
Is there the option to specify the user in the yml file or in the tsv file?
Any help would be great,
thanks
Alex
I am currently trying to setup an automated inplace import from a network share into omero server.
As a first step I try to implement the inplace import using the "bulk.yml" method.
The test scenario is like this:
omero-server:
mounted file-server in: /media/CCHL-User
Omero-ManagedRepository in /OMERO/ManagedRepository
Omero-binaries in /home/omerouser/OMERO.server/bin/omero
the following users are installed on the omero-server:
omero = admin
omerouser = account that runs omero server
inplace = inplace import user, with read/write access to ManagedRepository
Then I generated a bulk.yml file to import my test images. It looks like this:
- Code: Select all
---
continue: "true"
transfer: "ln_s"
exclude: "clientpath"
checksum_algorithm: "File-Size-64"
logprefix: "logs/"
output: "yaml"
path: "toImport.tsv"
columns:
- target
- path
Together with that I made the file "toImport.tvs" that looks like this:
- Code: Select all
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested+filtered01.zvi
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested+filtered02.zvi
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested+filtered03.zvi
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested02.zvi
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested03.zvi
Dataset:name:DermalStem /media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested01.zvi
I then tried to perform the import with the following command, logged in as omerouser:
/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulk.yml
the toImport.tsv is in the same location as the bulk.yml
I also tried the following command:
/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulk.yml
the toImport.tsv is in the same location as the bulk.yml -u 'Alex R'
Again no error message, but no upload either.
Here is the console output:
- Code: Select all
Server: [localhost:4064]
Username: [inplace]Alex R
Password:
Created session c0dc9d22-436e-4841-96ce-9afaa0d341a2 (Alex R@localhost:4064). Idle timeout: 10 min. Current group: system
2018-06-12 14:38:20,038 833 [ main] INFO ome.formats.importer.ImportConfig - OMERO Version: 5.4.5-ice36-b83
2018-06-12 14:38:20,095 890 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 5.7.3 revision: 34c7ca439a617601ce1ac5dcbdcdcfc6f0bfdaf1 date: 11 January 2018
2018-06-12 14:38:20,385 1180 [ main] INFO formats.importer.cli.CommandLineImporter - Adding exclusion: clientpath
2018-06-12 14:38:20,434 1229 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2018-06-12 14:38:20,443 1238 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2018-06-12 14:38:20,453 1248 [ main] INFO formats.importer.cli.CommandLineImporter - Setting output format: yaml
2018-06-12 14:38:20,467 1262 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-06-12 14:38:21,408 2203 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-06-12 14:38:22,130 2925 [ main] WARN del.enums.handlers.CorrectionEnumHandler - Unknown Correction value 'Achroplan' will be stored as "Other"
2018-06-12 14:38:22,491 3286 [ main] INFO ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 1065ms. (1083ms total) [0 unknowns]
2018-06-12 14:38:23,519 4314 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2018-06-12 14:38:24,711 5506 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2018-06-12 14:38:25,547 6342 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.4.5
2018-06-12 14:38:25,548 6343 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.4.5-ice36-b83
2018-06-12 14:38:25,548 6343 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_171
2018-06-12 14:38:25,548 6343 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2018-06-12 14:38:25,548 6343 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2018-06-12 14:38:25,549 6344 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 4.13.0-43-generic
2018-06-12 14:38:26,810 7605 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Dataset:name:DermalStem
2018-06-12 14:38:27,229 8024 [ main] INFO .importer.exclusions.ClientPathExclusion - ClientPath match for filename: media/CCHL-User/Alex/03-Microscopy/2018/2018-06-08-DermalStemCells_dissoiationTest/digested+filtered01.zvi
==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 0 errors in 0:00:01.366
#only for first file, the rest is identical, then:
==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 0 errors in 0:00:03.176
!! 06/12/18 14:39:13.406 error: 29 communicators not destroyed during global destruction.omerouser@comero:/media/CCHL-User$
Only at the end I get the error message: " error: 29 communicators not destroyed during global destruction"
Is there the option to specify the user in the yml file or in the tsv file?
Any help would be great,
thanks
Alex