Configuring the Managed Repository

[jmoore]

Hi Julia,

I ran into one issue trying out the configuration you're attempting, but with the steps below, I believe it will work for you.

1) temporarily remove the symlink that you created:

Code: Select all

rm /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01


2) Set the following configuration value for "omero.fs.repo.path" where the primary difference is that there's only one "//" in the name.

Code: Select all

bin/omero config set omero.fs.repo.path "Drobo_Ex01/%user%_%userId%//%year%-%month%/%day%/%time%"


3) Restart

Code: Select all

bin/omero admin restart


3) Try a simple import which will re-create the Drobo_Ex01 directory:

Code: Select all

touch /tmp/test.fake
bin/omero import /tmp/test.fake


4) Check that the file exists

Code: Select all

find /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/


5) Backup the contents of the newly created Drobo_Ex01 to the actual Drobo volume:

Code: Select all

mv /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/* /Volumes/Drobo_Ex01/


6) Re-link the Ex01 Drobo volume:

Code: Select all

ln -s /Volumes/Drobo_Ex01/ /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/


7) Re-try the simple import and check the Ex01 volume

Code: Select all

bin/omero import /tmp/test.fake
ls -ltra /Volumes/Drobo_Ex01/


Let us know how things go.
~Josh

[Lipplab]

Dear Josh,

I 've followed your steps.
When I'm creating a file with the terminal and upload it with the current user of the omero server benjaminS, than I'm able to see the file on the new destination.
But when I'm uoploading a file with the omero importer, the file is still uploading to the old destination.

I'm really sorry ..

Julia

Code: Select all

ome-www:bin benjaminS$ omero config get --hide-password
omero.data.dir=/Volumes/Drobo16TB/OMERO_DATA_PIC/
omero.db.name=omeroDB
omero.db.pass=********
omero.db.user=omeroDBuser
omero.fs.repo.path=Drobo_Ex01/%user%_%userId%/%year%-%month%/%day%/%time%
omero.web.application_server=development
omero.web.debug=True


Code: Select all

ome-www:bin benjaminS$ find /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//.DS_Store
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/.DS_Store
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/.DS_Store
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/17
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/17/.DS_Store
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/17/17-48-05.532
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/17/17-48-05.532/test.fake
/Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01//benjaminS_13/2016-12/17/17-48-05.532.log
ome-www:bin benjaminS$ mv /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/* /Volumes/Drobo_Ex01/
ome-www:bin benjaminS$ ln -s /Volumes/Drobo_Ex01/ /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/
ln: /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository//Drobo_Ex01: File exists
ome-www:bin benjaminS$ ln -s /Volumes/Drobo_Ex01/ /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/
ln: /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository//Drobo_Ex01: File exists
ome-www:bin benjaminS$ omero import /tmp/test.fake
Using session d309bf70-42ba-4c6a-9feb-3893e569f47e (benjaminS@134.96.155.200:4064). Idle timeout: 10 min. Current group: LIPPLAB
2016-12-17 17:54:32,608 263        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.2.5-ice35
2016-12-17 17:54:32,624 279        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.1.9 revision: c3065feb775a7a8c0cc2cf6e35979331cca2418b date: 14 April 2016
2016-12-17 17:54:32,663 318        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2016-12-17 17:54:32,993 648        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2016-12-17 17:54:33,055 710        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 60ms. (62ms total) [0 unknowns]
2016-12-17 17:54:33,448 1103       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to 134.96.155.200:4064
2016-12-17 17:54:33,966 1621       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.2.5
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.2.5-ice35
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_55
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2016-12-17 17:54:34,297 1952       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 10.10.5
2016-12-17 17:54:34,393 2048       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2016-12-17 17:54:34,827 2482       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2016-12-17 17:54:34,836 2491       [      main] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /tmp/test.fake...
2016-12-17 17:54:34,869 2524       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /tmp/test.fake
2016-12-17 17:54:34,891 2546       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /tmp/test.fake
2016-12-17 17:54:34,956 2611       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2016-12-17 17:54:35,059 2714       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 113576
2016-12-17 17:54:35,200 2855       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 113576
2016-12-17 17:54:35,384 3039       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 113576
2016-12-17 17:54:35,598 3253       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 113576
2016-12-17 17:54:35,622 3277       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 113576
2016-12-17 17:54:35,642 3297       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 113576
2016-12-17 17:54:35,775 3430       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /tmp/test.fake
Imported pixels:
111334
Other imported objects:
Fileset:18852
Image:111384
2016-12-17 17:54:35,776 3431       [l.Client-1] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0

==> Summary
1 file uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:01.390
ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo_Ex01/
total 32
drwx------   5 benjaminS  staff   170 14 Okt  2015 .Spotlight-V100
d-wx-wx-wt   3 benjaminS  staff   102  6 Dez 11:09 .Trashes
-rw-r--r--   1 benjaminS  staff     0  8 Dez 16:11 test.txt
drwx------   3 benjaminS  staff   102 12 Dez 14:46 .fseventsd
drwxrwxrwt@  9 root       admin   306 17 Dez 05:41 ..
-rw-r--r--@  1 benjaminS  staff  8196 17 Dez 17:37 .DS_Store
drwxr-xr-x   4 benjaminS  staff   136 17 Dez 17:48 benjaminS_13
drwxrwxr-x  10 benjaminS  staff   408 17 Dez 17:51 .
ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo_Ex01/

Code: Select all

ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01/
total 32
drwxr-xr-x  14 benjaminS  staff   476 17 Dez 17:48 ..
drwxr-xr-x   5 benjaminS  staff   170 17 Dez 17:56 .
drwxr-xr-x   4 benjaminS  staff   136 17 Dez 17:59 juliaS_368
drwxr-xr-x   4 benjaminS  staff   136 17 Dez 18:00 benjaminS_13
-rw-r--r--@  1 benjaminS  staff  6148 17 Dez 18:00 .DS_Store
ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo_Ex01/
total 32
drwx------   5 benjaminS  staff   170 14 Okt  2015 .Spotlight-V100
d-wx-wx-wt   3 benjaminS  staff   102  6 Dez 11:09 .Trashes
-rw-r--r--   1 benjaminS  staff     0  8 Dez 16:11 test.txt
drwx------   3 benjaminS  staff   102 12 Dez 14:46 .fseventsd
drwxrwxrwt@  9 root       admin   306 17 Dez 05:41 ..
drwxr-xr-x   4 benjaminS  staff   136 17 Dez 17:48 benjaminS_13
drwxrwxr-x  10 benjaminS  staff   408 17 Dez 17:51 .
-rw-r--r--@  1 benjaminS  staff  8196 17 Dez 17:56 .DS_Store


[jmoore]

Hi Julia,

the first thing I notice is that the config doesn't match exactly:

Code: Select all

omero.fs.repo.path=Drobo_Ex01/%user%_%userId%//%year%-%month%/%day%/%time%
omero.fs.repo.path=Drobo_Ex01/%user%_%userId%/%year%-%month%/%day%/%time%


I'm not sure that alone will explain what you're seeing, but we shouldn't try anything else until that's sorted. Unfortunately, the incorrect config will have created an entry in your database. If you can think of a name other than Drobo_Ex01 to use, then I'd suggest you try a fresh name. If you definitely want the name to be "Drobo_Ex01" then I'll help modify your DB.

But when I'm uploading a file with the omero importer, the file is still uploading to the old destination.

Can you show me the output of

Code: Select all

ls -ltra /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository


?

Thanks.
~Josh

[Lipplab]

Dear Josh,

I've changed the config. Is it ok, now?

Code: Select all

ome-www:bin benjaminS$ omero config get --hide-password
omero.data.dir=/Volumes/Drobo16TB/OMERO_DATA_PIC/
omero.db.name=omeroDB
omero.db.pass=********
omero.db.user=omeroDBuser
omero.fs.repo.path=Drobo_Ex01/%user%_%userId%//%year%-%month%/%day%/%time%
omero.web.application_server=development
omero.web.debug=True
ome-www:bin benjaminS$


The outpu of the -ls ltra command is.
I guess the link is missing.

Does it help just to create a new folder named like "Data_extent" on Drobo_Ex01? with the path Drobo_Ex01/Data_Extent?

Code: Select all

ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository
total 32
drwxr-xr-x   3 benjaminS  staff    102  5 Feb  2014 .omero
drwxr-xr-x   5 benjaminS  staff    170 19 Mai  2014 benjaminS_13
drwxr-xr-x   3 benjaminS  staff    102  7 Jul  2014 larsK_4
drwxr-xr-x   5 benjaminS  staff    170  1 Mai  2015 xinH_10
drwxr-xr-x   3 benjaminS  staff    102 21 Dez  2015 christinaT_502
drwxr-xr-x   5 benjaminS  staff    170 25 Feb  2016 uliW_454
drwxr-xr-x  17 benjaminS  staff    578  1 Aug 13:54 qinghaiT_20
drwxr-xr-x  19 benjaminS  staff    646  4 Aug 10:18 peterL_5
drwxr-xr-x   4 benjaminS  staff    136 26 Okt 11:47 elisabethK_307
drwxr-xr-x  11 benjaminS  staff    374  8 Dez 14:21 ..
drwxr-xr-x   9 benjaminS  staff    306  8 Dez 14:34 juliaS_368
drwxr-xr-x  14 benjaminS  staff    476 17 Dez 17:48 .
drwxr-xr-x   5 benjaminS  staff    170 17 Dez 17:56 Drobo_Ex01
-rw-r--r--@  1 benjaminS  staff  12292 17 Dez 18:00 .DS_Store
ome-www:bin benjaminS$


MAny Thanks Julia

[jmoore]

Hi Julia,

Code: Select all

omero.fs.repo.path=Drobo_Ex01/%user%_%userId%//%year%-%month%/%day%/%time%
omero.fs.repo.path=Drobo_Ex01/%user%_%userId%//%year%-%month%/%day%/%time%


I've changed the config. Is it ok, now?

Looks good.

The outpu of the -ls ltra command is.
I guess the link is missing.

Did something go wrong when you tried to move Drobo_Ex01 out of the way?

Does it help just to create a new folder named like "Data_extent" on Drobo_Ex01? with the path Drobo_Ex01/Data_Extent?

Let's get the symlink in place and then handle the new folder name next.

All the best,
~Josh

[Lipplab]

Dear Josh,

I removed the Drobo_Ex01 folder on Drono16TB.

I've made a new symlink

Code: Select all

ln -s /Volumes/Drobo_Ex01/ /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/

I've uploaded a new test file.

Code: Select all

ome-www:bin benjaminS$ touch /tmp/test.fake
ome-www:bin benjaminS$ omero import /tmp/test.fake
...
Other imported objects:
Fileset:19051
Image:111583
2016-12-19 11:26:32,817 5821       [l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0

==> Summary
1 file uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:03.584


And I'm testing the upload with the omero importer. At the moment , the upload is very low. It's due to the Drobo and it's file protecting because of one missing drive.
But it looks good. The file seems to be uploaded on the new destination.
Hopefully i'll get the new harddrive for the Drobo today. have a faster tranfering rate.

Thanks a lot
Julia

[Lipplab]

and the link seems to be fine now.

Code: Select all

ome-www:bin benjaminS$ ls -ltra /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01
lrwxr-xr-x  1 benjaminS  staff  20 19 Dez 11:24 /Volumes/Drobo16TB/OMERO_DATA_PIC/ManagedRepository/Drobo_Ex01 -> /Volumes/Drobo_Ex01/

Just in case, should I check the DataBase?

I've tried to create a new folder on the Drobo_Ex01. But It doesn't work to make this new directory as a new omero.fs.repo.path, although I've made a symlink.

But as long it works this way, it's fine.

Many thanks Julia

[jmoore]

Glad to hear it, Julia. Regarding the other issue (renaming Drobo_Ex1), as long as you importing from multiple users is working, we may not need to take further action. If you see any issues, though, please let us know and we'll help you to adjust things.

All the best,
~Josh.