Error on import: Directory exists but is not registered

[ratamero]

Dear all,

we're having a big issue here - after a few crashed imports, we're getting a "Error on import: Directory exists but is not registered: CheckedPath(petabyte)" error whenever trying to import anything. I've found a couple of previous cases where the same error was caused by an empty directory inside ManagedRepository (in our case, we would expect it to be the "petabyte" directory), and deleting the empty directory restored it.

Now here's the problem: We're using a custom omero.fs.repo.path that looks something like "petabyte/%groupId%/%user%_%userId%//%year%-%month%/%day%/%time%". So all our data is inside the "petabyte" folder (which is actually just a symlink to a mounted storage server share). It's not an empty directory - very much the opposite, in fact.

I've tried removing the symlink and importing data - works normally and creates a "petabyte" directory, as expected. However, "petabyte" has always been a symlink to the external storage server, and all our data is already sitting there. I'm trying to chase down the data from the crashed imports and thought maybe deleting it will sort it, but let me know if you have any other ideas. Worst case scenario is having to restore the server to its previous backup status.

best regards

Erick

[mtbc]

Dear Erick,

What does "bin/omero fs repos" report? Specifically, I am wondering if there's some wrong duplication there.

In general before making any manual change like adjusting directories under the server I would recommend a full backup in case the resulting behavior is to effect some troublesome state changes from which it is difficult to recover. Also be aware that we offer OMERO Docker images in which you might experiment to test the effects of various server/FS changes or recovery procedures in a safe place.

I wonder what originally happened to cause this problem. Could there have been some glitch in your networked storage that perhaps OMERO.server should have handled better? Nothing else is known to have changed that might have caused the problem?

Cheers,
Mark

[ratamero]

I think it might have run out of memory (I was running multiple imports in parallel when it happened). Another possibility is that it might have been a consequence of the server crashing when I tried to view a Slidebook file (Slidebook6Reader just loves to crash our server).

You're right, there's some strange duplication on the repos:

Code: Select all

# | Id     | UUID                                 | Type    | Path                                                 
---+--------+--------------------------------------+---------+------------------------------------------------------
0 | 57     | ee6b9e6a-a545-4269-916f-f65d89f01711 | Public  | /data/omerodata                                     
1 | 58     | ScriptRepo                           | Script  | /home/admin/OMERO.server-5.4.6-ice36-b87/lib/scripts
2 | 59     | e4910386-fcb6-4f3e-934b-d8e55c022f45 | Public  | /data/omerodata/ManagedRepository                   
3 | 2051   | 8f452952-9c48-49e1-981d-28d2a0659441 | Public  | /data/omerodata                                     
4 | 2052   | a552ca72-9238-4e8b-939e-ded1701e0ca9 | Public  | /data/omerodata/ManagedRepository                   
5 | 211101 | 958ce441-ce74-403e-a412-88e923764264 | Managed | /data/omerodata/ManagedRepository 

So what's the safe way to fix this?

[mtbc]

Dear Erick,

Out of curiosity, how is your external storage server mounted? Is it NFS v3 or somesuch? I wonder if at some point the storage disappeared under the server.

What are the contents of the binary repository's repo_uuid files? (There is probably one in .omero/repository/*/repo_uuid and in ManagedRepository/.omero/repository/*/repo_uuid.)

Which managed repository has the bulk of your data? Something like,

Code: Select all

psql -c "SELECT repo, COUNT(*) FROM originalfile WHERE repo IS NOT NULL GROUP BY repo ORDER BY COUNT(*) DESC" -U db_user omero_database


Also, I wonder about the petabyte directories now registered with OMERO,

Code: Select all

psql -c "SELECT id, path, repo FROM originalfile WHERE name = 'petabyte'" -U db_user omero_database


As background, you'll want the repo_uuid files to match what's in the repo column for the originalfile database table rows that list the image data that you actually want to keep.

Cheers,
Mark

[ratamero]

it's an SMB3 mount, IIRC. I'm pretty sure it hasn't disappeared at any given point.

I have /data/omerodata/.omero/repository/0f9bd6b7-c89d-4414-a13e-047ceaa85d4f/repo_uuid with the contents

Code: Select all

^@$8f452952-9c48-49e1-981d-28d2a0659441

and /data/omerodata/ManagedRepository/.omero/repository/0f9bd6b7-c89d-4414-a13e-047ceaa85d4f/repo_uuid with contents

Code: Select all

^@$958ce441-ce74-403e-a412-88e923764264

running the first postgres command have me

Code: Select all

                 repo                 | count 
--------------------------------------+--------
a552ca72-9238-4e8b-939e-ded1701e0ca9 | 193228
e4910386-fcb6-4f3e-934b-d8e55c022f45 |    234
ScriptRepo                           |    150
958ce441-ce74-403e-a412-88e923764264 |      6


and the second one gave me

Code: Select all

   id   | path |                 repo                 
--------+------+--------------------------------------
   2071 | /    | a552ca72-9238-4e8b-939e-ded1701e0ca9
211423 | /    | 958ce441-ce74-403e-a412-88e923764264


Thanks in advance!

[ratamero]

Quick update: I think it's sorted! I edited the repo_uuid files to correspond to the repos I wanted and it seems to be back to normal after a restart. I'll report back if it breaks again

[mtbc]

Great, thank you very much for letting us know, Erick. At some later point we may come back to this thread with further questions: we'd definitely like to help prevent users from getting into these states. If you notice any further problems or strangeness then indeed please don't hesitate to let us know the new state of things and we will try to help you sort them out.

Cheers,
Mark

[ratamero]

No worries. Thanks again, Mark!