Cleanse issue with 5.2.7

[jacques2020]

Hi,

when calling cleanse in omero 5.2.7 on debian 8 up to date, I get the error reproduced below. It is normally called after backup using the line

Code: Select all

sudo -u cedre-agent $OMERO_HOME/bin/omero -s localhost -u $USERNAME -w "$PASSWORD" -g CeDRE admin cleanse $BINARY_REPOSITORY

cedre-agent is the user running the server and owning omero files. OMERO_HOME is the path (contains a symlink).
I don't see any additional useful indication. Please ask me !

Thank you very much for your help

Best

Jacques

Code: Select all

Created session d6e3cd4c-7c1c-4fc9-aaed-6a6d18e40be5 (root@localhost:4064). Idle timeout: 60 min. Current group: CeDRE
Traceback (most recent call last):
  File "/usr2/local/OMERO/OMERO.server-Current//bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1438, in argv
    cli.invoke(args[1:])
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 952, in invoke
    stop = self.onecmd(line, previous_args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1029, in onecmd
    self.execute(line, previous_args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1111, in execute
    args.func(args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 595, in _check_admin
    return func(*args, **kwargs)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/plugins/admin.py", line 1796, in cleanse
    dry_run=args.dry_run)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 250, in cleanse
    delete_empty_dirs(proxy, root, client, dry_run)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 258, in delete_empty_dirs
    is_empty_dir(repo, '/', False, to_delete)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 285, in is_empty_dir
    for entry in repo.listFiles(directory):
AttributeError: 'NoneType' object has no attribute 'listFiles'

[mtbc]

Dear Jacques,

What output do you get from

Code: Select all

bin/omero fs repos

Cheers,
Mark

[jacques2020]

Dear Mark,

here is the output of fs repos, ran as root.

Code: Select all

Created session 8457d830-a219-4502-a43d-4c7a425f6727 (root@localhost:4064). Idle timeout: 60 min. Current group: system
# | Id      | UUID                                 | Type    | Path
---+---------+--------------------------------------+---------+-----------------------------------------------------------
0 | 12      | 2db7d60a-3884-40b6-990d-f89571e95497 | Public  | /OMERO
1 | 13      | ScriptRepo                           | Script  | /usr/local/OMERO/OMERO.server-5.2.7-ice35-b40/lib/scripts
2 | 201     | 47227125-a0dc-4c1c-be62-2ad594f4f6b6 | Public  | /OMERO
3 | 451     | 2e7ed308-e6fa-4fb4-b94a-5e598d993081 | Public  | /OMERO
4 | 850551  | de751af9-105a-48f8-bcff-298462f54c15 | Managed | /OMERO/ManagedRepository
5 | 1171667 | cb2486fd-022d-4e3d-968d-2eb19b67d57e | Public  | /OMERO/ManagedRepository

Thank you for your help !

Best

Jacques

[jmoore]

Hi Jacques,

it sounds like you are having the same issue as seen in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo#p16220, namely that the OMERO BinaryRepository was moved but the self-correction did not succeed. (Other less likely alternatives are that you ran two OMERO servers against the same directory, etc). Does any of that seem possible?

As a next step, we need to know which repositories are the correct ones. Could you run the following command against your PostgreSQL database?

Code: Select all

SELECT count(id), repo FROM originalfile WHERE mimetype <> 'Repository' AND repo IS NOT NULL GROUP BY repo;


Cheers,
~Josh.

[jacques2020]

Hi Josh,

and thank you for your reply.
Here is the output of the command:

Code: Select all

count |                 repo
-------+--------------------------------------
23364 | de751af9-105a-48f8-bcff-298462f54c15
    22 | ScriptRepo
(2 rows)

On a more general perspective,
1) there was only one omero server running on these data at a time but for recovery purpose, I started a second one with a 1 week older snapshot of the database at some point (after stopping the current one). After the recovery, I went back to the current one.
2) The repository was started with version 4 and migrated to version 5.
3) Eventually, I change the hardware containing the data and made a copy at that point. Eventually, the /OMERO path is a symlink pointing to the real mount point of the data NAS. When I upgraded the hardware, the symlink was replaced to point to an other mount point. But it was about 2 years ago while the cleanse issue appeared a few months ago.
I hope it helps to circumvent the possible causes.

Cheers
Jacques

[jmoore]

and thank you for your reply.

Sure. That's what we're here for

Here is the output of the command:

Code: Select all

count |                 repo
-------+--------------------------------------
23364 | de751af9-105a-48f8-bcff-298462f54c15
    22 | ScriptRepo
(2 rows)

This is actually quite good. i.e. only one of your repositories have any files associated with it. In a next step, we'll likely be deleting several of the empty repositories.

On a more general perspective,
1) there was only one omero server running on these data at a time but for recovery purpose, I started a second one with a 1 week older snapshot of the database at some point (after stopping the current one). After the recovery, I went back to the current one.
2) The repository was started with version 4 and migrated to version 5.
3) Eventually, I change the hardware containing the data and made a copy at that point. Eventually, the /OMERO path is a symlink pointing to the real mount point of the data NAS. When I upgraded the hardware, the symlink was replaced to point to an other mount point. But it was about 2 years ago while the cleanse issue appeared a few months ago.
I hope it helps to circumvent the possible causes.

Wow! That's quite the list. I'm not sure where to start to figure out which one of the steps may have led to the current situation. But I think the path forward is fairly straight-forward. One more question first:

What is shown by running:

Code: Select all

find /OMERO/.omero -name repo_uuid -exec cat {} \;
find /OMERO/ManagedRepository/.omero -name repo_uuid -exec cat {} \;


Cheers,
~Josh.

[jacques2020]

Here are the results of the commands

Code: Select all

find /OMERO/.omero -name repo_uuid -exec cat {} \;
$2e7ed308-e6fa-4fb4-b94a-5e598d993081

and

Code: Select all

find /OMERO/ManagedRepository/.omero -name repo_uuid -exec cat {} \;
$de751af9-105a-48f8-bcff-298462f54c15

Cheers
Jacques

[jmoore]

Ok. Good. That points to

Code: Select all

3 | 451     | 2e7ed308-e6fa-4fb4-b94a-5e598d993081 | Public  | /OMERO
4 | 850551  | de751af9-105a-48f8-bcff-298462f54c15 | Managed | /OMERO/ManagedRepository


being your effective (and active) repositories.

Now (as in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo#p16222) it's time to stop your server and take a backup. Once and only once that's done, you'll need to delete the old repositories as in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo&start=10#p16226:

Code: Select all

0 | 12      | 2db7d60a-3884-40b6-990d-f89571e95497 | Public  | /OMERO
2 | 201     | 47227125-a0dc-4c1c-be62-2ad594f4f6b6 | Public  | /OMERO
5 | 1171667 | cb2486fd-022d-4e3d-968d-2eb19b67d57e | Public  | /OMERO/ManagedRepository


which becomes:

Code: Select all

# remove the --dry-run unless you'd like us to look at what will be deleted.
bin/omero delete OriginalFile:12 OriginalFile:201 OriginalFile:1171667 --dry-run --report


~Josh.

[jacques2020]

Hello Mark,

The server is used by people in my lab. It is backed up (still running) once a day (config / database / data dir). Do I need a special backup performed without running the server simultaneously ?
Thank you

Cheers

Jacques

[mtbc]

Dear Jacques,

We would certainly advise backing up during actual downtime in a maintenance window to be sure of taking a cleanly consistent snapshot of the server state. Our documentation ought to make this clearer. Your regular backups will probably suffice but we occasionally recommend going the extra mile because repository deletion / management to recover from unexpected states is exactly the kind of situation that makes data loss rather more likely. We try to be very cautious about helping people to safely preserve their data on their OMERO production systems.

Cheers,
Mark