removepyramids on 5.4.4

[dsudar]

Hi team,

I just upgraded to 5.4.4 and wanted to check how many corrupted pyramids I might have.

First attempt fails because it clearly needs a wait:

Code: Select all

omero_user@omero:~$ omero admin removepyramids --endian=little --dry-run
Using session c6eefcb8-82f6-4136-8e72-11998383bbce (dsudar@localhost:4064). Idle timeout: 10 min. Current group: Sandbox
No more than 500 pyramids will be removed
Traceback (most recent call last):
  File "/usr/local/bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1620, in argv
    cli.invoke(args[1:])
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1095, in invoke
    stop = self.onecmd(line, previous_args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1172, in onecmd
    self.execute(line, previous_args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1254, in execute
    args.func(args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 644, in _check_admin
    return func(*args, **kwargs)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/plugins/admin.py", line 1016, in removepyramids
    limit=limit, wait=wait)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line 360, in removepyramids
    failontimeout=True)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/clients.py", line 899, in submit
    closehandle=True)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/clients.py", line 918, in waitOnCmd
    callback.loop(loops, ms)  # Throw LockTimeout
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/callbacks.py", line 256, in loop
    5000L, int(waited))
omero.LockTimeout: exception ::omero::LockTimeout
{
    serverStackTrace = None
    serverExceptionClass = None
    message = Command unfinished after 25.0 seconds
    backOff = 5000
    seconds = 25
}


But 2nd attempt with a 5000 wait also fails because it loses connection:

Code: Select all

omero_user@omero:~$ omero admin removepyramids --endian=little --dry-run --wait=5000
Using session c6eefcb8-82f6-4136-8e72-11998383bbce (dsudar@localhost:4064). Idle timeout: 10 min. Current group: Sandbox
No more than 500 pyramids will be removed
-! 03/07/18 18:58:15.461 warning: Proxy keep alive failed.
Traceback (most recent call last):
  File "/usr/local/bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1620, in argv
    cli.invoke(args[1:])
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1095, in invoke
    stop = self.onecmd(line, previous_args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1172, in onecmd
    self.execute(line, previous_args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 1254, in execute
    args.func(args)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/cli.py", line 644, in _check_admin
    return func(*args, **kwargs)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/plugins/admin.py", line 1016, in removepyramids
    limit=limit, wait=wait)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line 360, in removepyramids
    failontimeout=True)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/clients.py", line 899, in submit
    closehandle=True)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/clients.py", line 921, in waitOnCmd
    callback.close(closehandle)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/callbacks.py", line 316, in close
    self.handle.close()
  File "/usr/local/OMERO/OMERO.server/lib/python/omero_cmd_API_ice.py", line 860, in close
    return _M_omero.cmd.Handle._op_close.invoke(self, ((), _ctx))
Ice.ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero


Should I set the Ice connection to be much longer? If so, how?

Thanks,
- Damir

[jburel]

Hi Damir


Could you run in another terminal the following command

Code: Select all

omero sessions keepalive

and see if it helps

Cheers
Jmarie

[jburel]

Hi Damir
I have just run the command

Code: Select all

omero admin removepyramids --endian=little --dry-run --wait=5000


against our production server. It look ~15mins so I had to run

Code: Select all

omero sessions keepalive

to be sure I won't lose the connection.
It returns the expected result.
Note that you can also narrow down the number of pyramids to remove by using

Code: Select all

--imported-after DATE

Cheers

Jmarie

[dsudar]

Hi Jean-Marie,

Thanks, that indeed works for me as well. It outputs a list of 502 images it "Would remove pyramid for image xxxxx". However, when I inspect some of those images randomly, they look fine and render correctly. And the ones I looked at were imported at various time points between 2013 and very recently. Rather than killing off all those 502 pyramids and letting the server painfully re-create them all, is there a good way to pick out only the really corrupted ones? Or is it better to wipe all the ones found by the command?

Thanks,
- Damir

[jburel]

Hi Damir

Since the problem was introduced in 5.2 I reckon that out of the 502 only a small number of pyramids will be removed since the method will not remove the pre-FS pyramids. We currently do not offer a split between pre-FS and post-FS pyramids when running removepyramids with --dry-run

Finding the pyramids to remove is based on two criteria: endianness (due to bug introduced in 5.2) and imported after a given date
To find the pyramids corrupted due to the 5.2 bug, you could run removepyramids with --imported-after YYYY-MM-DD e.g. 2017-12-10 and --endian=little

The method does not support a time interval or the ability to specify a given pyramid id (id found by running with --dry-run)
Will that be useful to add?


The pyramids generated circa 2013 will most likely not be deleted (pre-FS)

Cheers
Jmarie

[dsudar]

Hi Jean-Marie,

Indeed by limiting the imported-after date range I can make it so it only finds a manageable number of pyramids to delete and so I let that happen. It deleted 32 pyramids as a small test.

However, I found it does not automatically re-create those deleted pyramids until one actually visits them. So that's good in the way that with a large remove, the server will not be busy creating pyramids for days afterwards but it's bad in the sense that users will now be confused because an image that used to "work" fine, now gives them suddenly a "pyramid being created" message. Is there way to cause them being generated from the output of the removepyramids command? I guess that the output isn't really in a good format for that but could be reformated with some manipulations.

I don't think there's a need for more options for the removepyramids method since this is not something one has to do often.

Thanks,
- Damir

[jburel]

Hi Damir

I wrote a little Python script that will re-regenerate the pyramids after running the "removepyramids" method
I send the output of the command line into a text file output.txt.

Code: Select all

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import omero
from omero.gateway import BlitzGateway
from omero.rtypes import rlist, rlong


file_name = 'output.txt'
ids = []
# extract the image ids
with open(file_name) as f:
    lines = f.readlines()
    for line in lines:
        line = line.strip('\n')
        if line.startswith("Pyramid removed for image"):
           for s in line.split():
               if s.isdigit():
                  ids.append(long(s))
# edit with your credentials
conn = BlitzGateway(admin_username, admin_password, host=hostname)
conn.connect()
# to allow cross group retrieval
conn.SERVICE_OPTS.setOmeroGroup('-1')

# asking for the thumbnail will trigger the pyramid generation
resp = conn.getThumbnailSet(rlist([rlong(i) for i in ids]))

conn._closeSession()


Hope it helps

Cheers

Jmarie