Making sure OMERO python scripts clean up properly

[graemeball]

I am writing an OMERO python script for long-running processing and have 2 questions.

Firstly, I am creating a BlitzGateway object:

Code: Select all

conn = BlitzGateway(client_obj=client)

and am wondering if/how I need to close it in a finally block before I do client.closeSession() -- do I need to use .seppuku(), or is this not necessary? (or should I be accessing the gateway in a different way in the first place?)

Secondly, it seems that if I start the script and then log out from my (web) client session, the script dies but my finally block is never reached, and temporary files are therefore not cleaned up. Is this expected behaviour? (or have I made a mistake?) Is there a way I can make sure my cleanup code is always run? If I start the script and close the browser tab without logging out, the session & script continue to run.

Note that I have been testing with a 5.0 server installation, so if there have been updates related to these issues since 5.0, please let me know and I will try with 5.2

Thanks, Graeme

[atarkowska]

Hi Graeme,

If you call seppuku it will kill omero session which will log user out.
I think your script could use tempfile

Code: Select all

import tempfile
with tempfile.NamedTemporaryFile() as temp:
    temp.write('Some data')
    temp.flush()

this should resolve the issues.

Ola

[graemeball]

Hi Ola,

Thanks for your reply. If tempfile does indeed clean up properly upon user logout (I have not tried), how is the cleanup triggered when logout happens / why does my finally block fail?

I cannot use a simple NamedTemporaryFile, because I need a temporary *directory* in a specific location. I see tempfile module can also create a temporary directory, but:
- other (remote) processes will be writing to this folder (hopefully as the same *user*... definitely same group)
- the python 2 version of 'tempfile' does not appear to clean up temp directories automatically using 'with':
http://stackoverflow.com/questions/6884 ... le-mkdtemp

Cheers,

Graeme

[atarkowska]

Hi Graeme,

If you want to create shared folder to read from or write to you will have to maintain it separately, for example using crone job. From the other hand did you consider using redis as a persistent store?

Ola

[graemeball]

OK, so if I understand correctly, OMERO.scripts do not really get a chance to clean up if the client session exits? (is the process they run in killed? I tried digging through the codebase a bit, but I don't think I can grasp it in a reasonable amount of time) It's a pity, but I guess I should be able to clean up the temp directories from the processing node side.

What is more worrying is: if the finally block is never reached, client.closeSession() will never be called. Hopefully though, this is automatically taken care of upon client session logout?

The temp directories will contain both a job description (basically .json), but also image data, which would not fit in a data/object store like redis.

Thanks again for your help,

Graeme

[atarkowska]

OK, so if I understand correctly, OMERO.scripts do not really get a chance to clean up if the client session exits? (is the process they run in killed? I tried digging through the codebase a bit, but I don't think I can grasp it in a reasonable amount of time) It's a pity, but I guess I should be able to clean up the temp directories from the processing node side.

What is more worrying is: if the finally block is never reached, client.closeSession() will never be called. Hopefully though, this is automatically taken care of upon client session logout?

I am not sure if I understand what is your question exactly?
When OMERO.web submit a job to the server to execute script this result in new session creation. Session will be killed and cleaned up regardless. It is true that OMERO.web shouldn't kill running jobs (script) on logout. I already reported that in https://trello.com/c/yc8YNhUa/117-omero ... -on-logout

As you are running 5.0 I am not able to help much, but I tested 5.2 and any exception raised in a body of a script should result in a finally block being called. If you are not sure add more debugging like:

Code: Select all

try:
   ...
except:
    client.setOutput("error", rstring(traceback.format_exc()))
finally:
    client.setOutput("finally",rstring("finally end"))
    client.closeSession()

For more details please look at Processor-0.log

As I said, for those cases where script won't finish sysadmin is responsible for deleting the any files regularly.

The temp directories will contain both a job description (basically .json), but also image data, which would not fit in a data/object store like redis.

Definitely JSON can be store in redis either as a plain string in dedicated key (or member/value of a set/list) or in a hash structure.

Ola