Prevent a processing script from timing out

[labriedion]

Hi,

I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:

Traceback (most recent call last):
File "bin/omero", line 125, in <module>
rv = omero.cli.argv()
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1438, in argv
cli.invoke(args[1:])
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 952, in invoke
stop = self.onecmd(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1029, in onecmd
self.execute(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1111, in execute
args.func(args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/plugins/script.py", line 429, in launch
rv = proc.getResults(3)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero_Scripts_ice.py", line 929, in getResults
return _M_omero.grid.ScriptProcess._op_getResults.invoke(self, ((waitSecs, ), _ctx))
Ice.UnknownException: exception ::Ice::UnknownException
{
unknown = ome.conditions.SessionTimeoutException: Session (started=2017-02-27 10:25:58.887, hits=296, last access=2017-02-27 11:25:54.553) exceeded timeToLive (3600000) by 61740 ms
at ome.services.sessions.state.SessionCache.getDataNullOrThrowOnTimeout(SessionCache.java:463)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:369)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:354)
at ome.services.sessions.SessionManagerImpl.getReferenceCount(SessionManagerImpl.java:598)
at ome.services.sessions.SessionManagerImpl.environment(SessionManagerImpl.java:723)
at ome.services.sessions.SessionManagerImpl.outputEnvironment(SessionManagerImpl.java:716)
at omero.grid.InteractiveProcessorI.getResults(InteractiveProcessorI.java:335)
at sun.reflect.GeneratedMethodAccessor1846.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy95.getResults(Unknown Source)
at ome.services.blitz.impl.ScriptProcessI.getResults(ScriptProcessI.java:109)
at sun.reflect.GeneratedMethodAccessor1845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy117.getResults(Unknown Source)
at omero.grid._ScriptProcessTie.getResults(_ScriptProcessTie.java:118)
at omero.grid._ScriptProcessDisp.___getResults(_ScriptProcessDisp.java:225)
at omero.grid._ScriptProcessDisp.__dispatch(_ScriptProcessDisp.java:324)
at IceInternal.Incoming.invoke(Incoming.java:222)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
at Ice.ConnectionI.message(ConnectionI.java:1213)
at IceInternal.ThreadPool.run(ThreadPool.java:321)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
at java.lang.Thread.run(Thread.java:745)

I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.

I also tried setting a timeout loop such as this one: https://github.com/openmicroscopy/openmicroscopy/blob/v.4.4.5/examples/OmeroClients/timeout.py. But that also didn't work, although I'm not sure if it should work or not.

Any ideas?

Thanks,
Étienne

[jmoore]

Hi,

Hi Étienne,

I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:

Code: Select all

...
    unknown = ome.conditions.SessionTimeoutException: Session (started=2017-02-27 10:25:58.887, hits=296, last access=2017-02-27 11:25:54.553) exceeded timeToLive (3600000) by 61740 ms
 


I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.

Several questions: What did you set the timeout to? Did you restart the server after changing the property? What version of the server are you running?

We definitely have examples of the following working:

Code: Select all

omero.scripts.timeout=172800000


Cheers,
~Josh

[labriedion]

Hi Josh,

Yes, I did change the script timeout value and restarted the server after. I set it at 36000000. I'll try with your value and report back.

My server is 5.2.7 running on CentOS 7.

(Also, an unrelated issue - when I reboot the server and I check the running sessions with "bin/omero admin sessions who", I see lots (30+) of OMERO.scripts sessions starting in a row, is that normal? They go away eventually.)

Thanks,
Étienne

[jmoore]

Yes, I did change the script timeout value and restarted the server after. I set it at 36000000. I'll try with your value and report back.

Thanks.

My server is 5.2.7 running on CentOS 7.

Noted.

(Also, an unrelated issue - when I reboot the server and I check the running sessions with "bin/omero admin sessions who", I see lots (30+) of OMERO.scripts sessions starting in a row, is that normal? They go away eventually.)

Yes, this is normal. That's the server calculating the params for each of the scripts by calling out to Python.

Cheers,
~Josh.

[labriedion]

Hi Josh,

It actually worked with your value! I guess I must've set the value wrong the first time or it wasn't long enough.

Thanks a lot!

Étienne

[jmoore]

Glad to hear it!
~J.