Prevent a processing script from timing out
Posted: Mon Feb 27, 2017 4:39 pm
Hi,
I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:
I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.
I also tried setting a timeout loop such as this one: https://github.com/openmicroscopy/openmicroscopy/blob/v.4.4.5/examples/OmeroClients/timeout.py. But that also didn't work, although I'm not sure if it should work or not.
Any ideas?
Thanks,
Étienne
I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:
Traceback (most recent call last):
File "bin/omero", line 125, in <module>
rv = omero.cli.argv()
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1438, in argv
cli.invoke(args[1:])
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 952, in invoke
stop = self.onecmd(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1029, in onecmd
self.execute(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1111, in execute
args.func(args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/plugins/script.py", line 429, in launch
rv = proc.getResults(3)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero_Scripts_ice.py", line 929, in getResults
return _M_omero.grid.ScriptProcess._op_getResults.invoke(self, ((waitSecs, ), _ctx))
Ice.UnknownException: exception ::Ice::UnknownException
{
unknown = ome.conditions.SessionTimeoutException: Session (started=2017-02-27 10:25:58.887, hits=296, last access=2017-02-27 11:25:54.553) exceeded timeToLive (3600000) by 61740 ms
at ome.services.sessions.state.SessionCache.getDataNullOrThrowOnTimeout(SessionCache.java:463)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:369)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:354)
at ome.services.sessions.SessionManagerImpl.getReferenceCount(SessionManagerImpl.java:598)
at ome.services.sessions.SessionManagerImpl.environment(SessionManagerImpl.java:723)
at ome.services.sessions.SessionManagerImpl.outputEnvironment(SessionManagerImpl.java:716)
at omero.grid.InteractiveProcessorI.getResults(InteractiveProcessorI.java:335)
at sun.reflect.GeneratedMethodAccessor1846.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy95.getResults(Unknown Source)
at ome.services.blitz.impl.ScriptProcessI.getResults(ScriptProcessI.java:109)
at sun.reflect.GeneratedMethodAccessor1845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy117.getResults(Unknown Source)
at omero.grid._ScriptProcessTie.getResults(_ScriptProcessTie.java:118)
at omero.grid._ScriptProcessDisp.___getResults(_ScriptProcessDisp.java:225)
at omero.grid._ScriptProcessDisp.__dispatch(_ScriptProcessDisp.java:324)
at IceInternal.Incoming.invoke(Incoming.java:222)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
at Ice.ConnectionI.message(ConnectionI.java:1213)
at IceInternal.ThreadPool.run(ThreadPool.java:321)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
at java.lang.Thread.run(Thread.java:745)
I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.
I also tried setting a timeout loop such as this one: https://github.com/openmicroscopy/openmicroscopy/blob/v.4.4.5/examples/OmeroClients/timeout.py. But that also didn't work, although I'm not sure if it should work or not.
Any ideas?
Thanks,
Étienne