Connection and Session timeouts

[bernhard]

Hi!

After migrating our client application to Omero4.1 we see omero session timeout exceptions we haven't seen before. Did something change in the session management and timeout settings?

A little bit background information: the application is run in interactive mode, this means the connection can be idle for a couple of minutes before new requests to the server are made. In order to guarantee valid service objects I automatically recreate an omero client object and a ServiceFactoryPrx if a timeout exception occurs:

Code: Select all

// session_, username_ and password_ are class members
// session is a ServiceFactoryPrx
ServiceFactoryPrx getSession() {
  if (session_) {
    try {
      session_->ice_ping();
    } catch (const Ice::ConnectionLostException& e) {
      // recreate session
      client_->closeSession();
      session_ = client_->createSession(username_, password_);
      session_->closeOnDestroy();
    }
  }
  return session_;
}


In order to avoid omero timeout exceptions at a higher level, I could add another handler for those exceptions but I'm wondering whether there is another option like setting the ice communication timeout value to the same as the omero session timeout.

Thanks for help! Bernhard

[jmoore]

Hi Bernhard,

To answer your question, there were many changes to the sessions, especially from the pre-4.x versions. Most likely, you are now experiencing the timeouts because synchronization (session reaping) is more aggressive.

Are you currently using a background thread to ping the service? That is certainly the preferred solution to keeping sessions alive. If you intended to use session-creation, however, adding a SessionTimeoutException stanza to your try/catch block should be fine.

Regardless, a SessionTimeoutException should only be thrown if the session actually times out, and not if the glacier timeout occurs. The default for both is 10 minutes, which seems longer than the waits your experiencing. You could try extending omero.sessions.timeout in etc/omero.properties and see if there is a change. You should then be getting the ConnectionLostException.

Hope that helps,
~Josh

[bernhard]

Hi Josh!

Thanks for you reply. Unfortunately, the additional catch handler for omero::SessionTimeoutException did not solve the issue. Hence I went forward and implemented a pinging thread followong the Ice chat example (I've put the code below for reference and if you like you can put that into your c++ examples if you like).

However, I still have problems with the session handling. My guess is that it is related to the two different timeout settings (glacier and omero session). When I change the glacier timeout in grid/template.xml to 10 seconds and perform a ping on the ServiceFactoryPrx every 5 seconds, the glacier session is not closed. However after 10 minutes I still get the omero::sessiontimeout exception or sometimes the following exception when I invoke methods on (even freashly created) service proxy:
Outgoing.cpp:422: Ice::ObjectNotExistException:
object does not exist:
identity: `session-45CE9E24-DB37-4F8A-A945-7B82A057B2C7/c8b31255-6ad2-4a78-ab5e-659dfb38caaf'
facet:
operation: ice_ping

It seems ServiceFactoryPrx::ice_ping is not enough to keep the omero session alive, what else do I have to do?

For my tests I tried to lower the omero.sessions.timeout setting in etc/omero.properties but it had no effect at all. What else is needed other than restarting the server for this configuration change?

[edit] I found a way via bin/omero config set omero.sessions.timout [/edit]

Thanks for help! Bernhard

Code: Select all

#ifndef SESSIONPINGTHREAD_H_INCLUDED
#define SESSIONPINGTHREAD_H_INCLUDED

#include <string>
#include <IceUtil/IceUtil.h>
#include <omero/API.h>

  /**
   * Thread class that pings an Omero ServiceFactoryPrx in regular intervals
   * in order to keep the session alive.
   */
  class SessionPingThread : public IceUtil::Thread {
   
  public:
   
    /**
     * Creates a new thread class that ping the given session every timeout seconds
     *
     * @param session - Omero session
     * @param timeout - timeout interval between ping invocations in seconds
     */
    SessionPingThread(const omero::api::ServiceFactoryPrx& session, long timeout);

    /**
     * Pings the session of this thread in regular intervals.
     */
    void run();
   
    /**
     * Signals the run method to finish.
     */
    void destroy();

    /**
     * Returns false in case of an error in the run method
     */
    bool failed() {
      return failed_;
    }

    /**
     * Returns the error message of an exception thrown in the run method
     */
    std::string getError() {
      return error_;
    }

  private:
    /// omero session to ping
    omero::api::ServiceFactoryPrx session_;
    /// timeout intervals in seconds
    IceUtil::Time timeout_;
    /// mutex used for conditinal variable
    IceUtil::Mutex mutex_;
    /// conditional variable to signal run method destroy invocation
    IceUtil::Cond condition_;
    /// flags to indicate destroy action and failure in run method
    bool destroy_, failed_;
    /// error message of exception throw in run method
    std::string error_;
  };
  /// shared pointer definition
  typedef IceUtil::Handle<SessionPingThread> SessionPingThreadPtr;
#endif

// Implementation file
#include "SessionPingThread.h"

SessionPingThread::SessionPingThread(const omero::api::ServiceFactoryPrx& session, long timeout) :
  session_(session),
  timeout_(IceUtil::Time::seconds(timeout)),
  destroy_(false),
  failed_(false) { }
 
void SessionPingThread::run() {

  while(!destroy_) {
    // in order to enable signaling use conditional variable
    IceUtil::Mutex::Lock lock(mutex_);
    if (!condition_.timedWait(lock, timeout_)) {
      // timeout expired, do ping
      // else timedWait returns true, when signaled to end, in that case no ping
      // needed - thread was destroyed
      try {
   session_->ice_ping();
        session_->keepAlive(NULL);
   failed_ = false;
      } catch (const Ice::Exception& e) {
   failed_ = true;
   error_ = e.what();
   break;
      }
    }
  }

}
 
void SessionPingThread::destroy() {
  destroy_ = true;
  // return from timedWait on this condition
  condition_.signal();
}



[jmoore]

Hey Bernhard,

Unfortunately, the additional catch handler for omero::SessionTimeoutException did not solve the issue.

What exception were you receiving, or what was the behavior then?

Hence I went forward and implemented a pinging thread followong the Ice chat example (I've put the code below for reference and if you like you can put that into your c++ examples if you like).

A ping thread is certainly what all of our clients do. ticket:1468 intended to add a convenience method for OmeroCpp. I'll most likely rework your code for that. Thanks.

However, I still have problems with the session handling. My guess is that it is related to the two different timeout settings (glacier and omero session). When I change the glacier timeout in grid/template.xml to 10 seconds and perform a ping on the ServiceFactoryPrx every 5 seconds, the glacier session is not closed. However after 10 minutes I still get the omero::sessiontimeout exception or sometimes the following exception when I invoke methods on (even freashly created) service proxy:
Outgoing.cpp:422: Ice::ObjectNotExistException:
object does not exist:
identity: `session-45CE9E24-DB37-4F8A-A945-7B82A057B2C7/c8b31255-6ad2-4a78-ab5e-659dfb38caaf'
facet:
operation: ice_ping

It seems ServiceFactoryPrx::ice_ping is not enough to keep the omero session alive, what else do I have to do?

Correct, it's not enough. ice_ping only checks if a service is still reachable, and like all methods prefixed with "ice_" has nothing to do with OMERO, i.e. it is an Ice-infrastructure method, and OMERO doesn't receive any notification about it.

Use instead: ServiceFactoryPrx.keepAlive: Documentation available here

For my tests I tried to lower the omero.sessions.timeout setting in etc/omero.properties but it had no effect at all. What else is needed other than restarting the server for this configuration change?

Right, etc/omero.properties are basically compile-time defaults. You'd have to

Code: Select all

jar uvf lib/server/blitz.jar

to make changes take affect, which is the meaning of:

# This properties file is compiled into blitz.jar and serves as a default
# for all values server-side values (client configuration happens differently).
# Any of these properties can be altered by using bin/omero config.

[edit] I found a way via bin/omero config set omero.sessions.timout [/edit]

Cool.

Cheers,
~josh

[bernhard]

Hi Josh!

Unfortunately, the additional catch handler for omero::SessionTimeoutException did not solve the issue.

What exception were you receiving, or what was the behavior then?

Actually, as I now understand that ice_ping does not touch the omero session, I recognized that I caught the exception in the wrong place, around a session->ice_ping.

Use instead: ServiceFactoryPrx.keepAlive: Documentation available here

That did the trick. You might want to make the documentation a bit clearer, though.In particular that this method must be called regularly to keep the session alive and that a None/null is a valid argument (at least that's what I do in the ping thread since I don't keep the service proxies around all the time). I updated the source code above.

Also the example on http://trac.openmicroscopy.org.uk/omero/wiki/OmeroClients#Whataretimeouts is a bit misleading as it does not mention the keepAlive method.

As always, thanks for your help!

Have a nice day! Bernhard

[jmoore]

Finally getting around to adding examples as a part of ticket 1814. Cheers, Josh.