Page 1 of 1

user cant login to OMERO becasue of Database issue

PostPosted: Wed Feb 14, 2018 8:57 am
by saleht
Hi OMERO Team,

i hat a weird issue, that after reboot my server could not the users to login.
then i restarted the omero service then works fine
i checked the Blitz-0.log file
i find database connection issue
Code: Select all
2018-02-14 08:56:06,987 INFO  [                ome.services.blitz.Entry] (      main) Creating OMERO.blitz. Please wait...
2018-02-14 08:56:08,206 INFO  [.s.ShutdownSafeEhcacheManagerFactoryBean] (      main) Initializing EHCache CacheManager
2018-02-14 08:56:08,441 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Found location in errorTimes: -1
2018-02-14 08:56:08,441 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Removing 0 from errorTimes
2018-02-14 08:56:08,441 WARN  [ome.services.db.SelfCorrectingDataSource] (      main) Registering error with list: Current size: 0
2018-02-14 08:56:08,442 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Sleeping for 0 then retry: 1
2018-02-14 08:56:08,442 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Found location in errorTimes: -1
2018-02-14 08:56:08,442 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Removing 0 from errorTimes
2018-02-14 08:56:08,443 WARN  [ome.services.db.SelfCorrectingDataSource] (      main) Registering error with list: Current size: 1
2018-02-14 08:56:08,443 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Sleeping for 1000 then retry: 2
2018-02-14 08:56:09,444 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Found location in errorTimes: -1
2018-02-14 08:56:09,444 INFO  [ome.services.db.SelfCorrectingDataSource] (      main) Removing 0 from errorTimes
2018-02-14 08:56:09,444 WARN  [ome.services.db.SelfCorrectingDataSource] (      main) Registering error with list: Current size: 2
2018-02-14 08:56:09,451 ERROR [ome.services.db.SelfCorrectingDataSource] (      main) Failed to acquire connection after retries=3
java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource 7f2ac4dd-e964-4d02-ac1b-766db2524a7c with 0 connection(s) (0 still available)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:262) ~[btm.jar:2.1.3]
        at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:83) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at ome.services.db.SelfCorrectingDataSource.call(SelfCorrectingDataSource.java:124) [server.jar:na]
        at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:97) [server.jar:na]
        at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) [server.jar:na]
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113) [spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:214) [spring-jdbc.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy4.createStatement(Unknown Source) [na:na]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:389) [spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455) [spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:463) [spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.query(SimpleJdbcTemplate.java:199) [spring-jdbc.jar:3.0.1.RELEASE]
        at ome.util.actions.PostgresSqlAction.dbVersion(PostgresSqlAction.java:171) [model-psql.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop.jar:3.0.1.RELEASE]
        at ome.util.SqlAction$LoggingSqlAction.invoke(SqlAction.java:85) [model-psql.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy3.dbVersion(Unknown Source) [na:na]
        at ome.services.util.DBPatchCheck.start(DBPatchCheck.java:55) [server.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1527) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1468) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1400) [spring-beans.jar:3.0.1.RELEASE]

    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:888) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1048) [spring-context.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:397) [spring-beans.jar:3.0.1.RELEASE]
        at ome.system.OmeroContext.getInstance(OmeroContext.java:202) [common.jar:na]
        at ome.services.blitz.Entry.start(Entry.java:189) [blitz.jar:na]
        at ome.services.blitz.Entry.main(Entry.java:146) [blitz.jar:na]
Caused by: java.sql.SQLException: unable to connect to non-XA resource org.postgresql.Driver
        at bitronix.tm.resource.jdbc.lrc.LrcXADataSource.getXAConnection(LrcXADataSource.java:108) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.jdbc.PoolingDataSource.createPooledConnection(PoolingDataSource.java:341) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.common.XAPool.createPooledObject(XAPool.java:283) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.common.XAPool.grow(XAPool.java:391) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:371) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:123) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91) ~[btm.jar:2.1.3]
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:258) ~[btm.jar:2.1.3]
        ... 98 common frames omitted
Caused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:239) ~[postgresql.jar:9.4]
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) ~[postgresql.jar:9.4]
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127) ~[postgresql.jar:9.4]
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) ~[postgresql.jar:9.4]
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) ~[postgresql.jar:9.4]
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41) ~[postgresql.jar:9.4]
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) ~[postgresql.jar:9.4]
        at org.postgresql.Driver.makeConnection(Driver.java:414) ~[postgresql.jar:9.4]
        at org.postgresql.Driver.connect(Driver.java:282) ~[postgresql.jar:9.4]
        at bitronix.tm.resource.jdbc.lrc.LrcXADataSource.getXAConnection(LrcXADataSource.java:104) ~[btm.jar:2.1.3]
        ... 105 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_131]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_131]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_131]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_131]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_131]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_131]
        at org.postgresql.core.PGStream.<init>(PGStream.java:61) ~[postgresql.jar:9.4]
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121) ~[postgresql.jar:9.4]
        ... 114 common frames omitted
2018-02-14 08:56:09,455 ERROR [          ome.services.util.DBPatchCheck] (      main)

***************************************************************************************
Error connecting to database table dbpatch. You may need to bootstrap.
See https://docs.openmicroscopy.org/latest/omero/sysadmins/server-upgrade.html
***************************************************************************************

ome.conditions.DatabaseBusyException: Cannot acquire connection
        at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:113) ~[server.jar:na]
        at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) ~[server.jar:na]
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:214) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy4.createStatement(Unknown Source) ~[na:na]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:389) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:463) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.query(SimpleJdbcTemplate.java:199) ~[spring-jdbc.jar:3.0.1.RELEASE]
        at ome.util.actions.PostgresSqlAction.dbVersion(PostgresSqlAction.java:171) ~[model-psql.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.util.SqlAction$LoggingSqlAction.invoke(SqlAction.java:85) ~[model-psql.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) ~[spring-aop.jar:3.0.1.RELEASE]
        at com.sun.proxy.$Proxy3.dbVersion(Unknown Source) ~[na:na]
        at ome.services.util.DBPatchCheck.start(DBPatchCheck.java:55) ~[server.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1527) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1468) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1400) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) [spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450) [spring-beans.jar:3.0.1.RELEASE]
     at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1048) [spring-context.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:397) [spring-beans.jar:3.0.1.RELEASE]
        at ome.system.OmeroContext.getInstance(OmeroContext.java:202) [common.jar:na]
        at ome.services.blitz.Entry.start(Entry.java:189) [blitz.jar:na]
        at ome.services.blitz.Entry.main(Entry.java:146) [blitz.jar:na]
2018-02-14 08:56:09,457 INFO  [.s.ShutdownSafeEhcacheManagerFactoryBean] (      main) Shutting down EHCache CacheManager
2018-02-14 08:56:09,459 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [OMERO.blitz], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'OMERO.blitz' defined in URL [jar:file:/mnt/data/OMERO.server-5.4.3-ice36-b77/lib/server/blitz.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'ome.server' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/mnt/data/OMERO.server-5.4.3-ice36-b77/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'currentDetails' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'roles' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPatchCheck' defined in class path resource [ome/services/startup.xml]: Invocation of init method failed; nested exception is ome.conditions.InternalException:
***************************************************************************************
Error connecting to database table dbpatch. You may need to bootstrap.
See https://docs.openmicroscopy.org/latest/omero/sysadmins/server-upgrade.html
***************************************************************************************

        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:409) ~[spring-beans.jar:3.0.1.RELEASE]
        at ome.system.OmeroContext.getInstance(OmeroContext.java:202) ~[common.jar:na]
        at ome.services.blitz.Entry.start(Entry.java:189) [blitz.jar:na]
        at ome.services.blitz.Entry.main(Entry.java:146) [blitz.jar:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'OMERO.blitz' defined in URL [jar:file:/mnt/data/OMERO.server-5.4.3-ice36-b77/lib/server/blitz.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'ome.server' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/mnt/data/OMERO.server-5.4.3-ice36-b77/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'currentDetails' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'roles' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPatchCheck' defined in class path resource [ome/services/startup.xml]: Invocation of init method failed; nested exception is ome.conditions.InternalException:
***************************************************************************************
Error connecting to database table dbpatch. You may need to bootstrap.
See https://docs.openmicroscopy.org/latest/omero/sysadmins/server-upgrade.html
***************************************************************************************

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:586) ~[spring-beans.jar:3.0.1.RELEASE]


i think the reason written here
https://www.postgresql.org/message-id/2 ... havleik.no

the question is if i did, what they worte
Code: Select all
REINDEX connect_idx;

could i had some bad impacts or not ?

cheers, Saleh

Re: user cant login to OMERO becasue of Database issue

PostPosted: Wed Feb 14, 2018 10:39 am
by jmoore
saleht wrote:i hat a weird issue, that after reboot my server could not the users to login.
then i restarted the omero service then works fine
...
Caused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused.


Can you review the order of your services? Did PostgreSQL get started _before_ OMERO?

i think the reason written here
https://www.postgresql.org/message-id/2 ... havleik.no

the question is if i did, what they worte
Code: Select all
REINDEX connect_idx;

could i had some bad impacts or not ?


Was this a hard reboot? i.e. did your server crash. If not, then I wouldn't expect so. Did you check the PostgreSQL log as well?

Cheers,
~Josh