Sometimes during log in 504 Gateway Time-out

[saleht]

Hi guys,

i am so glad that Omero works good now, and thats because you help me
i have a small issue but it is annoyed for the users

sometimes after requesting omero webpag, login page coming normal, but after putting user-name & password (from ldap users),
the page stay loading then gives 504 Gateway Time-out, if i wait a little or make refers, it will works after 10 sec
same issue with inisght app, if i closed and reopen it again works fine
any idea ???

thx in advanced

[atarkowska]

Hi,

Did you check var/log/Blitz-0.log around the same time? Do you see something like:

Code: Select all

ERROR [     o.s.blitz.fire.PermissionsVerifierI] (Server-183) Exception thrown while checking password for: user1234
ome.conditions.InternalException:  Wrapped Exception: (org.springframework.ldap.CommunicationException):
yourldapserver:389; nested exception is javax.naming.CommunicationException: yourldapserver:389 [Root exception is java.net.ConnectException: Connection timed out]

?

My guess is that your ldap server is not responding in default 60 sec https://www.openmicroscopy.org/site/sup ... gi-timeout and OMERO.web returns 504 Timeout as it didn't get response from OMERO.server on time.


Ola

[saleht]

Hi atarkowska ,

thx for info, i will check and tell you

[saleht]

Hi again,
if i checked ngnix Erro logs
i am getting upstream timed out 110 connection timed out

2016/12/19 11:25:51 [error] 1488#0: *98 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "GET /webclient/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/", host: "omero-cai.hhu.de"
2016/12/19 11:29:46 [error] 1488#0: *102 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "GET /webclient/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/", host: "omero-cai.hhu.de"
2016/12/19 13:07:36 [error] 5564#0: *10 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "GET /webclient/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/", host: "omero-cai.hhu.de"
2016/12/19 13:42:32 [error] 5918#0: *161 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "GET /webclient/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/", host: "omero-cai.hhu.de"
2016/12/19 13:45:17 [error] 5918#0: *198 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "POST /webclient/login/?url=%2Fwebclient%2F HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/login/?url=%2Fwebclient%2F", host: "omero-cai.hhu.de", referrer: "https://omero-cai.hhu.de/webclient/login/?url=%2Fwebclient%2F"
2016/12/19 13:47:24 [error] 5918#0: *223 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "POST /webclient/login/?url=%2Fwebclient%2F HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/login/?url=%2Fwebclient%2F", host: "omero-cai.hhu.de", referrer: "https://omero-cai.hhu.de/webclient/login/?url=%2Fwebclient%2F"
2016/12/19 13:48:29 [error] 5918#0: *223 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "POST /webclient/login/?url=%2Fwebclient%2F HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/login/?url=%2Fwebclient%2F", host: "omero-cai.hhu.de", referrer: "https://omero-cai.hhu.de/webclient/login/?url=%2Fwebclient%2F"
2016/12/19 13:55:17 [error] 5918#0: *235 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "POST /webclient/login/?url=%2Fwebclient%2F HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/login/?url=%2Fwebclient%2F", host: "omero-cai.hhu.de", referrer: "https://omero-cai.hhu.de/webclient/login/?url=%2Fwebclient%2F"

from :
Blitz-0.log

2016-12-19 13:59:15,412 INFO [ ome.services.util.ServiceHandler] (.Server-15) Excp: org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: AD.hhu.de:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
2016-12-19 13:59:15,412 ERROR [ ome.services.util.ServiceHandler] (.Server-15) Method interface ome.services.util.Executor$Work.doWork invocation took 929431
2016-12-19 13:59:15,413 ERROR [ o.s.blitz.fire.PermissionsVerifierI] (.Server-15) Exception thrown while checking password for:106de990-76bc-4958-8724-416c6a277472
ome.conditions.InternalException: Wrapped Exception: (org.springframework.ldap.PartialResultException):
nested exception is javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: AD.hhu.de:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:203) ~[spring-ldap-core.jar:1.3.0.RELEASE]
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:315) ~[spring-ldap-core.jar:1.3.0.RELEASE]
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:259) ~[spring-ldap-core.jar:1.3.0.RELEASE]
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:606) ~[spring-ldap-core.jar:1.3.0.RELEASE]
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:524) ~[spring-ldap-core.jar:1.3.0.RELEASE]
at ome.logic.LdapImpl.mapUserName(LdapImpl.java:197) ~[server.jar:na]
at ome.logic.LdapImpl.findExperimenter(LdapImpl.java:174) ~[server.jar:na]
at ome.logic.LdapImpl.createUser(LdapImpl.java:477) ~[server.jar:na]
at ome.logic.LdapImpl.createUser(LdapImpl.java:453) ~[server.jar:na]
at ome.security.auth.LdapPasswordProvider.checkPassword(LdapPasswordProvider.java:113) ~[server.jar:na]
at ome.security.auth.PasswordProviders.checkPassword(PasswordProviders.java:52) ~[server.jar:na]
at ome.logic.AdminImpl.checkPassword(AdminImpl.java:1197) ~[server.jar:na]
at ome.services.sessions.SessionManagerImpl$9.doWork(SessionManagerImpl.java:1102) ~[server.jar:na]
at sun.reflect.GeneratedMethodAccessor298.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at ome.security.basic.EventHandler.invoke(EventHandler.java:154) ~[server.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) ~[server.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) ~[server.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) ~[spring-aop.jar:3.0.1.RELEASE]
at com.sun.proxy.$Proxy72.doWork(Unknown Source) ~[na:na]
at ome.services.util.Executor$Impl.execute(Executor.java:443) ~[server.jar:na]
at ome.services.util.Executor$Impl.execute(Executor.java:387) ~[server.jar:na]
at ome.services.sessions.SessionManagerImpl.executeCheckPasswordRW(SessionManagerImpl.java:1097) ~[server.jar:na]
at ome.services.sessions.SessionManagerImpl.executeCheckPassword(SessionManagerImpl.java:1069) ~[server.jar:na]
at ome.services.sessions.SessionManagerImpl.executePasswordCheck(SessionManagerImpl.java:1045) ~[server.jar:na]
at ome.services.blitz.fire.PermissionsVerifierI.checkPermissions(PermissionsVerifierI.java:132) ~[blitz.jar:na]
at Glacier2._PermissionsVerifierDisp.___checkPermissions(_PermissionsVerifierDisp.java:114) [ice-glacier2.jar:na]
at Glacier2._PermissionsVerifierDisp.__dispatch(_PermissionsVerifierDisp.java:149) [ice-glacier2.jar:na]
at IceInternal.Incoming.invoke(Incoming.java:221) [ice.jar:na]

sometimes i am getting timeout if if i request the first page without any log in, just the page of OMERO
i believe i have a problem in nginx which make it behave like this,
can i put two ldap server in my configuration, if this will solve the issue ?

[jmoore]

Hi again,

Hi Saleh,

if i checked ngnix Erro logs
i am getting upstream timed out 110 connection timed out

Code: Select all

2016/12/19 11:25:51 [error] 1488#0: *98 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 134.99.190.4, server: omero-cai.hhu.de, request: "GET /webclient/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/", host: "omero-cai.hhu.de"


If you are seeing timeouts, you can certainly up the timeouts in your nginx configuration to prevent them, but they likely point to a deeper performance issue.

See http://howtounix.info/howto/110-connection-timed-out-error-in-nginx for example.

from :
Blitz-0.log

Code: Select all

2016-12-19 13:59:15,412 INFO  [        ome.services.util.ServiceHandler] (.Server-15)  Excp:    org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: AD.hhu.de:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
2016-12-19 13:59:15,412 ERROR [        ome.services.util.ServiceHandler] (.Server-15) Method interface ome.services.util.Executor$Work.doWork invocation took 929431
2016-12-19 13:59:15,413 ERROR [     o.s.blitz.fire.PermissionsVerifierI] (.Server-15) Exception thrown while checking password for:106de990-76bc-4958-8724-416c6a277472
ome.conditions.InternalException:  Wrapped Exception: (org.springframework.ldap.PartialResultException):
nested exception is javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: AD.hhu.de:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
        at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:203) ~[spring-ldap-


This looks like a known issue with Active Directories. Ola previously had a suggestion: https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=8158&p=17645#p17647. Did you give it a try?

sometimes i am getting timeout if if i request the first page without any log in, just the page of OMERO
i believe i have a problem in nginx which make it behave like this,
can i put two ldap server in my configuration, if this will solve the issue ?

I don't think putting two LDAP servers in the config will help, neither with the nginx issue nor the LDAP issue.

Cheers,
~Josh

[atarkowska]

socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']

Not an expert, but that error looks like the LDAP server dropped connection. Bad routing path? Try using the IP instead of the DNS resolution


Ola

[saleht]

thax for both,

i will try to change to IP Address.
and i don't believe that i have performance issue, i have 3 GB always free, my 4 Cpu with 8 cors, and i working alone on it, it is under test.

[saleht]

this issue solved here

viewtopic.php?f=5&t=8297