Bad Gateway error (again)

[alexr]

Dear forum,
I suddenly get the 502 Bad Gateway error. This has been described several times in the forum and I tried to find a solution in the previous posts without success.
So Omero server is running, I can access with the insight client, but I cannot access with the web client. OmeroWeb is also running. I tried several diagnostics, that I list below:

Output of Blitz.log

Code: Select all

2018-06-22 11:32:04,396 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1529659924351] time[45] tag[omero.call.success.ome.logic.ConfigImpl.getConfigValue]
2018-06-22 11:32:04,396 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Rslt:   OMERO.Glacier2/router:tcp -p 4063 -h @omero.host@
2018-06-22 11:32:04,403 INFO  [ ome.services.blitz.fire.SessionManagerI] (l.Server-5) Found session locally: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:04,405 INFO  [ ome.services.blitz.fire.SessionManagerI] (l.Server-5) Rejoining session ServiceFactoryI(session-ca31bcdb-9a52-48cd-947f-836a9922bfae/54c7cf1f-dfc6-42a2-b813-c4261ef245e4) (agent=OMERO.py.gateway;secure=False)
2018-06-22 11:32:04,412 INFO  [o.services.sessions.SessionContext$Count] (l.Server-4) -Reference count: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4=1
2018-06-22 11:32:04,412 INFO  [                      omero.cmd.SessionI] (l.Server-4) cleanupSelf(ServiceFactoryI(session-198c7aac-0c53-4768-84ae-2c11ff7e1e1b/54c7cf1f-dfc6-42a2-b813-c4261ef245e4)).
2018-06-22 11:32:04,412 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:54c7cf1f-dfc6-42a2-b813-c4261ef245e4/198c7aac-0c53-4768-84ae-2c11ff7e1e1bomero.api.ISession(omero.api._ISessionTie@43f8bca5)
2018-06-22 11:32:04,412 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: 198c7aac-0c53-4768-84ae-2c11ff7e1e1bomero.api.ISession
2018-06-22 11:32:04,412 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:54c7cf1f-dfc6-42a2-b813-c4261ef245e4/198c7aac-0c53-4768-84ae-2c11ff7e1e1bomero.api.IConfig(omero.api._IConfigTie@4a89e52c)
2018-06-22 11:32:04,412 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: 198c7aac-0c53-4768-84ae-2c11ff7e1e1bomero.api.IConfig
2018-06-22 11:32:04,417 INFO  [                      omero.cmd.SessionI] (l.Server-3) Added servant to adapter: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IAdmin(omero.api._IAdminTie@bc3e41cd)
2018-06-22 11:32:04,422 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Meth:   interface ome.api.IAdmin.lookupGroups
2018-06-22 11:32:04,422 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Args:   ()
2018-06-22 11:32:04,424 INFO  [         ome.security.basic.EventHandler] (l.Server-0)  Auth:   user=3,group=53,event=null(User),sess=54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:04,450 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1529659924422] time[27] tag[omero.call.success.ome.logic.AdminImpl.lookupGroups]
2018-06-22 11:32:04,450 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Rslt:   (ome.model.meta.ExperimenterGroup:Id_0, ome.model.meta.ExperimenterGroup:Id_1, ome.model.meta.ExperimenterGroup:Id_2, ... 3 more)
2018-06-22 11:32:04,466 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Executor.doWork -- ome.services.blitz.impl.ServiceFactoryI.isGuest[]
2018-06-22 11:32:04,466 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Args:   [null, InternalSF@244922133]
2018-06-22 11:32:04,468 INFO  [         ome.security.basic.EventHandler] (l.Server-2)  Auth:   user=3,group=53,event=null(User),sess=54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:04,470 INFO  [                 org.perf4j.TimingLogger] (l.Server-2) start[1529659924466] time[3] tag[omero.call.success.ome.services.blitz.impl.ServiceFactoryI$3.doWork]
2018-06-22 11:32:04,470 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Rslt:   false
2018-06-22 11:32:04,471 INFO  [                      omero.cmd.SessionI] (l.Server-2) Added servant to adapter: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IQuery(omero.api._IQueryTie@66e03835)
2018-06-22 11:32:04,475 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Meth:   interface ome.api.IQuery.findAllByQuery
2018-06-22 11:32:04,475 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Args:   [select pchl from GroupExperimenterMap as pchl left outer join fetch pchl.child as child left outer j, PARAMS:parent=1 ]
2018-06-22 11:32:04,477 INFO  [         ome.security.basic.EventHandler] (l.Server-6)  Auth:   user=3,group=53,event=null(User),sess=54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:04,493 INFO  [                 org.perf4j.TimingLogger] (l.Server-6) start[1529659924475] time[17] tag[omero.call.success.ome.logic.QueryImpl.findAllByQuery]
2018-06-22 11:32:04,493 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Rslt:   (ome.model.meta.GroupExperimenterMap:Id_4, ome.model.meta.GroupExperimenterMap:Id_6, ome.model.meta.GroupExperimenterMap:Id_8, ... 10 more)
2018-06-22 11:32:06,084 INFO  [                      omero.cmd.SessionI] (l.Server-1) Added servant to adapter: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.ISession(omero.api._ISessionTie@b0e2d08c)
2018-06-22 11:32:06,090 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Meth:   interface ome.api.ISession.closeSession
2018-06-22 11:32:06,091 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Args:   [ome.model.meta.Session:Hash_90834761]
2018-06-22 11:32:06,091 INFO  [o.services.sessions.SessionContext$Count] (2-thread-1) -Reference count: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4=0
2018-06-22 11:32:06,091 INFO  [ome.services.sessions.SessionManagerImpl] (2-thread-1) closeSession called and no more references: 54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:06,091 INFO  [ome.services.sessions.state.SessionCache] (2-thread-1) Destroying session 54c7cf1f-dfc6-42a2-b813-c4261ef245e4 due to : Remove session called
2018-06-22 11:32:06,091 INFO  [                 org.perf4j.TimingLogger] (2-thread-1) start[1529659924322] time[1769] tag[omero.session]
2018-06-22 11:32:06,096 INFO  [ ome.services.blitz.fire.SessionManagerI] (2-thread-1) Reaping 1 clients for 54c7cf1f-dfc6-42a2-b813-c4261ef245e4
2018-06-22 11:32:06,096 INFO  [                      omero.cmd.SessionI] (2-thread-1) doDestroy(ServiceFactoryI(session-ca31bcdb-9a52-48cd-947f-836a9922bfae/54c7cf1f-dfc6-42a2-b813-c4261ef245e4))
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Unregistered servant:54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IQuery(omero.api._IQueryTie@66e03835)
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Removed servant from adapter: ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IQuery
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Unregistered servant:54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IAdmin(omero.api._IAdminTie@bc3e41cd)
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Removed servant from adapter: ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.IAdmin
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Unregistered servant:54c7cf1f-dfc6-42a2-b813-c4261ef245e4/ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.ISession(omero.api._ISessionTie@b0e2d08c)
2018-06-22 11:32:06,097 INFO  [                      omero.cmd.SessionI] (2-thread-1) Removed servant from adapter: ca31bcdb-9a52-48cd-947f-836a9922bfaeomero.api.ISession
2018-06-22 11:32:06,097 INFO  [ ome.services.blitz.fire.SessionManagerI] (2-thread-1) Removing ServiceFactoryI(session-ca31bcdb-9a52-48cd-947f-836a9922bfae/54c7cf1f-dfc6-42a2-b813-c4261ef245e4)
2018-06-22 11:32:06,098 INFO  [                 org.perf4j.TimingLogger] (l.Server-5) start[1529659926091] time[7] tag[omero.call.success.ome.services.sessions.SessionBean.closeSession]
2018-06-22 11:32:06,098 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Rslt:   -2
2018-06-22 11:33:00,015 INFO  [ ome.services.blitz.fire.SessionManagerI] (2-thread-1) Performing requestHeartbeats


Status of nginx:

Code: Select all

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful


OmeroWeb status:

Code: Select all

OMERO.web status... [RUNNING] (PID 4196)


Tail of OMEROweb.log:

Code: Select all

2018-06-22 10:43:39,998 ERROR [                          django.request] (proc.04201) handle_uncaught_exception():256 HTTP 500 <WSGIRequest: GET '/favicon.ico'>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 108, in get_response
    response = middleware_method(request)
  File "/usr/local/lib/python2.7/dist-packages/django/middleware/common.py", line 74, in process_request
    if (not urlresolvers.is_valid_path(request.path_info, urlconf) and
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 646, in is_valid_path
    resolve(path, urlconf)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 521, in resolve
    return get_resolver(urlconf).resolve(path)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 365, in resolve
    for pattern in self.url_patterns:
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 401, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 395, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/omerouser/OMERO.server/lib/python/omeroweb/urls.py", line 84, in <module>
    __import__(urlmodule)
  File "/usr/local/lib/python2.7/dist-packages/omero_mapr/urls.py", line 36, in <module>
    DEFAULT_CONFIG = mapr_settings.CONFIG.iterkeys().next()
StopIteration


OmeroWeb config:

Code: Select all

Configuration:

    Configuration for OMERO.web takes place via the omero config commands. The
    configuration values which are checked are as below:

    Key                                  Default?  'Current value'
    omero.web.admins                     True      []
    omero.web.application_server         False     'wsgi-tcp'
    omero.web.application_server.host    True      '127.0.0.1'
    omero.web.application_server.max_re  True      0
    omero.web.application_server.port    True      4080
    omero.web.apps                       False     [u'omero_iviewer', u'omero_gallery', u'omero_forms', u'omero_figure', u'omero_mapr']
    omero.web.base_include_template      True      None
    omero.web.caches                     True      {u'default': {u'BACKEND': u'django.core.cache.backends.dummy.DummyCache'}}
    omero.web.chunk_size                 True      1048576
    omero.web.cors_origin_allow_all      True      False
    omero.web.cors_origin_whitelist      True      []
    omero.web.databases                  True      {}
    omero.web.debug                      True      False
    omero.web.index_template             True      None
    omero.web.logdir                     True      '/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/log'
    omero.web.login_logo                 True      '(unset)'
    omero.web.login_redirect             True      {}
    omero.web.login_view                 True      'weblogin'
    omero.web.middleware                 True      [{u'index': 1, u'class': u'django.middleware.common.BrokenLinkEmailsMiddleware'}, {u'index': 2, u'class': u'django.middleware.common.CommonMiddleware'}, {u'index': 3, u'class': u'django.contrib.sessions.middleware.SessionMiddleware'}, {u'index': 4, u'class': u'django.middleware.csrf.CsrfViewMiddleware'}, {u'index': 5, u'class': u'django.contrib.messages.middleware.MessageMiddleware'}]
    omero.web.open_with                  False     [[u'Image viewer', u'webgateway', {u'supported_objects': [u'image'], u'script_url': u'webclient/javascript/ome.openwith_viewer.js'}], [u'omero_iviewer', u'omero_iviewer_index', {u'supported_objects': [u'images', u'dataset', u'well'], u'script_url': u'omero_iviewer/openwith.js', u'label': u'OMERO.iviewer'}], [u'omero_figure', u'new_figure', {u'supported_objects': [u'images'], u'target': u'_blank', u'label': u'OMERO.figure'}]]
    omero.web.page_size                  True      200
    omero.web.ping_interval              True      60000
    omero.web.pipeline_css_compressor    True      None
    omero.web.pipeline_js_compressor     True      None
    omero.web.pipeline_staticfile_stora  True      'pipeline.storage.PipelineStorage'
    omero.web.prefix                     True      '(unset)'
    omero.web.public.cache.enabled       True      False
    omero.web.public.cache.key           True      'omero.web.public.cache.key'
    omero.web.public.cache.timeout       True      86400
    omero.web.public.enabled             True      False
    omero.web.public.get_only            True      True
    omero.web.public.password            True      '(unset)'
    omero.web.public.server_id           True      1
    omero.web.public.url_filter          True      <_sre.SRE_Pattern object at 0x7f4352f7a0c0>
    omero.web.public.user                True      '(unset)'
    omero.web.secret_key                 True      'M{msu-_9$P|Q-D;QK~%8jhZ\\npE3gT?b1{jc?T.>e(%sSm+@MO'
    omero.web.secure                     True      False
    omero.web.secure_proxy_ssl_header    True      []
    omero.web.server_list                True      [[u'localhost', 4064, u'omero']]
    omero.web.session_cookie_age         True      86400
    omero.web.session_cookie_domain      True      '(unset)'
    omero.web.session_cookie_name        True      '(unset)'
    omero.web.session_engine             True      'omeroweb.filesessionstore'
    omero.web.session_expire_at_browser  True      True
    omero.web.static_root                True      '/home/omerouser/OMERO.server-5.4.5-ice36-b83/lib/python/omeroweb/static'
    omero.web.static_url                 True      '/static/'
    omero.web.staticfile_dirs            True      []
    omero.web.template_dirs              True      []
    omero.web.thumbnails_batch           True      50
    omero.web.ui.center_plugins          False     [[u'Forms', u'forms/forms_init.js.html', u'omero_forms_panel'], [u'Forms', u'forms/forms_init.js.html', u'omero_forms_panel']]
    omero.web.ui.metadata_panes          True      [{u'index': 1, u'name': u'tag', u'label': u'Tags'}, {u'index': 2, u'name': u'map', u'label': u'Key-Value Pairs'}, {u'index': 3, u'name': u'table', u'label': u'Tables'}, {u'index': 4, u'name': u'file', u'label': u'Attachments'}, {u'index': 5, u'name': u'comment', u'label': u'Comments'}, {u'index': 6, u'name': u'rating', u'label': u'Ratings'}, {u'index': 7, u'name': u'other', u'label': u'Others'}]
    omero.web.ui.right_plugins           True      [[u'Acquisition', u'webclient/data/includes/right_plugin.acquisition.js.html', u'metadata_tab'], [u'Preview', u'webclient/data/includes/right_plugin.preview.js.html', u'preview_tab']]
    omero.web.ui.top_links               False     [[u'Data', u'webindex', {u'title': u'Browse Data via Projects, Tags etc'}], [u'History', u'history', {u'title': u'History'}], [u'Help', u'http://help.openmicroscopy.org/', {u'target': u'new', u'title': u'Open OMERO user guide in a new tab'}], [u'Forms Designer', u'omeroforms_designer', {u'target': u'new', u'title': u'Open OMERO.Forms in a new tab'}], [u'Forms Designer', u'omeroforms_designer', {u'target': u'new', u'title': u'Open OMERO.Forms in a new tab'}], [u'Figure', u'figure_index', {u'target': u'_blank', u'title': u'Open Figure in new tab'}]]
    omero.web.use_x_forwarded_host       True      False
    omero.web.viewer.view                True      'omeroweb.webclient.views.image_viewer'
    omero.web.webgateway_cache           True      '(unset)'
    omero.web.wsgi_args                  True      '(unset)'
    omero.web.wsgi_timeout               True      60
    omero.web.wsgi_workers               True      5


Omero Admin Diagnostics:

Code: Select all

================================================================================
OMERO Diagnostics (admin) 5.4.5-ice36-b83
================================================================================
       
Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      2.7.12    (/usr/bin/python)
Commands:   icegridnode --version          3.6.4     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.6.4     (/usr/bin/icegridadmin)
Commands:   psql --version                 9.6.9     (/usr/bin/psql)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 3311, enabled)
Server:     DropBox                        active (pid = 3324, enabled)
Server:     FileServer                     active (pid = 3325, enabled)
Server:     Indexer-0                      active (pid = 3326, enabled)
Server:     MonitorServer                  active (pid = 3328, enabled)
Server:     OMERO.Glacier2                 active (pid = 3329, enabled)
Server:     OMERO.IceStorm                 active (pid = 3337, enabled)
Server:     PixelData-0                    active (pid = 3344, enabled)
Server:     Processor-0                    active (pid = 3351, enabled)
Server:     Tables-0                       active (pid = 3354, enabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /home/omerouser/OMERO.server-5.4.5-ice36-b83/var/log exists
Log files:  Blitz-0.log                    489.0 MB      errors=3683 warnings=7228
Log files:  DropBox.log                    2.0 MB        errors=847  warnings=210
Log files:  FileServer.log                 5.0 KB       
Log files:  Indexer-0.log                  2.0 MB        errors=38   warnings=45 
Log files:  MonitorServer.log              3.0 MB        errors=3    warnings=6   
Log files:  OMEROweb.log                   8.0 MB        errors=461  warnings=13840
Log files:  PixelData-0.log                708.0 KB      errors=18   warnings=24 
Log files:  Processor-0.log                4.0 MB        errors=5    warnings=14 
Log files:  Tables-0.log                   82.0 KB       errors=6    warnings=27 
Log files:  TestDropBox.log                n/a
Log files:  master.err                     142.0 KB      errors=22   warnings=43 
Log files:  master.out                     0.0 KB       
Log files:  Total size                     511.68 MB


Environment:OMERO_HOME=(unset)             
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)         
Environment:OMERO_TMPDIR=(unset)           
Environment:PATH=/home/omerouser/bin:/home/omerouser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=(unset)               
Environment:LD_LIBRARY_PATH=(unset)       
Environment:DYLD_LIBRARY_PATH=(unset)     

OMERO SSL port:4064                           
OMERO TCP port:4063                           
OMERO data dir:'/OMERO'                       Exists? True   Is writable? True
OMERO temp dir:'/home/omerouser/omero/tmp'    Exists? True   Is writable? True   (Size: 0)

JVM settings: Blitz-${index}                -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index}              -Xmx1642m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index}            -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index}           -Xmx1642m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


Omero config:

Code: Select all

omero.data.dir=/OMERO
omero.db.name=comero_database
omero.db.pass=********
omero.db.user=omero_user
omero.web.application_server=wsgi-tcp
omero.web.apps=["omero_iviewer", "omero_gallery", "omero_forms", "omero_figure", "omero_mapr"]
omero.web.forms.priv.password=********
omero.web.forms.priv.user=formmaster
omero.web.open_with=[["Image viewer", "webgateway", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}], ["omero_iviewer", "omero_iviewer_index", {"supported_objects": ["images", "dataset", "well"], "script_url": "omero_iviewer/openwith.js", "label": "OMERO.iviewer"}], ["omero_figure", "new_figure", {"supported_objects": ["images"], "target": "_blank", "label": "OMERO.figure"}]]
omero.web.ui.center_plugins=[["Forms", "forms/forms_init.js.html", "omero_forms_panel"], ["Forms", "forms/forms_init.js.html", "omero_forms_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Forms Designer", "omeroforms_designer", {"target": "new", "title": "Open OMERO.Forms in a new tab"}], ["Forms Designer", "omeroforms_designer", {"target": "new", "title": "Open OMERO.Forms in a new tab"}], ["Figure", "figure_index", {"target": "_blank", "title": "Open Figure in new tab"}]]


Python packages installed:

Code: Select all

pip freeze
adium-theme-ubuntu==0.3.4
certifi==2018.4.16
chardet==3.0.4
decorator==4.0.6
Django==1.8.19
django-pipeline==1.3.20
django-redis==4.8.0
dnspython==1.12.0
funcsigs==0.4
futures==3.2.0
gunicorn==19.7.1
idna==2.7
image==1.5.24
importlib==1.0.4
Jinja2==2.8
Markdown==2.6.11
MarkupSafe==0.23
mock==1.3.0
netifaces==0.10.4
numexpr==2.4.3
numpy==1.11.0
omero-cli-render==0.3.0
omero-figure==4.0.0
omero-forms==1.0.2
omero-gallery==3.1.1
omero-iviewer==0.5.0
omero-mapr==0.2.3
omero-marshal==0.5.2
pbr==1.8.0
Pillow==3.1.2
pycrypto==2.6.1
pycurl==7.43.0
pygobject==3.20.0
pyliblzma==0.5.3
pysqlite==1.0.1
PyYAML==3.11
redis==2.10.6
reportlab==3.4.0
requests==2.19.1
rpm-python==4.12.0.1
scipy==0.17.0
six==1.10.0
tables==3.2.2
unity-lens-photos==1.0
urlgrabber==3.9.1
urllib3==1.23
virtualenv==15.0.1
yum-metadata-parser==1.1.4
zeroc-ice==3.6.4


On thing I realised when I read the older posts, is that echo $PYTHONPATH is empty. Is this relevant?

Any suggestions are welcome,
thnaks
Alex

[kennethgillen]

Hi alexr,

If this has just appeared with no config changes on your part, it could be the resource leak bug that existing in OMERO.web up to 5.4.6.

If you clear temp files out of the OMERO.web server, then restart OMERO.web, that should bring the OMERO.web client back.

There can be too many files for `rm *` to work, so you may want to use (carefully) use `find` with exec.

Code: Select all

sudo find /tmp -mindepth 1 -maxdepth 1 -name "sessionid*" -exec rm -f '{}'

Obviously this is a dangerous command, as I'm sure you appreciate! -name limits it to the session id files, and /tmp limits to the /tmp directory.

Best,

Kenny

[alexr]

Hi Kenny,
I just checked the /tmp folder. There is not much in there. There are no files with "sessionid" in their name. I also tried to issue your command, but I only get "missing argument for -exec.
Am I correct you mean the system tmp folder, located at the root of the file system?
Thanks
Alex

[kennethgillen]

Hi Alexr,

The /favicon.ico _could_ be a red herring, but - it would definitely be worth zipping up and sending us your whole OMEROweb.log and perhaps NGINX access & logs too.

You can submit them to http://qa.openmicroscopy.org.uk/qa/upload/ if you don't want to make them public to the world (given they contain IP addresses etc), and if doing that, as many of the other OMERO logs as you like, could be useful.

I'm surprised to see issues coming from `urlresolvers` for the favicon. For requests to that, I see in my NGINX logs:

Code: Select all

client-ip - - [22/Jun/2018:15:06:48 +0100] "GET /favicon.ico HTTP/1.1" 301 185 "-" "curl/7.46.0" "-"
client-ip - - [22/Jun/2018:15:06:48 +0100] "GET /favicon.ico HTTP/1.1" 302 5 "-" "curl/7.46.0" "-"
client-ip - - [22/Jun/2018:15:06:48 +0100] "GET /static/webgateway/img/ome.ico HTTP/1.1" 200 1150 "-" "curl/7.46.0" "-"

Unless you've been making changes to the system (patching, or changing OMERO.web configuration) I wouldn't expect OMEROweb to just stop working on it's own, unless there's something like a resource leak happening, hence why I thought to check for that known bug.

Yes, I did mean the O.S. /tmp directory - that's where I'd normally expect the file-based session files to appear. Seeing omero.web.session_engine True 'omeroweb.filesessionstore' in your OmeroWeb config output makes me think that the default file based session store is in use here, rather than something more advanced like Redis.

Colleagues who are experienced with mapr might be able to chime in with something concrete here looking at all the config you've already sent.

Many thanks,

Kenny

[alexr]

Hi Kenny,
I uploaded the requested log files. Also the ngnix log and access files. What I realised by looking at them is that there is a lot of unwanted access from all over the world. Maybe this is the reason for the troubles?
I have not changed anything on the web server or configuration. I am currently testing a python script that uses the BlitzGateway to connect to the server.
Thanks for your help, best wishes
Alex

[kennethgillen]

Hi Alex,

Could you please send more Blitz and NGINX logs?

What's interesting is around the time when the OMEROweb errors started to appear, i.e. 2018-06-21 14:13

It'd be really interesting to see what's going on with the rest of the system at that timepoint.

The /var/log/messages logfile might also give a clue.

Oh, and on the temp files front, I forgot the semicolon - sorry! i.e.

List the files (if there are any)..

Code: Select all

sudo find /tmp -mindepth 1 -maxdepth 1 -name "sessionid*" -exec ls -al '{}' \;


Delete the files listed ^

Code: Select all

sudo find /tmp -mindepth 1 -maxdepth 1 -name "sessionid*" -exec rm -f '{}' \;

But it won't have any effect if there aren't any sessionid file in there to delete, sorry.

Cheers,

Kenny

[kennethgillen]

Hello again,

Also, I can see that omero-mapr has been installed, but there seems to be no config for it in the output of "Omero config:" you supplied earlier.

If you haven't given mapr any config, it's likely to be the source of your problems.

If you're not using it, I'd try uninstalling it, and removing it from your OMERO.web config.

All the best,

Kenny

[ratamero]

Sorry to jump in like that - but I can confirm that installing omero.mapr and not having any config gave me a Bad Gateway error as well. Uninstalling mapr solved it. I hope that's the issue, since it's easily solvable!

[wmoore]

Files on QA https://www.openmicroscopy.org/qa2/qa/feedback/21560/

[alexr]

Dear Forum members,
thanks a lot for your help. Indeed it was the OMERO.mapr, that made the problem. After finishing the settings for OMERO.mapr, and restarting the omero web server it works again. I did obviously not see the omero.mapr effects after installation for more than a day and therefore missed that. Thanks again for the hint.
Alex