Open Microscopy Environment

The OME community
https://www.openmicroscopy.org/community/

Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=8010

Page 1 of 1

Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 2:37 am
by dsudar
Hi all,
I had been looking forward to 5.2.2 since it has many improvements for my use. Most of the upgrade from 5.2.1 on Ubuntu 14.04.4 went without a problem and OMERO itself is running fine. However when starting OMERO.web, it claims that it started but diagnostics says otherwise:
omero@quid:~$ omero web start
Post-processed 'omeroweb.viewer.min.css' as 'omeroweb.viewer.min.css'
Post-processed 'omeroweb.viewer.min.js' as 'omeroweb.viewer.min.js'

0 static files copied to '/home/omero/OMERO.server/lib/python/omeroweb/static', 764 unmodified, 2 post-processed.
Clearing expired sessions. This may take some time... [OK]
Starting OMERO.web... [OK]
omero@quid:~$ omero admin diagnostics

================================================================================
OMERO Diagnostics 5.2.2-ice35-b17
================================================================================

Commands: java -version 1.7.0 (/usr/bin/java)
Commands: python -V 2.7.6 (/usr/bin/python)
Commands: icegridnode --version 3.5.1 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.5.1 (/usr/bin/icegridadmin)
Commands: psql --version 9.4.5 (/usr/bin/psql -- 2 others)

Server: icegridnode running
Server: Blitz-0 active (pid = 20957, enabled)
Server: DropBox active (pid = 20980, enabled)
Server: FileServer active (pid = 20987, enabled)
Server: Indexer-0 active (pid = 20989, enabled)
Server: MonitorServer active (pid = 20990, enabled)
Server: OMERO.Glacier2 active (pid = 20988, enabled)
Server: OMERO.IceStorm active (pid = 21012, enabled)
Server: PixelData-0 active (pid = 20993, enabled)
Server: Processor-0 active (pid = 21014, enabled)
Server: Tables-0 active (pid = 21030, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /home/omero/OMERO.server/var/log exists
Log files: Blitz-0.log 1.0 MB errors=18 warnings=1
Log files: DropBox.log 1.0 KB
Log files: FileServer.log 0.0 KB
Log files: Indexer-0.log 1.0 KB
Log files: MonitorServer.log 1.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 1.0 KB errors=0 warnings=6
Log files: OMEROweb_brokenrequest.lock 0.0 KB
Log files: OMEROweb_brokenrequest.log 0.0 KB
Log files: PixelData-0.log 3.0 KB
Log files: Processor-0.log 63.0 KB errors=18 warnings=0
Log files: Tables-0.log 0.0 KB
Log files: TestDropBox.log n/a
Log files: master.err 0.0 KB
Log files: master.out 0.0 KB
Log files: Total size 1.30 MB


Environment:OMERO_HOME=/home/omero/OMERO.server
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-7-openjdk-amd64/bin:/usr/share/Ice-3.5.1:/usr/lib/postgresql/9.4/bin:/home/omero/OMERO.server/bin:/home/omero/omero-csv-tools/bin
Environment:PYTHONPATH=/usr/lib/pymodules/python2.7:
Environment:ICE_HOME=/usr/share/Ice-3.5.1
Environment:LD_LIBRARY_PATH=/usr/share/java:/usr/lib:
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/data/OMERO' Exists? True Is writable? True
OMERO temp dir:'/home/omero/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [NOT STARTED]
Django version: 1.8.8
omero@quid:~$ omero web status
OMERO.web status... [NOT STARTED]
omero@quid:~$


In nginx's error.log it says:
2016/03/01 18:07:28 [error] 22000#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:07:33 [error] 22000#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:14:41 [error] 23927#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:20:13 [error] 23927#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"


and OMEROweb.log only has some warnings:
2016-03-02 01:51:30,705 WARNI [ py.warnings] (proc.21270) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 01:51:31,510 WARNI [ django.security.SuspiciousSession] (proc.21276) decode():108 Session data corrupted
2016-03-02 01:51:31,510 WARNI [ django.security.SuspiciousSession] (proc.21276) decode():108 Session data corrupted
2016-03-02 01:52:25,553 WARNI [ py.warnings] (proc.21717) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 01:56:50,530 WARNI [ py.warnings] (proc.22046) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 02:15:06,726 WARNI [ py.warnings] (proc.23957) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib


and when pulling up a webpage, I only get a 502 - Bad Gateway page.

I tried restarting nginx. Also tried regenerating the nginx config file but it was almost identical to the one I already had. None of the log entries indicate anything to me why OMERO.web is staying up.

Help please ....
Thanks,
- Damir

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 6:46 am
by atarkowska
Hi Damir

Could you try:

Code: Select all
ps aux | grep django.pid
to make sure there are no processes running?

Code: Select all
bin/omero config set omero.web.wsgi_args -- " --error-logfile=/home/omero/OMERO.server/var/log/error.log".

and see if there is anything in this log file?

Could you also give more details about your configuration bin/omero config get --hide-password

Ola

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 7:26 am
by dsudar
Hi Ola,

Thanks for the quick response.

Code: Select all
root@quid:~# ps aux | grep django.pid
root     30065  0.0  0.0  11740   948 pts/1    S+   23:18   0:00 grep --color=auto django.pid
root@quid:~#


After setting:
Code: Select all
bin/omero config set omero.web.wsgi_args -- " --error-logfile=/home/omero/OMERO.server/var/log/error.log"


that error.log remains empty:
Code: Select all
omero@quid:~$ cat /home/omero/OMERO.server/var/log/error.log
cat: /home/omero/OMERO.server/var/log/error.log: No such file or directory

Then tried to do "omero web start" but still nothing in that error.log

Code: Select all
omero@quid:~$ omero config get --hide-password
omero.data.dir=/data/OMERO
omero.db.name=omerodb
omero.db.pass=********
omero.db.user=omero
omero.web.application_server=wsgi-tcp
omero.web.apps=["autotag", "tagsearch", "figure"]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero@quid:~$


Thanks,
- Damir

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 7:53 am
by dsudar
.. and after setting the omero.web.wsgi_args, omero config get reports:

Code: Select all
omero@quid:~$ omero config get --hide-password
omero.data.dir=/data/OMERO
omero.db.name=omerodb
omero.db.pass=********
omero.db.user=omero
omero.web.application_server=wsgi-tcp
omero.web.apps=["autotag", "tagsearch", "figure"]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero.web.wsgi_args= --error-logfile=/home/omero/OMERO.server/var/log/error.log


But still nothing in the that error.log

I just looked it up and noticed that that relates to gunicorn. And gunicorn is not running:

Code: Select all
root@quid:~# ps aux | grep gunicorn
root     30672  0.0  0.0  11740   940 pts/1    S+   23:50   0:00 grep --color=auto gunicorn


And the rest of OMERO is perfectly fine. I can connect with Insight with no trouble.

Thanks,
- Damir

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 9:10 am
by atarkowska
Could you join us on IRC, either via IRC client or https://webchat.freenode.net/, channel #ome. I will try to help you directly.

Ola

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 9:40 am
by atarkowska
Damir

Sorry I haven't realized the timezone difference ;-). If you cannot make to IRC early morning, then

1. I understand that you tried:

Code: Select all
/home/omero/OMERO.server/bin/omero config set omero.web.wsgi_args -- ' --error-logfile=/home/omero/OMERO.server/var/log/error.log'
/home/omero/OMERO.server/bin/omero web start

then
Code: Select all
    root@quid:~# ps aux | grep gunicorn
    root     30672  0.0  0.0  11740   940 pts/1    S+   23:50   0:00 grep --color=auto gunicorn

no web workers are running and
Code: Select all
    omero@quid:~$ cat /home/omero/OMERO.server/var/log/error.log
    cat: /home/omero/OMERO.server/var/log/error.log: No such file or directory


correct?
Coudl you try to change log level
Code: Select all
cd /home/omero/OMERO.server/
$ bin/omero config set omero.web.wsgi_args -- ' --log-level=DEBUG --error-logfile=/home/omero/OMERO.server/var/log/error.log'
$ bin/omero web start

Could you paste the full output of bin/omero web start?

I noticed you are running some of these commands as root and some as omero user. Please make sure you are starting omeroweb as omero user.

2. As the above didn't help, could you try

Code: Select all
$ export PYTHONPATH=/home/omero/OMERO.server/lib/python:$PYTHONPATH
$ gunicorn --bind 127.0.0.1:4080  omeroweb.wsgi:application


does this give any error?

Ola

Re: Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

PostPosted: Wed Mar 02, 2016 3:55 pm
by dsudar
All,

I just had a support session with Ola who figured out the problem:
Ubuntu 14.04.4's default gunicorn version is 17.5 but OMERO requires 19.3+.

So we forced an upgrade with pip:

Code: Select all
root@quid:/home/omero/OMERO.server# pip install --upgrade -r share/web/requirements-py27-nginx.txt


and that upgraded it to the compatible versions of gunicorn and django.

Now works fine.
Thanks Ola!

- Damir
All times are UTC
Page 1 of 1
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/