Omero Web: unhandled exception after upgrade

[Vanhecke]

Hi,

Lots of interesting output here... In summary:
- the Django fast CGI seems to not started
- there is no django module in python

As far as my understanding reaches: there seems to be another (faulty django) process running in the back. I suppose the solution would be reached If this could be killed (manually) and the web restarted on port 4080.

Error.log
about 39 error messages appear in /var/log/apache2/error.log when I access http://[server]/omero
All are pointing to /home/imaging/apps/OMERO/OMERO.server/var/omero.fcgi"

[server]/webclient/logout/ gives the same Unhandled exception ... and adds another 39 error messages

django

Code: Select all

omero web stop

gives the following output:

Code: Select all

Stopping OMERO.web... [FAILED]
Django FastCGI workers (PID 10348) not started?


followed by

Code: Select all

omero web start

gives:

Code: Select all

0 static files copied, 818 unmodified.
Starting OMERO.web... [OK]


(note that omero web stop gives now again the same output as above (including [FAILED]

Switching to develeopment does something:

Code: Select all

@svx-uo7680omero:~$ set omero.web.application_server development
@svx-uo7680omero:~$ omero config set omero.web True
@svx-uo7680omero:~$ omero web start
0 static files copied, 818 unmodified.
Starting OMERO.web... Validating models...

0 errors found
March 10, 2014 - 10:15:23
Django version 1.6, using settings 'omeroweb.settings'
Starting development server at http://localhost:4080/
Quit the server with CONTROL-C.
Error: That port is already in use.
@svx-uo7680omero:~$

Note that the version of Django is 1.6

I changed the port in the config.xml to 4081. The effect being that the webserver starts properly:

Code: Select all

imaging@svx-uo7680omero:~$ omero web start

0 static files copied, 818 unmodified.
Starting OMERO.web... Validating models...

0 errors found
March 10, 2014 - 10:51:55
Django version 1.6, using settings 'omeroweb.settings'
Starting development server at http://localhost:4081/
Quit the server with CONTROL-C.


However, the website did not appear using this URL:

Code: Select all

http://[server]:4081/omero

Python
The suggested line returns an error: no django module

Code: Select all

Python 2.7.3 (default, Jan  2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named django

Greetings,

Dimitri

[manics]

Could you try stopping your OMERO server (including web), and run the following to get a list of all python processes:

Code: Select all

ps -ef | grep python

I'm not sure whether your development server would be accessible from a remote host. If you can't run a browser directly could you try running

Code: Select all

curl -l http://localhost:4081

from a shell on your server, you should see a dump of the HTML.

Can you also try starting a python shell using omero:

Code: Select all

bin/omero shell

In [1]: import django

In [2]: print django.__file__


Thanks, Simon

[wmoore]

I wonder if you've still got web server running from before you tried to upgrade OMERO?

By listing processes (as Simon suggested) you should be able to see this and manually kill it. This will then free up port 4080.

If the development server is running as it seems, try to access the localhost url as it suggests:

http://localhost:4081/
or http://localhost:4081/webclient

Cheers,

Will.

[Vanhecke]

Hi Simon and Will,

I think we are getting very close here.

First IPython: print django.__file__ gives:
/home/imaging/apps/OMERO/OMERO.server/lib/python/django/__init__.pyc
(which looks like proper behaviour)

But the process status showed 5 lines that were fishy:

Code: Select all

imaging  24097     1  0 Feb20 ?        00:00:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/home/imaging/apps/OMERO/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400

There were 4 more like these. Interesting is also the date: 20th of February is when I updated the server.
I killed these (kill -9 24097) and restarted omero and the web server (omero admin start, omero web start). Got these outputs:

Code: Select all

imaging@svx-uo7680omero:~$ omero web start

0 static files copied, 818 unmodified.
Starting OMERO.web... Validating models...

0 errors found
March 10, 2014 - 14:10:45
Django version 1.6, using settings 'omeroweb.settings'
Starting development server at http://localhost:4080/
Quit the server with CONTROL-C.

(note that I reset the port to 4080 using omero config edit)
The process status now lists:

Code: Select all

imaging@svx-uo7680omero:~$ ps -ef | grep python
imaging   2839  2740  0 15:07 ?        00:00:00 python lib/python/fsDropBox.py --Ice.Config=/home/imaging/apps/OMERO/OMERO.server-5.0.0-ice34-b19/var/master/servers/DropBox/config/config
imaging   2842  2740  0 15:07 ?        00:00:00 python lib/python/fsServerFS.py --Ice.Config=/home/imaging/apps/OMERO/OMERO.server-5.0.0-ice34-b19/var/master/servers/FileServer/config/config
imaging   2846  2740  0 15:07 ?        00:00:00 python lib/python/fsServerMS.py --Ice.Config=/home/imaging/apps/OMERO/OMERO.server-5.0.0-ice34-b19/var/master/servers/MonitorServer/config/config
imaging   2872  2740  0 15:07 ?        00:00:00 python lib/python/runProcessor.py --Ice.Config=/home/imaging/apps/OMERO/OMERO.server-5.0.0-ice34-b19/var/master/servers/Processor-0/config/config
imaging   2881  2740  0 15:07 ?        00:00:00 python lib/python/runTables.py --Ice.Config=/home/imaging/apps/OMERO/OMERO.server-5.0.0-ice34-b19/var/master/servers/Tables-0/config/config
imaging   3442  3262  0 15:16 pts/3    00:00:00 grep python
imaging  31665 31523  0 11:38 pts/1    00:00:00 python /home/imaging/apps/OMERO/OMERO.server/bin/omero config edit
imaging  31866 31754  0 11:46 pts/2    00:00:00 python /home/imaging/apps/OMERO/OMERO.server/bin/omero config edit


So, this now at least shows the correct omero and ICE versions. Running curl on the localhost:

Code: Select all

[10/Mar/2014 14:18:33] "GET / HTTP/1.1" 301 0


(note: I had to open a second window to run this command since the web start did not gave my command line back). Starting the omeroweb (htto:/myserver:4080/omero) in a browser (I tried chrome and opera), however, gave an error:

Code: Select all

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

The error.log in var/log/apache2/ does not longer collects large error lines, but these 2 lines are added now each time I try to access the omero server through a browser:

Code: Select all

[Mon Mar 10 15:21:35 2014] [error] [client 134.21.78.134] (111)Connection refused: FastCGI: failed to connect to server "/home/imaging/apps/OMERO/OMERO.server/var/omero.fcgi": connect() failed
[Mon Mar 10 15:21:35 2014] [error] [client 134.21.78.134] FastCGI: incomplete headers (0 bytes) received from server "/home/imaging/apps/OMERO/OMERO.server/var/omero.fcgi"


More information about this error may be available in the server error log.[/code]

So, not longer the unhandled exception, but an internal server error. omero.fcgi exists but is indeed empty.

Cheers,

Dimitri

[manics]

Hi Dimitri

The development server is designed to be accessed directly, so going to port 4080 in your browser should work. It definitely won't work via Apache.

(note: I had to open a second window to run this command since the web start did not gave my command line back). Starting the omeroweb (htto:/myserver:4080/omero) in a browser (I tried chrome and opera), however, gave an error:

Code: Select all

    Internal Server Error

    The server encountered an internal error or misconfiguration and was unable to complete your request.

    Please contact the server administrator, webmaster@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.


That looks like an Apache error, which you shouldn't be seeing if you going to port 4080. Could you check what ports are in your Apache configs, and maybe restart everything?

Thanks

Simon

[Vanhecke]

Hi Simon,

It is with great pleasure and relief, that I can report that our omero server, including the web, is up and running as it should.

I stopped the omero web server (omero web stop).
I switched back to the fastcgi-tcp
Turned off the debug
restarted apache (apache2ctl restart).
and started the server (omero web start).

Note: The omero web server does not run in "fastcgi" configuration. But for now I am already happy we can access our data.

Thank you very much!


Dimitri

[manics]

Hi Dimitri, thanks for letting us know!
Simon