Problem after upgrade 5.06 to 5.1

[phm]

Hi,

After upgrading my server to 5.1 I can't start omero.
omero admin diagnostics :

================================================================================
OMERO Diagnostics 5.1.0-ice35-b40
================================================================================

Commands: java -version 1.7.0 (/usr/bin/java)
Commands: python -V 2.6.6 (/usr/bin/python)
Commands: icegridnode --version 3.5.1 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.5.1 (/usr/bin/icegridadmin)
Commands: psql --version 9.3.6 (/usr/pgsql-9.3/bin/psql -- 2 others)

Server: icegridnode running
Server: Blitz-0 active (pid = 9795, enabled)
Server: DropBox active (pid = 9815, enabled)
Server: FileServer active (pid = 9823, enabled)
Server: Indexer-0 active (pid = 9824, enabled)
Server: MonitorServer active (pid = 9825, enabled)
Server: OMERO.Glacier2 active (pid = 9827, enabled)
Server: OMERO.IceStorm active (pid = 9838, enabled)
Server: PixelData-0 active (pid = 9839, enabled)
Server: Processor-0 active (pid = 9845, enabled)
Server: Tables-0 inactive (disabled)
Server: TestDropBox inactive (enabled)

Log dir: /opt/OMERO.server/var/log exists

Log files: Blitz-0.log 158.0 KB errors=32 warnings=5
Log files: DropBox.log 9.0 KB errors=8 warnings=5
Log files: FileServer.log 1.0 KB
Log files: Indexer-0.log 8.0 MB errors=2196 warnings=549
Log files: MonitorServer.log 3.0 KB
Log files: OMEROweb.log 0.0 KB
Log files: OMEROweb_request.log 0.0 KB
Log files: PixelData-0.log 4.0 MB errors=1096 warnings=274
Log files: Processor-0.log 65.0 KB errors=34 warnings=18
Log files: Tables-0.log n/a
Log files: TestDropBox.log n/a
Log files: master.err 4.0 KB errors=6 warnings=8
Log files: master.out 0.0 KB
Log files: Total size 13.66 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=/tmp
Environment:PATH=/usr/pgsql-9.3/bin/:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin:/opt/OMERO.server/bin:/opt/dell/srvadmin/sbin:/usr/java/latest/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/tmp/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz -Xmx6400m -XX:MaxPermSize=1g # Settings({'system_memory': '32000', 'percent': '20'})
JVM settings: Indexer -Xmx3200m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})
JVM settings: Pixeldata -Xmx4800m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})
JVM settings: Repository -Xmx3200m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})

OMERO.web status... [RUNNING] (PID 9601)

And master.err

2015-04-13 14:00:35,863 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Executor.doWork -- ome.services.sessions.SessionManagerImpl.executeCheckPasswordRO[root]
2015-04-13 14:00:35,863 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Args: [null, InternalSF@702701196]
2015-04-13 14:00:35,867 INFO [ ome.security.basic.EventHandler] (l.Server-6) Auth: user=0,group=0,event=null(Sessions),sess=14417ecb-d786-40b6-a078-6f5dc159852f
2015-04-13 14:00:35,883 INFO [ org.perf4j.TimingLogger] (l.Server-6) start[1428926435863] time[19] tag[omero.call.success.ome.services.sessions.SessionManagerImpl$8.doWork]
2015-04-13 14:00:35,883 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Rslt: false
2015-04-13 14:00:35,885 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Executor.doWork -- java.lang.String.root[]
2015-04-13 14:00:35,885 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Args: [null, InternalSF@702701196]
2015-04-13 14:00:35,889 INFO [ ome.security.basic.EventHandler] (l.Server-6) Auth: user=0,group=0,event=null(Internal),sess=14417ecb-d786-40b6-a078-6f5dc159852f
2015-04-13 14:00:35,898 INFO [ org.perf4j.TimingLogger] (l.Server-6) start[1428926435885] time[13] tag[omero.call.success.ome.services.blitz.fire.PermissionsVerifierI$1.doWork]
2015-04-13 14:00:35,899 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Rslt: null
[omero@omero OMERO.server]$ tail var/log/master.err
from tables.utilsExtension import getPyTablesVersion, getHDF5Version
ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "lib/python/runTables.py", line 15, in <module>
import omero.tables
File "/opt/OMERO.server-5.1.0-ice35-b40/lib/python/omero/tables.py", line 32, in <module>
tables = __import__("tables") # Pytables
File "/usr/lib64/python2.6/site-packages/tables/__init__.py", line 30, in <module>
from tables.utilsExtension import getPyTablesVersion, getHDF5Version
ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory

Any idea ??
Philippe

[jmoore]

Could you send us more or all of var/log/Blitz-0.log?

Thanks,
~Josh.

[phm]

Could you send us more or all of var/log/Blitz-0.log?

Thanks,
~Josh.

tail var/log/Blitz-0.log
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) -------------------------------------------------
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) Stopping OMERO...
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) -------------------------------------------------
2015-04-13 14:30:06,641 INFO [ ome.services.pixeldata.PixelDataThread] ( main) Shutting down PixelDataThread
2015-04-13 14:30:06,641 INFO [ ome.services.eventlogs.EventLogQueue] ( main) Shutting down EventLogLoader
2015-04-13 14:30:06,641 INFO [ ome.services.fulltext.FullTextThread] ( main) Shutting down Full-Text Indexer
2015-04-13 14:30:06,642 INFO [ ome.services.eventlogs.EventLogQueue] ( main) Shutting down EventLogLoader
2015-04-13 14:30:06,642 INFO [ o.s.scheduler.SchedulerFactoryBean] ( main) Shutting down Quartz Scheduler
2015-04-13 14:30:06,645 INFO [.s.ShutdownSafeEhcacheManagerFactoryBean] ( main) Shutting down EHCache CacheManager
2015-04-13 14:30:06,646 INFO [ ome.services.blitz.Entry] ( main) Finished shutdown.

[sbesson]

Hi Philippe,

We may need lines before the server shutdown to find the nature of the error. Would it be possible for you to zip the Blitz logs and upload them to https://www.openmicroscopy.org/qa2/qa/upload/.

Best
Sebastien

[phm]

Hi Philippe,

We may need lines before the server shutdown to find the nature of the error. Would it be possible for you to zip the Blitz logs and upload them to https://www.openmicroscopy.org/qa2/qa/upload/.

Best
Sebastien

Hi Sebastien,

I uploaded the Blitz log

Thanks

Philippe

[jmoore]

Hi Philippe,

thanks for the log. It looks like there were some initial issues with the database having not been upgraded, but from what I can tell, the server is up and running:

Code: Select all

10994$ grep -E "(ERROR|Ready)" Blitz-0.log
2015-04-13 12:50:19,233 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 12:50:19,240 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:04:12,893 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:04:12,899 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:27:35,802 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:27:35,815 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:37:04,513 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:37:04,519 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:49:34,099 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:00:09,295 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:19:35,436 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:29:28,805 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:30:06,552 ERROR [                      omero.cmd.SessionI] (l.Server-3) Error destroying servant: 416cac6c-13eb-411f-a61f-ac77e85bf3ebomero.grid.SharedResources=ome.services.blitz.impl.SharedResourcesI@4190a6ce
2015-04-13 15:01:16,722 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.


What isn't working for you?
~Josh.

[phm]

Hi Philippe,

thanks for the log. It looks like there were some initial issues with the database having not been upgraded, but from what I can tell, the server is up and running:

What isn't working for you?
~Josh.

Hi Josh,

I did the database upgrade and cleaning and now the omero server is ok. However, the web part is down ???
Should I refactor the apache file ???


Thanks

Philippe

[sbesson]

Hi Philippe,

can you tell us more about your OMERO.web setup? Was the stanza regenerated using `bin/omero web config` as part of the upgrade? Is there anything either in the Web logs under var/log in the server or under the nginx/apache logs?

Best,
Sebastien

[phm]

Hi Philippe,

can you tell us more about your OMERO.web setup? Was the stanza regenerated using `bin/omero web config` as part of the upgrade? Is there anything either in the Web logs under var/log in the server or under the nginx/apache logs?

Best,
Sebastien

Hi Sebastien,

I think that's a apache config problem more than omero web config. My stanza was regenerated by omero web config apache > web_omero.conf (to be loaded after ssl.conf)

Code: Select all

###
# apache config for omero
# this file should be loaded *after* ssl.conf
#
# -D options to control configurations
#  OmeroWebClientRedirect - redirect / to /omero/webclient
#  OmeroWebAdminRedirect  - redirect / to /omero/webadmin
#  OmeroForceSSL - redirect all http requests to https

###
### Example SSL stanza for OMERO.web created 2015-04-14 09:50:59.536216
###

# Eliminate overlap warnings with the default ssl vhost
# Requires SNI (http://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI)
# support most later versions of mod_ssl and OSes will support it if you see
# "You should not use name-based virtual hosts in conjunction with SSL!!"
# or similar start apache with -D DISABLE_SNI and modify ssl.conf
#<IfDefine !DISABLE_SNI>
#  NameVirtualHost *:443
#</IfDefine>
#
## force https/ssl
#<IfDefine OmeroForceSSL>
#  RewriteEngine on
#  RewriteCond %{HTTPS} !on
#  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L]
#</IfDefine>
#
#<VirtualHost _default_:443>
#
#  ErrorLog logs/ssl_error_log
#  TransferLog logs/ssl_access_log
#  LogLevel warn
#
#  SSLEngine on
#  SSLProtocol all -SSLv2
#  SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
#  SSLCertificateFile /etc/pki/tls/certs/server.crt
#  SSLCertificateKeyFile /etc/pki/tls/private/server.key
#
#  # SSL Protocol Adjustments:
#  SetEnvIf User-Agent ".*MSIE.*" \
#    nokeepalive ssl-unclean-shutdown \
#    downgrade-1.0 force-response-1.0
#
#  # Per-Server Logging:
#  CustomLog logs/ssl_request_log \
#    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
#
#</VirtualHost>

###
### Stanza for OMERO.web created 2015-04-14 09:50:59.536216
###

FastCGIExternalServer "/opt/OMERO.server/var/omero.fcgi" -host 127.0.0.1:4080 -idle-timeout 60

<Directory "/opt/OMERO.server/var">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Directory "/opt/OMERO.server/lib/python/omeroweb/static">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Directory "/opt/OMERO.server/etc/templates/error">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Location /omero/>
    ErrorDocument 500 /omero/error/maintainance.html
</Location>

Alias /omero/error /opt/OMERO.server/etc/templates/error
Alias /static /opt/OMERO.server/lib/python/omeroweb/static
Alias /omero "/opt/OMERO.server/var/omero.fcgi/"


The OMEROweb.log is empty
When I start omero web I get these messages :

Code: Select all

Post-processed 'omeroweb.viewer.min.css' as 'omeroweb.viewer.min.css'
Post-processed 'omeroweb.viewer.min.js' as 'omeroweb.viewer.min.js'

0 static files copied, 745 unmodified, 2 post-processed.
Starting OMERO.web... [OK]


And apache error_log

Code: Select all

[Tue Apr 14 10:06:13 2015] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Tue Apr 14 10:06:13 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Apr 14 10:06:13 2015] [error] (13)Permission denied: FastCGI: apr_dir_open() failed
[Tue Apr 14 10:06:13 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Apr 14 10:06:13 2015] [notice] Digest: done
[Tue Apr 14 10:06:13 2015] [error] (13)Permission denied: FastCGI: apr_dir_open() failed
[Tue Apr 14 10:06:13 2015] [notice] FastCGI: process manager initialized (pid 3548)
[Tue Apr 14 10:06:13 2015] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Tue Apr 14 10:06:13 2015] [notice] mod_python: using mutex_directory /tmp
[Tue Apr 14 10:06:13 2015] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fastcgi/2.4.6 PHP/5.3.3 mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations


It seems that I have a problem with FastCGI authorization ???
And in the apache ssl_error_log

Code: Select all

[Tue Apr 14 10:13:42 2015] [error] [client 192.168.188.18] (13)Permission denied: access to /omero denied


Best

Philippe

[sbesson]

Hi Philippe

the following line in the error log:

Code: Select all

[Tue Apr 14 10:13:42 2015] [error] [client 192.168.188.18] (13)Permission denied: access to /omero denied


hints towards a file permissions issue. Could you check the Apache agent can access the /opt/OMERO.server/var directory? If not, you may want to:

• either fix the permissions of this folder to allow Apache to read it.
• or move omero.fcgi to another location that Apache can access, e.g. `/var/run/omero.fcgi`.

Best regards
Sebastien