Omero don't start after upgrade

[phm]

Hi,
I can't start omero after upgrade from 5.3.3 to 5.4.1.

Code: Select all

omero@omero:~$>omero admin start
Creating /home/omero/OMERO.server/var/master
Initializing /home/omero/OMERO.server/var/log
Creating /home/omero/OMERO.server/var/registry
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
................
.............
Failed to startup some components after 300 seconds
Calling "stop" on remaining components
Waiting on shutdown. Use CTRL-C to exit


omero@omero:~$>more OMERO.server/var/log/master.err
-! 12/06/17 15:26:57.450 OMERO.Glacier2: warning: unable to contact permissions verifier `BlitzVerifier@BlitzAdapters'
Reference.cpp:1637: Ice::NoEndpointException:
no suitable endpoint available for proxy `BlitzVerifier -t -e 1.1 @ BlitzAdapters'
-! 12/06/17 15:26:57.451 OMERO.Glacier2: warning: unable to contact session manager `BlitzManager@BlitzAdapters'
Reference.cpp:1637: Ice::NoEndpointException:
no suitable endpoint available for proxy `BlitzManager -t -e 1.1 @ BlitzAdapters'
!! 12/06/17 15:27:12.506 error: communicator not destroyed during global destruction.

omero@omero:~$>omero admin diagnostics

================================================================================
OMERO Diagnostics (admin) 5.4.1-ice36-b75
================================================================================

Commands: java -version 1.8.0 (/usr/bin/java)
Commands: python -V 2.7.5 (/usr/bin/python)
Commands: icegridnode --version 3.6.4 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.6.4 (/usr/bin/icegridadmin)
Commands: psql --version 9.6.5 (/usr/bin/psql)

Server: icegridnode not started

Log dir: /home/omero/OMERO.server/var/log exists
Log files: Blitz-0.log 20.0 KB errors=2 warnings=0
Log files: DropBox.log 1.0 KB errors=2 warnings=1
Log files: FileServer.log 0.0 KB
Log files: Indexer-0.log 15.0 KB errors=1 warnings=0
Log files: MonitorServer.log 0.0 KB
Log files: PixelData-0.log 15.0 KB errors=1 warnings=0
Log files: Processor-0.log 14.0 KB errors=4 warnings=8
Log files: Tables-0.log 16.0 KB errors=4 warnings=8
Log files: TestDropBox.log n/a
Log files: master.err 0.0 KB errors=1 warnings=2
Log files: master.out 0.0 KB
Log files: Total size 0.09 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=/tmp
Environment:PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/home/omero/OMERO.server/bin/
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

Cannot list deployed applications.
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/tmp' Exists? True Is writable? True (Size: 32819)

JVM settings: Blitz-${index} -Xmx5027m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx3351m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx5027m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx3351m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

I follow the procedures to do the upgrade as usual.

Philippe

[jmoore]

Hi Philippe,

can you share your Blitz-0.log with us?

~Josh

[phm]

Hi Philippe,

can you share your Blitz-0.log with us?

~Josh

Hi Josh,
I think that the problem come from my database. I did database upgrade before and may be it's not recommended. My database is 5.3.3 .
Philippe

[jmoore]

You'll definitely need to have your database on the correct version. Can you show us the content of the

Code: Select all

dbpatch

table?

You can start with the precheck from https://docs.openmicroscopy.org/omero/5.4.1/sysadmins/server-upgrade.html#upgrade-steps since it starts by determining the current version of the DB.

Cheers,
~Josh

[phm]

You'll definitely need to have your database on the correct version. Can you show us the content of the

Code: Select all

dbpatch

table?

You can start with the precheck from https://docs.openmicroscopy.org/omero/5.4.1/sysadmins/server-upgrade.html#upgrade-steps since it starts by determining the current version of the DB.

Cheers,
~Josh

Hi Josh,

The dbpatch command don't exist for me ??

omero@omero:~/OMERO.server$>dbpatch
-bash: dbpatch : commande introuvable

omero@omero:~/OMERO.server$>psql -h localhost -U db_omero omero_database < sql/psql/OMERO5.4__0/OMERO5.3__1-precheck.sql
Mot de passe pour l'utilisateur db_omero :
BEGIN
CREATE FUNCTION
ERREUR: wrong OMERO database version for this upgrade script
CONTEXTE : fonction PL/pgsql omero_assert_db_version(character varying,integer), ligne 12 à RAISE
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR: la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ROLLBACK

Philippe

[mtbc]

Dear Philippe,

The contents of the dbpatch table can be captured in a file:

Code: Select all

psql -h localhost -U db_omero -c 'SELECT * FROM dbpatch' -o /tmp/dbpatch omero_database

If you could zip up your /tmp/dbpatch file and your Blitz-0.log together and upload them to http://qa.openmicroscopy.org.uk/qa/upload/ then maybe we will be able to spot your problem.

Did anything odd happen when you were following the upgrade procedure steps, like unusual or concerning output from any of the commands that we had you type?

Cheers,
Mark

[phm]

Dear Philippe,

The contents of the dbpatch table can be captured in a file:

Code: Select all

psql -h localhost -U db_omero -c 'SELECT * FROM dbpatch' -o /tmp/dbpatch omero_database

If you could zip up your /tmp/dbpatch file and your Blitz-0.log together and upload them to http://qa.openmicroscopy.org.uk/qa/upload/ then maybe we will be able to spot your problem.

Did anything odd happen when you were following the upgrade procedure steps, like unusual or concerning output from any of the commands that we had you type?

Cheers,
Mark

Hi Marks,
Here the results of dbpatch on the database version before udpate (I restored the dump before upgrade):

omero@omero:~/OMERO.server$>more /tmp/dbpatch
id | currentpatch | currentversion | permissions | finished | message
| previouspatch | previousversion | external_id
----+--------------+----------------+-------------+----------------------------+----------------------------------------------------
------------------------------------------+---------------+-----------------+-------------
1 | 0 | OMERO5.0 | -52 | 2014-08-26 14:46:26.490616 | Database ready.
| 0 | OMERO5.0 |
2 | 1 | OMERO5.1 | -52 | 2015-04-13 13:49:10.384326 | Database updated.
| 0 | OMERO5.0 |
3 | 1 | OMERO5.1 | -52 | 2015-04-13 13:49:34.089304 | DB check DBEnumCheck: done for Bio-Formats revision
df26d52d34fe79589a79ff013ef0dc8f7a1bc0f7 | 1 | OMERO5.1 |
4 | 1 | OMERO5.1 | -52 | 2015-05-29 15:04:57.394871 | DB check DBEnumCheck: done for Bio-Formats revision
923bf020f921d8bfcef203f407fc94855e5b03f0 | 1 | OMERO5.1 |
5 | 1 | OMERO5.1 | -52 | 2015-05-29 15:04:57.413108 | DB check DBUnicodeUnitsCheck: done
| 1 | OMERO5.1 |
6 | 0 | OMERO5.2 | -52 | 2015-11-04 11:05:08.7575 | Database updated.
| 1 | OMERO5.1 |
7 | 0 | OMERO5.2 | -52 | 2015-11-04 11:05:55.999559 | DB check DBEnumCheck: done for Bio-Formats revision
6d102caae2bf4bd9284e81768b9d6990aefd6eb2 | 0 | OMERO5.2 |
8 | 0 | OMERO5.3 | -52 | 2017-08-17 18:00:24.724944 | Database updated.
| 0 | OMERO5.2 |
9 | 0 | OMERO5.3 | -52 | 2017-08-17 18:01:13.095333 | DB check DBEnumCheck: done for Bio-Formats revision
4c2f2b00a12aa8a3894c54459a4cf02d5b02e161 | 0 | OMERO5.3 |
10 | 0 | OMERO5.3 | -52 | 2017-08-17 18:01:13.758311 | DB check DBMappedEnumCheck: done for OMERO version
OMERO5.3__0 | 0 | OMERO5.3 |
(10 lignes)

And find the Blizt.log zipped in attachment.
Thanks
Philippe

[phm]

I restarted the "old" version of omero (5.3.3) omero start correctly however omeroweb send some errors :

omero@omero:~$>omero web start
Traceback (most recent call last):
File "manage.py", line 56, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 272, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 76, in load_command_class
return module.Command()
File "/usr/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 59, in __init__
self.storage.path('')
File "/usr/lib/python2.7/site-packages/django/utils/functional.py", line 213, in inner
self._setup()
File "/usr/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 311, in _setup
self._wrapped = get_storage_class(settings.STATICFILES_STORAGE)()
File "/usr/lib/python2.7/site-packages/django/core/files/storage.py", line 281, in get_storage_class
return import_by_path(import_path or settings.DEFAULT_FILE_STORAGE)
File "/usr/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_by_path
sys.exc_info()[2])
File "/usr/lib/python2.7/site-packages/django/utils/module_loading.py", line 21, in import_by_path
module = import_module(module_path)
File "/usr/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
__import__(name)
django.core.exceptions.ImproperlyConfigured: Error importing module pipeline.storage: "No module named pipeline.storage"
omero@omero:~$>omero admin diagnostics

================================================================================
OMERO Diagnostics 5.3.3-ice36-b63
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 5.4.0. See http://downloads.openmicroscopy.org/latest/omero for the latest version.

Commands: java -version 1.8.0 (/usr/bin/java)
Commands: python -V 2.7.5 (/usr/bin/python)
Commands: icegridnode --version 3.6.4 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.6.4 (/usr/bin/icegridadmin)
Commands: psql --version 9.6.5 (/usr/bin/psql)

Server: icegridnode running
Server: Blitz-0 active (pid = 19577, enabled)
Server: DropBox active (pid = 19601, enabled)
Server: FileServer active (pid = 19609, enabled)
Server: Indexer-0 active (pid = 19630, enabled)
Server: MonitorServer active (pid = 19610, enabled)
Server: OMERO.Glacier2 active (pid = 19634, enabled)
Server: OMERO.IceStorm active (pid = 19612, enabled)
Server: PixelData-0 active (pid = 19611, enabled)
Server: Processor-0 active (pid = 19614, enabled)
Server: Tables-0 active (pid = 19625, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /home/omero/OMERO.server/var/log exists
Log files: Blitz-0.log 193.0 MB errors=33 warnings=333
Log files: Blitz-0.log.1 524.0 MB errors=104 warnings=1907
Log files: DropBox.log 30.0 KB errors=20 warnings=15
Log files: FileServer.log 5.0 KB
Log files: Indexer-0.log 7.0 MB errors=832 warnings=286
Log files: MonitorServer.log 11.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 0.0 KB
Log files: PixelData-0.log 3.0 MB errors=457 warnings=171
Log files: Processor-0.log 2.0 MB errors=0 warnings=1
Log files: Processor-0.log.1 4.0 MB errors=38 warnings=85
Log files: Tables-0.log 128.0 KB errors=29 warnings=64
Log files: TestDropBox.log n/a
Log files: master.err 5.0 KB errors=6 warnings=5
Log files: master.out 0.0 KB
Log files: Total size 736.15 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=/tmp
Environment:PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/home/omero/OMERO.server/bin/
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/tmp' Exists? True Is writable? True (Size: 134059)

JVM settings: Blitz-${index} -Xmx5027m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx3351m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx5027m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx3351m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [NOT STARTED]
Django version: 1.6.11.6

And what's about Omero.py when we do the omero upgrade ???? Should we do Omero.py upgrade to ???

omero@omero:~$>ls -l
total 152952
drwx------. 4 omero omero 4096 23 août 17:54 omero
drwxrwxr-x 9 omero omero 4096 21 sept. 17:36 Omero_Backup
-rwxrw-r-- 1 omero omero 745 6 oct. 09:16 omero_clean_and_backup.sh
lrwxrwxrwx 1 omero omero 24 18 août 14:21 OMERO.py -> OMERO.py-5.3.3-ice36-b63
drwxr-xr-x 8 omero omero 4096 18 août 14:29 OMERO.py-5.3.3-ice36-b63
lrwxrwxrwx. 1 omero omero 29 17 août 15:56 OMERO.server -> OMERO.server-5.3.3-ice36-b63/
drwxr-xr-x. 9 omero omero 4096 17 août 15:58 OMERO.server-5.3.3-ice36-b63
drwxr-xr-x 9 omero omero 4096 22 nov. 20:41 OMERO.server-5.4.1-ice36-b75
-rw-rw-r-- 1 omero omero 156590179 22 nov. 21:42 OMERO.server-5.4.1-ice36-b75.zip
lrwxrwxrwx 1 omero omero 28 6 déc. 15:16 OMERO.server-new -> OMERO.server-5.4.1-ice36-b75
drwxr-xr-x 5 root root 4096 18 août 12:56 omerowebvenv

Chers,
Philippe

[kennethgillen]

I see omerowebvenv in the directory listing, but when you're trying to start OMERO.web, it doesn't look like you've got the virtualenv activated.

If the OMERO.web you're using is in OMERO.py, then yes, it'll need to be upgraded alongside OMERO.server.

Best,

Kenny

[phm]

Hi,
I'm go back to the version 5.3.3, the omero server and omero web work correctly, that mean that the database is ok for this version. Now if I want to upgrade to version 5.4.1 (omero server and omero.py) I have to check the database as described in the doc " Check ahead for upgrade issues". However I think there is a misunderstanding in the precheck command. Should we use sql script in the current version or the new version. For example if I run the preckeck from the current version there is not script for version 5.4.1 :

Code: Select all

omero@omero:~/OMERO.server$>psql -h localhost -U db_user omero_database < sql/psql/OMERO5.4__0/OMERO5.3__0-precheck.sql
-bash: sql/psql/OMERO5.4__0/OMERO5.3__0-precheck.sql: Aucun fichier ou dossier de ce type
omero@omero:~/OMERO.server$>ls sql/psql/
OMERO3__4    OMERO3A__11  OMERO3A__7   OMERO4.3__0  OMERO5.3DEV__1   OMERO5.3DEV__2  OMERO5.3DEV__8
OMERO3__5    OMERO3A__2   OMERO3A__8   OMERO4.4__0  OMERO5.3DEV__10  OMERO5.3DEV__3  OMERO5.3DEV__9
OMERO3__6    OMERO3A__3   OMERO3A__9   OMERO5.0__0  OMERO5.3DEV__11  OMERO5.3DEV__4
OMERO3__7    OMERO3A__4   OMERO4__0    OMERO5.1__1  OMERO5.3DEV__12  OMERO5.3DEV__5
OMERO3A__1   OMERO3A__5   OMERO4.1__0  OMERO5.2__0  OMERO5.3DEV__13  OMERO5.3DEV__6
OMERO3A__10  OMERO3A__6   OMERO4.2__0  OMERO5.3__0  OMERO5.3DEV__14  OMERO5.3DEV__7



And if I run the script from the folder of version 5.4.1 :

omero@omero:~/OMERO.server-5.4.1-ice36-b75$>psql -h localhost -U db_omero omero_database < sql/psql/OMERO5.4__0/OMERO5.3__0-precheck.sql
Mot de passe pour l'utilisateur db_omero :
BEGIN
CREATE FUNCTION
omero_assert_db_version
-------------------------

(1 ligne)

DROP FUNCTION
CREATE FUNCTION
CREATE FUNCTION
assert_db_server_prerequisites
--------------------------------

(1 ligne)

DROP FUNCTION
DROP FUNCTION
SELECT 484276
CREATE INDEX
CREATE FUNCTION
assert_no_filepath_array_duplicates
-------------------------------------

(1 ligne)

DROP FUNCTION
DROP MATERIALIZED VIEW
status
-----------------------------------------------------------
+
+
+
YOUR DATABASE IS READY FOR UPGRADE TO VERSION OMERO5.4__0+
+
+

(1 ligne)

ROLLBACK

It sound good..
Which is also not the case with :

Code: Select all

omero@omero:~/OMERO.server-5.4.1-ice36-b75$>psql -h localhost -U db_omero omero_database < sql/psql/OMERO5.4__0/OMERO5.3__1-precheck.sql
Mot de passe pour l'utilisateur db_omero :
BEGIN
CREATE FUNCTION
ERREUR:  wrong OMERO database version for this upgrade script
CONTEXTE : fonction PL/pgsql omero_assert_db_version(character varying,integer), ligne 12 à RAISE
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ERREUR:  la transaction est annulée, les commandes sont ignorées jusqu'à la fin du bloc
de la transaction
ROLLBACK



The database is well in UTF8 :

Code: Select all

postgres@omero:~$>psql -l
                                    Liste des bases de données
      Nom       | Propriétaire | Encodage | Collationnement | Type caract. |    Droits d'accès     
----------------+--------------+----------+-----------------+--------------+-----------------------
omero_database | db_omero     | UTF8     | fr_FR.UTF-8     | fr_FR.UTF-8  |
postgres       | postgres     | UTF8     | fr_FR.UTF-8     | fr_FR.UTF-8  |
template0      | postgres     | UTF8     | fr_FR.UTF-8     | fr_FR.UTF-8  | =c/postgres          +
                |              |          |                 |              | postgres=CTc/postgres
template1      | postgres     | UTF8     | fr_FR.UTF-8     | fr_FR.UTF-8  | =c/postgres          +
                |              |          |                 |              | postgres=CTc/postgres
(4 lignes)



Now should I have to upgrade the database ?? Because in the documentation the sentence is not clear :

Warning This section only concerns users upgrading from a 5.3 or earlier server. If upgrading from a server, you do not need to upgrade the database.

Philippe