Upgrade from 5.0.1 to 5.0.6 do not work !

[rudy]

Hello everybody,

A few months ago, in order to test omero with several biomedical images, I installed omero (version OMERO.server-5.0.1-ice34-b21) on an openvz container/vm (debian 7 x86 on a proxmox server) and after following the installation instructions, it worked fine.

Recently, due to the security issue, I decided to upgrade to the last version (I followed the upgrade instructions) but the upgrade didn't work and I have this error.

-! 01/13/15 16:25:23.103 icegridnode: warning: exception occurred while deactiva
ting `OMERO.IceStorm' using process proxy:
Network.cpp:1243: Ice::ConnectionRefusedException:
connection refused: Connection refused
Could not create the Java virtual machine.
Could not create the Java virtual machine.
-! 01/13/15 16:25:39.208 OMERO.Glacier2: warning: unable to contact permissions
verifier `BlitzVerifier@BlitzAdapters'
Reference.cpp:1566: Ice::NoEndpointException:
no suitable endpoint available for proxy `BlitzVerifier -t @ BlitzAdapters'
-! 01/13/15 16:25:39.212 OMERO.Glacier2: warning: unable to contact session mana
ger `BlitzManager@BlitzAdapters'
Reference.cpp:1566: Ice::NoEndpointException:
no suitable endpoint available for proxy `BlitzManager -t @ BlitzAdapters'
Could not create the Java virtual machine.

Error occurred during initialization of VM
Could not reserve enough space for object heap

The Blitz log file was not created but it seems to be a problem with ICE. Do you have any idea what would be the reason of this error ? Maybe, I missed some point in the upgrade or even in the original installation .

After I rollbacked to the old version of omero, everything worked fine without changing anything.

I thank you in advance for your help.

Kind regards,
Rudy

[manics]

Hi Rudy

It sounds like you're having memory issues. Could you post the output of

Code: Select all

omero admin diagnostics
omero admin jvmcfg
omero config get


(Remove any passwords from the last command)

Could you also see if there's any clues in var/log/Blitz-0.log? Feel free to attach it.

Thanks, Simon

[rudy]

Hi Simon,

Thank you very much for your reply. I run the commands and you can see the results below.
NB: I only change the name of the main folder to switch between the two commands.
The file Blitz-0.log is not created.

================================================================================
OMERO Diagnostics 5.0.6-ice34-b53
================================================================================

Commands: java -version 1.6.0 (/usr/bin/java)
Commands: python -V 2.7.3 (/usr/bin/python)
Commands: icegridnode --version 3.4.2 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.4.2 (/usr/bin/icegridadmin)
Commands: psql --version 9.1.14 (/usr/bin/psql)

Server: icegridnode not started

Cannot list deployed applications.

Log dir: /home/omero/OMERO.server-5.0.1-ice34-b21/var/log exists

Log files: Blitz-0.log n/a
Log files: DropBox.log 11.0 KB errors=16 warnings=8
Log files: FileServer.log 2.0 KB
Log files: Indexer-0.log n/a
Log files: MonitorServer.log 3.0 KB
Log files: OMEROweb.log 12.0 KB errors=10 warnings=10
Log files: OMEROweb_request.log 0.0 KB
Log files: Processor-0.log 109.0 KB errors=57 warnings=28
Log files: Tables-0.log 118.0 KB errors=57 warnings=28
Log files: TestDropBox.log n/a
Log files: master.err 6.0 KB errors=0 warnings=21
Log files: master.out 2.0 KB errors=24 warnings=0
Log files: Total size 0.27 MB

Error while parsing logs

Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=(unset)
Environment:PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO data dir: '/OMERO' Exists? True Is writable? True
OMERO temp dir: '/home/omero/omero/tmp' Exists? True Is writable? True (Size: 0)
OMERO.web status... [NOT STARTED]

JVM Settings:
============
blitz=-Xmx2576m -XX:MaxPermSize=1g
indexer=-Xmx1717m -XX:MaxPermSize=1g
pixeldata=-Xmx2576m -XX:MaxPermSize=1g
repository=-Xmx1717m -XX:MaxPermSize=1g


Config get:
===========
omero.db.name=omero_database
omero.db.pass=xxxxx
omero.db.user=omero
omero.web.application_server=fastcgi-tcp
omero.web.force_script_name=/omero
omero.web.server_email=xxx@xxx

Maybe, there is some problem with the jvm memory (?) I will try to have a look at this because this functionnality is new.

Thank you for your help,
Kind regards,
Rudy

[rudy]

Thank you very much Simon for indicating me the new command jvmcfg. Indeed, it seems the problem was the implementation of jvmcfg by percent (new memory management since version 5.0.2).
By manually specifying the memory, it seems to work now.
I used an openvz container with debian x86, so, the memory for a process should be below 4gb even if a container can use more than 4Gb (my container can use til 16Gb ram ).

With 16Gb ram, what would be the best memory configuration ?

JVM Settings:
============
blitz=-Xmx2048m -XX:MaxPermSize=512m # Settings({'strategy': 'manual', 'percent': '10', 'perm_gen': '512m', 'heap_size': '2048m'})
indexer=-Xmx2048m -XX:MaxPermSize=512m # Settings({'heap_size': '2048m', 'perm_gen': '512m', 'strategy': 'manual'})
pixeldata=-Xmx2048m -XX:MaxPermSize=512m # Settings({'strategy': 'manual', 'perm_gen': '512m', 'heap_size': '2048m'})
repository=-Xmx2048m -XX:MaxPermSize=512m # Settings({'strategy': 'manual', 'perm_gen': '512m', 'heap_size': '2048m'})


Omerco config get
================
omero.db.name=omero_database
omero.db.pass=xxxx
omero.db.user=omero
omero.jvmcfg.heap_size=2048m
omero.jvmcfg.percent.blitz=10
omero.jvmcfg.perm_gen=512m
omero.jvmcfg.strategy=manual
omero.jvmcfg.strategy.indexer=manual
omero.web.application_server=fastcgi-tcp
omero.web.force_script_name=/omero
omero.web.server_email=xx@xx

Thank you again,
Rudy

[sbesson]

Hi Rudy,

on a production OMERO server with 16GB RAM, we have the following JVM memory settings in place:

Code: Select all

-bash-4.1$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx5900m -XX:MaxPermSize=1g # Settings({'percent': '35'})
indexer=-Xmx1685m -XX:MaxPermSize=1g
pixeldata=-Xmx1685m -XX:MaxPermSize=1g # Settings({'percent': '10'})
repository=-Xmx1685m -XX:MaxPermSize=1g
-bash-4.1$ bin/omero config get | grep jvmcfg
omero.jvmcfg.percent.blitz=35
omero.jvmcfg.percent.pixeldata=10


Note this OMERO.server does not have many large images so there the `Pixeldata` service is almost unused. If the server needed to manage large images, then the value of `omero.jvmcfg.percent.pixeldata` would need to be increased.

Best,
Sebastien

[rudy]

Hi Sebastien,

Thank you very much for this interesting information. I think the problem with my server is the used of a 32bits image (openvz) even if it can use 16Gb (for all processes) but for one process it should be less than 4Gb and currently the process java (32bits) is running with 3Gb ram ... So I'm currently at the limit of memoy. For information, I installed omero on a 32bits openvz container because we first tested this server on a old virtualization server (only 4Gb ram) which was upgraded a few months ago.

I'll try to create a 64bits omero server once I have time ... (maybe by using a qemu virtualization)

Thank you again for your reply.
Regards,
Rudy