OMERO.web startup problem

[claybreshears]

Encountered a problem while following the installation and set up instructions for OMERO.web at https://docs.openmicroscopy.org/omero/5 ... ce3.6.html

I have previously installed the OMERO.server (5.4.4 on CentOS 7 with Ice3.6) on the same machine and been able to use it from the OMERO.insight client.

For the OMERO.web installation, at the step where the root attempts to start the omero-web.service with the copied file /etc/systemd/system/omero-web.service in place:

Code: Select all

systemctl start omero-web.service

I get the following error:

Code: Select all

Job for omero-web.service failed because the control process exited with error code. See "systemctl status omero-web.service" and "journalctl -xe" for details.


Looking at the "status" I receive:

Code: Select all

[root@compute-0-6 ~]# systemctl status omero-web.service -l
● omero-web.service - OMERO.web
   Loaded: loaded (/etc/systemd/system/omero-web.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2018-03-19 13:25:55 PDT; 31s ago
  Process: 3553605 ExecStart=/home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web start (code=exited, status=2)

Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: downloaded a distribution, please provide which link you
Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: used.
Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: ********************************************************************************
Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: Debugging Info:
Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: --------------
Mar 19 13:25:55 compute-0-6.internal.com python[3553605]: CWD=/
Mar 19 13:25:55 compute-0-6.internal.com systemd[1]: omero-web.service: control process exited, code=exited status=2
Mar 19 13:25:55 compute-0-6.internal.com systemd[1]: Failed to start OMERO.web.
Mar 19 13:25:55 compute-0-6.internal.com systemd[1]: Unit omero-web.service entered failed state.
Mar 19 13:25:55 compute-0-6.internal.com systemd[1]: omero-web.service failed.


The nginx service is running and the 'curl' check earlier in the instructions was passed. I had tried this with the omero system user (omero) in and out of the virtual environment. Same results for both.

Any ideas on where I might have messed up the process or if I hadn't set a config file correctly? Can you tell what the "code=exited, status=2" is triggered by?

Thanks.

--clay

[manics]

Hi Clay. Could you clarify for me, does it fail for you in all situations, or only when running via systemd? In particular, do the following work when run as the omero system user on the command line? In both cases start from a fresh shell, not in a virtualenv.

Code: Select all

/home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web start


Code: Select all

. /home/omero/omerowebvenv/bin/activate
/home/omero/OMERO.py/bin/omero web start


Is there anything in your OMEROweb.log? Does turning on debug logging before starting give any more info?

Code: Select all

omero config set omero.web.debug True

[claybreshears]

The first command does not work. It generates the following:

Code: Select all

[omero@compute-0-6 ~]$ /home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web start
Post-processed 'omeroweb.viewer.min.css' as 'omeroweb.viewer.min.css'
Post-processed 'omeroweb.viewer.min.js' as 'omeroweb.viewer.min.js'

0 static files copied to '/home/omero/OMERO.py/lib/python/omeroweb/static', 604 unmodified, 2 post-processed.
Clearing expired sessions. This may take some time... [OK]
Starting OMERO.web... Traceback (most recent call last):
  File "/home/omero/OMERO.py/bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/home/omero/OMERO.py/lib/python/omero/cli.py", line 1620, in argv
    cli.invoke(args[1:])
  File "/home/omero/OMERO.py/lib/python/omero/cli.py", line 1095, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/omero/OMERO.py/lib/python/omero/cli.py", line 1172, in onecmd
    self.execute(line, previous_args)
  File "/home/omero/OMERO.py/lib/python/omero/cli.py", line 1254, in execute
    args.func(args)
  File "/home/omero/OMERO.py/lib/python/omero/install/windows_warning.py", line 26, in wrapper
    return func(self, *args, **kwargs)
  File "/home/omero/OMERO.py/lib/python/omero/plugins/web.py", line 81, in wrapper
    return func(self, *args, **kwargs)
  File "/home/omero/OMERO.py/lib/python/omero/plugins/web.py", line 601, in start
    rv = self.ctx.popen(args=runserver, cwd=location)  # popen
  File "/home/omero/OMERO.py/lib/python/omero/cli.py", line 1345, in popen
    stdout=stdout, stderr=stderr)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


The second does work and I can get the nice web client log in screen. (Even when I have this up and running, I can't log in due to a version mismatch error. But we can poke at that after this is put aside.)

I looked in the OMEROweb.log, but couldn't make head nor tails from it. Also, I'd been doing all the above experiments and have turned on the web debug True some time along the way. So, I noted the time as 23:30 UTC and tried to start with the "systemctl start omero-web.service" command to see if that would generate something in the log file. I didn't see anything with the time stamp. I've attached the log file for your perusal.

--clay

[manics]

Hi Clay, I forgot with the virtualenv the path isn't updated if it's not activated. That's why the systemd service contains an Environment line. Could you verify this works:

Code: Select all

PATH=/home/omero/omerowebvenv/bin:/bin:/usr/bin /home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web start

This should be equivalent to what the systemd service runs.

Simon

[claybreshears]

Simon -

Yes, that works. (I changed the space to a ':' before the last directory path and made the whole thing one command.)

It's not always well formatted and sometimes the graphics are missing, but I am presented with the login screen.

Does this give you a clue about what I need to set up with system root to enable the systemd service to work?

--clay

[manics]

Hi Clay

If the command works the systemd service should work! Can you:

• Reboot your system and start omero-web with systemd
• Check your journald logs for omero-web
  

  Code: Select all

  sudo journalctl -u omero-web

• If that doesn't show anything useful try browsing the whole journald logs
• Do you have SELinux enabled? If so check /var/log/audit/audit.log, if it looks like there are warnings run
  

  Code: Select all

  sudo cat /var/log/audit/audit.log | audit2why

• List anything else that's running on your system, or any relevant customisations to the base OS if you're not using a standard CentOS installation image.

Thanks, Simon

[claybreshears]

Had some local networking problems, so couldn't connect to machines reliably for a few days. But, back on the case.

The host system is not running SELinux.

Code: Select all

[root@compute-0-6 ~]# sestatus
SELinux status:                 disabled


Rebooting didn't help either. (In fact, the OMERO.server wasn't able to start from the system boot up. I expect these are related.)

Looking at the 'journalctl -xe' output on a wide screen, I saw these bits...

Code: Select all

[root@compute-0-6 ~]# journalctl -xe
Mar 29 13:38:24 compute-0-6.internal.com dbus-daemon[1014]: dbus[1014]: [system] Successfully activated service 'org.freedesktop.problems'
Mar 29 13:39:00 compute-0-6.internal.com polkitd[1070]: Registered Authentication Agent for unix-process:1457426:26268021 (system bus name :1.1190 [/usr/bin/pktty
. . .<snip>. . .
-- Unit omero-web.service has begun starting up.
Mar 29 13:39:11 compute-0-6.internal.com python[1457525]: ********************************************************************************
Mar 29 13:39:11 compute-0-6.internal.com python[1457525]: ERROR: Could not import omero.cli! (/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by


And I've seen that error before while I tried to install OMERO.server. Top solve that installation error, I set the LD_LIBRARY_PATH variable to /usr/local/lib64. Setting that in root and trying to start omero-web.service doesn't help.

Then I noticed that the path to the C++ library (as shown in the error message) is fixed to /lib64. The version of this library is '6.0.19' while the version that is installed in my /usr/local/lib64 is '6.0.22' and this latter version seems to be the one with the needed `CXXABI_1.3.8'.

Code: Select all

[root@compute-0-6 ~]# ls -l /lib64/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Mar 13 10:23 /lib64/libstdc++.so.6 -> libstdc++.so.6.0.19
[root@compute-0-6 ~]# ls -l /usr/local/lib64/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Jan 23  2017 /usr/local/lib64/libstdc++.so.6 -> libstdc++.so.6.0.22


If this is the cause of the errors I'm having starting the OMERO.server and omero-web.service, short of setting up the symbolic link in /lib64/libstdc++.so.6 to point at the same thing my /usr/local/lib64/libstdc++.so.6 points to, is there some way to fix the path issue in the start up scripts for these two components? If that's not the problem, any other ideas on what I should look into?

Thanks.

[claybreshears]

To answer the other question that was asked, it seems that there was an alteration to the system.

The installed version of gcc with Centos7 was (apparently) 4.8.5. Someone has "upgraded" the platform I'm using to host OMERO with gcc version 6.3.0. The path to the compiler is /usr/local/bin. This leads me to believe that it was this installation that installed the /usr/local/lib64 directory.

The error that I was having installing OMERO.server and now with booting up omero-web.service was "fixed" in gcc 4.9.2 (according to several online sources). Thus the trick of setting the LD_LIBRARY_PATH to /usr/local/lib64 fixed the installation. However, I'm still "stuck" with the hardwired /lib64 as the expected location for libstdc++.so.6 that omero-web.service requires (which does not have the needed version of module).

Just FYI.

[rleigh]

Dear Clay,

You might want to look at https://www.softwarecollections.org/en/ ... toolset-4/, for future reference. This provides a safe way to use a more recent compiler, without compromising the functionality of the base system. There's also a more recent https://www.softwarecollections.org/en/ ... toolset-7/ but I haven't tried it out yet.

The problem, as you have found, is that the old libstdc++ in /lib64 is missing symbols which are present in the new version in /usr/local/lib. devtoolset-4 works around this limitation by dynamically linking against the base /lib64 version, and also statically linking against the new version. This brings the missing symbols into any compiled libraries and executables, allowing them to run on the unmodified base system. This is the RedHat supported way of using a new compiler on the platform, and it's what we use ourselves to build on CentOS 7.

Simply source ". /opt/rh/devtoolset-4/enable" and you will get the new compiler version.

To fix up your system, I would suggest doing the following:

• pip uninstall all python modules built with the GCC in /usr/local
• remove the GCC from /usr/local
• install devtoolset-4
• activate devtoolset-4
• reinstall/rebuild all python modules

This should get you to a working state, but you will need to rebuild everything that was built with GCC 6, and remove every last trace of the compiler and its libraries from /usr/local.

Hope that helps,
Roger