restart server after each user

[Alan H]

I currently am administering a 4.4 server running on OSX 10.8. Since this started out as a trial, I have insight, importer and editor installations on the same machine, and until today everybody in the lab just sat down at this one machine to import images and annotate them. Today I installed insight, editor and importer on 3 laptops (OSX 10.7 or 10.8) owned by people in the lab. They all connect to the Omero server; their users can log in.

Generally, whenever a user connects via insight, which is what they've been using so far, a second user cannot connect until the first user closes insight and we restart the server on the OSX terminal command line. Once that's done..shut down, then restart the server, the next user can connect. All of the current users are Administrators, and there are no groups, other than Administrator, on the server.

Oddly, today after installing editor/insight/importer on one laptop, I opened up editor and created a dummy protocol which I then saved to the server successfully in my own account, even though another user was still connected. I then connected as our PI, using her username and PW and that worked, too. I thought the problem was mysteriously solved until the first user logged off (she was connected remotely through her laptop) and POW....nobody can connect again until we stop and restart the server.

Any suggestions on how to resolve this? It's not feasible to stop and restart the omero server every time a new user needs to import images or edit.

[Alan H]

I referred to the Omero Administrator account, up above. .... properly, this is the Omero root account. sorry!!

..as in, I can't connect as Omero root until the server is stopped and started again. So the issue isn't just with users, it's with the root account as well.

[atarkowska]

Hi Alan

Could you please clarify how your account and group structure look like? You could easily get it from Webadmin. Something like:

groups:

Code: Select all

1    user    Private (rw----)    -    
0    system    Private (rw----)    -    
10    Lab1    Read-Only (rwr---)    description1   
20    Lab2    Read-Annotate (rwra--)    description2   


users:

Code: Select all

0    root root    root    -    admin active    
10    John Smith    jsmith    -    active   
20    John2 Smith2    jsmith2    -    ldap active


What user(s) do you use to connect to the server while failing? I need as much as possible details in order to reproduce it.

Could you also try to reproduce the same problem and once you get to the stage you can't log in just look into omero/var/log/Blitz-0.log what happens exactly?

Ola

[Alan H]

OK, I'm not running the web server, so not sure what to say about getting user and group details from webadmin.

OK regarding omero/var/log/Blitz-0.log

hm.... OK, the omero folder is in the home directory of the user who installed the package as (user = obrienlab).

There is no var directory in that directly.....omero/var doesn't exist.

obrienlab/omero/log exists, but there's no Blitz-0.log in there. The only file that's in there is omeroinsight.log

sr13-bd789141b2:obrienlab obrienlab$ ls
Desktop Downloads Library Music Public omero
Documents Dropbox Movies Pictures logfile var
sr13-bd789141b2:obrienlab obrienlab$ cd omero
sr13-bd789141b2:omero obrienlab$ ls -la
total 16
drwx------ 7 obrienlab staff 238 May 16 10:03 .
drwxr-xr-x+ 20 obrienlab staff 680 May 16 16:55 ..
-rw-r--r--@ 1 obrienlab staff 6148 Apr 29 14:10 .DS_Store
drwxr-xr-x 3 obrienlab staff 102 May 16 15:42 editor
drwxr-xr-x 2 obrienlab staff 68 May 16 10:03 filesOmero
drwxr-xr-x 3 obrienlab staff 102 Apr 16 14:37 log
drwx------ 3 obrienlab staff 102 Apr 15 17:42 tmp
sr13-bd789141b2:omero obrienlab$ cd log
sr13-bd789141b2:omero obrienlab$ ls
sr13-bd789141b2:log obrienlab$ omeroinsight.log




OK, so now what? I bet I'm looking in the wrong place!

[atarkowska]

Hi Alan

There is no problem you are not running webserver and you cannot access Webadmin. You can go to the database directly and run the following query:

Code: Select all

select e.id, e.omename, g.id, g.name, g.permissions from experimenter as e ,experimentergroup as g , groupexperimentermap as gem
where gem.parent = g.id and gem.child = e.id

omero/var/log/Blitz-0.log refers to the log on the server side. You can find server log by the following command:

Code: Select all

find / -name Blitz-0

The one you mentioned is client specific.

What version of OMERO do you use exactly?
Ola

[Alan H]

version is 4.4.6

OK, found Blitz0.log, it's...uh...quite large!! However, there are a series of exceptions that show up...lots of them. They are "bean creation exceptions".. ------Here's the first one in the cascade of "bean errors" ---------

2013-05-21 10:33:49,044 INFO [ng.ShutdownSafeEhcacheManagerFactoryBean] ( main) Shutting down EHCache CacheManager
2013-05-21 10:33:49,044 ERROR [ ome.services.blitz.Entry] ( main) Error on startup.
org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [OMERO.blitz], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'OMERO.blitz' defined in URL [jar:file:/usr/local/Cellar/omero/4.4.6/lib/server/blitz.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'ome.server' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/usr/local/Cellar/omero/4.4.6/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'groupChmodStrategy' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'omeroSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omeroSessionFactory' defined in class path resource [ome/services/hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [ome/services/hibernate.xml]: Invocation of init method failed; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection
at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:409)
at ome.system.OmeroContext.getInstance(OmeroContext.java:203)
at ome.services.blitz.Entry.start(Entry.java:233)
at ome.services.blitz.Entry.main(Entry.java:151)

-----------Since I last wrote, we are having problems getting the server to start. First things, first. Oh, joy.-----------


2013-05-21 10:33:44,771 INFO [ ome.services.blitz.Entry] ( main) Creating OMERO.blitz. Please wait...
2013-05-21 10:33:46,041 INFO [ng.ShutdownSafeEhcacheManagerFactoryBean] ( main) Initializing EHCache CacheManager
2013-05-21 10:33:48,037 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Found location in errorTimes: -1
2013-05-21 10:33:48,037 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Removing 0 from errorTimes
2013-05-21 10:33:48,037 WARN [ome.services.db.SelfCorrectingDataSource] ( main) Registering error with list: Current size: 0
2013-05-21 10:33:48,037 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Sleeping for 0 then retry: 1
2013-05-21 10:33:48,038 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Found location in errorTimes: -1
2013-05-21 10:33:48,038 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Removing 0 from errorTimes
2013-05-21 10:33:48,038 WARN [ome.services.db.SelfCorrectingDataSource] ( main) Registering error with list: Current size: 1
2013-05-21 10:33:48,038 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Sleeping for 1000 then retry: 2
2013-05-21 10:33:49,039 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Found location in errorTimes: -1
2013-05-21 10:33:49,039 INFO [ome.services.db.SelfCorrectingDataSource] ( main) Removing 0 from errorTimes
2013-05-21 10:33:49,040 WARN [ome.services.db.SelfCorrectingDataSource] ( main) Registering error with list: Current size: 2
2013-05-21 10:33:49,040 ERROR [ome.services.db.SelfCorrectingDataSource] ( main) Failed to acquire connection after retries=3
java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource 3fa3e287-8aca-49e9-ba42-4d84fc362f74 with 0 connection(s) (0 still available)

[Alan H]

Aaaaand...finally, I suppose this might give a clue, huh?? sheesh....

-bash: bin/omero: No such file or directory
sr13-bd789141b2:~ obrienlab$ omero admin diagnostics

================================================================================
OMERO Diagnostics 4.4.6
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 4.4.7 See http://trac.openmicroscopy.org.uk/omero for the latest version

Commands: java -version 1.6.0 (/usr/bin/java)
Commands: python -V 2.7.2 (/usr/bin/python)
Commands: icegridnode --version 3.3.1 (/usr/local/bin/icegridnode)
Commands: icegridadmin --version 3.3.1 (/usr/local/bin/icegridadmin)
Commands: psql --version 9.1.5 (/usr/bin/psql -- 2 others)

Server: icegridnode not started

Cannot list deployed applications.

Log dir: /usr/local/Cellar/omero/4.4.6/var/log exists

Log files: Blitz-0.log 147.0 MB errors=20535 warnings=5948
Log files: DropBox.log 78.0 KB errors=92 warnings=47
Log files: FileServer.log 16.0 KB
Log files: Indexer-0.log 427.0 MB errors=147672 warnings=43
Log files: MonitorServer.log 17.0 KB
Log files: OMEROweb.log n/a
Log files: PixelData-0.log 430.0 MB errors=148334 warnings=24
Log files: Processor-0.log 1.0 MB errors=444 warnings=3107
Log files: Processor-0.log.1 4.0 MB errors=1246 warnings=8724
Log files: Processor-0.log.2 4.0 MB errors=1246 warnings=8720
Log files: Processor-0.log.3 4.0 MB errors=1237 warnings=8690
Log files: Tables-0.log n/a
Log files: TestDropBox.log n/a
Log files: master.err 55.0 KB
Log files: master.out 0.0 KB
Log files: Total size 1022.44 MB

Parsing Blitz-0.log:[line:119] => Server restarted <=
Parsing Blitz-0.log:[line:3471] => Server restarted <=
Parsing Blitz-0.log:[line:14304] => Server restarted <=
Parsing Blitz-0.log:[line:21255] => Server restarted <=
Parsing Blitz-0.log:[line:133743] => Server restarted <=
Parsing Blitz-0.log:[line:136444] => Server restarted <=
Parsing Blitz-0.log:[line:186985] => Server restarted <=
Parsing Blitz-0.log:[line:194426] => Server restarted <=
Parsing Blitz-0.log:[line:196700] => Server restarted <=
Parsing Blitz-0.log:[line:200563] => Server restarted <=
Parsing Blitz-0.log:[line:203097] => Server restarted <=
Parsing Blitz-0.log:[line:233084] => Server restarted <=
Parsing Blitz-0.log:[line:279374] => Server restarted <=
Parsing Blitz-0.log:[line:319097] => Server restarted <=
Parsing Blitz-0.log:[line:376469] => Server restarted <=
Parsing Blitz-0.log:[line:413762] => Server restarted <=
Parsing Blitz-0.log:[line:489790] => Server restarted <=
Parsing Blitz-0.log:[line:544826] => Server restarted <=
Parsing Blitz-0.log:[line:585351] => Server restarted <=
Parsing Blitz-0.log:[line:627484] => Server restarted <=
Parsing Blitz-0.log:[line:638966] => Server restarted <=
Parsing Blitz-0.log:[line:678062] => Server restarted <=
Parsing Blitz-0.log:[line:709984] => Server restarted <=
Parsing Blitz-0.log:[line:722572] => Server restarted <=
Parsing Blitz-0.log:[line:743091] => Server restarted <=
Parsing Blitz-0.log:[line:759813] => Server restarted <=
Parsing Blitz-0.log:[line:764634] => Server restarted <=
Parsing Blitz-0.log:[line:777462] => Server restarted <=
Parsing Blitz-0.log:[line:791603] => Server restarted <=
Parsing Blitz-0.log:[line:835839] => Server restarted <=
Parsing Blitz-0.log:[line:865646] => Server restarted <=
Parsing Blitz-0.log:[line:903072] => Server restarted <=
Parsing Blitz-0.log:[line:1112184] => Server restarted <=
Parsing Blitz-0.log:[line:1141900] => Server restarted <=
Parsing Blitz-0.log:[line:1149472] => Server restarted <=
Parsing Blitz-0.log:[line:1154399] => Server restarted <=
Parsing Blitz-0.log:[line:1160266] => Server restarted <=
Parsing Blitz-0.log:[line:1174784] => Server restarted <=
Parsing Blitz-0.log:[line:1184051] => Server restarted <=
Parsing Blitz-0.log:[line:1211763] => Server restarted <=
Parsing Blitz-0.log:[line:1212796] => Server restarted <=
Parsing Blitz-0.log:[line:1214101] => Server restarted <=
Parsing Blitz-0.log:[line:1216101] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1216284] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1216525] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1216766] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1217007] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1217248] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1217489] Your postgres hostname and/or port is invalid
Parsing Blitz-0.log:[line:1217730] Your postgres hostname and/or port is invalid

Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO data dir: '/Users/obrienlab/var/OMERO.data' Exists? True Is writable? True
OMERO.web status... [NOT STARTED]

[Alan H]

All right, I tried initializing the postgre sql database - no love...


sr13-bd789141b2:var obrienlab$ psql -h localhost -U omero -W USERNAME
Password for user omero:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (fe80::1) and accepting
TCP/IP connections on port 5432?


-------------------

I've checked to see if the server is listening on Port 5432

sr13-bd789141b2:~ obrienlab$ netstat -an | egrep '5432.*LISTEN'

and it's not. GRRR. Found nothing. So I found postgresql.conf and took a look.

sr13-bd789141b2:postgres obrienlab$ cat postgresql.conf

here's the relevant stuff.

#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

#listen_addresses = 'localhost' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
#port = 5432 # (change requires restart)
max_connections = 20 # (change requires restart)
# Note: Increasing max_connections costs ~400 bytes of shared memory per
# connection slot, plus lock space (see max_locks_per_transaction).
#superuser_reserved_connections = 3 # (change requires restart)
#unix_socket_directory = '' # (change requires restart)
#unix_socket_group = '' # (change requires restart)
#unix_socket_permissions = 0777 # begin with 0 to use octal notation
# (change requires restart)
#bonjour = off # advertise server via Bonjour
# (change requires restart)
#bonjour_name = '' # defaults to the computer name
# (change requires restart)

# - Security and Authentication -

#authentication_timeout = 1min # 1s-600s
#ssl = off # (change requires restart)
#ssl_ciphers = 'ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH' # allowed SSL ciphers
# (change requires restart)
#ssl_renegotiation_limit = 512MB # amount of data between renegotiations
#ssl_cert_file = 'server.crt' # (change requires restart)
#ssl_key_file = 'server.key' # (change requires restart)
#ssl_ca_file = '' # (change requires restart)
#ssl_crl_file = '' # (change requires restart)
#password_encryption = on
#db_user_namespace = off

# Kerberos and GSSAPI
#krb_server_keyfile = ''
#krb_srvname = 'postgres' # (Kerberos only)
#krb_caseins_users = off

# - TCP Keepalives -
# see "man 7 tcp" for details

#tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds;
# 0 selects the system default
#tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds;
# 0 selects the system default
#tcp_keepalives_count = 0 # TCP_KEEPCNT;
# 0 selects the system default




-------------------------

I don't see anything wrong! Now, what?

[wmoore]

Previously on my Mac I've had problems with postgres failing to restart when my laptop has not been shut down properly. So psql fails etc.

A fix that has worked for me (if I remember correctly) is to remove the pid file

Code: Select all

$ cat /usr/local/var/postgres/postmaster.pid
253
/usr/local/var/postgres
  5432001     65536
$ rm /usr/local/var/postgres/postmaster.pid
$ psql omero
psql (9.0.4)
Type "help" for help.

omero=#



[mtbc]

Did you try checking for any clues in var/postgres/server.log?