cannot upload images after upgrade

[bernie]

Hi,

we've just upgrade our omero server from:

omero v 5.1.2 running on centos 6 to omero 5.2.6 running on centos 7

13TB Repository help in attached storage.

The upgrade seemed to go well. Users can logon and view images. OMERO.web works. However users cannot upload new images. Uploads fails almost immediately status 'Preparing upload'.

In Blitz.log I see an associated error:

2016-11-27 20:26:19,697 ERROR [ o.h.e.def.AbstractFlushingEventListener] (.Server-11) Could not synchronize database state with session
org.hibernate.exception.SQLGrammarException: could not insert collection: [ome.model.fs.Fileset.jobLinks#32066]
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:92) ~[hibernate-core.jar:na]
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core.jar:na]
at org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1244) ~[hibernate-core.jar:na]
at org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:58) ~[hibernate-core.jar:na]

e.t.c.

A new test database can be created (using the attached storage as a new repository) with new users and they can import images as normal.

A complication is that the server is in AD in which there is an AD user omero. However, we are using a local omero account using the same uid gid as in the previous omero server before the upgrade, and these ids are different from the AD omero user. The server is configured so that local users take precedence. Also the local omero group as been rename omero_g. I mention this just in case it has a bearing.

We have a short period of time in which we can try and trouble shoot before we should revert to the previous install.

Any help would be greatly appreciated,

Bernie

[jmoore]

Hi Bernie,

In Blitz.log I see an associated error:

2016-11-27 20:26:19,697 ERROR [ o.h.e.def.AbstractFlushingEventListener] (.Server-11) Could not synchronize database state with session
org.hibernate.exception.SQLGrammarException: could not insert collection:

Can you send this entire file?

A complication is that the server is in AD in which there is an AD user omero. However, we are using a local omero account using the same uid gid as in the previous omero server before the upgrade, and these ids are different from the AD omero user. The server is configured so that local users take precedence. Also the local omero group as been rename omero_g. I mention this just in case it has a bearing.

Is it possible that the upgraded server no longer has write access to the /OMERO (or similar) directories?

Cheers,
~Josh

[bernie]

Good morning Josh,

thanks for your reply. Can you remind me of your upload site please and I'll upload the Blitz.log.

User omero can write to the repository:

bash-4.2$ whoami
omero
bash-4.2$ touch /OMERO/ManagedRepository/test
bash-4.2$ ls -l /OMERO/ManagedRepository/test
-rw-r--r-- 1 omero omero_g 0 Nov 28 08:11 /OMERO/ManagedRepository/test

I also tried a new user but had the same issue,

Bernie

[mtbc]

Dear Bernie,

One can zip the logs up and upload them to http://qa.openmicroscopy.org.uk/qa/upload/

Cheers,
Mark

[bernie]

The server is managed by puppet but this is off.

Also here's the diagnostics:

bash-4.2$ cd ~/OMERO.server
bash-4.2$ bin/omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
bash-4.2$ bin/omero admin diagnostics

================================================================================
OMERO Diagnostics 5.2.6-ice36-b35
================================================================================

Commands: java -version 1.8.0 (/bin/java)
Commands: python -V 2.7.5 (/bin/python)
Commands: icegridnode --version 3.6.3 (/bin/icegridnode)
Commands: icegridadmin --version 3.6.3 (/bin/icegridadmin)
Commands: psql --version 9.6.1 (/bin/psql)

Server: icegridnode running
Server: Blitz-0 active (pid = 1133, enabled)
Server: DropBox active (pid = 1171, enabled)
Server: FileServer active (pid = 1174, enabled)
Server: Indexer-0 active (pid = 1175, enabled)
Server: MonitorServer active (pid = 1215, enabled)
Server: OMERO.Glacier2 active (pid = 1178, enabled)
Server: OMERO.IceStorm active (pid = 1182, enabled)
Server: PixelData-0 active (pid = 1193, enabled)
Server: Processor-0 active (pid = 1204, enabled)
Server: Tables-0 active (pid = 1213, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /var/omero/OMERO.server-5.2.6-ice36-b35/var/log exists
Log files: Blitz-0.log 140.0 MB errors=450 warnings=112
Log files: DropBox.log 31.0 KB errors=2 warnings=12
Log files: FileServer.log 4.0 KB
Log files: Indexer-0.log 73.0 KB
Log files: MonitorServer.log 55.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 474.0 KB errors=0 warnings=3027
Log files: OMEROweb_brokenrequest.lock 0.0 KB
Log files: OMEROweb_brokenrequest.log 253.0 KB errors=0 warnings=1512
Log files: PixelData-0.log 54.0 KB
Log files: Processor-0.log 1.0 MB errors=316 warnings=55
Log files: Tables-0.log 26.0 KB errors=4 warnings=8
Log files: TestDropBox.log n/a
Log files: master.err 8.0 KB errors=5 warnings=3
Log files: master.out 0.0 KB
Log files: Total size 142.77 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/var/omero/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [ERROR] You are deploying OMERO.web using Apache and mod_wsgi. OMERO.web does not provide any management for the daemon process which communicates with Apache child processes using UNIX sockets to handle a request. Please check Apache directly.

[bernie]

Hi,

I've uploaded the Blitz.log to your upload site.

Interestingly I ran: bin/omero admin cleanse --dry-run /OMERO

I created a new user,' upgrade', to see if a new user would be able to import files (it can't) but the files appear in the list of those that would be removed (as well as some others):

Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-39.929/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-41.329/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-42.364/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-33.385/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-34.549/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-35.610/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-22.639/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-23.804/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-24.847/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-25.034/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-26.233/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-27.361/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-26-19.437/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-40-18.950/

Could it be that omero can read the db but not write to it?

Bernie

[mtbc]

Dear Bernie,

While this isn't of any immediate help to you, I notice that you seem to be running some 9.6 version of PostgreSQL and OMERO 5.2 supports only up to 9.4 as at http://www.openmicroscopy.org/site/supp ... postgresql - I wonder if that's a clue as to your troubles.

Cheers,
Mark

[bernie]

Hi Mark,

Hmm.. I shall drop down to 9.4. I thought I saw that 5.2 supported 9.4+ somewhere which I took to mean 9.4 or above, but I didn't see this, which is unequivocal. Thank you, Bernie

[bernie]

Thanks Mark, dropping the PostgreSQL version down to 9.4.3 fixed the problem,

Bernie

[mtbc]

Dear Bernie,

Thank you very much for letting us know that your OMERO system is now working as it should. Also, my apologies: I was slow to recognize that you probably ran into the issue described at http://www.openmicroscopy.org/site/supp ... s-optional so running the OMERO5.2-5.1-format-syntax-patch.sql script may have more easily solved your problem. While we do not yet do any formal testing with PostgreSQL beyond 9.4 this script provided with OMERO 5.2.6 fixes the only later-PostgreSQL issue we are aware of so far.

Cheers,
Mark