All Imports failing...

[RickB]

Imports were working fine on our system up until yesterday. We had a user submit an import of 250 or so ndpi images, which started OK, but failed about 75% through... it looks like we ran into the servers open files limit. So I increased the open files limit (via limits.conf), restarted the server for good measure, and now all imports fail immediately, without even providing an exception. Nothing in master.err either, although I do see a strange message in Blitz.log about 5 repos (I would expect only 1)... Omero version 5.1.2-ice34-b45 - Thanks for any advice...

[kennethgillen]

Hi Rick,

To let the community help you further, can you please paste the output of the "config get", "admin diagnostics" and "fs repos" commands.

Also the relevant sections of your Blitz-0.log file would be fantastic, too. If you're not happy adding your Blitz log to pastebin or the likes, feel free to zip up all your logs to send to just the OME team at http://qa.openmicroscopy.org.uk/qa/upload/

Code: Select all

OMERO-CURRENT/bin/omero config get --hide-password

Code: Select all

OMERO-CURRENT/bin/omero admin diagnostics

Code: Select all

OMERO-CURRENT/bin/omero fs repos

All the best,

Kenny

[RickB]

Thanks very much, uploading logs now, here is the requested output:

-bash-4.1$ bin/omero config get --hide-password
omero.data.dir=/OMERO/
omero.db.name=omero
omero.db.pass=********
omero.db.poolsize=75
omero.db.user=omero
omero.ldap.base=dc=jax,dc=org
omero.ldap.config=true
omero.ldap.password=********
omero.ldap.urls=ldap://bhdc01.jax.org:3268
omero.ldap.user_filter=(memberOf=CN=OMERO-USERS,OU=Unix,OU=Groups,DC=jax,DC=org)
omero.ldap.user_mapping=omeName=sAMAccountName,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=CN=svc-omero,OU=ServiceAccounts,DC=jax,DC=org
omero.web.application_host=http://omeroweb.jax.org:80/
omero.web.application_server=fastcgi-tcp
omero.web.email_host=smtp.jax.org
omero.web.server_email=it-sysadmin@jax.org


-bash-4.1$ bin/omero admin diagnostics

================================================================================
OMERO Diagnostics 5.1.2-ice34-b45
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 5.1.3 See http://trac.openmicroscopy.org.uk/omero for the latest version

Commands: java -version 1.7.0 (/usr/bin/java)
Commands: python -V 2.6.6 (/usr/bin/python)
Commands: icegridnode --version 3.4.2 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.4.2 (/usr/bin/icegridadmin)
Commands: psql --version not found

Server: icegridnode running
Server: Blitz-0 active (pid = 10984, enabled)
Server: DropBox active (pid = 10998, enabled)
Server: FileServer active (pid = 11007, enabled)
Server: Indexer-0 active (pid = 11005, enabled)
Server: MonitorServer active (pid = 11015, enabled)
Server: OMERO.Glacier2 active (pid = 11021, enabled)
Server: OMERO.IceStorm active (pid = 11045, enabled)
Server: PixelData-0 active (pid = 11022, enabled)
Server: Processor-0 active (pid = 11039, enabled)
Server: Tables-0 active (pid = 11030, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /local/omerouser/OMERO.server-5.1.2-ice34-b45/var/log exists

Log files: Blitz-0.log 215.0 MB errors=517 warnings=2889
Log files: Blitz-0.log.1 524.0 MB errors=406 warnings=13754
Log files: Blitz-0.log.2 524.0 MB errors=92 warnings=11248
Log files: DropBox.log 43.0 KB errors=8 warnings=18
Log files: FileServer.log 6.0 KB
Log files: Indexer-0.log 10.0 MB errors=2357 warnings=465
Log files: MonitorServer.log 20.0 KB
Log files: OMEROweb.log n/a
Log files: PixelData-0.log 5.0 MB errors=1504 warnings=245
Log files: Processor-0.log 80.0 KB errors=28 warnings=24
Log files: Tables-0.log 85.0 KB errors=32 warnings=24
Log files: TestDropBox.log n/a
Log files: master.err 80.0 KB errors=240 warnings=104
Log files: master.out 0.0 KB
Log files: Total size 1280.40 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/java/jdk1.7.0_17/bin/
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO/' Exists? True Is writable? True
OMERO temp dir:'/local/omerouser/omero/tmp' Exists? True Is writable? True (Size: 24957028112)

JVM settings: Blitz -Xmx1238m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer -Xmx825m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Pixeldata -Xmx1238m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository -Xmx825m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [NOT STARTED]

-bash-4.1$ bin/omero fs repos
Using session 0af3664b-31a0-4108-8fea-48ec70405e3a (root@localhost:4064). Idle timeout: 10 min. Current group: Shopland Lab
# | Id | UUID | Type | Path
---+-------+--------------------------------------+--------+-----------------------------------------------------------
0 | 14 | ScriptRepo | Script | /local/omerouser/OMERO.server-5.1.2-ice34-b45/lib/scripts
1 | 15 | b69eafe9-ea9e-4b72-8b99-841ae2fdfa39 | Public | /OMERO
2 | 6951 | d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc | Public | /OMERO/ManagedRepository
3 | 22201 | a3feed69-f91e-490d-ab5c-bb2d0b6e8104 | Public | /OMERO
4 | 22251 | 5197275f-5d47-4ed7-a0e1-115133189771 | Public | /OMERO

[RickB]

logs uploaded, filename is log.tar.bz2

[mtbc]

Rick,

I am sorry to hear of your woes. In /OMERO there will be .omero directories: could you check if any of those have a lock file in them? (If so then they should be pretty obvious.)

viewtopic.php?f=5&t=7812&p=15792&hilit=repos#p15792 looks like it might be relevant. I am wondering if the multiple repositories issue is simply a direct result of the previous ulimit problems; at least, without newly imported files, all the current ones are presumably in your original repository.

Cheers,
Mark

[RickB]

Thanks Mark, I don't see any lock files in /OMERO/.omero/, there is a lock file in /OMERO/ManagedRepository/.omero/repository/e346fa72-0c94-477e-ac75-cdad37293e86

[RickB]

I think my repos are broken or at least mismatched, I see directories for the base repo and managed repository in the filesystem:

/OMERO/.omero/repository/e346fa72-0c94-477e-ac75-cdad37293e86
/OMERO/ManagedRepository/.omero/repository/e346fa72-0c94-477e-ac75-cdad37293e86

But these don't seem to match what's listed from the fs repo command:

bash-4.1$ /local/omerouser/OMERO.server/bin/omero fs repos
Using session 7a3bc733-9f12-4719-a87e-3f806a70b7f9 (root@localhost:4064). Idle timeout: 10 min. Current group: Shopland Lab
# | Id | UUID | Type | Path
---+-------+--------------------------------------+--------+-----------------------------------------------------------
0 | 14 | ScriptRepo | Script | /local/omerouser/OMERO.server-5.1.2-ice34-b45/lib/scripts
1 | 15 | b69eafe9-ea9e-4b72-8b99-841ae2fdfa39 | Public | /OMERO
2 | 6951 | d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc | Public | /OMERO/ManagedRepository
3 | 22201 | a3feed69-f91e-490d-ab5c-bb2d0b6e8104 | Public | /OMERO
4 | 22251 | 5197275f-5d47-4ed7-a0e1-115133189771 | Public | /OMERO

I haven't renamed or moved any repos, any ideas how to straighten this out?

Thanks again,
Rich

[mtbc]

Dear Rick,

Experimenting at my end, when I have a running OMERO I too have a .lock file and the UUID of its directory doesn't appear to match the repository UUIDs. Then, I shut down OMERO and the .lock file disappears.

If you use psql on the omero database, you might try,

Code: Select all

SELECT hash, name, path FROM originalfile WHERE mimetype = 'Repository';


to verify the output from the fs repos command, and,

Code: Select all

SELECT count(id), repo FROM originalfile WHERE mimetype <> 'Repository' AND repo IS NOT NULL GROUP BY repo;


to check how many files are in each repository.

It may be safe to stop the server and adjust the originalfile table: delete the database row recording that duplicate repository and, if it had any files in it, to update those files' repo column to move them to the original repository instead.

Do not try this unless you are confident about SQL and have fully backed up your OMERO installation: see https://www.openmicroscopy.org/site/sup ... store.html. I really am not sure what the result will be and I don't want you to lose any data.

Unfortunately those on our open-source side who know best how to deal with these issues are presently busy with MULTIMOT stuff in Belgium: my suggestion here is but a guess, but I appreciate how dreadfully inconvenient it is to have your OMERO installation so disabled, so I wanted to fully disclose to you whatever ideas I had in the meantime.

Cheers,

Mark

[RickB]

Thanks Mark, The original repo looks OK, and appears to contain all of our data as expected:

-bash-4.1$ psql -d omero
psql (9.3.6)
Type "help" for help.

omero=# SELECT hash, name, path FROM originalfile WHERE mimetype = 'Repository';
hash | name | path
--------------------------------------+-------------------+----------------------------------------------------
5197275f-5d47-4ed7-a0e1-115133189771 | OMERO | /
a3feed69-f91e-490d-ab5c-bb2d0b6e8104 | OMERO | /
ScriptRepo | scripts | /local/omerouser/OMERO.server-5.1.2-ice34-b45/lib/
b69eafe9-ea9e-4b72-8b99-841ae2fdfa39 | OMERO | /
d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc | ManagedRepository | /OMERO/
(5 rows)

omero=# SELECT count(id), repo FROM originalfile WHERE mimetype <> 'Repository' AND repo IS NOT NULL GROUP BY repo;
count | repo
-------+--------------------------------------
8778 | d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc
16 | ScriptRepo
(2 rows)



It seems like it should be safe to delete the DB rows for the duplicate repos, but my SQL confidence is low - backups can be made easily enough via snapshots, so I'm willing to try it with some guidance.

[kennethgillen]

backups can be made easily enough via snapshots, so I'm willing to try it with some guidance.

Hi Rich,

I'd just like to clarify your use of the word snapshot there. If you're talking about Virtual Machine snapshots, I'd recommend _not_ relying on that technology for a safe backup of OMERO. For several reasons, which I won't go into in this post.

The only way to get a reliable, safe backup of OMERO is as described in our Sysadmin docs:

https://www.openmicroscopy.org/site/sup ... store.html

1: Server needs to be off or guaranteed read-only (any writes ongoing during the backup make the state inconsistent),
2: Database dump,
3: Copy of the binary repository files on disk which match the state in the DB dump,
4: Copy of any server configuration in the OMERO binary distribution folder.

All the best,

Kenny