recently one of our users noticed a problem when trying to create a movie export using the "publishing options" -> "make movie" function. However, he just got an error message saying there was "no processor" available.
Digging around on the machine, I figured that the "tmp" directory (<OMEROroot>/omero/tmp/) contains huge amounts of big files (files named "__omero_export__[0-9]{18}.ome.tif"), filling up the partition where the directory resides. In our case that's a separate one, containing nothing but the "omero/tmp/" and "var/log/" directories of OMERO (*not* the system's ones).
In addition, I found that the "processor" Python process ate up all the memory available on that machine (32GB overall):
- Code: Select all
$ ps fu -e | grep runProcessor
omeronas 3764 2.1 93.6 34488308 30774112 ? Sl Sep27 582:55 \_ python lib/python/runProcessor.py --Ice.Config=/home/omeronas/OMERO.server-4.4.5-ice34-b20/var/master/servers/Processor-0/config/config
So, where does all the data in the "tmp" directory come from, and what happened that prevented the directory from being cleaned up? Clearly, most of the stuff was placed there yesterday when our user tried to create the movies, but there is also some stuff sitting there already since march:
- Code: Select all
du --time -sh tmp/*
3,2G 2013-03-26 11:49 14980@bc2-omero01
245M 2013-09-11 17:20 28414@bc2-omero01.example.com
148K 2013-09-17 15:50 28448
20G 2013-10-15 14:05 3734@bc2-omero01.example.com
16G 2013-10-15 15:10 3764
Is the behaviour of the Python process expected, or is this rather a bug that was uncovered by running out of space in the tmp dir?
And last (but most important): how should we recover right now? I'd just try to stop the OMERO services as usual making sure the berserk python process is gone, (re)move the ome.tiffs from the "tmp" directory and then start the services again.
Many thanks,
Niko
