How to speed-up Pyramid data creation?

[rpoehlmann]

Dear OME-Team,

unfortunately we are facing once again some issues with Pyramid data creation on our OMERO server.

A user discovered some TIF images that he had imported into OMERO before Christmas and for which no Pyramid data were calculated. When having a closer look at our system it turned out that apparently since quite some time no Pyramid data had been created at all.

After a restart of the OMERO services, Pyramid data creation re-started as well, but now there seems to be an substantial backlog of unprocessed data and overall progress is very slowly.

A (truncated) PixelData Log file is attached.

Code: Select all

JVM Settings:
============
blitz=-Xmx96000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '40', 'strategy': 'percent'})
indexer=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})
pixeldata=-Xmx72000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '30', 'strategy': 'percent'})
repository=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})


PixelData Settings:

Code: Select all

omero.pixeldata.cron=* * * * * ?
omero.pixeldata.repetitions=2
omero.pixeldata.threads=12


Today, I tried to better understand the coresponding parameters and started playiing around with them, however I was not able to substantially improve Pyramid data creation speed.
Our server has 12 CPU cores, 256 GB RAM, is running CentOS 7.1, OMERO 5.2.5-ice35. The OMERO data store resides on a GPFS file-system.

• I played around with the no. of pixeldata threads, however they seem to stay always the same independent from the actual settings I chose (or I did something wrong which I of course cannot exclude). I can always see 5 _controlling_ threads and a total no. of threads via "ps -eLF" with a name "-Domero.name=PixelData-0" of 42?
  
• I understood the "omero.pixeldata.repetitions=2" option in a way that it could be beneficial to deal with a backlog of pixel to process and therefore changed it to "2". However, no effect as far as I can judge compared to the default value before. I tried to find more info about this setting but did not really succeed. What am I doing wrong here?
• I do not really understand in which order missing pixel data will be processed. When looking at the currently processed data as they appear in the PixelData-0.log file, it seems to be a real mixture of older and newer data, and not e.g. from the oldest towards the current data?
• Processing seems to be very slow as seen by an 1.9 MB file which took 15 min. to complete (extract from PixelData-0.log):
  Import Date: 2016-12-14 16:21:04
  Dimensions (XY): 1600 x 1200
  Pixels Type: uint8
  Pixels Size (XYZ) (µm):
  84.67 x 84.67
  Z-sections/Timepoints: 1 x 1
  

  Code: Select all

  du -sh /export/omero/OMERO/ManagedRepository/schibe02_2704/2016-12/14/17-21-03.765/008-HA_HA.tif
1.9M   /export/omero/OMERO/ManagedRepository/schibe02_2704/2016-12/14/17-21-03.765/008-HA_HA.tif
[omeronas@omero02 OMERO.server]$ grep "2017-01-23" var/log/PixelData-0.log |  grep "%)" | grep 335562
2017-01-23 13:25:45,074 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1/4797 (0%).
2017-01-23 13:27:13,995 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 480/4797 (9%).
2017-01-23 13:29:04,254 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 959/4797 (19%).
2017-01-23 13:30:26,455 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1438/4797 (29%).
2017-01-23 13:31:30,678 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1917/4797 (39%).
2017-01-23 13:33:22,824 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 2396/4797 (49%).
2017-01-23 13:34:56,985 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 2875/4797 (59%).
2017-01-23 13:35:53,342 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 3354/4797 (69%).
2017-01-23 13:37:37,510 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 3833/4797 (79%).
2017-01-23 13:39:19,983 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 4312/4797 (89%).
2017-01-23 13:40:25,867 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 4791/4797 (99%).


  
  => could it be that we a facing file-system performance issues (although GPFS should be quite fast)?
• Is there a way to determine the amount of data having no Pyramid data yet? Just curious to know what has accumulated over the past weeks ...
• Any ideas _why_ the PixelData threads died/stopped working and how this could be prevented/monitored?

Any help/feedback/recommendation would be greatly appreciated to shed some light on this topic.

Many thanks in advance for your support, best,
-Rainer

[mtbc]

Dear Rainer,

Many good questions. I suspect that we will need to draw upon Josh's expertise but perhaps there are also useful hints on https://www.openmicroscopy.org/site/sup ... mance.html.

Could you zip up your server log directory and upload to http://qa.openmicroscopy.org.uk/qa/upload/ and mention when anything especially interesting happened in them? Also, which image formats are you seeing trouble with?

My immediate thoughts with the dying PixelData threads might be shortage of memory or a transient glitch in the underlying mounted storage volume. I wonder what others' experience is.

Cheers,
Mark

[rpoehlmann]

Dear Mark,

thanks a lot for your reply.

I uploaded the logs server zip file "OMERO-logs.zip" (see ID 17503, https://www.openmicroscopy.org/qa2/qa/f ... d3d8c62438).

In the meantime Pyramid data creation finished. Nevertheless it would be very valuable to better understand how it can be tweaked and/or prevented from failure.

The affected image formats were TIF files (not extensively tested/validated).

Thanks a lot for your support, cheers,
-Rainer

[mtbc]

Dear Rainer,

Your server logs certainly show all manner of interesting activity so thank you for those. Regarding pixels service unhappiness the log lines that catch my eye would include those from the morning of 2016-12-13 in Blitz-0.log.4 where we have OutOfMemoryError reports from the server and I/O errors from the database backend. Additionally the PixelData logs often show "Disk quota exceeded" messages. I'd therefore suspect that your server has been suffering from both memory and disk issues at times.

Josh can probably help with your questions about how to speed up the pyramid generation. He might also have some ideas about how to handle any corrupt pyramids you may have been left with, given your "Invalid TIFF file" messages in Blitz-0.log.1. I hope that the above observations help you to figure out why things went wrong in the first place though.

Cheers,
Mark

[jmoore]

Hi Rainer,

Trying to catch up:

• If you think that corrupt Pyramid files have been produced, take a look at `bin/omero admin fixpyramids -h` for a place to get started. (See this thread for an example)
• Regarding processing ordering, the next image chronologically per user is picked. This is to prevent the queue from being filled by any one user.
• Before trying to optimize for performance, I'd suggest we take care of the memory issues as well as the disk quotas since it will be difficult to debug otherwise.

All the best,
~Josh

[rpoehlmann]

Hi Josh,

thanks for your reply and the info about processing order.

• I checked (and fixed) corrupted Pyramid files:
  

  Code: Select all

  [omeronas@omero02 OMERO.server]$ bin/omero admin fixpyramids  /export/omero/OMERO/
Using session 3b17de97-84c2-40b8-8ed7-a8b99f481db0 (root@localhost:4064). Idle timeout: 200 min. Current group: system
Removing 197125_pyramid
Removing 271825_pyramid


  
  Only a few ones, not really big impact
• We should not have any disk quota issues
  

  Code: Select all

  Filesystem Fileset    type             KB      quota      limit   in_doubt    grace
bc2_fs07   share_omero USR         no limits             


  
  However, I will doublecheck this with the storage admins!
• I would be glad to further adjust/improve memory settings in particular if those are short for Pyramid file generation. Any input/recommendation are greatly appreciated.

Best,
-Rainer

[jmoore]

Hi Rainer,

what is the current memory settings for your processes:

Code: Select all

$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


~Josh.

[rpoehlmann]

Hi Josh,

Code: Select all

./bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx96000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '40', 'strategy': 'percent'})
indexer=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})
pixeldata=-Xmx72000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '30', 'strategy': 'percent'})
repository=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})


The server has 256GB RAM in total

Thanks & cheers,
-Rainer

[jburel]

Hi Rainer

The server seems to be well provisioned.
The fact that the pyramid generation was getting slow might have been due to the faulty ones
and also due to the backlog of pyramid to generate.
Do you still have pyramids to generate in the queue?
Hopefully the speed will improve when the queue is empty.
Unfortunately we do not have visual tools to monitor that.

Cheers

Jmarie

[rpoehlmann]

Hi Jean-Marie,

thanks for your reply. And, it's good to know that JVM settings seems to be more or less fine

The fact that the pyramid generation was getting slow might have been due to the faulty ones
and also due to the backlog of pyramid to generate.

Currently, everything is fine, no Pyramid data backlog.
At least for me, the take home message from our discussion seems to be: if possible try to avoid large backlogs in pyramid data generation. Since our backlog had been caused by a silently dying Pixeldata service, the best would be to detect those cases as soon as possible. And, to regularly scan for faulty ones (cause or effect?).

How to best monitor those scenarios is still puzzling me, but we will find a solution hopefully ...

Anyhow, thanks to everybody for the help, I think we can consider this topic as solved.

Cheers,
-Rainer