Slow pyramid calculation

[evenhuis]

Hi,

we are experiencing really slow pyramid generation times on some files which has lead to a massive backlog on our server.

I suspect it is due to some files which are either corrupted and/or aren't handled well by bioformats.

Quick questions

• should this be cross posted to the bioformats forum?
• is there way to disable pyramid generation for these problems files while we sort it out?

More details below.

Cheers,

Chris

Forum searches
I've had a look through the forums for some tips:

• pyramid creation on large image - out of memory https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=8380&p=19098&hilit=pyramid#p19098
  
• How to speed-up Pyramid data creation? https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=8198&p=17921&hilit=pyramid+speed#p17921

Server setup
From these I ran the 'remove pyramids' script (no problem files) and found an OOM error in the log do I upped the memory for the pixel data. The memory is

Code: Select all

[omero@prdomeapp01 OMERO.server]$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx3764m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx2509m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx3764m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx2509m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOption

It takes about 20hours to create the pyramid, which is surprising because the file is only 16MB.

After the pyramid has been generated there's a bunch if errors which looks like the connection to the database was dropped - perhaps because it took so long the connection timed out?

Code: Select all

2018-10-31 11:51:37,448 WARN  [   o.s.jdbc.support.SQLErrorCodesFactory] (2-thread-4) Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.postgresql.util.PSQLException: This connection has been closed.
snip
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
snip
2018-10-31 11:51:37,485 ERROR [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Failed to handle pixels 20806


The file itself
The file is a 16MB tif ( 8-bit 11424 x 16640 ) and looks to be generated by Hamamatsu scanner and JPEG copmpressed.

It can easily opened with Preview on MacOS, Inkscape struggles a bit, but Fiji really chokes on it (100% CPU for 10min with no sign of progress). It does eventually open but it must have taken 30min or so.

Demo server
I uploaded the file to the demo server and it seems to being a longtime to create the pyramids there to. The ID is 54506.

The file itself and server logs
The file itself and some the server logs are uploaded to the qa server, the id is 27045

[evenhuis]

Demo server update
The pyramid was calculated sometime during night for Image ID: 54506. It would interesting to find out how long it took from the PixelData log.

So it seems we are experiencing two problems,

• Slow generation of the pyramid for this file. It takes imageJ ~30min to open it and I imagine the pyramid calculation involves opening it >10 times.
• The database connection being dropped on server, maybe due to a timeout someplace
• The PixelData service halting after the database connection error.

Cheers,

Chris

[Dominik]

Hi Chris,

thanks for providing the example file. I was able to replicate the issue but I couldn't track it down yet, what the reason for extremly slow pyramid generation (and the high memory consumption) is. We'll working on it and can hopefully provide more information and a bugfix soon.

Regards,
Dominik

[evenhuis]

Hi Dominick,

Thanks for reply. Is there anyway to skip these files for pyramid generation - I suspect there are about 200 of these in the queue at the moment.

Cheers,

Chris

[jmoore]

Hi Chris,

joining this thread late: you can skip pending pyramid generations by disabling the pixel data thread, bumping the configuration setting, and then re-enabling:

Code: Select all

$ bin/omero admin ice server disable  PixelData-0
$ bin/omero admin ice server stop  PixelData-0


Code: Select all

postgres=# select * from configuration where name like 'pixel%';
                 name                  | value
---------------------------------------+-------
pixelDataEventLogLoader.v1.current_id | -1
(1 row)

postgres=# select max(id) from pixels;
max
-----
   1
(1 row)

update configuration set value = '1' where name = 'pixelDataEventLogLoader.v1.current_id';
UPDATE 1


Code: Select all

$ bin/omero admin ice server enable  PixelData-0


However, either while you are doing that or even before, you might consider if it's possible to give the pixeldata process more memory. (docs)

Cheers,
~Josh

[evenhuis]

Hi Josh,

thanks for the info. We have about a 1,000 of these files so incrementing the counter like that isn't going to be fun.

We have decided to delete these dodgy files, however now we are running into problems with open files. When I select a batch of 32 files to delete this results in 7,000 files being opened and not released for at least an hour. I'll update in the morning if/when they are released. Update - the 7K still have not been released after 8 hour.

What has been happening is that we have been running into the file open limit (~20K for our system) which degrades the system enough to need a restart.

Is this typical for deleting largish amounts of files or is this specific to this dodgy file?

Cheers,

Chris

[mtbc]

Will take this up via viewtopic.php?f=5&t=8680, thanks!

Cheers,
Mark