OMERO.web 5.4.0 plate thumbnail display hangs long time

[dsudar]

Hi team,

In OMERO.web 5.4.0 it takes many minutes for a thumbnail display for a plate with 20 by 35 wells to show up and the web browser is non-responsive during that time.
This new and annoying behavior started after I upgraded one of our production servers to 5.4.0. This is on Ubuntu 16.04 and I upgraded from 5.3.4.

The standard workflow is as follows: Image acquired with a Nikon HCA system in nd2 format on a per-well basis are imported by CLI. Each nd2 file contains 700 images which are the contents of a single well from an 8-well plate. I have a script that takes the Dataset of 700 images and creates a pseudo-plate using the Dataset_To_Well script. Doing this allows me to show the entire 20 by 35 grid of cell spots that we printed into the bottom of each well.

Since the upgrade to 5.4.0 every time (also 2nd or later times the same well) it takes minutes for the thumbnails to show up and during that time the server is not at all busy and no warnings or anything shows up in the logs. The only thing that appears busy is the browser which is unresponsive. But again does not show much activity in the Windows Task Manager. This happens with any browser I tested. Once it finally renders the thumbnails, everything appears fine again until the next set is attempted to open.

I'll be happy to provide one of the nd2 files and associated little script to generate such a 20 by 35 spots in-"well" plate. But they are 7 - 12GB in size so I will use the alternative upload mechanism.

My little bash script is as simple as this:

Code: Select all

dsid=`omero obj new Dataset name=$wname`
omero import -d $dsid <filename>
dsidonly=`echo $dsid | sed s/Dataset://g`
omero script launch <Dataset_To_Well.py script> IDs=$dsidonly First_axis='column' First_Axis_Count=20 Column_Names='number' Row_Names='letter'
omero delete $dsid


I'm assuming some Javascript being generated is tripping up the browser but I have no idea where to look for that. Note that displaying these thumbnails in OMERO.Insight works fine and fast.

Thanks for any insights.
Cheers,
- Damir

[wmoore]

Hi Damir,

I'm sorry to hear that this has got worse for you. I'm not aware of any changes in 5.4.0 to the thumbnail rendering behaviour, but we'll continue to investigate...

The current webclient behaviour is to load thumbnails in batches of 50. So your 700 thumbnails are trying to load in 14 simultaneous calls to /webgateway/get_thumbnails/. These are taking a long time to return and during this time, other requests will be held up, hence the webclient being unresponsive.

I tried a similar workflow to you in 5.4.0, using 700 images of size 1392 x 1040 and I also saw a significant delay during thumbnail creation, but only around 20 secs instead of minutes.

Once the thumbnails have been generated on the server, they should be cached so that subsequent requests are faster. So something's not right here.

I'll try this with 5.3.4 too and see if there's any noticeable difference in behaviour or performance.
And see if others on the team have any suggestions.


Will.

[wmoore]

Hi Damir,

I've confirmed that thumbnails are faster to load in 5.3.4.
It seems that thumbnails are not getting cached in the 5.4.0 OMERO server. I've created a ticket to look into this further. https://trello.com/c/2fQFE3Lb/99-bug-thumbnail-caching
Apologies for this regression. Hopefully we should have a fix soon.

Regards,

Will.

[dsudar]

Hi Will,

Thanks for checking right away and glad you were able to replicate the issue. My 700 images are 1600 by 1600 and 4 channels each so that larger size probably adds to the delay. I uploaded such a nd2 file to the Glencoe upload server named: PlateLI8X00631_Well1_Seq0000.nd2

And a brief clarification: for me, while the webclient is blocked, actually the entire browser application (Firefox or Chrome) is blocked or responding extremely slowly. And interestingly, the thumbnail view for smaller plates (e.g. 96 or 384 wells) loads quickly or at least I don't see a noticeable delay.

Cheers,
- Damir

[wmoore]

Hi Damir,

Thanks for the file upload. NB: for OME team ref, the file is now at squig:/ome/data_repo/inbox/dsudar-2017-10-26.
We'll let you know when we have more info.

Regards,

Will.

[dsudar]

Hi Will,

As far as I can see version 5.4.1 did not incorporate any fixes for this issue, right? Do you have an idea whether this can be resolved relatively easily? Is there anything I can do to help track it down?

Thanks,
- Damir

[jmoore]

Hi Damir,

You're right, there isn't yet a fix for this. For the moment, we still don't know what the underlying issue is, so the ETA for a fix is unknown. Other than having someone bisect or otherwise step through the code, I'm not sure there's much you can do.

It's definitely still on the list of critical issues that we will be looking into.

Sorry I don't have better news.
~Josh.

[mtbc]

Dear Damir,

You should find that Will has fixed this issue in the forthcoming OMERO.web 5.4.2. If you find that the problem persists even in that then do let us know. Thank you for providing various helpful clues about the bug!

Cheers,
Mark

[dsudar]

Hi Mark, thanks for the update and that's great news. I see from Trello that 5.4.2 is imminent so I'll upgrade as soon as it's available.
Cheers,
- Damir