Browser caching generated thumbs

[PaulVanSchayck]

Dear all,

I've got a problem with browser caching of generated thumbnails of pyramided files in the OMERO.web client.

Right now, after importing, the dataset will be full with thumbnails with the click icon, i.e. pixel pyramid being generated. However once finished pressing the "Refresh" button from within OMERO.web (not browser refresh!) will not trigger the browser of requesting the thumbnail again. Thumbnails are then left on the click icon, even though they are really finished and ready to show. A hard browser refresh will show the new images.

I've been thinking of a solution for this. What would be an option is appending the current time to the request string to /render_thumbnail/ like this /render_thumbnail/size/96/1234/?time=1406204196. This would however cause a lot of extra requests.


Another, but slightly related issue has been that the processing of pyramid files seems to stop after a few files, unless the thumbnails are requested again. Is this intended or unexpected behavior? This does not always seem to happen, but maybe only when the importer is called from an OMERO.py script. Maybe this problem description is too minimal. Tell me if you need more information.

Thanks,

Paul

[wmoore]

HI Paul,

We could certainly use some technique like you suggest to reload all the thumbnails on 'refresh'. I'd have to see how tricky it is to only refresh the thumbnails on the currently opened dataset.

Processing of pyramid files shouldn't stop until complete.
Can you could repeat it yourself a few times to see if it's reproducible, E.g from a python script.
If so, try looking at the logs in var/log/PixelData-0.log when the processing stops, and while you're refreshing thumbnails etc.

Code: Select all


$ tail -f dist/var/log/PixelData-0.log
...
2014-08-11 09:48:24,829 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-4) File is a raw codestream not a JP2.
2014-08-11 09:48:24,904 INFO  [              loci.formats.in.TiffReader] (2-thread-4) Checking comment style
2014-08-11 09:48:24,904 INFO  [          loci.formats.in.BaseTiffReader] (2-thread-4) Populating OME metadata
2014-08-11 09:48:27,075 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Added StatsInfo:751 for ome.model.core.Channel:Id_801 - C:0 Max:255.0 Min:27.0
2014-08-11 09:48:27,078 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Added StatsInfo:752 for ome.model.core.Channel:Id_802 - C:1 Max:255.0 Min:0.0
2014-08-11 09:48:27,081 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Added StatsInfo:753 for ome.model.core.Channel:Id_803 - C:2 Max:255.0 Min:0.0
2014-08-11 09:48:27,081 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) HANDLED EventLog:23062(entityId=451) [23056 ms.]
2014-08-11 09:48:27,082 ERROR [        ome.services.util.ServiceHandler] (2-thread-4) Method interface ome.services.util.Executor$Work.doWork invocation took 23066
2014-08-11 09:48:27,096 INFO  [             ome.io.nio.FilePathResolver] (2-thread-1) Metadata only file, resulting path: /OMERO/ManagedRepository/will_3/2014-08/11/09-47-55.158/4k4k.png



regards,

WIll.

[PaulVanSchayck]

Hi Will,

Took a while to get back to you.

Here is a typical case:
http://pastebin.com/3X3ySyz0

At 14:02:48 the processing stops, at 14:06 the processing continues again after I refresh the browser. This also happens for the Insight client, and there pressing the refresh button will restart pixel generation.

What I notice from the logs is that the "Metadata only file, resulting path:" message appears several times for the same image throughout the log.

Thanks for taking a look,

Paul

[wmoore]

Hi Paul,

Did you confirm whether the stalling of Pyramid generation depends on how you import? E.g. via Insight or command line? (or commandline via script)?

Also, could you give us an idea of what size the png images are (sizeX, sizeY) and how many you're importing each time (do you ever see this stall if it's one image at a time)?

Also, what's your OMERO version? Any other info that might be useful?

I'll create a ticket for someone who knows about pyramid generation, although there's a few people away just now, so there might be a delay... https://trac.openmicroscopy.org.uk/ome/ticket/12526

Thanks,

Will.

[PaulVanSchayck]

Dear Will,

The stalling does not depend on the way of import it seems. I can replicate it using the insight import, the cli importer or the cli import from an OMERO.script.

The PNG images are ranging from 3600px by 2675px to 5004 px by 4478 px, so not all images are in fact pyramided. There are 40 of them. But I'm pretty sure I've seen it with different numbers as well. I don't think I've ever seen it stall when I was only importing one image.

The OMERO version is 5.0.3, both clients and server.

The only other observation I have is that quite often that once pixel generation stalls that the number of images being done in the next round (until it stalls again) is 5. This is the same number as the reported in the log file (2-thread-1 - 2-thread-5).

Thanks,

Paul

[jburel]

Dear Paul

Thanks for the detailed analysis,.
Several people are currently on holidays, we will investigate the issue and hopefully have it fixed for the next major release.(5.1.0)
If you wish to trac the ticket created by Will. Let us know

Regards

Jmarie

[PaulVanSchayck]

Dear Jmarie,

Thank you. Yes I would like to be added to the cc list of the ticket.

I did some further tests, and it does not seem to be limited to PNG files. Also for JPG files this happens. The only other thing in my setup which is I think a bit more exotic is that the /OMERO/MangedRepository and /OMERO/Pixels directories are symlinks to NFS shares.


Coming back to the thumbnail caching issue within the webclient. Doing something simple as changing in omeroweb/webclient/templates/webclient/data/containers_icons.html

Code: Select all

<img id="{{ c.id }}" src="{% url 'render_thumbnail_resize' 96 c.id %}" alt="image" title="{{ c.name }}{% if not c.isOwned %}, owned by {{ c.getOwner.getNameWithInitial }}{% endif %}"/>

to

Code: Select all

<img id="{{ c.id }}" src="{% url 'render_thumbnail_resize' 96 c.id %}#{% now 'H:i:s' %}" alt="image" title="{{ c.name }}{% if not c.isOwned %}, owned by {{ c.getOwner.getNameWithInitial }}{% endif %}"/>

Fixes the issue. However, this indeed does cause all thumbs to be requested each time you press refresh. Is it perhaps possible to know in that template file whether a thumbnail has been (re)generated. This would allow to trigger a selective refresh.

Thanks,

Paul

[wmoore]

Hi Paul,

Just discussed this with Jean-Marie...
It seems that there is a 'version' attribute of thumbnails in the DB, so there may be a way to include this in the html and only get a thumbnail if the current version isn't already cached.

Although there is the possibility that refreshing the thumbnail may actually generate another new version on the server, so we'd have to experiment a bit and see if we can achieve what we want.

I'll look into it anyway,

Will

[PaulVanSchayck]

Dear Will,

Coming back to the pyramid generation stalling, I've been trying to debug this further. First of all it does not seem to be related to NFS. I've placed a test user and corresponding Pixel data dir back onto the local filesystem (using symlinks), and that made no difference.

So, tracing the code I came to this file
https://github.com/openmicroscopy/openm ... oader.java

I would like to view the output of the debug statements in that file, but I've been unable to edit etc/logback.xml in such way that these lines become visible in var/log/PixelData-0.log

Thanks,

Paul

[sbesson]

Hi Paul,

after looking at the template for PixelData log file, the configuration file you need to modify is actually etc/logback-indexing.xml. Thus, adding a line like

Code: Select all

  <logger name="ome.services.pixeldata" level="DEBUG"/>


to this configuration file should be enough to enable the DEBUG level for the services you are interested in.

Best,
Sebastien