Full viewer not loading the whole image

[pmcarlton]

I'm using server version 4.4.8-ice34, and the web client on both Safari (OSX 10.8.3) and Firefox.
Web server details: fastcgi-tcp, nginx

All image thumbnails look fine, but double-clicking on an image (or clicking "Full viewer") usually results in an image window with a partially loaded image in it—the top ⅓ to ⅔ or so (randomly varying on each reload) of the image is shown, and the bottom is blank. (See the image: ). The client still "knows" about the data, as the line profile indicates, but it's not showing up in the window. The datatype is UInt16.

Sometimes fiddling with image display properties, or max. projection, etc., will get it to load the whole image, but that can go away. I've checked Blitz.log and OMEROweb.log and saw no obvious errors.

Any ideas?
Thanks a lot!
-Pete Carlton, Kyoto U.

[atarkowska]

Hi Pete,

First of all, could you run OMERO.insight and see whether you have the same problem?

In addition Could you please give us more details about environment you are running omero server. Please run

Code: Select all

bin/omero admin diagnostics

Could you also check how much memory do you have allocated for server processes in OMERO_HOME/etc/grid/template.xml

Code: Select all

<option>-Xmx512M</option>
<!--
       MaxPermSize needs to be set for most operating systems using Hibernate 3.3+
       with any largish heap size. The value below should be sufficient for most
       cases. The value will be added to the "-Xmx" value for the total memory used
       by the JVM (here: 512+128 = 640MB).

       See #4670
-->
<option>-XX:MaxPermSize=128m</option>

Might be worth to increase it to 2048M by changing the following:

Code: Select all

<option>-Xmx2048M</option>

Could you also send us part of the logfiles: Blitz-0.log OMEROweb.log & OMEROweb_request.log while you are running full viewer.

Thanks
Ola

[pmcarlton]

Hello Ola, and thanks for the quick reply.

1. OMERO.insight (v4.4.8-ice33-b256) does not seem to have the same problem; it loads the full image. However, thumbnails and images take much longer to load (more precisely, the image does not load until all the thumbnails for its dataset have loaded…but that's another issue, I guess.)

2. Diagnostics - I uploaded a .zip file with the requested logs, etc.

pmcarlton-2013-05-29.zip

logs, diagnostics, etc.

(34.98 KiB) Downloaded 148 times

The first bit is:

Code: Select all

OMERO Diagnostics 4.4.8-ice34-b256
Commands:   java -version                  1.6.0     (/usr/bin/java)
Commands:   python -V                      2.7.3     (/usr/bin/python)
Commands:   icegridnode --version          3.4.2     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.4.2     (/usr/bin/icegridadmin)
Commands:   psql --version                 9.1.9     (/usr/bin/psql -- 2 others)


The recent errors from the Blitz-0.log are as follows: (I think these errors relate to the long thumbnail generation times for OMERO.insight, not the web client)

Code: Select all

2013-05-28 12:19:42,012 ERROR [        ome.services.util.ServiceHandler] (2-thread-2) Method interface ome.services.util.Executor$Work.doWork invocation took 45697
2013-05-28 14:10:07,362 ERROR [        ome.services.util.ServiceHandler] (erver-2243) Method interface ome.services.util.Executor$Work.doWork invocation took 18918
2013-05-28 14:10:07,417 ERROR [        ome.services.util.ServiceHandler] (erver-2243) Method interface omeis.providers.re.RenderingEngine.renderProjectedCompressed invocation took 18973
2013-05-29 09:03:17,324 ERROR [        ome.services.util.ServiceHandler] (erver-2256) Method interface ome.api.ThumbnailStore.getThumbnailByLongestSideSet invocation took 87058
2013-05-29 09:09:15,907 ERROR [        ome.services.util.ServiceHandler] (erver-2264) Method interface ome.api.ThumbnailStore.getThumbnailByLongestSideSet invocation took 165545
2013-05-29 09:36:20,424 ERROR [                 ome.system.UpgradeCheck] (      main) Error reading from url: http://upgrade.openmicroscopy.org.uk?version=4.4.8-ice34-b256;os.name=Linux;os.arch=amd64;os.version=3.2.0-44-generic;java.runtime.version=1.6.0_32-b05;java.vm.vendor=Sun+Microsystems+Inc. "connect timed out"
2013-05-29 09:40:57,952 ERROR [        ome.services.util.ServiceHandler] (l.Server-7) Method interface ome.api.ThumbnailStore.getThumbnailByLongestSideSet invocation took 50275

3. Memory -- yes, the memory set in etc/grid/templates.xml (nb: not template.xml) was 512Mb, and I tried setting it to 2048Mb and restarting, but the behavior did not improve.

Thanks for your time.
—Pete

[atarkowska]

Hi Pete,

Thank you for all your logs. First of all my guest about memory issues were correct as in OMEROweb_request you can see:

Code: Select all

serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded...
Initialization of bean failed; nested exception is java.lang.OutOfMemoryError: Java heap space...


I am afraid I can't say anything else from OMEROweb_request.log as it match only period from 2013-05-23 02:17:10 to 2013-05-28 06:09:40.
Could you please send me more recent logs? Ideally would be if you could send me logs starting from the last restart (after memory was changed in etc/grid/template.xml) containing startup information, like:

Code: Select all

2013-05-29 10:11:40,865 INFO  [                ome.services.blitz.Entry] (      main) Creating OMERO.blitz. Please wait...
2013-05-29 10:11:43,713 INFO  [ng.ShutdownSafeEhcacheManagerFactoryBean] (      main) Initializing EHCache CacheManager
2013-05-29 10:11:47,668 INFO  [  ome.services.fulltext.FullTextAnalyzer] (      main) Initialized FullTextAnalyzer
2013-05-29 10:11:51,899 INFO  [          ome.services.util.DBPatchCheck] (      main) Verified database patch: OMERO4.4__0
2013-05-29 10:11:52,772 INFO  [        ome.services.db.DatabaseIdentity] (      main) Using LSID format: urn:lsid:export.openmicroscopy.org:%s:277d25c7-1e3a-4940-b829-b3cc146845e5_%s%s
2013-05-29 10:11:54,821 INFO  [                 org.perf4j.TimingLogger] (      main) start[1369818712895] time[1926] tag[ome.io.nio.SimpleBackOff.256X256]
2013-05-29 10:11:55,521 INFO  [                 org.perf4j.TimingLogger] (      main) start[1369818714822] time[699] tag[ome.io.nio.SimpleBackOff.256X256]
2013-05-29 10:11:56,554 INFO  [                 org.perf4j.TimingLogger] (      main) start[1369818715521] time[1033] tag[ome.io.nio.SimpleBackOff.256X256]
...

Also could you copy everything matching the same time from OMEROweb.log and OMEROweb_request.log

In order to do so, I would stop the omero web and omero server, backup logs from OMERO_HOME/var/log just in case there are any important information and delete entire var directory. Then set debugging mode on web by the following command:

Code: Select all

bin/omero config set omero.web.debug True

start again omero server and web. Try to reproduce the display issue and then send me entire log directory including all logfiles.

Ola

[pmcarlton]

Hi,
Okay, I issued `admin stop` and `web stop`, removed `var/`, and restarted.
The issue is still occurring with the web client. I'm attaching the full log output.

PMCarlton_2013-05-30.tar.gz

(55.39 KiB) Downloaded 149 times

Somehow, the OMEROweb_request.log file is not getting appended to (though it is created) - its size is 0.
In the var/log directory I also included the /var/log/nginx/access_log and error_log files.

Thanks,
Pete

[atarkowska]

Thank you for your effort and log files. You mentioned: "usually results in an image window with a partially loaded image in it—the top ⅓ to ⅔ or so (randomly varying on each reload) of the image is shown, and the bottom is blank". Does it mean that if you open the same image in the Viewer it will always load the same piece of the same image or is random load for exactly the same image?

Ola

[pmcarlton]

> if you open the same image in the Viewer it will always load the same piece of the same image or is random load for exactly the same image?

Each time I reload the exact same image it shows a different piece—it always loads at least some of it, and sometimes it loads the whole image, and the rest of the time it's partial. That is the same whether I just hit ⌘R to reload the image, or log out and log in again and load the same image.

I did notice some changes with the "Quality" drop-down menu though:
If I set it to "Low", then it does immediately display the entire image.
If I change it from "Normal" to "High", the amount of partial image that gets shown decreases (i.e., if ~⅔ of the image is shown, it'll change to ~⅓ or ~¼ or so).
If I then change it right back to "Normal", it will re-display the same part that it showed before I switched to "High". If I switch back many times, it will eventually show the entire image.
But then, when I change the Z section in "Normal" or "High" quality mode, it becomes a partial image again.

-Pete

[atarkowska]

Hi Pete,

From nginx error log you can see:

Code: Select all

2013/05/30 10:23:21 [crit] 1677#0: *10652 open() "/home/pcarlton/apps/OMERO/OMERO.server/var/nginx_tmp/0000000098" failed (2: No such file or directory) while reading upstream, client: 10.229.31.193, server: pcarlton-dell-ubuntu, request: "GET /webclient/render_image/1369/18/0/?c=1|257:2287$0000FF,2|176:1282$00FF00,3|161:1172$FF0000&m=c&p=normal&ia=0&q=0.9&zm=100&x=0&y=0 HTTP/1.1", upstream: "fastcgi://0.0.0.0:4080", host: "10.228.43.240:8080", referrer: "http://10.228.43.240:8080/webclient/img_detail/1369/"


Are you runnign nginx as a system daemon?
If so, it might not have write permissions on /home/pcarlton/apps/OMERO/OMERO.server/var/nginx_tmp/
So I guess you needs to make nginx_tmp writeably by whatever user nginx is run as, or world-writeable.


What version of Python Image Library do you use? Did you try to reinstall/upgrade PIL as this is what webclient uses to load image into the viewport.

I also came across many post that certain versions of various webbrowser may cause that bug. Did you try reproduce that issue on the different computer, such different OS (Windows, Mac OS X), web browser (IE, FF, Chrome)?

Ola

[pmcarlton]

Ah, that may well be it — nginx is running as user `www-data`. I just made nginx_tmp mode 777 and now the whole images are being displayed even in High quality mode (did not have to restart the server for that to happen). I will see if the fix is permanent or if I need to make changes to how nginx is run, but for now it looks like the problem has been fixed, so we can close this. (I've confirmed it works on Safari, Chrome and Firefox on the Mac). Thanks very much for your help.
—Pete