We're Hiring!

Possible time-out issues on multiple file download.

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Possible time-out issues on multiple file download.

Postby i.munro » Thu Feb 26, 2015 2:05 pm

Hello all

Yesterday one of the team(Petr) was kind enough to try downloading a large-ish set of original files from our public OMERO server at

https://cisbic.bioinformatics.ic.ac.uk/ ... oject-3706

This failed consistently for him with:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Apache/2.2.15 (Scientific Linux) Server at cisbic.bioinformatics.ic.ac.uk Port 80

Attempting the same download from within IC we were only able to reproduce the same error on one occasion. The download was successful every other time.

On asking our sysadmin to examine the logs I received the reply pasted below.

Can someone please suggest how best to proceed in order to confirm that the issue has been correctly identified?

Regards

Ian



"Having just said that, I've spotted something in the daily log summary email from cell's httpd server which looks like it might tie in with the correct time (pasted below...) - there were a few more earlier instances of this error which sounds like your problem description. This doesn't give us the error reported from the backend server process, but is essentially saying that the backend process didn't return the correct HTML headers in a sufficiently timely manner. Accessing the URL in the error returns an HTTP 500 (internal server error) response. Could this query have been generated indirectly from the URL they were trying to access?

I think what the problem is here is that the creation of a zip archive by the backend is taking too long, and FastCGI is giving up waiting for a response before it is complete. You can see that the underlying process works ok by removing most of the image IDs so it just looks for a few, in which case it works fine. The time taken for it to return is also unsuprisingly dependent upon how many images are in the query,

This kind of problem normally occurs because the server process does not return any data before it has completed everything necessary at the backend to return the requested data. A good old fashioned kludge for this kind of problem is to immediately return the HTTP headers to the client which prevents it timing out and generating a 500 error, and then get on with generating the content to be returned. It is still possible for the request to then timeout if it takes too long to return the requested data, but we are talking minutes rather than seconds, and if that turns out to be a problem then we can adjust HTTP timeout values on the server.

This could, of course, just be a red herring, but is something that I guess needs to be fixed anyway....

Cheers,
James


[Wed Feb 25 17:32:33 2015] [error] [client 192.168.0.38] FastCGI: incomplete headers (0 bytes) received from server "/opt/omero/var/omero.fcgi", referer: https://cisbic.bioinformatics.ic.ac.uk/ ... ge-119275| image-119295|image-119319|image-119387|image-119388|image-119413|image-119273|image-119274|image-119278|image-119279|image-119281|image-119282|image-119283|image-119285|image-119286|image-119287|image-119288|image-119289|image-119290|image-119291|image-119294|image-119296|image-119297|image-119300|image-119301|image-119302|image-119303|image-119304|image-119305|image-119306|image-119307|image-119308|image-119309|image-119310|image-119311|image-119312|image-119313|image-119314|image-119315|image-119316|image-119320|image-119321|image-119325|image-119328|image-119329|image-119333|image-119334|image-119335|image-119317|image-119324|image-119327|image-119344|image-119353|image-119355|image-119374|image-119395|image-119412|image-119422|image-119318|image-119322|image-119323|image-119330|image-119331|image-119332|image-119339|image-119340|image-119346|image-119347|image-119349|image-119351|image-119352|image-119364|image-119336|image-119338|image-119343|image-119345|image-119348|image-11935 0|image-119354|image-119356|image-119357|image-119358|image-119359|image-119360|image-119361|image-119362|image-119337|image-119382|image-119419|image-119341|image-119342|image-119370|image-119379|image-119420|image-119421|image-119363|image-119365|image-119368|image-119369|image-119373|image-119376|image-119377|image-119378|image-119380|image-119381|image-119383|image-119384|image-119385|image-119386|image-119366|image-119367|image-119371|image-119372|image-119375|image-119390|image-119392|image-119393|image-119399|image-119400|image-119404|image-119409|image-119417|image-119389|image-119391|image-119394|image-119396|image-119397|image-119398|image-119401|image-119402|image-119403|image-119405|image-119406|image-119407|image-119408|image-119410|image-119411|image-119414|image-119415|image-119416|image-119418&fileCount=200
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: Possible time-out issues on multiple file download.

Postby atarkowska » Mon Mar 02, 2015 9:44 am

Hi Ian,

Recently we have made some adjustment to apache config and to how the downloading files. I will look at that closer. In the meantime could you send us your apache config?

Ola
atarkowska
 
Posts: 327
Joined: Mon May 18, 2009 12:44 pm

Re: Possible time-out issues on multiple file download.

Postby i.munro » Mon Mar 02, 2015 9:51 am

Hi Ola

Thanks. Will do. I should have added that this is a 5.0.6 server.

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: Possible time-out issues on multiple file download.

Postby i.munro » Tue Mar 03, 2015 9:18 pm

Passing this on from James:

Hi Ian,

Our httpd.conf file 'includes' a number of other files from a conf.d directory. In addition to the main httpd.conf, the relevant additional parts are going to be (comments removed for brevity):

omero.conf

SetEnv HTTPS on
Redirect 403 /static/webstart/jars/jcl-over-slf4j-1.7.5.jar
Redirect 403 /static/webstart/jars/ehcache-core-2.6.2.jar
FastCGIExternalServer "/opt/omero/var/omero.fcgi" -host 0.0.0.0:4080
<Directory "/opt/omero/var">
Options -Indexes FollowSymLinks
Order allow,deny
Allow from all
</Directory>

<Directory "/opt/omero/lib/python/omeroweb/static">
Options -Indexes FollowSymLinks
Order allow,deny
Allow from all
</Directory>
Alias /static /opt/omero/lib/python/omeroweb/static
Alias /omero "/opt/omero/var/omero.fcgi/"

fastcgi.conf

User apache
Group apache
LoadModule fastcgi_module modules/mod_fastcgi.so
FastCgiIpcDir /var/run/mod_fastcgi
FastCgiConfig -idle-timeout 20 -maxClassProcesses 1

httpd.conf

ServerTokens OS
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 60
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
<IfModule worker.c>
StartServers 4
MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Listen 80

LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule version_module modules/mod_version.so

Include conf.d/*.conf
User apache
Group apache
ServerAdmin root@localhost
UseCanonicalName Off
DocumentRoot "/var/www/html"

<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>

<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

<IfModule mod_userdir.c>
UserDir disabled
</IfModule>

DirectoryIndex index.html index.html.var
AccessFileName .htaccess

<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>

TypesConfig /etc/mime.types
DefaultType text/plain

<IfModule mod_mime_magic.c>
MIMEMagicFile conf/magic
</IfModule>

HostnameLookups Off
ErrorLog logs/error_log
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog logs/access_log combined

ServerSignature On

Alias /icons/ "/var/www/icons/"
<Directory "/var/www/icons">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

<IfModule mod_dav_fs.c>
DAVLockDB /var/lib/dav/lockdb
</IfModule>
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable Charset=UTF-8
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README.html
HeaderName HEADER.html
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW
ForceLanguagePriority Prefer Fallback
AddDefaultCharset UTF-8
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
AddHandler type-map var
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
Alias /error/ "/var/www/error/"

<IfModule mod_negotiation.c>
<IfModule mod_include.c>
<Directory "/var/www/error">
AllowOverride None
Options IncludesNoExec
AddOutputFilter Includes html
AddHandler type-map var
Order allow,deny
Allow from all
LanguagePriority en es de fr
ForceLanguagePriority Prefer Fallback
</Directory>
</IfModule>
</IfModule>
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: Possible time-out issues on multiple file download.

Postby atarkowska » Mon Mar 09, 2015 9:18 am

Hi Ian,

My apology for a slow response. I'd recommend to update apache config as timeout was extended in recent config templates to 60 sec. You can set up your own value...
See: https://github.com/openmicroscopy/openm ... mplate#L58

Code: Select all
FastCGIExternalServer "omero_path/var/omero.fcgi" -host 127.0.0.1:4080 -idle-timeout 60


That won't solve the problem related to the long zipping but definitely resolve issue for a smaller set.

Please also note that there are new changes in webserver template coming with 5.1. All of that will be mentioned on upgrade page.

Ola
atarkowska
 
Posts: 327
Joined: Mon May 18, 2009 12:44 pm

Re: Possible time-out issues on multiple file download.

Postby i.munro » Tue Mar 10, 2015 11:02 am

Hi Ola

The time-out change that you suggested seems to have done the trick, at least several members of the OME team were able to download the data in question without problem. Thanks

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest