Possible time-out issues on multiple file download.
Posted: Thu Feb 26, 2015 2:05 pmHello all
Yesterday one of the team(Petr) was kind enough to try downloading a large-ish set of original files from our public OMERO server at
https://cisbic.bioinformatics.ic.ac.uk/ ... oject-3706
This failed consistently for him with:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Apache/2.2.15 (Scientific Linux) Server at cisbic.bioinformatics.ic.ac.uk Port 80
Attempting the same download from within IC we were only able to reproduce the same error on one occasion. The download was successful every other time.
On asking our sysadmin to examine the logs I received the reply pasted below.
Can someone please suggest how best to proceed in order to confirm that the issue has been correctly identified?
Regards
Ian
"Having just said that, I've spotted something in the daily log summary email from cell's httpd server which looks like it might tie in with the correct time (pasted below...) - there were a few more earlier instances of this error which sounds like your problem description. This doesn't give us the error reported from the backend server process, but is essentially saying that the backend process didn't return the correct HTML headers in a sufficiently timely manner. Accessing the URL in the error returns an HTTP 500 (internal server error) response. Could this query have been generated indirectly from the URL they were trying to access?
I think what the problem is here is that the creation of a zip archive by the backend is taking too long, and FastCGI is giving up waiting for a response before it is complete. You can see that the underlying process works ok by removing most of the image IDs so it just looks for a few, in which case it works fine. The time taken for it to return is also unsuprisingly dependent upon how many images are in the query,
This kind of problem normally occurs because the server process does not return any data before it has completed everything necessary at the backend to return the requested data. A good old fashioned kludge for this kind of problem is to immediately return the HTTP headers to the client which prevents it timing out and generating a 500 error, and then get on with generating the content to be returned. It is still possible for the request to then timeout if it takes too long to return the requested data, but we are talking minutes rather than seconds, and if that turns out to be a problem then we can adjust HTTP timeout values on the server.
This could, of course, just be a red herring, but is something that I guess needs to be fixed anyway....
Cheers,
James
[Wed Feb 25 17:32:33 2015] [error] [client 192.168.0.38] FastCGI: incomplete headers (0 bytes) received from server "/opt/omero/var/omero.fcgi", referer: https://cisbic.bioinformatics.ic.ac.uk/ ... ge-119275| image-119295|image-119319|image-119387|image-119388|image-119413|image-119273|image-119274|image-119278|image-119279|image-119281|image-119282|image-119283|image-119285|image-119286|image-119287|image-119288|image-119289|image-119290|image-119291|image-119294|image-119296|image-119297|image-119300|image-119301|image-119302|image-119303|image-119304|image-119305|image-119306|image-119307|image-119308|image-119309|image-119310|image-119311|image-119312|image-119313|image-119314|image-119315|image-119316|image-119320|image-119321|image-119325|image-119328|image-119329|image-119333|image-119334|image-119335|image-119317|image-119324|image-119327|image-119344|image-119353|image-119355|image-119374|image-119395|image-119412|image-119422|image-119318|image-119322|image-119323|image-119330|image-119331|image-119332|image-119339|image-119340|image-119346|image-119347|image-119349|image-119351|image-119352|image-119364|image-119336|image-119338|image-119343|image-119345|image-119348|image-11935 0|image-119354|image-119356|image-119357|image-119358|image-119359|image-119360|image-119361|image-119362|image-119337|image-119382|image-119419|image-119341|image-119342|image-119370|image-119379|image-119420|image-119421|image-119363|image-119365|image-119368|image-119369|image-119373|image-119376|image-119377|image-119378|image-119380|image-119381|image-119383|image-119384|image-119385|image-119386|image-119366|image-119367|image-119371|image-119372|image-119375|image-119390|image-119392|image-119393|image-119399|image-119400|image-119404|image-119409|image-119417|image-119389|image-119391|image-119394|image-119396|image-119397|image-119398|image-119401|image-119402|image-119403|image-119405|image-119406|image-119407|image-119408|image-119410|image-119411|image-119414|image-119415|image-119416|image-119418&fileCount=200
Yesterday one of the team(Petr) was kind enough to try downloading a large-ish set of original files from our public OMERO server at
https://cisbic.bioinformatics.ic.ac.uk/ ... oject-3706
This failed consistently for him with:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Apache/2.2.15 (Scientific Linux) Server at cisbic.bioinformatics.ic.ac.uk Port 80
Attempting the same download from within IC we were only able to reproduce the same error on one occasion. The download was successful every other time.
On asking our sysadmin to examine the logs I received the reply pasted below.
Can someone please suggest how best to proceed in order to confirm that the issue has been correctly identified?
Regards
Ian
"Having just said that, I've spotted something in the daily log summary email from cell's httpd server which looks like it might tie in with the correct time (pasted below...) - there were a few more earlier instances of this error which sounds like your problem description. This doesn't give us the error reported from the backend server process, but is essentially saying that the backend process didn't return the correct HTML headers in a sufficiently timely manner. Accessing the URL in the error returns an HTTP 500 (internal server error) response. Could this query have been generated indirectly from the URL they were trying to access?
I think what the problem is here is that the creation of a zip archive by the backend is taking too long, and FastCGI is giving up waiting for a response before it is complete. You can see that the underlying process works ok by removing most of the image IDs so it just looks for a few, in which case it works fine. The time taken for it to return is also unsuprisingly dependent upon how many images are in the query,
This kind of problem normally occurs because the server process does not return any data before it has completed everything necessary at the backend to return the requested data. A good old fashioned kludge for this kind of problem is to immediately return the HTTP headers to the client which prevents it timing out and generating a 500 error, and then get on with generating the content to be returned. It is still possible for the request to then timeout if it takes too long to return the requested data, but we are talking minutes rather than seconds, and if that turns out to be a problem then we can adjust HTTP timeout values on the server.
This could, of course, just be a red herring, but is something that I guess needs to be fixed anyway....
Cheers,
James
[Wed Feb 25 17:32:33 2015] [error] [client 192.168.0.38] FastCGI: incomplete headers (0 bytes) received from server "/opt/omero/var/omero.fcgi", referer: https://cisbic.bioinformatics.ic.ac.uk/ ... ge-119275| image-119295|image-119319|image-119387|image-119388|image-119413|image-119273|image-119274|image-119278|image-119279|image-119281|image-119282|image-119283|image-119285|image-119286|image-119287|image-119288|image-119289|image-119290|image-119291|image-119294|image-119296|image-119297|image-119300|image-119301|image-119302|image-119303|image-119304|image-119305|image-119306|image-119307|image-119308|image-119309|image-119310|image-119311|image-119312|image-119313|image-119314|image-119315|image-119316|image-119320|image-119321|image-119325|image-119328|image-119329|image-119333|image-119334|image-119335|image-119317|image-119324|image-119327|image-119344|image-119353|image-119355|image-119374|image-119395|image-119412|image-119422|image-119318|image-119322|image-119323|image-119330|image-119331|image-119332|image-119339|image-119340|image-119346|image-119347|image-119349|image-119351|image-119352|image-119364|image-119336|image-119338|image-119343|image-119345|image-119348|image-11935 0|image-119354|image-119356|image-119357|image-119358|image-119359|image-119360|image-119361|image-119362|image-119337|image-119382|image-119419|image-119341|image-119342|image-119370|image-119379|image-119420|image-119421|image-119363|image-119365|image-119368|image-119369|image-119373|image-119376|image-119377|image-119378|image-119380|image-119381|image-119383|image-119384|image-119385|image-119386|image-119366|image-119367|image-119371|image-119372|image-119375|image-119390|image-119392|image-119393|image-119399|image-119400|image-119404|image-119409|image-119417|image-119389|image-119391|image-119394|image-119396|image-119397|image-119398|image-119401|image-119402|image-119403|image-119405|image-119406|image-119407|image-119408|image-119410|image-119411|image-119414|image-119415|image-119416|image-119418&fileCount=200