Open Microscopy Environment

Hi,

I have been testing the export of large files using the new Big Tiff functionality. I have found that the files fail to export when they get really big. In most cases the stack trace showed that the the temporary ome.tiff file on the server side had been deleted when the client was trying to read it.

I added some logging to org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMETiff(...):

Code: Select all

    synchronized File exportImageAsOMETiff(File f, long imageID)
        throws DSAccessException, DSOutOfServiceException
    {
        isSessionAlive();
        FileOutputStream stream = null;
        DSAccessException exception = null;
        try {
            ExporterPrx store = null;
            stream = new FileOutputStream(f);
           
            try {
                //synchronized(new Object()) {
                    store = getExporterService();
                    if (store == null) store = getExporterService();
                    store.addImage(imageID);
                    log.info("Exporting image ID " + imageID);
                   
                    long offset = 0;
                    try {
                        long size = store.generateTiff();
                        log.info("Extracting " + size + " bytes");
                        for (offset = 0; (offset+INC) < size;) {
                            stream.write(store.read(offset, INC));
                            offset += INC;
                        }   
                        stream.write(store.read(offset, (int)(size-offset)));
                        stream.close();
                        log.info("Successfully read " + size + " bytes");
                    } catch (Exception e) {
                        log.info("Partially read " + offset + " bytes");
                        if (stream != null)
                            stream.close();
                        if (f != null) {
                            f.delete();
                            f = null;
                        }
                        exception = new DSAccessException(
                                "Cannot generate the image as an OME-TIFF", e);
                    }
                //}
            } finally {
                try {
                    if (store != null) store.close();
                } catch (Exception e) {}
                if (exception != null) throw exception;
                return f;
            }
        } catch (Throwable t) {
            if (f != null) f.delete();
            if (t instanceof DSAccessException)
                throw (DSAccessException)t;
            throw new DSAccessException(
                    "Cannot export the image as an OME-TIFF", t);
        }
    }


This basically records at what point the file export fails. In all cases I get logging like this:

Code: Select all

grep -C 3 Partial /tmp/omeroinsight.log

2011-10-03 16:08:48,763 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Exporting image ID 20402
2011-10-03 16:12:31,047 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Extracting 5075447505 bytes
2011-10-03 16:19:16,008 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Partially read 4023386112 bytes
2011-10-03 16:19:16,352 ERROR [penmicroscopy.shoola.env.ui.ExportLoader] (entQueue-0) Data Retrieval Failure: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot generate the image as an OME-TIFF
        at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMETiff(OMEROGateway.java:7000)
        at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.exportImageAsOMETiff(OmeroImageServiceImpl.java:1458)
--

2011-10-04 13:13:29,631 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Exporting image ID 20402
2011-10-04 13:17:31,901 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Extracting 5075447505 bytes
2011-10-04 13:23:51,590 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Partially read 3913023488 bytes
2011-10-04 13:23:51,917 ERROR [penmicroscopy.shoola.env.ui.ExportLoader] (entQueue-0) Data Retrieval Failure: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot generate the image as an OME-TIFF
        at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMETiff(OMEROGateway.java:7000)
        at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.exportImageAsOMETiff(OmeroImageServiceImpl.java:1458)
--

2011-10-04 16:25:20,502 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Exporting image ID 20402
2011-10-04 16:29:29,221 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Extracting 5075447505 bytes
2011-10-04 16:35:51,390 INFO  [nmicroscopy.shoola.env.data.OMEROGateway] ( Thread-23) Partially read 3793223680 bytes
2011-10-04 16:35:51,640 ERROR [penmicroscopy.shoola.env.ui.ExportLoader] (entQueue-0) Data Retrieval Failure: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot generate the image as an OME-TIFF
        at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMETiff(OMEROGateway.java:7009)
        at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.exportImageAsOMETiff(OmeroImageServiceImpl.java:1458)


So it takes about 4 minutes to create the file on the server and then 6 minutes of downloading most of the file before it fails. The stack trace for the error says the file on the server is missing:

Code: Select all

Caused by: omero.InternalException
    serverStackTrace = "java.lang.RuntimeException: java.io.FileNotFoundException: /tmp/omero/tmp/omero_omero/865@omero.lifesci.susx.ac.uk/__omero_export__5467077110343421763.ome.tiff (No such file or directory)
                           at ome.services.blitz.impl.ExporterI.read(ExporterI.java:558)


However I noted that in the case where the export fails the logging timestamps are 10 minutes and 30 seconds apart (give or take a few seconds). In the case of large files (>4GB) that I have successfully extracted, the total timing is less than 10 minutes.

When I watch the file on the server it is generated OK. The file is then downloaded and appears to be generated OK on the client until the 10 minute threshold. It then fails and the incomplete client-side file is deleted.

Q. Is there a timeout occurring somewhere between the client and server that cuts off a working process?

I had a similar problem in my Python code and solved this by setting the following:

Code: Select all

    conn = BlitzGateway(username, password)
    conn.getSession().setTimeToIdle(rlong(60*60*1000))


Q. Is there a way to increase the idle time for the session held by the Insight Java client? Alternatively can the client somehow ping the server while the download is in progress?

Thanks,

Alex

Hi,

A bit more investigation led me to set a property on the OMERO server:

Code: Select all

omero config set omero.sessions.timeout 1200000


This is an increase from the default of 60000 millisecond (or 10 minutes).

After restarting the server I managed to get an image to export in 13 minutes. Previously this would have failed. Insight did lose the connection to the server but after prompting to reconnect it managed to successfully reconnect.

So I may have found a solution to export my big images. However I am not sure why Insight dropped the connection but the export process finished.

Regards,

Alex

Hi Alex
Thanks for the information, the services should be kept alive. I have created a ticket.
http://trac.openmicroscopy.org.uk/ome/ticket/6928.
I will look at the problem

There is currently no way in the UI to increase the idle time, that's a requirement that should not be difficult to add.

Thanks again
Jmarie

Hi Alex
The "synchronized" part of the method needs to be reviewed, that's the source of the problem.

jmarie