Opening files while they are being written

[barseno]

Hi,
I'd like to write an application that performs image processing on very large stacks while they are being acquired. For that I would need to be able to access the file while it is being written. From my tests it seems as if bioformats can't do this right now. Can you think of an easy way to implement that or do you think it's not feasible at all? Help is very appreciated. Thanks!

[wmoore]

Hi,

I believe the solution that some have used for this is to write single-plain images as separate files during acquisition & analysis. Then stitch together when you are done.

Will

[barseno]

Thanks for your quick reply. Unfortunately I have no influence on the acquisition software (and it's supposed to be compatible with as many as possible).

[wmoore]

Could you tell us what the acquisition software is and what the file format is?

Then I'll have to pass this over to Melissa, who has a better knowledge of file formats than I do.

Thanks,

Will.

[mlinkert]

Hi,

I'd like to write an application that performs image processing on very large stacks while they are being acquired. For that I would need to be able to access the file while it is being written. From my tests it seems as if bioformats can't do this right now.

Whether or not this is possible depends entirely upon the format. If all of the metadata (width and height of images, number of total images, etc.) is written first, e.g. into some sort of header file, then it's likely possible.

However, in order to help you make this work, we will need as much information as possible. Can you send a few datasets, preferably in various stages of acquisition? Can you send any of the test code that you are using, so that we know exactly what your code is expecting? Would you rather have Bio-Formats report the number of expected planes (regardless of how many have been acquired already), or only the number of planes that have been acquired so far?

If needed, I can provide you with an SFTP server to which datasets can be uploaded.

Regards,
-Melissa

[barseno]

Hi,

thanks for your help.

Can you send any of the test code that you are using, so that we know exactly what your code is expecting?

The structure is very simple, like this (in pseudo code):

Code: Select all

while(acquisitionRunning) // tested e.g. by change of file size within certain interval
    new framePackage
    while(framePackage.framesMissing > 0)
        readToPackage(min(framesAvailable,framePackage.framesMissing))
        if(acquisitionRunning == false)
            framePackage.framesMissing = 0
        if(framePackage.framesMissing == 0)
            process(framePackage)


Would you rather have Bio-Formats report the number of expected planes (regardless of how many have been acquired already), or only the number of planes that have been acquired so far?

As seen in the code above I do only care about the frames (/planes) already acquired. However, I will need the frames in (acquisition-) order!

Could you tell us what the acquisition software is and what the file format is?

As I said, I hope it to be working with as many acquisition programs as possible. For now the most important one would be Nikon's ND2 format.

From my understanding it should be possible to achieve it in theory, as long as the file format specifies plane information (as in size & type) in the beginning of the file and then adds new planes in order. Also, it would not work if it used compression that is not being done frame wise (although it sounds weird to me do not do it frame wise, I don't know what's common though).

[mlinkert]

Hi,

As seen in the code above I do only care about the frames (/planes) already acquired. However, I will need the frames in (acquisition-) order!

That shouldn't be a problem then.

As I said, I hope it to be working with as many acquisition programs as possible. For now the most important one would be Nikon's ND2 format.

From my understanding it should be possible to achieve it in theory, as long as the file format specifies plane information (as in size & type) in the beginning of the file and then adds new planes in order.

For ND2 specifically, this should mostly work. You may run into some trouble if you need to know the specific timepoint/Z section/channel/etc. of a particular plane, but I think it would work to just retrieve the pixel data.

However, we're not really set up to test this ourselves. If you can send a set of files in various stages of acquisition, that would help a lot in testing. If that's not possible, then it would help to see the full text of any error messages that you are receiving. I will send information privately on how to send files to us.

Also, it would not work if it used compression that is not being done frame wise (although it sounds weird to me do not do it frame wise, I don't know what's common though).

Frame-by-frame compression is the most common approach. In some cases (e.g. ICS) you will see the entire set of pixel data compressed at once; ICS is rarely (never?) used as an acquisition format, though.

The one somewhat common case that I can think of that really won't work at all is if the file is an MS OLE document; Zeiss ZVI, Hamamatsu/SimplePCI .cxd, and Olympus FV1000 .oib all fall into this category.

Regards,
-Melissa