We're Hiring!

Olympus raw file support is incomplete (.oir)

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Olympus raw file support is incomplete (.oir)

Postby andalman » Wed Sep 06, 2017 7:26 pm

OME Bioformats support for the Olympus raw file format (.oir) is incomplete. When a .oir file reaches a size larger than 1Gb, Olympus automatically begins splitting the file into sub-files, e.g. 'datafile.oir' will contain time points 0 through N, then 'datafile_00001' will contain time points N through M, etc... These files need to be loaded/treated as one large file, but OME Bioformats does not do this properly. (Note: the sub-files cannot simply be loaded as separate .oir files. See my previous post).

I would greatly appreciate Bioformats correcting this issue. If I upload a complete example of a large split .oir file, could someone work on this?

Note, the Olympus Viewer ImageJ plugin handles these split files correctly, but it is not scriptable or usable outside of ImageJ.
andalman
 
Posts: 6
Joined: Fri Sep 01, 2017 6:45 pm

Re: Olympus raw file support is incomplete (.oir)

Postby sbesson » Thu Sep 07, 2017 3:48 pm

Hi Aaron,

OME Bioformats support for the Olympus raw file format (.oir) is incomplete. When a .oir file reaches a size larger than 1Gb, Olympus automatically begins splitting the file into sub-files, e.g. 'datafile.oir' will contain time points 0 through N, then 'datafile_00001' will contain time points N through M, etc... These files need to be loaded/treated as one large file, but OME Bioformats does not do this properly. (Note: the sub-files cannot simply be loaded as separate .oir files. See my previous post).


Linking back to viewtopic.php?f=13&t=8360 for the previous post mentioned above. From the OME standpoint, a short summary is that the Olympus OIR reader has no support for OIR datasets composed of multiple files such as the ones discussed above and in the previous thread. This is primarily due to the fact we had no such representative samples when the initial OIRReader was developed for Bio-Formats 5.5.0.

I would greatly appreciate Bioformats correcting this issue. If I upload a complete example of a large split .oir file, could someone work on this?


From the OME side, we can certainly record this format support addition in our public backlog. Given our current agenda and deadlines, it is unlikely the open-source team will have the time to prioritize this issue in the upcoming months.
It is possible that someone in the growing Bio-Formats contributing community can and wants to get involved. For this to happen, sharing representative datasets publicly is probably the best starting point if possible (see http://downloads.openmicroscopy.org/images/Olympus-OIR/ for instance).

Note, the Olympus Viewer ImageJ plugin handles these split files correctly, but it is not scriptable or usable outside of ImageJ.


Knowing that the Olympus Viewer ImageJ plugin works on this file is certainly great to hear since this is the reference external software we used to validate OIR filesets during the initial reader development.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: Olympus raw file support is incomplete (.oir)

Postby andalman » Thu Sep 07, 2017 8:28 pm

Sebastien, Thank you for the reply.

Obviously, it would be great if the OME team could prioritize this fix after the current deadlines pass. Olympus is shipping more and more systems that use this format and datasets larger than a 1Gb are no longer uncommon.

To this end, and in case the public is interested in working on this, I have hosted an example of a multipart .oir file on AWS. Here are the links to the 4 parts:
https://s3.amazonaws.com/edu.stanford.c ... tipart.oir
https://s3.amazonaws.com/edu.stanford.c ... part_00001
https://s3.amazonaws.com/edu.stanford.c ... part_00002
https://s3.amazonaws.com/edu.stanford.c ... part_00003
andalman
 
Posts: 6
Joined: Fri Sep 01, 2017 6:45 pm

Re: Olympus raw file support is incomplete (.oir)

Postby hailstonem » Mon Nov 06, 2017 11:19 am

Hi all,

I've also come up against this problem, using the new FV3000 which has very limited support for other file formats, and am struggling to deal with some quite large (60GB) datasets. I'd be happy to provide one (or a smaller test dataset) if it's useful, but it looks like Aaron already has.

Anyway, just wanted to add my vote for extended OIR support and am happy to test out any modifications to make it work.

Martin
hailstonem
 
Posts: 2
Joined: Thu Nov 02, 2017 4:00 pm

Re: Olympus raw file support is incomplete (.oir)

Postby sbesson » Thu Nov 09, 2017 6:04 pm

Hi Martin (and Aaron),

Thanks for notifying us of the importance of this issue. We have starting the investigation of the modifications required in the reader to support large OIR datasets- see this card for more details.

Having extra datasets to confirm the validity of this preliminary work would definitely be useful. If you can generate a fileset below 2GB, feel free to upload it at http://qa.openmicroscopy.org.uk/qa/upload/. Otherwise we can send you private instructions for uploading your data to a FTP server.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: Olympus raw file support is incomplete (.oir)

Postby hailstonem » Mon Nov 20, 2017 2:18 pm

Hi Sebastien,

Thanks for your help on this, it is very much appreciated. Most of the imaging I do is much larger than 2GB and I am happy to send a representative dataset that is ~30GB via FTP.

I am currently attempting to upload this one, which is under the 2GB limit:
20171020_JupGFPHisRFP_L3_30xSI_galvo_p5z_p1xy_better_than_nyquist.oir
20171020_JupGFPHisRFP_L3_30xSI_galvo_p5z_p1xy_better_than_nyquist_00001

However, as it is not supported correctly by OMERO, I'm unsure as to how successful this will be. My own uploads to OMERO result in a truncated dataset, which is not the intention.

Regards,
Martin
hailstonem
 
Posts: 2
Joined: Thu Nov 02, 2017 4:00 pm

Re: Olympus raw file support is incomplete (.oir)

Postby mlinkert » Mon Nov 20, 2017 2:36 pm

Hi Martin,

Thank you again for offering to send a few datasets. FTP instructions for large files have been sent to you via private message.

-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests