BF command line tools // problem splitting Volocity series

[ehrenfeu]

Dear all,

we're experiencing problems using Bio-Formats to convert a Volocity (.mvd2) container to OME-TIFF.

We used the command line tools to extract single series into separate files per timepoint. Separating timepoints was done for multiple reasons, e.g. the whole series has an overall size of ~25GB, which would make the usage of BigTIFF necessary, which then again doesn't work properly with some of our other tools (Imaris, ...). On top of that, of course speed also matters and already opening all timepoints of the series for instance in FiJi takes more than half an hour. So separating timepoints was the way to go:

Code: Select all

bfconvert -bigtiff -no-upgrade -series 35 infile.mvd2 outfile_series-%s.ome.tif

After converting the series, we end up with 210 individual files, each about 120 MB in size (as expected). As explained above, opening them with FiJi takes ages (even though everything is on very fast local SSD drives), so we'd like to open the timepoints individually (for example to figure out where the interesting part happens). However, we didn't manage to do so. The Bio-Formats reader in FiJi always wants to read the whole series, no matter what options we chose in the dialog. Moving one of the individual files to a separate directory and then opening it with FiJi results in the following error message:

Code: Select all

loci.formats.FormatException: Unmatched UUID: urn:uuid:23dc5fe4-68df-4ace-95f4-9a954284d33d
   at loci.formats.in.OMETiffReader.initFile(OMETiffReader.java:449)
   at loci.formats.FormatReader.setId(FormatReader.java:1238)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:482)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:146)
   at loci.plugins.in.Importer.showDialogs(Importer.java:141)
   at loci.plugins.in.Importer.run(Importer.java:79)
   at loci.plugins.LociImporter.run(LociImporter.java:81)
   at ij.IJ.runUserPlugIn(IJ.java:184)
   at ij.IJ.runPlugIn(IJ.java:151)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at ij.Command.runPlugIn(Command.java:146)
   at ij.Command.runCommand(Command.java:95)
   at ij.Executer.run(Executer.java:64)
   at ij.IJ.run(IJ.java:250)
   at ij.macro.Functions.doRun(Functions.java:580)
   at ij.macro.Functions.doFunction(Functions.java:83)
   at ij.macro.Interpreter.doStatement(Interpreter.java:219)
   at ij.macro.Interpreter.doStatements(Interpreter.java:207)
   at ij.macro.Interpreter.run(Interpreter.java:104)
   at ij.macro.Interpreter.run(Interpreter.java:74)
   at ij.macro.Interpreter.run(Interpreter.java:85)
   at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:105)
   at ij.IJ.runMacro(IJ.java:103)
   at Action_Bar.run(Action_Bar.java:800)
   at java.lang.Thread.run(Thread.java:662)


Trying to read one of the timepoint.ome.tif files with Imaris gives us completely strange results with messed up z-information, channels and basically everything. So we can't use Imaris as a workaround either.

Being very desperate, I even tried Huygens Essential (the free version can be used as a converter for certain formats), but it complains about being unable to increase the image memory to ~48GB. So it clearly wants to open the full timeseries as well (even though there is just one file in the directory, and no matter which one of the series).

Any help is very much appreciated!
~Niko

[mlinkert]

we're experiencing problems using Bio-Formats to convert a Volocity (.mvd2) container to OME-TIFF.

We used the command line tools to extract single series into separate files per timepoint. Separating timepoints was done for multiple reasons, e.g. the whole series has an overall size of ~25GB, which would make the usage of BigTIFF necessary, which then again doesn't work properly with some of our other tools (Imaris, ...). On top of that, of course speed also matters and already opening all timepoints of the series for instance in FiJi takes more than half an hour. So separating timepoints was the way to go:

Code: Select all

bfconvert -bigtiff -no-upgrade -series 35 infile.mvd2 outfile_series-%s.ome.tif


That command won't split the timepoints into separate files - it will just convert series #35 into a single file. Did you try something like this instead?:

Code: Select all

bfconvert -bigtiff -no-upgrade -series 35 infile.mvd2 outfile_series-%s-t%t.ome.tif


After converting the series, we end up with 210 individual files, each about 120 MB in size (as expected). As explained above, opening them with FiJi takes ages (even though everything is on very fast local SSD drives), so we'd like to open the timepoints individually (for example to figure out where the interesting part happens). However, we didn't manage to do so. The Bio-Formats reader in FiJi always wants to read the whole series, no matter what options we chose in the dialog.

The "Specify range for each series" option should do what you want - just make sure to set the "T Begin" and "T End" options to the same value to open a single timepoint.

Moving files from a multi-file dataset is definitely not recommended - every file has all of the metadata for the entire dataset, so moving one of the files to a separate directory means that the reader will still look for all of the other files.

If "Specify range for each series" doesn't work, please let us know.

-Melissa

[ehrenfeu]

That command won't split the timepoints into separate files - it will just convert series #35 into a single file. Did you try something like this instead?:

Code: Select all

bfconvert -bigtiff -no-upgrade -series 35 infile.mvd2 outfile_series-%s-t%t.ome.tif


Sorry, that was a copy-paste error when creating this post - I actually used the correct "%t" (otherwise I probably wouldn't have received the 210 individual images)

The "Specify range for each series" option should do what you want - just make sure to set the "T Begin" and "T End" options to the same value to open a single timepoint.

It does, thanks for the hint (how could I miss that one?) - but the results are unfortunately not corresponding to the ones we can see using the original viewer (Volocity). At least they're consistent with the result in Imaris, so for one time it's at least not a Bitplane problem

I opened one of the series, specifying the range of 1 to 10 for T, but I do not yet fully understand what the problem is. Both channels and all z-slices show the same information (should be different), the first timepoint contains data from one channel, the remaining 9 contain data from the other channel. Really weird, I have to have a deeper look at this after lunch...

As a first (quick) try for a workaround, I used bfconvert again to really extract just one timepoint by adding the "-timepoint 0" parameter. Conversion works without any error, unfortunately FiJi complains with this message when trying to read the resulting file:

Code: Select all

java.lang.ArithmeticException: / by zero
   at loci.formats.MetadataTools.populatePixelsOnly(MetadataTools.java:277)
   at loci.formats.MetadataTools.populateMetadata(MetadataTools.java:235)
   at loci.formats.MetadataTools.populatePixels(MetadataTools.java:136)
   at loci.formats.in.OMETiffReader.initFile(OMETiffReader.java:810)
   at loci.formats.FormatReader.setId(FormatReader.java:1238)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:482)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:146)
   at loci.plugins.in.Importer.showDialogs(Importer.java:141)
   at loci.plugins.in.Importer.run(Importer.java:79)
   at loci.plugins.LociImporter.run(LociImporter.java:81)
   at ij.IJ.runUserPlugIn(IJ.java:184)
   at ij.IJ.runPlugIn(IJ.java:151)
   at ij.IJ.runPlugIn(IJ.java:140)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:192)


Sorry for participating in today's apocalypse

~Niko

[ehrenfeu]

As a first (quick) try for a workaround, I used bfconvert again to really extract just one timepoint by adding the "-timepoint 0" parameter. Conversion works without any error, unfortunately FiJi complains with this message when trying to read the resulting file:

Code: Select all

java.lang.ArithmeticException: / by zero
   at loci.formats.MetadataTools.populatePixelsOnly(MetadataTools.java:277)
   at loci.formats.MetadataTools.populateMetadata(MetadataTools.java:235)
   at loci.formats.MetadataTools.populatePixels(MetadataTools.java:136)
   at loci.formats.in.OMETiffReader.initFile(OMETiffReader.java:810)
   at loci.formats.FormatReader.setId(FormatReader.java:1238)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:482)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:146)
   at loci.plugins.in.Importer.showDialogs(Importer.java:141)
   at loci.plugins.in.Importer.run(Importer.java:79)
   at loci.plugins.LociImporter.run(LociImporter.java:81)
   at ij.IJ.runUserPlugIn(IJ.java:184)
   at ij.IJ.runPlugIn(IJ.java:151)
   at ij.IJ.runPlugIn(IJ.java:140)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:192)


So I guess that's just for the same reason as you stated above? Extracting parts of a dataset? The reason why I'm asking this is because FiJi shows the very same message when trying to use the "specify range" option on a dataset that is not yet fully converted (e.g. just the first 50 of the overall 210 separate files exist yet) ?

[mlinkert]

My apologies for not following up on this sooner.

To summarize, there are two open issues here:

1) Converting a single timepoint (or channel, or Z section) to OME-TIFF using 'bfconvert' results in an invalid file.

2) Converting a whole series to multiple files is successful, but the converted files have images that are wrong and/or out of order.

If there are other problems that I missed, please correct me; otherwise, fixes for both of those problems are now under review here:

https://github.com/openmicroscopy/bioformats/pull/328