Replicate BioFormats bfconvert with Fiji/ImageJ

[Joshua]

Hi All,

I'm trying to convert some large files (Aperio SVS and Olympus VSI) using the command line bfconvert. When I do it takes ~12 hours for a sample dataset I have. If I use Fiji to open and export as OME-TIF it's about 2 minutes.

How do I replicate the way Fiji is saving this with bfconvert? Is there a way to spy on the methods fiji uses in general so I could create other workflows to save?

Thanks in advance!

-Joshua

[mlinkert]

Hi Joshua,

I'm trying to convert some large files (Aperio SVS and Olympus VSI) using the command line bfconvert. When I do it takes ~12 hours for a sample dataset I have. If I use Fiji to open and export as OME-TIF it's about 2 minutes.

How do I replicate the way Fiji is saving this with bfconvert? Is there a way to spy on the methods fiji uses in general so I could create other workflows to save?

'bfconvert' with no options will convert all of images in the source file to an OME-TIFF; when you export from Fiji, only the images in the currently focused window will be exported. Most likely, you'll want to use the '-series' option, and maybe the '-crop' option too, like this:

Code: Select all

bfconvert -series 4 -crop 0,0,512,512 source.svs converted.ome.tiff


That should give you the same results as if you had opened source.svs in Fiji (with the "Crop on import" option chosen), selected the 5th image in the "Bio-Formats Series Options" window, and set the crop options to X = 0, Y = 0, width = 512, height = 512, and then exported the resulting image.

If you aren't using the "Crop on import" option, then just this should work:

Code: Select all

bfconvert -series 4 source.svs converted.ome.tiff


...just be sure to adjust the series index accordingly.

Note too that we have plans to make OME-TIFF writing faster in general, as noted here:

http://trac.openmicroscopy.org.uk/ome/ticket/9329

Regards,
-Melissa

[Joshua]

Hey Melissa,

Unfortunately I've tried most combinations of switches to get this to work and it's just so much slower than Fiji or Omero server for importing (as a test appliance). Is there a way to spy on what fiji is doing?

J

[mlinkert]

Unfortunately I've tried most combinations of switches to get this to work and it's just so much slower than Fiji or Omero server for importing (as a test appliance). Is there a way to spy on what fiji is doing?

Looking more carefully at the code behind bfconvert, I think I see the problem. There is now a fix pending review for inclusion into the official builds:

https://github.com/melissalinkert/biofo ... dfd7f2e2c0

That fix brings conversion time with bfconvert down from a couple of hours to a couple of minutes with the few SVS files that I've tried.

[Joshua]

Hey Melissa,

Good to hear you made some positive improvements in the code! I took the changes and merged them into the latest source and compiled with

Code: Select all

'ant tools'

and it yielded a compiled loci_tools.jar. I then swapped out the .jar in the bftools dir and tried to run again which yielded the following error message,

[CellSens VSI] -> ../output.ome.tiff [OME-TIFF]
Exception in thread "main" loci.formats.FormatException: Image plane too large. Only 2GB of data can be extracted at one time. You can workaround the problem by opening the plane in tiles; for further details, see: http://www.openmicroscopy.org/site/supp ... -this-mean
at loci.formats.FormatReader.openBytes(FormatReader.java:760)
at loci.formats.ImageReader.openBytes(ImageReader.java:400)
at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:549)
at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:508)
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:456)
at loci.formats.tools.ImageConverter.main(ImageConverter.java:640)
Caused by: java.lang.IllegalArgumentException: Invalid array size: 0 x 29 x 1 x 2
at loci.common.DataTools.safeMultiply32(DataTools.java:883)
at loci.common.DataTools.allocate(DataTools.java:862)
at loci.formats.FormatReader.openBytes(FormatReader.java:757)

To get around this error I edited the bfconvert to give java some more mem, 16384m to be exact but it still errors out. This box has 32GB of ram so it should be OK. Since the array size *isn't* negative I'm not sure where to go. I'm not 100% sure I patched correctly as well.


-J

[mlinkert]

That error is partly due to the bug fixed by this commit:

https://github.com/melissalinkert/biofo ... 460090dd4d

(also not yet in the official builds, but you're welcome to try it out by building from source), and partly due to the fact that the tile sizes are not recorded properly for CellSens data, which is part of this ticketed bug:

http://trac.openmicroscopy.org.uk/ome/ticket/9376

Does the same command work correctly with an .svs input file?

[Joshua]

The command I used was,

Code: Select all

sh bfconvert -bigtiff -compression LZW  ../53676.svs ../output.ome.tiff

When used on an SVS file of 635MB it returns,

Reading IFDs
Populating metadata
Populating OME metadata
Expected positive value for PhysicalSizeX; got 0.0
Expected positive value for PhysicalSizeY; got 0.0
[Aperio SVS] -> ../../svsToTiff/output.ome.tiff [OME-TIFF]
Series 0: converted 1/1 planes (100%)
Series 1: converted 1/1 planes (100%)
Series 2: converted 1/1 planes (100%)
Series 3: converted 1/1 planes (100%)
Series 4: converted 1/1 planes (100%)
Series 5: converted 1/1 planes (100%)
Series 6: converted 1/1 planes (100%)
[done]
3980.517s elapsed (769.2857+567421.7ms per plane, 932ms overhead)

That's not too shabby! Much improved. Not too sure about the PhysX and Y sizes though.

Regarding vsi file conversion:
I applied the changes to the CellSens format reader and the changes to the imagereader to calc the area and then issue the same command as above on a 1MB vsi or corresponding 136MB ets file but it yields the familiar Can't open large images error. Even when I drop this file onto our head cluster node and give java 64GB of ram it still doesn't work. Confused as the cellsens is smaller in file size as well as pixel area.

-J

[mlinkert]

The command I used was,

Code: Select all

sh bfconvert -bigtiff -compression LZW  ../53676.svs ../output.ome.tiff


When used on an SVS file of 635MB it returns,

Reading IFDs
Populating metadata
Populating OME metadata
Expected positive value for PhysicalSizeX; got 0.0
Expected positive value for PhysicalSizeY; got 0.0
[Aperio SVS] -> ../../svsToTiff/output.ome.tiff [OME-TIFF]
Series 0: converted 1/1 planes (100%)
Series 1: converted 1/1 planes (100%)
Series 2: converted 1/1 planes (100%)
Series 3: converted 1/1 planes (100%)
Series 4: converted 1/1 planes (100%)
Series 5: converted 1/1 planes (100%)
Series 6: converted 1/1 planes (100%)
[done]
3980.517s elapsed (769.2857+567421.7ms per plane, 932ms overhead)

That's not too shabby! Much improved. Not too sure about the PhysX and Y sizes though.

OK, great! The PhysicalSizeX and PhysicalSizeY warnings are nothing to be concerned about in this case.

Regarding vsi file conversion:
I applied the changes to the CellSens format reader and the changes to the imagereader to calc the area and then issue the same command as above on a 1MB vsi or corresponding 136MB ets file but it yields the familiar Can't open large images error. Even when I drop this file onto our head cluster node and give java 64GB of ram it still doesn't work. Confused as the cellsens is smaller in file size as well as pixel area.

Sorry, which changes specifically? If you just merged this commit:

https://github.com/melissalinkert/biofo ... 460090dd4d

that wouldn't be sufficient; you would need to do something like this (assuming you are in the root of your clone of bioformats.git, and 'origin' points to openmicroscopy/bioformats.git on GitHub):

Code: Select all

git fetch origin
git remote add melissalinkert git://github.com/melissalinkert/bioformats.git
git fetch melissalinkert
git checkout origin/develop
git merge melissalinkert/sprint1.2-bug-fixes
ant clean tools


If you follow those steps, do you still see an error message with the CellSens files?

-Melissa

[Joshua]

Works!! Very quickly at that (5 mins for large file).

The only question I have remaining is when using bfconvert and the compression switch, how do I figure out all the supported combinations? I would like to use JPEG2000 lossless compression for these files but can't seem to find a list of available options or syntax. It would be useful to learn of the different combinations I can use.

Thanks so much for VSI support!

-J

[mlinkert]

Works!! Very quickly at that (5 mins for large file).

Glad to hear it!

The only question I have remaining is when using bfconvert and the compression switch, how do I figure out all the supported combinations? I would like to use JPEG2000 lossless compression for these files but can't seem to find a list of available options or syntax. It would be useful to learn of the different combinations I can use.

This is not really exposed in bfconvert, but you can see the list of compression types for each output format by reading the relevant source code. For OME-TIFF, that would be:

https://github.com/openmicroscopy/biofo ... r.java#L69

and

https://github.com/openmicroscopy/biofo ... nType.java

So to get lossless JPEG-2000 compression, using the '-compression JPEG-2000' option should work.

-Melissa