Memory issues with bfconvert

[jmichael]

Hello,

I am trying to convert a .czi file to .tiff with 'bfconvert' and am getting failures due to Java memory issues.

My code is

Code: Select all

module load bioformats
fdir=/scratch_space/jmichael/LightMicroscopy/data
fname=filename

cmd="time bfconvert -bigtiff $fdir/$fname.czi $fname.tiff"
echo "Command is $cmd"
$cmd


with output

Code: Select all


Command is time bfconvert -bigtiff /scratch_space/jmichael/LightMicroscopy/data/filename.czi filename.tiff
Output file filename.tiff exists.
Do you want to overwrite it? ([y]/n)
y
/scratch_space/jmichael/LightMicroscopy/data/filename.czi
ZeissCZIReader initializing /scratch_space/jmichael/LightMicroscopy/data/filename.czi
[Zeiss CZI] -> filename.tiff [Tagged Image File Format]
Tile size = 1024 x 1024
   Series 0: converted 1/1 planes (100%)
Tile size = 1024 x 1024
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
   at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:323)
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:273)
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270)
   at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:744)
   at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:630)
   at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:559)
   at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)
627.21user 18.15system 9:19.73elapsed 115%CPU (0avgtext+0avgdata 2140928maxresident)k
0inputs+2336outputs (0major+6825080minor)pagefaults 0swaps


If I edit my original code with

Code: Select all

export BF_MAX_MEM=30g

then I get a bit further but my my job still fails due to Java memory issues:

Code: Select all

Command is time bfconvert -bigtiff /scratch_space/jmichael/LightMicroscopy/data/filename.czi filename.tiff
Output file filename.tiff exists.
Do you want to overwrite it? ([y]/n)
y
/scratch_space/jmichael/LightMicroscopy/data/filename.czi
ZeissCZIReader initializing /scratch_space/jmichael/LightMicroscopy/data/filename.czi
[Zeiss CZI] -> filename.tiff [Tagged Image File Format]
Tile size = 1024 x 1024
   Series 0: converted 1/1 planes (100%)
Tile size = 1024 x 1024
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
   at java.util.Arrays.copyOf(Arrays.java:3236)
   at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
   at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
   at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135)
   at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:346)
   at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:273)
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270)
   at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:744)
   at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:630)
   at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:559)
   at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)
570.11user 19.40system 9:41.87elapsed 101%CPU (0avgtext+0avgdata 30623312maxresident)k
0inputs+2320outputs (0major+6664982minor)pagefaults 0swaps


How can I go about changing the use of bfconvert so that memory is not an issue? Please note that I am running on Linux with very large memory machines (multiple TB of RAM) so actual memory should not be an issue.

[sbesson]

Hi,

a couple of questions related to your issue:

• which version of Bio-Formats are you using?
• what are the dimensions and the pixel type of the CZI image you are trying to convert?
• have you tried running the bfconvert utility with smaller tile sizes using -tilex and -tiley to see if that would prevent the OutOfMemory issues?

Best,
Sebastien

[jmichael]

which version of Bio-Formats are you using?

I am using 5.3.2

what are the dimensions and the pixel type of the CZI image you are trying to convert?

This happens on two different attempts. First is, 190 MB, 1170 tiles, 43419 x 42274 pixels and the second is 5 GB, 1491 tiles, 145248 x 55319. We are using 16 bit pixels.

have you tried running the bfconvert utility with smaller tile sizes using -tilex and -tiley to see if that would prevent the OutOfMemory issues?

I have actually and it's still an error but no longer a memory error. I tried tilex and tiley at 512 and get a slightly different error:

Code: Select all

Command is time bfconvert -tilex 512 -tiley 512 /scratch_space/jmichael/LightMicroscopy/data/filename.czi filename_%x_%y_%m.tiff
/scratch_space/jmichael/LightMicroscopy/data/filename.czi
ZeissCZIReader initializing /scratch_space/jmichael/LightMicroscopy/data/filename.czi
[Zeiss CZI] -> filename_%x_%y_%m.tiff [Tagged Image File Format]
Tile size = 512 x 512
Series 0: converted 1/1 planes (100%)
Tile size = 512 x 512
Exception in thread "main" loci.formats.FormatException: Plane index:1 must be < 1
at loci.formats.FormatWriter.checkParams(FormatWriter.java:388)
at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:215)
at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:744)
at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:630)
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:559)
at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)
1190.74user 76.47system 20:53.02elapsed 101%CPU (0avgtext+0avgdata 9728560maxresident)k
0inputs+3216outputs (0major+18611092minor)pagefaults 0swaps


[rleigh]

For the last error you posted, please could you upload a file demonstrating this problem to http://qa.openmicroscopy.org.uk/qa/upload/ so that we can take a look at reproducing and fixing this?

For the general out of memory problem, it's also possible to set the BF_MAX_MEM environment variable to something large e.g. 8g to ensure that there is sufficient memory to hold the metadata and pixel data needed in memory for the conversion.


Kind regards,
Roger

[jmichael]

I don't believe I have permission to upload the images.

As per the memory issue - please see the original question. I am using BF_MAX_MEM to 30 GB and still getting memory issues.

[dgault]

Hi,

I have been able to spend time debugging and profiling the memory usage of the image converter command line tool today. Although I was able to find some CZI files which caused a large increase in memory, due to a large increase in the size of the tiff file being written out, I was unable to find any substantial memory leaks which would result in the issues you saw.

The first OutOfMemory exception is occurring as the TiffSaver is attempting to allocate an array larger than the max possible size (In this case it would 2^31-1). The dimensions you listed shouldn't exceed this, the fact that the first series appears to have been written successfully also seems odd that it only fails at the second series.

The second error you saw when you set the tiling parameters also seems to have successfully converted the first series and failed on the second.

Would it be possible to try and run the bfconvert on only the second series, using the option '-series 1', to verify that the issue is related to that specific series?

With Thanks,
David Gault

[jmichael]

As expected this fails very quickly at the second series:

Code: Select all

Command is time bfconvert -tilex 512 -tiley 512 -series 1 /scratch_space/jmichael/LightMicroscopy/data/filename.czi filename_%x_%y_%m.tiff
/scratch_space/jmichael/LightMicroscopy/data/filename.czi
ZeissCZIReader initializing /scratch_space/jmichael/LightMicroscopy/data/filename.czi
[Zeiss CZI] -> filename_%x_%y_%m.tiff [Tagged Image File Format]
Tile size = 512 x 512
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
   at java.util.ArrayList.rangeCheck(ArrayList.java:653)
   at java.util.ArrayList.get(ArrayList.java:429)
   at ome.xml.model.Pixels.getChannel(Pixels.java:632)
   at ome.xml.meta.OMEXMLMetadataImpl.getChannelSamplesPerPixel(OMEXMLMetadataImpl.java:1232)
   at loci.formats.FormatWriter.getSamplesPerPixel(FormatWriter.java:470)
   at loci.formats.out.TiffWriter.getPlaneCount(TiffWriter.java:451)
   at loci.formats.FormatWriter.setId(FormatWriter.java:353)
   at loci.formats.out.TiffWriter.setId(TiffWriter.java:154)
   at loci.formats.ImageWriter.setId(ImageWriter.java:481)
   at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:714)
   at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:630)
   at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:559)
   at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)
18.24user 5.17system 0:07.60elapsed 307%CPU (0avgtext+0avgdata 5843024maxresident)k
0inputs+1536outputs (0major+157844minor)pagefaults 0swaps


[dgault]

The IndexOutOfBoundsException being thrown when using "-series 1" is unfortunately a separate issue. The channel metadata looks like it is set on the first series and it is unable to find it. This should default to a sensible value rather than throwing the exception though, I will log this a separate bug to be fixed.

That still doesn't explain the initial OutOfMemory exceptions though. The point at which the exception occurs is when allocating a new array with a size far larger than it should be.

Code: Select all

     
      tileWidth = (int) ifd.getTileWidth();
      tileHeight = (int) ifd.getTileLength();
      int tilesPerRow = (int) ifd.getTilesPerRow();
      int rowsPerStrip = (int) ifd.getRowsPerStrip()[0];
      int stripSize = rowsPerStrip * tileWidth * bytesPerPixel;
      nStrips =
        ((w + tileWidth - 1) / tileWidth) * ((h + tileHeight - 1) / tileHeight);

      if (interleaved) stripSize *= nChannels;
      else nStrips *= nChannels;

      stripBuf = new ByteArrayOutputStream[nStrips];
      DataOutputStream[] stripOut = new DataOutputStream[nStrips];
      for (int strip=0; strip<nStrips; strip++) {
        stripBuf[strip] = new ByteArrayOutputStream(stripSize);  ** Point at which the exception is occurring **
        stripOut[strip] = new DataOutputStream(stripBuf[strip]);
      }


Are you able to find the Tiff IFD entries in the file for the following tags?
- ROWS_PER_STRIP (IFD Tag 278)
- TILE_WIDTH (IFD Tag 322)

[jmichael]

I can try to follow the TIFF formatting structure and write some C code to pull out the IFD's, but the chances of me screwing this up without a source of validation are pretty high.

How would you recommend going about finding these tags without writing my own source?

[dgault]

I have created a test branch of the Bio-Formats code to add some extra debugging to the ImageConverter: https://github.com/dgault/bioformats/co ... 52593cccad

This will print out extra statements with the values we are interested in.

You can clone this branch locally from https://github.com/dgault/bioformats/tr ... terExample
From there you can build it using the command

Code: Select all

ant tools

Once complete you can use bfconvert from the tools folder of this branch to test the file