[C++] Compression with writer

[thomas_braun]

Hi,

I'm in the process of writing an application which converts from a proprietary dataformat to OME-TIFF using the C++ libraries (ome-files-cpp 0.2.2, ome-common 5.3.2, bioformats 5.2.3).

I played around with compression

Code: Select all

auto writer = make_shared<ome::files::out::OMETIFFWriter>();
auto compressionTypes = writer->getCompressionTypes();

std::cout << "List of supported compression types" << std::endl;
for(auto compType : compressionTypes)
   std::cout << "Compression: " << compType << std::endl;


but the set of supported compression types is always empty.

Is my compilation of the OME libraries faulty or incomplete?

Thanks,
Thomas

[rleigh]

Compression support is not yet completed, which is why an empty list is returned. Most of the pieces are in place for basic compression formats, but it needs some final work to (1) fill the list with the supported compression types introspected from libtiff and (2) enable the compression when writing.

If using the platform-provided libtiff, it should also support jpeg compression, but other compression types such as jpeg2000 will need extra support. If using the super-build, it's currently limited to zlib and compression types built into libtiff; we would need to add libjpeg and other compression libraries to the superbuild (I'm currently working on libjpeg-turbo for this purpose).

I have added initial support for this tentatively as an item for the 0.3.0 release: https://trello.com/c/IQdFu4Ls/53-add-ti ... on-support. If there's time to fit it in I'll do this, or else it will be done in a later point release.


Kind regards,
Roger

[rleigh]

Dear Thomas,

I have opened an initial pull request adding compression support, which you can find here:

https://github.com/ome/ome-files-cpp/pull/42

This adds support for LZW, Deflate and any other codecs which libtiff was compiled with e.g. JPEG, and all of the other built in ones.

If you wanted to make use of this prior to it being included in a release, please feel free to cherry-pick this commit. I would certainly be interested in any feedback you might have.


Kind regards,
Roger

[thomas_braun]

Hi Roger,

I'm just came back to this project of mine.

I've compiled and installed the latest version of ome-files-cpp from the repository.
Now the list of compression types is filled but I'm not able to use compression.

I've modified one of the supplied examples:

Code: Select all

/*
* #%L
* OME-BIOFORMATS C++ library for image IO.
* Copyright © 2015 Open Microscopy Environment:
*   - Massachusetts Institute of Technology
*   - National Institutes of Health
*   - University of Dundee
*   - Board of Regents of the University of Wisconsin-Madison
*   - Glencoe Software, Inc.
* %%
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice,
*    this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
*    this list of conditions and the following disclaimer in the documentation
*    and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* The views and conclusions contained in the software and documentation are
* those of the authors and should not be interpreted as representing official
* policies, either expressed or implied, of any organization.
* #L%
*/

#include <iostream>

#include <ome/files/CoreMetadata.h>
#include <ome/files/Version.h>
#include <ome/files/MetadataTools.h>
#include <ome/files/VariantPixelBuffer.h>
#include <ome/files/out/OMETIFFWriter.h>
#include <ome/xml/meta/OMEXMLMetadata.h>

#include <ome/compat/memory.h>

#include <ome/common/filesystem.h>

using boost::filesystem::path;
using ome::compat::make_shared;
using ome::compat::shared_ptr;
using ome::files::dimension_size_type;
using ome::files::fillMetadata;
using ome::files::CoreMetadata;
using ome::files::DIM_SPATIAL_X;
using ome::files::DIM_SPATIAL_Y;
using ome::files::DIM_SUBCHANNEL;
using ome::files::FormatWriter;
using ome::files::MetadataMap;
using ome::files::out::OMETIFFWriter;
using ome::files::PixelBuffer;
using ome::files::PixelBufferBase;
using ome::files::PixelProperties;
using ome::files::VariantPixelBuffer;
using ome::xml::model::enums::PixelType;
using ome::xml::model::enums::DimensionOrder;

namespace
{

  /* write-example-start */
  shared_ptr< ::ome::xml::meta::OMEXMLMetadata>
  createMetadata()
  {
    // OME-XML metadata store.
    shared_ptr< ::ome::xml::meta::OMEXMLMetadata> meta(make_shared< ::ome::xml::meta::OMEXMLMetadata>());

    // Create simple CoreMetadata and use this to set up the OME-XML
    // metadata.  This is purely for convenience in this example; a
    // real writer would typically set up the OME-XML metadata from an
    // existing MetadataRetrieve instance or by hand.
    std::vector<shared_ptr<CoreMetadata> > seriesList;
    shared_ptr<CoreMetadata> core(make_shared<CoreMetadata>());
    core->sizeX = 512U;
    core->sizeY = 512U;
    core->imageCount = 1U;
    core->sizeC.clear(); // defaults to 1 channel with 1 subchannel; clear this
    core->sizeC.push_back(3U); // replace with single RGB channel
    core->pixelType = ome::xml::model::enums::PixelType::UINT16;
    core->interleaved = false;
    core->bitsPerPixel = 12U;
    core->dimensionOrder = DimensionOrder::XYZTC;
    seriesList.push_back(core);
    // seriesList.push_back(core);

    fillMetadata(*meta, seriesList);

    return meta;
  }
  /* write-example-end */

  /* pixel-example-start */
  void
  writePixelData(FormatWriter& writer,
                 std::ostream& stream)
  {
    // Total number of images (series)
    dimension_size_type ic = 1;
    // stream << "Image count: " << ic << '\n';

    // Loop over images
    for (dimension_size_type i = 0 ; i < ic; ++i)
      {
        // Change the current series to this index
        writer.setSeries(i);

        // Total number of planes.
        dimension_size_type pc = 1;
        // stream << "\tPlane count: " << pc << '\n';

        // Loop over planes (for this image index)
        for (dimension_size_type p = 0 ; p < pc; ++p)
          {
            // Pixel buffer; size 512 × 512 with 3 subchannels of type
            // uint16_t.  It uses the native endianness and has a
            // storage order of XYZTC without interleaving
            // (subchannels are planar).
            shared_ptr<PixelBuffer<PixelProperties<PixelType::UINT16>::std_type> >
              buffer(make_shared<PixelBuffer<PixelProperties<PixelType::UINT16>::std_type> >
                     (boost::extents[512][512][1][1][1][3][1][1][1],
                      PixelType::UINT16, ome::files::ENDIAN_NATIVE,
                      PixelBufferBase::make_storage_order(DimensionOrder::XYZTC, false)));

            // Fill each subchannel with a different intensity ramp in
            // the 12-bit range.  In a real program, the pixel data
            // would typically be obtained from data acquisition or
            // another image.
            for (dimension_size_type x = 0; x < 512; ++x)
              for (dimension_size_type y = 0; y < 512; ++y)
                {
                  PixelBufferBase::indices_type idx;
                  std::fill(idx.begin(), idx.end(), 0);
                  idx[DIM_SPATIAL_X] = x;
                  idx[DIM_SPATIAL_Y] = y;

                  idx[DIM_SUBCHANNEL] = 0;
                  buffer->at(idx) = (static_cast<float>(x) / 512.0f) * 4096.0f;
                  idx[DIM_SUBCHANNEL] = 1;
                  buffer->at(idx) = (static_cast<float>(y) / 512.0f) * 4096.0f;
                  idx[DIM_SUBCHANNEL] = 2;
                  buffer->at(idx) = (static_cast<float>(x+y) / 1024.0f) * 4096.0f;
                }

            VariantPixelBuffer vbuffer(buffer);
            // stream << "PixelBuffer PixelType is " << buffer->pixelType() << '\n';
            // stream << "VariantPixelBuffer PixelType is " << vbuffer.pixelType() << '\n';
            // stream << std::flush;

            // Write the the entire pixel buffer to the plane.
            writer.saveBytes(p, vbuffer);

            // stream << "Wrote " << buffer->num_elements() << ' ' << buffer->pixelType() << " pixels\n";
          }
      }
  }
  /* pixel-example-end */

}

int
main(int argc, char *argv[])
{
  try
    {
      std::cout << ome::files::release_version << std::endl;

      if (argc > 1)
        {
          // Portable path
          path filenameBase(argv[1]);

          // Create TIFF writer
          shared_ptr<FormatWriter> writer(make_shared<OMETIFFWriter>());

          auto compressionTypes = writer->getCompressionTypes();

          std::cout << "List of supported compression types" << std::endl;
          int i = 0;
          for(auto compType : compressionTypes)
          {
               std::cout << "Compression: " << compType << std::endl;

              /* writer-example-start */
              // Create metadata for the file to be written.
              shared_ptr< ::ome::xml::meta::MetadataRetrieve> meta(createMetadata());

              // Set writer options before opening a file
              writer->setMetadataRetrieve(meta);
              writer->setInterleaved(false);

              path filename(filenameBase);
              filename += std::to_string(i++);
              filename += ".tiff";

              writer->setCompression(compType);

              // Open the file
              writer->setId(filename);

              // Write pixel data
              writePixelData(*writer, std::cout);

              // Explicitly close writer
              writer->close();
              /* writer-example-end */
          }
        }
      else
        {
          std::cerr << "Usage: " << argv[0] << " ome-xml.xml\n";
          std::exit(1);
        }
    }
  catch (const std::exception& e)
    {
      std::cerr << "Caught exception: " << e.what() << '\n';
      std::exit(1);
    }
  catch (...)
    {
      std::cerr << "Caught unknown exception\n";
      std::exit(1);
    }
}


And the looked at what tiffinfo tells me

$make ex && ./ex test
make: 'ex' is up to date.
0.2.3-28-gee87809
List of supported compression types
Compression: AdobeDeflate
Compression: CCITT Group 3
Compression: CCITT Group 4
Compression: CCITT RLE
Compression: CCITT RLE/W
Compression: Deflate
Compression: ISO JBIG
Compression: JPEG
Compression: LZMA
Compression: LZW
Compression: PackBits
Compression: default

$tiffinfo test*tiff | grep Compression
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None
Compression Scheme: None

Any hints what I'm doing wrong?

Thanks,
Thomas

[rleigh]

Dear Thomas,

I'm afraid you're doing everything correctly; it's an oversight on my part, so my apologies for that. The compression support was enabled in the TIFF code and the MinimalTIFFWriter writer, but not in the OMETIFFWriter. I forgot they had independent setupIFD methods, so only updated one of them. Please see https://github.com/ome/ome-files-cpp/pull/51 for a PR I just opened to rectify this.

One point to note from your example: not all compression types support all pixel types. When you call

Code: Select all

auto compressionTypes = writer->getCompressionTypes();

you might be better off using

Code: Select all

auto compressionTypes = writer->getCompressionTypes(PixelType::UINT16);

which will give you the types which are supported for 16-bit unsigned integers only. Some of the older algorithms in the list are limited to 1- or 2-bit greyscale (the CCITT and PackBits ones). Deflate and LZW are safe to use for all pixel types though.


Kind regards,
Roger

[thomas_braun]

Dear Roger,

thanks for the quick fix. I've tested PR and now it works.

I also ran into the problem, as you already hinted, that not all compression types are supported for every pixel type.

Thanks again!

Thomas