.nd2 -> .tif conversion issue (#2)

[user]

The uploaded file (test_T17_XY9_Z1.nd2) contains 17 time points, 9 series (X/Y points), and a single z-slice per point. The file opens properly using Fiji/Bioformats; however, bfconvert fails when converting to both .tiff and .ome.tiff:

Converting to .tiff (bfconvert -series 0 test_T17_XY9_Z1.nd2 tmp1.tiff) gives the correct number of time points (17), but the image data is corrupted.

Converting to .ome.tiff (bfconvert -series 0 test_T17_XY9_Z1.nd2 tmp1.ome.tiff) gives correct image data, but the final time point fails to load (16 instead of 17 time points).

Please advise on how both conversion issue can be resolved. Thank you.

PS: The same problem occurs for images with a single time point and multiple z-slices (i.e., conversion to .tiff produces a file with the correct number of slices but corrupted data, and conversion to .ome.tiff produces a file with uncorrupted pixels but missing one slice in the z-stack).

PPS: For another file having 6 timepoints, 14 series, and 31 z-points (test_T6_XY14_Z31.nd2, added to the upload queue), the following behavior is observed:

$ for ext in tiff ome.tiff; do bfconvert -series 0 -timepoint 0 test_T6_XY14_Z31.nd2 test_XY0_T0_Z31.${ext}; done
test_XY0_T0_Z31.tiff (corrupted data, correct number of fields, as above)
test_XY0_T0_Z31.ome.tiff (uncorrupted data, incorrect number of fields, as above)
$ for ext in tiff ome.tiff; do bfconvert -series 0 -z 0 test_T6_XY14_Z31.nd2 test_XY0_T6_Z0.${ext}; done
test_XY0_T6_Z0.tiff (conversion fails because of an invalid index; see below for exception)
test_XY0_T6_Z0.ome.tiff (conversion fails because of an invalid index; see below for exception)

Exception in thread "main" java.lang.IllegalArgumentException: Invalid image index: 31/6
at loci.formats.FormatTools.getZCTCoords(FormatTools.java:641)
at loci.formats.FormatTools.getFilename(FormatTools.java:1124)
at loci.formats.FormatTools.getFilename(FormatTools.java:1101)
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:537)
at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)

[dgault]

Hi,

Can you confirm which version of Bio-Formats you are currently using?
Also I was unable to find the mentioned uploaded files on our QA system, would it be possible to upload them again to https://www.openmicroscopy.org/qa2/qa/upload/

The first issue mentioned with the incorrect image counts looks like it may be related to a bug with Tiff writing which we have found present in Bio-Formats 5.3.0. We are currently working on putting a fix in place for this issue and testing is ongoing(linked below).
https://github.com/openmicroscopy/biofo ... -267576295

The second issue with the IllegalArgumentException is something which we will have to further investigate and test. I will link it to the investigation mentioned in the previous thread viewtopic.php?f=13&t=8180.

David

[user]

Hello,

The version I am using is:
Version: 5.3.0
Build date: 11 December 2016
VCS revision: 5.3.0

I re-added both files to the upload queue.

Thanks.

[sbesson]

Hi,

many thanks for reporting the corruption issue while writing to TIFF or OME-TIFF files. These were effectively cause by a regression in the TIFF writing functionality of Bio-Formats.

I would expect the first issue reported in this thread to be solved by Bio-Formats 5.3.1 which was just released. Could you upgrade your command-line tools to use this new version when exporting data and let us know if there is still corruption.

As mentioned by David, the second issue i.e. the indexing error while converting files using the command-line tools utility has been ticketed and we will try to have a look at it early 2017.

Best,
Sebastien

[user]

Hello,

Thanks for posting a bug fix. I can confirm that, when using bfconvert from bioformats v5.3.1, extraction of a time series from a 2D file (multi-X/Y series, multi-t) or a z-stack from a 3D file (multi-X/Y series, multi-t, multi-z) work (when writing to .ome.tiff but not to .tiff), whereas extraction of a time series from the same 3D file fails.

(1) extraction of a temporal sequence from a 2D .nd2 file with multiple series and multiple timepoints (but no z-stacks):

result: no java errors; both .tiff and .ome.tiff have the correct number of planes, but pixel data are corrupted for the former

example:

Code: Select all

bfconvert -series 0 test_T17_XY9_Z1.nd2 test_T17_XY0_Z1.tiff
        [outcome: corrupt data]
bfconvert -series 0 test_T17_XY9_Z1.nd2 test_T17_XY0_Z1.ome.tiff
        [outcome: successful conversion]


(2a) extraction of a z-stack from a 3D .nd2 file with multiple series, multiple timepoints, and multiple z-slices:

result: no java errors; both .tiff and .ome.tiff have the correct number of planes, but pixel data are corrupted for the former (i.e., same result as #1)

example:

Code: Select all

bfconvert -series 0 -timepoint 0 test_T6_XY14_Z31.nd2 test_T0_XY0_Z31.tiff
        [outcome: corrupt data]
bfconvert -series 0 -timepoint 0 test_T6_XY14_Z31.nd2 test_T0_XY0_Z31.ome.tiff
        [outcome: successful conversion]


(2b) extraction of a temporal sequence from a 3D .nd2 file with multiple series, multiple timepoints, and multiple z-slices:

result: java exception raised (Exception in thread "main" java.lang.IllegalArgumentException: Invalid image index: 31/6); both .tiff and .ome.tiff contain only one plane

example:

Code: Select all

bfconvert -series 0 -z 0 test_T6_XY14_Z31.nd2 test_T6_XY0_Z0.tiff
        [outcome: a single-page tiff]
bfconvert -series 0 -z 0 test_T6_XY14_Z31.nd2 test_T6_XY0_Z0.ome.tiff
        [outcome: a single-page tiff]


(3) Finally, for the issue reported in my first post (viewtopic.php?f=13&t=8180), with a 2D .nd2 input file (having multiple channels and multiple z-slices), writing a single-channel z-stack to either .tiff or .ome.tiff fails (Exception in thread "main" java.lang.IllegalArgumentException: Invalid image index: 24/22).

example:

Code: Select all

bfconvert -channel 0 field001.nd2 field001.tiff
        [outcome: corrupt pixel data, incorrect number of planes (9/22), correct channel]
bfconvert -channel 0 field001.nd2 field001.ome.tiff
        [outcome: same]


Thank you for your efforts to resolve these problems.

[sbesson]

Hi,

thanks for the quick feedback on the Bio-Formats 5.3.1 and the detailed information on the testing,

The IllegalArgumentException exceptions thrown in the last two scenarios are definitely part of the issue reported in the Trac ticket. We will review this code and get back to you.

One question regarding the TIFF/OME-TIFF pixel corruption. I ran the commands detailed in 1 and 2 using the provided nd2 files. However, I could not see any obvious difference between the two images when opening using a graphical client like Fiji. I also used the following code in MATLAB to open the TIFF data and this resulted in identical matrices

Code: Select all

info = imfinfo('test_T17_XY0_Z1.ome.tiff');
for k = 1:numel(info)
    I(:,:,k) = imread('test_T17_XY0_Z1.ome.tiff',k);
    I2(:,:,k) = imread('test_T17_XY0_Z1.tiff',k);
end
size(I)
isequal(I, I2)

info = imfinfo('test_T0_XY0_Z31.ome.tiff');
for k = 1:numel(info)
    I3(:,:,k) = imread('test_T0_XY0_Z31.ome.tiff',k);
    I4(:,:,k) = imread('test_T0_XY0_Z31.tiff',k);
end
size(I3)
isequal(I3, I4)


Can you give us more information about the way corruption appears on your system? And potentially more information about the environment (Operating System, Java version) allowing us to reproduce?

Best,
Sebastien

[user]

Hello,

I am running Debian 8 and JDK8 (openjdk 1.8). More detailed system information is included at the bottom of this message.

I ran your MATLAB code and also obtain true for the isequal() statements. Visualizing the images in MATLAB confirms this fact (following block adapted from the MATLAB code you posted):

Code: Select all

info = imfinfo('test_T17_XY0_Z1.ome.tiff');
for k = 1:numel(info)
    I1(:,:,k) = imread('test_T17_XY0_Z1.ome.tiff',k);
    I2(:,:,k) = imread('test_T17_XY0_Z1.tiff',k);
end
[y1,x1,z1]=size(I1);
isequal(I, I2)

info = imfinfo('test_T0_XY0_Z31.ome.tiff');
for k = 1:numel(info)
    I3(:,:,k) = imread('test_T0_XY0_Z31.ome.tiff',k);
    I4(:,:,k) = imread('test_T0_XY0_Z31.tiff',k);
end
[y3,x3,z3]=size(I3);
isequal(I3, I4)

addpath('~/MATLAB/sc'); % cf. https://www.mathworks.com/matlabcentral/fileexchange/16233-sc-powerful-image-rendering
figure('name','<-- I1 (.ome.tiff) | I2 (.tiff) -->'), imdisp(cat(2,reshape(I1,y1,x1,1,z1),reshape(I2,y1,x1,1,z1)),'Size',1);
figure('name','<-- I3 (.ome.tiff) | I4 (.tiff) -->'), imdisp(cat(2,reshape(I3,y3,x3,1,z3),reshape(I4,y3,x3,1,z3)),'Size',1);

The .tiff images appear corrupted when I view them in either ImageJ or Fiji. However, I just checked them in MATLAB, per your suggestion, as well as with the imagemagick suite and they are fine with both of those. I don't know what tiff libraries are used by ImageJ/Fiji and MATLAB, but I have libtiff5 v.4.0.7-1 on my system.

I attached an image demonstrating the corruption to which I alluded (see below). The left half is a montage of the .ome.tiff (I1) and the right half is a montage of the .tiff (I2). Interestingly, the very same files display normally in MATLAB. In any case, I suppose this latter issue should be raised with the ImageJ development team rather than the bioformats one. Thanks.

test_T17_XY0_Z1_OME-left_TIFF-right.jpg (224.66 KiB) Viewed 5308 times

---------

Detailed system information:

Code: Select all

$ uname -a
  Linux 4.8.0-2-amd64 #1 SMP Debian 4.8.11-1 (2016-12-02) x86_64 GNU/Linux
$ java -version
  openjdk version "1.8.0_111"
  OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3-b14)
  OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

[sbesson]

Hi,

thanks for the additional information. I can certainly reproduce your observations when opening the converted TIFF file using File > Open in my local ImageJ/Fiji distribution.

Using the Bio-Formats importer i.e. the Bio-Formats TIFF reader seems to be sufficient to read the file correctly. Hopefully this is a suitable workaround if you are using ImageJ/Fiji to read the converted TIFF files. Raising the other issue with the ImageJ development team definitely seems like the best course of action.

Best,
Sebastien

[user]

Note: for the unresolved issue (https://trac.openmicroscopy.org/ome/ticket/13316#ticket), the culprit lines are:

Code: Select all

int sizeC = retrieve.getPixelsSizeC(series).getValue();
int sizeT = retrieve.getPixelsSizeT(series).getValue();
int sizeZ = retrieve.getPixelsSizeZ(series).getValue();
int[] coordinates = FormatTools.getZCTCoords(order.getValue(), sizeZ, sizeC, sizeT, sizeZ*sizeC*sizeT, image);

in components/formats-api/src/loci/formats/FormatTools.java

Image is an int which ranges from 0 to the total number of images minux 1 (sizeZ*sizeT*sizeC). However, the sizes in these lines from FormatTools.java are *not* those for the entire image file, but rather for the subset to be written (e.g., in my case, dealing with a CxZxT = 1x31x145 input .nd2 file, and requesting a single z-stack at one channel/timepoint value, then the inputs fed to getZCTCoords are sizeC=1, sizeZ=31, sizeT=1, whereas the image integer can be from 0 to 4494 (assuming it's 0-based). so the exception handling in FormatTools.getZCTCoords will almost always fail.

In a previous commit (https://github.com/dgault/bioformats/bl ... Tools.java), you use the following line instead:

Code: Select all

int[] coordinates = r.getZCTCoords(image);

This works well, and so as a temporary workaround, I did the following in FormatTools.java: I re-pass the IFormatReader from the initial getFilename call to the subsequent one, and then use the reader's getZCTCoords instead of the function in FormatTools:

Code: Select all

  public static String getFilename(int series, int image, IFormatReader r,
      String pattern, boolean padded) throws FormatException, IOException
  {
     MetadataStore store = r.getMetadataStore();
     MetadataRetrieve retrieve = store instanceof MetadataRetrieve ?
       (MetadataRetrieve) store : new DummyMetadata();
     return getFilename(series, image, r, retrieve, pattern, padded);
  }

  public static String getFilename(int series, int image, IFormatReader r, MetadataRetrieve retrieve,
      String pattern, boolean padded) throws FormatException, IOException
  {
     ...
     //int[] coordinates = FormatTools.getZCTCoords(order.getValue(), sizeZ, sizeC, sizeT, sizeZ*sizeC*sizeT, image);
    int[] coordinates = r.getZCTCoords(image);
     ...
  }



Especially since I'm not relying on the auto-naming feature, this presents no problems, but I think it's fine with auto-naming as well. Seeing as I know very little about MetadataStore, I wasn't able to figure out how to get the correct values for sizeZ/sizeC/sizeT using the retrieve object. All I know about MetadataStore is that it extends something from .ome.xml.meta which I couldn't find in bioformats or here (https://github.com/ome/ome-model/tree/m ... e/xml/meta). Fom components/formats-api/src/loci/formats/meta/MetadataStore.java:

Code: Select all

package loci.formats.meta;
public interface MetadataStore extends ome.xml.meta.MetadataStore { }


[dgault]

Hi,

Thank you for the excellent investigation work and glad to here that you have managed to get a work around sorted for the time being. The behaviour you have uncovered is rather unusual, as usually the values returned from the reader and metadataStore should be identical. The fact that they are returning different values in itself suggested that something else going wrong in this instance.

I was able to do some further debugging this morning and it appears that the root of the issue is in the ImageConverter itself. When using the converter tool with the -z option but without -series, the converter was inadvertently overwriting sizeZ on the readers metadataStore object while setting up the writer. This is a bug in the ImageConverter and I have some initial code which appears to resolve the issue. I shall carry out some further testing and hopefully have a code PR open later this week to put a fix in place.

Thanks again for the detailed investigation which has been a great help.

David Gault