Calculating downsamples for a pyramid

[pbankhead]

Hi,

Is there a 'right' way to calculate a value representing how much each resolution of a pyramid is downsampled relative to the full resolution image?

What I've tried:

• Calculating it based on the ratio of the image sizes (in pixels), i.e. width & height
• Calculating it based on the physical pixel size (in microns) for each resolution

The first kind of works... usually. But it fails for some .vsi files (at least), and I assume that there may be additional padding/cropping in some formats across different resolutions. This method can also result in sufficiently different downsample values in x and y to be uncomfortable.

The second I haven't figured out how to do. I can use

Code: Select all

IMetadata meta = service.createOMEXMLMetadata();
...
Length xSize = meta.getPixelsPhysicalSizeX(series);
Length ySize = meta.getPixelsPhysicalSizeY(series);


but as far as I can tell, the required int requires the series and gives the full-resolution image pixel size and I don't see how I can get pixel sizes on a per-resolution basis. I also saw this thread, which leads me to think it might not be the way to go anyway.

Ultimately, QuPath requires a full-resolution pixel size and then an array of downsample values for each resolution in the pyramid... and I'm looking for the best way to populate that array.

Thanks!

Pete

[jmoore]

Hey Pete,

Is there a 'right' way to calculate a value representing how much each resolution of a pyramid is downsampled relative to the full resolution image?

I probably can't give you an as encouraging answer as you'd like. There are per format issues like https://trello.com/c/INs2Dw2I/147-incorrect-physical-pixel-sizes-with-pyramids, and in general, we (and formats) don't support variations in scale factor between resolutions in the same pyramid.

That being said, try out the code below. If it matches what you've already tried, then we'll need to dig deeper.

Code: Select all

import loci.common.DebugTools;
import loci.formats.ImageReader;
import loci.formats.CoreMetadata;
import loci.formats.MetadataTools;
import loci.formats.meta.IMetadata;

import ome.units.quantity.Length;

import java.util.Arrays;
import java.util.List;


public class Test {
   public static void main(String[] args) throws Exception {
      DebugTools.setRootLevel("info");
      for (String filename : args) {
         examinePyramids(filename);
      }
   }

   public static void examinePyramids(String filename) throws Exception {
      IMetadata ome = MetadataTools.createOMEXMLMetadata();
      ImageReader reader = new ImageReader();
      reader.setMetadataStore(ome);
      reader.setFlattenedResolutions(false);
      reader.setId(filename);


      List<CoreMetadata> cms = reader.getCoreMetadataList();
      int series = reader.getSeriesCount();
      System.out.println("Series count: " + series);

      for (int s = 0; s < series; s++) {
         System.out.println("Series " + s);
         reader.setSeries(s);

         int resolutions = reader.getResolutionCount();
         System.out.println("\tResolution count: " + resolutions);
         CoreMetadata core = null;
         List<Length> sizes = null;
         for (int i = 0; i < resolutions; i++) {
            int coreIndex = reader.seriesToCoreIndex(reader.getSeries()) + i;
            CoreMetadata cm = cms.get(coreIndex);
            System.out.println(String.format("\tResolution %s", i));
            System.out.println(String.format("\t\tDimensions: %s x %s", cm.sizeX, cm.sizeY));
            if (i == 0) {
               sizes = storedSizes(ome);
               core = cm;
            } else {
               calculatedSizes(core, sizes, cm);
            }
         }
      }
   }

   public static void printLength(Length l, String where, String what, Double how) {
      System.out.print(String.format("\t\t%s %s: %s %s", where, what, l.value(), l.unit().getSymbol()));
      if (how != null) {
         System.out.println(String.format(" (ratio: %s)", how));
      } else {
         System.out.println();
      }
   }

   public static List<Length> storedSizes(IMetadata ome) {
      Length phyX = ome.getPixelsPhysicalSizeX(0);
      Length phyY = ome.getPixelsPhysicalSizeY(0);
      printLength(phyX, "Stored", "X", null);
      printLength(phyY, "Stored", "Y", null);
      return Arrays.asList(phyX, phyY);
   }

   public static void calculatedSizes(CoreMetadata core, List<Length> sizes, CoreMetadata cm) {
      double ratioX = Math.ceil(core.sizeX / cm.sizeX);
      double ratioY = Math.ceil(core.sizeY / cm.sizeY);
      Length phyX0 = sizes.get(0);
      Length phyY0 = sizes.get(1);
      Length phyX = new Length(ratioX * phyX0.value().doubleValue(), phyX0.unit());
      Length phyY = new Length(ratioY * phyY0.value().doubleValue(), phyY0.unit());
      printLength(phyX, "Est.", "X", ratioX);
      printLength(phyY, "Est.", "Y", ratioY);
   }
}


The first kind of works... usually. But it fails for some .vsi files (at least), and I assume that there may be additional padding/cropping in some formats across different resolutions. This method can also result in sufficiently different downsample values in x and y to be uncomfortable.

Are these files you can show us? (Or at least similar output as here).

For me, the files from http://downloads.openmicroscopy.org/images/SVS/ produce:

Code: Select all

bf@2f59cb2751af:/tmp$ java -cp bftools/bioformats_package.jar:. Test /svs/77917.svs
15:41:48.782 [main] INFO  loci.formats.ImageReader - SVSReader initializing /svs/77917.svs
15:41:48.788 [main] INFO  loci.formats.in.MinimalTiffReader - Reading IFDs
15:41:48.865 [main] INFO  loci.formats.in.MinimalTiffReader - Populating metadata
15:41:48.878 [main] INFO  l.formats.in.JPEG2000MetadataParser - Unknown JPEG 2000 box 0x2f00 at 16
15:41:48.879 [main] INFO  l.formats.in.JPEG2000MetadataParser - File is a raw codestream not a JP2.
15:41:48.961 [main] INFO  loci.formats.in.BaseTiffReader - Populating OME metadata
Series count: 3
Series 0
   Resolution count: 5
   Resolution 0
      Dimensions: 96999 x 45667
      Stored X: 0.253 µm
      Stored Y: 0.253 µm
   Resolution 1
      Dimensions: 24249 x 11416
      Est. X: 1.012 µm (ratio: 4.0)
      Est. Y: 1.012 µm (ratio: 4.0)
   Resolution 2
      Dimensions: 6062 x 2854
      Est. X: 4.048 µm (ratio: 16.0)
      Est. Y: 4.048 µm (ratio: 16.0)
   Resolution 3
      Dimensions: 3031 x 1427
      Est. X: 8.096 µm (ratio: 32.0)
      Est. Y: 8.096 µm (ratio: 32.0)
   Resolution 4
      Dimensions: 1024 x 482
      Est. X: 23.782 µm (ratio: 94.0)
      Est. Y: 23.782 µm (ratio: 94.0)
Series 1
   Resolution count: 1
   Resolution 0
      Dimensions: 428 x 424
      Stored X: 0.253 µm
      Stored Y: 0.253 µm
Series 2
   Resolution count: 1
   Resolution 0
      Dimensions: 1280 x 431
      Stored X: 0.253 µm
      Stored Y: 0.253 µm


Cheers,
~Josh

[pbankhead]

Hi Josh,

Thanks very much indeed for this, sorry for the huge delay in trying this out and replying.

Your code does indeed do something similar to mine, but rather more nicely. This prompted me to do a larger revision.

I’m afraid I can’t share the images since they are not mine; I realise that’s not very helpful, although pretty much all of my .vsi woes (and, as far as I can tell from this forum, many of yours ) come from Oli. He may be able to provide some examples, if he has not done so already.

In any case, using the sample data from https://biop.epfl.ch/TOOL_VSI_Reader.html and your code, I get

Code: Select all

Series count: 4
Series 0
   Resolution count: 7
   Resolution 0
      Dimensions: 17149 x 7031
      Stored X: 3.4765997395808945 µm
      Stored Y: 3.476473408326059 µm
   Resolution 1
      Dimensions: 8575 x 3516
      Est. X: 6.952794044705699 µm (ratio: 1.9998833819)
      Est. Y: 6.9519580585965155 µm (ratio: 1.9997155859)
   Resolution 2
      Dimensions: 4288 x 1758
      Est. X: 13.903966635625235 µm (ratio: 3.9993003731)
      Est. Y: 13.903916117193031 µm (ratio: 3.9994311718)
   Resolution 3
      Dimensions: 2144 x 879
      Est. X: 27.80793327159813 µm (ratio: 7.9986007463)
      Est. Y: 27.807832234386062 µm (ratio: 7.9988623436)
   Resolution 4
      Dimensions: 1072 x 440
      Est. X: 55.6158665428486 µm (ratio: 15.9972014925)
      Est. Y: 55.5524648497068 µm (ratio: 15.9795454545)
   Resolution 5
      Dimensions: 536 x 220
      Est. X: 111.23173308604485 µm (ratio: 31.9944029851)
      Est. Y: 111.10492969976124 µm (ratio: 31.9590909091)
   Resolution 6
      Dimensions: 268 x 110
      Est. X: 222.46346617174206 µm (ratio: 63.9888059701)
      Est. Y: 222.2098593995225 µm (ratio: 63.9181818182)
Series 1
   Resolution count: 7
   Resolution 0
      Dimensions: 17487 x 22534
      Stored X: 3.4765997395808945 µm
      Stored Y: 3.476473408326059 µm
   Resolution 1
      Dimensions: 8744 x 11267
      Est. X: 6.952801880961511 µm (ratio: 1.9998856359)
      Est. Y: 6.952946816652118 µm (ratio: 2.0)
   Resolution 2
      Dimensions: 4372 x 5634
      Est. X: 13.905603761575362 µm (ratio: 3.9997712717)
      Est. Y: 13.90465952835255 µm (ratio: 3.9996450124)
   Resolution 3
      Dimensions: 2186 x 2817
      Est. X: 27.811207523498386 µm (ratio: 7.9995425435)
      Est. Y: 27.8093190567051 µm (ratio: 7.9992900248)
   Resolution 4
      Dimensions: 1093 x 1409
      Est. X: 55.62241504664911 µm (ratio: 15.9990850869)
      Est. Y: 55.59890119454858 µm (ratio: 15.9929027679)
   Resolution 5
      Dimensions: 547 x 705
      Est. X: 111.14314377707012 µm (ratio: 31.9689213894)
      Est. Y: 111.11893869968584 µm (ratio: 31.9631205674)
   Resolution 6
      Dimensions: 274 x 353
      Est. X: 221.88065564248166 µm (ratio: 63.8211678832)
      Est. Y: 221.9230928703395 µm (ratio: 63.835694051)
Series 2
   Resolution count: 8
   Resolution 0
      Dimensions: 31747 x 40401
      Stored X: 3.4765997395808945 µm
      Stored Y: 3.476473408326059 µm
   Resolution 1
      Dimensions: 15874 x 20201
      Est. X: 6.952980466936935 µm (ratio: 1.9999370039)
      Est. Y: 6.952774722527223 µm (ratio: 1.9999504975)
   Resolution 2
      Dimensions: 7937 x 10101
      Est. X: 13.90596093387387 µm (ratio: 3.9998740078)
      Est. Y: 13.904861119659023 µm (ratio: 3.9997029997)
   Resolution 3
      Dimensions: 3969 x 5051
      Est. X: 27.808418224234103 µm (ratio: 7.9987402368)
      Est. Y: 27.80696934656962 µm (ratio: 7.9986141358)
   Resolution 4
      Dimensions: 1985 x 2526
      Est. X: 55.60282716997039 µm (ratio: 15.9934508816)
      Est. Y: 55.6029303917688 µm (ratio: 15.9940617577)
   Resolution 5
      Dimensions: 993 x 1263
      Est. X: 111.14965954937942 µm (ratio: 31.970795569)
      Est. Y: 111.2058607835376 µm (ratio: 31.9881235154)
   Resolution 6
      Dimensions: 497 x 632
      Est. X: 222.07567793258306 µm (ratio: 63.8772635815)
      Est. Y: 222.23576292689526 µm (ratio: 63.9256329114)
   Resolution 7
      Dimensions: 249 x 316
      Est. X: 443.25948567268813 µm (ratio: 127.4979919679)
      Est. Y: 444.4715258537905 µm (ratio: 127.8512658228)
Series 3
   Resolution count: 1
   Resolution 0
      Dimensions: 620 x 212
      Stored X: 3.4765997395808945 µm
      Stored Y: 3.476473408326059 µm

The estimated values are similar for all resolutions, and so quite usable. I currently take the average of both x & y ratios as the downsample for the level; this appears to match the approach used in OpenSlide https://github.com/openslide/openslide/blob/7b99a8604f38280d14a34db6bda7a916563f96e1/src/openslide.c#L272

I’m not sure that it matches the behavior in OMERO, which may use the lower ratio - I base that on https://github.com/openmicroscopy/openmicroscopy/blob/v5.4.6/components/insight/SRC/org/openmicroscopy/shoola/env/rnd/data/ResolutionLevel.java#L96 but am not sure if that’s the right code to be comparing.

In any case, the difference here would be small. However, I do have one image with the following:

Code: Select all

Resolution count: 10
   Resolution 0
      Dimensions: 145240 x 91172
      Stored X: 3.4697641272849924 µm
      Stored Y: 3.4698876995499828 µm
   Resolution 1
      Dimensions: 72620 x 45056
      Est. X: 6.939528254569985 µm (ratio: 2.0)
      Est. Y: 7.021408943136313 µm (ratio: 2.0235262784)
   Resolution 2
      Dimensions: 36310 x 22528
      Est. X: 13.87905650913997 µm (ratio: 4.0)
      Est. Y: 14.042817886272626 µm (ratio: 4.0470525568)
   Resolution 3
      Dimensions: 18155 x 11264
      Est. X: 27.75811301827994 µm (ratio: 8.0)
      Est. Y: 28.085635772545253 µm (ratio: 8.0941051136)
   Resolution 4
      Dimensions: 9078 x 5632
      Est. X: 55.51316830206504 µm (ratio: 15.9991187486)
      Est. Y: 56.171271545437506 µm (ratio: 16.1882102273)
   Resolution 5
      Dimensions: 4539 x 2816
      Est. X: 111.02633660413008 µm (ratio: 31.9982374972)
      Est. Y: 112.34254309052801 µm (ratio: 32.3764204545)
   Resolution 6
      Dimensions: 2270 x 1408
      Est. X: 222.00376292804893 µm (ratio: 63.9823788546)
      Est. Y: 224.68508618140302 µm (ratio: 64.7528409091)
   Resolution 7
      Dimensions: 1135 x 704
      Est. X: 444.00752585644483 µm (ratio: 127.9647577093)
      Est. Y: 449.37017236280604 µm (ratio: 129.5056818182)
   Resolution 8
      Dimensions: 568 x 352
      Est. X: 887.2333483219143 µm (ratio: 255.7042253521)
      Est. Y: 898.7403447256121 µm (ratio: 259.0113636364)
   Resolution 9
      Dimensions: 284 x 176
      Est. X: 1774.4666966438285 µm (ratio: 511.4084507042)
      Est. Y: 1797.480689450877 µm (ratio: 518.0227272727)

Previously, QuPath simply failed to open the image because it gave up when it noticed that the calculated x & y downsampling values deviated substantially.

It appears the ‘damage’ is done at the first downsampling, where pixels are lost vertically (i.e. 91172/2 = 45586, but the height of the second level is 45056). Consequently whenever I rescale my tile requests for different pyramid levels under the assumption that the image is indeed 145240 x 91172, I end up requesting pixels that are beyond the bounds of the images at the lower levels. This produces, at least, strange boundary artefacts when zooming in and out.

As far as I can tell, it is ‘safe’ to assume that the downsample values for .vsi files are strictly powers of 2 and so I can handle this as a special case. I can also identify when pixels appear missing and effectively decrease the width or height of the full resolution to restrict it to only include pixels valid at all resolutions. I need to investigate further how this behaves across more images, but in principle I can work around it if necessary.

Nevertheless, given that the downsampling ratios based on image dimensions can vary somewhere between a little and a lot, I was hoping there was some solution I was missing. Of course it's fine if not, but either way I’d like to match the QuPath behavior to however this may be handled in other software - for which any suggestions are welcome.

Thanks again,

Pete

[pbankhead]

Sorry, one other thing...

Your code contains:

Code: Select all

Length phyX = ome.getPixelsPhysicalSizeX(0);
Length phyY = ome.getPixelsPhysicalSizeY(0);

Should this really have the series number, rather than 0, to return the appropriate resolution?

At least I have been passing the series number previously, and that appears to work. But I do find myself a bit confused between the use of 'series' and 'image' in general, and suspect I'm missing something conceptually.

[dgault]

Hi Pete, yeah in the example given it shouldn't be hardcoded to 0 and you should use the series.

The different terminology can be confusing but the image index in the metadata API will be the same as the series index in the reader and writer API. For example if you switch flattened resolutions on you should still see

Code: Select all

reader.getSeriesCount()

; and

Code: Select all

ome.getImageCount();

return the same values.

[pbankhead]

Great, thanks! Now I can check that this matches what I was actually doing... I think it does, since it appeared to be working, but it helps a lot to better understand why.

[pbankhead]

Thanks again to the OME team for all your help, here and at the OME meeting.

I've updated the QuPath extension and written a blog post about it here.

It remains a work in progress, and any fixes and suggestions are welcome.