new reader struggle

[Albin]

Code: Select all

    public static void main(String[] args) throws IOException {
        boolean little = false;
        String id = "/Users/Hedwall/Desktop/C0003094.ISQ";
        in = new RandomAccessInputStream(id);
        in.order(little);
        int[] test = new int[32];
        for(int i=0; i<test.length; i++) {
            test[i] = in.readInt();
        }
         for(int j=0; j<test.length;j++){
            System.out.println(test[j]);
            }
    }

Made a very simple read for the first 32 int, the values that I receive are very weird and I don't understand why since no optional type work either. Any tips on how to solve the type problem?

//Albin

[dgault]

It appears that the byte order is little Endian, so setting the stream order before reading the int values appears to provide sensible values:

Code: Select all

            String imageName = in.readString(40); //name at char[40]
            in.seek(0);
            String first16 = this.in.readString(16);
            boolean identifier = first16.startsWith("CTDATA-HEADER_V1");
            CoreMetadata m = this.core.get(0);

            if (identifier = true) {

                //in.seek(43);
                //in.skipBytes(27);
                /**check position for seek() function against MatLab, which value is correct */
              in.order(!in.isLittleEndian());
                int dataType = (int) in.readUnsignedInt();
                int numBytes = (int) in.readUnsignedInt();
                int numBlocks = (int) in.readUnsignedInt();
                int patientIndex = (int) in.readUnsignedInt();
                int scnanerId = (int) in.readUnsignedInt();
                int creationDate1 = (int) in.readUnsignedInt();
                int creationDate2 = (int) in.readUnsignedInt();
             
                //dimx_p is the dimension in pixels, dimx_um the dimension in microns.
                m.sizeX = (int) in.readInt(); //dimx_p [12] @44
                //in.skipBytes(4);
                m.sizeY = (int) in.readUnsignedInt(); //dimy_p [13] @48
                //in.skipBytes(4);
                m.sizeZ = (int) in.readUnsignedInt(); //dimz_p [14] = nr of slices in current series  @52


which provides the values:

Code: Select all

first16 = "CTDATA-HEADER_V1"
dataType = 3
numBytes = 341789184
numBlocks = 667557
sizeX = 1040
sizeY = 1040
sizeZ = 158


Note: I also added seek(0) after reading the image name which now provides the correct value for the first 16 chars

The next error after that is for a missing instrument when setting the DetectorOffset, for that you will need to create a Detector first which can be done using

Code: Select all

        String detectorID = MetadataTools.createLSID("Detector", 0, i);
        store.setDetectorID(detectorID, 0, i);


[Albin]

Hi thanks for help:)


I have now tried the things you pointed out and also tried to put some random numbers into offsets and setDetectorSettingsOffset, but nothing happens to the image which means that we are not sure if the store settings is even used. Any tips on how to try this out?


The image result by image launcher should be a circle and not two semi-circles which we think is an offset error. Also we get very random pixel values inside the semicircles which seems to mean either wrong data type or one byte offset error. The area outside the circles are probably 0s and therefore not effected by the things mentioned above.

Would love to upload the image to but it is to large.


//Albin

[Albin]

Code: Select all

package loci.formats.in;


import java.io.FileInputStream;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;

import loci.common.Location;
import loci.common.RandomAccessInputStream;
import loci.formats.*;
import loci.formats.meta.MetadataStore;

import loci.formats.tools.ImageInfo;
import ome.units.quantity.Length;
import ome.units.quantity.Time;
import ome.units.UNITS;



//the images are 16-bit unsigned integers, 1024x1024, with a 512-byte header.

public class ISQReader extends FormatReader {


    // -- Constants --
    private static final String ISQ_MAGIC_STRING = "CTDATA-HEADER_V1";
    private static boolean useMappedByteBuffer;
    private static final String ISQ_FILE = "";


    // -- Fields --
    private RandomAccessInputStream pixelFile;
    public int[] offsets;
    private long pixelOffset;
    private short nDimensions;

    // -- Constructor --
    /** Constructs a new ISQ reader. */
    public ISQReader() {
        super("ISQ", "ISQ");
        LOGGER.info("@constructor");
        suffixSufficient = false;
        domains = new String[]{"X-ray Imaging"};
        datasetDescription = "a single .isq file";
    }


    // -- IFormatReader API methods --


    /* @see loci.formats.IFormatReader#isThisType(RandomAccessInputStream) */
    public boolean isThisType(RandomAccessInputStream stream) throws IOException {
        final int blockLen = 512;
        LOGGER.info("@type checker");
        if (!FormatTools.validStream(stream, blockLen, false)) {
            return false;
        } else {
            //double check this, not sure if correct
            stream.seek(508L);
            String lastFour = stream.readString(4);
            return lastFour.equals("ISQ");
        }

    }

    /** @see loci.formats.IFormatReader#openBytes(int, byte[], int, int, int, int) */
    //Obtains a sub-image of the specified image plane into a pre-allocated byte array.
    public byte[] openBytes(int no, byte[] buf, int x, int y, int w, int h) throws FormatException, IOException {
        FormatTools.checkPlaneParameters(this, no, buf.length, x, y, w, h);
        this.in.seek((long)(this.offsets[no]));

        for(int row = h - 1; row >= 0; --row) {
            this.in.skipBytes(x);
            this.in.read(buf, row * w, w);
            this.in.skipBytes(this.getSizeX() - w - x);
        }
        //this.readPlane(this.in, x, y, w, h,0, buf);

        //LOGGER.info("@openBytes method");
        return buf;
    }

    /** @see loci.formats.IFormatReader#close(boolean) */
    public void close(boolean fileOnly) throws IOException {
        super.close(fileOnly);
        if (!fileOnly) {
            this.offsets = null;
        }
    }
    // fler cases att täcka?


    // -- Internal FormatReader API methods --

    /** @see loci.formats.FormatReader#initFile(String) */
    @Override
    protected void initFile(String id) throws FormatException, IOException {
        super.initFile(id);
        this.in = new RandomAccessInputStream(id);
        this.in.order(true);
        boolean isSigned = true;

        LOGGER.info("Looking for header file");
        //String header = this.in.readString(128);

            /*String magic = in.readString(16);
            if (magic != ISQ_MAGIC_STRING) {
                throw new FormatException("Header file not found.");
            }*/

        LOGGER.info("Reading header");

            int [] META = new int[32];
            for(int i=0; i<META.length; i++){
                META[i] = this.in.readInt();
            }

            String imageName = this.in.readString(40); //name at char[40]
            this.in.seek(0);
            String first16checker = this.in.readString(16);
            if(first16checker.startsWith("CTDATA-HEADER_V1")) {

                int dataType = META[4];
                int numBytes = META[5];
                int numBlocks = META[6];
                int patientIndex = META[7];
                int scannerId = META[8];
                //String creation_date = in.readByteToString(4);
                CoreMetadata m = (CoreMetadata)this.core.get(0);

                //dimx_p is the dimension in pixels, dimx_um the dimension in microns.
                m.sizeX = META[11]; //dimx_p [12]
                //in.skipBytes(4);
                m.sizeY = META[12]; //dimy_p [13]
                //in.skipBytes(4);
                m.sizeZ = META[13]; //dimz_p [14] = nr of slices in current series

                if (this.getSizeT() == 0) {
                    m.sizeT = 1;
                }
                if (this.getSizeC() == 0) {
                    m.sizeC = 1;
                }
                if (this.getSizeZ() == 0) {
                    m.sizeZ = 1;
                }

                int x_um = META[15]; //dimx_um [15]
                int y_um = META[16]; //dimy_um [16]
                int z_um = META[17]; //dimz_um [17]

                if (getImageCount() == 0) {
                    m.imageCount = this.getSizeZ() * this.getSizeT() * this.getSizeC();
                } else {
                    m.imageCount = getImageCount();
                }
                this.offsets = new int[this.getImageCount()];


                /** 1. se till att få effekt när vi ändrar offset
                 * 2. Bruset på bilden kan vara orsakat av 1 bytes fel
                 * 3. Flytta bilden en halva till, så "cirkeln" blir hel*/

                LOGGER.info("Calculating image offsets");
                int offset;

                offset =  512 - 1;

                for (int i = 0; i < this.getSizeC(); ++i) {
                    for (int j = 0; j < this.getSizeZ(); j++) {
                        this.offsets[i * this.getSizeZ() + j] = offset + j * this.getSizeX() * this.getSizeY();

                    }
                    offset += this.getSizeX() * this.getSizeY() * this.getSizeZ();
                }

                this.addGlobalMeta("Image name", imageName);
                //this.addGlobalMeta("Creation date", date);
                this.addGlobalMeta("Number of dimensions", this.nDimensions);
                this.addGlobalMeta("first16", first16checker);
                this.addGlobalMeta("dataType", dataType);
                this.addGlobalMeta("nr_of_bytes", numBytes);
                this.addGlobalMeta("nr_of_blocks", numBlocks);
                this.addGlobalMeta("patient index", patientIndex);
                this.addGlobalMeta("scanner id", scannerId);
                //this.addGlobalMeta("creation date", creation_date);

                LOGGER.info("Populating metadata");
                m.seriesMetadata = getSeriesMetadata();
                m.dimensionOrder = "XYZCT";
                //m.rgb = false;
                m.thumbSizeY = 128;
                m.thumbSizeX = 128;
                m.pixelType = FormatTools.pixelTypeFromBytes(2, isSigned, false);
                m.metadataComplete = true;

            LOGGER.info("Populating OME metadata");
            // The metadata store we're working with.
            MetadataStore store = this.makeFilterMetadata();
            MetadataTools.populatePixels(store,this);
            store.setImageName(imageName, 0);
            //RandomAccessFile r = new RandomAccessFile(id,"r");
            if (this.getMetadataOptions().getMetadataLevel() != MetadataLevel.MINIMUM) {
                String instrumentID = MetadataTools.createLSID("Instrument", new int[]{0});
                store.setInstrumentID(instrumentID, 0);
                store.setImageInstrumentRef(instrumentID, 0);
                // populate Dimensions data
                //Formats the input value for the physical size into a length in microns
                Length sizeX = FormatTools.getPhysicalSizeX((double) x_um);
                Length sizeY = FormatTools.getPhysicalSizeY((double) y_um);
                Length sizeZ = FormatTools.getPhysicalSizeZ((double) z_um);
                if (sizeX != null) {
                    store.setPixelsPhysicalSizeX(sizeX, 0);
                }
                if (sizeY != null) {
                    store.setPixelsPhysicalSizeY(sizeY, 0);
                }
                if (sizeZ != null) {
                    store.setPixelsPhysicalSizeZ(sizeZ, 0);
                }
                for (int i = 0; i < this.getSizeC(); i++) {
                    store.setDetectorSettingsOffset((double) this.offsets[i], i, 0);
                    String detectorID = MetadataTools.createLSID("Detector", new int[]{0, i});
                    store.setDetectorID(detectorID, 0, i);
                    store.setDetectorType(this.getDetectorType("Other"), 0, i);
                    store.setDetectorSettingsID(detectorID, 0, i);

                    }

                }
            }

        }

    }




Why is the image only displayed at every second plane? It seems to work fine otherwise...

[dgault]

Apologies for the slow response in getting back to this, you seem to be making pretty good progress with the reader. I can reproduce the issue with every other frame as well, debugging through it shows the pixels are int16 and currently the openBytes is only reading a single byte per pixel.

So for both the offset calculation and the openBytes you will need to multiply by the number of bytes per pixel. This can be found using :

Code: Select all

int bpp = FormatTools.getBytesPerPixel(getPixelType());


The offsets would then be as below. Though it is worth noting that the pixelType would need to be set before the offset calculation, currently it occurs afterwards :

Code: Select all

                m.pixelType = FormatTools.pixelTypeFromBytes(2, isSigned, false);
                int bpp = FormatTools.getBytesPerPixel(getPixelType());
                for (int i = 0; i < this.getSizeC(); ++i) {
                    for (int j = 0; j < this.getSizeZ(); j++) {
                        this.offsets[i * this.getSizeZ() + j] = offset + j * this.getSizeX() * this.getSizeY() * bpp;

                    }
                    offset += this.getSizeX() * this.getSizeY() * this.getSizeZ() * bpp;
                }


And openBytes would use :

Code: Select all

        for(int row = h - 1; row >= 0; --row) {
            this.in.skipBytes(x);
            this.in.read(buf, row * w * bpp, w * bpp);
            this.in.skipBytes(this.getSizeX() - w - x);
        }


[Albin]

Hi thanks for replying:)!

Yes, I have got the reader to work now but in another way. Should I change it and use the code you provided to follow some kind of standard?

Also my goal is to implement this reader into Bioformats package, how do I do this? Read that you need to pass some test-suite before making a pull request. That's it or do I need to fill in some documentation as well?


kind regards
//Albin

[Albin]

This is what the test-suite gives me:

Albins-MacBook-Pro:test-suite Hedwall$ ant -Dtesting.directory=/Users/Hedwall/bioformats/components/formats-gpl/target/classes/loci/formats/in/metamorph test-all
Buildfile: /Users/Hedwall/bioformats/components/test-suite/build.xml
[echo] isSnapshot = true

mvn-init:

init-dependencies:
[echo] Loading dependency paths from file: /Users/Hedwall/bioformats/components/test-suite/build/compile-dependencies.xml

init-dependencies:
[echo] Loading dependency paths from file: /Users/Hedwall/bioformats/components/test-suite/build/test-dependencies.xml

init-dependencies:
[echo] Loading dependency paths from file: /Users/Hedwall/bioformats/components/test-suite/build/runtime-dependencies.xml

init-title:
[echo] ----------=========== bio-formats-testing-framework ===========----------

init-timestamp:

init:

copy-resources:

compile:

test-all:
[testng] 22:34:08,355 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
[testng] 22:34:08,355 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
[testng] 22:34:08,355 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/Users/Hedwall/bioformats/components/test-suite/logback.xml]
[testng] 22:34:08,576 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
[testng] 22:34:08,579 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
[testng] 22:34:08,640 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.sift.SiftingAppender]
[testng] 22:34:08,642 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SIFT]
[testng] 22:34:08,658 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [loci.tests.testng] to DEBUG
[testng] 22:34:08,658 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
[testng] 22:34:08,658 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SIFT] to Logger[ROOT]
[testng] 22:34:08,659 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[loci.tests.testng]
[testng] 22:34:08,659 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
[testng] 22:34:08,660 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@1786f9d5 - Registering current configuration as safe fallback point
[testng] 22:34:08,688 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [loci.tests.testng.TimestampedLogFileAppender]
[testng] 22:34:08,690 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [logfile-main]
[testng] 22:34:08,751 |-INFO in loci.tests.testng.TimestampedLogFileAppender[logfile-main] - File property is set to [target/bio-formats-test-main-2018-07-04_22-34-08.log]
[testng] [2018-07-04 22:34:08,757] [main] No base directory specified.
[testng] [2018-07-04 22:34:08,757] [main] Please specify a directory containing files to test:
[testng] [2018-07-04 22:34:08,757] [main] ant -Dtestng.directory="/path/to/data" test-all
[testng] [TestNG] Running:
[testng] Bio-Formats software test suite
[testng]
[testng]
[testng] ===============================================
[testng] Bio-Formats software test suite
[testng] Total tests run: 0, Failures: 0, Skips: 0
[testng] ===============================================
[testng]

BUILD SUCCESSFUL
Total time: 2 seconds

Feels like I should specify that my reader is the one to be used but where?

[dgault]

If you want to update the code you have on GitHub I would be happy to give it an initial review and test.

The test suite refers to the daily tests which we run against a large repository of filesets to ensure that new additions have not broken any existing behaviour. As this is a new reader we wont have any existing sample files configured for this format. If you are curious as to what the tests look like or how the process works then the first step is to generate the configurations for a fileset. This would be done by running the below command:

Code: Select all

ant -f components/test-suite/build.xml gen-config -Dtestng.directory=/path/to/apache_repo/QA_ID/ -Dtestng.configDirectory=/path/to/data_repo_config/curated/FORMAT_NAME/qa-QA_ID/


You can point both paths to a folder containing the filesets. Once run this will generate a .bioformats file containing the metadata values for the dataset. Then anytime you run the test suite again as you originally have it will compare the results with this configuration file and notify you if there have been any breakages.

When it comes to integrating a new reader into Bio-Formats itself there are a number of things we would have to take into account. The fact that there is a public spec for the format header is a good start. The other main considerations would be the level of demand for the reader, having a suitable sample set of files in our repository to run our daily tests against, and the future cost of being able to maintain the reader with bug fixes and new versions etc.

[Albin]

The code on Git is now updated : https://github.com/Hedwall/ISQReader/blob/master/src


Alright cool thanks! Will look into it:)



//Albin

[dgault]

Thanks, I retested with the latest code and it appears to be working well. Other than some small formatting changes the code itself is looking pretty good also. If useful it may also be worth populating some of the additional metadata fields from the header also. Feel free to let me know if you are making any updates or changes to the reader.