importing dicom files from CTScanner

[derekeh]

From the directory structure output from the CTScanner I have created an OMERO dataset for each of the SE directories.

dicom_output.png (27.79 KiB) Viewed 1299 times

Within each of these directories I have created a pattern file which contains a single line in the form of IM<x-y> where IM is the generic name of the image and x and y are the sequence numbers of the files.

eg. IM<1-301> indicates that there are 301 files in the directory named IM1 up to IM301

You can see the idea from the screenshot of my database table (not omeroDB)

databaseEntries.png (110.79 KiB) Viewed 1299 times

On uploading this pattern file to OMERO, the log file for the import shows the pattern file being read from PA2/ST1/SE3

in which there are 301 files. (This is correct)

Then the importer imports a single image from each of PA2/ST1/SE2 and SE3

followed by reading all 376 files that are in both PA2/ST1/SE4 and SE5

This results in 3 series being created in OMERO

omeroDataset.png (72.51 KiB) Viewed 1299 times

The first one is a single image (looks like the first image of the 301 in SE3

The second is a high resolution sequence with 376 images in the sequence (SE4)

The third is a lower resolution sequence with 376 images in the sequence (SE5)

So I was expecting the pattern file IM<1-301> just to load the 301 images from SE3.

Is there something in a DICOM file in SE3 that links to these other sequences in SE4 and SE5?

The obvious thing I can see is the dcm:study_decription attribute but why does it not load SE3?

I'm sure that this has something to do with the way that the OMERO importer works but is it reading something from the DICOM files themselves?

Any help/pointers greatly appreciated.
OMERO 5.4.0
My application is Java.
Within that I am calling the CommandLineImporter to do the uploads as I have experienced compiling issues when using the OMEROMetadataStoreClient (Another story!)

Derek

[sbesson]

Hi Derek,

The first one is a single image (looks like the first image of the 301 in SE3

The second is a high resolution sequence with 376 images in the sequence (SE4)

The third is a lower resolution sequence with 376 images in the sequence (SE5)

So I was expecting the pattern file IM<1-301> just to load the 301 images from SE3.

Is there something in a DICOM file in SE3 that links to these other sequences in SE4 and SE5?

The obvious thing I can see is the dcm:study_decription attribute but why does it not load SE3?

There are a few modalities like DICOMDIR file structures under which DICOM files will be detected and grouped together by the Bio-Formats reader.

I'm sure that this has something to do with the way that the OMERO importer works but is it reading something from the DICOM files themselves?

[/quote][/quote]

Can these files be shared with us via QA? If this is not possible due to the sensitive information, a good starting point would be to know which files are grouped together at import time. For each DICOM file, the list of the files detected by `setId` can be computed using e.g. the Bio-Formats command line tools

Code: Select all

showinf -nopix -nogroup/path/to/file.dcm


or the OMERO command-line interface:

Code: Select all

bin/omero import -f /path/to/file.dcm


Best,
Sebastien

[derekeh]

Hi Sebastian,
Thanks for your reply.
I've attached the output for bin/omero import -f /path/to/SE3/IM1
and I can see that it lists the files for the 3 series that have been imported.

output.zip

(2.54 KiB) Downloaded 71 times

My strategy has been to treat each of the SE directories as a separate sequence.
I grab the dcm:studyDescription and dcm:seriesDescription from one of the files in the sequence (they all share the same values as far as I can see) and use this to create a dataset with name and description.
I then load the pattern file from the SE directory into that dataset.

Can I override the importer so that it does not look in other directories for linked files?
Or do you think I need to approach this in a different way altogether?

Note: If I isolate each of the SE directories from the DICOM output file structure then they load fine using the pattern file in each directory. However, I would prefer not to go down this road if possible and retain the file structure as it is.
ps. The file structure I have illustrated previously with PA[n] at the top is itself a subdirectory of the original CT output which includes the DICOMDIR file

Thanks
Derek

[derekeh]

Now I'm getting a bit confused.
I've attached 2 output files. 1 using bioformats -nogroup as a parameter and the other without the -nogroup parameter.

bf_out.zip

(4.74 KiB) Downloaded 77 times

I cannot see any reference to the other sequences (SE directories) in either output if that is what you were suggesting.
I get the same outputs when loading the pattern file instead of the image.

Also, when I import the pattern file into ImageJ using the bioformats importer, it loads just the single sequence.
ie. the files that are in the same directory as the pattern file, which is what I want.

So, why is Omero trying to load in what it perceives as the 'grouped' files?
I don't want it to do that.

I've tried removing the DICOMDIR level of files thinking Omero was reading the DICOMDIR file, but this did not solve the problem. It is still reading in 3 series from the SE2, SE4 and SE5 directories.

What are your referring to by 'setId' ?

Thanks again
Derek

[sbesson]

Hi Derek,

I've attached 2 output files. 1 using bioformats -nogroup as a parameter and the other without the -nogroup parameter.

I cannot see any reference to the other sequences (SE directories) in either output if that is what you were suggesting.
I get the same outputs when loading the pattern file instead of the image.

Also, when I import the pattern file into ImageJ using the bioformats importer, it loads just the single sequence.
ie. the files that are in the same directory as the pattern file, which is what I want.

Thanks for the additional testing, it is very informative. As far as I can tell, it get to the same conclusion as you i.e. your version of Bio-Formats does the right thing in terms of grouping.

So, why is Omero trying to load in what it perceives as the 'grouped' files?
I don't want it to do that.

You mentioned earlier your application was built using OMERO 5.4.0. Which version of Bio-Formats was used for the testing in the command-line and the ImageJ testing? It might be your local testing used a more recent of Bio-Formats with some changes. Also is your code available publicly?

What are your referring to by 'setId' ?

This refers to this API which initializes the reader from a give file, including the detection and the grouping of the dataset.

Best,
Sebastien

[derekeh]

Hi Sebastien,
Thanks for you reply.
Yes, the version of BioFormats used in the commandLine and in imageJ (using Fiji) was the latest version.
6.0.0

Can I send you my application code for the upload privately?

Can I update my version of Omero (5.4.0) to use the BioFormats 6 package or would I have to update to the latest version of Omero itself? (Presuming that this is the problem)

Derek

[derekeh]

Actually, I can post the relevant piece of code here.

Code: Select all

public String upload_via_omero_cli(String[] paths, long datasetID) {
            String RET="OK";
            ImportConfig config = new ImportConfig();           
            config.email.set("");
            config.sendFiles.set(true);
            config.sendReport.set(false);
            config.contOnError.set(false);
            config.debug.set(false);
            config.hostname.set(props.getProperty("server"));
            config.port.set(Integer.parseInt(props.getProperty("port")));
            config.username.set(props.getProperty("db_user"));
            config.password.set(props.getProperty("db_passwd"));
            config.target.set("Dataset:"+datasetID);
            boolean setting=false;
            CommandLineImporter cli=null;
            try {
                cli = new CommandLineImporter(config,paths,setting);
                cli.start();
            }
            catch(Exception e){RET="ERROR: "+e.getMessage();}
            finally {
                if(cli!=null)
                    cli.cleanup();
            }
            return RET;
    }


I can confirm that String[] paths contains one entry which is the absolute path to the pattern file in the SE directory and that datasetID is valid as are all the property values.
Not exactly sure what the 'settings' boolean does. API says 'If only getting a list of used files'.
From within my application, the info from ImportConfig is that it is using OMERO Version: 5.4.0-ice36-b74
and the output from config.getBioFormatsVersion() is:
BioFormats version: version: 5.7.1 revision: 4d44544dc5ab8824fd3d15e74211ae97aac7e3c0 date: 20 September 2017

Hope this helps.
Derek

[sbesson]

Yes, the version of BioFormats used in the commandLine and in imageJ (using Fiji) was the latest version.
6.0.0

Understood. This probably explains why your command-line tools and Fiji tests work. Specifically, I suspect this change included the fix for your DICOM use case.

Can I update my version of Omero (5.4.0) to use the BioFormats 6 package or would I have to update to the latest version of Omero itself? (Presuming that this is the problem)

At the moment, there is no production-grade relase of OMERO shipping Bio-Format 6 but upgrading is certainly on the roadmap of OMERO 5.5.0. If you are working against a test development server, it should be possible to upgrade an OMERO 5.4 server to use Bio-Formats 6. If you tell us more about your deployment, we can give some instructions.

Best,
Sebastien

[derekeh]

What, specifically, do you need to know about my deployment?

My current installation is for development purposes only at present, but I would hope to move that to production within a couple of months.
The installation was carried out as per instructions for the 5.4.0 release.
OS is Centos 7.

My application is JSF2.2 based constructed using Maven in Netbeans 8. (java 8)
It is deployed to Glassfish 5 and viewed on Nginx 1.12.2
The Maven build has dependencies: blitz.jar, glacier2.jar and ice.jar extracted from the OMERO.java-5.4.0-ice36-b74 download package.

Everything is running on the single linux machine. Omero, postgres, nginx, glassfish.
Hope that helps.
Let me know if there is anything else you need to know to enable me to upgrade this installation to BioFormats6.

Derek

[derekeh]

In the meantime I have created a workaround whereby I

copy the relevant SE directory to /tmp

point the CommandLineImporter to the /tmp/SE directory and load the sequence into the OMERO Dataset

delete /tmp/SE

Bit of extra file movement but that works
Derek