96 well plate upload - C01 cellomics file format

[svcostes]

Hi everyone,

I have just tested Arrayscan High-Content Analysis microscope from Thermo Scientific (formerly known as cellomics) and I acquired ~2000 images from a 96 well plate. This is ~2GB of images. The images have all the same initial name, and what changes at the end are a few strings identifying the well and the image # for that well. All the images are saved under the same directory. I tried to upload these images via the import GUI of OMERO client. It takes about 1 hour per image to upload... After running it over night, it only loaded 17 images, and created a bunch of orphaned images (thousands) and I cannot access anything. If trying to upload the images from separate directories (one image per directory), then I can upload the image in 6 seconds and it shows up in the screen tab. It seems that OMERO is doing a TOC for each image, as it reads each individual image (my interpretation). I tried command line and same problem...

Except writing a python script to read each image individually, anyone else got this problem and figured a solution?

Thx,

Sylvain

[rleigh]

Dear Sylvain,

Which directories and/or files did you select to do the import? What happens if you just import one file from the screen? Do the rest then get automatically imported? If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.

If you are able to zip up and upload the screen privately then we can take a look at the import for you and improve any bugs or performance problems you are seeing. You can do this here: http://qa.openmicroscopy.org.uk/qa/upload/. However, if you have any trouble uploading over http due to the size of the data involved, we can provide you with details for uploading over ssh/sftp.

Kind regards,
Roger

[svcostes]

Dear Roger,

I tried importing only one file and you are right, part of the problem is that each file upload lets OMERO look into the directory and see all the other file with similar names. Thus, it gets stuck. If you instead load only one file, OMERO also upload all the other files automatically. However, results are very messed up:
1. The original image has a suffix for each channel that looks like: d0.CO1, d1.CO1, d2.CO1 (where d0 is image for channel 0, d1 is image for channel 1, etc...). However the upload ends up creating 3 images with the same name d0.C01. So, once loaded, one cannot differentiate which image is which channel until the image is open and the stain is recognized.
2. All the images ended up being sent to the orphan directory. I then had to copy them into an experiment directory.
3. If I load one image separately by first putting it into a separate directory alone, then OMERO loads the image very quickly and shows where that image is located into a 96 well plate format under the screen tab. So, metadata are completely accurate this way. However, this only works at the single image level.
4. As I tried many times to upload all the images before, the system loaded tens of thousands of images in the orphaned directory. I had to write a script to delete them as OMERO would crash while attempting to do it via the GUI. There are however left overs information I cannot get rid of anymore in OMERO: i.e. screen tabs has a bunch of 96 well plate experiments related to the one experiment I tried to upload many times. These docs are empty and but cannot be deleted... The image tab has also some problems: there are tens of thousand images being listed but I cannot delete them either. While trying to delete, it crashes and I sent the crash log earlier last week.

I am going to start uploading the data for you to look at it.

thx,

Sylvain

[rleigh]

Dear Sylvain,

Thanks for uploading the data. Sorry for the delay in replying--it took a while to test importing and deleting all the data.

I can certainly reproduce the problems you reported here, and I've opened a ticket (#11148) to track this, and you'll be notified of any changes to the ticket. Just a few clarifications needed:

If d0/d1/d2 are the channels, what are o1/o2? Analysis overlays, overview scan, or something else? I think the main problem here is that the CellomicsReader isn't handling these properly.

Regarding the points in your previous post:
All the images ending up in the orphaned images folder are images which we didn't handle importing properly. I agree that the naming here--prefixing with the image selected for import rather than the screen name--is not ideal, but the real problem here is that they are orphaned in the first place! This is something we can look at fixing.

Regarding the deletion of the orphaned images, I'll open a separate ticket about that--the speed is awfully slow and I've also seen odd failures here. I found that moving all the images into a dataset and then deleting the dataset was orders of magnitude faster--I'm just investigating this in more detail.

Regards,
Roger

[rahulmekala]

Dear Sylvain,

Which directories and/or files did you select to do the import? What happens if you just import one file from the screen? Do the rest then get automatically imported? If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.

If you are able to zip up and upload the screen privately then we can take a look at the import for you and improve any bugs or performance problems you are seeing. You can do this here: http://qa.openmicroscopy.org.uk/qa/upload/. However, if you have any trouble uploading over http due to the size of the data involved, we can provide you with details for uploading over ssh/sftp.

Kind regards,
Roger

i am facing with the same problem where i am unable to upload more than 2000 images through import.CLI,it does not work even a single image also, please provide sufficient information to me..
thanks

[mtbc]

Dear Rahul,

Let's get you importing a single image before we work on thousands. This thread is about a problem with Cellomics (*.C01 files) and recognizing their channels (original image or analysis results) properly; we fixed it back in 2013. So, I am guessing that your current problem may be something else.

Cheers,
Mark

[mtbc]

Dear Rahul,

As you mention elsewhere you are using OMERO 5.0: I can confirm that the fix for the problem on this thread was included in Bio-Formats/OMERO 5.0.0.

Cheers,
Mark

[rahulmekala]

Hi all i am getting the error while uploading the cellomics dataset into omero database it is uploading only one image instead of the rest then get automatically imported when selected one image file
If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.




but it does not happens in my case.
i have decided into three types folder d0,d1,d2 and uploading one folder it uploaded as individual images . each file has the sepate view of having field#1 onlyy.....


please help me

[sbesson]

Hi,

the last post is likely related to the same issue as the one describe in this thread. So let's stick to one thread to minimize confusion.

Reading it, one question occurs to me: can you confirm your Cellomics filesets organized into separate folders with one folder per field? Looking at the CellomicsReader, I think this this data layout is not currently supported as the Cellomics reader expects all files to be part of the same directory.

Best,
Sebastien