Slow Extracting of Series from Large LIF Files

[Cammyron]

Hi All,

I have some large LIF files (100gb and up) that contain 100-1000 series in them of 100MB each or so. I have a bit of code written for ImageJ that extracts each in turn and saves it. The only issue is it is very very slow. It takes about 2 minutes to open and save each series. The main delay seems to be BioFormats analysing the LIF file each time (i assume building the index of the series in it) before it can open one to save it. Is there a away around this?

Thanks

The code is here

Code: Select all

setBatchMode(true);
fs=File.separator;

pathFile=File.openDialog("Select LIF File to Process");
savePath=getDirectory("Select/Create Output Directory");


run("Bio-Formats Macro Extensions");
Ext.setId(pathFile);
Ext.getSeriesCount(seriesCount);

for(l=43;l<seriesCount;l++){
   Ext.setSeries(l);
   run("Bio-Formats Importer", "open=["+pathFile+"] autoscale color_mode=Default view=Hyperstack stack_order=XYCZT series_"+(l));
   nameStore=getTitle();
   getDateAndTime(year, month, dayOfWeek, dayOfMonth, hour, minute, second, msec);
   print(hour+":"+minute+":"+second+" - Processing Series "+l+" of "+seriesCount);
   //imageName=getInfo("image.filename");
   currentCount=l;
   currentCount=d2s(currentCount,0);
   currentLength=lengthOf(currentCount);
   if(currentLength==1){
      saveName="000"+currentCount;
   }
   if(currentLength==2){
      saveName="00"+currentCount;
   }
   if(currentLength==3){
      saveName="0"+currentCount;
   }
   if(currentLength==4){
      saveName=currentCount;
   }
   
   saveAs("Tiff", savePath+saveName+".tif");
   
   run("Close All");
   
   wait(500);
   call("java.lang.System.gc");
   wait(500);
}
print("DONE!");

[dgault]

Hi,

One change which may help to improve the performance is to remove the need to run the importer for each series. When using the macro extensions, the call to Ext.setId(pathFile) is enough to initialise the file. Once this is done you can loop through the series count and open the ImagePlus with Ext.openImagePlus.

From the macro you provided this would simply mean replacing:

Code: Select all

run("Bio-Formats Importer", "open=["+pathFile+"] autoscale color_mode=Default view=Hyperstack stack_order=XYCZT series_"+(l));

with:

Code: Select all

Ext.openImagePlus(pathFile);

This should hopefully reduce the long processing times which you are seeing.

David Gault

[Cammyron]

Hi David,

Thanks for the suggestion. Unfortunately hasn't made any difference. It still take 2 min a series to extract and is still analysising the lif file on each loop.

Cheers

Cam

[dgault]

Hi Cam,

Using macros the other simple approach would be to open a range of series at a time depending on what performance can be handled.

The other alternative would be to create a simple plugin for the operation which would be able to make use of the caching Bio-Formats uses, this would cache the file after the initial initialisation and speed up and future access. Unfortunately this is not currently made use of by the macro extensions but that is something we will look at adding in the future. I have begun putting together a sample of such a plugin which can be found at the link below:
https://github.com/dgault/example-legac ... orter.java

David Gault