Issue with Matlab parfor and Bioformats bfGetplane

[nhuebsch]

I saw someone else had a problem incorporating Bioformats functions with parfor loops in Matlab. Not sure if this is solveable but I hope so (I also posted on another part of the forum by mistake but I think this is the correct place). I am running a code that processes stacks two images at a time to look at movements. I can use parfor loops if I first load the entire stack of images into ram with bfGetplane. However, if I try to use bfGetplane within a parfor loop, I get this error:

Error using bfGetPlane (line 49)
The value of 'r' is invalid. It must satisfy the function: @(x)isa(x,'loci.formats.IFormatReader')&&~isempty(x.getCurrentFile()).

when the command is : " I = bfGetPlane(reader,i)" (i) = the "ith" plane.

Is there a solution to this issue, or can you recommend a work-around that doesn't involve keeping the entire image series in active RAM?

thanks.

[sbesson]

Hi,

your error seems to indicate an issue with the initialization of the reader within each parallel loop. Would it be possible for you to share the code or at least the functional block allowing to reproduce your issue?

As Bio-Formats readers are not thread safe, it is not possible to initialize a reader outside a parfor loop and pass it to each parallel worker. Instead, each worker should initialize a Bio-Formats reader, perform work on the image such as reading and processing a particular plane and close the reader. The Bio-Formats MATLAB documentation provides a minimal example of how to use Bio-Formats in a parfor loop.

Let us know if there is anything in this snippet that could be expanded to clarify the use cases

Best,
Sebastien

[nhuebsch]

Thank you Sebastian. I will try the instructed method. Here is the code fragment (below). I commented out the way I was doing before (before, I would call up the entire image stack outside parfor, and open individual pieces within).

For this code we are doing block matching optical flow, which is very computationally intensive, hence our need to take advantage of the parallel computing toolbox.

thanks,

Nate

>>>>>
imgPNumber = i;


I1 = stack(:,:,tzerostrain);
imgInit = double(ResizeImage(I1,imageSize, mbSize));

%Test if file is there:
% Can't currently do this test b/c this syntax doesn't work for
% BioFormats
% % if not(exist(reader))
% % error('GetMotionVectors:FileMissing','File missing');
% % end
%I2 = stack(:,:,imgPNumber);
dummy = bfGetPlane(reader,i);
I2 = double(dummy(:,:,1)); % convert from rgb to gray just in case
%
imgNext = double(ResizeImage(I2,imageSize, mbSize));

[sbesson]

Hi Nate,

using the parallel toolbox for block matching makes complete sense. I was able to reproduce your error with the following pseudo-code:

Code: Select all

bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
% Initialize a reader outside parfor loop
reader = bfGetReader(path);
parfor i =1:4
    % Throws an error
    I= bfGetPlane(reader, i);
    disp(mean(I(:)));
end
reader.close();


As mentioned previously, moving the initialization/closing of the reader steps within the parfor block should be sufficient to run the code in parallel:

Code: Select all

bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
parfor i =1:4
    % Initialize a reader, retrieve the plane and close the reader
    reader = bfGetReader(path);
    I = bfGetPlane(reader,i);
    disp(mean(I(:)));
    reader.close();
end


Initializing a reader for each parallel execution might become a bottleneck depending on the nature and/or size of the data you are working with. If this turns out to be the case, you might want to cache the initialized reader before running the parallel computation using the Memoizer class as described in the documentation example.

Best,
Sebastien

[nhuebsch]

Thank you Sebastian. Doing the memoization as you suggest works pretty well.

Do you know offhand if the memoization can be done prior to a function call (e.g. so that the function passes the reader object after memoization)? Or must the memoization be done within the same function?

thank you.

[sbesson]

Hi Nate,

as long as there is no serialization within your function itself (like an internal parfor loop), you should be able to pass the initialized reader to your subfunction as follows:

Code: Select all

bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
parfor i =1:4
    % Initialize a reader, retrieve the plane and close the reader
    reader = bfGetReader(path);
    runMyAnalysis(reader, i);
    reader.close();
end


Best,
Sebastien