bfopen; An unexpected network error occurred

[Kouichi_C_Nakamura]

Although I don't know how to reproduce this, but while I'm using `bfopen.m` of Bio-Formats MATLAB toolbox, I often (but not always) encounter the error below. This is apparently not due to MATLAB code, because the same code works well usually and only after running the code for some time I encounter this. A workaround for this is to restart MATLAB and it always works so far.

But if you could tell what's causing this, and stop this happening, it would be better.

Thanks in advance.

Best,
Kouichi

Code: Select all

Java exception occurred:
java.io.IOException: An unexpected network error occurred
   at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
   at sun.nio.ch.FileDispatcherImpl.pread(Unknown Source)
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
   at sun.nio.ch.IOUtil.read(Unknown Source)
   at sun.nio.ch.FileChannelImpl.readInternal(Unknown Source)
   at sun.nio.ch.FileChannelImpl.read(Unknown Source)
   at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:127)
   at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:112)
   at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:629)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:133)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165)
   at loci.common.Location.getHandle(Location.java:416)
   at loci.common.Location.getHandle(Location.java:372)
   at loci.common.Location.getHandle(Location.java:353)
   at loci.common.Location.getHandle(Location.java:336)
   at loci.common.Location.checkValidId(Location.java:441)
   at loci.formats.ImageReader.getReader(ImageReader.java:177)
   at loci.formats.ImageReader.setId(ImageReader.java:839)
   at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
   at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
   at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
   at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)



[dgault]

If this only occurs after running for a period of time then I suspect that this is caused by a memory issue. It might be worth trying to free up memory from variables after they have been used, using clearvars (https://www.mathworks.com/help/matlab/r ... rvars.html) would be one approach.

Are you running as part of one long continuous program or is multiple runs before the issue occurs?

[Kouichi_C_Nakamura]

Thank you. Because it says network error, I could not associate it with the memory issue. I can have good control of memory in MATLAB workspace using `clear` or `clearvars`. But could it be related to memory usage in Java or anything else that I can't see in MATLAB workspace?

I found an article about "Why does ImageJ not release any memory back to the system?" It seems that the issue is more about Java rather than ImageJ, so I suspect that Bio-Formats can be suffering from the same issue.

http://imagej.net/Troubleshooting#OutOfMemoryError

In that case, `clear` or `clearvars` in MATLAB side wouldn't help. Something like below might?

https://stackoverflow.com/questions/324 ... e-collecti

So far I'm not quite sure, but I think it's more about multiple runs.

[dgault]

It is possible that the issue may also be an error with being able to access and read the file. Is it possible that another program or running code might be locking the file while Bio-Formats is attempting to read it?

Do you have a sample MATLAB program which I could use to try and reproduce the issue? Also are the files you are reading being stored locally on the same machine or are you accessing them remotely?

[Kouichi_C_Nakamura]

File accessibility .... mmm, I can't think of anything particular, though I can't completely deny the possibility. I feel like it doesn't really fit the fact that relaunching MATLAB is always the solution.

I'd love to share the data and code, but at the moment I don't have a good way to reproduce this issue. Once I find one, I'll let you know.

[Kouichi_C_Nakamura]

I've just run the following code to open 3.7 GB TIFF file (14510 * 32510 * 4 pixels, 16 bit), and after around 10 cycles, MATLAB crashed. I could not confirm whether this was the same error as above, but it could.

I had set Java Heap Memory size to 25 GB in advance.

Code: Select all

for i = 1:20

    disp(i)
    b = bfopen(file);

end


If you can get `copytoImagePlus.m` https://github.com/kouichi-c-nakamura/copytoImagePlus and have `ImageJ-MATLAB` working, then you can create the same size of data by the following:

Code: Select all

I = zeros(14510 ,32510,4,'uint16');
imp = copytoImagePlus(I);
ij.IJ.saveAsTiff(imp,file)


I can send you data if you like.

[dgault]

I can confirm that using the same loop that I have also been able to reproduce the MATLAB crash, though as you mentioned I have not yet been able to verify if this is the same issue as the original error. I will continue testing and debugging this further to see if I can find the root cause of these issues.

[Kouichi_C_Nakamura]

Thanks a lot for the effort.

This may not be an issue of Bio-Formats. Since I've been recently experiencing bizarre behavior of ImageJ Java objects when manipulated from MATLAB. https://uk.mathworks.com/matlabcentral/ ... oning-code

During 35 `for` loop iterations using Bio-Formats and ImageJ from MATLAB, I experienced some 10~15 crashes of MATLAB.

[dgault]

Thanks for the update, this is certainly something we will keep an eye on and continue investigating.