Page 1 of 1

Memoizer not working in MATLAB 2017b

PostPosted: Tue Sep 11, 2018 11:51 pm
by jmfrank
Hi there,

I've seen countless examples of initiating the memoizer reader in matlab. However, following even the simplest example here (https://docs.openmicroscopy.org/bio-for ... b-dev.html) does not work on my system. I'm wondering if it's my version of matlab / java/ or ome?

Basically, I never see the creation of the cached reader file and reader.isSavedToMemo always returns to false despite setting the minimum elapsed time to zero.

Maybe I'm missing something obvious?

I tested this on LSM files.

MATLAB 2017B
Ubuntu 16.04 LTS
Using the bfmatlab tools. bfUpgradeCheck returns 'up to date'.
MATLAB ver returns: Java Version: Java 1.8.0_121-b13

Re: Memoizer not working in MATLAB 2017b

PostPosted: Wed Sep 12, 2018 7:40 am
by sbesson
Hi,

I just tested the following minimal code on MATLAB R2017b / OSX with Bio-Formats 5.9.2 and confirmed a `.test.fake.bfmemo` file is created both from the debug logging and on disk

Code: Select all
[~,v] = bfCheckJavaPath()
bfInitLogging('DEBUG');
r = loci.formats.Memoizer(bfGetReader(), 0);
r.setId('test.fake');
r.close();


Could you try and run the same snippet and see if you get a similar positive result on your instance? If not, having the MATLAB version, the Java version as well as the output of the logging would certainly help us troubleshoot this issue further.

Best,
Sebastien

Re: Memoizer not working in MATLAB 2017b

PostPosted: Wed Sep 12, 2018 4:08 pm
by jmfrank
I still don't see a memofile in the current directory.

The only output from the lines you gave me is:

v =

'5.7.3'

Maybe there's something wrong with the logger? I do see inconsistent warnings about initializing the log4j properly. I think the warnings go away after one call to bioformats. The warnings look something like this:
log4j:WARN No appenders could be found for logger (loci.formats.ClassList).
log4j:WARN Please initialize the log4j system properly.

Are these warnings indicating something critical to the memoizer isn't working? Everything else in my bioformats works fine, so I had ignored these errors until now. Now that I think about it, I suppose the memoizer works using logging? I'm new to using Java so excuse my lack of understanding.

The output in MATLAB from calling ver is:
> ver

MATLAB Version: 9.3.0.713579 (R2017b)
MATLAB License Number: STUDENT
Operating System: Mac OS X Version: 10.13.5 Build: 17F77
Java Version: Java 1.8.0_121-b13 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
-----------------------------------------------------------------------------------------------------
MATLAB Version 9.3 (R2017b)
Box Counting Version 1.10
Computer Vision System Toolbox Version 8.0 (R2017b)
Control System Toolbox Version 10.3 (R2017b)
Curve Fitting Toolbox Version 3.5.6 (R2017b)
DSP System Toolbox Version 9.5 (R2017b)
GUI Layout Toolbox Version 2.3.2 (R2018a)
Image Processing Toolbox Version 10.1 (R2017b)
Instrument Control Toolbox Version 3.12 (R2017b)
Optimization Toolbox Version 8.0 (R2017b)
Parallel Computing Toolbox Version 6.11 (R2017b)
Signal Processing Toolbox Version 7.5 (R2017b)
Statistics and Machine Learning Toolbox Version 11.2 (R2017b)
Symbolic Math Toolbox Version 8.0 (R2017b)
Wavelet Toolbox Version 4.19 (R2017b)

Re: Memoizer not working in MATLAB 2017b

PostPosted: Wed Sep 12, 2018 4:42 pm
by jmfrank
I noticed my bioformats wasn't the current version (despite bfupgrade check telling me I was up to date...).

I downloaded 5.9.2 and now things are starting to work! I'm seeing what looks like proper debugging output. Not sure why it wasn't working before though...

I see a line that says "deleting invalid memo file: /Users/franklin/Documents/MATLAB/.test.fake.bfmemo". So maybe memoizer was previously working using that default directory. The new bioformats is again saving files to that directory. I thought the default directory was the same as where the file is located?

I switch between my mac and ubuntu for work and I access my image data from a server. Will memofiles work seamlessly between OSX and linux? I.e. can OSX read a memofile created by my ubuntu machine and vice-versa? If not, then I can see how it would make sense to have a local directory with all memofiles on each machine. If both machines can read memofiles, then how can I force the memofile to be saved in the image directory?

Here's the output the test :
>> [~,v] = bfCheckJavaPath()
bfInitLogging('DEBUG');
r = loci.formats.Memoizer(bfGetReader(), 0);
r.setId('test.fake');
r.close();

v =

'5.9.2'

Could not find loci.formats.in.SlideBook6Reader
Could not find loci.formats.in.ScreenReader
processor is INTEL_64 os.arch is x86_64
architecture is OSX_64 os.name is mac os x
architecture is OSX_64 os.name is mac os x
platform specific path is META-INF/lib/osx_64/
mappedLib is libturbojpeg.dylib
URL is jar:file:/Users/franklin/Dropbox/code_bank/matlab/bfmatlab/bioformats_package.jar!/META-INF/lib/osx_64/libturbojpeg.dylib
URL path is file:/Users/franklin/Dropbox/code_bank/matlab/bfmatlab/bioformats_package.jar!/META-INF/lib/osx_64/libturbojpeg.dylib
Extracting 'jar:file:/Users/franklin/Dropbox/code_bank/matlab/bfmatlab/bioformats_package.jar!/META-INF/lib/osx_64/libturbojpeg.dylib' to '/var/folders/jk/m4cr4wdj49nfqj_62czqct0c0000gn/T/libturbojpeg3582672045927558887.dylib'
deleting invalid memo file: /Users/franklin/Documents/MATLAB/.test.fake.bfmemo
Kryo Exception: java.lang.ArrayIndexOutOfBoundsException: -2
Serialization trace:
objectives (loci.formats.in.OIRReader)
readers (loci.formats.ImageReader)
reader (loci.formats.DimensionSwapper)
reader (loci.formats.FileStitcher)
helper (loci.formats.in.FilePatternReader)
readers (loci.formats.ImageReader)
reader (loci.formats.ChannelFiller)
reader (loci.formats.ChannelSeparator)
start[1536769476478] time[195] tag[loci.formats.Memoizer.loadMemo]
Loaded properties from: services.properties
Added interface interface loci.formats.services.POIService and implementation class loci.formats.services.POIServiceImpl
Added interface interface loci.formats.services.MDBService and implementation class loci.formats.services.MDBServiceImpl
Added interface interface loci.formats.services.JPEGTurboService and implementation class loci.formats.services.JPEGTurboServiceImpl
Added interface interface ome.codecs.services.LuraWaveService and implementation class ome.codecs.services.LuraWaveServiceImpl
Added interface interface loci.formats.services.JAIIIOService and implementation class loci.formats.services.JAIIIOServiceImpl
Added interface interface loci.formats.services.WlzService and implementation class loci.formats.services.WlzServiceImpl
Added interface interface loci.formats.services.JHDFService and implementation class loci.formats.services.JHDFServiceImpl
Added interface interface loci.formats.services.NetCDFService and implementation class loci.formats.services.NetCDFServiceImpl
Added interface interface loci.formats.services.EXIFService and implementation class loci.formats.services.EXIFServiceImpl
Added interface interface loci.formats.services.MetakitService and implementation class loci.formats.services.MetakitServiceImpl
Added interface interface loci.formats.services.LuraWaveService and implementation class loci.formats.services.LuraWaveServiceImpl
Added interface interface loci.formats.services.OMEXMLService and implementation class loci.formats.services.OMEXMLServiceImpl
Added interface interface ome.codecs.services.JAIIIOService and implementation class ome.codecs.services.JAIIIOServiceImpl
Added interface interface loci.formats.services.JPEGXRService and implementation class loci.formats.services.JPEGXRServiceImpl

java.io.FileNotFoundException: test.fake (No such file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:130)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165)
at loci.common.Location.getHandle(Location.java:416)
at loci.common.Location.getHandle(Location.java:372)
at loci.common.Location.getHandle(Location.java:353)
at loci.common.Location.getHandle(Location.java:336)
at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:125)
at loci.formats.FormatReader.isThisType(FormatReader.java:614)
at loci.formats.ImageReader.getReader(ImageReader.java:188)
at loci.formats.ImageReader.setId(ImageReader.java:839)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:690)

{ MANY REPEATS OF FileNotFoundException }

FakeReader initializing test.fake
FakeReader initializing test.fake
loci.formats.in.FakeReader.initFile(test.fake)
saved to temp file: /Users/franklin/Documents/MATLAB/.test.fake.bfmemo2637976983648812026
start[1536769476734] time[95] tag[loci.formats.Memoizer.saveMemo]
saved memo file: /Users/franklin/Documents/MATLAB/.test.fake.bfmemo (32485 bytes)
start[1536769476432] time[398] tag[loci.formats.Memoizer.setId]
Location.mapFile: embedded-stream.raw -> null
Location.mapFile: embedded-stream.raw -> null

Re: Memoizer not working in MATLAB 2017b

PostPosted: Thu Sep 13, 2018 12:19 pm
by sbesson
Hi

I downloaded 5.9.2 and now things are starting to work! I'm seeing what looks like proper debugging output. Not sure why it wasn't working before though...


Very glad to hear that. I am unclear on why it was not working before as I do not know any significant change would have happened in Memoizer API or the Bio-Formats MATLAB bindings. But I assume we can move forward with your working set up.

I see a line that says "deleting invalid memo file: /Users/franklin/Documents/MATLAB/.test.fake.bfmemo". So maybe memoizer was previously working using that default directory. The new bioformats is again saving files to that directory. I thought the default directory was the same as where the file is located?


The deleting line is expected as you upgraded to a new minor version of Bio-Formats and memo files were not backwards-compatible. Regarding the location, your expectation is absolutely correct. My example was using fake files which are special in the sense the files do not need to exist on disk. In that case, the current directory is used as the location to store memo files.

You should be able to confirm that for any file on disk, the directory should be used e.g.

Code: Select all
touch /tmp/test.fake


Calling

Code: Select all
r.setId('/tmp/test.fake');


should create a memo file under /tmp with the default constructor.

I switch between my mac and ubuntu for work and I access my image data from a server. Will memofiles work seamlessly between OSX and linux? I.e. can OSX read a memofile created by my ubuntu machine and vice-versa? If not, then I can see how it would make sense to have a local directory with all memofiles on each machine. If both machines can read memofiles, then how can I force the memofile to be saved in the image directory?


What you suggest might work but will require some testing and will be need to satisfy a few conditions:

  • the Bio-Formats version used on both OSX and Linux should be compatible in terms of reader serialization i.e. ideally same version or at least same minor version
  • the directories containing memo files are stored should be accessible and writeable by the user on both machines
  • for shared folders, you will want the mount point to be exactly the same as the memo file will sometimes cache filepath locations. Having different mount points could end up in a broken situation where the reader would be loaded from the memo file but the pixel data would be unavailable

Most of the restrictions above arise from the fact the Bio-Formats memoization API was largely driven by the OMERO 5 use case i.e. when a repository of cache files for a set of image filesets is fully maintained by a single version of Bio-Formats in a server in a central location.
Sharing memo directories between machines or even between users in a traditional filesystem group are concepts that have been discussed but never prototyped and properly designed so far.

Best,
Sebastien