bfmatlab - bfGetReader fails for large files

[akatsov]

Hello

I'm trying out bfmatlab but cannot initialize a reader for large image files (100+GB).

Specifically, Matlab hangs or throws a java heap exception when I call bfGetReader. The problem happens at line 89: r.setId(id);

Is there an alternative way to initialize a reader? Can bfmatlab handle large files, and if so what's the right way to read them?

Thanks,

Alex

[sbesson]

Hi Alex,

in general, bfmatlab should be able to cope with large image files without throwing Java heap exceptions.
Some information is necessary for us to help you with your issue:
- which version of Bio-Formats are you using?

Code: Select all

>> char(loci.formats.FormatTools.VERSION)


- which file format are you opening?
- how much Java memory is there available on MATLAB?

Code: Select all

java.lang.Runtime.getRuntime().maxMemory/(1024*1024)


Best,
Sebastien

[akatsov]

Hi Sebastien
Thanks for the quick reply. Here's more info:
1. ver: 5.0.3
2. file format: PCORAW. No problem reading files up to 50 GB, but problems with bigger files.
3. 500 MB java heap. Does initialization require space only for metadata, or more?
Thanks,
Alex

[sbesson]

Hi Alex,

for very large files (~100GB), could you try increase your heap size to a larger value e.g. 1024MB and see if this fixes the error?

Best
Sebastien

[akatsov]

Yes -- that fixed it. Thank you. Is there any other method to read large files that's less memory intensive?

Thanks again,

Alex

[sbesson]

Hi Alex,

no, to be able to read files with Bio-Formats, initializing the reader with `setId()` is mandatory. For many formats, including TIFF for instance, the memory requirements are expected to scale linearly with the size of the file so upping the JVM heap size to ~1G is expected for files of this order of magnitude.

Best,
Sebastien