BIoFormatsCache size

[ratamero]

Dear all,

at the moment we're running our OMERO server on a machine with very limited storage, which means we're constantly getting Nagios warnings about running out of space - the main offender is always BioFormatsCache, which means every warning involves me having to ssh into the server and delete it. So two questions:

1) Is it possible to set the maximum size of BioFormatsCache?
2) If not, any idea how much storage I should provision for that?

Thanks in advance!

Erick

[sbesson]

Hi Erick,

The BioFormatsCache directory contains the cached readers for all the filesets imported in the ManagedRepository. If these cache files do not exist or are invalid e.g. following a server upgrade, they will be recreated on-the-fly when the server tries to access the raw data.

In terms of the size, it is hard to give precise numbers. In general, the expected disk usage will depend on two elements:
- the number of filesets imported in your database
- the nature of the file formats as certain readers cache much more metadata than others

If you want to maintain your current solution of deleting the cache directory, an option might be to adjust omero.pixeldata.memoizer_wait to a very large value. This should effectively disable the creation of the Bio-Formats cache files.

Best,
Sebastien

[kennethgillen]

Hi Erick,

I'm not the expert here, but removing the Bio-Formats caches will have a performance impact for your end-users - as Sebastien mentioned, the re-generation of the files on-the-fly, so it might be worth considering a server migration to a system with more local disk.

Best regards,

Kenny

[ratamero]

For the moment I'll probably settle for a middle-of-the-road solution: setting memoizer_wait to something vaguely sensible so that large files get cached, but small ones don't (our lack of space issue right now comes from TBs of data being imported in the form of many many very small TIFFs). Hopefully it won't impact performance too much and it should save a lot of space. In the meantime, it's time to go back to IT Services to beg for more storage, I guess. Thanks!

[ratamero]

Update: we managed to point the BioFormatsCache folder to our external storage server, where all the data sits. As long as cache doesn't hit 200TB we should be fine. Thanks again!