Advanced configuration // memory settings

[ehrenfeu]

Dear all,

the advanced configuration documentation about JVM memory states that this is most likely the part that should be adjusted on a production machine.

Do you have any hints about the values to use here? As we donated 32G of memory to our OMERO machine I definitely think we need to increase the default 512 MB

What else should we adjust from the default configuration?

• omero.db.poolsize?
• PostgreSQL settings?
• ...?

Thanks,
Niko

[jmoore]

Hi Niko,

2 GB is a good starting size for the Blitz-0 process, and for the PixelData-0 process if you are going to be working much with big images (3000x3000 or bigger). Much beyond 8GB for OMERO might cause issues with garbage collection, but it might be worth trying 16 nonetheless.

If you increase the number of max_connections in postgresql.conf then you can increase omero.db.poolsize to match, but leave yourself some extra handles just in case. 50 or a 100 in postgresql.con and then 40 or 80 in omero.db.poolsize would be sensible.

For that to work, you will need to make sure you have sufficient shared memory on your system. PostgreSQL will warn you if you don't. There are a number of other memory settings in postgresql.conf which you can tune, but best to read on PG websites about the best way forward there.

Hope that helps,
~Josh.

P.S. A non-memory related suggestion is to provide as many separate disks to PostgreSQL as you can. It can easily use three.

[ehrenfeu]

Hi Josh,

2 GB is a good starting size for the Blitz-0 process, and for the PixelData-0 process if you are going to be working much with big images (3000x3000 or bigger). Much beyond 8GB for OMERO might cause issues with garbage collection, but it might be worth trying 16 nonetheless.

ok, we'll give it a try. Currently we don't have many large images (at least not the single planes), it's more a matter of huge overall datasets (many z-slices, many positions, many timepoints).

How would we notice issues with the garbage collector? Slowing down something? Java crashes? ...?

Thanks
~Niko

[jmoore]

How would we notice issues with the garbage collector? Slowing down something? Java crashes? ...?

Tell-tale signs would include session timeout exceptions or possibly OutOfMemory exceptions under excessively high-load. If you suspect something like this is the case, you can configure GC logging in etc/grid/template.xml in the BlitzTemplate server-template element by adding some combination of:

Code: Select all

         <option>-verbose:gc</option>
         <option>-XX:+PrintGCTimeStamps</option>
         <option>-XX:+PrintGCDetails</option>


just about the <option>-Xmx512M</option> element. This will print to master.out any number of details:

Code: Select all

var/log $ tail -n 10 master.out
18.349: [GC 18.349: [ParNew: 19087K->2112K(19136K), 0.0165337 secs] 253332K->242921K(291232K), 0.0165864 secs] [Times: user=0.05 sys=0.01, real=0.01 secs]
18.366: [CMS-concurrent-preclean: 0.024/0.041 secs] [Times: user=0.10 sys=0.01, real=0.04 secs]
18.366: [GC[YG occupancy: 2454 K (19136 K)]18.366: [Rescan (parallel) , 0.0046657 secs]18.371: [weak refs processing, 0.0392330 secs] [1 CMS-remark: 240809K(272096K)] 243263K(291232K), 0.0440594 secs] [Times: user=0.06 sys=0.00, real=0.05 secs]
18.410: [CMS-concurrent-sweep-start]
18.538: [CMS-concurrent-sweep: 0.124/0.128 secs] [Times: user=0.25 sys=0.00, real=0.13 secs]
18.538: [CMS-concurrent-reset-start]
18.540: [CMS-concurrent-reset: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
18.645: [GC 18.646: [ParNew: 19136K->2111K(19136K), 0.0072078 secs] 189969K->173451K(303860K), 0.0072623 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
19.492: [GC 19.492: [ParNew: 19135K->1773K(19136K), 0.0100257 secs] 190475K->174624K(303860K), 0.0100993 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
20.734: [GC 20.734: [ParNew: 18797K->1664K(19136K), 0.0065431 secs] 191648K->174515K(303860K), 0.0066109 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]


Making this change requires restarting the server.

Cheers,
~Josh

[ehrenfeu]

Thanks again, Josh!