java.lang.OutOfMemoryError: unable to create new native thre

[jwarren]

Thanks Josh - I've got our memory changed to 8G instead of 4G as we were running the python client script, db and omero server on the same machine. I'll get back to you if this doesn't fix things.

[jmoore]

Hi Jonathan,

looking at your `ps` output, there's just the one python process for your upload_to_omero2.py work, the rest all belong to the server, so that doesn't seem to be an issue. The individual memory configurations for OMERO that are being used in that output (512M, 256M, and 256M) aren't sufficient for production, though.

Starting with the 5.0.3 release, there's a new configuration, https://www.openmicroscopy.org/site/support/omero5/sysadmins/server-performance.html#jvm-memory-settings, that you might want to take a look at. A recent thread on the forums also showed examples.

All the best.
~Josh.

[jwarren]

Hi Josh

Many thanks for the help. I can confirm that the performance of omero and this issue is much better since upgrading to .04 and increasing the memory available to omero using the new config options.

However we do still get this error although less frequently.

So before I start monitoring and fine tuning- what is the recommended memory for an omero installation? I'm guessing 8G for the whole machine is too little still? I guess this maybe like saying "how long is a piece of string" - but what do people generally end up using? We will have hundreds of thousands of 2D images and soon we will also have large 3D image sets as well in this mix.

[kennethgillen]

So before I start monitoring and fine tuning- what is the recommended memory for an omero installation? I'm guessing 8G for the whole machine is too little still? I guess this maybe like saying "how long is a piece of string"

Hi,

Memory requirements are covered in our docs in the System Requirements [1] and Performance and monitoring [2] sections. If we can improve these - please let us know!

For what it's worth, for a 16TB OMERO server at the College of Life Science at the University of Dundee, I have OMERO configured to use 17GB of system RAM, with an 8.5GB Blitz, 3.4G Indexer, 3.4G PixelData using the following OMERO config:

Code: Select all

omero.jvmcfg.percent.blitz=50
omero.jvmcfg.percent.indexer=20
omero.jvmcfg.percent.pixeldata=20
omero.jvmcfg.system_memory=17000

what do people generally end up using? We will have hundreds of thousands of 2D images and soon we will also have large 3D image sets as well in this mix.

It certainly would be interesting to hear what others in the community are using.

[1] https://www.openmicroscopy.org/site/sup ... ments.html
[2] https://www.openmicroscopy.org/site/sup ... mance.html

Best,
Kenny

[jwarren]

Thanks Kenny - I'll look into those docs.

I also have some processor output from our systems guys before they rebooted our machine while omero was crashed.

# ps -eLf|grep mi_adm|wc -l
1026

most of them are my upload_to_omero2 python script which seems to be running 658 processes - this seems a bit excessive? Is there a way to limit these processes started? As I understand it the script relies on omero to hand me back a session and not start up new ones (script previously posted on this thread) :

# ps -eLf|grep mi_adm|grep
/net/isilonP/public/rw/homes/mi_adm/python/upload_to_omero2.py|wc -l
658
# ps -eLf|grep mi_adm|grep icegridnode|wc -l
23
# ps -eLf|grep mi_adm|grep "python manage.py runfcgi"|wc -l
174

[jmoore]

most of them are my upload_to_omero2 python script which seems to be running 658 processes - this seems a bit excessive? Is there a way to limit these processes started? As I understand it the script relies on omero to hand me back a session and not start up new ones (script previously posted on this thread)

Note: your ps command shows 658 threads rather than processes. That's not nearly as bad. Ice definitely creates a number of threads even if you do not. You can certainly limit these if you'd like to try that, though going too low (approx. < 5) could lead to deadlocks.

See https://github.com/openmicroscopy/openmicroscopy/blob/develop/etc/ice.config#L126 for more information.

Cheers,
~Josh.

[jwarren]

Thanks for the reply Josh and Kenny.

I've upgraded to using 12G (configured using the percentage strategy) in total to see how that goes as I have inferred from your replies that this or more is pretty standard.

[jwarren]

solved this easily in the end by sitting down and looking at what was going on.
Global CLI and connection variables used for all the uploads using the same session fixed it.

My suggestion for documentation would be a clear simple python example of how to upload images as one user? Then maybe a more complicated one with multiple users that you have? This also should be more prominent in the docs as we had some collaborators who claimed the reason they didn't use omero is because they couldn't work out how to load images into it.

As Im now uploading 78,000 images I would like to turn off the output from the upload process to be one line per file at the moment it looks like this:

Imported pixels:
Other imported objects:
Fileset:9673
Image:9673
Using session b5a5abc4-327f-4b13-9d84-0b0532e1480c (root@ves-ebi-cf:4064). Idle timeout: 10.0 min. Current group: public_group
2014-11-14 16:52:16,560 188 [ main] INFO ome.formats.importer.ImportConfig - OMERO Version: 5.0.2-ice35-b26
2014-11-14 16:52:16,573 201 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 5.0.2-ice35-b26 revision: 0c4215a date: 27 May 2014
2014-11-14 16:52:27,072 10700 [ main] ERROR ome.system.UpgradeCheck - Error reading from url: http://upgrade.openmicroscopy.org.uk?ve ... ystems+Inc. "connect timed out"
9674


I tried many variations of the debug param including having it not set but to no avail this i thought would work:

import_args.extend(["-d", str(dsId),"--debug", "ERROR"])

How do i get rid of the UpgradeCheck connection error? I tried setting it to empty on the server but that didn't work?
bin/omero config set omero.upgrades.url

Any tips welcome - thanks!

[jmoore]

Hi Jonathan,

solved this easily in the end by sitting down and looking at what was going on.
Global CLI and connection variables used for all the uploads using the same session fixed it.

Does that mean that that frequent CLI connections were causing a problem? Can you point us to an example of what was required if you consider it a workaround?

My suggestion for documentation would be a clear simple python example of how to upload images as one user? Then maybe a more complicated one with multiple users that you have? This also should be more prominent in the docs as we had some collaborators who claimed the reason they didn't use omero is because they couldn't work out how to load images into it.

Duly noted (and agreed). If you have anything you consider to be of general use, though, we can likely find a place to put it.

As Im now uploading 78,000 images I would like to turn off the output from the upload process to be one line per file at the moment it looks like this:

By modifying the root logger from "warn" to "error" in etc/logback.xml, you can remove essentially all logging:

Code: Select all

  <root level="error">
    <appender-ref ref="stderr"/>
  </root>


That will still leave more than just the summary. Everything that goes to stdout or stderr directly without passing through the logback system, will be printed. At the moment, there's no flag to modify this.

How do i get rid of the UpgradeCheck connection error? I tried setting it to empty on the server but that didn't work?
bin/omero config set omero.upgrades.url

To simply remove the logging, put this in your etc/logback.xml file:

Code: Select all

  <logger name="ome.system.UpgradeCheck" level="OFF" additivity="false"/>


That doesn't fix the overarching problem though. We'll look further into what's happening there.

Cheers,
~Josh