java.lang.OutOfMemoryError: unable to create new native thre

[jwarren]

Hi

Is this a common error below when importing data via python? I get this pretty often. I set the heap size to 1g as per some documentation although it doesn't show what it was set to prior to me setting it?

Any top tips?

2014-09-04 11:05:38,193 11265 [ main] ERROR formats.importer.cli.CommandLineImporter - Error during import process.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[na:1.6.0_32]
at java.lang.Thread.start(Thread.java:679) ~[na:1.6.0_32]
at IceInternal.Timer.<init>(Timer.java:128) ~[ice.jar:na]
at IceInternal.Instance.finishSetup(Instance.java:904) ~[ice.jar:na]
at Ice.CommunicatorI.finishSetup(CommunicatorI.java:299) ~[ice.jar:na]
at Ice.Util.initialize(Util.java:212) ~[ice.jar:na]
at omero.client.init(client.java:385) ~[blitz.jar:na]
at omero.client.<init>(client.java:290) ~[blitz.jar:na]
at omero.client.createClient(client.java:472) ~[blitz.jar:na]
at ome.formats.OMEROMetadataStoreClient.unsecure(OMEROMetadataStoreClient.java:712) ~[blitz.jar:na]
at ome.formats.OMEROMetadataStoreClient.initialize(OMEROMetadataStoreClient.java:677) ~[blitz.jar:na]
at ome.formats.importer.ImportConfig.createStore(ImportConfig.java:303) ~[blitz.jar:na]
at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:125) ~[blitz.jar:na]
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:719) ~[blitz.jar:na]

Thanks

Jonathan.

[kennethgillen]

Hi,

I'd be inclined to try increasing your Blitz process heap size to much larger than 1GB if you have the system memory available. I maintain an 8GB Blitz myself for a college server, if that's any use.

It's also worth noting that with OMERO 5.0.3 we've simplified process memory configuration: see https://www.openmicroscopy.org/site/sup ... mance.html.

Others on the team and in the community may well have other advice, especially those using python imports more often.

Let us know how you get on!

All the best,

Kenny

[jmoore]

Hi Jonathan,

what Kenny suggests is certainly the best practice when the out of memory is being thrown server-side, but you're running into on the client. This has been seen before, e.g. http://trac.openmicroscopy.org.uk/ome/ticket/8447.

Can you share the Python code that you are using to invoke the CLI?

Cheers,
~Josh.

[jwarren]

Hi Josh

The code is pretty much copied from https://github.com/ome/omego/blob/master/auto_import.py I also posted the code recently as when I was running the code on my mac it gave a mac specific error viewtopic.php?f=6&t=7577
code snippet:

Code: Select all

print "printing user info"+omeroUser, group, filename
    #global cli
    print "-"*100
    #if cli is None:
        #print "cli is none!!!!!"
    cli = omero.cli.CLI()
    cli.loadplugins()
    cli.invoke(["login", omeroUser+'@'+omeroHost, "-w", passw, "-C"], strict=True)
    #cli.invoke(["login", "%s@localhost" % user, "-w", passw, "-C"], strict=True)
    cli.invoke(["sessions", "group", group], strict=True)
    import_args = ["import"]
    if dataset is not None:
        dsId = create_containers(cli, dataset,omeroHost, project)
        print "dsId="+str(dsId)
        import_args.extend(["-d", str(dsId),"--debug", "DEBUG"])#,"--debug", "ALL"])
    else:
        print "dataset is None!!!!!!!!!!!!!!!!!!!!"
    import_args.append(filename)
    print " import args="
    print import_args
    cli.invoke(import_args, strict=True)


[jwarren]

Full code here:

Code: Select all

#!/usr/bin/python

import os
import requests
import json
from xml.dom.minidom import parseString
import glob
import shutil
import sys
import os.path
import sys, getopt
import mysql.connector
from mysql.connector import errorcode
import omero.all
import omero.rtypes
import omero
import omero.cli
from omero.gateway import BlitzGateway
from omero.rtypes import wrap
from omero.model import DatasetI, ProjectI
from common import splitString

responseFailed=0
numberOfImageDownloadAttemps=0
totalNumberOfImagesWeHave=0
numFoundInSolr=0
cli=None



uniqueUris=set()
def main(argv):
    solrUrl="""http://wwwdev.ebi.ac.uk/mi/impc/dev/solr/experiment/select?q=observation_type:image_record&fq=(download_file_path:*mousephenotype.org*%20AND%20!download_file_path:*.pdf)&fl=id,download_file_path,phenotyping_center,pipeline_stable_id,procedure_stable_id,datasource_name,parameter_stable_id&wt=json&indent=on&rows=10000000"""
    #below url for testing one paremeter only
    #solrUrl="""http://wwwdev.ebi.ac.uk/mi/impc/dev/solr/experiment/select?q=observation_type:image_record&fq=(download_file_path:*mousephenotype.org*%20AND%20parameter_stable_id:IMPC_XRY_034_001)&fl=id,download_file_path,phenotyping_center,pipeline_stable_id,procedure_stable_id,datasource_name,parameter_stable_id&wt=json&indent=on&rows=1000000"""
    #note cant split this url over a few lines as puts in newlines into url which doesn't work
    rootDestinationDir='/nfs/komp2/web/images/impc'#'/Users/jwarren/images/impc'#'/nfs/komp2/web/images/impc/images'
    omeroHost = ''
    omeroPort =
    omeroUser = ''
    omeroPass = '
    group='public_group'
    print("running python upload to omero for impc images")


    try:
          opts, args = getopt.getopt(argv,"d:s:h:p:u::db:pass:ohost:oport:ouser:opass",[])
    except getopt.GetoptError:
        print 'upload_to_omero2.py -d <rootDestinationDir> -s <solrUrl> -h <komp2host> -p <komp2Port> -db <komp2db> -u <komp2User> -pass <komp2Pass> -ohost <omerohost> -oport <omero port> -ouser <omero user> -opass <omero pass>'
        sys.exit(2)
    for opt, arg in opts:
        print 'upload_to_omero2.py -d <rootDestinationDir> -s <solrUrl> -h <komp2host> -p <komp2Port> -db <komp2db> -u <komp2User> -pass <komp2Pass> -ohost <omerohost> -oport <omero port> -ouser <omero user> -opass <omero pass>'
        if opt in ("-d", "--rootDestinationDir"):
            rootDestinationDir = arg
        elif opt in ("-s", "--solrUrl"):
             solrUrl = arg
    print 'rootDestinationDir is "', rootDestinationDir
    print 'solrUrl', solrUrl
    deleteImagesAlreadyInOmero(omeroHost, omeroPort, omeroUser, omeroPass)
    alreadyInOmero=getImagesAlreadyInOmero(omeroHost, omeroPort, omeroUser, omeroPass, group)
    print 'alreadyInOmero size=', len(alreadyInOmero)
    runWithSolrAsDataSource(solrUrl, rootDestinationDir, alreadyInOmero, omeroUser, omeroPass, omeroHost, omeroPort, group)
   

def getImagesAlreadyInOmero( omeroHost, omeroPort, omeroUsername, omeroPass, group ):
    alreadyInOmero=set()
   
    from omero.gateway import BlitzGateway


    # Connect to the Python Blitz Gateway
    # =============================================================
    # Make a simple connection to OMERO, printing details of the
    # connection. See OmeroPy/Gateway for more info
    conn = BlitzGateway(omeroUsername, omeroPass, host=omeroHost, port=omeroPort, group=group)
    connected = conn.connect()


    # Check if you are connected.
    # =============================================================
    if not connected:
        import sys
        sys.stderr.write("Error: Connection not available, please check your user name and password.\n")
        sys.exit(1)


    # Using secure connection.
    # =============================================================
    # By default, once we have logged in, data transfer is not encrypted (faster)
    # To use a secure connection, call setSecure(True):

    # conn.setSecure(True)         # <--------- Uncomment this


    # Current session details
    # =============================================================
    # By default, you will have logged into your 'current' group in OMERO. This
    # can be changed by switching group in the OMERO.insight or OMERO.web clients.

    user = conn.getUser()
    print "Current user:"
    print "   ID:", user.getId()
    print "   Username:", user.getName()
    print "   Full Name:", user.getFullName()

    print "Member of:"
    for g in conn.getGroupsMemberOf():
        print "   ID:", g.getId(), " Name:", g.getName()
    group = conn.getGroupFromContext()
    print "Current group: ", group.getName()

    print "Other Members of current group:"
    for exp in conn.listColleagues():
        print "   ID:", exp.getId(), exp.getOmeName(), " Name:", exp.getFullName()

    print "Owner of:"
    for g in conn.listOwnedGroups():
        print "   ID:", g.getName(), " Name:", g.getId()

    # New in OMERO 5
    print "Admins:"
    for exp in conn.getAdministrators():
        print "   ID:", exp.getId(), exp.getOmeName(), " Name:", exp.getFullName()

    # The 'context' of our current session
    ctx = conn.getEventContext()
    # print ctx     # for more info
    print ctx
    # The only_owned=True parameter limits the Projects which are returned.
    # If the parameter is omitted or the value is False, then all Projects
    # visible in the current group are returned.
    print "\nList Projects:"
    print "=" * 50
    my_expId = conn.getUser().getId()
    for project in conn.listProjects(my_expId):
        print_obj(project)
        for dataset in project.listChildren():
            print_obj(dataset, 2)
            for image in dataset.listChildren():
                #print_obj(image, 4)
                fileset = image.getFileset()
                print 'fileset=', fileset
                if fileset is not None:
                    print 'image id=', image.getId()
                    filesetId=fileset.getId()
                    query = 'SELECT clientPath FROM FilesetEntry WHERE fileset.id = :id'
                    params = omero.sys.ParametersI()
                    params.addId(omero.rtypes.rlong(filesetId))
                    for path in conn.getQueryService().projection(query, params):
                        fullPath=path[0].val
                        if splitString in fullPath:
                            print 'path in omero=' +fullPath
                            relativeOmeroUrl=fullPath.split(splitString,1)[1]
                            print "adding to hash relative omero url="+relativeOmeroUrl
                            alreadyInOmero.add(relativeOmeroUrl)
    # Close connection:
    # =================================================================
    # When you are done, close the session to free up server resources.
    conn._closeSession()
    return alreadyInOmero

def print_obj(obj, indent=0):
    """
    Helper method to display info about OMERO objects.
    Not all objects will have a "name" or owner field.
    """
    print """%s%s:%s  Name:"%s" (owner=%s)""" % (\
            " " * indent,
            obj.OMERO_CLASS,\
            obj.getId(),\
            obj.getName(),\
            obj.getOwnerOmeName())

def runWithSolrAsDataSource(solrUrl, rootDestinationDir, alreadyInOmero, omeroUser, omeroPass, omeroHost, omeroPort, group):
#need to get these passed in as arguments - the host and db name etc for jenkins to run
#first get the list of download urls and the data source, experiment, procdure and parameter and observation id for the images
    v = json.loads(requests.get(solrUrl).text)
    docs=v['response']['docs']
    numFoundInSolr=v['response']['numFound']
    for doc in docs:
        download_file_path=doc['download_file_path']
        datasource_id=doc['datasource_name']
        phenotyping_center=doc['phenotyping_center']
        #experiment=doc['experiment']
        pipeline_stable_id=doc['pipeline_stable_id']
        observation_id=doc['id']
        procedure_stable_id=doc['procedure_stable_id']
        parameter_stable_id=doc['parameter_stable_id']
        processFile(alreadyInOmero,observation_id, rootDestinationDir,phenotyping_center,pipeline_stable_id, procedure_stable_id, parameter_stable_id, download_file_path,  omeroUser, omeroPass, omeroHost, omeroPort, group)
           
    print 'number found in solr='+str(numFoundInSolr)+' number of failed responses='+str(responseFailed)+' number of requests='+str(numberOfImageDownloadAttemps)+' total totalNumberOfImagesWeHave='+str(totalNumberOfImagesWeHave)
   
def createDestinationFilePath(rootDestinationDir, phenotyping_center, pipeline_stable_id, procedure, parameter, download_file_path):
    directory="/".join([rootDestinationDir,phenotyping_center, pipeline_stable_id,procedure,parameter])
    return directory

def processFile(alreadyInOmero, observation_id,  rootDestinationDir, phenotyping_center,pipeline_stable_id, procedure, parameter, downloadFilePath,  omeroUser, omeroPass, omeroHost, omeroPort, group):
        global totalNumberOfImagesWeHave
        global responseFailed
        global numberOfImageDownloadAttemps
        directory = createDestinationFilePath(rootDestinationDir, phenotyping_center, pipeline_stable_id, procedure,parameter, downloadFilePath)
        print 'directory='+directory
        print 'downloadFilePath='+downloadFilePath
        dstfilename=directory+"/"+str(downloadFilePath.split('/')[-1])
        print 'dstfilename='+dstfilename
        calculatedFilePath=dstfilename.split(splitString,1)[1]
        #print 'calculatedFilePath='+calculatedFilePath
        if calculatedFilePath in alreadyInOmero:
            print 'calculatedFilePath is already in Omero '+calculatedFilePath
        else:
            print 'should load into omero '+calculatedFilePath
            slashSplit=calculatedFilePath.split("/")
            project=slashSplit[0]
            print 'project='+project
            dataset=slashSplit[3]
            print 'dataset='+dataset
            loadIntoOmero(dstfilename, project, dataset,  omeroUser, omeroPass, omeroHost, omeroPort, group)


        if dstfilename in uniqueUris:
            print '---------------------!!!!!!!!!!error the filePath is not unique and has been specified before:'+dstfilename
        uniqueUris.add(dstfilename)
        print 'dstfilename '+dstfilename
       
def loadIntoOmero(fullNfsUrl, project, dataset,  omeroUser, omeroPass, omeroHost, omeroPort, group):
    global numFoundInSolr
    print 'need to load into omero here'
    do_import(omeroUser, omeroPass, omeroHost, omeroPort,  fullNfsUrl, dataset, project, omeroPass, group)
    print len(uniqueUris), '/',numFoundInSolr
    print "import finished"


def do_import(omeroUser, omeroPass, omeroHost, omeroPort,  filename, dataset=None, project=None, passw=None, group="public_group"):
    print "printing user info"+omeroUser, group, filename
    #global cli
    print "-"*100
    #if cli is None:
        #print "cli is none!!!!!"
    cli = omero.cli.CLI()
    cli.loadplugins()
    cli.invoke(["login", omeroUser+'@'+omeroHost, "-w", passw, "-C"], strict=True)
    #cli.invoke(["login", "%s@localhost" % user, "-w", passw, "-C"], strict=True)
    cli.invoke(["sessions", "group", group], strict=True)
    import_args = ["import"]
    if dataset is not None:
        dsId = create_containers(cli, dataset,omeroHost, project)
        print "dsId="+str(dsId)
        import_args.extend(["-d", str(dsId),"--debug", "DEBUG"])#,"--debug", "ALL"])
    else:
        print "dataset is None!!!!!!!!!!!!!!!!!!!!"
    import_args.append(filename)
    print " import args="
    print import_args
    cli.invoke(import_args, strict=True)


def create_containers(cli, dataset, omeroHost, project=None):
    """
    Creates containers with names provided if they don't exist already.
    Returns Dataset ID.
    """
    sessionId = cli._event_context.sessionUuid
    conn = BlitzGateway(host=omeroHost)
    conn.connect(sUuid = sessionId)
    params = omero.sys.Parameters()
    params.theFilter = omero.sys.Filter()
    params.theFilter.ownerId = wrap(conn.getUser().getId())
    #print "ownerId="+conn.getUser().getId()
    #project=None
    from omero.rtypes import rstring
    d = None
    prId = None
    if project is not None:
        p = conn.getObject("Project", attributes={'name': project}, params=params)
        if p is None:
            print "Creating Project:", project
            p = omero.model.ProjectI()
            p.name = wrap(str(project))
            prId = conn.getUpdateService().saveAndReturnObject(p).id.val
        else:
            print "Using Project:", project, p
            prId = p.getId()
            # Since Project already exists, check children for Dataset
            for c in p.listChildren():
                if c.getName() == dataset:
                    d = c

    if d is None:
        d = conn.getObject("Dataset", attributes={'name': dataset}, params=params)

    if d is None:
        print "Creating Dataset:", dataset
        d = omero.model.DatasetI()
        d.name = wrap(str(dataset))
        dsId = conn.getUpdateService().saveAndReturnObject(d).id.val
        if prId is not None:
            print "Linking Project-Dataset..."
            link = omero.model.ProjectDatasetLinkI()
            link.child = omero.model.DatasetI(dsId, False)
            link.parent = omero.model.ProjectI(prId, False)
            conn.getUpdateService().saveObject(link)
    else:
        print "Using Dataset:", dataset, d
        dsId = d.getId()
    return dsId

def deleteImagesAlreadyInOmero( omeroHost, omeroPort, omeroUsername, omeroPass ):
     
    from omero.gateway import BlitzGateway


    # Connect to the Python Blitz Gateway
    # =============================================================
    # Make a simple connection to OMERO, printing details of the
    # connection. See OmeroPy/Gateway for more info
    conn = BlitzGateway(omeroUsername, omeroPass, host=omeroHost, port=omeroPort)
    connected = conn.connect()


    # Check if you are connected.
    # =============================================================
    if not connected:
        import sys
        sys.stderr.write("Error: Connection not available, please check your user name and password.\n")
        sys.exit(1)


    # Using secure connection.
    # =============================================================
    # By default, once we have logged in, data transfer is not encrypted (faster)
    # To use a secure connection, call setSecure(True):

    # conn.setSecure(True)         # <--------- Uncomment this


    # Current session details
    # =============================================================
    # By default, you will have logged into your 'current' group in OMERO. This
    # can be changed by switching group in the OMERO.insight or OMERO.web clients.
    print "Deleting projects and datasets and images!!!!!!!!!!!!!!!!!!!!!"
    user = conn.getUser()
    print "Current user:"
    print "   ID:", user.getId()
    print "   Username:", user.getName()
    print "   Full Name:", user.getFullName()

    print "Member of:"
    for g in conn.getGroupsMemberOf():
        print "   ID:", g.getName(), " Name:", g.getId()
    group = conn.getGroupFromContext()
    print "Current group: ", group.getName()

    print "Other Members of current group:"
    for exp in conn.listColleagues():
        print "   ID:", exp.getId(), exp.getOmeName(), " Name:", exp.getFullName()

    print "Owner of:"
    for g in conn.listOwnedGroups():
        print "   ID:", g.getName(), " Name:", g.getId()

    # New in OMERO 5
    print "Admins:"
    for exp in conn.getAdministrators():
        print "   ID:", exp.getId(), exp.getOmeName(), " Name:", exp.getFullName()

    # The 'context' of our current session
    ctx = conn.getEventContext()
    # print ctx     # for more info
    print ctx
    # The only_owned=True parameter limits the Projects which are returned.
    # If the parameter is omitted or the value is False, then all Projects
    # visible in the current group are returned.
    print "\nList Projects:"
    print "=" * 50
    my_expId = conn.getUser().getId()
   
    for project in conn.listProjects(my_expId):
        print_obj(project)
        object_ids=[project.getId()]
        deleteChildren = True
        handle = conn.deleteObjects("Project", object_ids,\
deleteAnns=True, deleteChildren=deleteChildren)
    # Close connection:
    # =================================================================
    # When you are done, close the session to free up server resources.
    conn._closeSession()

if __name__ == "__main__":
    main(sys.argv[1:])


[jwarren]

We've also noticed many processes seem be running on the server by the user that is running omero. So much so that the VM refused to let that user ssh on to the machine this morning?

[jmoore]

Could you send us list of the processes? It sounds like general resource exhaustion is what you're running into. How much physical memory is on the host and how much is currently used/free?
~Josh

[jwarren]

So after restarting the vm and currently running the import python script:
from top:
Tasks: 168 total, 1 running, 167 sleeping, 0 stopped, 0 zombie
Cpu(s): 37.0%us, 2.2%sy, 0.0%ni, 59.7%id, 0.5%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 3920752k total, 3807800k used, 112952k free, 47884k buffers
Swap: 4128760k total, 59740k used, 4069020k free, 871168k cached

[jwarren]

from a ps -U ( I tried a pstree but too many characters to post to forum):

PID TTY TIME CMD
7045 ? 00:00:02 java
7073 pts/0 00:00:00 ps
19538 ? 00:00:00 sshd
19539 pts/0 00:00:00 bash
20203 ? 00:00:00 sshd
20204 pts/2 00:00:00 bash
20229 ? 00:00:00 sshd
20230 pts/3 00:00:00 bash
21505 pts/2 00:00:00 tail
22950 ? 00:00:01 icegridnode
22991 ? 00:05:22 java
23003 ? 00:00:06 python
23004 ? 00:00:00 python
23005 ? 00:00:02 python
23007 ? 00:00:41 java
23014 ? 00:00:22 glacier2router
23016 ? 00:00:03 python
23017 ? 00:00:31 java
23022 ? 00:00:00 icebox
23244 pts/3 00:00:00 less
23279 ? 00:00:00 python
23280 ? 00:00:00 python
23281 ? 00:00:00 python
23282 ? 00:00:00 python
23283 ? 00:00:01 python
23284 ? 00:00:00 python
23401 ? 00:00:01 sshd
23402 ? 00:00:00 bash
23409 ? 00:01:14 python

[jmoore]

Are most of the python processes your invocations? Could you try `ps auxw -H` and try uploading as an attachment or sending as a private email? Cheers, ~Josh