inconsistent error behavior when processing many images

[jennBakal]

hi,

i'm running some code to process features from within bin/omero shell that's exhibiting some strange behavior (it's similarly strange when run as a python script):

Code: Select all

for iid in list:
  filename='iid'+str(iid)+ '_fieldfeatures.h5'
  query = 'select f.id from originalfile f where f.name = \''+ filename +'\''
  output=subprocess.Popen(['psql','omero2','-c',query],stdout=subprocess.PIPE)
  if '0 row' in output.communicate()[0]:
    features = pslid.calculateFeatures( session, gateway, iid )
    pslid.linkFeatures( session, iid, features )
    now=datetime.datetime.now()
    print str(iid),now.hour,now.minute,now.second
    PrintToFile1(str(iid)+' '+str(now.hour)+' '+str(now.minute)+' '+str(now.secon\
d))
    del features
  else:
    print 'iid '+str(iid)+' already calculated'
  del filename, query, output


if len(list)< approx 110, this code runs fine.
however, at some point > 100, the code either stalls or causes an error. the error is not always the same, but at least a couple times it was a heap space error. this latest time, i got the following error a couple times:

Code: Select all

2011-04-05 04:44:46,870 ERROR [            ome.services.throttling.Task] (l.Server-5) Failed to invoke exception()


this also happens if i run (for example) list A with 50 iids and then list B with 100 iids.
if i don't restart the server before rerunning the script or restarting omero shell, an error, not always the same as the one that crashed it, occurs. i tried putting del statements in the pslid.calculateFeatures and pslid.linkFeatures functions as well but that didn't seem to make a difference.

any suggestions would be appreciated.

[jmoore]

Hi Jenn,

what all are you doing in your "pslid.calculateFeatures()" method? I assume writing several files? Are you closing any stateful services that you may have? In general, any service which has a close method will need to have close called, especially if you are in a loop. If there's nothing that you can close yourself, there may be a leak in OMERO that will need to be taken care of. Perhaps you could attach your Blitz-0.log?

Best wishes,
~Josh

[icaoberg]

This is calculateFeatures()

Code: Select all

def calculateFeatures( session, gateway, iid ):
    """
    Calculates and returns features vector given a valid
    image identification (iid). It currently calculates SLF34.
    @param session
    @param gateway
    @param image id (iid)
    @return features' vector
    """

    img=pyslic.Image()
    img.label=iid
    channels=['protein','dna']
    slice=0
    timepoint=0
    for c in channels:
        channel_num=channels.index(c)
        img.channels[c]=channel_num
        img.channeldata[c]=getPlane(session,gateway,iid,slice,channel_num,timepoint)
    img.loaded=True
    features = []
    ids = ["SLF27.66","SLF27.67","SLF27.68","SLF27.69","SLF27.70","SLF27.71","SLF27.72","SLF27.73","SLF27.74",$
    values = pyslic.computefeatures(img,'field-dna+')
    features = [ ids, values ]
    del img,ids,values
    return features


Get plane is your method

Code: Select all

def getPlane( session, gateway, iid, slice, channel, timepoint ):
   rawPixelsStore = session.createRawPixelsStore()
   image = gateway.getImage( iid )
   pid = image.getPixels( 0 ).getId().getValue();
   pixels = session.getPixelsService().retrievePixDescription(pid);
   rawPixelsStore.setPixelsId( pid, True )
   plane = utils.downloadPlane( rawPixelsStore, pixels, slice, channel, timepoint );
   return plane


Calculate features does not create files.

Thanks
Ivan

[jmoore]

Hi Ivan,

getPlane definitely has a memory leak in it. In service returned from session via a "create" method (e.g. session.createRawPixelsStore()) should be closed as soon as possible.

You say that it's our getPlane method. Where did it come from?

Cheers,
~Josh.

[jennBakal]

In service returned from session via a "create" method (e.g. session.createRawPixelsStore()) should be closed as soon as possible.

are you saying that getPlane should look something like this:

Code: Select all

rawPixelsStore=session.createRawPixelsStore()
[do some stuff with rawPixelsStore]
plane=utils.downloadPlane(rawPixelsStore,pixels,slice,channel,timepoint)
rawPixelsStore.close()      <--- or whatever is used to close rawPixelsStore
return plane


thanks,
jenn

[jmoore]

Jenn, exactly. But I'd add to do it in a try/finally block:

Code: Select all

def getPlane( session, gateway, iid, slice, channel, timepoint ):
   image = gateway.getImage( iid )
   pid = image.getPixels( 0 ).getId().getValue();
   pixels = session.getPixelsService().retrievePixDescription(pid);

   rawPixelsStore = session.createRawPixelsStore()
   try:
       rawPixelsStore.setPixelsId( pid, True )
       plane = utils.downloadPlane( rawPixelsStore, pixels, slice, channel, timepoint );
       return plane
    finally:
        rawPixelsStore.close()


Another way to do it would be to pass in the rawPixelsStore to getPlane so that you can re-use it for all the planes. The method above is safe, but a bit wasteful.
Cheers,
~Josh