Ice Memory Limit Exception

[ebbwdan]

Hi

I'm getting the following error from our server:

Traceback (most recent call last):
File "./script", line 544, in <module>
runAsScript()
File "./script", line 533, in runAsScript
robj, message = make_images_from_rois(conn, parameterMap)
File "./script", line 365, in make_images_from_rois
newImage, newDataset, link, new_ids = process_image(conn, iId, parameterMap)
File "./script", line 265, in process_image
description=description, sourceImageId=imageId)
File "/home/omero/OMERO.server-5.0.2-ice35-b26/lib/python/omero/gateway/__init__.py", line 2896, in createImageFromNumpySeq
raise exc
Ice.MemoryLimitException: exception ::Ice::MemoryLimitException
{
reason = requested 773974123 bytes, maximum allowed is 67108864 bytes (see Ice.MessageSizeMax)
}

}

I am attempting to grab a large roi (~2e4x2e4) from a very large slide scanner image and create a new image in the server from this. To avoid heap space issues I am grabbing the roi in small blocks but I am then having to rebuild the final dataset in memory from the blocks before writing the whole to the server using createImageFromNumpySeq.

We have increased the heap space to cover ourselves (we don't get heap space errors) and we have set the MessageSizeMax to 1024MB but still the error states that this is 64MB. Is this a hard-coded maximum value?

Is there a better way to approach this kind of thing? Incremental writing of the image to the server?

Thanks in advance for any help/suggestions!

Cheers,

Dan.

[jmoore]

Hi Dan,

We have increased the heap space to cover ourselves (we don't get heap space errors) and we have set the MessageSizeMax to 1024MB but still the error states that this is 64MB. Is this a hard-coded maximum value?

Hard-coded in the sense of global and configured at startup, yes. Ice wants to have a fixed limit to the size of array that will be passed over the wire. Here you're trying to send 700MB. You could bump it to 1GB to workaround, but this could obviously lead to other problems. (Lost connections, etc)

Ultimately, this is a bug in the uploadPlane inner function of createImageFromNumpySeq. It will need to upload the image in blocks rather than the full plane.

I've filed a ticket: https://trac.openmicroscopy.org.uk/ome/ticket/12459

If you'd like to try to come up with a workaround before that is completed, we can try to do so here on the forums.

Cheers,
~Josh.

[ebbwdan]

Hi Josh,

Thanks for the response! We have fixed up setting MessageSizeMax but I'm definitely taking your advice and writing the new image as tiles. My plan is to make a block generator which I can pass to something like 'createImageFromNumpySeq' and use 'setTile' on the 'rawPixelStore'. Does that seem sensible?

Cheers,

Dan.

[jmoore]

It does, indeed.

[ebbwdan]

Hi Josh,

Here's where I got to. I created a tile generator which I then loop over inside my method 'createImageFromNumpyTileSeq':

Code: Select all

        for theTile in range(sizeTiles):
            x,y,w,h = planeCoords.next()
            for theZ in range(sizeZ):
                for theC in range(sizeC):
                    for theT in range(sizeT):
                        plane = zctPlanes.next()
                        if image == None: # use the first plane to create image.
                            image, dtype = createImage(plane, channelList)
                            pixelsId = image.getPrimaryPixels().getId().getValue()
                            rawPixelsStore.setPixelsId(pixelsId, True, conn.SERVICE_OPTS)
                        uploadPlane(plane, theZ, theC, theT, x, y, w, h, dtype)


I create the new image after getting the first plane of the first tile (exactly as in 'createImageFromNumpySeq' but instead of using the plane shape to set the pixel buffer size I instead pass in the total size of the final image I want to make out of the tiles. I am now running into the following error:

ROMIO pixel buffer only supports full row writes

Because my image size is not the same size as the tile size. Some pointers about how to proceed would be very much appreciated! Thank you!

Cheers,

Dan.

[wmoore]

Hi Dan,

It looks like this script from Josh might be useful to you.

https://gist.github.com/joshmoore/816cdf461669a7e06d46

It creates a large tiled image and uploads tiles as numpy 2D arrays one at a time.

Currently it's just creating fake tiles from a numpy function, but you should be able to convert it to grab the appropriate tile from your larger image.

Let us know how you get on,

Will

[ebbwdan]

Hi Will and Josh,

That does the job nicely, thank you! It seems that the error I was getting:

ROMIO pixel buffer only supports full row writes

is raised when trying to use 'setTile' on an image with a shape < 4096 and a tile shape != image shape.

Cheers,

Dan.

[ebbwdan]

Just to clarify, I am testing on vm so I was keeping the ROIs relatively small. I ran into this repeatedly with 'setTile':

ROMIO pixel buffer only supports full row writes

until I tried an ROI with a shape >= 4096.

Cheers,

Dan.

[ebbwdan]

Hi Will and Josh,

Another naive question. When I grab an ROI from an image in my script, am I correct in thinking that an image pyramid does not get generated?

And is this possible? The ROIs potentially could be really big and now that I can successfully create tiled images from ROIs, I was wondering if I could force the creation of a pyramid to make the image display in the viewer a bit faster.

Cheers,

Dan.

[jmoore]

Hi Dan,

When I grab an ROI from an image in my script, am I correct in thinking that an image pyramid does not get generated?

A pyramid is only generated when either 1) an original image file is uploaded to the server after which the pyramid is generated in a background queue or 2) an image without an original file is uploaded in tiles in which case the act of uploading generates the pyramid in real time. After that, nothing would trigger the creation of a pyramid, unless the server setting for maximum tile size were lowered.

The ROIs potentially could be really big and now that I can successfully create tiled images from ROIs, I was wondering if I could force the creation of a pyramid to make the image display in the viewer a bit faster.

Here I'm a bit confused. What does really big mean? i.e. how big are the ROIs and how big is the image (are the images)? If you uploaded an image in tiles that was larger than 3000x3000, you should already have a pyramids file. You can check under /OMERO/Pixels for a file ending in "_pyramid".

Cheers,
~Josh