retrive error Ice.MemoryLimitException

[jiez]

Hi all,

I face another problem when retrieving data by omero_client API (Java) from OMERO server.

what I try to do is retrieving 'projects', 'datasets' and 'images' names & descriptions with image thumbnails. Additionally very simple image tags (e.g., date, experimental description, gene) are to be extracted.

A few users have limited images (3000 jpeg) that everything runs fine, but one user has a large number of images (20G jpeg) that causes Ice.MemoryLimitException problem.

I run the java application on a laptop with 2GB memory (ubuntu 32-bit), here is the error msg:

connection is set up and userId is:352**.

Exception in thread "main" Ice.UnknownLocalException
unknown = "Ice::MemoryLimitException
Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeInt(BasicStream.java:955)
at IceInternal.BasicStream.writeInstance(BasicStream.java:1793)
at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1712)
at omero.api._AMD_IContainer_loadContainerHierarchy.ice_response(_AMD_IContainer_loadContainerHierarchy.java:31)
at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at ome.services.throttling.Task.response(Task.java:63)
at ome.services.throttling.Callback.run(Callback.java:57)
at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
at ome.services.blitz.impl.ContainerI.loadContainerHierarchy_async(ContainerI.java:179)
at omero.api._IContainerTie.loadContainerHierarchy_async(_IContainerTie.java:134)
at omero.api._IContainerDisp.___loadContainerHierarchy(_IContainerDisp.java:194)
at omero.api._IContainerDisp.__dispatch(_IContainerDisp.java:637)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._IContainerDelM.loadContainerHierarchy(_IContainerDelM.java:585)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:730)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:702)
at..........




I am not sure this is from my laptop or from the server (Debian 64-bit with 96 GB memory), but in omero.server.4.3.4/var/log/master.err

similar memory limit error

10:37:41:974 warning: Blitz-0-Ice.ThreadPool.Server-5: dispatch exception:
identity: c86f648c-c54c-4fa3-a34d-2c3af43b8fe3/2b82933d-cba3-46b8-9fc1-2102c638270aomero.api.IContainer
facet:
operation: loadContainerHierarchy
Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeInt(BasicStream.java:955)
at IceInternal.BasicStream.startWriteSlice(BasicStream.java:531)
at omero.model.Event.__write(Event.java:742)
at IceInternal.BasicStream.writeInstance(BasicStream.java:1809)
at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1712)
at omero.api._AMD_IContainer_loadContainerHierarchy.ice_response(_AMD_IContainer_loadContainerHierarchy.java:31)
at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at ome.services.throttling.Task.response(Task.java:63)
at ome.services.throttling.Callback.run(Callback.java:57)
at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
at ome.services.blitz.impl.ContainerI.loadContainerHierarchy_async(ContainerI.java:179)
at omero.api._IContainerTie.loadContainerHierarchy_async(_IContainerTie.java:134)
at omero.api._IContainerDisp.___loadContainerHierarchy(_IContainerDisp.java:194)
at omero.api._IContainerDisp.__dispatch(_IContainerDisp.java:637)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971).


The similar problem reported previously was about importing big files, not like this.

Can anyone help ?

many thanks

jiez

[jmoore]

You'll need to load the images in batches (or "pages") in order to prevent the memory exceptions. When dealing with large lists or wide images, there will always be a need to iterate over smaller bundles.

Can you show us the code your using so we can make a concrete suggestion?

Cheers,
~Josh

[jiez]

thanks Josh.

Basically I follow the template from here:

http://www.openmicroscopy.org/site/supp ... /Java.html

to retrieve 'projects', 'datasets', or 'images'. The error comes up when processing 'project', so i paste a simplified version.

public static void retrieveProjects (long lon_userId) throws IOException, ServerError
{
IContainerPrx proxy = myEntry.getContainerService();
ParametersI param = new ParametersI();
long userId =lon_userId; // entryUnencrypted.getAdminService().getEventContext().userId;
param.exp(omero.rtypes.rlong(userId));
param.leaves(); //indicate to load the images
//param.noLeaves(); //no images loaded, this is the default value.


List<IObject> results = proxy.loadContainerHierarchy(Project.class.getName(), new ArrayList<Long>(), param);
//You can directly interact with the IObject or the Pojos object.
//Follow interaction with the Pojos.

Iterator<IObject> i = results.iterator();

ProjectData project;
Set<DatasetData> datasets;

Iterator<DatasetData> j;
DatasetData dataset;


while (i.hasNext()) {
project = new ProjectData((Project) i.next()); // ONE PROJECT;

vProject = project.getName();
vProjectId = project.getId();

String project_temp = project_df;
if (project_df.equals(project_ND)) // do not need project check;
{project_temp =vProject;}


if (vProject.equals (project_temp)) {
datasets = project.getDatasets(); // ALL DATASETS IN ONE PROJECT;
j = datasets.iterator();
while (j.hasNext()) {
dataset = j.next(); // ONE DATASET IN ONE PROJECT;
vDataSet = dataset.getName();
vDataSetId = dataset.getId();

System.out.print("start retrieving project:"+vProject+".\n\n");
String dataset_temp = dataset_df;
if (dataset_df.equals(dataset_ND)) // do not check dataset;
{dataset_temp =vDataSet;}

if (vDataSet.equals(dataset_temp))
{ System.out.print("start retrieving dataset:"+vDataSet+".\n\n");
// retrieveImages (dataset);
} // end of dataset check;
} // end while j ;
} // end of if vProject check;
} // end while i;

} // end of projects;



public static void getDataForAnnot(String argv[]) throws ServerError, IOException

{
Jparse jparse = new Jparse();
JparseInfo jparseInfo = new JparseInfo();
HashMap key_vs_value = jparse.getJparseHashMap(jparseInfo,argv);

String hostname_ = ((String)key_vs_value.get(jparseInfo.hostName_k)).replace("null","");
String port_ = ((String)key_vs_value.get(jparseInfo.portName_k)).replace("null","");
String user_ = ((String)key_vs_value.get(jparseInfo.userName_k)).replace("null","");
String passwd_ = ((String)key_vs_value.get(jparseInfo.passwd_k)).replace("null","");
String project_ = ((String)key_vs_value.get(jparseInfo.project_k)).replace("null","");
String dataset_ = ((String)key_vs_value.get(jparseInfo.dataset_k)).replace("null","");
String thumbSize_ = ((String)key_vs_value.get(jparseInfo.thumbSize_k)).replace("null","");
String thumbPath_ = ((String)key_vs_value.get(jparseInfo.thumbPath_k)).replace("null","");
String thumbType_ = ((String)key_vs_value.get(jparseInfo.thumbType_k)).replace("null","");
String fileName_ = ((String)key_vs_value.get(jparseInfo.descrpFileName_k)).replace("null","");

hostName = hostname_;
portNo = Integer.parseInt(port_);
userId = user_;
passwd = passwd_;
project_df = project_;
// System.out.print("project_ is:"+project_+"***************************************.\n\n");
if (project_df == null) {project_df =project_ND;}
else
{if ((project_df.toLowerCase().equals("null")) || (project_df.toLowerCase().equals("none")) || (project_df.toLowerCase().equals("")))
{project_df = project_ND;} } // project else;
// System.out.print("project_df is:"+project_df+"**.\n\n");
dataset_df = dataset_;
// System.out.print("dataset_ is:"+dataset_+"***************************************.\n\n");
if (dataset_df == null) {dataset_df =dataset_ND;}
else
{if ((dataset_df.toLowerCase().equals("null")) || (dataset_df.toLowerCase().equals("none")) || (dataset_df.toLowerCase().equals("")))
{dataset_df = dataset_ND;} } // project else;
// System.out.print("dataset_df is:"+dataset_df+"***************************************.\n\n");
thumbSize = Integer.parseInt(thumbSize_);
thumbPath = thumbPath_;
thumbType = thumbType_;
descrpFileName = fileName_;


try {
myClient = new client (hostName, portNo);
myEntry = myClient.createSession(userId,passwd);

long userId =myEntry.getAdminService().getEventContext().userId;
long groupId = myEntry.getAdminService().getEventContext().groupId;

System.out.print("The connection is set up and userId is:"+userId+"**.\n\n");
// System.out.print("***********************.\n\n");
// System.out.print("********************888888**.\n\n");

vFileImageAttached = new FileWriter (descrpFileName);
bw = new BufferedWriter(vFileImageAttached);


retrieveProjects (userId);
// retrieveDatasets (userId);

myClient.closeSession();
bw.close();
vFileImageAttached.close();
}
catch (ServerError e) {System.err.print(e);}
catch (CannotCreateSessionException c) {System.err.print(c);}
catch (PermissionDeniedException p) {System.err.print(p);}




} // end of forPivot;


public static void main(String argv[]) throws ServerError, IOException

{
getDataForAnnot(argv);



} // end of main;



Jparse & JparseInfo for parsing parameters (not shown here).

Again, it works fine with most users but not this one (20G jpeg).

thanks.

[jmoore]

Hi Jiez,

I've taken your snippet and turned it into a runnable gist, https://gist.github.com/4123795, which batch loads the image IDs which you can then take to perform other operations.

The output for me begins:

Code: Select all

The connection is set up and userId is:0**.

groupId is:0**.

start retrieving project:test.
start retrieving dataset:test:5.
   Loaded 3 images in page 1
   Loaded 3 images in page 2
   Loaded 3 images in page 3
   Loaded 1 images in page 4
start retrieving dataset:test:6.
   Loaded 3 images in page 1
   Loaded 3 images in page 2
   Loaded 3 images in page 3
   Loaded 1 images in page 4
start retrieving dataset:test:8.
...


Hope that helps.
~Josh

[jiez]

Josh, thanks.

it works fine now.

Another help, as I understand, when connect to the server by

Client myClient = new client (hostName, portNo);
ServiceFactoryPrx myEntry = myClient.createSession(userName,passwd);

each individual username & password are required.

If I want to retrieve all users' data as a admin or group owner, without asking for password each time, is there any way doing this?

jiez

[jmoore]

If I want to retrieve all users' data as a admin or group owner, without asking for password each time, is there any way doing this?

If the userName you enter is either a system admin or a group admin (i.e. the owner of a group), then yes, you will be able to load data for other users. It depends on which method you are using.

You may want to take a look at some of the following resources:

• https://www.openmicroscopy.org/site/support/faq/omero-clients/omero-permissions-security-level
• https://www.openmicroscopy.org/site/support/omero4/sysadmins/server-permissions.html
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/InsightMultiGroups-4.4.mov
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/WebMultiGroups-4.4.mov
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/InsightMultiGroups-4.4.mov

Cheers,
~Josh

[jiez]

I think the easy method would be I log in as a admin, obviously I can get my own userId and groupid, then retrieve my data through projects (userId) .

not sure how to get other users' Ids based on groupid, or by what method ?

[jmoore]

IAdmin.lookupGroups will return you all groups and then all users linked to those groups. Alternatively, you can use a different query method and load images for more than one user at a time.

Cheers,
~Josh

[jiez]

thanks josh.

how to get Experimenter Id (long) from omero::model::Experimenter ?

it sounds no getId op from omero::model::Experimenter.

[jiez]

I can figure out from Iadmin.lookupGroups() to omero::model::Experimenter, but cannot figure out how to obtain Experimenter Id (userId) (long) from here.

Can i have an example by IQuery to obtain all userIds related to one shared group, if groupid is known ?

many thanks

Jiez