Open Microscopy Environment

Hi all,

I face another problem when retrieving data by omero_client API (Java) from OMERO server.

what I try to do is retrieving 'projects', 'datasets' and 'images' names & descriptions with image thumbnails. Additionally very simple image tags (e.g., date, experimental description, gene) are to be extracted.

A few users have limited images (3000 jpeg) that everything runs fine, but one user has a large number of images (20G jpeg) that causes Ice.MemoryLimitException problem.

I run the java application on a laptop with 2GB memory (ubuntu 32-bit), here is the error msg:

connection is set up and userId is:352**.

Exception in thread "main" Ice.UnknownLocalException
unknown = "Ice::MemoryLimitException
Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeInt(BasicStream.java:955)
at IceInternal.BasicStream.writeInstance(BasicStream.java:1793)
at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1712)
at omero.api._AMD_IContainer_loadContainerHierarchy.ice_response(_AMD_IContainer_loadContainerHierarchy.java:31)
at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at ome.services.throttling.Task.response(Task.java:63)
at ome.services.throttling.Callback.run(Callback.java:57)
at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
at ome.services.blitz.impl.ContainerI.loadContainerHierarchy_async(ContainerI.java:179)
at omero.api._IContainerTie.loadContainerHierarchy_async(_IContainerTie.java:134)
at omero.api._IContainerDisp.___loadContainerHierarchy(_IContainerDisp.java:194)
at omero.api._IContainerDisp.__dispatch(_IContainerDisp.java:637)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._IContainerDelM.loadContainerHierarchy(_IContainerDelM.java:585)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:730)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:702)
at..........




I am not sure this is from my laptop or from the server (Debian 64-bit with 96 GB memory), but in omero.server.4.3.4/var/log/master.err

similar memory limit error

10:37:41:974 warning: Blitz-0-Ice.ThreadPool.Server-5: dispatch exception:
identity: c86f648c-c54c-4fa3-a34d-2c3af43b8fe3/2b82933d-cba3-46b8-9fc1-2102c638270aomero.api.IContainer
facet:
operation: loadContainerHierarchy
Ice.MemoryLimitException
reason = (null)
at IceInternal.BasicStream.expand(BasicStream.java:2145)
at IceInternal.BasicStream.writeInt(BasicStream.java:955)
at IceInternal.BasicStream.startWriteSlice(BasicStream.java:531)
at omero.model.Event.__write(Event.java:742)
at IceInternal.BasicStream.writeInstance(BasicStream.java:1809)
at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1712)
at omero.api._AMD_IContainer_loadContainerHierarchy.ice_response(_AMD_IContainer_loadContainerHierarchy.java:31)
at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at ome.services.throttling.Task.response(Task.java:63)
at ome.services.throttling.Callback.run(Callback.java:57)
at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
at ome.services.blitz.impl.ContainerI.loadContainerHierarchy_async(ContainerI.java:179)
at omero.api._IContainerTie.loadContainerHierarchy_async(_IContainerTie.java:134)
at omero.api._IContainerDisp.___loadContainerHierarchy(_IContainerDisp.java:194)
at omero.api._IContainerDisp.__dispatch(_IContainerDisp.java:637)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971).


The similar problem reported previously was about importing big files, not like this.

Can anyone help ?

many thanks

jiez

You'll need to load the images in batches (or "pages") in order to prevent the memory exceptions. When dealing with large lists or wide images, there will always be a need to iterate over smaller bundles.

Can you show us the code your using so we can make a concrete suggestion?

Cheers,
~Josh

thanks Josh.

Basically I follow the template from here:

http://www.openmicroscopy.org/site/supp ... /Java.html

to retrieve 'projects', 'datasets', or 'images'. The error comes up when processing 'project', so i paste a simplified version.

public static void retrieveProjects (long lon_userId) throws IOException, ServerError
{
IContainerPrx proxy = myEntry.getContainerService();
ParametersI param = new ParametersI();
long userId =lon_userId; // entryUnencrypted.getAdminService().getEventContext().userId;
param.exp(omero.rtypes.rlong(userId));
param.leaves(); //indicate to load the images
//param.noLeaves(); //no images loaded, this is the default value.


List<IObject> results = proxy.loadContainerHierarchy(Project.class.getName(), new ArrayList<Long>(), param);
//You can directly interact with the IObject or the Pojos object.
//Follow interaction with the Pojos.

Iterator<IObject> i = results.iterator();

ProjectData project;
Set<DatasetData> datasets;

Iterator<DatasetData> j;
DatasetData dataset;


while (i.hasNext()) {
project = new ProjectData((Project) i.next()); // ONE PROJECT;

vProject = project.getName();
vProjectId = project.getId();

String project_temp = project_df;
if (project_df.equals(project_ND)) // do not need project check;
{project_temp =vProject;}


if (vProject.equals (project_temp)) {
datasets = project.getDatasets(); // ALL DATASETS IN ONE PROJECT;
j = datasets.iterator();
while (j.hasNext()) {
dataset = j.next(); // ONE DATASET IN ONE PROJECT;
vDataSet = dataset.getName();
vDataSetId = dataset.getId();

System.out.print("start retrieving project:"+vProject+".\n\n");
String dataset_temp = dataset_df;
if (dataset_df.equals(dataset_ND)) // do not check dataset;
{dataset_temp =vDataSet;}

if (vDataSet.equals(dataset_temp))
{ System.out.print("start retrieving dataset:"+vDataSet+".\n\n");
// retrieveImages (dataset);
} // end of dataset check;
} // end while j ;
} // end of if vProject check;
} // end while i;

} // end of projects;



public static void getDataForAnnot(String argv[]) throws ServerError, IOException

{
Jparse jparse = new Jparse();
JparseInfo jparseInfo = new JparseInfo();
HashMap key_vs_value = jparse.getJparseHashMap(jparseInfo,argv);

String hostname_ = ((String)key_vs_value.get(jparseInfo.hostName_k)).replace("null","");
String port_ = ((String)key_vs_value.get(jparseInfo.portName_k)).replace("null","");
String user_ = ((String)key_vs_value.get(jparseInfo.userName_k)).replace("null","");
String passwd_ = ((String)key_vs_value.get(jparseInfo.passwd_k)).replace("null","");
String project_ = ((String)key_vs_value.get(jparseInfo.project_k)).replace("null","");
String dataset_ = ((String)key_vs_value.get(jparseInfo.dataset_k)).replace("null","");
String thumbSize_ = ((String)key_vs_value.get(jparseInfo.thumbSize_k)).replace("null","");
String thumbPath_ = ((String)key_vs_value.get(jparseInfo.thumbPath_k)).replace("null","");
String thumbType_ = ((String)key_vs_value.get(jparseInfo.thumbType_k)).replace("null","");
String fileName_ = ((String)key_vs_value.get(jparseInfo.descrpFileName_k)).replace("null","");

hostName = hostname_;
portNo = Integer.parseInt(port_);
userId = user_;
passwd = passwd_;
project_df = project_;
// System.out.print("project_ is:"+project_+"***************************************.\n\n");
if (project_df == null) {project_df =project_ND;}
else
{if ((project_df.toLowerCase().equals("null")) || (project_df.toLowerCase().equals("none")) || (project_df.toLowerCase().equals("")))
{project_df = project_ND;} } // project else;
// System.out.print("project_df is:"+project_df+"**.\n\n");
dataset_df = dataset_;
// System.out.print("dataset_ is:"+dataset_+"***************************************.\n\n");
if (dataset_df == null) {dataset_df =dataset_ND;}
else
{if ((dataset_df.toLowerCase().equals("null")) || (dataset_df.toLowerCase().equals("none")) || (dataset_df.toLowerCase().equals("")))
{dataset_df = dataset_ND;} } // project else;
// System.out.print("dataset_df is:"+dataset_df+"***************************************.\n\n");
thumbSize = Integer.parseInt(thumbSize_);
thumbPath = thumbPath_;
thumbType = thumbType_;
descrpFileName = fileName_;


try {
myClient = new client (hostName, portNo);
myEntry = myClient.createSession(userId,passwd);

long userId =myEntry.getAdminService().getEventContext().userId;
long groupId = myEntry.getAdminService().getEventContext().groupId;

System.out.print("The connection is set up and userId is:"+userId+"**.\n\n");
// System.out.print("***********************.\n\n");
// System.out.print("********************888888**.\n\n");

vFileImageAttached = new FileWriter (descrpFileName);
bw = new BufferedWriter(vFileImageAttached);


retrieveProjects (userId);
// retrieveDatasets (userId);

myClient.closeSession();
bw.close();
vFileImageAttached.close();
}
catch (ServerError e) {System.err.print(e);}
catch (CannotCreateSessionException c) {System.err.print(c);}
catch (PermissionDeniedException p) {System.err.print(p);}




} // end of forPivot;


public static void main(String argv[]) throws ServerError, IOException

{
getDataForAnnot(argv);



} // end of main;



Jparse & JparseInfo for parsing parameters (not shown here).

Again, it works fine with most users but not this one (20G jpeg).

thanks.

Hi Jiez,

I've taken your snippet and turned it into a runnable gist, https://gist.github.com/4123795, which batch loads the image IDs which you can then take to perform other operations.

The output for me begins:

Code: Select all

The connection is set up and userId is:0**.

groupId is:0**.

start retrieving project:test.
start retrieving dataset:test:5.
   Loaded 3 images in page 1
   Loaded 3 images in page 2
   Loaded 3 images in page 3
   Loaded 1 images in page 4
start retrieving dataset:test:6.
   Loaded 3 images in page 1
   Loaded 3 images in page 2
   Loaded 3 images in page 3
   Loaded 1 images in page 4
start retrieving dataset:test:8.
...


Hope that helps.
~Josh

Josh, thanks.

it works fine now.

Another help, as I understand, when connect to the server by

Client myClient = new client (hostName, portNo);
ServiceFactoryPrx myEntry = myClient.createSession(userName,passwd);

each individual username & password are required.

If I want to retrieve all users' data as a admin or group owner, without asking for password each time, is there any way doing this?

jiez

jiez wrote:If I want to retrieve all users' data as a admin or group owner, without asking for password each time, is there any way doing this?

If the userName you enter is either a system admin or a group admin (i.e. the owner of a group), then yes, you will be able to load data for other users. It depends on which method you are using.

You may want to take a look at some of the following resources:

• https://www.openmicroscopy.org/site/support/faq/omero-clients/omero-permissions-security-level
• https://www.openmicroscopy.org/site/support/omero4/sysadmins/server-permissions.html
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/InsightMultiGroups-4.4.mov
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/WebMultiGroups-4.4.mov
• http://cvs.openmicroscopy.org.uk/snapshots/movies/omero-4-4/mov/InsightMultiGroups-4.4.mov

Cheers,
~Josh

I think the easy method would be I log in as a admin, obviously I can get my own userId and groupid, then retrieve my data through projects (userId) .

not sure how to get other users' Ids based on groupid, or by what method ?

IAdmin.lookupGroups will return you all groups and then all users linked to those groups. Alternatively, you can use a different query method and load images for more than one user at a time.

Cheers,
~Josh

thanks josh.

how to get Experimenter Id (long) from omero::model::Experimenter ?

it sounds no getId op from omero::model::Experimenter.

I can figure out from Iadmin.lookupGroups() to omero::model::Experimenter, but cannot figure out how to obtain Experimenter Id (userId) (long) from here.

Can i have an example by IQuery to obtain all userIds related to one shared group, if groupid is known ?

many thanks

Jiez