Client searching

[johnjhufnagle]

Hi,
I'm developing a public facing website that will perform searching over an OMERO image store. Will the Java/CPP/Python client api's and server support a large throughput? If we have 25k or 50k for an image count and a vague search matches on 5k or 10k (or more) images is there a way in the api to deal with pagination from the web tier or does the web tier have to bring back the entire List<> of 5 or 10k ImageI objects and maintain the list in memory and paginate on that side? If this kind of process can be accomplished with the Search client side object how would I set that up?
Thanks for your help
John

[johnjhufnagle]

I was digging around in the OMERO code and it looks like OMERO uses the Hibernate/Lucene option rather than Lucene on its own?
So refering back to my original question...if from the client side I issue a query that matches 15K images would all 15K Image objects be 're-constituted' by Hibernate/Lucene and then returned back over the wire to my client? If so do you think it would be possible for me to 'adjust' the server code to only have the query on the server side instantiate a collection 15K Image id's rather than the entire Image object and return that id colletion to the client. In that case I could then simply issue 20 calls back to the OMERO server to get the details about those 20 images being shown on the nth page of the search results list.
Any thoughts?
Thanks
John

[jmoore]

Hi John,

The Search API provides for forward-scrolling. The default batch size is currently set to 1000, meaning that each call to search.results() will at most return 1000 objects. You can increase or decrease this number via setBatchSize.

If you would like to reduce the returned data size, you can use setReturnUnloaded which will pass back objects with only the id set.

Cheers,
~Josh.

[johnjhufnagle]

Thanks Josh that sounds like something I could work with

[johnjhufnagle]

Is there a way on the SearchPrx api to get the total size. The setBatchSize() works as expected except that it is a process of 'next'ing through those batches. Anyway to get the total count?

[jmoore]

Not at the moment (as of 4.1). Feel free to open a ticket under the "Search" component on trac.

[johnjhufnagle]

Thanks Josh.
I was looking through the wiki and I have a question(s)

In a multithread Java client, if I execute:

1. omero.client c = new omero.client(map); //do this just once @startup

Can multiple threads share the ServiceFactoryPrx returned by a succeeding?:
omero.api.ServiceFactoryPrx s = c.createSession();

Concerning multiple threads that wish to use a SearchPrx. For now I expect I will run the search and return only the Image id's in an 'unloaded' fashion (I'll 'next' through the entire id resultset to produce an id collection I can keep around) and then use IQuery.find() to return the details for each page of images as I need them.

Would it be best to create my stateful SearchPrx objects (createSearchService()) each time at the start of an Image search and release it once I have the collection of image id's or is it costly to create and perhaps just create a pool of SearchPrx objects at startup and just use them as needed by each thread.

Similarly since I will be using stateless IQueryPrx object can I use one instance of that across threads or should I have each thread perform a 'getQueryService()'?

Thanks
John

[jmoore]

In a multithread Java client, if I execute:

1. omero.client c = new omero.client(map); //do this just once @startup

Can multiple threads share the ServiceFactoryPrx returned by a succeeding?:
omero.api.ServiceFactoryPrx s = c.createSession();

Definitely. The service factory and the services returned are all thread-safe, though there are of course operations which you will want to serialize.

Concerning multiple threads that wish to use a SearchPrx. ... Would it be best to create my stateful SearchPrx objects (createSearchService()) each time at the start of an Image search and release it once I have the collection of image id's or is it costly to create and perhaps just create a pool of SearchPrx objects at startup and just use them as needed by each thread.

Creating Search service instances is definitely not costly. Running many large queries and then keeping the results server-side would be a bad idea, but once you next'd the results, the resources will be freed. Then you can either release or reuse it. Nevertheless, you will have to be careful with multple threads.

Similarly since I will be using stateless IQueryPrx object can I use one instance of that across threads or should I have each thread perform a 'getQueryService()'?

One stateless instance per omero.client object is sufficient. In fact, two calls to getQueryService will return handles to the same server-side object.

Cheers,
~Josh.