Search index performance

[unibasscott]

Hi All,

We have the new version of the OMERO 4.4.5 server now setup and running in Basel, however we have
a few questions we have not been able to find answers for on the site.

Is the search indexing in anyway affected by the load of the import process? With our new users wishing to upload their existing data into OMERO and the size of an individual scientist data around a few terabytes. Are there any implications with the search indexing process in importing?

We have read the following page in the documentation http://www.openmicroscopy.org/site/supp ... g-indexing but are unsure if it would be necessary to stop the indexing and have to re-index the database for such a "big" import.

Many thanks,

IMCF Basel

[jmoore]

Hi Scott,

Is the search indexing in anyway affected by the load of the import process? With our new users wishing to upload their existing data into OMERO and the size of an individual scientist data around a few terabytes. Are there any implications with the search indexing process in importing?

The size of the import itself is less of an issue than the metadata attached (number of images/plates/wells/planes/etc). Of course, if the binary IO is fully occupying the server, then yes, indexing will be slower.

We have read the following page in the documentation ... but are unsure if it would be necessary to stop the indexing and have to re-index the database for such a "big" import.

It's hard to say. If you see that the indexer is not keeping up, or if the server is overall experiencing issues, it's certainly safe to disable the indexer:

Code: Select all

bin/omero admin ice server disable Indexer-0   # Prevents restart on stop
bin/omero admin ice server stop Indexer-0


And then start it up again later:

Code: Select all

bin/omero admin ice server enable Indexer-0  # start is called automatically


If you feel then that the indexer would never be able to catch back up, you can index specific types/tables (usually images) and then restart the indexer.

Perhaps the best way forward is to try some import and see how the indexer fairs.

Cheers,
~Josh.

[ehrenfeu]

Hi Josh,

thanks for the update, this already helps a lot. As we made sure to have the database, the repository and the generic stuff (OS, OMERO installation) all on different I/O subsystems, I have the feeling we're on the safe side here

But we'll see...

One more thing:

And then start it up again later:

Code: Select all

bin/omero admin ice server disable Indexer-0  # start is called automatically


I guess this was more meant to be "enable" instead of "disable"?

Thanks again,
Niko

[jmoore]

I guess this was more meant to be "enable" instead of "disable"?

Indeed. Fixed.
Thanks,
~J.