Replicating an OMERO instance to get TEST environment

[rpoehlmann]

Dear all,

I am currently trying to clone/replicate an OMERO instance in order to obtain a TEST environment for rehearsal of OS (CentOS 6->7) and OMERO (5.1.x->5.2.x) version upgrade. While doing so, I stumbled across several issues I would like to address to the community:

• Is there kind of an official procedure how to clone an instance?
• How can it be avoided that PROD and cloned TEST instance will automatically build-up an OMERO grid upon service startup? No, I do not want this!
• How can specific network interfaces be expluded on an OMERO server from being used for internal service connections? E.g. the cloned TEST server has an Infiniband adapter used for connection to a storage device. Apparantly, this one was discovered during OMERO service startup and now also shows up in the node-connection list
  

  Code: Select all

  4901 | BlitzManager -t -e 1.1:tcp -h 10.41.26.240 -p 36663:tcp -h 10.1.26.240 -p 36663:tcp -h 131.152.26.240 -p 36663 |        -1
20 |                            |       | 2016-06-08 10:07:06.515    | d8396784-53df-4dd4-874c-8f637605c392 |       0 |

  
  Can this be prevented somehow, e.g. by explicitely excluding the Infiniband interface 10.41.26.240 in the above example from being used?

Many thanks in advance for your help.

Best regards,
-Rainer

[jmoore]

Dear all,

Hi Rainer,

Is there kind of an official procedure how to clone an instance?

That terminology is not used, but "Backup and Restore is what we use ourselves. This includes migrating the configuration from one installation to the next. Alternatively, we try to use some form of virtualization (docker, virtualbox, openstack) when we know more cloning is necessary.

How can it be avoided that PROD and cloned TEST instance will automatically build-up an OMERO grid upon service startup? No, I do not want this!

OMERO uses a number of ports for finding the various services that are run. If you attempted to start 2 OMERO instances on the same machine, I would expect an error in at least one of them saying that a particular port (like 4063, 4064, or 4061) could not be bound.

How can specific network interfaces be expluded on an OMERO server from being used for internal service connections? E.g. the cloned TEST server has an Infiniband adapter used for connection to a storage device. Apparantly, this one was discovered during OMERO service startup and now also shows up in the node-connection list
...
Can this be prevented somehow, e.g. by explicitely excluding the Infiniband interface 10.41.26.240 in the above example from being used?[/list]

The Ice.Default.Host property can be used to pick a single interface. If unset, all interfaces are used.

Code: Select all

bin/omero config set Ice.Default.Host 10.41.26.240


Cheers,
~Josh.

[rpoehlmann]

Hi Josh,

thanks a lot for your reply.

That terminology is not used, but "Backup and Restore is what we use ourselves. This includes migrating the configuration from one installation to the next. Alternatively, we try to use some form of virtualization (docker, virtualbox, openstack) when we know more cloning is necessary.

Virtualization is not really "feasible" here since I need to prepare/rehearse for 2 upgrade tasks:

• migrate the existing productive system onto new hardware
• upgrade to the most recent OMERO version

Therefore, I wanted to first "backup and restore" the existing setup onto the new hardware, get it running there (as a test, non-productive), then rehearse the OMERO upgrade. If everything is fine, then repeat the procedure but this time real.

OMERO uses a number of ports for finding the various services that are run. If you attempted to start 2 OMERO instances on the same machine, I would expect an error in at least one of them saying that a particular port (like 4063, 4064, or 4061) could not be bound.

Here, we have 2 instances on 2 distinct pieces of hardware. However, when OMERO on the the 2nd hardware (the new server) was started, certain features (image import) did not work correctly and we found from the logs that OMERO no. 2 apparently tries to communicate with OMERO no. 1 although we never ever told it to do so.
Now we are blocking those communication attempts by firewall rules on OMERO no. 1: do not allow any connections from OMERO no. 2. Nevertheless, OMERO no. 2 still tries when importing a new image (which fails!):

Blitz0.log:

2016-06-09 18:08:37,638 INFO [ ome.services.blitz.fire.Registry] (erver-1317) Found 3 repo(s) : [InternalRepository-ScriptRepo -t -e 1.1:tcp -h 131.152.26.240 -p 56624, InternalRepository-7ac6874a-d7b7-4237-888f-8ecef88ddb74 -t -e 1.1:tcp -h 10.1.0.172 -p 52072:tcp -h 131.152.25.172 -p 52072, InternalRepository-f49e4e5d-9854-4ab7-b547-2ca9df884cf6 -t -e 1.1:tcp -h 10.1.0.172 -p 52072:tcp -h 131.152.25.172 -p 52072]

It should find only 1 repo at 131.152.26.240 and nothing else?!? All other IP's (10.1.0.172 + 131.152.25.172)are from OMERO no 1.

OMERO Config (which does not mention any other repo!):

Code: Select all

Ice.Default.Host=131.152.26.240
Ice.IPv6=0
omero.data.dir=/export/omero/OMERO
omero.db.gist=localhost
omero.db.name=omerodb_dev
omero.db.pass=************************************
omero.db.poolsize=80
omero.db.user=omerouser
omero.jvmcfg.max_system_memory=96000
omero.jvmcfg.percent.blitz=30
omero.jvmcfg.percent.indexer=10
omero.jvmcfg.percent.pixeldata=30
omero.jvmcfg.percent.repository=10
omero.jvmcfg.strategy=percent
omero.jvmcfg.system_memory=96000
omero.ldap.base=SOME_LDAP_BASE
omero.ldap.config=true
omero.ldap.password=****************************
omero.ldap.urls=ldap://SOME_LDAP:3268
omero.ldap.user_mapping=omeName=sAMAccountName,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=SOME_NAME
omero.mail.config=true
omero.mail.from=OMERONEW at IMCF <nikolaus.ehrenfeuchter@unibas.ch>
omero.pixeldata.threads=4
omero.sessions.timeout=600000
omero.upgrades.url=
omero.web.admins=[["Rainer Poehlmann", "rainer.poehlmann@unibas.ch"], ["Niko Ehrenfeuchter", "nikolaus.ehrenfeuchter@unibas.ch"]]
omero.web.application_server=fastcgi-tcp
omero.web.apps=["figure", "autotag", "tagsearch"]
omero.web.email_host=smtp.unibas.ch
omero.web.email_subject_prefix=OMERONEW@Biozentrum Notification
omero.web.server_email=omeronew-biozentrum@unibas.ch
omero.web.server_list=[["omeronew.biozentrum.unibas.ch", 4064, "OMERONEW@BZ"]]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero.web.webstart_host=omeronew.biozentrum.unibas.ch

Where could this other repo come from? From the database? (I of course dumped/restores the DB from OMERO no. 1 to no. 2). Puzzled!!!

The Ice.Default.Host property can be used to pick a single interface. If unset, all interfaces are used.

Great: I tried it and it worked (see above config list)!

Cheers,
-Rainer

[jmoore]

Most importantly:

Great: I tried it and it worked (see above config list)!

Glad to hear it. A theory as to the proxy: when you migrated, did you copy your entire installation, i.e. including the var/ directory?

Cheers,
~Josh.

[rpoehlmann]

A theory as to the proxy: when you migrated, did you copy your entire installation, i.e. including the var/ directory?

Yes, clearing out only the nginx_tmp and the log directory.

Code: Select all

[omeronas@omero02 var]$ ls -ltr
total 16
lrwxrwxrwx 1 omeronas omeronas   29 Jun 17  2015 log -> /scratch/OMERO.server/var/log
drwxr-x--- 3 omeronas omeronas 4096 Jun 17  2015 registry
drwxr-x--- 5 omeronas omeronas 4096 Jun 17  2015 master
srwxr-xr-x 1 omeronas omeronas    0 May 23 17:35 django_fcgi.sock
-rw-r--r-- 1 omeronas omeronas    6 Jun  8 16:36 django.pid
drwx------ 2 omeronas omeronas 4096 Jun  8 17:44 nginx_tmp

From your question I kind of conclude that this apparently was not a brilliant idea, was it?
Should I give it another try without it, starting over from scratch again?

Thanks,
-Rainer

[jmoore]

It's an interesting question, Rainer. I've never seen the behavior your suggesting, but on the flip-side, I've never explicitly tested for it. All of var is considered volatile, and in fact there are cases were we suggest removing it. Try deleting it or backing it up (mv var var.bak) as a part of your cloning step. If the problem you were seeing goes away, we'll add an explicit statement to the backup-and-restore page.

Thanks for the heads up!
~Josh.

[rpoehlmann]

Hi Josh,
after quite some time of investigation, going back and forth, I guess I now have a clue what was actually happening to cause this issue we observed. And, of course as usual kind of a "self-made" problem!

Starting scenario as follows:

• setting up a TEST system to rehearse OS and OMERO version upgrade
• to also have some data present for the TEST system I rsynced part of the original data repository onto a new volume connected to the TEST server
• replicating the OMERO installation PROD => TEST by more or less following the official backup & restore procedure

Now, what apparently happened was the following:

• I forgot to also copy over the repo_uuid from the .omero subdirectory within the ManageRepository folder from PROD to TEST
• When starting up OMERO, every thing was "fine", images could be displayed (=read), however *no* images could be uploaded
• Upon the first image upload attempt, OMERO realized that the repository was there but its UUID (a new repo_uuid had been created) did not match the one within the restored database. Error:
  

  Code: Select all

  ome.formats.importer.ImportLibrary - Error on import: Directory exists but is not registered: CheckedPath(poehlman_2)

  
  
• As a consequence, OMERO TEST started "looking around" for other available ManagedRepositories and found OMERO PROD running in the same subnet!
• However, upload into this "erroneously clustered" repo on PROD also failed giving a weird error message (unfortunately not the clear one mentioned above I finally found much much later) and misleadingly pointing us into the wrong direction: OMERO spontaneously trying to form clustered services?!?

Everything solved now, I have a TEST system with connected (partially duplicated) TEST repository and even image upload does work!

Sorry again for the long delay in not replying, and thanks again for your help.

Best regards,
-Rainer