Scheduler/HPC cluster Integration

[ChrisB]

Hi All,
To save doubling up on resources our intention is integrate Omero into our HPC cluster, which means getting it to submit all processing jobs via a scheduler. Resources such as GPU (on a tesla) would be ran via a resource request.

My question is whether this is supported in the current version of Omero and if not, how big a task would you estimate it to be in order to set it up?

Any documentation/pointers would be welcome.
Thanks,

Chris.

[jmoore]

Hi Chris,

currently dispatching to a cluster queue is not supported out of the box. Two options available are either A) to launch small processes on each node which will then consume OMERO jobs or B) to implement a different Processor service which uses your cluster. Which jobs are you looking to run on the cluster?

Best,
~Josh

[a.herbert]

I am interested in using cluster resources for our OMERO installation. This would be to run intensive processing analysis jobs on images. The analysis invariably requires running a command-line program on input images.

My currently solution for OMERO scripts is to have the script do the initial verification of input parameters and set up the job. This is then submitted to a cluster farm. I use Torque PBS but it should work with any farm setup. The software is then installed on the nodes to be able to access all the information they require from OMERO.

However for my principle processing task direct access to OMERO is not required since I have not modified the command-line tool to read OMERO image files directly from the server. Instead I use the OMERO script to export images to a suitable file format which the analysis tool can read from common file access space. This set-up allows me to run parallel processing on large multi-dimensional images using multiple serial jobs in a compute farm.

The final part of this workflow is a script that runs after all other jobs are complete. This script loads results back into the OMERO server, collating individual planes into a result image.

This system does work but it was a pain to create. The final stage has checks to verify the individual jobs were a success and can e-mail error reports upon failure. It will also e-mail the user when it has finished so that they know how their jobs are doing.

Currently my compute farm is a single 16 core machine. This is because demand for the service is not very high and so I have not integrated into the University HPC cluster. The waiting time for nodes to come up on that system is too long due to the demand. It is better to run my own server (which is usually not very stressed) to ensure I get nodes immediately.

The obvious advantage of the farm system is that a user can select to process all the images in a dataset and the queue manager nicely processes through the work. They get e-mails when each image is done.

It would be interesting to see a better way to integrate OMERO into compute farms.

Alex

[jmoore]

Hi Alex,

we certainly agree that having a clean and simple integration between cluster resources and OMERO would be ideal, but it's an extremely complex thing to do in the general case while keeping everyone happy. Would it be possible to look over what you've done and see where existing code could be re-used in your situation and where your code could be reused by others?

~Josh

[a.herbert]

Hi Josh,

I can send you the code. It would be good to try and work on a reuseable framework.

Alex

[icaoberg]

that is how we trigger feature calculation in our omero.server. we modified the bridge so that everytime an image gets imported to omero.server, an omeroscript is generated and submitted to a queue.

so far our only problem is that we cannot directly login to omero from omero.shell, so we created a special account for this purpose and save the username/password in a place that can be read by such scripts.

[liz]

Dear all

Just wondering if OMERO currently supports the integration to the HPC cluster utilizing SGE/PBS.. without 'self-coding'? Was looking everywhere for info on this but to no avail. Perhaps I may have missed.

Would appreciate any pointers or clarification.

Many thanks!

[jmoore]

Hi Liz,

Just wondering if OMERO currently supports the integration to the HPC cluster utilizing SGE/PBS.. without 'self-coding'? Was looking everywhere for info on this but to no avail. Perhaps I may have missed.

No, you didn't miss anything. Though there have been a few people who have dabbled with an OMERO/cluster mix, there's no official support at the moment. If you can outline what you'd like to do specifically (# of nodes you have, # of images you're looking to process, etc), we may be able to make a concrete suggestion.

All the best,
~Josh.

[liz]

Thanks, josh for your reply.

We're trying to set up OMERO and see if we could utilize the current HPC setup we have (1+3 compute nodes connected to GPFS storage).

We also have 1 new-ish server (IBM PowerEdge T620 with 16GB RAM and about 2TB of storage) + 2 workstation (64-bit, 16GB RAM and 32-bit, 3GB RAM)

I read the recommended set up and I think we would probably be starting off on a smaller scale first.

But was just wondering if you could please advise on how we could utilize our current HPC + the new stuffs (if necessary)? Or at best, utilize the GPFS storage. What sort of architecture design should we deploy?

My test setup is currently down but I remember seeing somewhere that we could configure some node setting. But I didn't really understand that part if it is said that the cluster setup is not supported. Is that supposed to be linked?

Not too sure if that made sense, but please let me know if you require any further information.

Many thanks!

liz

[jmoore]

Hi Liz,

But was just wondering if you could please advise on how we could utilize our current HPC + the new stuffs (if necessary)? Or at best, utilize the GPFS storage. What sort of architecture design should we deploy?

It really depends on what you want to do:

• If you want to just make the most of the machines you have for a vanilla OMERO install, then likely splitting up the services (Java, PostgreSQL) between the machines will be the best way forward. (A)
• If you are planning on using OMERO.scripts, likely starting a separate processor on each would be more useful. (B)
• If you want high-availability, you can mount the same GPFS directory on at least 2 nodes and start the Java service on both. (C)
• If you plan on using some more compute-intensive operations, adding either a proper scheduler and/or MPI might be what you'll need/want.(D)

My test setup is currently down but I remember seeing somewhere that we could configure some node setting. But I didn't really understand that part if it is said that the cluster setup is not supported. Is that supposed to be linked?

The documentation for starting extra nodes etc is under https://www.openmicroscopy.org/site/support/omero5/sysadmins/grid.html which is mostly useful for B and partly C above. However, we're happy to add more depending on your use case(s).

Cheers,
~Josh