PBS queue for OMERO shell python jobs

[bhcho]

Hi,

I'd like to submit jobs into queue using PBS for my python code.
Normally, I first starts an IPython interpreter session by (bin/omero shell) and run my code within the session by

Code: Select all

In [1]: execfile('mycode.py')

But I need to be online to complete my job this way.

When I tried

Code: Select all

$ python mycode.py

then, I cannot import omero.

So, is there anyway to run my code without starting the IPython session (bin/omero shell)?

I tried something like

Code: Select all

$bin/omero shell mycode.py

but it does not work.

Thanks in advance,
BK

[jmoore]

Hi BK,

Probably your best bet is to pass a session key to your script via qsub. First, login locally:

Code: Select all

$ bin/omero login USER@HOST
Previous session expired for USER on HOST:4064
Password:
Created session d89d42c5-b50e-4294-8b2b-ebf1b4778fd8 (USER@HOST:4064). Idle timeout: 10.0 min. Current group: GROUP


Then submit with a qsub variable list:

Code: Select all

OMERO_SESSION=d89d42c5-b50e-4294-8b2b-ebf1b4778fd8 qsub -v OMERO_SESSION ...


In your script, then, you can reference $OMERO_SESSION.

Alternatively, you can stage-in the active sessions file:

Code: Select all

$ bin/omero sessions file
/Users/YOU/omero/sessions/HOST/USER/d89d42c5-b50e-4294-8b2b-ebf1b4778fd8
$ cat `bin/omero sessions file`
omero.sess=d89d42c5-b50e-4294-8b2b-ebf1b4778fd8
omero.user=USER
omero.host=HOST


But that may only be practical if you are already staging files in or out: http://www.clusterresources.com/torquedocs/6.3filestaging.shtml

Cheers,
~Josh

[bhcho]

Thanks Josh,

Actually, it could be simply resolved by adding the OMERO_DIST/lib/python folder into the python path such as

Code: Select all

import sys
Path = '/OMERO_DIST/lib/python/'
sys.path.append(Path)

but when I submit multiple jobs, each of which connects to the server with the same account and retrieve data from the same table,
I got the following error.

Traceback (most recent call last):
File "test__pattern_1.py", line 108, in <module>
data, message = contentDB.retrieveDB(conn, 'slf33')
File "/xxx/lib/python/contentDB.py", line 690, in retrieveDB
values = table.read(chunk_col[col], chunk_row[row][0], chunk_row[row][-1]+1)
File "/xxx/lib/python/omero_Tables_ice.py", line 699, in read
return _M_omero.grid.Table._op_read.invoke(self, ((colNumbers, start, stop), _ctx))
Ice.UnknownLocalException: exception ::Ice::UnknownLocalException
{
unknown = TcpTransceiver.cpp:158: Ice::ConnectionLostException:
connection lost: recv() returned zero
}

Do you have any idea with this error message?

BK

[jmoore]

That looks like the session is being closed at some point. Could you show us your code? One option is to run in the controlling terminal (where you run qsub): bin/omero sessions keepalive

Code: Select all

$ bin/omero sessions keepalive
Using session d342b8d5-adca-4f29-acd4-bb5c860f79e8 (USER@HOST:4064). Idle timeout: 10.0 min. Current group: GROUP
Running keep alive every 60 seconds
Press enter to cancel.


but this will depend on why the session is being closed.

Cheers,
~Josh