Unable to run the script

[bhcho]

Hi all,

I have problem with my script.
Very often time, I can't launch my script, which I can launch again right after restarting the server.

the error message is

"Unable to run the script"
org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data. Cannot run the script

more details are

java.lang.Exception: java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot run the script.
at org.openmicroscopy.shoola.env.data.views.ProcessBatchCall.doStep(ProcessBatchCall.java:101)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot run the script.
at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:802)
at org.openmicroscopy.shoola.env.data.OMEROGateway.runScript(OMEROGateway.java:6672)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.runScript(OmeroImageServiceImpl.java:902)
at org.openmicroscopy.shoola.env.data.views.calls.ScriptRunner$1.initialize(ScriptRunner.java:74)
at org.openmicroscopy.shoola.env.data.views.ProcessBatchCall.doStep(ProcessBatchCall.java:99)
... 6 more
Caused by: org.openmicroscopy.shoola.env.data.ProcessException: Cannot run script with ID:7655
at org.openmicroscopy.shoola.env.data.OMEROGateway.runScript(OMEROGateway.java:515)
at org.openmicroscopy.shoola.env.data.OMEROGateway.runScript(OMEROGateway.java:6670)
... 9 more
Caused by: omero.InternalException
serverStackTrace = "Traceback (most recent call last):
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/decorators.py", line 61, in exc_handler
rv = func(*args, **kwargs)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/processor.py", line 819, in processJob
prx, process = self.process(client, session, job, current, params, iskill = True)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/decorators.py", line 28, in handler
return func(*args, **kwargs)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/processor.py", line 858, in process
process = ProcessI(self.ctx, "python", properties, params, iskill, omero_home = self.omero_home)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/processor.py", line 117, in __init__
self.make_files()
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/processor.py", line 140, in make_files
self.dir = create_path("process", ".dir", folder = True)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/temp_files.py", line 338, in create_path
return manager.create_path(prefix, suffix, folder = folder)
File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/temp_files.py", line 266, in create_path
name = tempfile.mkdtemp(prefix = prefix, suffix = suffix, dir = self.dir)
File "/usr/lib64/python2.5/tempfile.py", line 328, in mkdtemp
_os.mkdir(file, 0700)
OSError: [Errno 13] Permission denied: '/afs/cs.cmu.edu/user/bhcho/omero/tmp/omero_bhcho/31342/processBF51VO.dir'
"
serverExceptionClass = ""
message = "Internal exception"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._IScriptDelM.runScript(_IScriptDelM.java:508)
at omero.api.IScriptPrxHelper.runScript(IScriptPrxHelper.java:662)
at omero.api.IScriptPrxHelper.runScript(IScriptPrxHelper.java:634)
at org.openmicroscopy.shoola.env.data.OMEROGateway.runScript(OMEROGateway.java:511)
... 10 more

at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:159)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:246)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:218)
at org.openmicroscopy.shoola.env.ui.ActivityComponent.showException(ActivityComponent.java:480)
at org.openmicroscopy.shoola.env.ui.ActivityComponent.actionPerformed(ActivityComponent.java:957)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

is there any problem with my OMERO setting?

[jmoore]

BK,

I think this is still a problem with your OMERO_TEMPDIR:

OSError: [Errno 13] Permission denied: '/afs/cs.cmu.edu/user/bhcho/omero/tmp/omero_bhcho/31342/processBF51VO.dir'

My guess is you see this only sporadically because sometimes your Kerberos credentials are present for the process which is running OMERO, and at other times they are not. I know you have previously set OMERO_TEMPDIR as described under TempFileManager, but you will either need to make that permanent in your environment via .bashrc, .profile, or similar or remember to set it before everything invocation to "bin/omero admin ...".

Hope that helps,
~Josh

[bhcho]

Thanks Josh,

After I set OMERO_TEMPDIR as environment variable, I have not got the error message like

"Unable to run the script"
org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data. Cannot run the script

But I get inconsistent results when I run the script.
like the attached image, those two messages are from the same script.
Sometimes I get what I expected, but some other times, I get "No result returned"

Moreover, when I click "Error" from the output message, it says,

No handlers could be found for logger "omero.util.TempFileManager"

Because I don't write any file in the script, I don't understand why I have this error.
And that error is consistently happening whether I get the correct result or not.

FYI, the script is

Code: Select all

import sys
import omero
import omero.clients
from omero.rtypes import *
from omero.rtypes import wrap, rstring, rlong, rint, robject
import omero.scripts as scripts
import omero.constants
from omero.rtypes import *
import omero_api_Gateway_ice    # see http://tinyurl.com/icebuserror
import omero.util.script_utils as scriptUtil
import os
from collections import defaultdict
from omero.util.temp_files import create_path



def SearchImgsbyFullText(client, value):
    """
    Search images within the client account by the "value" of the attached text (xml) file
   
    @param client         The OMERO client
    @param value          query value
    """
    search = client.sf.createSearchService()
    search.onlyType("Image")
    search.byFullText(value)
    ids = [x.id.val for x in search.results()]
    return ids

def list_intersection(list1, list2):
    bag = defaultdict(int)
    for elt in list1:
        bag[elt] += 1

    result = []
    for elt in list2:
        if elt in bag:
            # remove elt from bag, making sure
            # that bag counts are kept positive
            if bag[elt] == 1:
                del bag[elt]
            else:
                bag[elt] -= 1
            result.append(elt)

    return result


def runAsScript():

    client = scripts.client('Search_by_Context.py', """Search Images by Context Meta Data""",

    # 1. Protocol
    scripts.String("Fixation", optional=True, grouping="1.1",
        description="Fixation"),
    scripts.String("Permeabilization", optional=True, grouping="1.2",
        description="Permeabilization"),
    scripts.String("Substrate", optional=True, grouping="1.3",
        description="Substrate"),                       
    scripts.String("Title", optional=True, grouping="1.4",
        description="Title"),                   

    # 2. Cell Type
    scripts.String("Cell_Type_Name", optional=True, grouping="2.1",
        description="Cell Type Name"),
    scripts.String("Organism", optional=True, grouping="2.2",
        description="Organism"),
    scripts.String("atcc_number", optional=True, grouping="2.3",
        description="atcc number"),                       
    scripts.String("cell_mesh_heading", optional=True, grouping="2.4",
        description="cell_mesh_heading"), 
    scripts.String("cell_mesh_tree_number", optional=True, grouping="2.5",
        description="cell_mesh_tree_number"),

    version = "4.2.1",
    authors = ["Baek Hwan Cho", "Murphy Lab"],
    institutions = ["Carnegie Mellon University"],
    contact = "bhcho@cmu.edu",
    )

    try:
        session = client.getSession()

##    # create the services we're going to need
##    metadataService = session.getMetadataService()
##        queryService = session.getQueryService()
##        updateService = session.getUpdateService()
##        rawFileStore = session.createRawFileStore()




        # process the list of args above.
        parameterMap = {}
        for key in client.getInputKeys():
            if client.getInput(key):
                parameterMap[key] = client.getInput(key).getValue()

        query_queue = []
       
        try:
            Fixation = parameterMap["Fixation"]
            query_string = "Slide.Protocol.Fixation:" + str(Fixation)
            query_queue.append(query_string)
        except:
            Fixation = None

        try:   
            Permeabilization = parameterMap["Permeabilization"]
            query_string = "Slide.Protocol.Permeabilization:" + str(Permeabilization)
            query_queue.append(query_string)
        except:
            Permeabilization = None

        try:   
            Substrate = parameterMap["Substrate"]
            query_string = "Slide.Protocol.Substrate:" + str(Substrate)
            query_queue.append(query_string)
        except:
            Substrate = None

        try:
            Title = parameterMap["Title"]
            query_string = "Slide.Protocol.Title:" + str(Title)
            query_queue.append(query_string)
        except:
            Title = None

        searched_img_IDs = []
        search = client.sf.createSearchService()
        search.onlyType("Image")
        for q in query_queue:
            search.byFullText(q)
            try:
                ids = [x.id.val for x in search.results()]
                searched_img_IDs.append(ids)
            except:
                tmp=[]

           

        # search the common IDs
        final_searched_img_IDs = []
        if searched_img_IDs != []:
            id_list1 = searched_img_IDs[0]

            for id_list2 in searched_img_IDs:
                result = list_intersection(id_list1, id_list2)
                id_list1 = result
                   
            final_searched_img_IDs = id_list1
       
        txt =  " ".join(map(str,final_searched_img_IDs))       
        client.setOutput("Message", rstring("Script Ran OK.  list = %s" % txt))

    finally:
        client.closeSession()

if __name__ == "__main__":
    runAsScript()


[jmoore]

After I set OMERO_TEMPDIR as environment variable, I have not got the error message like

"Unable to run the script"
org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data. Cannot run the script

But I get inconsistent results when I run the script.
like the attached image, those two messages are from the same script.
Sometimes I get what I expected, but some other times, I get "No result returned"

This seems very odd. If possible, please stop your server, delete or backup the logs under var/log, and then restart with OMERO_TEMPDIR set, so we can try to figure out what's happening.

Moreover, when I click "Error" from the output message, it says,

No handlers could be found for logger "omero.util.TempFileManager"

Because I don't write any file in the script, I don't understand why I have this error.
And that error is consistently happening whether I get the correct result or not.

This isn't too suprising. When you import temp_files in your script, the module tries to log debug messages and this warning gets sent to standard out. You can try adding:

Code: Select all

import logging
logging.basicConfig()


to make this message go away. Alternatively, you can remove the import from your code, since you don't seem to use it.

Cheers,
~Josh

[bhcho]

Hi Josh,

I solved the "temp_files" problem by removing the import.

But I still get the same "No result returned" message like the picture in the previous post.
And I attached the log files here.

Thanks in advance,
BK

[jmoore]

Hi BK,

in Processor-0.log, I see the following:

Code: Select all

2011-02-25 15:58:18,694 INFO  [                            omero.remote] (Dummy-5   )  Meth: ProcessorI.willAccept
2011-02-25 15:58:18,739 INFO  [                            omero.remote] (Dummy-5   )  Rslt: True
2011-02-25 15:58:18,884 INFO  [                            omero.remote] (Dummy-6   )  Meth: ProcessorI.processJob
2011-02-25 15:58:18,885 INFO  [              omero.processor.ProcessorI] (Dummy-6   ) processJob: Session = a76ba8b7-ff77-431d-b829-2611aa4ffda8, JobId = 2753
2011-02-25 15:58:18,969 INFO  [                omero.processor.ProcessI] (Dummy-6   ) Created a76ba8b7-ff77-431d-b829-2611aa4ffda8 in /usr0/local/omero.server/OMERO.server-Beta-4.2.1/tmp/omero/tmp/omero_bhcho/5752/processlOL_uh.dir
2011-02-25 15:58:18,978 INFO  [              omero.processor.ProcessorI] (Dummy-6   ) Downloaded file: 8401
2011-02-25 15:58:18,984 INFO  [                omero.processor.ProcessI] (Dummy-6   ) <proc:6706,rc=-,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Activated
2011-02-25 15:58:19,026 INFO  [                            omero.remote] (Dummy-6   )  Rslt: A9698F94-266F-4D73-BE48-FD4F64EA9678 -t @ Processor-0.ProcessorAdapter
2011-02-25 15:58:19,029 INFO  [                            omero.remote] (Dummy-7   )  Meth: ProcessI.registerCallback
2011-02-25 15:58:19,029 INFO  [                            omero.remote] (Dummy-7   )  Rslt: None
2011-02-25 15:58:19,039 INFO  [                            omero.remote] (Dummy-10  )  Meth: ProcessI.registerCallback
2011-02-25 15:58:19,039 INFO  [                            omero.remote] (Dummy-10  )  Rslt: None
2011-02-25 15:58:19,042 INFO  [                            omero.remote] (Dummy-10  )  Meth: ProcessI.poll
2011-02-25 15:58:19,042 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=-,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Polling
2011-02-25 15:58:19,042 INFO  [                            omero.remote] (Dummy-10  )  Rslt: None
2011-02-25 15:58:19,546 INFO  [                            omero.remote] (Dummy-10  )  Meth: ProcessI.poll
2011-02-25 15:58:19,546 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Polling
2011-02-25 15:58:19,546 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Deactivating
2011-02-25 15:58:19,547 INFO  [                            omero.remote] (Dummy-10  )  Meth: ProcessI.shutdown
2011-02-25 15:58:19,547 INFO  [                            omero.remote] (Dummy-10  )  Rslt: None
2011-02-25 15:58:19,641 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Changed job status from Running to Finished
2011-02-25 15:58:19,645 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : No stdout
2011-02-25 15:58:19,645 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : No stderr
2011-02-25 15:58:19,650 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Lived 0s. Deactivation took 0s.
2011-02-25 15:58:19,650 INFO  [                omero.processor.ProcessI] (Dummy-10  ) <proc:6706,rc=0,uuid=a76ba8b7-ff77-431d-b829-2611aa4ffda8> : Callback processFinished
2011-02-25 15:58:19,651 INFO  [                            omero.remote] (Dummy-10  )  Rslt: object #0 (::omero::RInt)


Can you confirm that the original file 8401 is the script you are trying to launch? From the log, it looks like it ran successfully, but produced no stdout or stderr. Could you perhaps try adding some logging? You'll then be able to view from Insight what is printed.

Cheers,
~Josh

[bhcho]

Hi J,

Yes, the original file 8401 was the script I was trying to launch.
Could you tell me how to add some logging?
is it like the following?

Code: Select all

import logging
logging.basicConfig()

can I just use the second line for once in the beginning of my script?

BK

[jmoore]

Hi BK,

that's not necessary. Anything you print to stdout will show up in the log:

Code: Select all

...
print "Searching for %s",  search_key
...
print "Found list1: %s", ", ".join(list1)
...


Cheers,
~Josh

[bhcho]

I printed out the list.

when I run the script, the same thing happen. Sometimes "No results returned" and some other times correct results are shown.

Even though "No results returned" happened, I now can click "Info" and I download it (I could not directly view it from Insight, even though there is "View" option). when I look at it, it says the founded IDs. It means the script run successfully, but no output passed to the "output message box".

Another strange thing is now the Error message happens again
"No handlers could be found for logger "omero.util.TempFileManager"


Best,
BK

[jmoore]

Can you attach the new script, Processor-0.log with output from a successful ans well as a failed run, and examples of the output? Somehow we need to correlate the occurrences of "No results found" with something in the logs/output.

~Josh.