generating/attaching/retrieving a non-OMERO related file

[bhcho]

Hi All,

I'm trying to generate an output file from my Script in OMERO.Insight and attach it to a dataset.
Later from another Script, I also want to retrieve that output file, given the dataset ID.

Could anyone help me about this?
I'm trying to generate a pickle file that packs a complex data as the following python snippet

Code: Select all

# for generating a pickle file
import pickle
output = open('model.pkl','wb')
pickle.dump(final_model, output)  # final_model: a complex data variable
output.close()

# for retrieving the pickle file
import pickle
pkl_file = open('model.pkl','rb')
final_model2 = pickle.load(pkl_file)

BK

[wmoore]

Hi,

This is quite common behavior.
All the 'Figure' scripts E.g. 'Thumbnail Figure' save their output as attachments to E.g Image or Dataset.
In fact, you can do it with a single call: scriptUtil.uploadAndAttachFile(), documented here http://hudson.openmicroscopy.org.uk/job ... AttachFile

Below is some sample code from Thumbnail Figure script. See http://trac.openmicroscopy.org.uk/omero ... _Figure.py

Code: Select all

import omero.util.script_utils as scriptUtil
format = "image/png"

parent = gateway.getDataset(Id, False)    # parent can be omero.model.ImageI or DatasetI or ProjectI

output = "thumbnailFigure.png"   
figure.save(output, "PNG")        # simply writes to a file in the current directory

# uploads the file to the server, attaching it to the 'parent' Project/Dataset as an OriginalFile annotation,
# with the figLegend as the description. Returns the fileAnnotation object
fileAnnotation = scriptUtil.uploadAndAttachFile(queryService, updateService, rawFileStore, parent, output, format, figLegend)

client.setOutput("File_Annotation", robject(fileAnnotation))


If you want to be able to identify the file uniquely E.g. from your second script, you could give the file a meaningful namespace. E.g. 'omero.constants.metadata.NSMOVIE' (see the method docs above).

To download the file again, you can use the downloadFile(), also from the script_utils package.
http://hudson.openmicroscopy.org.uk/job ... wnloadFile
But you first need to retrieve your file - probably using the queryservice.

I just cobbled together this example, so I'm not sure it works - but hopefully you should be able to work the rest out for yourself.

Code: Select all

dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = %s" % (datasetId, yourNameSpace) , None)
                   
originalFile = dal.child.file
           
for file in dataset.
filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
print "Saved file at:", filePath
f = open(filePath)


Hope that helps,

Will

[bhcho]

Hi Will,

Thanks so much for the answer.
The attachment was successful, but for the retrieving, i have the following error.
I used the namespace of "omero.constants.metadata.CLASSIFIER" for the file.
and the dataset ID 452 clearly has the attached file of "final_model.pkl" when I look at the OMERO.Insight

Could you figure out which part has the problem with my code?
the code is

Code: Select all

            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = %s" % (ClassfierdatasetID, namesp_classfier) , None)

Code: Select all

        * Traceback (most recent call last):
        *   File "./script", line 399, in <module>
        *     runAsScript()
        *   File "./script", line 367, in runAsScript
        *     "fileAnn.ns = %s" % (ClassfierdatasetID, namesp_classfier) , None)
        *   File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_api_IQuery_ice.py", line 132, in findByQuery
        *     return _M_omero.api.IQuery._op_findByQuery.invoke(self, ((query, params), _ctx))
        * omero.QueryException: exception ::omero::QueryException
        * {
        *     serverStackTrace = ome.services.query.QueryException: Illegal query:select dal from DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER
        * Invalid path: 'omero.constants.metadata.CLASSIFIER' [select dal from ome.model.annotations.DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER]
        *       at ome.services.query.StringQuery.buildQuery(StringQuery.java:60)
        *       at ome.services.query.Query.doInHibernate(Query.java:222)
        *       at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
        *       at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
        *       at ome.logic.QueryImpl.execute(QueryImpl.java:141)
        *       at ome.logic.QueryImpl.findByQuery(QueryImpl.java:337)
        *       at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        *       at ome.security.basic.EventHandler.invoke(EventHandler.java:157)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:111)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        *       at $Proxy52.findByQuery(Unknown Source)
        *       at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        *       at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:83)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        *       at $Proxy52.findByQuery(Unknown Source)
        *       at sun.reflect.GeneratedMethodAccessor579.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
        *       at ome.services.throttling.Callback.run(Callback.java:56)
        *       at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        *       at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
        *       at ome.services.blitz.impl.QueryI.findByQuery_async(QueryI.java:92)
        *       at omero.api._IQueryTie.findByQuery_async(_IQueryTie.java:113)
        *       at omero.api._IQueryDisp.___findByQuery(_IQueryDisp.java:342)
        *       at omero.api._IQueryDisp.__dispatch(_IQueryDisp.java:508)
        *       at IceInternal.Incoming.invoke(Incoming.java:159)
        *       at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
        *       at Ice.ConnectionI.message(ConnectionI.java:972)
        *       at IceInternal.ThreadPool.run(ThreadPool.java:577)
        *       at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
        *       at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
        *
        *     serverExceptionClass = ome.services.query.QueryException
        *     message = Illegal query:select dal from DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER
        * Invalid path: 'omero.constants.metadata.CLASSIFIER' [select dal from ome.model.annotations.DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER]
        * }
        *


[jmoore]

BK,

in the formatting two apostrophes apparently got lost. Try:

Code: Select all

            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = '%s' " % (ClassfierdatasetID, namesp_classfier) , None)


In general, though, it is safer to use Parameters:

Code: Select all

            params = omero.sys.ParametersI()
            params.addLong("id", ClassifierdatasetID);
            params.addString("ns", namesp_classfier)
            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = :id and " \
                    "fileAnn.ns = :ns ", params)


Cheers,
~Josh

[bhcho]

Hi Josh,

Thanks, and now I got the following error.
my code is as follows.
I'm sorry about so many detailed questions.

BK

Code: Select all

            params = omero.sys.ParametersI()
            params.addLong("id", ClassfierdatasetID);
            params.addString("ns", namesp_classfier)

            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = :id and " \
                    "fileAnn.ns = :ns ", params)

            originalFile = dal.child.file
           
            filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
            print "Saved file at:", filePath
            f = open(filePath)

* filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/script_utils.py", line 245, in downloadFile
* fileSize = originalFile.getSize().getValue()
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_model_OriginalFileI.py", line 294, in getSize
* self.errorIfUnloaded()
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_model_OriginalFileI.py", line 47, in errorIfUnloaded
* raise _omero.UnloadedEntityException("Object unloaded:"+str(self))
* omero.UnloadedEntityException: Object unloaded:object #0 (::omero::model::OriginalFile)
* {
* _id = object #1 (::omero::RLong)
* {
* _val = 7026
* }
* _details = <nil>
* _loaded = False
* _version = <nil>
* _pixelsFileMapsSeq =
* {
* }
* _pixelsFileMapsLoaded = False
* _pixelsFileMapsCountPerOwner =
* {
* }
* _path = <nil>
* _size = <nil>
* _atime = <nil>
* _mtime = <nil>
* _ctime = <nil>
* _sha1 = <nil>
* _mimetype = <nil>
* _annotationLinksSeq =
* {
* }
* _annotationLinksLoaded = False
* _annotationLinksCountPerOwner =
* {
* }
* _name = <nil>
* }
*

[wmoore]

Hi,

I think you need to load (fetch) the file in the query.
Try adding "join fetch fileAnn.file" in this context...

Code: Select all

join fetch dal.child as fileAnn
join fetch fileAnn.file
join dal.parent as dataset


See if that works!

[bhcho]

Thanks Will and Josh,

It finally works!!!!

BK

[wmoore]

Final thought: You should probably choose a nicer namespace that would enable someone else getting info from your server to know what application etc the data came from. This avoids confusion with namespaces that are part of the OMERO clients' behavior etc.

Something like:

AppName.scriptName.Variable

Whatever seems sensible to you!

Will.

[bhcho]

Hi all,

I'd like to ask a follow-up question.
Currently, I'm retrieving the non-OMERO attachment file from a dataset by
(for a specific namespace)

Code: Select all

                params = omero.sys.ParametersI()
                params.addLong("id", My_datasetID);
                params.addString("ns", My_namesp)
                dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                        "join fetch dal.child as fileAnn " \
                        "join fetch fileAnn.file " \
                        "join dal.parent as dataset " \
                        "where dataset.id = :id and " \
                        "fileAnn.ns = :ns ", params)
                originalFile = dal.child.file
                filePath = scriptUtil.downloadFile(rawFileStore, originalFile)

1.
it seems like "originalFile" is a file object. but how can I get the file name?

2.
If there are multiple files with the same namespace, how can I get all the files at once (and their filenames)?

3.
how can I retrieve just one file with a specific file name?


thanks in advance,
BK

[jmoore]

Hi BK,

1. it seems like "originalFile" is a file object. but how can I get the file name?

omero.model.OriginalFile objects have a "name" and a "path" field. You may also want to use "mimetype".

2. If there are multiple files with the same namespace, how can I get all the files at once (and their filenames)?

Use something like:

Code: Select all

                        "join fetch fileAnn.file as file " \
                        "join dal.parent as dataset " \
                        "where dataset.id = :id and " \
                        "fileAnn.ns = :ns and file.name = :name", params)


3. how can I retrieve just one file with a specific file name?

What do you mean? findByQuery always returns on element.

thanks in advance,
BK

Cheers,
~Josh