command line importer fails

[helmerj]

Hi

After updating my OMERO test system from 4.2.2 to 4.3.4 I can import images using the GUI importer. But the import fails on the commandline:

1. I have created a project and inside the project two dataasets with ids 1 and 2.
2. using the latest importer (4.3.3) I try to import a set of images to the created dataset(s):

Code: Select all

./importer-cli -s 192.168.2.181 -u importer -w THE_PASSWORD -d 1 /home/helmerj/rails/metaxpress/transfer-script/input_all/20100825\ rbilly_A*.tif


This results in the following error:

Code: Select all

java.lang.RuntimeException: omero.SecurityViolation
    serverStackTrace = "ome.conditions.SecurityViolation: Cannot read ome.model.containers.Dataset:Id_1
                                at ome.security.basic.BasicACLVoter.throwLoadViolation(BasicACLVoter.java:112)
                                at ome.security.CompositeACLVoter.throwLoadViolation(CompositeACLVoter.java:86)
                                at ome.security.ACLEventListener.onPostLoad(ACLEventListener.java:104)
                                at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:250)
                                at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:898)
                                at org.hibernate.loader.Loader.doQuery(Loader.java:773)
                                at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
                                at org.hibernate.loader.Loader.loadEntity(Loader.java:1953)
                                at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86)
                                at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76)
                                at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3270)
                                at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496)
                                at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477)
                                at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
                                at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:147)
                                at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1082)
                                at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:1020)
                                at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:176)
                                at org.hibernate.Hibernate.initialize(Hibernate.java:428)
                                at ome.logic.QueryImpl$1.doInHibernate(QueryImpl.java:174)
                                at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
                                at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
                                at ome.logic.QueryImpl.get(QueryImpl.java:157)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                                at ome.security.basic.EventHandler.invoke(EventHandler.java:150)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
                                at $Proxy74.get(Unknown Source)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                                at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:83)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
                                at $Proxy74.get(Unknown Source)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
                                at ome.services.throttling.Callback.run(Callback.java:56)
                                at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
                                at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
                                at ome.services.blitz.impl.QueryI.get_async(QueryI.java:110)
                                at omero.api._IQueryTie.get_async(_IQueryTie.java:127)
                                at omero.api._IQueryDisp.___get(_IQueryDisp.java:170)
                                at omero.api._IQueryDisp.__dispatch(_IQueryDisp.java:516)
                                at IceInternal.Incoming.invoke(Incoming.java:159)
                                at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
                                at Ice.ConnectionI.message(ConnectionI.java:972)
                                at IceInternal.ThreadPool.run(ThreadPool.java:577)
                                at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
                                at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
                        "
    serverExceptionClass = "ome.conditions.SecurityViolation"
    message = "Cannot read ome.model.containers.Dataset:Id_1"
        at ome.formats.OMEROMetadataStoreClient.getTarget(OMEROMetadataStoreClient.java:2197)
        at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:278)
        at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:141)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:395)
Caused by: omero.SecurityViolation
    serverStackTrace = "ome.conditions.SecurityViolation: Cannot read ome.model.containers.Dataset:Id_1
                                at ome.security.basic.BasicACLVoter.throwLoadViolation(BasicACLVoter.java:112)
                                at ome.security.CompositeACLVoter.throwLoadViolation(CompositeACLVoter.java:86)
                                at ome.security.ACLEventListener.onPostLoad(ACLEventListener.java:104)
                                at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:250)
                                at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:898)
                                at org.hibernate.loader.Loader.doQuery(Loader.java:773)
                                at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
                                at org.hibernate.loader.Loader.loadEntity(Loader.java:1953)
                                at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86)
                                at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76)
                                at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3270)
                                at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496)
                                at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477)
                                at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
                                at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:147)
                                at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1082)
                                at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:1020)
                                at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:176)
                                at org.hibernate.Hibernate.initialize(Hibernate.java:428)
                                at ome.logic.QueryImpl$1.doInHibernate(QueryImpl.java:174)
                                at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
                                at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
                                at ome.logic.QueryImpl.get(QueryImpl.java:157)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                                at ome.security.basic.EventHandler.invoke(EventHandler.java:150)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
                                at $Proxy74.get(Unknown Source)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                                at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:83)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
                                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
                                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
                                at $Proxy74.get(Unknown Source)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:597)
                                at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
                                at ome.services.throttling.Callback.run(Callback.java:56)
                                at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
                                at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
                                at ome.services.blitz.impl.QueryI.get_async(QueryI.java:110)
                                at omero.api._IQueryTie.get_async(_IQueryTie.java:127)
                                at omero.api._IQueryDisp.___get(_IQueryDisp.java:170)
                                at omero.api._IQueryDisp.__dispatch(_IQueryDisp.java:516)
                                at IceInternal.Incoming.invoke(Incoming.java:159)
                                at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
                                at Ice.ConnectionI.message(ConnectionI.java:972)
                                at IceInternal.ThreadPool.run(ThreadPool.java:577)
                                at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
                                at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
                        "
    serverExceptionClass = "ome.conditions.SecurityViolation"
    message = "Cannot read ome.model.containers.Dataset:Id_1"
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at java.lang.Class.newInstance0(Class.java:372)
        at java.lang.Class.newInstance(Class.java:325)
        at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
        at IceInternal.BasicStream.throwException(BasicStream.java:1632)
        at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
        at omero.api._IQueryDelM.get(_IQueryDelM.java:547)
        at omero.api.IQueryPrxHelper.get(IQueryPrxHelper.java:660)
        at omero.api.IQueryPrxHelper.get(IQueryPrxHelper.java:632)
        at ome.formats.OMEROMetadataStoreClient.getTarget(OMEROMetadataStoreClient.java:2193)
        ... 3 more



Server IP, User name, and password if copied from this command and used in the GUI do work and allow me to import the same files.

Any help on this would be greatly appreciated!

Cheers Juergen

[cxallan]

Dataset:1 is not in the default group that the command line importer is logging you in to so you cannot access the Dataset. You've got a couple options:

• Use bin/omero login and bin/omero import to get you in to the right group
• Set your default group to the same group that contains Dataset:1

[cxallan]

First off the OME team agrees with you and that's what's currently slated for 4.4.0's group based permissions changes which include chgrp, cross group querying, etc.

Dataset:1 is not in the default group that the command line importer is logging you in to so you cannot access the Dataset. You've got a couple options:

• Use bin/omero login and bin/omero import to get you in to the right group
• Set your default group to the same group that contains Dataset:1

Hi

thanks for the quick reply and the workaround, but that does not make much sense...

1. I can create a dataset as a user, so if I provide authentication with that username and password I should have access to it!

Agreed, however that is not how group permissions containment has worked since OMERO 4.2.0.

2. I can use the gui importer to import images into it so that application has no issue with the dataset belonging not to the default group

It has exactly the same issue. If you switched groups you would not even be able to see the Dataset in the GUI for the purposes of selecting it for import.

3. I cannot use the commandline importer to import to the same dataset as I have a group issue?

You can absolutely use the command line importer. You just must follow the rules of the permissions system. If Dataset:1 is in Group:1, you must log in to Group:1 otherwise you will not be able to link imported images to it.

I think the commandline importer should be changed to be in line with the gui importer's behaviour. Having differences between tools is not really recommendable application behaviour now is it? Plus iof this is a desired application behaviour it shoud be written in bold letters across the documentation... Which is might actually do I have not checked.

You can read all about the permissions system here:

• http://www.openmicroscopy.org/site/support/omero4/server/permissions/
• http://trac.openmicroscopy.org.uk/ome/wiki/Delete

For what it's worth these exact same restrictions were present from 4.2.0, were outlined in the release e-mails, are included in the upgrade instructions for 4.2.0 and are further outlined on the features page:

• http://www.openmicroscopy.org/site/products/feature-list

These include amongst the documentation outlined above, several movies outlining the permissions system in a user friendly manner.

[helmerj]

Hi

thanks for the quick reply and the workaround, but that does not make much sense. So I can only import into datasets that are in the default system group when using the commandline importer but the gui version can import into any project and dataset?

Having these inconsistancies between tools is very very confusing application behaviour.

I do I enjoy a fine tuned persission system as any other, so if this is a desired application behaviour it shoud be written in bold letters across the documentation... Which it might actually does, I have not checked

Oh well try to make it work...

Cheers Juergen

[helmerj]

You can absolutely use the command line importer. You just must follow the rules of the permissions system. If Dataset:1 is in Group:1, you must log in to Group:1 otherwise you will not be able to link imported images to it.

But that is the confusing thing. I am using an account and it is in both existing groups on my server, I have system (default, I suppose) and CSIR. My user importer is in both groups otherise I could not have created a data set when being logged in as that user. I have created a dataset in the CSIR group but still have no access to it although my user is in the group!

Cheers Juergen

[cxallan]

You can absolutely use the command line importer. You just must follow the rules of the permissions system. If Dataset:1 is in Group:1, you must log in to Group:1 otherwise you will not be able to link imported images to it.

But that is the confusing thing. I am using an account and it is in both existing groups on my server, I have system (default, I suppose) and CSIR. My user importer is in both groups otherise I could not have created a data set when being logged in as that user. I have created a dataset in the CSIR group but still have no access to it although my user is in the group!

Cheers Juergen

That's just simply not how it works. You can only be logged in to one group at a time. If you created the Dataset in the CSIR group you must be logged in to that group to link imported images to it.

[helmerj]

That's just simply not how it works. You can only be logged in to one group at a time. If you created the Dataset in the CSIR group you must be logged in to that group to link imported images to it.

hi!

So shouldn't I be able top specify a group on the commandline if this behaviour is "just like that"?

Because with this behaviour I can only use the importer on the commandline if

1. a user is in a single group
2. the user is in multiple group but the dataset is created in the default group.

That is not very flexible...
Cheers Juergen

[cxallan]

That's just simply not how it works. You can only be logged in to one group at a time. If you created the Dataset in the CSIR group you must be logged in to that group to link imported images to it.

hi!

So shouldn't I be able top specify a group on the commandline if this behaviour is "just like that"?

You can, via the mechanisms that I outlined with bin/omero login and bin/omero import, these wrap the command line import with all these pieces of functionality. If you are set on using importer-cli, then create the session however you wish and use the -k command line argument to pass a logged in session through.

Because with this behaviour I can only use the importer on the commandline if

1. a user is in a single group
2. the user is in multiple group but the dataset is created in the default group.

That is not very flexible...
Cheers Juergen

Again, that's just simply not true. I've given you at least two ways to work in that manner if that's what you want to do. The command line importer does not replicate all functionality of OMERO.insight or OMERO.importer, it is designed to be used in tandem with bin/omero or with your own scripts.

Here's a concrete example:

Code: Select all

bin/omero -s <hostname> -u <username> -g <group_id> -w <password> login
bin/omero import <your_command_line_importer_arguments>
...
# To examine the sessions you have available, get session keys, etc. ...
bin/omero sessions list
...
# To change your active group...
bin/omero sessions group ...
...
# If you insist on using importer-cli then login via bin/omero login as above and...
importer-cli -k <session_key_from_sessions_list> <your_command_line_importer_arguments>
# The above even allows you to do things like have session re-use, etc.


If you want to do something more exotic then be concrete with an example and we'll be happy to help you.

[helmerj]

That's just simply not how it works. You can only be logged in to one group at a time. If you created the Dataset in the CSIR group you must be logged in to that group to link imported images to it.

hi!

So shouldn't I be able top specify a group on the commandline if this behaviour is "just like that"?

You can, via the mechanisms that I outlined with bin/omero login and bin/omero import, these wrap the command line import with all these pieces of functionality. If you are set on using importer-cli, then create the session however you wish and use the -k command line argument to pass a logged in session through.

Because with this behaviour I can only use the importer on the commandline if

1. a user is in a single group
2. the user is in multiple group but the dataset is created in the default group.

That is not very flexible...
Cheers Juergen

Again, that's just simply not true. I've given you at least two ways to work in that manner if that's what you want to do. The command line importer does not replicate all functionality of OMERO.insight or OMERO.importer, it is designed to be used in tandem with bin/omero or with your own scripts.

Here's a concrete example:

Code: Select all

bin/omero -s <hostname> -u <username> -g <group_id> -w <password> login
bin/omero import <your_command_line_importer_arguments>
...
# To examine the sessions you have available, get session keys, etc. ...
bin/omero sessions list
...
# To change your active group...
bin/omero sessions group ...
...
# If you insist on using importer-cli then login via bin/omero login as above and...
importer-cli -k <session_key_from_sessions_list> <your_command_line_importer_arguments>
# The above even allows you to do things like have session re-use, etc.


If you want to do something more exotic then be concrete with an example and we'll be happy to help you.

Hi!

Point taken, that works fine for my needs.

Thanks
Juergen

[cxallan]

Awesome.

When we have more of the "make this nicer" functionality present in the development line for inclusion in 4.4.0 we'll definitely be interested in your feedback.