image import python script

[bhcho]

Hi all,

I want to import lots of images by python script.

1.
Could anyone show me a snippet of how to import images by python script?

2.
Is it possible to specify the dataset ID when I import an image using the script?

3.
does this way to import images calls my custom Bridge (extensions.jar)?

Best,
BK

[icaoberg]

Dr. Cho

1. To import an image from omero import use the following line

Code: Select all

omero -s localhost -u "omero" -w "omero" -p 4064 import -d 1601 001_1_001.ome.tif


2. In the above example the option -d 1601 tells the code to import it to dataset 1601

3. It calls the same bridge.

Ivan

Hi all,

I want to import lots of images by python script.

1.
Could anyone show me a snippet of how to import images by python script?

2.
Is it possible to specify the dataset ID when I import an image using the script?

3.
does this way to import images calls my custom Bridge (extensions.jar)?

Best,
BK

[bhcho]

Thanks Ivan

[bhcho]

I have follow-up questions.

1.
Does this connection have a timeout?
if no, does it mean that the connection can run forever?
if yes, how long is the default timeout and how can I control it (by another option)?

2.
After several successful trials with the command above, i got the following outofmemory error.
Could anyone tell me why this happens?

Using session 7b5da05e-05d3-454a-8d0e-fa3c5077e26a (demo@localhost). Idle timeout: 10.0 min. Current group: Researchers
2011-11-09 12:16:41,489 0 [ main] INFO ome.formats.importer.ImportConfig - OMERO Version: Beta4.2.1-r8614-Beta4.2-b41
2011-11-09 12:16:42,468 979 [ main] WARN ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to Beta-4.3.3 See http://trac.openmicroscopy.org.uk/omero for the latest version
2011-11-09 12:16:42,478 989 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 4.2.1-DEV revision: 7178 date: 9 November 2010
2011-11-09 12:16:42,493 1004 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2011-11-09 12:16:42,879 1390 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2011-11-09 12:16:43,964 2475 [ main] INFO ome.formats.importer.ImportCandidates - 2 file(s) parsed into 2 group(s) with 2 call(s) to setId in 1078ms. (1085ms total) [0 unknowns]
2011-11-09 12:16:44,036 2547 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2011-11-09 12:16:44,753 3264 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2011-11-09 12:16:44,793 3304 [ main] ERROR formats.importer.cli.CommandLineImporter - Error during import process.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at IceInternal.ThreadPool.<init>(ThreadPool.java:91)
at Ice.ObjectAdapterI.<init>(ObjectAdapterI.java:823)
at IceInternal.ObjectAdapterFactory.createObjectAdapter(ObjectAdapterFactory.java:183)
at Ice.CommunicatorI.createObjectAdapter(CommunicatorI.java:71)
at omero.client.init(client.java:364)
at omero.client.<init>(client.java:255)
at omero.client.createClient(client.java:415)
at ome.formats.OMEROMetadataStoreClient.unsecure(OMEROMetadataStoreClient.java:587)
at ome.formats.OMEROMetadataStoreClient.initialize(OMEROMetadataStoreClient.java:552)
at ome.formats.importer.ImportConfig.createStore(ImportConfig.java:292)
at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:88)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:365)

Best,
BK

[bhcho]

I have another type of error when I import images via the script.

Code: Select all

Created session 0a7a9e0a-c2fa-47ae-a428-18d803c41848 (performance@localhost). Idle timeout: 10.0 min. Current group: Researchers
2011-11-19 21:00:05,783 0          [     main] INFO          ome.formats.importer.ImportConfig  - OMERO Version: Beta4.2.1-r8614-Beta4.2-b41
2011-11-19 21:00:09,166 3383      [     main] WARN                    ome.system.UpgradeCheck  - UPGRADE AVAILABLE:Please upgrade to Beta-4.3.3 See http://trac.openmicrosc$
2011-11-19 21:00:09,175 3392      [     main] INFO          ome.formats.importer.ImportConfig  - Bioformats version: 4.2.1-DEV revision: 7178 date: 9 November 2010
2011-11-19 21:00:09,187 3404      [     main] INFO   formats.importer.cli.CommandLineImporter  - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2011-11-19 21:00:09,571 3788      [     main] INFO     ome.formats.importer.ImportCandidates  - Depth: 4 Metadata Level: MINIMUM
2011-11-19 21:00:22,475 16692      [     main] INFO     ome.formats.importer.ImportCandidates  - 400 file(s) parsed into 400 group(s) with 400 call(s) to setId in 12311ms.$
2011-11-19 21:00:22,542 16759      [     main] INFO      ome.formats.OMEROMetadataStoreClient  - Attempting initial SSL connection to localhost:4064
2011-11-19 21:00:23,226 17443      [     main] INFO      ome.formats.OMEROMetadataStoreClient  - Insecure connection requested, falling back
2011-11-19 21:00:23,258 17475      [     main] ERROR  formats.importer.cli.CommandLineImporter  - Error during import process.
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at IceInternal.ThreadPool.<init>(ThreadPool.java:91)
        at Ice.ObjectAdapterI.<init>(ObjectAdapterI.java:823)
        at IceInternal.ObjectAdapterFactory.createObjectAdapter(ObjectAdapterFactory.java:183)
        at Ice.CommunicatorI.createObjectAdapter(CommunicatorI.java:71)
        at omero.client.init(client.java:364)
        at omero.client.<init>(client.java:255)
        at omero.client.createClient(client.java:415)
        at ome.formats.OMEROMetadataStoreClient.unsecure(OMEROMetadataStoreClient.java:587)
        at ome.formats.OMEROMetadataStoreClient.initialize(OMEROMetadataStoreClient.java:552)
        at ome.formats.importer.ImportConfig.createStore(ImportConfig.java:292)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:88)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:365)

[jmoore]

Hi BK,

interesting. I've personally never seen an OOM so early in the import process. What JVM are you using and on what operating system? What type of files are these and how large?

To try to get past this point, you'll either need to increase the memory for the process:

Code: Select all

export JAVA_OPTS=-Xmx1000M


or loop over all the files in Python as opposed to passing them all to the importer at once. However, since I'm not sure what's causing the OOM, it's not clear if this will fix the situation. I've filed a ticket, and anything you can tell us about these files would be greatly appreciated.

Cheers,
~Josh

[bhcho]

Hi Josh,

1.
for Java,

java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)

for OS version,

Linux version 2.6.28.4-002-STD64 (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP PREEMPT Fri Jul 3 20:28:31 EDT 2009

2.
I will try to use the JAVA memory option, and let you know if I still have the same problem.

3.
for looping over all the files in Python, do you mean I call the shell script for each file from python?

Code: Select all

omero -s localhost -u "omero" -w "omero" -p 4064 import -d 1601 001_1_001.ome.tif

I think this will create a session everytime I call this script (which might be a problem).
Can you tell me about a python script (not shell script) that imports images like the shell script above, not creating a session for each image (to a dedicated dataset)?

Best,
BK

[jmoore]

The use of "omero import" shouldn't re-create a session, but instead will re-use the same one as long as its active. If you'd prefer to use a python-script, however, you can do the following:

Code: Select all

import omero
import omero.cli

def method():
            cli = omero.cli.CLI()
            cli.loadplugins()
            cmd = ["-s", self.host, "-p", str(self.port), "-k", key, "import"]
            ## To re-direct output to a file
            ## cmd.extend([str("---errs=%s"%t), str("---file=%s"%to)])
            cmd.extend(shlex.split(self.importArgs))
            cmd.append(fileName)
            logging.debug("cli.invoke(%s)" % cmd)
            cli.invoke(cmd)
            retCode = cli.rv


This is from https://github.com/openmicroscopy/openmicroscopy/blob/master/components/tools/OmeroFS/fsDropBoxMonitorClient.py#L608

Cheers,
~Josh.

[bhcho]

After several iterations of image importing and calculation (executed from my custom Bridge), OMERO.web looked slow so I checked Indexer log and I found lots of errors like below.

2011-11-23 17:57:30,526 INFO [ ome.services.fulltext.FullTextIndexer] (3-thread-4) INDEXED 13 objects in 1 batch(es) [38508 ms.]
2011-11-23 17:57:30,528 ERROR [ ome.services.util.ServiceHandler] (3-thread-4) Method interface ome.services.util.Executor$Work.doWork invocation took 38511
2011-11-23 17:57:33,075 INFO [ ome.services.fulltext.FullTextIndexer] (3-thread-1) INDEXED 1 objects in 1 batch(es) [1056 ms.]
2011-11-23 18:25:00,020 INFO [ome.services.sessions.state.SessionCache] (3-thread-3) Waiting for synchronization
2011-11-23 18:25:02,346 INFO [ome.services.sessions.state.SessionCache] (3-thread-5) Synchronizing session cache. Count = 2
2011-11-23 18:25:02,361 INFO [ome.services.sessions.state.SessionCache] (3-thread-5) Synchronization took 15 ms.
2011-11-23 18:55:04,018 INFO [ome.services.sessions.state.SessionCache] (3-thread-4) Waiting for synchronization
2011-11-23 18:55:05,350 INFO [ome.services.sessions.state.SessionCache] (3-thread-2) Synchronizing session cache. Count = 2
2011-11-23 18:55:05,367 INFO [ome.services.sessions.state.SessionCache] (3-thread-2) Synchronization took 17 ms.
2011-11-23 19:25:08,018 INFO [ome.services.sessions.state.SessionCache] (3-thread-1) Waiting for synchronization

from Blitz log,

2011-11-25 11:24:33,282 ERROR [ ome.services.blitz.impl.ServiceFactoryI] (l.Server-9) Error destroying servant: 80:2:cf:74:2845654e:133d13e045d:-1cd5omero.api.ThumbnailStore=om$
java.lang.RuntimeException: omero.InternalException
serverStackTrace = "java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeLast(LinkedList.java:144)
at ome.security.basic.CurrentDetails.logout(CurrentDetails.java:180)
at ome.security.basic.BasicSecurityWiring.logout(BasicSecurityWiring.java:110)
at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:85)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy69.close(Unknown Source)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:227)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
serverExceptionClass = "java.util.NoSuchElementException"
message = (null)
at ome.services.blitz.impl.AbstractAmdServant$1.ice_exception(AbstractAmdServant.java:191)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:255)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
Caused by: omero.InternalException
serverStackTrace = "java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeLast(LinkedList.java:144)
at ome.security.basic.CurrentDetails.logout(CurrentDetails.java:180)
at ome.security.basic.BasicSecurityWiring.logout(BasicSecurityWiring.java:110)
at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:85)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy69.close(Unknown Source)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:227)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
serverExceptionClass = "java.util.NoSuchElementException"
message = (null)
at omero.util.IceMapper.handleException(IceMapper.java:1262)
... 12 more

Do you have any idea about this?

Best,
BK

[jmoore]

It would look like something has happened to your indexer, though what you show here are just symptoms not the cause. Could you attach your entire log, and possibly the master.out and master.err files?

Thanks,
~Josh