Open Microscopy Environment

Hi Josh

I've looked at the code from here : https://github.com/openmicroscopy/openm ... ry.py#L235

I'm trying to set the session to use an already setup public_group - how do I do that?

The below code works apart from setting the group bit:

Code: Select all

print "initiating ImageImport object"
        self.root = omero.client(omeroHost)  # ok because adds self
        #self.__clients.add(self.root)
        self.root.setAgent("OMERO.py.root_test")
        self.root.createSession("user", "pass")
        group=self.group_and_name("public_group")
        self.root.ctx["omero.group"]=group
        print "user="+str(self.root)


It would also be great to have an example of uploading an image using soft linking via this api?

Cheers

Jonathan.

Hi Jonathan,

to switch between groups, you may want to use the setSecurityContext(group) method on the initialized session (see https://github.com/openmicroscopy/openm ... ry.py#L488 for an example).

In order to upload an image, you should be able to use a CLI object like in this tread using the key of the session. You can then pass the soft link transfer options to the import command:

Code: Select all

cli = omero.cli.CLI()
cli.loadplugins()
import_args = ["import", "-s", hostname, "-k", "%s" % sessionId]
import_args += ["---errs", stderr, "---file", stdout, file_to_import]
import_args += ["--", "--transfer=ln_s"]
cli.invoke(import_args, strict=True)


Best,
Sebastien

Great - thanks.

I was using the CLI before - but after speaking to Josh in person last week he suggested using the client as in the test script shown above. Loading each file using the CLI was taking too long for us as the Java JVM boots up for each image...

Using the CLI on a per directory basis was fast - but at some points we need more fine control as loading a directory results in the same images being uploaded again. I wish/ is there a "don't load files of same name" option to the CLI?

Hi Jonathan,

jwarren wrote:I was using the CLI before - but after speaking to Josh in person last week he suggested using the client as in the test script shown above. Loading each file using the CLI was taking too long for us as the Java JVM boots up for each image...

Using the CLI and using the omero.cli.CLI object should be quite close in terms of time. Both will need to create a JVM and that was primarily what I wanted to prevent.

Using the CLI on a per directory basis was fast - but at some points we need more fine control as loading a directory results in the same images being uploaded again. I wish/ is there a "don't load files of same name" option to the CLI?

At the moment, there's not such a "don't load files of same name" option, but it's an interesting idea. Thanks. That being said, what sequence of commands is leading to multiple imports? Unless you're recursively descending the same directory multiple times, I wouldn't expect that to happen.

Cheers,
~Josh.

Hi Josh

Unfortunately we go into the same directories often to upload data as the directory the file is in is the only way to associate the correct omero id with our external db image record. When we have new images then we have to scan the directories to see if there are new ones in them.

Using the CLI with directory loading (not per file basis) and no_thumbnails argument was 6 times faster than on a file basis and thumbnails. Using the CLI using the directory as an argument still falls over after a few thousand images (complained of lost connection) and then I have to restart the job. This means we need to go through the same directories in case the directory contents was only half loaded... resulting in duplicates or triplicates in omero (depending on the number of times we've already run the importer).

How difficult would it be to put a filename check in the importer? Should I try it? Any pointers?
Writing this importer in java and using the java client code itself maybe faster for me to write and use?


Hope this makes sense?

jwarren wrote:Unfortunately we go into the same directories often to upload data as the directory the file is in is the only way to associate the correct omero id with our external db image record. When we have new images then we have to scan the directories to see if there are new ones in them.

Ah, ok. Sorry for having not realized that. What you're doing is sounding a bit more like dropbox now, but of course, only partially, which explains how/why you're script has developed as it has. Speaking of which, is that visible on github?

Using the CLI with directory loading (not per file basis) and no_thumbnails argument was 6 times faster than on a file basis and thumbnails.

Just for comparison, how many files (how large each) and how many seconds is that?

Using the CLI using the directory as an argument still falls over after a few thousand images (complained of lost connection) and then I have to restart the job.

Hmmmm.....that's surprising unless your calling Python script is timing out the session that it created. Looking at that again would be helpful.

This means we need to go through the same directories in case the directory contents was only half loaded... resulting in duplicates or triplicates in omero (depending on the number of times we've already run the importer).

How difficult would it be to put a filename check in the importer? Should I try it? Any pointers?

This should be pretty straight-forward. Perhaps we need to work out some more of the details first though. (For example, would checking on the sha1 suffice? Filenames may not be unique for everyone)

Writing this importer in java and using the java client code itself maybe faster for me to write and use?

That's certainly up to you. But, for example, if you'd like to have a go at the duplicate detection, I can point you to the right location (roughly https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/cli/CommandLineImporter.java#L332).

Cheers,
~Josh

Jonathan,

I've tried to scope out a "file exclusion" flag for the importer under https://github.com/openmicroscopy/openm ... /pull/3510 -- thoughts welcome (preferably on the PR).

~Josh.