cannot download archived file

[ebbwdan]

Hi

Some time ago I posted asking about how to deal with creating new images from very large ROIs (~20,000x20,000 pixels) annotated to very large slide scanner images (~100,000x60,000 pixels) using the OMERO scripting service. Josh and Will helped to write a function whereby pixel data is read tile-by-tile and placed in a new image tile-by-tile.

I am now revisiting this work and I am finding that I cannot download my newly created image. The option to download the archived file is not available and I cannot download as either ome-tiff or standard tiff as the image is too big (the error message I receive states as much).

I have tried a slightly different approach where I write a tiled tiff and import that back to OMERO but still I cannot download the archived file. I need some suggestions for how I can get the data back please. I was thinking about using a script to export but I can't see in the API where I would able to do this for an original archived file.

Many thanks in advance!

Cheers,

Dan.

[jmoore]

Hi again Dan,

since it's been a while, could you tell us what version you are currently using, and how all (insight, web, command-line, API) you're trying and need to download the file?

All the best,
~Josh

[ebbwdan]

Hi Josh,

Thanks for the response!

We are running:

5.0.6-ice35-b53

I realise now that the subject line isn't quite accurate (sorry!). The thing which I hadn't realised before was that I never get an archived file when running the scripting service, even when it is an image that I upload from outside OMERO (see case 2 below). That is ok when the data is small enough that an OME-TIFF can be exported but is a problem in the following two cases:

1. My script to crop ROIs from slide scanner images generates new images well in excess of 3k x 3k which then cannot be exported as OME-TIFF (big tiff error).

2. I am running a script which runs an ImageJ plugin to stitch multiple fields of view together. I download data to a temp location, ImageJ executes and I upload newly generated data (tile-by-tile if required). Again I cannot export the data as OME-TIFF if the new image is > 3k x 3k and there is no archived image.

Cheers,

Dan.

[wmoore]

Hi Dan,

Certainly if you create a new image (or new tiled image) via the API by using setPlane() or setTile() to upload the pixel data, then this image will not have any Original Files.

However, if you were to use the API to do a regular OMERO import of a big tiled tiff (E.g. use Python to call the command-line importer as discussed viewtopic.php?f=6&t=5149&p=10271) then you would have the auto-generated pyramid AND the original image on the server.

One option that might work for you is to have a client-side Python script that iterates through all the tiles, using getTile() to write a client-side tiff image (I remember from our previous discussion viewtopic.php?f=6&t=7565 that you have some experience of writing tiled tiffs). Would this workflow give you everything you need, or is this not practical for multiple users etc?
A possible alternative is to run this 'export' script via the scripting service, writing a downloadable file that can then be downloaded from the clients (in the same way as Batch_Image_Export.py generates a zip for download). This makes the functionality widely available in the clients but creates large file annotations on the server, which are not automatically cleaned up (although giving the annotations a unique namespace makes it quite easy to delete them periodically via the API).

Hope this helps - or at least gets us closer to a solution. Josh is away just now but might have some other ideas if this isn't going to work.

Cheers,

Will.

[ebbwdan]

Hi Will,

Many thanks for the response and the ideas. My first thought was to use the 'Batch_Image_Export.py' approach but I have to admit that I hadn't at first fully appreciated that it was zip up the files and annotate them. As a slight aside we have noticed that downloading files using the webclient creates a zip file and this can fail when the image is very large (looks like a timeout). I would potentially trying to zip a file that would have a size of anything up to 1Gb; would timeouts be any issue.

I like the idea of using the CLI and I wish I'd thought of it! This is much better than the approach I am taking at the moment (trying to adapt a third party module). I will let you know how I get on.

Thanks again!

Cheers,

Dan.

[ebbwdan]

Hi Will,

I've tested the CLI and I think it is definitely the way to go. If possible I would like to log in using a session key rather than having to pass a username and password. I can see the session uuid if use "getEventContext" but I couldn't find a method to get the id from the context. I'm sure I've missed something so maybe you can point me in the right direction.

Cheers,

Dan.

[jmoore]

Hi Dan,

I think I'm missing something. The session uuid value that you find in the EventContext is the one you should use in combination with "bin/omero login -k" or even "bin/omero download -k". There won't be a method on that object, you just access the value:

Code: Select all

eventContext = client.sf.getEventContext()
session_key = eventContext.sessionUuid


Alternatively (and slightly more efficient):

Code: Select all

client.getSessionId()


Cheers,
~Josh.

[ebbwdan]

Hi Josh,

Thanks for the response. Not you missing something, it was me completely missing that the sessionId is an attribute.

From the command line I have no problem logging in at the command line using the CLI and a session key but when I try this in a script I am getting an EOFerror from raw_input. I am running:

Code: Select all

cli.invoke(["sessions", "login", "-k", "%s" % sessionId], strict=True)

And am getting:

File "./script", line 333, in do_import
cli.invoke(["sessions", "login", "-k", "%s" % sessionId], strict=True)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 791, in invoke
stop = self.onecmd(line, previous_args)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 860, in onecmd
self.execute(line, previous_args)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 940, in execute
args.func(args)
File "/home/omero/OMERO.server/lib/python/omero/plugins/sessions.py", line 324, in login
server, name, port = self._get_server(store, name)
File "/home/omero/OMERO.server/lib/python/omero/plugins/sessions.py", line 637, in _get_server
rv = self.ctx.input("Server: [%s]" % defserver)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 408, in input
rv = raw_input(prompt)
EOFError: EOF when reading a line

I notice that when run at the command line I get the server name returned which then requires confirmation. Is this causing the problem when run from the script?

Is there a better way to do this? I went this route because otherwise I need to pass a username and password.

Cheers,

Dan.

[sbesson]

Hi Dan,

to be able to use the session key argument you also need to specify the server hostname. Else the CLI will try and ask for it interactively and fail in a non-interactive session. I would expect the following to fix your issue:

Code: Select all

hostname = client.getProperty('omero.host')
sessionId = client.getSessionId()
cli.invoke(["sessions", "login", "-s", %s" % hostname, "-k", "%s" % sessionId], strict=True)


Best,
Sebastien

[ebbwdan]

Hi Sebastien,

Many thanks for the reply!

Code: Select all

hostname = client.getProperty('omero.host')

returns an empty string but it works if I set hostname to 'localhost'.

One more question. When the script completes - I get an 'Error' link in the client the contents of which seems to be the output from the CLI. Is there a way to stop that happening?

Also, is there an easy OMERO way to get the ID of the newly imported image?

Cheers,

Dan.