In place import : changing the symlink name

[thusharaw]

I'm using in place import to store some files with OMERO and we have a perhaps unique situation that at the point of doing the import, the file name that we are linking to is an undecipherable hash - we use the actual file name as a key/value pair that can be retrieved but when we look at the files with omero web, we see the hash on the left pane and the only way to get to the original file name is by expanding the 'key value pairs' tab on the right and searching for the original file name.

Now I agree the simpler solution would be to pass the original file name to the in place import command, but for reasons I'd rather not get into, that is not an option for us.

Here is an idea of what the files look like, once imported to OMERO:

Code: Select all

/omero/managed/omero_2/2017-06/
├── 02
├── 26
  ├── 22-49-39.354
│   │   └── 8898a7abf72344d18bc5301d525cf15f -> /file-storage-service/data/dev/5f/8898a7abf72344d18bc5301d525cf15f
│   ├── 22-49-39.354.log
│   ├── 22-49-47.433
│   │   └── da81347a829244ff84911640409cfe2c -> /file-storage-service/data/dev/2c/da81347a829244ff84911640409cfe2c
│   ├── 22-49-47.433.log
│   ├── 22-49-55.292
│   │   └── a2c5652bdc4b497f894cc166ec49e923 -> /file-storage-service/data/dev/23/a2c5652bdc4b497f894cc166ec49e923

So in omero web, we would see hashes like 8898a7abf72344d18bc5301d525cf15f on the left pane.
We would like the hash to be changed to something readable - which we know at the time of doing the in place import.

So the ask is if it is possible to have the in place import use a user supplied symlink, where we supply it specifically to the tool. So then the above file structure could look like this:

Code: Select all

/omero/managed/omero_2/2017-06/
├── 02
├── 26
  ├── 22-49-39.354
│   │   └── stem_cell_30_days.tiff -> /file-storage-service/data/dev/5f/8898a7abf72344d18bc5301d525cf15f
│   ├── 22-49-39.354.log
│   ├── 22-49-47.433
│   │   └── stem_cell_60_days.tiff -> /file-storage-service/data/dev/2c/da81347a829244ff84911640409cfe2c
│   ├── 22-49-47.433.log
│   ├── 22-49-55.292
│   │   └── differentiated_1_day.tiff -> /file-storage-service/data/dev/23/a2c5652bdc4b497f894cc166ec49e923

thanks,
thushara (S/W Engineer @ Allen Institute for Cell Science)

[wmoore]

Hi,

I don't think that there's a way to do the symlinking that you're asking for, but there are alternative ways to update the name so that it will display the way you want in the clients.
The easiest from the user's point of view would be to create an OMERO.script that can be run from the web (or Insight) client to update image names after import. See https://www.openmicroscopy.org/site/sup ... s/scripts/ for an intro to scripts.

You say that you have the actual file name saved as a Key/Value pair on each image? How is this populated?
Or do you mean the Original Metadata (under the Acquisition tab) that is coming from the file itself?
As a starting point, this script is probably quite useful: https://github.com/ppouchin/omero-user- ... _colors.py
It reads the original metadata and uses particular values to set channel colours for the image. It would be easy to look for a different key/value in the original metadata and use it to set the image name.

Or if the image name is really a Key/Value annotation (known as Map Annotation on the server) then you can read these just as easily. Although if you're already adding these annotations in a different step then maybe you could do the image re-naming at the same time?

Either way, let us know if a script sounds like a useful solution and if you need any help with it,

Regards,

Will.

[mtbc]

Further, you may be interested to know that the file transfer methods are designed to be extensible so it is possible to write some Java code to teach the server to adjust the file naming as desired -- see http://www.openmicroscopy.org/site/supp ... e-transfer for this. It may not be too hard to adapt OMERO's SymlinkFileTransfer class for your needs.

Note that OMERO's ManagedRepositoryI.suggestImportPaths uses a path sanitizer from ServerFilePathTransformer to convert arbitrary filenames to something safely legal for the server's underlying filesystem in effecting the behavior described in http://www.openmicroscopy.org/site/supp ... file-names -- you may find it convenient to reuse some of that code if the desired filenames are not guaranteed to be "safe".

Cheers,
Mark

[thusharaw]

Thank you, Will and Mark. Appreciate the input.

>> You say that you have the actual file name saved as a Key/Value pair on each image? How is this populated?
>> Or do you mean the Original Metadata (under the Acquisition tab) that is coming from the file itself?

The key/value pairs are populated as part of our service as Omero annotations. So they are not in the original metadata of the file. But our service has access to them.

It would be sufficient to have the omero viewer show the original file name. Rather than using a script, I would like to do that in the same scala/java code that does the file importing.

One thing I noticed was that in the omero insight viewer, I can click the edit icon on the top right, and change the file name. Does the java api allow this? I tried to get a pointer to the Image object and set the name, with some code like this:

Code: Select all

val imageObj = new ImageI(imageId, true)
imageObj.setName(originalFileName)

but OMERO threw a omero.gateway.exception.DSAccessException on saving the object:

Code: Select all

     "message": "Cannot access data. \nCannot update the object.",
      "class": "omero.gateway.exception.DSAccessException",
      "stack": [
        "omero.gateway.facility.Facility.handleException(Facility.java:305)",
        "omero.gateway.facility.DataManagerFacility.saveAndReturnObject(DataManagerFacility.java:344)",
        "org.alleninstitute.ois.helpers.OmeroAPI.addKeyValue(OmeroAPI.scala:266)",
        "org.alleninstitute.ois.helpers.OmeroAPI.$anonfun$inplaceImport$8(OmeroAPI.scala:83)",
        "scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:12)",
        "scala.collection.immutable.List.map(List.scala:272)",
        "org.alleninstitute.ois.helpers.OmeroAPI.$anonfun$inplaceImport$7(OmeroAPI.scala:80)",
        "scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)",
        "scala.collection.immutable.List.foreach(List.scala:378)",
        "scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)",
        "scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)",
        "scala.collection.immutable.List.flatMap(List.scala:341)",
        "org.alleninstitute.ois.helpers.OmeroAPI.$anonfun$inplaceImport$4(OmeroAPI.scala:78)",
        "scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)",
        "scala.collection.immutable.List.foreach(List.scala:378)",
        "scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)",
        "scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)",
        "scala.collection.immutable.List.flatMap(List.scala:341)",
        "org.alleninstitute.ois.helpers.OmeroAPI.inplaceImport(OmeroAPI.scala:77)",
        "org.alleninstitute.ois.helpers.InplaceImporter.importImage(InplaceImporter.scala:138)",
        "org.alleninstitute.ois.helpers.InplaceImporter.$anonfun$importDataset$1(InplaceImporter.scala:262)",
        "org.alleninstitute.ois.helpers.InplaceImporter.$anonfun$importDataset$1$adapted(InplaceImporter.scala:260)",
        "scala.collection.immutable.List.foreach(List.scala:378)",
        "org.alleninstitute.ois.helpers.InplaceImporter.importDataset(InplaceImporter.scala:260)",
        "org.alleninstitute.ois.helpers.InplaceImporter.$anonfun$handleNonEmptyFieldList$2(InplaceImporter.scala:348)",
        "scala.util.Success.$anonfun$map$1(Try.scala:251)",
        "scala.util.Success.map(Try.scala:209)",
        "scala.concurrent.Future.$anonfun$map$1(Future.scala:287)",
        "scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)",
        "scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)",
        "org.uowlog.UOWFutureOnCompleteWrapper.$anonfun$apply$2(Continuity.scala:44)",
        "org.uowlog.UnitOfWork.apply(UnitOfWork.scala:468)",
        "org.uowlog.UOWFutureOnCompleteWrapper.apply(Continuity.scala:42)",
        "scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)",
        "scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:140)",
        "java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)",
        "java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)",
        "java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)",
        "java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)"
      ]
    },
    {
      "message": "null",
      "class": "omero.ValidationException",
      "stack": [
        "sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java)",
        "sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)",
        "sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)",
        "java.lang.reflect.Constructor.newInstance(Constructor.java:423)",
        "java.lang.Class.newInstance(Class.java:442)",
        "IceInternal.BasicStream.createUserException(BasicStream.java:2779)",
        "IceInternal.BasicStream.access$300(BasicStream.java:14)",
        "IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298)",
        "IceInternal.BasicStream.throwException(BasicStream.java:2291)",
        "IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399)",
        "omero.api.IUpdatePrxHelper.end_saveAndReturnObject(IUpdatePrxHelper.java:927)",
        "omero.api.IUpdatePrxHelper.saveAndReturnObject(IUpdatePrxHelper.java:805)",
        "omero.api.IUpdatePrxHelper.saveAndReturnObject(IUpdatePrxHelper.java:792)",
        "omero.gateway.facility.DataManagerFacility.saveAndReturnObject(DataManagerFacility.java:341)",
        "org.alleninstitute.ois.helpers.OmeroAPI.addKeyValue(OmeroAPI.scala:266)",
        "... omitting 36 common frames"
      ]
    }


I was wondering if there is a different API to allow for this?

[wmoore]

I think you'll need to load the Image first before updating the name and saving this.

For example at https://docs.openmicroscopy.org/omero/5 ... /Java.html

Code: Select all

BrowseFacility browse = gateway.getFacility(BrowseFacility.class);
ImageData image = browse.getImage(ctx, imageId);


Hope that helps,

Will.

[thusharaw]

Thanks Will.

Perhaps I'm not saving the image correctly? Here is my console showing that after saving the image, the image name has not changed:

Code: Select all

scala> val imageId = 103
imageId: Int = 103

scala> val browse = gateway.getFacility(classOf[BrowseFacility])
browse: omero.gateway.facility.BrowseFacility = omero.gateway.facility.BrowseFacility@25569ff0

scala> val image = browse.getImage(ctx, imageId)
image: omero.gateway.model.ImageData = omero.gateway.model.ImageData (id=103)

scala> image.setName("changeme")

scala> val imageObj = new ImageI(imageId, false)
imageObj: omero.model.ImageI = omero.model.ImageI@36cfd887

scala> dm.saveAndReturnObject(ctx, imageObj)
res16: omero.model.IObject = omero.model.ImageI@3731202a

scala> image.getName
res17: String = changeme

scala> val changedImage = browse.getImage(ctx, imageId)
changedImage: omero.gateway.model.ImageData = omero.gateway.model.ImageData (id=103)

scala> changedImage.getName
res18: String = 73b0c9fcb69e4601ae83b591e1da4259.czi

scala>


While the in memory object preserves the name change, after saving and retrieving it back, it reverts back to the old name.

thanks,

[jmoore]

Hi Thushara,

Perhaps I'm not saving the image correctly? Here is my console showing that after saving the image, the image name has not changed:

See the comments below:

Code: Select all

## This loaded & modified object should be saved
scala> image.setName("changeme")                           

## This object isn't loaded, therefore the save is a no-op.
scala> val imageObj = new ImageI(imageId, false)           
imageObj: omero.model.ImageI = omero.model.ImageI@36cfd887


Cheers,
~Josh.

[thusharaw]

Thanks Josh - yeah, I was missing the asImage function that would have given me the right object to pass to the save method:

Code: Select all

val image = browse.getImage(ctx, imageId)
image.setName(f)
dm.saveAndReturnObject(ctx, image.asImage())