Upgrade from 4.4 to 5.0

[ppouchin]

Hello !

I've seen that OMERO 5 is now out, and this is great !
I'm really eager to use it and upgrade our server... !

However, I was wondering one thing: when I first set up the server (years ago), I chose to use the DropBox feature as a way to archive original files (and let them easily accessible to the users).

Of course, I could just keep the "pixels" as they are stored now into OMERO 5, but in order to free some space, I was wondering if there would be a way to use the original files stored in the DropBox to replace the "Pixel" data while preserving all the annotations... like... a python script re-importing all the images and comparing their names to re-annotate them... ? (brute-force style)

I have a feeling it's going to be hard, and probably not worth it... but still... I wanted to ask.

[jmoore]

We haven't (yet) developed anything to do this automatically. Do you know the mapping from files under DropBox to the images in OMERO? What file formats do you have? If you have some less critical data that you would be willing to test this on, we could try to come up with a script that would work.

Cheers,
~Josh.

[ppouchin]

Do you know the mapping from files under DropBox to the images in OMERO?

What do you mean by "mapping"... ? I'm not sure I understand...

Each user's home resides in the DropBox folder, and when they put a file there, it's automatically imported into OMERO. All I know is that the only link between the original files and the images in OMERO is their name: the image name is the full path to the file (e.g. "/data/OMERO/DropBox/<username>/path/to/the/image"), although some images might have been renamed to only keep the relative path from the user's folder...

What file formats do you have? If you have some less critical data that you would be willing to test this on, we could try to come up with a script that would work.

Cheers,
~Josh.

In the DropBox folder, we mainly have LIF from Leica (2643) and (old) LSM from Zeiss (2878), but there are TIFF as well (1173). A few PNG (450), JPG (153), MDB (69) and some rare IMS (40) also made it to the server.

For now, we only use 5.1 TB for both DropBox and OMERO. I have access to a bigger server with several TB available, so I may do a backup of the OMERO data (folder & database) before trying anything.
I can also use a VM to do some tests... Although I cannot do that for 2 TB of data...

[jmoore]

What do you mean by "mapping"... ? I'm not sure I understand...

By "mapping", I mean knowing how to programmatically take the names under /data/OMERO/DropBox and find the images that were imported from them. If use of Image.name works, that could be a start. Alternatively, you would need to do them in batches, passing the directory and the image IDs to the script.

In the DropBox folder, we mainly have LIF from Leica (2643) and (old) LSM from Zeiss (2878), but there are TIFF as well (1173). A few PNG (450), JPG (153), MDB (69) and some rare IMS (40) also made it to the server.

This is good to know, thanks. Let us have a look at we'll let you know.

Cheers,
~Josh.

[ppouchin]

Thank you for your quick answers.

By "mapping", I mean knowing how to programmatically take the names under /data/OMERO/DropBox and find the images that were imported from them. If use of Image.name works, that could be a start. Alternatively, you would need to do them in batches, passing the directory and the image IDs to the script.

In the past, I used a small & dirty script to generate some commands in order to force a re-import in case DropBox had ceased to function and imports had been dropped.
It did work, but I've not used that for quite some time.
Also, there was a limit: if a file had been deleted in OMERO but was still in the folder, I was not able to know that it had been imported and deleted.

It did allow me to match images with files, but it may not be helpful in this case.

Edit: Also, I realize now that it might have been a bit faster had I removed matched images from the list of images. Although there might be even faster ways to do that than a double loop...

[jmoore]

I'll certainly take a look. Cheers! ~Josh

[ppouchin]

Also, a user recently had to delete around 3000 images, and I noticed it took quite some time, and that it did not complete properly. Doing that by batches of around 700~800 worked better.

So I'm curious if we are able to somehow import images in OMERO 5 and copy the annotations from the old images to the newly imported files, it might be necessary to slow both the import and delete processes (through pauses every couple hundreds of images)...

Otherwise, I fear the server may not be able to withstand the load...

But maybe I'm shortsighted and don't see the proper way to go at it...

[wmoore]

Hi,

You might find some useful info in this post: viewtopic.php?f=4&t=7397 which is mostly about performance of "Move group" but also applies to large Deletes.

I'm afraid we don't have any built-in functionality for replacing one image with a newly imported one.
It should be easy enough to write a script to transfer annotations from old images to new ones.
Here is one I wrote a while back for taking Tags from one Dataset and applying them to the contained Images or other Datasets or Images. https://gist.github.com/will-moore/5777962. However, it doesn't quite fit your requirements, since you'd need to do some matching of images by name to transfer Tags from one to the other.
You'd also lose any info about who added the annotations and when.

NB: I noticed that in your script you filter "Images I own" after retrieving them from OMERO. A better way of doing this would be:

Code: Select all

userId = conn.getUserId()
params = omero.sys.Parameters()
params.theFilter = omero.sys.Filter()
params.theFilter.ownerId = rlong(userId)
images = conn.getObjects("Image", params=params)


[jmoore]

Please follow https://trac.openmicroscopy.org.uk/ome/ticket/12176 to monitor the progress of any 4.4/5.0 migration work.

Thanks,
~Josh

[ppouchin]

Please follow https://trac.openmicroscopy.org.uk/ome/ticket/12176 to monitor the progress of any 4.4/5.0 migration work.

Thanks,
~Josh

Ok. Thank you.
(I'll try to do some tests in the weeks/months to come...)