Inplace Import of CV7000S

[gudlap]

Sebastien -

This is a follow-up to an earlier thread. Furthermore, per Sebastien's suggestion, I'm moving this PM to public forum for future reference.

Looks like the latest version of Bioformats (6.0) started supporting import datasets acquired on Yokogawa CellVoyager CV7000/CV7000S. Thanks to all the OMERO/Bioformats developers for implementing the importer for this High-throughput microscope.

Is there a way to test this new functionality in OMERO.Server for our datasets from CV7000S?

We've tired the latest OMERO.Server docker image (release 5.4.10) from docker hub. But based on the import logs, it looks like the latest OMERO.Server (rel. 5.4.10) is using Bioformats 5.9.2.

My question(s):
1) Could we just upgrade ONLY the "bioformats" component on the OMERO.Server 5.4.10 to the latest version to test out the functionality? Looks like the user in this thread managed to test OMERO.Server 5.4.9 with Bioformats 6.0. Any pointers?

2) Is there a docker build for OMERO.Server (version 5.5.0) that is in "Alpha/Beta/RC" phase that we can use to test out the functionality?

3) We have also noticed some (unexpected??) behavior while testing inplace import feature. Specifically, if we use the following command to do a inplace import of a "source" folder OMERO.Server does NOT complain/check for "duplication" of the source folder. That is, we can import the "source" folder multiple times. What are we doing wrong? Is this somewhat related to thread?

Here is the relevant part of the code (dockerized OMERO.Server) for inplace import that we used.

Code: Select all

# Environment variable have been substituted with "dummy" values.
OMERO="/opt/omero/server/OMERO.server/bin/omero"
PREFIX="omero-5_4_10"
OMERO_PI_LAB="G1"
OMERO_PI_NAME="PI ABC"
OMERO_PI_USERNAME="piuname"
OMERO_USER_NAME="USER ABC"
OMERO_USER_USERNAME="useruname"
OMERO_EXPERIMENT_NAME="Exp_031"
IMPORT_PATH="/OMERO/inplace/PI_ABC/User_ABC/Exp_031_20171206_114408/"
USER_PATH="User_ABC"

docker exec $PREFIX-server $OMERO import -s localhost -p 4064 -g $OMERO_PI_LAB -u $OMERO_USER_USERNAME -w $OMERO_USER_USERNAME --skip all -- --no-stats  -- --auto-close -- --transfer=ln_s $IMPORT_PATH -- -T "regex:+name:^.*${USERPATH}/(?<Container1>[^/]*)/.*"


Thank you.

Regards,

Prabhakar (Reddy) Gudla

High-throughput Imaging Facility
Laboratory of Receptor Biology and Gene Expression
Center for Cancer Research, National Cancer Institute
National Institutes of Health, Rockville, MD USA

[sbesson]

Hi Prabhakar,

This is a follow-up to an earlier thread. Furthermore, per Sebastien's suggestion, I'm moving this PM to public forum for future reference.

Thanks for doing so.

1) Could we just upgrade ONLY the "bioformats" component on the OMERO.Server 5.4.10 to the latest version to test out the functionality? Looks like the user in this thread managed to test OMERO.Server 5.4.9 with Bioformats 6.0. Any pointers?

Upgrading only Bio-Formats in an OMERO 5.4 server is certainly a possibility and we have used this internally for pre-testing pyramids. The generic way to perform such an upgrade is to replace all the Bio-Formats JARs under `lib/server` and `lib/client` if you are using the server for importing. In the case of Bio-Formats 6 there have been multiple components updated that would need to be substitued. Would a modified Docker image be sufficient or useful or are you working against a non-containerized environment?
Obviously, modifying a production environment has all types of caveats in terms of testing and support.

2) Is there a docker build for OMERO.Server (version 5.5.0) that is in "Alpha/Beta/RC" phase that we can use to test out the functionality?

More generally, you can see more about the roadmap of OMERO.server 5.5.0 here. We are working through various features via development milestones. Upgrading Bio-Formats is one of the latter milestones at the moment. Once this is done, we can certainly make pre-release Docker images of the server and web deployment availabe.

3) We have also noticed some (unexpected??) behavior while testing inplace import feature. Specifically, if we use the following command to do a inplace import of a "source" folder OMERO.Server does NOT complain/check for "duplication" of the source folder. That is, we can import the "source" folder multiple times. What are we doing wrong? Is this somewhat related to thread?

You might want to try the `excludePath` option comparing the path on disk to what is stored in the DB. e.g.

Code: Select all

bin/omero import --exclude=clientPath

This is used in IDR import configuration files to prevent multiple in-place imports.

Best,
Sebastien

[gudlap]

Greetings Sebastien -

The generic way to perform such an upgrade is to replace all the Bio-Formats JARs under `lib/server` and `lib/client` if you are using the server for importing.

I can try that. Is there list of jar files that I should focus on? Sorry, never done this before.

Would a modified Docker image be sufficient or useful or are you working against a non-containerized environment?

That would be great. We are testing the functionality using Docker images in a non-production environment.

The "--exclude=clientpath" worked like a charm. I glossed over this section in the documentation

Best,

Prabhakar (Reddy) Gudla

[sbesson]

Hi,

I can try that. Is there list of jar files that I should focus on? Sorry, never done this before.

No worries. Minimally the three files you want to replace under both locations are `formats-gpl.jar`, `formats-bsd.jar` and `formats-api.jar`. After replacing these files by the versions 6.0.0, restarting your server should be sufficient to take into account the new library.

We are testing the functionality using Docker images in a non-production environment

You might want to take a look at https://doi.org/10.5281/zenodo.2594964. This is a work in-progress docker compose service. It includes a modified version of the `omero-server` Docker image which updates all the Bio-Formats JARs before starting the server. Feedback and comments welcome.

The "--exclude=clientpath" worked like a charm. I glossed over this section in the documentation

Great, thanks for letting us know.

Best,
Sebastien