Moving large project to another group

[jmoore]

Hi Tristan,

I'm currently repeating your experiment with 100 copies of the file you linked to. One question I have: did the moves you attempted come directly after the import? Since these TIFFs are fairly large (6K x 6K), the first thing the server does is to generate pixel pyramids for them at multiple resolutions. This processing overhead is certainly a portion of the CPU consumption you were seeing. Once the processing is done for all 100 images, I'll test the move operations and see how it goes.

Cheers,
~Josh

[jmoore]

Hi again, Tristan,

having let the background processing finish, a move of 100 images in a single dataset on my local system takes 3-5 minutes. My expectation then is that the background work was causing much of the issues you saw. If you see any other notably slow actions, please let us know.

Cheers,
~Josh

[Sethur]

Hi Josh,

I'm not entirely sure any more how much time I gave the system after the import before I started the move but it is very possible that I did it right away. I will repeat this experiment tomorrow.

On this note: Can you give me a recommendation on the memory settings for the different OMERO.server processes. Currently, I set Blitz to 8 GB and Indexer as well as PixelData to 1 GB. Our total system memory is 16 GB. Do you think this is a good choice for a setting with approx. 100 users where 10 might work concurrently?

Cheers,

Tristan

[jmoore]

On this note: Can you give me a recommendation on the memory settings for the different OMERO.server processes. Currently, I set Blitz to 8 GB and Indexer as well as PixelData to 1 GB. Our total system memory is 16 GB. Do you think this is a good choice for a setting with approx. 100 users where 10 might work concurrently?

Those are certainly reasonable numbers. Going much over 8GB for the Blitz process probably won't help linearly unless you also start tuning other Java options (GC, etc). 1GB for the other 2 processes is also a good starting point. In 5.0.0, there will hopefully be a number of fixes to make the Indexer need less, but if you can spare 1GB for it, it certainly won't hurt.

If you notice that you are having any issues with larger images (3K x 3K or larger), then check the PixelData log. In outside circumstances, you may need to increase the memory for it. Going for 2GB from the start if you can spare might be slightly safer.

Cheers,
~Josh.

[Sethur]

Hi Josh,

I repeated the move operation of 100 x 55 MB tiffs now after upgrading to 5.0.0-rc2 and it took approx. 4 mins. Also, the server was still responsive during the operation, so I think as long as nobody tries to move extremely large projects, we are fine (assuming this scales linearly).

Cheers,

Tristan

[jmoore]

Thanks for letting us know, Tristan.

Cheers,
~Josh