OME5 import takes hours preparing upload

[hroque]

Hi all,

I believe this question/bug might be related to a previous ticket (https://www.openmicroscopy.org/qa2/qa/feedback/7651/) and also a related discussion here on the forum (http://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7351) about uploading large files form a ERS PE system.

I decided to open a new topic because these filesets keep on causing problems even after the upgrade to OME5.

My question is: is the uploading supposed to stay for hours (literally I've pressed the upload buttom 2.5h ago) in the preparing upload status?
This is concerning a 37Gb folder with ~55k files.
I thought that in OME5 the files would be checked and if ok just copied across and then the server would take care of the rest.

Cheers and thanks,
Helio

[jmoore]

Hi all,

Hi Helio,

My question is: is the uploading supposed to stay for hours (literally I've pressed the upload buttom 2.5h ago) in the preparing upload status?
This is concerning a 37Gb folder with ~55k files.
I thought that in OME5 the files would be checked and if ok just copied across and then the server would take care of the rest.

Could you send us the insight log or logs that you think pertain to the upload? (~/omero/log/omeroinsight.log*) You're certainly right that the "prepare upload" phase should not be taking hours and hours. What file are choosing in the importer? Are you clicking on the entire directory?

My assumption is that something has gone wrong but that the state has not been properly updated. Would be great if we could track that down.

Thanks for all your help!
~Josh

[hroque]

Hi Josh,

I'm sending here the log. I will also ask Douglas to send the log from the server side to see if there is anything there.
the preparing upload status did change (after ~12h) and it uploaded the whole folder (taking ~6h) but it failed finally in the metadata import. I've had asked Douglas to up the memory on the server and he did so not sure this is coming up again...

In any case hope the log does give you some hints...

Here is the link for the log file: https://www.dropbox.com/s/5lpe76zc11u7p ... nsight.log

Cheers,
Helio

PS.: Yes I always just select the entire folder and upload it instead of selecting the one file.

[jmoore]

Hi Josh,

Hi Helio,

I'm sending here the log. I will also ask Douglas to send the log from the server side to see if there is anything there.

That's great, thanks. There look to be several attempts in the file; could you suggest some rough times to look at?

the preparing upload status did change (after ~12h) and it uploaded the whole folder (taking ~6h)

With one of the sample filesets that Douglas provided, I definitely saw long times in scanning the files. From what I could tell, most of this came from the number of files in a directory (+75K). Tomorrow I'll file a ticket so Melissa can see if there's anything that can be optimized, but for the moment, it is simply quite a long process. (Picking the .htm file or similar might speed it up though).

but it failed finally in the metadata import. I've had asked Douglas to up the memory on the server and he did so not sure this is coming up again...

I chatted briefly with Douglas. We do need to make sure that the 5rc1 server is setup as we had previously. The server log would tell us what went wrong.

Cheers,
~Josh