importing time

[bhcho]

Hi all,

I'm trying to measure the time for importing.
when I use the OMERO.importer, the status says "prepping", "analyzing", "updating DB", transfer(1/3, 2/3, 3/3), "thumbnailing", and "done".
However, it takes too much time to go to the next image.
That is, it only took about 5 seconds (from "prepping" to "done") for importing an image, but it took another about 10 seconds before it started "prepping" for the next image.
I'm currently using 4.2.1.

Could you tell me about what OMERO is doing at the cooling time?
is this because I'm using 4.2.1? (may be solved from 4.2.2?)

Best,
BK

[jmoore]

Hi BK,

could you attach the importer.log file after one of these incidents? Does it always occur? Only for particular file types? How many files are in your queue? Is it only the transaction from the first file to the second file, or also the transition from the second to the third file?

Cheers,
~Josh

[bhcho]

Hi Josh,

attached is the importer.log file.

Does it always occur?

Yes, it does.

Only for particular file types?

when I tried for ome-tif, tif, and bmp, it always happened.

How many files are in your queue?

from 25 images to 75 images. But it seems there is always the cooling time, even though the cooling time span varies a little bit.

Is it only the transaction from the first file to the second file, or also the transition from the second to the third file?

As I said, it's consistent along the entire queue.

BK

[jmoore]

Hi BK,

could you possibly attach your var/log/Tables-0.log file as well?

Thanks,
~Josh

[bhcho]

Attached.

BK

[jmoore]

Thanks, BK.
It looks like the "Import History" functionality is causing the delays:

Code: Select all

2011-04-04 16:19:18,867 INFO  [                            omero.remote] (Dummy-11  )  Meth: TableI.update
2011-04-04 16:19:26,182 INFO  [                     omero.tables.TableI] (Dummy-11  ) Updated 5861 row(s) of data to Table-...


That's a 7 second window in which the import information is saved. To test this, could you run the following and then re-try import:

Code: Select all

bin/omero admin ice server disable Tables-0
bin/omero admin ice server stop Tables-0


If that's the case, we'll need to look into getting the fixes described under http://trac.openmicroscopy.org.uk/ome/ticket/4872.

Cheers,
~Josh

[bhcho]

Thanks Josh,

After I stopped Tables-0, there was no idle time before it moves to the next image.
what does this Tables-0 do? updating import history?
Do I need to restart this service to complete the importing? and how?

Best,
BK

[jmoore]

Hi BK,

Tables-0 is the service which provides the OMERO.tables API, so leaving it turned off is not really an option in your case. The importer is also using the service to store your import history, but unfortunately there's no runtime way to turn this off (other than disabling the service itself).

If you're willing to compile your own client, you can comment out the line:
http://git.openmicroscopy.org/?p=ome.git;a=blob;f=components/tools/OmeroImporter/src/ome/formats/importer/gui/HistoryTableStore.java;h=6e30f16f5af888f3d41214e948b867abe5faa0d9;hb=4_2#l127

and the importer will not use the tables service. Alternatively, you can use the command-line importer for large imports, until we've fixed #4872.

Cheers,
~Josh