OMERO in place import, excesive memory requirements

[thusharaw]

I'm running this on a new ome.tiff that does not contain the bad markup as far as I could tell:

Code: Select all

BF_MAX_MEM=12g ./showinf -nopix /allen/aics/microscopy/PRODUCTION/PIPELINE_4_OptimizationAutomation/3500000794/ZSD3/100X_zstack/converted/3500000794_100X_20170404_4-Scenes-18_F07_P24.ome.tiff
Checking file format [OME-TIFF]
Initializing reader
OMETiffReader initializing /allen/aics/microscopy/PRODUCTION/PIPELINE_4_OptimizationAutomation/3500000794/ZSD3/100X_zstack/converted/3500000794_100X_20170404_4-Scenes-18_F07_P24.ome.tiff
Reading IFDs
Populating metadata
Initialization took 14.904s

Reading core metadata
filename = /allen/aics/microscopy/PRODUCTION/PIPELINE_4_OptimizationAutomation/3500000794/ZSD3/100X_zstack/converted/3500000794_100X_20170404_4-Scenes-18_F07_P24.ome.tiff
Series count = 1
Series #0 :
   Image count = 490
   RGB = false (1)
   Interleaved = false
   Indexed = false (false color)
   Width = 924
   Height = 624
   SizeZ = 70
   SizeT = 1
   SizeC = 7
   Thumbnail size = 128 x 86
   Endianness = intel (little)
   Dimension order = XYCZT (certain)
   Pixel type = uint16
   Valid bits per pixel = 16
   Metadata complete = true
   Thumbnail series = false
   -----
   Plane #0 <=> Z 0, C 0, T 0
   Plane #243 <=> Z 34, C 5, T 0
   Plane #244 <=> Z 34, C 6, T 0
   Plane #245 <=> Z 35, C 0, T 0
   Plane #246 <=> Z 35, C 1, T 0
   Plane #247 <=> Z 35, C 2, T 0
   Plane #489 <=> Z 69, C 6, T 0


Reading global metadata

Reading metadata


The CommandLineImporter is able to process it fine with 8G - I think I need to increase memory for blitz, fulltext, pixeldata - not sure which one will need more, as I see all services active and CommandLineImporter is simply waiting for the processing there to be done, I think...

Here is a typical run:

Code: Select all

omero@prd-aics-dcv-006-omero_server:/omero$ JAVA_OPTS=-Xmx8G /omero/OMERO.server/bin/omero import -s aics-06 -p 4064 -u omero -w omero -- --transfer=ln_s /allen/programs/allencell/data/proj0/bd/e357e57bee6d4bc39080122978745bbdCreated session 4d45e811-9552-4577-b494-d5bd7de447fa (omero@aics-06:4064). Idle timeout: 10 min. Current group: guest
2017-09-22 17:28:03,095 225        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.3.3-ice36-b63
2017-09-22 17:28:03,110 240        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.5.2 revision: 4c2f2b00a12aa8a3894c54459a4cf02d5b02e161 date: 23 August 2017
2017-09-22 17:28:03,157 287        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2017-09-22 17:28:03,161 291        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2017-09-22 17:28:03,459 589        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2017-09-22 17:28:25,023 22153      [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 21548ms. (21564ms total) [0 unknowns]
2017-09-22 17:28:26,251 23381      [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.3.4. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
2017-09-22 17:28:26,295 23425      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to aics-06:4064
2017-09-22 17:28:26,663 23793      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2017-09-22 17:28:26,999 24129      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.3.3
2017-09-22 17:28:27,000 24130      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.3.3-ice36-b63
2017-09-22 17:28:27,000 24130      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_144
2017-09-22 17:28:27,000 24130      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2017-09-22 17:28:27,001 24131      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2017-09-22 17:28:27,001 24131      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.4.0-81-generic
2017-09-22 17:28:27,097 24227      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2017-09-22 17:28:27,510 24640      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2017-09-22 17:28:27,527 24657      [      main] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /allen/programs/allencell/data/proj0/bd/e357e57bee6d4bc39080122978745bbd...
2017-09-22 17:28:27,611 24741      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /allen/programs/allencell/data/proj0/bd/e357e57bee6d4bc39080122978745bbd
2017-09-22 17:28:32,430 29560      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /allen/programs/allencell/data/proj0/bd/e357e57bee6d4bc39080122978745bbd
2017-09-22 17:28:34,971 32101      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2017-09-22 17:28:35,046 32176      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 3999
Sep 22, 2017 5:29:34 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
Sep 22, 2017 5:31:04 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
Sep 22, 2017 5:35:34 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.



I will upload this new ome.tiff to the q/a location.

[thusharaw]

Further update:
With the following changes to the heap allocations I was able to import this image:

Code: Select all

omero@prd-aics-dcv-006-omero_server:/omero$ OMERO.server/bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx4206m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'percent': '25'})
indexer=-Xmx2019m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'percent': '12'})
pixeldata=-Xmx4206m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'percent': '25'})
repository=-Xmx1682m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
omero@prd-aics-dcv-006-omero_server:/omero$

[rleigh]

The OME-XML is valid, but there is what looks like a problem with some of the annotations:

Code: Select all

        <Device Adcs="&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-16&quot;?&gt; #13; #10;&lt;ArrayOfRtAdcConfiguration xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;&gt; #13; #10;  &lt;RtAdcConfiguration&gt; #13; #10;    &lt;BaseConfiguration&gt; #13; #10;      &lt;Address&gt;176&lt;/Address&gt; #13; #10;      &lt;Index&gt;0&lt;/Index&gt; #13; #10;      &lt;InterfaceBoard&gt;0&lt;/InterfaceBoard&gt; #13; #10;      &lt;Name&gt;Adc_1&lt;/Name&gt; #13; #10;      &lt;RealtimeIndex&gt;176&lt;/RealtimeIndex&gt; #13; #10;   
...


It looks like the XML is being escaped as text rather than being processed as actual XML. This isn't the case for all of the annotations, or even the entire annotations with the escaping, so likely a bug in the handling of some sub-elements.

Regarding the memory usage, I've checked with bio-formats and it requires a JVM size of 1.5g to be able to process the metadata which is indeed quite excessive. A large part of the reason for this is the deserialisation of this XML into a DOM tree, the application of XSL transforms (if needed) and then the conversion of the DOM tree into a tree of OME-XML model objects. This does have a cost, but it's possible we can improve the resource usage. I have created a trello card to track this.


Kind regards,
Roger