Populate_Metadata script in 5.3.3

[dsudar]

Hi team,
I was trying the Populate_Metadata.py script on a 384 well plate with a csv file that starts:

Code: Select all

Well,Compound,Concentration
A01,Media,0
A02,Media,0
A03,Media,0
A04,Media,0
A05,Media,0
A06,Media,0
A07,Media,0
A08,Media,0
A09,Media,0
A10,Media,0
A11,Media,0
A12,Media,0
A13,Media,0
A14,Media,0
A15,Media,0
A16,Media,0
A17,Media,0
A18,Media,0
A19,Media,0
A20,Media,0
A21,Media,0
A22,Media,0
A23,Media,0
A24,Media,0
B01,Media,0
B02,leucovorin,20000
B03,Panotinib,137.17
B04,Panotinib,45.72
B05,Pertuzumab+Herceptin,1.23
B06,Oxalipatin,740.74
B07,Lenvatinib,1234.57
etc....


The errors I get say:

Code: Select all

WARNING:omero.util.populate_metadata:Skipping table row 0! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L
WARNING:omero.util.populate_metadata:Skipping table row 1! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L
WARNING:omero.util.populate_metadata:Skipping table row 2! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L

etc....


On another plate using the exact same csv annotation file, it doesn't give errors but nothing appears in the Tables tab for the wells or images in each well. For both plates, a bulk_annotations attachment does get created.

I believe this all worked fine under 5.2.8 and I recently upgraded both my production servers to 5.3.3 (and will upgrade to 5.3.4 shortly). Am I doing something wrong or is something different between 5.2.x and the 5.3.3?

Thanks,
- Damir

[dsudar]

Hi team,

Quick update on my query: turns out that I apparently had an unfortunate and incompatible combination of versions of PyTables, HDF5, numexpr, and/or numpy installed. Especially when installing and upgrading with pip, it is pretty easy to get into such a situation and there's not much that lets you know. Any thoughts how to check you have something compatible?

Anyway, by downgrading some of those pip packages, I was able to get the Populate_Metadata.py script to work correctly again.

Thanks,
- Damir

[manics]

Hi Damir

What operating system are you using? We've had problems with pytables/numpy/numexpr etc in the past. With recent distributions (e.g. CentOS 7) we recommend using the distribution packages where possible, and only using pip to install packages which are too old or non-existent.

Simon

[manics]

I've just had another though. pytables should have a built in test method:

Code: Select all

import tables
tables.test()


If most of the tests pass it's probably working.

[dsudar]

Hi Simon,

This is on Ubuntu 14.04 LTS. This server has been through many upgrades starting with being Ubuntu 12.04 LTS and running OMERO 4.4.5. So it can probably do with some cleaning up and indeed there has been some mix of apt-get installed packages and pip-installed ones. On my list of to-dos.

I actually did run the tables.test() and that passed even while the Populate_Metadata.py script was not doing what it should do. I'll have a close look at the Tables.log and Blitz.log to see if I find any clues.

Cheers,
- Damir