Manage and Organise

[jhehl]

Dear all,

just started today with OMERO and I have a question regarding data organisation. I know that one have to create projects and datasets. Is it also possible to create subprojects and subdatasets in projects and datasets to get a broader hierachical scale?

Thanks

Joachim

[jmoore]

Hi Joachim,

it's not possible to create possible to create sub-datasets or projects. There's a strict Project->Dataset->Image hierarchy. Instead, an image can be in multiple datasets, and datasets can be in multiple projects. So, if you were planning on making the structure:

Code: Select all

MyProject/
  Controls/
     A
     B
   DrugX/
     C
     D


You could instead use either:

Code: Select all

MyProject/
  Controls-A
  Controls-B
  DrugX-C
  DrugX-D


or have

Code: Select all

MyProject/
  Controls
  DrugX
  A
  B


with the some images in A & Controls, B & Controls, etc.

Hope that helps.
~Josh

[albot]

Hi,

I'd like to comment on this rather old post. Recently, we have started a pilot project of OMERO at VIB.
Pilot users were rather to very positive.

One major remarks was the fact that there is only one project level.
All of our pilot users find it interesting to have another hierarchical level above the projects.
I can of course understand that the project - dataset - image hierarchy is governed by your data model.

Wouldn't there be a way to add another type of level above the project ?

Cheers,
Alex

[dpwrussell]

At first I thought that clinging on to methodologies of using deep nested directory structures was what was causing objections here, but once I saw how much data people were collecting over time, I tend to agree that they just wont be able to keep it organised with the current level of nesting.

If they have hundreds of datasets in projects spanning years with derivative datasets this is going to get messy fast with only 2 levels. I know the database is very happy with that, but the human trying to use the data isn't so happy.

Also, as purely a visual aid, working sets would be nice in insight and the web client, like those in eclipse.

[wmoore]

I'm in two minds about whether extra levels would be a good idea (quite apart from the amount of code that would need changing if another container became the top-level one).

In ISA-TAB http://isa-tools.org/ they use Investigation, Study, Assay as a 3-level containers. You could equate Study & Assay to our Project & Dataset, suggesting that a 3 levels is something of a useful number / convention. Adding a 3rd level to OMERO is not like making a file-system and could be of benefit.

One way you can emulate this to a certain extent in the current OMERO clients is to add Tags to Projects to get an extra level of hierarchy (and even Tag-sets to the Tag for yet another level). The only limitation is that we don't have a way of filtering Tags for ones that are used on Projects vv other Tags that are used on Images etc. But if adding levels to the P/D/I hierarchy (by tagging Projects) is your priority, then this would be a good way to go.

Other ways that we are considering for managing large amounts of data is to search / browse by other means. E.g. show my latest work (like facebook) or browse by a bunch of other criteria (E.g. iTunes - No file system but you can have Artist, Album, Genre etc.).

Perhaps our current failing (the reason so many people want more levels) is that they're used to them, AND so far we haven't really worked on these alternatives very much. There's not many times when you really want to see ALL your data at once. Or am I missing something?

Thanks for the feedback. Any more suggestions etc would be most welcome...

Will.

[albot]

Hi,
I just received some feedback from our colleagues:

citation:
"I tried to use tags to add another layer in the project hierarchy and I'm quite happy with it. What I did was the following: I have a project called "candidates analysis" and in there several data sets that correspond to different types of experiments. I'm running each of these experiments on several mutants, and every time I take pictures for both control and mutant. So I simply imported all the pictures for a given experiment in the same data set, regardless of the genotype I tested, and I tagged subsets of pictures to mark them with the mutant they correspond to (I could also have tagged them with the day the experiment was performed, or anything else). I was happy to see that the fact several pictures have the same name (for example, the control is always the same) wasn't a problem. Then I just need to go to the tag tab to retrieve only the subset of pictures that correspond to one experiment done on one mutant. I can also add other tags to some of these pictures, for example to note which one I should use to build a figure.
So, in brief I think that the OMERO developer was right and that the only concern here is to get used to this new way of sorting the data."

So, it appears that people might be able to adapt their way of working. Indeed, the strong point of OMERO is the tagging so that you can filter and sort by tags.
This is usually one of the reasons why people work with hierarchies where after a while they don't find their way through.

Still, I'm quite positive about one additional level as Will suggested (Investigation) - this would give more flexibility to the OMERO users. By combining one additional level and tagging, you would have the best of two worlds.

Cheers,
Alex

[LCThompson]

I agree that tagging can be used as a viable workaround in some situations. However, choosing not to add more directory levels is making the decision for us. Why not just allow a few more levels, and then let each one of us decide what system makes the most sense for our needs?

I could absolutely make do with a well designed tagging system, but I am not building this for me. I am building it for collaborators and outsiders who need to be able to really simply and intuitively find what I need them to find. Furthermore, if we ever generate a set of images that could not possibly be logically arranged without more levels then I will be out of luck and really frustrated.

I absolutely understand that adjusting the data system would be difficult, so if this update never happens I doubt anyone would be angry. But if you guys could get this done it would be great!

[jburel]

Thanks for your valuable comments, I have linked the post to https://trac.openmicroscopy.org.uk/ome/ticket/10233

We will be shortly working on improving the tagging workflow. Hopefully that will facilitate some of the solutions described in the thread.

Thanks again.
Jmarie

[David]

Despite the fact that I come from a computing background that started in dos command line, and then windows, therefore the hierarchical folder structure makes sense in my head; I would advocate NOT adding more folders. The problem being that if you gave people an additional folder level, they will stick with the out-dated concept of folders, and the bigger problem is that they will eventually find that 3 levels is not enough and want a forth added, then a fifth then a sixth.

Instead I would say that having a combination of folders and tags is confusing and we should have one or the other not both. So get rid of folders altogether and only have tags thereby allowing/forcing people to embrace the tag methodology and have the left hand explorer panel show image objects based on “Smart Folders” that get populated dynamically based on tag filtering.

If implemented sensibly there is no reason that data cannot be represented that looks like the classic hierarchical folder view but under the bonnet it is implemented using smart views and can also have the added benefit of object existing in multiple “places” at once.

So to summarise, the answer is to have hierarchical tags not hierarchical folders.

Regards

David Strachan
Beatson Institute for Cancer Research