DropBox import to Datasets

[frias]

Hi!

We generate quite a lot of images during one experiment, i.e. 150 images per condition. Is it possible to use the dropbox feature to import the images straight into specific data sets? I.e. by employing a directory structure like

USER/PROJECT/DATASET/IMAGES

und the Dropbox folder?

Moreover, is it possible to organize data sets in a tree like

PROJECT/EXPERIMENT1/CONDITION1
PROJECT/EXPERIMENT1/CONDITION2

and so on?

Best & many thanks,

Sebastian

[cblackburn]

Hi Sebastian,

To answer your second question first, the hierarchy is currently limited to just two levels, Project/Dataset.

On the first point, as shipped DropBox only imports images as orphaned images, but I've pushed a branch to github based on v4.4.4,

https://github.com/ximenesuk/openmicros ... ataset_4_4

which contains a modification to MonitorClientI that does a limited import to Dataset. It uses the first two directories following the username in the DropBox path, if they both exist, as a Project/Dataset pair, creating the Project and Dataset if necessary. It then imports the images in the folder Dataset into the corresponding Dataset.

This has not been robustly tested and doesn't cater for all use cases. However, it should give you some ideas on extending this class for yourself if you wish to.

As the only modified file is a Python file, fsDropBoxMonitorClient.py, it could be copied into dist/lib/python/ without the need to rebuild your server. But if you do build your own server then you will need to ensure this file is components/tools/OmeroFS/ if you want to rebuild with these changes.

If you do use this then any feedback would be very helpful. If you need any further help let me know.

Cheers,

Colin

[ppouchin]

Hi,


I tried this modification, but it doesn't find my project/dataset and creates new ones.
And then, when the import starts, I get this :

Code: Select all

serverStackTrace = "ome.conditions.SecurityViolation: User 2 is not a member of group 0 and cannot login

It's as if it's looking for the project/dataset in the root account, does not find them so creates new ones and then the user cannot upload to it...

[cblackburn]

Hi,

Would it be possible for you to zip up the log files (dist/var/log)and put them somewhere so that I can take a look?

Cheers,

Colin

[ppouchin]

Of course.

You can find the logs here :
http://srv-gred.u-clermont1.fr/~pouchinp/logs.tar.gz

The log level might be low though.

[cblackburn]

Thanks, I'll take a look and hopefully provide a solution later.

Cheers,

Colin

[frias]

Haven't tried yet, but your patch is exactly what I was looking for! I will stay tuned for the error to be fixed.

Great... and what a support!

Best,

Sebastian

[frias]

Is there a solution yet? I tried as well and could only produce crashes of dropbox. If of any help, I will provide the logs...

This feature would be so cool!!!

[cblackburn]

Hi,

Sorry for the delay but I am working on a solution. I'll post here once it is available.

Cheers,

Colin

[cblackburn]

Hi,

sorry for the delay but the script at:

https://github.com/ximenesuk/openmicroscopy/tree/dropbox_dataset_4_4

should now import into projects and datasets based on the first two subdirectories under the DropBox user name. It will create new projects and datasets on demand if they don't exist. The imports will be made into the user's default group. Note that as it is possible for a user to have identically named projects and datasets, the import will be made into the first one returned by the underlying query. This is still just an illustrative modification so please use with care and any feedback would be appreciated.

I've raised a ticket to examine this idea further and to perhaps include the group as an option,

https://trac.openmicroscopy.org.uk/ome/ticket/9899

Hopefully the ticket can be slated for inclusion in a release in the near future.

Cheers,

Colin