User quotas meanwhile possible?

[Sethur]

Hi,

somewhere in the OMERO docs I indistinctly remember to have read that there is no way to set user quotas in OMERO at this point.

Do you have any plans to fix that in the near future? We would certainly need this feature for our implementation, since we cannot allow a certain type of users to upload large amounts of data.

Cheers,

Tristan

[cblackburn]

Hi Tristan,

We have no immediate plans to implement quotas within OMERO though it is something we would like to look at.

You could certainly use the OMERO 5 Managed Repository to implement quotas to some degree via some scripted solution. The template path, which determines the location for uploaded files at import time, can be configured to allow for user or group level monitoring of used drive space. See:

http://www.openmicroscopy.org/site/support/omero5/sysadmins/fs-upload-configuration.html

for some information on this. Note that this page will be significantly improved on release of OMERO 5.0.3.

There are a few caveats though.

If users import big images then pyramid files may be generated following import. These files are not stored under the ManagedRepository but rather under Pixels. If users upload file annotations then these are stored under Files. Also, if you use some form of in-place import then you would need to factor that into any calculations.

Cheers,

Colin

[Sethur]

Hi Colin,

I was considering using direct file system quotas to enforce quotas on OMERO, but this would probably lead to a cryptic error message when a user tries to exceed his quota. Also, users would not be able to check how much space they have already used up.

If you think about the amount of data that some long-term projects might put on a CFU's servers over time, especially when everything is uploaded without regard to the merit of the data, I think OMERO internal quotas are an highly important feature. Enforcing quotas has multiple benefits:

a) Users who pay less or no money (because of special arragements) for CFU services can be restricted in the amount of storage space they are provided with. Note that every MB of space that a user occupies normally also has to be backed-up, so providing space to users is really a cost issue and it is not a good idea to have unlimited space for everybody.

b) Having a quota in place will make users think about what to upload and what to keep on the system. Some data can easily be identified as useless or redundant. Currently, such data will never be deleted since space is for free. This is especially problematic, since adding more and more space to an existing OMERO implementation is already only possible up to a certain point since (if I am not mistaken) an OMERO Managed Repository has to reside on a single partition.

c) If necessary, individual users can be prevented from uploading any more data while still giving them access to their current projects/datasets.

Cheers,

Tristan

[jmoore]

Hi Tristan,

This is especially problematic, since adding more and more space to an existing OMERO implementation is already only possible up to a certain point since (if I am not mistaken) an OMERO Managed Repository has to reside on a single partition.

If you include a mount-point at the top of your managed repository as described during the FS Workshop in Paris, there shouldn't be a limit to how many disks you can mount currently. That being said, support for multiple managed repositories is certainly not unthinkable.

The same certainly goes for quotes, too! We understand the utility, but simply haven't had the capacity to implement it.

Cheers,
~Josh.

[kennethgillen]

Hi Folks,

This is especially problematic, since adding more and more space to an existing OMERO implementation is already only possible up to a certain point since (if I am not mistaken) an OMERO Managed Repository has to reside on a single partition.

I'd also like to add that Linux LVM [1] can be used to abstract the underlying partition layout from OMERO. We've had good success using LVM and XFS at CLS Dundee on a 16TB (and growing) OMERO implementation. New partitions can be added, the LVM logical volumes extended and the filesystem grown fully on-line with no downtime.

Of course, this doesn't help with respect to Quotas, but just thought I'd mention how we at Dundee have addressed that particular issue of growth of an OMERO server.

[1] http://tldp.org/HOWTO/LVM-HOWTO/

Best,

Kenny

[Sethur]

Hi Kenny,

thanks for the hint with LVM, did you manage to extend your LVM partition on the fly? Or do you need to copy everything to somewhere else before you can extend?

@Josh: You are right, this would certainly be a solution after moving some files and updating the DB. Can you point me to a manual on how to update the DB for a new storage scheme? Also, can you tell me what the "X" in driveX means? How does the system know which drive to choose?

Best,

Tristan

[kennethgillen]

Hi Tristan,

did you manage to extend your LVM partition on the fly?

With LVM, we have indeed added new partitions to the 'volume group' on the fly, and resize 'logical volume' then the file system where the OMERO binary repository's mount point is also on the fly.

Best,

Kenny

[Sethur]

Thanks Kenny, that's certainly good to know.

[jmoore]

You are right, this would certainly be a solution after moving some files and updating the DB. Can you point me to a manual on how to update the DB for a new storage scheme?

To test this scenario, we added a command which will move a single fileset in 5.0.3:

Code: Select all

bin/omero fs rename $FILESET_ID


This can be done in a loop to achieve what you're asking for, but post-release we'll also working on a SQL template that will do the same thing in bulk for the "drive-addition" use case.

Also, can you tell me what the "X" in driveX means? How does the system know which drive to choose?

Whatever drive name you choose must be configured in the server. E.g:

Code: Select all

bin/omero config set omero.fs.repo.path drive-002/...
bin/omero admin restart


At that point, all new imports will go to "drive-002". The same procedure would have to be followed to later move to drive-003, etc.

~J