Tools around or integrating with Rucio
These tools currently hardcode some settings, but are in general usable to check usage and quotas or adjust quotas in an automated way.
Most of them require a valid VOMS proxy certificate and correct environment setup (including a working version of Rucio).
Function which can be used in your shell environment to get a VOMS proxy certificate and set up the X509_USER_PROXY environment variable correctly.
It also sets up the Rucio home correctly, currently for the VOs ATLAS and BELLE.
A rucio_homes directory is part of this repository with the configurations for these VOs. Note the differing LICENSE (see COPYRIGHT).
Shell script massaging the output from get_accs_usage.py to produce output like:
name mail files size/TB quota/TB
cernid [email protected] 10652 120.728 170
The usage and quota are measured in the same way, i.e. if multiple people order the same datasets, they will all get this accounted to their personal quota (but storage will only have the sata once). This means the sum of the sizes is usually larger than the total storage consumption.
Shell script massaging the output from get_accs_usage2.py to produce output like:
name mail size/TB perm/TB temp/TB
cernid [email protected] 79.4263 79.4263 0
The first column matches the total usage, derived from locks owned by the user to the configured RSE. The perm and temp columns are filled by checking for associated rules which target the given RSE, and summing up the bytes for rules which are permanent (i.e. expires_at is not set) and temporary (i.e. expires_at is set). This idea follows the approach outline in this issue.
Note that the first column is created by summarizing locks and the latter column is only filled if a matching rule targeting the RSE is found. This means that rules targeting an expression such as (type=LOCALGROUPDISK) may not be associated. For this reason, the first column does not have to match the sum of the latter two columns --- a mismatch indicates that an expression is used in the rule (or transfers are still ongoing).
Tool to extract the data also shown at on the Pleiades Monitor. Output like:
Accounts Space total Space persistent Space temporary
cernid 69.316 69.316 0.000
cernid,cernid 44.992 24.992 20.000
Data ordered by several persons or multiple times by one person is shown with grouped CERN IDs, so this does not match the Rucio quota calculation, but the sum of the sizes should match the total storage consumption.
Note that persistent relates to data placed by rules which do not have an expiration set, while temporary relates to data placed by rules with expiration. See this PR for details.
Same as pleiades_quick.sh, but creates a simple CSV output with the cokumns:
Accounts;Space persistent;Space temporary
for cunsmption by other tools. Can e.g. be visualized with pleiades.html.
Read a CSV file produced e.g. by pleiades_csv.sh and displays it graphically via Javascript.
Can be inspected via GitHub pages here.
Tool which gets and adjusts quotas. Rules to be documented (basically, it rounds up usage after adding a safety margin, and sets a minimum default quota different for users and admins).
This tool has parameters:
| Parameter | Description |
|---|---|
| -v | Verbose mode, strongly recommended. |
| --dry-run | Perform a dry-run, i.e. only show what would be done. Strongly recommended before applying changes. |
Simple tool cross-checking whether users not in a cloud have quota on an RSE and users in a cloud are missing quota on an RSE (to cross-check e.g. LOCALGROUPDISK quotas).