Welcome! These are a few scripts that we use at SURF to work with the Spider compute cluster.
ADA is being rewritten from Bash to Python. You can check out progress at https://github.com/sara-nl/dcache-pyclient. The Bash version (in this git repo) is no longer actively developed.
ADA is a client that talks to the dCache storage system API to work with data in dCache.
- List directory and file information
- List file checksums
- Rename, move and delete files and directories
- Work with labels on files: set labels, remove them, and search directories for files with labels
- Work with file metadata (extended attributes): set attributes, delete them, and search directories for files with certain attributes
- Stage files (restore from tape), check whether they are online or not
- Show your disk and tape quotas
- Subscribe to server-sent-events to set up automated workflows
- Upload and download files
Many of these operations can be done recursively. For authentication, ADA supports X509, tokens (macaroons and OIDC) and basic auth (username/password), depending on the dCache configuration.
When something goes wrong, Ada tries to provide clear, understandable and useful error messages, so you can help yourself, or otherwise your support team will be able to help you better.
With the --debug flag, Ada provides a wealth of information that may help you to troubleshoot problems. If you are a developer, this will help you to understand how to talk to the dCache API.
- ADA can upload and download files, but not in bulk, and it's not as efficient as Rclone.
- ADA depends on dCache. The dCache system you work with may have limitations that impact ADA.
- Ada is currently implemented in bash. Bash has limitations; for instance, trying to stage more than 1500 files in a single operation may fail.
ADA has been tested on Linux and MacOS. It is pre-installed and ready to use on the Spider compute cluster. If you want to use ADA elsewhere, you can clone this repository:
git clone https://github.com/sara-nl/SpiderScripts.git
cd SpiderScripts
Install dependencies (if not already installed on your system):
# MacOS
brew install jq rclone bash
# Redhat/Rocky/Alma
dnf install jq rclone
There are also optional dependencies to run tests and create macaroons:
brew install shunit2 (or "wget https://raw.githubusercontent.com/kward/shunit2/refs/heads/master/shunit2")
pip install pymacaroons
wget https://raw.githubusercontent.com/sara-nl/GridScripts/master/view-macaroon -P ada
wget https://raw.githubusercontent.com/sara-nl/GridScripts/master/get-macaroon -P ada
The commands now are only available in the installed folder. To make them accessible from any location on your computer do:
chmod +x ./ada
sudo ln -s ./ada /usr/local/bin/ada
sudo ln -s ./get-macaroon /usr/local/bin/get-macaroon
sudo ln -s ./view-macaroon /usr/local/bin/view-macaroon
Test with a new terminal and type in ada --help.
TODO
To test the installation, run:
tests/unit_test.sh
The unit tests will perform a dry-run, i.e. commands are not actually sent to the dCache API, but simply printed and compared to what is expected.
The integration test actually executes commands on the dCache API. Set up a configuration file tests/test.conf based on tests/test_example.conf. Then run:
tests/integration_test.sh
For an overview of the commands and options, run:
ada --help
Read further how to use ADA in the Spider Documentation.