Support multiple credentials in proxy config

[arttor]

Problem statement

Throughout this document, a user is an S3 identity — one access key + secret key.

Today the worker and the proxy use the same credential configuration. For each storage you list one credential per user:

storages:
  main:
    credentials:
      admin: { accessKeyID: …, secretAccessKey: … }
  backup:
    credentials:
      admin: { accessKeyID: …, secretAccessKey: … }

This user is also the identity that routing and replication policies are keyed by — it is used everywhere in Chorus to set up and track a replication.

This works well for the worker

The worker only needs one full-access (admin) user per storage: full read on the source to copy any object, full write on the destination to write it. One credential per storage is enough, and the worker never needs the credentials of the individual users who upload data.

It does not work for the proxy

The proxy is the S3 endpoint that clients connect to instead of the storages. For every incoming request it must:

1. Authenticate the caller — verify the request's signature, which requires that user's secret key.
2. Find the policy — routing and replication are keyed by user, so it maps the request to its user.
3. Forward the request — route it to a backend storage and re-sign it for that storage, which requires that user's keys on the target storage (the S3 signature is bound to the hostname, so the request cannot be forwarded unchanged).

The mechanism itself works today. The limitation is the config structure: it allows only one credential per replication, so a replication corresponds to the admin user alone. There is no way to set the other accessing users' credentials as part of that same replication — so when one of them modifies an object, the proxy cannot map it to the replication.

Example: replication is configured for the admin user, but admin's buckets can also be modified by other users through the proxy. When one of those users changes an object, we currently cannot map that change to the replication policy.

Keys may differ between storages

Whether a user can have the same secret/access keys on every storage depends on the backend:

• Ceph RGW lets you set arbitrary access/secret keys, so for RGW → RGW you can provision every storage with identical keys per user.
• AWS generates keys for you and does not let you choose them, so the same user will have different keys on different AWS storages.

Migration to Ceph is the main use case, but the config must still support the case where the keys differ across storages.

What the proxy credential config needs

What the worker calls a user actually corresponds to a tenant — an account. The proxy, which deals with many users, should let you set many user credentials grouped under the same tenant. Policies are keyed by tenant, so one policy covers all of its users. Concretely the proxy config needs to:

• a) hold many user credentials per tenant;
• b) map each user credential to its tenant, and to its equivalent credential on the other storage (an alias), for backends that do not allow setting arbitrary keys;
• c) since migration to RGW (where keys can match across storages) is the main use case, we can consider a simplified form for when a user's credentials are identical on all storages.