[FEATURE] Recommended production architecture for Chorus S3 Proxy, TLS, Ceph RGW VIPs, and rclone replication behavior

[saif-0987]

Hello,

We are evaluating Chorus for bucket-level replication between two Ceph RGW S3 storage systems.

Our intended architecture is:

S3 Client
  → HTTPS endpoint / TLS load balancer
  → Chorus S3 Proxy
  → Main Ceph RGW
  → Chorus Worker
  → Follower Ceph RGW

Current setup:

• Chorus Proxy listens on HTTP port 9669.
• Nginx terminates HTTPS on port 443 and forwards requests to 127.0.0.1:9669.
• Proxy authentication uses AWS Signature V4.
• Main and follower are configured as separate S3-compatible storage endpoints.
• Redis is used for replication tasks.
• We enable replication at bucket level using:

chorctl repl add \
  -u user1 \
  -b <bucket-name> \
  -f main \
  -t follower

Testing is successful:

• Client creates a bucket through the Chorus proxy.
• Client uploads objects through the proxy.
• Objects are written to main storage.
• Existing objects are replicated to follower after enabling replication.
• New objects uploaded through the proxy are replicated automatically.

We would appreciate clarification on the recommended production architecture:

1. TLS for the Chorus S3 Proxy

The official Docker Compose example exposes Chorus Proxy over HTTP on port 9669. Does Chorus support native TLS for its S3 endpoint?

If not, is the recommended deployment to place one of the following in front of Chorus?

• Nginx
• HAProxy

and when using a reverse proxy, are there any required settings beyond preserving:

• Host
• Authorization
• Original request URI/query string
• Request body streaming
  This is particularly important for AWS Signature V4 validation.

2. Ceph RGW endpoint design

For production, should the main and follower storage addresses point to stable RGW VIP/load-balancer endpoints rather than individual RGW daemon IPs?

3. Rclone replication behavior

We understand that Chorus uses rclone-related storage functionality internally, but we would like to understand the exact replication behavior.

For initial replication after running chorctl repl add:

• Does Chorus perform an operation comparable to rclone copy?
• Does it perform a full rclone sync?
• Or does it scan the source and create individual object-copy tasks?

For live writes through the proxy:

• Does the proxy first complete the write to main and then enqueue a replication event?
• Does the worker subsequently read that object from main and upload it to follower?
• Are deletes and overwrites processed as individual events?
• Is a full bucket scan avoided for each live event?

4. Writes that bypass the proxy

If an application writes directly to main RGW instead of through Chorus Proxy, will Chorus miss the live replication event?

Should production access therefore enforce:

Clients → Chorus Proxy only
Chorus Proxy/Worker → RGW VIPs

with direct RGW access restricted to Chorus and administrators?

5. High availability

For an HA deployment with multiple Chorus Proxy and Worker instances:

• Should a VIP/load balancer be placed in front of the proxy instances?
• Can all proxy and worker instances safely share the same Redis backend and storage configuration?
• Are there recommended load-balancer health checks for the Chorus S3 proxy?

Any recommended reference architecture or production deployment example would be very helpful.

Relevant documentation reviewed:
https://github.com/clyso/chorus/blob/main/service/proxy/README.md
https://github.com/clyso/chorus/blob/main/service/proxy/config.yaml
https://github.com/clyso/chorus/blob/main/docker-compose/README.md
https://github.com/clyso/chorus/blob/main/docker-compose/s3-credentials.yaml

Any suggestion or clarification would be appreciated.

Thank you.
Mohammad Saif

[arttor]

The official Docker Compose example exposes Chorus Proxy over HTTP on port 9669. Does Chorus support native TLS for its S3 endpoint?

Not at the moment, but we will be happy to accept contribution for this feature. For production setup helm chart provides option to configure ingress with tls.

For production, should the main and follower storage addresses point to stable RGW VIP/load-balancer endpoints rather than individual RGW daemon IPs?

There is no such requirement. Any s3-compatible endpoint is supported.

We understand that Chorus uses rclone-related storage functionality internally

Not anymore. Rclone was removed in #160 . Chorus using s3 client direcly to copy objects. We have our custom copy logic to retain orginal object versionID when syncing versioned bucket to RGW and some other features. Rclone ignores or modifies some data because it is designed to sync between any types of storages, not only S3. So it is not used in chorus anymore.

Does Chorus perform an operation comparable to rclone copy?

yes, but per object. Chorus lists all objects in soruce bucket to create syncObject task and copies object with tags and acl to destination if etag is not match.

Or does it scan the source and create individual object-copy tasks?

yes.

Does the proxy first complete the write to main and then enqueue a replication event?

yes. task created only after main returned 2xx response to proxy. worker will process all initial migration tasks and then event tasks created by proxy.

Are deletes and overwrites processed as individual events?

yes but there is only one type of event for object payload modification. event contains only object name - it not says if object was added, updated or deleted. worker checks object in destination: if there is no object in source, it deletes also in dest; if object exists in soruce, it copies to dest if etag is different.
there is a separate event for PutObjecAcl or PutObjectTag - worker will compare only tags or only acls between src and dest in a similar way.

Is a full bucket scan avoided for each live event?

yes - event scoped to object name. only bucket create/delete events are scoped to bucket.

If an application writes directly to main RGW instead of through Chorus Proxy, will Chorus miss the live replication event?

yes. for this setup you can configure chorus agent with s3 bucket notifications instead of proxy.

Can all proxy and worker instances safely share the same Redis backend and storage configuration?

they must use the same redis instance and same config (see helm setup - it creates deployments with multiple replicas for worker and proxy). all proxy and worker instance are coordinated over redis by sharing the same replication and routing policy.

Any recommended reference architecture or production deployment example would be very helpful.

at the moment, we dont have such documentation. for production deployment it is better to check helm chart, but we will be happy to receive some feedback about your setup to improve documenation or deployment model.