[BUG] Diff check reports inconsistency due to multipart ETag differences after successful replication

[sarwottamdev]

Bug Report

Description

Chorus reports a bucket pair as not consistent after replication has successfully completed, even though the object content is identical on both source and destination buckets.

The diff report identifies differences based on object ETags, but manual verification of the object data shows matching MD5 checksums on both endpoints.

Expected Behavior

After replication reaches 100% completion and the source and destination objects contain identical data, the diff report should mark the bucket pair as consistent.

If multipart ETag differences are expected between source and destination implementations, the diff report should either account for this or provide a way to distinguish ETag differences from actual data inconsistencies.

Actual Behavior

Replication completes successfully:

[root@sc-container-vm ceph-migration-tool]# chorctl --address localhost:9670 repl

NAME                                              PROGRESS                 OBJECTS     EVENTS     LAG       PAUSED     AGE       ARCHIEVED     HAS_SWITCH
root:main:test-bucket->follower:test-bucket       [##########] 100.0 %     12/12       17/17      0s        false      1h25m     false         -

However, the diff report still shows:

[root@sc-container-vm ceph-migration-tool]# chorctl --address localhost:9670 diff check main:test-bucket follower:test-bucket --user=root
Diff check has been created.

[root@sc-container-vm ceph-migration-tool]# chorctl --address localhost:9670 diff report main:test-bucket follower:test-bucket

CHECK
READY:        true
QUEUED:       3
COMPLETED:    3
CONSISTENT:   false
VERSIONED:    false
IGNORE SIZES: false
IGNORE ETAGS: false

and reports the following differences:

PATH                SIZE          ETAG                                    follower     main
/big_file-1.txt     419430400     17f3dd8971fed8446c62b52f7885494f-25     ✓            X
/big_file.txt       419430400     72675617ef314026526051b4300b3e23-50     X            ✓
/big_file.txt       419430400     5fc013e1a83186d4dee4ca43966d9dbb-25     ✓            X
/big_file-1.txt     419430400     a7ff6018a45bbb18f4f73278cb163f7a-50     X            ✓

The overall status remains:

[root@sc-container-vm ceph-migration-tool]# chorctl --address localhost:9670 diff

STORAGES                                   STATUS             STATS
follower:test-bucket, main:test-bucket     not consistent     3/3

However, downloading the same object from both source and destination endpoints and calculating the MD5 checksum produces identical results:

aws s3 cp s3://test-bucket/big_file-1.txt - \
  --endpoint-url http://10.64.87.202 | md5sum

7121f2156552bebf34423cb2de4ee978  -

aws s3 cp s3://test-bucket/big_file-1.txt - \
  --endpoint-url http://10.64.87.207:8000 | md5sum

7121f2156552bebf34423cb2de4ee978  -

This indicates that the object content is synchronized even though Chorus reports the bucket pair as inconsistent.

Steps to Reproduce

1. Configure Chorus replication between two Ceph RGW buckets.
2. Upload large multipart objects to the source bucket.
3. Wait for replication to complete successfully.
4. Run a diff check:

chorctl --address localhost:9670 diff check main:test-bucket follower:test-bucket --user=root

5. View the diff report:

chorctl --address localhost:9670 diff report main:test-bucket follower:test-bucket

6. Observe that Chorus reports:

CONSISTENT: false

7. Download the same object from both source and destination endpoints and compare the checksums:

aws s3 cp s3://<bucket>/<object> - --endpoint-url <source-endpoint> | md5sum

aws s3 cp s3://<bucket>/<object> - --endpoint-url <destination-endpoint> | md5sum

8. Observe that the checksums match even though the diff report shows inconsistencies.

Environment

• Chorus Version: chorctl version "development" (Built on "not set" from Git SHA "not set")
• S3 Provider: Ceph RGW
• Chorus Setup: Docker Compose
• Other Details:
  • Redis version: 7.4.10
  • Replication shows 12/12 objects and 17/17 events processed.
  • Replication lag is 0s.

Additional Context

The issue appears to affect multipart-uploaded objects where source and destination ETags differ (for example, multipart suffixes -25 vs -50) even though the resulting object content is identical.

This may indicate that the diff-check logic relies on ETag comparison and can produce false-positive inconsistencies when multipart ETags differ between source and destination implementations.

From the reported output:

/big_file-1.txt     419430400     17f3dd8971fed8446c62b52f7885494f-25     ✓            X
/big_file-1.txt     419430400     a7ff6018a45bbb18f4f73278cb163f7a-50     X            ✓

and

/big_file.txt       419430400     5fc013e1a83186d4dee4ca43966d9dbb-25     ✓            X
/big_file.txt       419430400     72675617ef314026526051b4300b3e23-50     X            ✓

the object sizes match, while only the multipart ETags differ. The actual object content verified via MD5 is identical on both source and destination systems.

[aiivashchenko]

Hi @sarwottamdev, this is a known limitation that, as you already mentioned, comes from the way how etags are computed for multipart uploads.
As it is Ceph, I believe that suffixes correspond to the amount of parts client and server have negotiated for the upload, 25 in first case and 50 in another.
In case of multipart upload, iirc, etag is computed as md5 sum of etags for each part, therefore they differ.

Name/key, size and etag are parts of data available from ListObjectsV2 method for comparison.
Suppose, anything else would require a deeper inspection of an object.
First option is to HEAD the object and compare checksums if present. Two problems here: 1) checksums for multipart uploads are usually also composite, means will share similar flaw, 2) considering that call is made against 3rd party S3 API, it can be billable.
Second option would be to compare objects byte-to-byte. Bandwidth heavy, long, expensive and pretty fragile, especially for big buckets.
Both are considered as possible future improvements, but there is no timeline.

Currently, to overcome this, diff offers two flags, to skip etag and size checks if needed.