CNDB-17829: Add possibility to configure the Digest file checksum type

[cbornet]

What is the issue

https://github.com/riptano/cndb/issues/17829

What does this PR fix and why was it fixed

This PR adds the possibility to configure the type of checksum written together with SSTables.
The config property sstable_digest_type can be set to CRC32, CRC32C and CRC64NVME (default CRC32). Respectively, a Digest.crc, Digest.crc32c or Digest.crc64nvme file will be written when an SSTable is committed to disk.

PR in CNDB: https://github.com/riptano/cndb/pull/17907

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR and ticket in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the IBM copyright header instead of the Apache License one (no DataStax copyright any longer)

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cbornet]

Benchmark results on an AWS writer node:

     [java] Benchmark                             (bufferSize)  Mode  Cnt     Score   Error  Units
     [java] ChecksumBench.benchAwsCrtCrc64nvme              31  avgt    5   125.886 ? 0.548  ns/op
     [java] ChecksumBench.benchAwsCrtCrc64nvme             131  avgt    5   135.875 ? 0.279  ns/op
     [java] ChecksumBench.benchAwsCrtCrc64nvme             517  avgt    5   165.602 ? 0.349  ns/op
     [java] ChecksumBench.benchAwsCrtCrc64nvme            2041  avgt    5   275.670 ? 0.357  ns/op
     [java] ChecksumBench.benchCrc32                        31  avgt    5    18.024 ? 0.112  ns/op
     [java] ChecksumBench.benchCrc32                       131  avgt    5    19.346 ? 0.029  ns/op
     [java] ChecksumBench.benchCrc32                       517  avgt    5    40.198 ? 0.039  ns/op
     [java] ChecksumBench.benchCrc32                      2041  avgt    5   115.017 ? 1.180  ns/op
     [java] ChecksumBench.benchCrc32c                       31  avgt    5    16.405 ? 0.056  ns/op
     [java] ChecksumBench.benchCrc32c                      131  avgt    5    14.802 ? 0.046  ns/op
     [java] ChecksumBench.benchCrc32c                      517  avgt    5    33.205 ? 0.067  ns/op
     [java] ChecksumBench.benchCrc32c                     2041  avgt    5   110.146 ? 1.221  ns/op
     [java] ChecksumBench.benchMd5                          31  avgt    5   300.002 ? 0.836  ns/op
     [java] ChecksumBench.benchMd5                         131  avgt    5   543.571 ? 0.972  ns/op
     [java] ChecksumBench.benchMd5                         517  avgt    5  1281.668 ? 1.968  ns/op
     [java] ChecksumBench.benchMd5                        2041  avgt    5  4238.461 ? 6.114  ns/op
     [java] ChecksumBench.benchPureJavaCrc64nvme            31  avgt    5    68.671 ? 0.097  ns/op
     [java] ChecksumBench.benchPureJavaCrc64nvme           131  avgt    5   190.400 ? 2.529  ns/op
     [java] ChecksumBench.benchPureJavaCrc64nvme           517  avgt    5   720.848 ? 6.939  ns/op
     [java] ChecksumBench.benchPureJavaCrc64nvme          2041  avgt    5  2571.147 ? 3.881  ns/op

[michaelsembwever]

Is this planned to be upstreamed or always rebased ?
is there a PR for main-5.0 ?  (helps give a clue upfront about future rebasing costs – it touches a few method signatures)

(ftr, saying yes to upstreaming doesn't mean do, or block on, it now.  but it's worth creating the C* ticket and get the idea of this approach in first. it can also help smoke out critical questions from other sources…)

[cbornet]

My current feeling would be to not upstream as I'm not sure there would be an interest in the broader community for this. But I'm happy to do it if there is.
I'll do a PR on main-5.0 as well, no problem.

[michaelsembwever]

My current feeling would be to not upstream as I'm not sure there would be an interest in the broader community for this.

Can you elaborate on this (please). (No objection, just appreciate having your reasoning, logical or intuition, noted.)
Is it that this is (most likely) only applicable post CEP-36: Make Cassandra's data store on Object storage(Cloud Storage) ?

[cbornet]

Indeed, in the context of this CEP, that's interesting. Thanks for pointing that out!

[cbornet]

PR on main-5.0 branch: #2442

[Copilot]

Pull request overview

This PR introduces a configurable checksum algorithm for SSTable digest files, expanding beyond the existing CRC32 digest to support CRC32C and CRC64-NVMe. The goal is to allow operators to select the digest checksum type via a new config property, and have SSTable commit/verification logic read/write the corresponding digest component.

Changes:

• Add sstable_digest_type configuration (CRC32 default, plus CRC32C and CRC64NVME) and route SSTable digest writing to the selected digest component filename.
• Extend checksum support (ChecksumType) with CRC32C and CRC64NVME (AWS CRT-backed when available, otherwise pure-Java CRC64NVME).
• Update verification / validation paths and add new unit + distributed tests and a microbenchmark for the new checksum types.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/java/org/apache/cassandra/config/Config.java	Adds SSTableDigestType enum and sstable_digest_type config field.
src/java/org/apache/cassandra/config/DatabaseDescriptor.java	Adds getter/setter for the new digest type config.
src/java/org/apache/cassandra/utils/ChecksumType.java	Adds CRC32C/CRC64NVME checksum types and AWS CRT presence detection.
src/java/org/apache/cassandra/utils/PureJavaCRC64NVME.java	Introduces a pure-Java CRC64-NVMe implementation.
src/java/org/apache/cassandra/io/sstable/format/SortedTableWriter.java	Selects digest file component + checksum type based on config when constructing data writers.
src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java	Updates the Big format writer’s component set for digest component handling.
src/java/org/apache/cassandra/io/sstable/Component.java	Adds new digest component types for CRC32C and CRC64NVME.
src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java	Adds digest-validator discovery across digest variants.
src/java/org/apache/cassandra/io/util/DataIntegrityMetadata.java	Generalizes digest validation to accept digest file + checksum type.
src/java/org/apache/cassandra/io/util/ChecksumWriter.java	Generalizes full-file checksum computation to use the selected checksum type.
src/java/org/apache/cassandra/io/util/ChecksummedSequentialWriter.java	Threads configured checksum type into digest writing for uncompressed data files.
src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java	Threads configured checksum type into digest writing for compressed data files.
src/java/org/apache/cassandra/io/compress/EncryptedSequentialWriter.java	Updates ChecksumWriter construction with explicit checksum type.
src/java/org/apache/cassandra/db/compaction/Verifier.java	Switches verification to use the digest-validator discovered from available digest component.
src/java/org/apache/cassandra/db/streaming/ComponentManifest.java	Includes new digest variants in the set of streamable components (filtered by existence).
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java	Adds new digest variants to supported/streaming component sets.
src/java/org/apache/cassandra/io/sstable/format/trieindex/TrieIndexFormat.java	Adds new digest variants to supported/streaming component sets for BTI format.
test/unit/org/apache/cassandra/utils/PureJavaCRC64NVMETest.java	Adds unit tests for the pure-Java CRC64NVME implementation.
test/unit/org/apache/cassandra/db/VerifyTest.java	Adds verification tests for CRC32C/CRC64NVME digest files.
test/distributed/org/apache/cassandra/distributed/test/SSTableDigestTest.java	Adds distributed test coverage for digest component selection by config.
test/microbench/org/apache/cassandra/test/microbench/ChecksumBench.java	Adds JMH benchmarks for CRC32C/CRC64NVME and MD5.
test/unit/org/apache/cassandra/io/util/ChecksummedSequentialWriterTest.java	Updates tests for the new ChecksummedSequentialWriter constructor signature.
test/unit/org/apache/cassandra/io/util/ChecksummedRandomAccessReaderTest.java	Updates tests for the new ChecksummedSequentialWriter constructor signature.
test/unit/org/apache/cassandra/io/sstable/ComponentTest.java	Adds parsing test coverage for new digest component representations.
test/unit/org/apache/cassandra/db/ScrubTest.java	Updates scrub test to use maybeGetDigestValidator().
src/java/org/apache/cassandra/cache/AutoSavingCache.java	Updates cache writer creation for new ChecksummedSequentialWriter constructor.
build.xml	Adds optional/provided dependency on AWS CRT (aws-crt) for CRC64NVME.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cbornet]

If software.amazon.awssdk.crt.checksums.CRC64NVME is found but new CRC64NVME() fails, it's probably better to let it crash than fallback...

[plpesvc-ds]

❌ Build ds-cassandra-pr-gate/PR-2434 rejected by Butler

---

5 regressions found
See build details here

---

Found 5 new test failures

Test	Explanation	Runs	Upstream
o.a.c.config.DatabaseDescriptorRefTest.testDatabaseDescriptorRef (compression)	REGRESSION	🔵🔴	0 / 30
o.a.c.index.sai.QueryContextTest.testWideTableWithStatics[eb] (compression)	REGRESSION	🔴🔵	0 / 30
o.a.c.index.sai.cql.LuceneUpdateDeleteTest.testOverwriteWithTTL[fa] (compression)	REGRESSION	🔵🔴	0 / 30
o.a.c.index.sai.cql.VectorCompaction100dTest.testOneToManyCompactionTooManyHoles[version=ec enableNVQ=false]	REGRESSION	🔴⚪	0 / 30
o.a.c.index.sai.cql.VectorSiftSmallTest.testRerankKZeroOrderMatchesFullPrecisionSimilarity[version=ed enableNVQ=true enableFused=false]	NEW	🔴🔴	0 / 30

---

Found 9 known test failures