CNDB-18201: support streaming with mixed MS versions

[driftx]

What is the issue

Fixes #18201

What does this PR fix and why was it fixed

MessagingService.accept_streaming now accepts a range between VERSION_40 and current_version. Inbound streaming uses the negotiated version and outbound streaming stores it as a Netty channel attribute that control/file messages use for serialization.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR and ticket in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the IBM copyright header instead of the Apache License one (no DataStax copyright any longer)

[adelapena]

It seems testMixedMessagingVersionUncompressedStreamingDs12ToDs11 is failing.

How would this work when we have a mix of nodes/writers without this patch and the current messaging version, and writers without this patch and an upgraded messaging version? Should we wait until all the involved writers have this patch to bump the messaging version?

[driftx]

It seems testMixedMessagingVersionUncompressedStreamingDs12ToDs11 is failing.

It failed to reach schema agreement, I think it was an infra problem. The test passes for me locally.

How would this work when we have a mix of nodes/writers without this patch and the current messaging version, and writers without this patch and an upgraded messaging version? Should we wait until all the involved writers have this patch to bump the messaging version?

Yes, I think we need this patch in before we can bump the version.

[adelapena]

It seems testMixedMessagingVersionUncompressedStreamingDs12ToDs11 is failing.

It failed to reach schema agreement, I think it was an infra problem. The test passes for me locally.

For some reason the two Ds12ToDs11 variants consistently fail to reach schema agreement when I run them locally. But since they work locally for you and in CI, I guess it should be something related to my environment.

Is there a CNDB PR bumping the CC version to include this?

[driftx]

For some reason the two Ds12ToDs11 variants consistently fail to reach schema agreement when I run them locally. But since they work locally for you and in CI, I guess it should be something related to my environment.

If it's failing for you too, that concerns me. I looped it but could not reproduce, however I pushed an update that uses a slightly relaxed call for schema creation, let's see how that fares.

Is there a CNDB PR bumping the CC version to include this?

Not yet, let me create one after CI finishes on this update.

[adelapena]

If it's failing for you too, that concerns me. I looped it but could not reproduce, however I pushed an update that uses a slightly relaxed call for schema creation, let's see how that fares.

I was running it with J11, and with that Java version and the last changes it still consistently timeouts. However with J22 it passes most of the time, although I have still hit occasional timeouts while waiting for the schema agreement in one of those two tests. I have hit 2 failures in 7 runs. When they work they pass in ~9 seconds, so I'm not sure is a question of waiting for longer.

[driftx]

I was running it with J11, and with that Java version and the last changes it still consistently timeouts.

I switched to J11 and was able to loop 50 times.

However with J22 it passes most of the time, although I have still hit occasional timeouts while waiting for the schema agreement in one of those two tests. I have hit 2 failures in 7 runs. When they work they pass in ~9 seconds, so I'm not sure is a question of waiting for longer.

I can loop many times with J22 too, and it looks like it passed in CI this time. Not being a question of waiting longer makes me feel like there is meat on that bone though, any chance you can dig in and see what's hanging it? In the last commit I removed the requirement of a single schema version being present, now only requiring that the table is present everywhere, so if that is failing easily then something seems fundamentally wrong.

[adelapena]

So far I haven't found anything. Unfortunately I'm not very familiar with schema propagation. This is the output of a failed run: testMixedMessagingVersionStreamingDs12ToDs11_output.txt

I have run the test on unpatched main, only until the creation of the keyspace, and to my surprise it occasionally hits the same timeout, although much less often.

[driftx]

I added ds12 to ds12 tests, see if either of these fail the schema creation for you. If so, then I think we can safely eliminate the streaming as the cause (which already seems to be the case from your log too.)

[adelapena]

A few runs of the ds12 to ds12 tests pass for me.

I have been running the tests in idea. However, they pass dozen of times when running them in the command line, in a loop, with:

ant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.test.StreamingTest -Dtest.methods=testMixedMessagingVersionStreamingDs12ToDs11 -Dno-build-test=true

and

ant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.test.StreamingTest -Dno-build-test=true

So it might be some problem with my env, and the CI failure was just a normal timeout?

I have tried two different versions of IDEA. Did you run them on IDEA?

[driftx]

A few runs of the ds12 to ds12 tests pass for me.

Was that in IDEA?

So it might be some problem with my env, and the CI failure was just a normal timeout?

I think that's likely. I went back to the original commit using schemaChange since that is used widely. Let's see how CI fares this time.

I have tried two different versions of IDEA. Did you run them on IDEA?

Yes, I could not reproduce the failures in IDEA.

[adelapena]

A few runs of the ds12 to ds12 tests pass for me.

Was that in IDEA?

Yes, both.

Definitely this looks like an issue on my end. It's quite surprising though that it happens much more often with the ds12 to ds11 versions. I'll take another look on Monday and let go if nothing comes up.

[adelapena]

It seems the failure is due to something on my env, and I guess the only failure on the multiple CI runs was due to a legitimate timeout, so I'm not blocking this any longer.

The failure of SlowSAIQueryLoggerTest will go out after rebase. The rest of CI looks good to me.

We need the matching CNDB PR before merging this, probably with a simple test to verify that it works on Astra too.

[driftx]

The failure of SlowSAIQueryLoggerTest will go out after rebase.

We shall see, I had to rebase for the CNDB PR.

[adelapena]

The failure of SlowSAIQueryLoggerTest will go out after rebase.

We shall see, I had to rebase for the CNDB PR.

I think it's the one fixed by https://github.com/riptano/cndb/issues/17291, it passes locally after local rebase.

[plpesvc-ds]

❌ Build ds-cassandra-pr-gate/PR-2476 rejected by Butler

---

3 regressions found
See build details here

---

Found 3 new test failures

Test	Explanation	Runs	Upstream
o.a.c.distributed.test.sai.SlowSAIQueryLoggerTest.testSlowSAIQueryLogger	REGRESSION	🔵🔴	0 / 30
o.a.c.index.sai.cql.VectorCompaction100dTest.testOneToOneCompaction[version=ec enableNVQ=false]	REGRESSION	🔴⚪	0 / 30
o.a.c.index.sai.cql.VectorSiftSmallTest.testRerankKZeroOrderMatchesFullPrecisionSimilarity[ec true]	REGRESSION	🔴⚪	0 / 30

---

Found 6 known test failures

[driftx]

Looks like it passed. The CNDB PR is at https://github.com/riptano/cndb/pull/18345