CNDB-11070: Limited backport of CASSANDRA-19534

[jkni]

What is the issue

Unbounded queue length at the native transport stage can caused large backlogs of requests that the system processes, even though clients may no longer expect a response.

What does this PR fix and why was it fixed

This PR implements a limited backport of CNDB-11070, introducing the notion of a native request timeout that can shed messages with excessive queue times at the NTR stage as well as async read/write stages, if enabled. Cross-node message timeouts are also now respected earlier in the mutation verb handler.

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits

[JeremiahDJordan]

Nice! This should give us a lot of the benefits of the original ticket without all of the painful refactoring that happened.

I think this is a good path forward. Should add some tests.

[jkni]

Basic test coverage added, and further ad-hoc testing shows that this interacts well with Stargate query handling (in that Message creation happens early in Stargate handling, so Message creation time is still a reasonable approximation of entry in the system). This is ready for review. Please take a look if interested @JeremiahDJordan @jtgrabowski @sbtourist

[sonarcloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1393 rejected by Butler

---

11 new test failure(s) in 7 builds
See build details here

---

Found 11 new test failures

Test	Explanation	Branch history	Upstream history
t.TestConstants.test_cql_reserved_keywords	failing	🔴🔴🔴🔴🔴🔴🔴
...Completion.test_complete_in_select_limit_clause	failing	🔴🔴🔴🔴🔴🔴🔴
t.TestCqlshUnicode.test_unicode_desc	regression	🔴🔴🔵🔵🔵🔴🔵
t.TestCqlshUnicode.test_unicode_identifier	failing	🔴🔴🔴🔴🔴🔴🔴
...nTest.testMultipleCompactionsDifferentWs_Static	failing	🔴🔴🔴🔴🔴🔴🔴
...i.s.c.VectorSiftSmallTest.testMultiSegmentBuild	failing	🔴🔴🔴🔴🔴🔴🔴
...int,int>>>,wide=false,scenario=COMPACTED_QUERY]	flaky	🔵🔴🔵🔵
...t,wide=false,scenario=MEMTABLE_QUERY]	flaky	🔵🔴🔵🔵
...titionsTest.testServiceTopPartitionsSingleTable	flaky	🔵🔴🔵🔵
o.a.c.t.TransportTest.testAsyncTransport	flaky	🔴🔵🔴🔴🔴🔵🔴
o.a.c.u.b.BinLogTest.testTruncationReleasesLogS...	flaky	🔵🔴🔵🔴🔴🔵🔵

---

Found 14 known test failures

[jtgrabowski]

Left few questions, please take a look.

[jtgrabowski]

Consider avoiding this mutable state in Message. The following code could be a part of lambda passed to supplyAsync above.

[jtgrabowski]

Should have something simlar for other verbs? Maybe reads?

[jkni]

Reads handle this much more extensively through Monitorable implementation (https://issues.apache.org/jira/browse/CASSANDRA-7392). CASSANDRA-19534 added this as a very minimal load-shedding point for expired mutations, but reads already have much more extensive infrastructure to handle this.

[jtgrabowski]

So async queue time includes sync queue time? Could you please make the relation clear in the comments?

[jkni]

Yes, will do.