CNDB-11532: Adaptive compression

[pkolaczk]

This commit introduces a new AdaptiveCompressor class.

AdaptiveCompressor uses ZStandard compression with a dynamic compression level based on the current write load. AdaptiveCompressor's goal is to provide similar write performance as LZ4Compressor for write heavy workloads, but a significantly better compression ratio for databases with a moderate amount of writes or on systems with a lot of spare CPU power.

If the memtable flush queue builds up, and it turns out the compression is a significant bottleneck, then the compression level used for flushing is decreased to gain speed. Similarly, when pending compaction tasks build up, then the compression level used for compaction is decreased.

In order to enable adaptive compression:

• set -Dcassandra.default_sstable_compression=adaptive JVM option to automatically select AdaptiveCompressor as the main compressor for flushes and new tables, if not overriden by specific options in cassandra.yaml or table schema
• set flush_compression: adaptive in cassandra.yaml to enable it for flushing
• set AdaptiveCompressor in Table options to enable it for compaction

Caution: this feature is not turned on by default because it
may impact read speed negatively in some rare cases.

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits

[blambov]

I'm not very happy with this name, and generally with the use of "FAST" for "FLUSH" and "GENERAL" for "COMPACTION", but we can't change the enum at this point, can we?

[pkolaczk]

Im not happy either... However if I change the naming, then we should change it for all the things consistently and this is a part of the config (we use "fast" in cassandra.yaml). :(

[blambov]

This can use EnumSet.of.

Further, IMHO it makes better sense to return both here; or rather, add a new method to adapt the compressor for a certain use, e.g.

default ICompressor forUse(Uses use)
{
    return recommendedUses.contains(use) ? this : null;
}

in ICompressor, overridden to change params and recreate in AdaptiveCompressor.

[blambov]

Actually, such a change is also necessary to preserve encryption for HCD.

[pkolaczk]

+1 to EnumSet.of

However, I don't agree with returning both uses here.
This is because a particular AdaptiveCompressor can be either adapting to "flush pressure" or "compaction pressure" but not both. I don't want to accidentally end up using AdaptiveCompressor configured for compaction (aka GENERAL use) in flush, so it cannot advertise as FAST_COMPRESSION unless configured for flushing.
The use is locked at the moment of creating the compressor. It could be actually two separate compressor classes, each for different use, but because they share like 99% of logic and the only real difference is defaults + pressure source, there is likely no point in separating them.

[pkolaczk]

But on the second thought, indeed, if someone configures flushCompression as table and then AdaptiveCompression is selected in the table schema... this would result in using a wrong source of pressure. So adding this forUse API looks like a good idea to me.

[blambov]

This is precisely why I described an adaptation method: we do want to use a compressor derived from the current one, suitable for the flush usage. This can be done without requiring any changes to existing ICompressor implementations via the default method above, and is important for this patch, but also critical for preserving encryption when DSE moves to this code base.

[blambov]

This doesn't appear to agree with the javadoc. The effect of val should be stronger for higher n, not weaker.

E.g. I believe calling update(val) twice results in (1-alpha)^2 * current + (1-(1-alpha)^2) * val, which is not the same as (1 - alpha^2) * current + alpha^2 * val, which this translates to.

[pkolaczk]

Right. I got it reversed.

[blambov]

It would be better to port over CASSANDRA-13890 which introduces a compaction metric for the current compaction rate (with a few aggregations, e.g. 1-minute rate).

[blambov]

I'm curious why not port over the few other changes in CASSANDRA-13890. Has the code diverged too much?

[pkolaczk]

I'm curious why not port over the few other changes in CASSANDRA-13890. Has the code diverged too much?

Will take a look. The merge wasn't clean and I didn't initially want to spend time on something that is not essential for this feature.

[sonarcloud]

Quality Gate failed

Failed conditions
73.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1432 rejected by Butler

---

10 new test failure(s) in 12 builds
See build details here

---

Found 10 new test failures

Test	Explanation	Branch history	Upstream history
...lureTest.testFailedBkdReaderOnMultiIndexesQuery	regression	🔴🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...lureTest.testFailedKeyReaderOnMultiIndexesQuery	regression	🔴⚫⚫⚫⚫⚫⚫	⚫⚫⚫⚫⚫⚫⚫
...Test.testFailedRangeIteratorOnMultiIndexesQuery	regression	🔴🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...reTest.testFailedTermsReaderOnMultiIndexesQuery	regression	🔴🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
....s.f.SnapshotTest.shouldTakeAndRestoreSnapshots	regression	🔴🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...yIteratorTest.testTotalAndReadBytesManySSTables	flaky	🔵🔴🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...t.n.SetGetCompactionThroughputTest.testMaxValue	failing	🔴
...t.n.SetGetCompactionThroughputTest.testPositive	failing	🔴
...n.SetGetCompactionThroughputTest.testUpperBound	failing	🔴
o.a.c.t.n.SetGetCompactionThroughputTest.testZero	failing	🔴

---

Found 110 known test failures