STAR-1872: Parallelize UCS compactions per output shard #1342
Conversation
| partCommittedOrAborted(); | ||
| } | ||
|
|
||
| private void partCommittedOrAborted() |
There was a problem hiding this comment.
what about passing here the partial transaction as a parameter ?
instead of having only a number partsToCommitOrAbort we could have a reference to all the child transactions and ensure that we don't count the same transaction twice and when we commit all of the children are in the expected state
There was a problem hiding this comment.
You can't count the same transaction twice because of the protections in PartialLifecycleTransaction.
There could be some value in knowing which part did not complete, but because we don't have timeouts on these things (and actually can't, as compactions can last days) there's no obvious place to surface that information.
| } | ||
| else | ||
| { | ||
| final CompressionMetadata compressionMetadata = getCompressionMetadata(); |
There was a problem hiding this comment.
are we exercising this branch in the unit tests ?
There was a problem hiding this comment.
There's a list of TODOs in UnifiedCompactionStrategy.createAndAddTasks where this is next in line.
| return tasks; | ||
| } | ||
|
|
||
| private <T> List<T> splitSSTablesInShards(Collection<SSTableReader> sstables, |
There was a problem hiding this comment.
what about making this method static and writing specific unit tests to cover all of the cases?
|
The PR is not yet ready for review. |
blambov
force-pushed
the
STAR-1872
branch
2 times, most recently
from
9512132
to
6cc862f
Compare
This splits compactions that are to produce more than one output sstable into tasks that can execute in parallel. Such tasks share a transaction and have combined progress and observer. Because we cannot mark parts of an sstable as unneeded, the transaction is only applied when all tasks have succeeded. This also means that early open is not supported for such tasks. At this time the new parallelization mechanism is not taken into account by the thread allocation scheme, and thus some levels may take more resources than they should. Because of this limitation (which should be fixed in the near future), the new behaviour is off by default. Also: - Adds a flag to combine non-overlapping sets in major compactions to reshard data, as major compactions can can now be executed as a parallelized operation. - Changes SSTable expiration to be done in a separate getNextBackgroundCompactions round to improve the efficiency of expiration (separate task can run quickly and remove the relevant sstables without waiting for a compaction to end). - Applies small-partition-count correction in ShardManager.calculateCombinedDensity.
|
The patch is now ready for review. |
|
❌ Build ds-cassandra-pr-gate/PR-1342 rejected by Butler8 new test failure(s) in 16 builds Found 8 new test failures
Found 100 known test failures |
This splits compactions that are to produce more than one
output sstable into tasks that can execute in parallel.
Such tasks share a transaction and have combined progress
and observer. Because we cannot mark parts of an sstable
as unneeded, the transaction is only applied when all
tasks have succeeded. This also means that early open
is not supported for such tasks.
At this time the new parallelization mechanism is not taken
into account by the thread allocation scheme, and thus
some levels may take more resources than they should.
Because of this limitation (which should be fixed in the
near future), the new behaviour is off by default.
Also:
Adds a flag to combine non-overlapping sets in major
compactions to reshard data, as major compactions can
can now be executed as a parallelized operation.
Changes SSTable expiration to be done in a separate
getNextBackgroundCompactions round to improve the
efficiency of expiration (separate task can run quickly
and remove the relevant sstables without waiting for
a compaction to end).
Applies small-partition-count correction in
ShardManager.calculateCombinedDensity.