Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STAR-1872: Parallelize UCS compactions per output shard #1342

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Commits on Oct 29, 2024

  1. STAR-1872: Implements multi-task compactions

    This splits compactions that are to produce more than one
    output sstable into tasks that can execute in parallel.
    Such tasks share a transaction and have combined progress
    and observer. Because we cannot mark parts of an sstable
    as unneeded, the transaction is only applied when all
    tasks have succeeded. This also means that early open
    is not supported for such tasks.
    
    At this time the new parallelization mechanism is not taken
    into account by the thread allocation scheme, and thus
    some levels may take more resources than they should.
    Because of this limitation (which should be fixed in the
    near future), the new behaviour is off by default.
    
    Also:
    - Adds a flag to combine non-overlapping sets in major
      compactions to reshard data, as major compactions can
      can now be executed as a parallelized operation.
    
    - Changes SSTable expiration to be done in a separate
      getNextBackgroundCompactions round to improve the
      efficiency of expiration (separate task can run quickly
      and remove the relevant sstables without waiting for
      a compaction to end).
    
    - Applies small-partition-count correction in
      ShardManager.calculateCombinedDensity.
    blambov committed Oct 29, 2024
    1 Configuration menu
    Copy the full SHA
    b6295c0 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. 1 Configuration menu
    Copy the full SHA
    3353269 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. Configuration menu
    Copy the full SHA
    6a850fe View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Configuration menu
    Copy the full SHA
    ef1b009 View commit details
    Browse the repository at this point in the history
  2. Test fixes

    blambov committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3136567 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8dfb4f7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6a90b29 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. sonarcloud

    blambov committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e071215 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Configuration menu
    Copy the full SHA
    f3786c0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e7d7241 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    94afba6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    69a37d0 View commit details
    Browse the repository at this point in the history