[CORE-8161] `storage`: add `min.cleanable.dirty.ratio`, schedule compaction by `dirty_ratio` #24991

WillemKauf · 2025-01-31T17:48:09Z

Based on PR #24649.

Instead of unconditionally compacting all logs during a round of housekeeping, users may now optionally schedule log compaction in the log_manager using the cluster/topic property min_cleanable_dirty_ratio/min.cleanable.dirty.ratio.

As mentioned in the above PR,

The dirty ratio of a log is defined as the ratio between the number of bytes in "dirty" segments and the total number of bytes in closed segments.
Dirty segments are closed segments which have not yet been cleanly compacted- i.e, duplicates for keys in this segment could be found in the prefix of the log up to this segment.

By setting the min.cleanable.dirty.ratio on a per topic basis, users can avoid unnecessary read/write amplification during compaction as the log grows in size.

A housekeeping scan will still be performed every log_compaction_interval_ms, and the log's dirty_ratio will be tested against min.cleanable.dirty.ratio in determining it's eligibility for compaction. Additionally, logs are now compacted in descending order according to their dirty ratio, offering a better "bang for buck" heuristic for compaction scheduling.

Backports Required

Release Notes

Improvements

Allow for optional scheduling of compaction via min_cleanable_dirty_ratio

vbotbuildovich · 2025-02-01T00:26:01Z

Retry command for Build#61464

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/describe_topics_test.py::DescribeTopicsTest.test_describe_topics_with_documentation_and_types
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":2}

vbotbuildovich · 2025-02-01T01:55:27Z

CI test results

test results on build#61464

test_id	test_kind	job_url	test_status	passed
gtest_raft_rpunit.gtest_raft_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61464#0194be30-4fe3-4201-a506-11eca43a36e9	FLAKY	1/2
kafka_server_rpfixture.kafka_server_rpfixture	unit	https://buildkite.com/redpanda/redpanda/builds/61464#0194be30-4fe3-4201-a506-11eca43a36e9	FAIL	0/2
rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61464#0194be87-f8fb-4fbd-9207-0641084924fe	FLAKY	1/2
rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types	ducktape	https://buildkite.com/redpanda/redpanda/builds/61464#0194be87-f8f9-4197-b75d-9e2b0e669087	FAIL	0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61464#0194be87-f8fb-4fbd-9207-0641084924fe	FLAKY	1/2
rptest.tests.partition_force_reconfiguration_test.NodeWiseRecoveryTest.test_node_wise_recovery.dead_node_count=2	ducktape	https://buildkite.com/redpanda/redpanda/builds/61464#0194be87-f8fb-4fbd-9207-0641084924fe	FAIL	0/20
rptest.tests.retention_policy_test.RetentionPolicyTest.test_changing_topic_retention_with_restart.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/61464#0194be87-f8f9-4197-b75d-9e2b0e669087	FLAKY	1/3
test_compat_rpunit.test_compat_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61464#0194be2d-b364-4273-8107-604c1a24978e	FAIL	0/2

test results on build#61480

test_id	test_kind	job_url	test_status	passed
gtest_raft_rpunit.gtest_raft_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61480#0194c30c-66c7-425f-882b-a62f163db808	FLAKY	1/2
gtest_raft_rpunit.gtest_raft_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61480#0194c30c-66c8-4db9-a2da-7dc5f0262638	FLAKY	1/2
rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61480#0194c368-0af7-4d62-86e0-9ef129bfeb88	FLAKY	1/2
rptest.tests.offset_for_leader_epoch_archival_test.OffsetForLeaderEpochArchivalTest.test_querying_archive	ducktape	https://buildkite.com/redpanda/redpanda/builds/61480#0194c368-0af6-4ff1-92d6-ad77df6dc1f7	FLAKY	1/5
rptest.tests.partition_force_reconfiguration_test.NodeWiseRecoveryTest.test_node_wise_recovery.dead_node_count=2	ducktape	https://buildkite.com/redpanda/redpanda/builds/61480#0194c368-0af7-4d62-86e0-9ef129bfeb88	FAIL	0/20

test results on build#61769

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61769#0194ed98-e02d-49fb-815b-8a92ef12fe4a	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61769#0194ed9d-c07e-41c7-ad37-873a07105423	FLAKY	1/2
rptest.tests.mirror_maker_test.TestMirrorMakerService.test_consumer_group_mirroring.source_type=kafka	ducktape	https://buildkite.com/redpanda/redpanda/builds/61769#0194ed98-e02c-45df-bba3-2bf6f621197c	FAIL	0/20
rptest.tests.partition_movement_test.PartitionMovementTest.test_availability_when_one_node_down	ducktape	https://buildkite.com/redpanda/redpanda/builds/61769#0194ed98-e02c-45df-bba3-2bf6f621197c	FLAKY	1/2

test results on build#61775

test_id	test_kind	job_url	test_status	passed
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61775#0194effa-4c71-49e2-a5c3-06cce4574f93	FLAKY	1/2
rptest.tests.mirror_maker_test.TestMirrorMakerService.test_consumer_group_mirroring.source_type=kafka	ducktape	https://buildkite.com/redpanda/redpanda/builds/61775#0194eff5-6746-4bf1-aa34-e78dd6b67522	FAIL	0/20
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic	ducktape	https://buildkite.com/redpanda/redpanda/builds/61775#0194effa-4c73-4464-9bfb-d54df885c3c0	FLAKY	1/2
test_compat_rpunit.test_compat_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61775#0194efb0-cf5d-48a4-9e93-707da0399798	FLAKY	1/2

test results on build#61791

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61791#0194f362-3879-4838-b35b-39ecd5051d85	FLAKY	1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/61791#0194f362-387a-4e5d-9f71-62729ec6a9da	FLAKY	1/4
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61791#0194f362-387b-41cd-954d-e30904bab43d	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=10	ducktape	https://buildkite.com/redpanda/redpanda/builds/61791#0194f362-387b-41cd-954d-e30904bab43d	FLAKY	1/4
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61791#0194f362-3879-4838-b35b-39ecd5051d85	FLAKY	1/2

test results on build#61800

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f54b-039e-4ea0-858f-4f25d07def12	FLAKY	1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f539-0f33-48da-a1a6-50bb0c690fe9	FLAKY	1/3
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f539-0f34-4233-b49f-a1d8f5323962	FLAKY	1/3
rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f54b-039e-4ea0-858f-4f25d07def12	FLAKY	1/2
rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True.truncate_point=start_offset	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f54b-039c-47ca-bd97-e67be6e3dac2	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61800#0194f539-0f33-48da-a1a6-50bb0c690fe9	FLAKY	1/2

test results on build#61803

test_id	test_kind	job_url	test_status	passed
rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f69c-0782-4a80-b2b3-956b1e980961	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=10	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f69c-0780-4965-bfd9-81ca3854161e	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f69c-0781-4818-a649-7df74eb7ce0f	FLAKY	1/2
rptest.tests.partition_force_reconfiguration_test.NodeWiseRecoveryTest.test_node_wise_recovery.dead_node_count=2	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f697-41c0-46d0-a978-c550d40c25dd	FLAKY	1/2
rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f69c-0780-4965-bfd9-81ca3854161e	FLAKY	1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic	ducktape	https://buildkite.com/redpanda/redpanda/builds/61803#0194f69c-0780-4965-bfd9-81ca3854161e	FLAKY	1/2

test results on build#62093

test_id	test_kind	job_url	test_status	passed
rptest.tests.cloud_storage_chunk_read_path_test.CloudStorageChunkReadTest.test_read_when_segment_size_smaller_than_chunk_size	ducktape	https://buildkite.com/redpanda/redpanda/builds/62093#01952733-d99c-45a3-a6d9-94da98ad4a69	FLAKY	1/2
rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete	ducktape	https://buildkite.com/redpanda/redpanda/builds/62093#01952737-3bec-46af-bfd3-6bc6a1680779	FLAKY	1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/62093#01952733-d99d-421e-a70d-370a764abd1c	FLAKY	1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade	ducktape	https://buildkite.com/redpanda/redpanda/builds/62093#01952733-d99b-4c50-a628-7614ed3a1123	FLAKY	1/2

test results on build#62314

test_id	test_kind	job_url	test_status	passed
rptest.tests.datalake.compaction_test.CompactionTest.test_compaction.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FLAKY	1/3
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=100.storage_compaction_key_map_memory_kb=131072	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FAIL	0/20
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FAIL	0/20
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=10	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FAIL	0/20
rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954081-97f3-407f-aac6-635bf27d36ce	FLAKY	1/3
rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954081-97f0-4939-9d74-9a6fd41664e4	FAIL	0/20
rptest.transactions.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FLAKY	1/2
rptest.transactions.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=delete.workload=Workload.TX_UNIQUE_KEYS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62314#01954084-fd03-4248-9ae6-04bd07df4ace	FLAKY	1/4

test results on build#62339

test_id	test_kind	job_url	test_status	passed
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_creating_and_listing_migrations	ducktape	https://buildkite.com/redpanda/redpanda/builds/62339#01954357-c2f6-444d-8a7c-fa50cca84313	FLAKY	1/2
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/62339#0195435b-61cc-4d72-84ca-a19dae5272fd	FLAKY	1/2
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62339#0195435b-61cc-4d72-84ca-a19dae5272fd	FLAKY	1/2
rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62339#01954357-c2f6-4753-a177-a4c3e016c730	FLAKY	1/4
rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/62339#01954357-c2f4-4f98-a86b-ba51aadf2152	FLAKY	1/9

test results on build#62381

test_id	test_kind	job_url	test_status	passed
rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True	ducktape	https://buildkite.com/redpanda/redpanda/builds/62381#019545d3-8d64-46f5-ab09-4a472c6d423e	FLAKY	1/2
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/62381#019545bf-4a30-4dde-a082-e589ed45c885	FLAKY	1/2
rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact	ducktape	https://buildkite.com/redpanda/redpanda/builds/62381#019545bf-4a31-4573-a494-b8462c8bcdb4	FLAKY	1/2

vbotbuildovich · 2025-02-01T23:40:34Z

Retry command for Build#61480

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":2}

src/v/storage/probe.cc

src/v/cluster/metadata_cache.cc

src/v/storage/log_manager.cc

src/v/config/configuration.cc

tests/rptest/tests/describe_topics_test.py

WillemKauf · 2025-02-08T10:18:19Z

Force push to:

rebase to upstream/dev post base PR merge

WillemKauf · 2025-02-10T01:00:55Z

Force push to:

Use map of ntp_to_compaction_heuristic to sort _logs_list in log_manager.cc for compaction ordering
Add compaction_scheduling storage fixture test

src/v/storage/log_manager.cc

vbotbuildovich · 2025-02-10T04:45:33Z

Retry command for Build#61769

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/mirror_maker_test.py::TestMirrorMakerService.test_consumer_group_mirroring@{"source_type":"kafka"}

WillemKauf · 2025-02-10T11:48:08Z

Force push to:

Make recommended docs changes
Add log_housekeeping_meta::bitflags::compaction_checked and use it in main compacting loop in log_manager.cc

vbotbuildovich · 2025-02-10T15:47:08Z

Retry command for Build#61775

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/mirror_maker_test.py::TestMirrorMakerService.test_consumer_group_mirroring@{"source_type":"kafka"}

WillemKauf · 2025-02-10T22:19:50Z

test_compat is failing, and it's a little strange. Every so often we get a somewhat truncated double back from serde:

Expected: min_cleanable_dirty_ratio: {0.9553594929310676}

Decoded: min_cleanable_dirty_ratio: {0.9553594929310675}

This is the only diff between expected and decoded in the serde test.

Do we have a serde bug with double floating-point precision values?

WillemKauf · 2025-02-11T12:16:27Z

Force push to:

Add min_cleanable_dirty_ratio_validator to kafka config utils
Add test_min_cleanable_dirty_ratio_validation ducktape test
Represent min_cleanable_dirty_ratio cluster property as a decimal, not a percentage to align with Kafka

This approach has the added benefit of not having to sort the entire `_logs_list`. However, we now book-keep two data structures instead of one. Pointers to `housekeeping_meta` are held, but there are no concerns about concurrent removal since no scheduling points exist while the order is being mutated.

WillemKauf · 2025-02-26T03:16:40Z

i think we need more thought put into the default. the simple answer is to use a setting that retains the existing behavior.

Yeah, that would mean disabled by default. I'm not sure what's better here (maintaining the current status quo, or imposing a new sensible default value).

I think the min.cleanable.dirty.ratio is something that all compact topics should eventually consider setting, which is why I consider having it not disabled by default to be sensible.

I could maybe see leaving it disabled at the cluster level, but defaulting to create new topics with the kafka value 0.5 as a valid option as well.

WDYT?

dotnwat · 2025-02-26T03:28:20Z

I could maybe see leaving it disabled at the cluster level, but defaulting to create new topics with the kafka value 0.5 as a valid option as well.

maybe choose something "close" to the existing behavior. so if existing behavior is 0, then maybe something like 0.2 or whatever. then we can be sure that the code is executing in tests, but not yet have to tackle any sort of tuning project?

WillemKauf · 2025-02-26T04:14:52Z

maybe choose something "close" to the existing behavior. so if existing behavior is 0, then maybe something like 0.2 or whatever. then we can be sure that the code is executing in tests, but not yet have to tackle any sort of tuning project?

Yeah, 0.2 feels like a good value to start with. Ducktape doesn't actually have any complaints with it set to 0.5 which is cool.

I'm open to whatever you feel is best.

dotnwat · 2025-02-26T04:24:10Z

Yeah, 0.2 feels like a good value to start with. Ducktape doesn't actually have any complaints with it set to 0.5 which is cool.
I'm open to whatever you feel is best.

cool. let's just leave it at 0.5 and we can discuss further when we add the max lag configuration option, and maybe consider integrating with space management.

WillemKauf · 2025-02-26T04:26:45Z

(Force pushed to correct value in describe_topics_test.py after change in cluster default from std::nullopt to 0.5)

dotnwat · 2025-02-26T04:43:43Z

(Force pushed to correct value in describe_topics_test.py after change in cluster default from std::nullopt to 0.5)

just curious about this, i don't see any recent force pushes

WillemKauf · 2025-02-26T04:45:39Z

It's a bit above our most recent conversation, sorry 🙂

vbotbuildovich · 2025-02-26T07:18:49Z

Retry command for Build#62314

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/shadow_indexing_compacted_topic_test.py::ShadowIndexingCompactedTopicTest.test_upload@{"cloud_storage_type":1}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":131072}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":10}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":3}

WillemKauf · 2025-02-26T14:49:44Z

Well, maybe ducktape doesn't like 0.5 as the cluster default after all.

I can tweak the failing tests to set their min_cleanable_dirty_ratio to std::nullopt, if we feel strongly that we want to keep 0.5 as the default moving forward.

WillemKauf · 2025-02-26T15:55:39Z

Force push to:

Set min_cleanable_dirty_ratio for ducktape tests (these all expect unconditional compacting of the log, so set the property to 0.0 throughout)

tests/rptest/tests/log_compaction_test.py

tests/rptest/clients/kafka_cli_tools.py

Most of these tests expect/need unconditional compacting of the log. Set the `min_cleanable_dirty_ratio` at the topic/cluster level in these tests to mimic the old compaction scheduling behavior before this configuration was added.

WillemKauf · 2025-02-27T03:33:16Z

Force push to:

Remove added ducktape test for future PR.

dotnwat · 2025-02-27T05:13:39Z

Cool. But what was this change?

https://github.com/redpanda-data/redpanda/compare/851be256a2e6bce3296e5797e83da6b3d2ce0d2c..a9cb0be0ad25b879a00aa69bef64461076c5793b

I don't recall it being mentioned

WillemKauf · 2025-02-27T05:17:46Z

That change was an amend to the commit for the new ducktape test which was removed. It's no longer in this PR.

WillemKauf · 2025-02-27T13:01:24Z

CI flakes are:

https://redpandadata.atlassian.net/issues/CORE-8500
https://redpandadata.atlassian.net/issues/CORE-9114
https://redpandadata.atlassian.net/issues/CORE-7129

Looks good to me

WillemKauf requested a review from a team as a code owner January 31, 2025 17:48

github-actions bot added the area/redpanda label Jan 31, 2025

WillemKauf force-pushed the min_cleanable_dirty_ratio branch 2 times, most recently from db600d4 to 78b2928 Compare January 31, 2025 21:02

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 78b2928 to 7a6aab3 Compare February 1, 2025 19:43

WillemKauf changed the title ~~storage: schedule compaction by dirty_ratio~~ storage: add min.cleanable.dirty.ratio, schedule compaction by dirty_ratio Feb 1, 2025

WillemKauf changed the title ~~storage: add min.cleanable.dirty.ratio, schedule compaction by dirty_ratio~~ [CORE-8161] storage: add min.cleanable.dirty.ratio, schedule compaction by dirty_ratio Feb 1, 2025

dotnwat reviewed Feb 4, 2025

View reviewed changes

src/v/storage/probe.cc Outdated Show resolved Hide resolved

src/v/cluster/metadata_cache.cc Show resolved Hide resolved

src/v/storage/log_manager.cc Show resolved Hide resolved

asimms41 reviewed Feb 7, 2025

View reviewed changes

src/v/config/configuration.cc Outdated Show resolved Hide resolved

asimms41 reviewed Feb 7, 2025

View reviewed changes

src/v/config/configuration.cc Outdated Show resolved Hide resolved

asimms41 reviewed Feb 7, 2025

View reviewed changes

tests/rptest/tests/describe_topics_test.py Outdated Show resolved Hide resolved

asimms41 reviewed Feb 7, 2025

View reviewed changes

tests/rptest/tests/describe_topics_test.py Outdated Show resolved Hide resolved

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 7a6aab3 to 8b2ac50 Compare February 8, 2025 10:17

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 8b2ac50 to 0110d6c Compare February 10, 2025 00:45

WillemKauf commented Feb 10, 2025

View reviewed changes

src/v/storage/log_manager.cc Outdated Show resolved Hide resolved

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 0110d6c to e99aa91 Compare February 10, 2025 11:47

github-actions bot added the area/build label Feb 10, 2025

WillemKauf force-pushed the min_cleanable_dirty_ratio branch 2 times, most recently from 1df39dd to 70c37b7 Compare February 11, 2025 12:14

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 70c37b7 to 1af7ac0 Compare February 11, 2025 18:15

WillemKauf dismissed dotnwat’s stale review via 6697885 February 26, 2025 03:08

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from ce067cd to 6697885 Compare February 26, 2025 03:08

WillemKauf requested a review from dotnwat February 26, 2025 04:26

dotnwat previously approved these changes Feb 26, 2025

View reviewed changes

WillemKauf dismissed dotnwat’s stale review via 04d355e February 26, 2025 15:54

dotnwat previously approved these changes Feb 26, 2025

View reviewed changes

WillemKauf enabled auto-merge February 26, 2025 18:06

WillemKauf disabled auto-merge February 26, 2025 18:06

WillemKauf dismissed dotnwat’s stale review via 851be25 February 26, 2025 21:16

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from 851be25 to a9cb0be Compare February 26, 2025 23:05

dotnwat reviewed Feb 27, 2025

View reviewed changes

tests/rptest/tests/log_compaction_test.py Outdated Show resolved Hide resolved

tests/rptest/tests/log_compaction_test.py Outdated Show resolved Hide resolved

tests/rptest/clients/kafka_cli_tools.py Outdated Show resolved Hide resolved

WillemKauf force-pushed the min_cleanable_dirty_ratio branch from a9cb0be to cba07b8 Compare February 27, 2025 03:32

dotnwat approved these changes Feb 27, 2025

View reviewed changes

dotnwat merged commit f28e1bf into redpanda-data:dev Feb 27, 2025
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-8161] `storage`: add `min.cleanable.dirty.ratio`, schedule compaction by `dirty_ratio` #24991

[CORE-8161] `storage`: add `min.cleanable.dirty.ratio`, schedule compaction by `dirty_ratio` #24991

WillemKauf commented Jan 31, 2025 •

edited

Loading

vbotbuildovich commented Feb 1, 2025 •

edited

Loading

vbotbuildovich commented Feb 1, 2025 •

edited

Loading

vbotbuildovich commented Feb 1, 2025

WillemKauf commented Feb 8, 2025

WillemKauf commented Feb 10, 2025

vbotbuildovich commented Feb 10, 2025

WillemKauf commented Feb 10, 2025

vbotbuildovich commented Feb 10, 2025

WillemKauf commented Feb 10, 2025 •

edited

Loading

WillemKauf commented Feb 11, 2025

WillemKauf commented Feb 26, 2025

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025 •

edited

Loading

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

vbotbuildovich commented Feb 26, 2025 •

edited

Loading

WillemKauf commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

WillemKauf commented Feb 27, 2025

dotnwat commented Feb 27, 2025

WillemKauf commented Feb 27, 2025

WillemKauf commented Feb 27, 2025 •

edited

Loading

[CORE-8161] storage: add min.cleanable.dirty.ratio, schedule compaction by dirty_ratio #24991

[CORE-8161] storage: add min.cleanable.dirty.ratio, schedule compaction by dirty_ratio #24991

Conversation

WillemKauf commented Jan 31, 2025 • edited Loading

Backports Required

Release Notes

Improvements

vbotbuildovich commented Feb 1, 2025 • edited Loading

Retry command for Build#61464

vbotbuildovich commented Feb 1, 2025 • edited Loading

CI test results

vbotbuildovich commented Feb 1, 2025

Retry command for Build#61480

WillemKauf commented Feb 8, 2025

WillemKauf commented Feb 10, 2025

vbotbuildovich commented Feb 10, 2025

Retry command for Build#61769

WillemKauf commented Feb 10, 2025

vbotbuildovich commented Feb 10, 2025

Retry command for Build#61775

WillemKauf commented Feb 10, 2025 • edited Loading

WillemKauf commented Feb 11, 2025

WillemKauf commented Feb 26, 2025

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025 • edited Loading

dotnwat commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

vbotbuildovich commented Feb 26, 2025 • edited Loading

Retry command for Build#62314

WillemKauf commented Feb 26, 2025

WillemKauf commented Feb 26, 2025

WillemKauf commented Feb 27, 2025

dotnwat commented Feb 27, 2025

WillemKauf commented Feb 27, 2025

WillemKauf commented Feb 27, 2025 • edited Loading

[CORE-8161] `storage`: add `min.cleanable.dirty.ratio`, schedule compaction by `dirty_ratio` #24991

[CORE-8161] `storage`: add `min.cleanable.dirty.ratio`, schedule compaction by `dirty_ratio` #24991

WillemKauf commented Jan 31, 2025 •

edited

Loading

vbotbuildovich commented Feb 1, 2025 •

edited

Loading

vbotbuildovich commented Feb 1, 2025 •

edited

Loading

WillemKauf commented Feb 10, 2025 •

edited

Loading

WillemKauf commented Feb 26, 2025 •

edited

Loading

vbotbuildovich commented Feb 26, 2025 •

edited

Loading

WillemKauf commented Feb 27, 2025 •

edited

Loading