storage: Always initialize ingestion statistics #30975
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, async storage workers would calculate the as_of of a dataflow from the since of the dataflow's output's remap shards. If there was more than one distinct remap shard among the outputs, then the storage worker would panic. It's expected that the only collection that will ever have a remap shard is the ingestion collection itself. Furthermore, we are planning to remove the ingestion collection from the outputs of the dataflow (in fact there's already a feature flag that does this). If the ingestion is removed from the outputs, then no output will have a remap shard, and the as_of will always be empty. This commit simplifies the existing as_of calculation and fixes the as_of calculation when the ingestion collection is removed from the outputs. It does this by calculating the as_of directly from the ingestion's remap shard. Additionally, it asserts that if any of the outputs have a remap shard, then it must be equal to the ingestion's remap shard. Works towards resolving #MaterializeInc/database-issues/issues/8620
Previously, collection statistics were only initialized for ingestion dataflow outputs. When the `force_source_table_syntax` flag is enabled, the ingestion collection is excluded from the ingestion dataflow outputs. As a result, statistics are never created for the ingestion collection. This causes later parts of the code to panic because it is assumed that all ingestion collections have statistics initialized. This commit fixes the issue by ensuring that statistics are always initialized for ingestion collections, even if it's not included in the dataflow outputs. Works towards resolving #MaterializeInc/database-issues/issues/8620
jkosh44
force-pushed
the
force-primary-export-stats
branch
from
b84e290 to
9ade587
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, collection statistics were only initialized for ingestion
dataflow outputs. When the
force_source_table_syntaxflag is enabled,the ingestion collection is excluded from the ingestion dataflow
outputs. As a result, statistics are never created for the ingestion
collection. This causes later parts of the code to panic because it is
assumed that all ingestion collections have statistics initialized.
This commit fixes the issue by ensuring that statistics are always
initialized for ingestion collections, even if it's not included in
the dataflow outputs.
Works towards resolving #MaterializeInc/database-issues/issues/8620
Motivation
This PR adds a known-desirable feature.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.