Scan Delete Support Part 3: ArrowReader::build_deletes_row_selection implementation
#951
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sdd
changed the title
ArrowReader::build_deletes_row_selection implementation
sdd
force-pushed
the
feat/build-deletes-row-selection-implementation
branch
3 times, most recently
from
f4b6d94 to
a52fe50
Compare
sdd
force-pushed
the
feat/build-deletes-row-selection-implementation
branch
from
a52fe50 to
4017a0e
Compare
* refactor: only pass row groups metadata rather than entire parquet metadata to . This makes it easier to test as we don't need to mock up a full
…letes that cross page group boundaries
sdd
force-pushed
the
feat/build-deletes-row-selection-implementation
branch
from
2fc5f70 to
26dc78f
Compare
sdd
changed the title
ArrowReader::build_deletes_row_selection implementationArrowReader::build_deletes_row_selection implementation
…haviour when processing a row selection that ends before the end if the stream
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Third part of delete file read support. See #630
Builds on top of #950 and should be merged after that one.
build_deletes_row_selectioncomputes aRowSelectionfrom a&[usize]representing the indexes of rows in a data file that have been marked as deleted by positional delete files that apply to the data file being read.The resulting
RowSelectionwill be merged with aRowSelectionresulting from the scan's filter predicate (if present) and supplied to theParquetRecordBatchStreamBuilderso that deleted rows are omitted from theRecordBatchStreamreturned by the reader.NB: I encountered quite a few edge cases in this method and the logic is quite complex. There is a good chance that a keen-eyed reviewer would be able to conceive of an edge-case that I haven't covered.