Added documentation for SortMergeJoin #13469
Conversation
|
I would be adding this short video to make the dev/user familiar with SMJ concepts https://www.youtube.com/watch?v=jiWCPJtDE2c |
| /// will fail under the same conditions. Multiple record batches of buffered could be | ||
| /// present in memory/disk during the exectution. | ||
| /// | ||
| /// Only one record batch of streamed input will be present in the memory at all times. There is no |
There was a problem hiding this comment.
it depends on the batch size and there can batch_size of streamed rows currently lives in memory.
For the buffered the buffered batch lives until BufferedState switches and due to presorted inputs the algorithm understands the buffered batch is not needed anymore and it gets released
There was a problem hiding this comment.
Thanks @athultr1997 this is good start
@korowa @viirya do you have anything to add on it?
| /// partitions. | ||
| /// Join execution plan that executes equi-join predicates on multiple partitions using Sort-Merge | ||
| /// join algorithm and applies an optional filter post join. Can be used to join arbitrarily large | ||
| /// inputs where one or both of the inputs don't fit in the available memory. |
There was a problem hiding this comment.
where one or both of the inputs don't fit in the available memory.
Hmm, is this true?
There was a problem hiding this comment.
streamed - Always taken one batch at a time.
buffered - Has spilling support.
Hence, the inputs don't have to fit in memory.
Also, I think this was the vision behind SMJ: #1599 (comment).
| /// | ||
| /// Buffered input is buffered for all record batches having the same value of join key. | ||
| /// If the memory limit increases beyond the specified value and spilling is enabled, | ||
| /// buffered batches could be spilled to disk. If spilling is disabled, the execution |
There was a problem hiding this comment.
Is there a config for spilling? Shall we mention it here too?
There was a problem hiding this comment.
One has to enable the disk manager when creating the RuntimeEnv during the creation of TaskContext.
let runtime = RuntimeEnvBuilder::new()
.with_memory_limit(100, 1.0)
.with_disk_manager(DiskManagerConfig::NewOs)
.build_arc()?;
I am not sure if we should mention it here, since this is all the way over in RuntimeEnv and there are multiple strategies to enable the disk manager.
Which issue does this PR close?
Closes #10357
Rationale for this change
What changes are included in this PR?
Added documentation for SortMergeJoin
Are these changes tested?
No tests needed since only documentation is added.
Are there any user-facing changes?
No