This repository has been archived by the owner on Jun 20, 2024. It is now read-only.
Add a test for ScatterGather showing failed deferred access with map #361
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This shows a bug when we use
scatter.map(f).map(g)
where the new task built in the second map (byg
) requires that the task created in the first map (byf
) has successfully been executed. If you look atgroupBy
orflatMap
inSubScatter
, you'll see that the defer execution of their method until previous tasks have completed. If you look atSecondarySubScatter
, you'll see the source of the behavior for.map.map
:The
chain
method is recursively called on each trackedSubScatter
(i.e.subs
), after applyingf
to thesource
. For each of thosesubs
, theirf
method (i.e.g
) will be applied to the provided task, but the provided task may not have been executed. I think we want to have a wrapper that defers chaining until the source task has completed successfully. We also callsubs.foreach(_.connect())
inSubScatter
which relies on thechain
already being called.