Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add resume suggestion to parallel runner #1830

Open
jmholzer opened this issue Sep 5, 2022 · 1 comment
Open

Add resume suggestion to parallel runner #1830

jmholzer opened this issue Sep 5, 2022 · 1 comment
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@jmholzer
Copy link
Contributor

jmholzer commented Sep 5, 2022

Description

See #1795 for the original discussion and context surrounding this issue.

Currently, when Kedro is run using SequentialRunner and a Node creates an exception, Kedro will suggest resuming the run from the nearest Nodes with persisted input. This saves the user a great deal of time that would otherwise be wasted by running earlier Nodes whose output was successfully saved. Currently, no suggestion is made in the same case when Kedro is run using ParallelRunner.

This is because, when a Node reaches an exception while using ParallelRunner, other nodes may still be in the process of running. The 'end state' of Nodes which have finished running when the exception is reached is not guaranteed due to the inherent stochasticity of the sequence with which Nodes are executed in a parallel scheme. Therefore, if 'resume-suggestion' logic were applied to ParallelRunner, the suggestion would be inconsistent between runs, with the correct resume scenario suggestion only generated sporadically.

Context

This change will suggest a resume scenario to users who use ParallelRunner, saving them a great deal of time that would otherwise be wasted in running Nodes unnecessarily.

Possible Implementation

Ensuring that all nodes that can be run are run before the 'end state' is reached and the exception is generated using (for example) joins.

@jmholzer jmholzer added the Issue: Feature Request New feature or improvement to existing feature label Sep 5, 2022
@noklam
Copy link
Contributor

noklam commented Oct 3, 2022

Related to #503, It's likely a pre-requisite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Status: No status
Development

No branches or pull requests

3 participants