Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demo slicing pipeline final #2036

Merged
merged 182 commits into from
Sep 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
182 commits
Select commit Hold shift + click to select a range
2ef2a16
set actions - filters
Jun 14, 2024
eb0f871
set reducers - filters
Jun 14, 2024
c59ef7b
set reducers - filtered-pipeline
Jun 14, 2024
d748246
user getFilteredPipeline in disabled
Jun 14, 2024
81d234f
set eslint
Jun 14, 2024
15d04c7
first setup in flowchart
Jun 14, 2024
676d83d
include isDisabledViaFilters in disabled
Jun 14, 2024
71c051e
highlight multi-select nodes
Jun 19, 2024
3a9afa3
use local state instead
Jun 19, 2024
3dcf748
remove extra action
Jul 4, 2024
4109119
shifting the logic around for shift hold the node
Jul 5, 2024
02ff6cc
create a separate state for highlighting node
Jul 8, 2024
5c6ffa3
firt working version of highlighting the node list
Jul 9, 2024
c97838a
remove the extra selector and action
Jul 9, 2024
6839ece
first draft of highlighting the node list
Jul 9, 2024
b474ad7
hide the slicing button
Jul 9, 2024
6d5c9de
remove extra node selector
Jul 10, 2024
494b93c
set click outside to clear highlighting node
Jul 10, 2024
f36c3cb
slice pipeline component
Jul 10, 2024
2288662
handle click on the third node
Jul 10, 2024
01a8da3
reset filter nodes on click on the node list
Jul 10, 2024
eaac898
create utils function for filtering nodes in draw.js
Jul 11, 2024
21a8e2c
adding comments for componentDidUpdate
Jul 11, 2024
659e034
remove un-used selectors
Jul 11, 2024
0e2db35
rename filteredPipeline state and classname
Jul 15, 2024
c732a4a
include filters in normalize data
Jul 15, 2024
8ae4648
update normalize data
Jul 16, 2024
03e0dd8
test for reducers
Jul 16, 2024
5873442
write test for selector
Jul 16, 2024
f3fb63d
Merge remote-tracking branch 'origin' into feature/slicing-pipeline
Jul 16, 2024
0ff7dc1
changes to test the tests locally
Jul 16, 2024
deca337
responsive cta button
Jul 16, 2024
f20783d
update failed test
Jul 16, 2024
49d9183
fix failed e2e tests
Jul 17, 2024
a0300c7
adding transformX value for button
Jul 17, 2024
c51aab0
add slice btn functionality
Jul 17, 2024
d8e650f
remove unused props
Jul 17, 2024
1351cfc
fix console error where the same id applies twice
Jul 17, 2024
2f1f742
Merge branch 'feature/slicing-pipeline' into feature/slicing-pipeline…
Jul 17, 2024
f1e7aa7
prevent resetting on onclick slice btn
Jul 17, 2024
95e1cbb
first working version of dilerting node list
Jul 17, 2024
e4b0c8a
filtering node list on the left
Jul 18, 2024
91b9ff0
filter out modular pipeline if no nodes selected
Jul 22, 2024
e25fc07
include reset button
Jul 22, 2024
30a2b5a
merge main and solve conflicts
Jul 22, 2024
f4f83fa
cy:test close the hint feature so can click on a node
Jul 22, 2024
e66f9a9
run command selector
Jul 22, 2024
2f0fd74
include command copier
Jul 22, 2024
d071c01
UI for command line
Jul 23, 2024
562b453
avoid reseting when clicking on a node while filters is activated
Jul 23, 2024
00e461e
Add comment
Jul 23, 2024
6d7876a
test to cover the run-command selector
Jul 23, 2024
0ab76ce
no holdshift click on nodes when filters are applied
Jul 24, 2024
77f71fc
rename component and apply light theme
Jul 25, 2024
d52fc69
move theme to shared variable
Jul 25, 2024
2d2f5d7
fix typo
Jul 25, 2024
3fd21ab
close the hint feature in cy test
Jul 25, 2024
eb698b6
lower case for one of the cy test
Jul 25, 2024
23bed33
Merge branch 'main' into feature/slicing-pipeline
Jul 25, 2024
ef69cc4
remove pretify name as its already set
Jul 25, 2024
e759b99
Merge branch 'feature/slicing-pipeline' into feature/slicing-pipeline…
Jul 25, 2024
e8e6888
update component name
Jul 25, 2024
59d87ff
fix merge conflict and update classname for FilteredPipelineActionBar
Jul 25, 2024
a55a281
update classnames to match the new component name
Jul 25, 2024
2cada8a
change the styling for filtered nodes
Jul 25, 2024
a5d38cb
refactor the logic to make it clearer
Jul 25, 2024
7b7e5b8
temporary solution for styling highlighting state
Jul 25, 2024
85143aa
update name for selector and flowchart, draw and node-list components
Jul 26, 2024
749ce01
rename for state and action
Jul 26, 2024
71e1460
update name for action bar
Jul 26, 2024
f49ab77
update name for css
Jul 26, 2024
acc6490
add notification
SajidAlamQB Jul 26, 2024
e479e0b
Merge branch 'feature/slicing-pipeline' into feature/slicing-notifica…
SajidAlamQB Jul 26, 2024
0541966
rename
SajidAlamQB Jul 26, 2024
831ce9c
Use SlicedPipelineActionBar
SajidAlamQB Jul 26, 2024
0a335b8
Merge remote-tracking branch 'origin' into feature/slicing-pipeline
Jul 29, 2024
7fdb27f
Merge branch 'feature/slicing-pipeline' into feature/slicing-pipeline…
Jul 29, 2024
aacc097
fix other merge conflict
Jul 29, 2024
4f05e64
Update flowchart.test.js
SajidAlamQB Jul 29, 2024
981766e
styling for long run command
Jul 29, 2024
8d31f39
adjust some styling
Jul 29, 2024
ead4fad
remove the styling for node list
Jul 29, 2024
62f5c80
Merge branch 'feature/slicing-pipeline' into feature/slicing-pipeline…
Jul 29, 2024
8306bcc
update failed test and use visibleSidebar instead
Jul 29, 2024
eeb366f
fix typo
Jul 31, 2024
72e8863
tidy up code
Jul 31, 2024
9b3b8be
Merge branch 'main' into feature/slicing-pipeline
Jul 31, 2024
1d5f50b
Merge branch 'feature/slicing-pipeline' into feature/slicing-pipeline…
Jul 31, 2024
f0bbcae
update name in test
Jul 31, 2024
55a0b13
Merge branch 'feature/slicing-pipeline' into feature/slicing-notifica…
SajidAlamQB Jul 31, 2024
e4cd8db
add tests for util functions
Jul 31, 2024
914225a
update logic for selector nodes
Jul 31, 2024
332624b
remove --over-write class on row as it's causing a bug
Jul 31, 2024
9340874
merge
SajidAlamQB Jul 31, 2024
5e02618
Update flowchart.js
SajidAlamQB Jul 31, 2024
13f1d95
remove overwrite styling as it causing a bug
Jul 31, 2024
dd0d30b
Merge branch 'feature/slicing-pipeline--run-command' into feature/sli…
SajidAlamQB Jul 31, 2024
5da0790
Revert "Merge branch 'feature/slicing-pipeline--run-command' into fea…
SajidAlamQB Jul 31, 2024
aad8325
add the classname back as accidentally removed
Jul 31, 2024
926195e
include 2 new props in flowchart test
Jul 31, 2024
007cf93
change the position when meta panel opens
Aug 1, 2024
294097e
update some colour with the new design changes
Aug 1, 2024
11b5f83
selecting a node from bottom to top, the order should not matter
SajidAlamQB Aug 1, 2024
f9b931d
Update flowchart.js
SajidAlamQB Aug 1, 2024
c04e1c9
Merge branch 'feature/slicing-pipeline' into feature/slicing-notifica…
Aug 1, 2024
9e9f453
center the notification
Aug 1, 2024
c56d718
center the action bar
Aug 1, 2024
1a8eef9
using existing function to update state
Aug 1, 2024
d9fd907
use redux instead of local state
Aug 1, 2024
51d47b1
tidy up code
SajidAlamQB Aug 1, 2024
04fba60
update colours
Aug 1, 2024
8159a40
only style from to pipeline if both of them are defined
Aug 2, 2024
1035f6c
fix the fading styling
Aug 2, 2024
f92a3c1
add test
SajidAlamQB Aug 5, 2024
889a8ee
simplify the code
Aug 5, 2024
640d7f4
changes based on review
SajidAlamQB Aug 6, 2024
9d7d9ce
update variable names
Aug 7, 2024
9fcc5c1
adding comments to explain
Aug 7, 2024
1c58884
rename ActionBar to slicePipelineActionBar
SajidAlamQB Aug 7, 2024
c43e561
reorder code
Aug 7, 2024
bf8a0d6
update comment and function name
Aug 7, 2024
cc1e41a
set minimum transformX value
Aug 7, 2024
687653f
update function name in test
Aug 7, 2024
084f98c
Adjust the transform value to 0.3s
Aug 7, 2024
3c79047
update config value for calculating postion
Aug 7, 2024
f156e64
update positioning logic
Aug 7, 2024
b27a1e2
removed unncesscary this.props.slicedPipeline
Aug 7, 2024
007dbac
include extra check for data.name
Aug 8, 2024
142b09d
add || rather than &&
Aug 8, 2024
ce0cb7c
add hover style for the slice button
Aug 8, 2024
a41cea9
hover style for reset button
Aug 8, 2024
d608dc3
hover style for copy button
Aug 8, 2024
dd3ff70
fix padding for action bar
Aug 8, 2024
d767948
first working version of box shadow
Aug 9, 2024
7854294
fix position and transition issue
Aug 9, 2024
1bd2e67
fix the issue when selecting on the node again the second time
Aug 9, 2024
660d2b9
set button heigh and position of the tooltip
Aug 12, 2024
25dd7c9
update slicing notification when metadata panel is closed
SajidAlamQB Aug 12, 2024
dc72b5d
Update flowchart.js
SajidAlamQB Aug 12, 2024
5eab977
remove line at end of notification bar
SajidAlamQB Aug 12, 2024
f54b885
persist from node from user selection
SajidAlamQB Aug 12, 2024
b2a2aeb
merge 3 different branch and fix merege conflict
Aug 12, 2024
5da4f3c
box shadow using after class
Aug 12, 2024
0196b3a
fix the from node bug
Aug 12, 2024
44a6e49
increase box shadow length
Aug 12, 2024
a09349b
fix bug
Aug 12, 2024
f384401
update buffer name
Aug 13, 2024
000ea07
update function name
Aug 13, 2024
e2e06b4
Merge branch 'main' into demo-slicing-pipeline-final
Aug 14, 2024
e809035
fix merge commit
Aug 14, 2024
405f42e
fix merge conflict
Aug 15, 2024
787e88f
separate action bar and notification
Aug 15, 2024
a6c116b
remove unused mixin
Aug 15, 2024
33cc248
include extra check for run command length
Aug 15, 2024
c7e212c
include test for notification
Aug 15, 2024
167c51c
fix bug on highlighting node list in slicing mode
Aug 15, 2024
485f711
new animation for action bar
Aug 15, 2024
26c09b8
fix bug about the number of sliced pipeline
Aug 15, 2024
3159f81
add new props in test
Aug 15, 2024
a5bb98c
Feature/slicing pipeline add notification (#2003)
SajidAlamQB Aug 16, 2024
d950b4c
highlight modularpipeline when collasped in slicing mode
Aug 16, 2024
4b2041f
update slice button color
Aug 16, 2024
473448d
fix merge conflict
Aug 16, 2024
627be66
removed unused props
Aug 20, 2024
7b27c8b
update name for slicePipeline
Aug 20, 2024
872ab61
update test with new name for SET_SLICE_PIPELINE
Aug 20, 2024
3d636d8
get runCommand based on type of the element
Aug 21, 2024
423e847
update run command to get from metadata
Aug 22, 2024
b973f79
add comments
Aug 22, 2024
7bc7e87
Merge branch 'main' into demo-slicing-pipeline-final
Aug 23, 2024
b64fc27
update test for sliced-pipeline
Aug 23, 2024
f726040
update run-command test with new logic
Aug 23, 2024
692c1ac
Disable slicing feature in embedded mode (#2072)
jitu5 Sep 3, 2024
c0282f2
Telemetry/slicing pipeline (#2054)
Huongg Sep 4, 2024
31b6e25
fix the highlight of modular pipeline
Sep 4, 2024
68ca886
fix the position of run command
Sep 4, 2024
d42d897
Merge branch 'main' into demo-slicing-pipeline-final
Sep 4, 2024
e1a5e10
use 1.5 instead 2 for divided value
Sep 4, 2024
e2bc438
Documentation: slicing pipeline in kedro-viz (#2043)
Huongg Sep 5, 2024
f56e722
update release note
Sep 5, 2024
af82be8
Merge branch 'main' into demo-slicing-pipeline-final
Sep 5, 2024
e1677e5
Merge branch 'main' into demo-slicing-pipeline-final
Sep 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
"curly": ["error"],
"valid-typeof": ["error"],
"camelcase": "error",
"id-length": ["error", { "min": 3, "exceptions": ["_","a","b","d","e","i","j","k","x","y","id","el","pi","PI","up"] }],
"no-var": ["error"],
"id-length": ["error", { "min": 3, "exceptions": ["_","a","b","d","e","i","j","k","x","y","id","el","pi","PI","up","to"] }], "no-var": ["error"],
"lines-between-class-members": ["error", "always"]
}
}
Expand Down
1 change: 1 addition & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Please follow the established format:
## Major features and improvements

- Introduce `onActionCallback` prop in Kedro-Viz react component. (#2022)
- Slice a pipeline functionality. (#2036)

## Bug fixes and other changes

Expand Down
7 changes: 4 additions & 3 deletions cypress/tests/ui/flowchart/flowchart.cy.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,14 @@ describe('Flowchart DAG', () => {

beforeEach(() => {
cy.enablePrettyNames(); // Enable pretty names using the custom command
cy.wait(500);
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved
cy.get('.feature-hints__close').click(); // Close the feature hints so can click on a node
cy.wait(500);
});

it('verifies that users can expand a collapsed modular pipeline in the flowchart. #TC-23', () => {
const modularPipelineText = 'feature_engineering';
const taskNodeText = 'Create Derived Features';

cy.enablePrettyNames();
const taskNodeText = 'create_derived_features';

// Assert before action
cy.get('.pipeline-node > .pipeline-node__text').should(
Expand Down
7 changes: 4 additions & 3 deletions cypress/tests/ui/flowchart/menu.cy.js
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
// All E2E Tests Related to Flowchart Menu goes here.

import { prettifyName } from '../../../../src/utils';

describe('Flowchart Menu', () => {
beforeEach(() => {
cy.enablePrettyNames(); // Enable pretty names using the custom command
cy.wait(500);
cy.get('.feature-hints__close').click(); // Close the feature hints so can click on a node
cy.wait(500);
});

it('verifies that users can select a section of the flowchart through the drop down. #TC-16', () => {
Expand Down Expand Up @@ -144,7 +145,7 @@ describe('Flowchart Menu', () => {
.invoke('text')
.then((focusedNodesText) =>
expect(focusedNodesText.toLowerCase()).to.contains(
prettifyName(nodeToFocusText).toLowerCase()
nodeToFocusText
)
);
cy.get('.pipeline-node--active > .pipeline-node__text').should(
Expand Down
3 changes: 2 additions & 1 deletion cypress/tests/ui/flowchart/panel.cy.js
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ describe('Pipeline Minimap Toolbar', () => {
cy.__waitForPageLoad__(() => {
let initialZoomValue;
let zoomInValue;

cy.get('@zoomScale')
.invoke('text')
.then((text) => {
Expand Down Expand Up @@ -155,6 +155,7 @@ describe('Pipeline Minimap Toolbar', () => {
cy.get('@zoomScale')
.invoke('text')
.should((text) => {
initialZoomValue = parseFloat(text.replace('%', ''));
expect(initialZoomValue).to.be.eq(parseFloat(text.replace('%', '')));
});
});
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Take a look at the <a href="https://demo.kedro.org/" target="_blank" rel="noopen
kedro-viz_visualisation
share_kedro_viz
preview_datasets
slice_a_pipeline
experiment_tracking
```

Expand Down
33 changes: 33 additions & 0 deletions docs/source/slice_a_pipeline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Slice a pipeline

Slicing a pipeline in Kedro refers to creating a subset of a pipeline's nodes, which can help in focusing on specific parts of the pipeline. There are two primary ways to achieve this:

1. **Programmatically with the Kedro CLI.** This method is suitable for those comfortable with command-line tools. Detailed steps on how to achieve this are available in the kedro documentation: [Slice a Pipeline](https://docs.kedro.org/en/stable/nodes_and_pipelines/slice_a_pipeline.html).

2. **Visually through Kedro-Viz:** This approach allows you to visually select and slice pipeline nodes, which then generates a run command for executing the slice within your Kedro project.

## Benefits of Kedro-Viz slicing

- **Visual Representation:** View the relationships between nodes and identify which ones are part of your slice.
- **Immediate Command Generation:** Get a ready-to-use CLI command for executing the sliced pipeline.
- **Interactive Control:** Visually select and reset slices with a couple of clicks.

## Steps to slice in Kedro-Viz

Kedro-Viz offers a user-friendly visual interface for slicing pipelines. Follow these steps to use the slicing feature:

1. **Select elements in the flowchart:** In Kedro-Viz, select two elements to set the boundaries for your slice:
- Click on the first node you want to include.
- Hold the Shift key and select the second node.

![](./images/slice_pipeline_multiple_click.gif)

2. **Highlighted selection:** The flowchart will highlight all nodes between the selected elements, and the corresponding nodes in the list on the left will also be highlighted.

3. **View the run command:** After selecting the nodes, Kedro-Viz generates a CLI command for the sliced pipeline. You can copy this command and use it directly in your Kedro project to run the slice.

4. **Slice the pipeline:** When you're ready, click the "Slice" button. This opens a new view where you can directly interact with the sliced pipeline.

5. **Reset:** To discard your selection and return to the full pipeline view, click the "Reset" button. This will clear the slice and restore the default view.

![](./images/slice_pipeline_slice_reset.gif)
28 changes: 28 additions & 0 deletions src/actions/slice.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
export const APPLY_SLICE_PIPELINE = 'APPLY_SLICE_PIPELINE';

export const applySlicePipeline = (apply) => {
return async function (dispatch) {
dispatch({
type: APPLY_SLICE_PIPELINE,
apply,
});
};
};

export const SET_SLICE_PIPELINE = 'SET_SLICE_PIPELINE';

export const setSlicePipeline = (from, to) => {
return async function (dispatch) {
dispatch({
type: SET_SLICE_PIPELINE,
slice: { from, to },
});
};
};

export const RESET_SLICE_PIPELINE = 'RESET_SLICE_PIPELINE';

export const resetSlicePipeline = () => ({
type: RESET_SLICE_PIPELINE,
slice: { from: null, to: null },
});
2 changes: 1 addition & 1 deletion src/components/app/app.scss
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
--color-bg-1: #{colors.$white-800};
--color-bg-2: #{colors.$grey-0};
--color-bg-3: #{colors.$white-200};
--color-bg-4: #{colors.$white-0};
--color-bg-4: #{colors.$white-200};
--color-bg-5: #{colors.$white-600};
--color-bg-alt: #{colors.$black-700};
--color-bg-list: #{colors.$white-100};
Expand Down
1 change: 0 additions & 1 deletion src/components/app/app.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ import { localStorageName } from '../../config';
import { prepareNonPipelineState } from '../../store/initial-state';
import reducer from '../../reducers/index';
import { TOGGLE_GRAPH_LOADING } from '../../actions/graph';
import { prettifyName } from '../../utils/index';

describe('App', () => {
const getState = (wrapper) => wrapper.instance().store.getState();
Expand Down
38 changes: 38 additions & 0 deletions src/components/flowchart/draw.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,20 @@ const toSinglePoint = (value) => parseFloat(value).toFixed(1);
*/
const limitPrecision = (path) => path.replace(matchFloats, toSinglePoint);

/**
* Creates a mapping of node IDs to a boolean indicating if the node ID is included in the given values.
* @param {Array} nodes - Array of nodes to process.
* @param {Array} values - Array of values to check against node IDs.
* @returns {Object} An object mapping node IDs to booleans.
*/
function createNodeStateMap(nodes, values) {
const valueSet = new Set(values); // Convert to Set for efficient lookup
return nodes.reduce((acc, { id }) => {
acc[id] = valueSet.has(id);
return acc;
}, {});
}

/**
* Render layer bands
*/
Expand Down Expand Up @@ -135,7 +149,20 @@ export const drawNodes = function (changed) {
nodes,
focusMode,
hoveredFocusMode,
isSlicingPipelineApplied,
} = this.props;
const {
from: slicedPipelineFromId,
to: slicedPipelineToId,
range,
} = this.state.slicedPipelineState;

const slicedPipelineFromTo =
slicedPipelineFromId &&
slicedPipelineToId &&
createNodeStateMap(nodes, [slicedPipelineFromId, slicedPipelineToId]);

const slicedPipelineRange = createNodeStateMap(nodes, range);

const isInputOutputNode = (nodeID) =>
focusMode !== null && inputOutputDataNodes[nodeID];
Expand Down Expand Up @@ -241,6 +268,17 @@ export const drawNodes = function (changed) {
allNodes
.classed('pipeline-node--active', (node) => nodeActive[node.id])
.classed('pipeline-node--selected', (node) => nodeSelected[node.id])
.classed(
'pipeline-node--sliced-pipeline',
(node) => !isSlicingPipelineApplied && slicedPipelineRange[node.id]
)
.classed(
'pipeline-node--from-to-sliced-pipeline',
(node) =>
!isSlicingPipelineApplied &&
slicedPipelineFromTo &&
slicedPipelineFromTo[node.id]
)
.classed(
'pipeline-node--collapsed-hint',
(node) =>
Expand Down
Loading
Loading