Improve Third-Party Deployment Plugins Reliability and Compatibility

[DimedS]

Overview

Part of #4317. This issue focuses on addressing user challenges with Kedro's third-party connection plugins for deployment. We aim to ensure that these plugins remain reliable and compatible with the latest Kedro versions while offering a flexible approach to managing platform-specific features.

User Insights and Challenges

• Kubeflow Plugin: Users are exploring ways to use the Kubeflow plugin for compiling and deploying Kedro pipelines but face issues with outdated functionality.
• Databricks Integration: Some users prefer interacting with the Databricks API manually via Asset Bundles to achieve greater flexibility, rather than using the Kedro-Databricks plugin.
• Argo Workflows: Translating Kedro pipelines into Argo Workflows manifests is a significant challenge.
• SageMaker Plugin: The Kedro-SageMaker plugin was not used due to incompatibility with Kedro 0.19, limiting its application for certain projects.

Each connection plugin includes two key functionalities:

• Converting Kedro pipelines for different platforms
• Handling API interactions to upload and execute the converted pipelines

Goals

Ensure Long-Term Compatibility: Maintain and update third-party plugins regularly to align with the latest Kedro releases.

Possible Solutions

• Split Plugin Functionality: Divide each plugin's functionality into two parts:
  • Pipeline Conversion: Kedro maintainers will support and maintain the part responsible for converting Kedro pipelines to various platforms.
  • API Interactions: The community will handle the API interaction component, responsible for uploading and running the converted pipelines on specific platforms.
• Resource Optimization: By splitting responsibilities, Kedro maintainers can focus on the core functionality (pipeline conversion), saving resources while ensuring essential compatibility, and empowering the community to maintain API-specific integrations.

[DimedS]

Summarised the current state of deployment plugins, highlighting the node grouping functionality:

https://www.notion.so/mckinsey/1423043f5dcf80b7a6cdf21901ff55c2?v=7958cdee2674405b82de8dd318d9fa1f