kedro flask integration

[milanpatel]

Is there any way to integrate kedro pipeline with flask Rest API?
I am trying to create pipeline for model prediction and I have little complicated model pipeline with multiple neural network models and ensembler.
I am using Sequential Runner to run the model pipeline.

[datajoely]

So today we don't have anything that does this for you as part of a main project - kedro-grpc exists as an open source project which you could use, or perhaps implement your own flask equivalent. using this as basis.

[lucasjamar]

Have you seen this tutorial already? https://www.youtube.com/watch?v=z7MIq-B4hPA

[mjmare]

Where has DataEngineerOne gone? His videos are so useful (but Kedro has gone through quite a bit of change since).

[datajoely]

Unfortunately his career has taken him places where he no longer has time for this sort of thing :( We are looking to hire a full time DevRel so if anyone reading this wants to get in touch, please do!

[mjmare]

Good for him, a pity for the community

[Selasi3]

How did you solve that issue using the said tutorial:: its not working for me

[lucasjamar]

Heres an example of how to launch a kedro pipeline from a flask app

from pathlib import Path
from typing import Any, Dict

from kedro.framework.session import KedroSession
from kedro.framework.startup import _add_src_to_path, _get_project_metadata
from flask import Flask

def create_new_kedro_session(
    extra_params: Dict[str, Any] = None, env: str = None
) -> KedroSession:
    """Helper function to initialize a new `KedroSession` inside of another python file.
    """
    project_path = Path.cwd()
    metadata = _get_project_metadata(project_path)
    _add_src_to_path(metadata.source_dir, project_path)
    new_session = KedroSession.create(
        metadata.package_name, project_path, extra_params=extra_params, env=env
    )
    return new_session


initial_session = create_new_kedro_session()
context = initial_session.load_context()
catalog = context.catalog
server = Flask(__name__)


@server.route("/hello_world/", methods=["GET"])
def hello_world():
    new_session = create_new_kedro_session(extra_params={"train_test_ratio": 0.8})
    new_session.run()
    data = catalog.load("final_results")
    return data.to_csv(index=False)
    
    

[datajoely]

So this thread is super interesting - a long time the Kedro core team built something internally but there wasn't much demand so the prototype was discarded. I now believe it may be time for the community to align around a single solution going forward.

The folks at Indicium have this kedro-fast-api (and excellent blog post): https://bitbucket.org/indiciumtech/kedro_fast_api/src/master/
@Galileo-Galilei has the kedro-serving repo: https://github.com/Galileo-Galilei/kedro-serving
@mmchougule has the kedro-grpc repo: https://github.com/mmchougule/kedro-grpc-server

The three usecases I would love to see out of this:
(1) Instrumenting runs from other services in a wider system
(2) Consuming run outputs from other systems
(3) Streamlit / Dash catalog integration