Decide to vendor kedro-viz or require user to install

[noklam]

Context

#70 (comment)

There is question whether we need to also vendor the relevant dependencies,
viz itself is too big and we have done a lot of work to optimise this. At
the moment we do not need a viz backend server but merely using the library
code to generates some JSON.

If we choose to ask user install viz, the minimum is this should not cause
any server panic. That is, for non flowchart users they should be able to
keep their workflow without forcing to install viz.

Requirement

• kedro_viz is an optional dependency in any case, extension should works with (for any feature available in 0.1.0)

[noklam]

@jitu5 and I have done some research, ideally we would like to vendor kedro-viz into the extensions for 2 reasons:

1. Avoid user to install kedro-viz manually (easier to onboard)
2. Avoid version conflict, we can control which viz version is being used in Kedro extension.

After some work, we figured out there are 3 "core" dependencies needed for kedro-viz:

• networkx>=2.5
• kedro-viz
• pydantic>=2.0.0 (and pydantic_core)

The first 2 are easy to vendor into the extension, but pydantic is tricky because it's per platform + python version. I don't have a deep understanding in how Python search for share libraries. It seems to work if the extension packaged the same platform + python version as the user environment (we tested this work for 1 specific version).

However, it's not easier to packaged all combinations that pydantic_core support: https://pypi.org/project/pydantic-core/#files

As a result, we can achieve 2 by vendoring kedro-viz source code itself, but we cannot avoid 1 completely. This creates a slightly weird situation, as we need to ask user to install pydantic with pip install pydantic>=2.0.0 in order to make this integration work. (it may be nicer if we can make it works with kedro-viz[core] but that requires breaking change on viz.

@astrojuanlu

[astrojuanlu]

I get that it's not possible to tell VSCode "install these Python extensions in the extension environment with pip", right? (Since then pip would figure out the proper wheel to download)

If that's the case, in the understanding that there's no ideal solution in the short term, I'd rather ask the user to pip install kedro-viz rather than half-vendoring it.

[jitu5]

@astrojuanlu @noklam
I figured a way to install pydantic and orjson with pip just before extension gets activated.
Please refer 5beaca2 I tested this and it worked for me.

Today I was testing on fresh virtual env and realised that there are "6" core Kedro-Viz dependencies we needed:

• kedro-viz
• networkx>=2.5
• pydantic>=2.0.0 (and pydantic_core)
• orjson
• sqlalchemy>=1.4, <3
• fastapi>=0.100.0,<0.200.0

Below 4 will be packaged with extension

• networkx>=2.5
• kedro-viz
• sqlalchemy>=1.4, <3
• fastapi>=0.100.0,<0.200.0

Below 2 will be installed in the extension environment with pip in bundled/libs

• pydantic>=2.0.0 (and pydantic_core)
• orjson

Please review 5beaca2 and let me know wdyt.

[noklam]

@jitu5 I don't think it's a good idea to install package into user environment. I like the direction of what @astrojuanlu suggested.

I get that it's not possible to tell VSCode "install these Python extensions in the extension environment with pip", right?

Provided there is a way to know where are all extension installed, and we have permission to do pip install --target <location>, in my local environment, it would be
/Users/Nok_Lam_Chan/.vscode/extensions/kedro.kedro-0.2.0-rc1/bundled/lib, I am not sure how resilient this is in Windows and other OS. If we are okay that this may breaks in edge case, we can try to implement this.

[jitu5]

@jitu5 I don't think it's a good idea to install package into user environment. I like the direction of what @astrojuanlu suggested.

I get that it's not possible to tell VSCode "install these Python extensions in the extension environment with pip", right?

Provided there is a way to know where are all extension installed, and we have permission to do pip install --target <location>, in my local environment, it would be /Users/Nok_Lam_Chan/.vscode/extensions/kedro.kedro-0.2.0-rc1/bundled/lib, I am not sure how resilient this is in Windows and other OS. If we are okay that this may breaks in edge case, we can try to implement this.

@noklam here in the code 5beaca2 We are not installing it in the user environment but in /bundled/libs. However, you are right; we need to ensure it works properly on Windows and other OS.

[noklam]

Sorry I overlook that, if we can just get the path of lsp_server.py or extension.ts, then we can find the relative path to the folders.