kedro not importing installed module when pipeline is run

[meaningfromdata]

I'm trying to use some great expectations functions in my kedro pipeline by creating data validation nodes (I know there's a kedro hook but want to try this way first). I've pip installed great expectations into the conda environment with kedro and have been able to import great expectations in other non-kedro test code I've written in this environment without a problem. However, when I run the project pipeline from the project root folder with "kedro run" I am getting a "ModuleNotFoundError".

great_expectations is installed in the site-packages directory (Anaconda3\envs\project_env\lib\site-packages) for this environment and this directory appears when I print sys.path from the environment. Also, the import works fine when I run it from the nodes.py file in the kedro project. It seems that it's only when I run the pipeline that the import failure occurs. Any help getting this module imported without issue would be appreciated. I am using kedro 0.17.6 with Python 3.7.11

[antonymilne]

Hello @meaningfromdata! If you do which kedro, where does it point to - the one in your environment?

[meaningfromdata]

Hi, @AntonyMilneQB. which kedro lists the correct environment, however in traceback of kedro run where I am getting the import error, all of the site-packages\kedro\framework\session or framework\project files are still being sourced from an old environment with kedro 0.17.6 that I cloned before installing great expectations in the new environment. Could this be causing the issue? Should I avoid use of conda clone? I confess, I'm not quite sure what the proper procedure is to install required packages with kedro while I'm still figuring out which packages/versions will work together. It's hard to specify a priori all the packages I want to include in the requirements.txt file.

[meaningfromdata]

@AntonyMilneQB I just created a new conda environment from scratch and did pip install -r requirements.txt with as many packages as I currently anticipate needing for the project (including great expectations == 0.14.2). This allowed me to run the pipeline and import great_expectations without an error. So it seems that using conda clone to clone a working kedro environment before adding a new package of interest for the project with pip install is not a good idea.

[antonymilne]

Ah yes, this sounds like an issue with conda clone. My guess would be that the console scripts entrypoint used by setuptools is not very compatible with conda, so that when you do conda clone it can't update its reference to the kedro in the new environment.

This is actually very useful to know, because I've seen similarly mysterious things happening before and couldn't figure out what the original source of the problem was. It seems very plausible that this was due to people creating environments using conda clone rather than conda new. FYI @datajoely

So my recommended flow would be:

1. Use conda new
2. Keep track of dependencies using kedro build-reqs which uses pip-tools under the hood
3. Once you've got the packages you want installed, if you want to recreate the environment then do conda new and pip-sync the requirements.txt generated by kedro build-reqs

Generally speaking I use pip-tools for dependency management and conda just for separating different virtual environments, and things work well.

[antonymilne]

Small disclaimer if you do use kedro build-reqs: due to a pip-tools bug you'll need to use pip-tools 6.5 if you're on pip>=22.

[meaningfromdata]

@AntonyMilneQB Thanks, good to know what your recommended environment setup is and seems to confirm process I just stumbled into through trial and error. Will continue testing and post again should anything else turn out to be broken.