Programming Example with IPython Notebook

[hunhoffe]

First attempt at an ipython notebook for a programming example.

I'm not attempting to solve all problems with ipython notebooks, but rather provide a basis that might become a more comprehensive solution with further development.

How it works:

• The notebook contains the design code (e.g. aie2.py) and test code (test.py)
• The notebook calls aiecc, not through the python interface, but through ! commands in the notebook which are executed directly. So this is not an end-to-end python solution - I do this purposefully to demonstrate that using notebooks does not demand an end-to-end python (even though that would be the ideal experience for the programming examples, in my opinion).
• To execute, the notebook needs an ipykernel which is build around the venv (either sandbox or ironenv); I've updated the quick_setup.sh script and the setup_python_packages.sh to do this automatically.
• To test, I use jupyter nbconvert to convert the notebook into a python script, and then run it through new makefile targets(make run_notebook and make clean_notebook), which are then called in a lit test.
• Adds a git pre-commit hook (defined in .pre-commit-config.yaml) using the pre-commit tool. The pre-commit will scrub notebooks of output and most metadata, but allow one to check the scrubbing before the commit can succeed. The action uses the nbstripout tool. The pre-commit that is installed depends on the venv, so I standardized the venv name between the two setup paths (quick_setup.sh and setup_python_packages.sh); this will also help standardize information in READMEs about choosing the ipykernel name used for ipython notebooks.

VSCode

Even with creating an ipykernel based on the mlir-aie venv, I have not successfully found out how to run the notebook in VS Code. It does work for me locally using a jupyter server started at the root of mlir-aie in a terminal where I've run the env setup script.

There are two problems: Python path finding mlir-aie/install/python and other environment variables set by env_steup.sh. The first problem is fairly easily solved through using a .pth file in the site-packages directory of the venv.

The other environment variables are more problematic. Things I have tried to get vscode working:

• Adding env values to the ipykernel definition. I don't know why this doesn't work, it feels like it should, but I tried like so by editing the setup_env.sh script:

  # Create ipykernel with associated environment variables
  python3 -m ipykernel install --user --name ironenv \
    --env MLIR_AIE_INSTALL_DIR "$MLIR_AIE_INSTALL_DIR" \
    --env LLVM_INSTALL_DIR "$LLVM_INSTALL_DIR" \
    --env PEANO_INSTALL_DIR "$PEANO_INSTALL_DIR" \
    --env PATH "$PATH" \
    --env LD_LIBRARY_PATH "$LD_LIBRARY_PATH"

• Setting envs in the vscode settings for jupyter startup commands... I think maybe this works, but it gets ugly really fast so I don't consider this a better solution than running the jupyter server yourself. When I say ugly, I mean a solution like below:

  import os, subprocess
  from subprocess import check_output
  output = check_output("source /scratch/ehunhoff/mlir-aie/utils/env_setup.sh /scratch/ehunhoff/mlir-aie/install /scratch/ehunhoff/mlir-aie/llvm/install /scratch/ehunhoff/mlir-aie/my_install/llvm-aie; env -0", shell=True, executable="/bin/bash")
  envs = output.split(b'\x00')
  # then parse out each env and set it individually somehow? May or may not work, but it's not great.

[hunhoffe]

Note to self: look into github commit hook to scrub jupyter notebook outputs automatically

[hunhoffe]

The pre-commit setup I used is a little annoying (it's very visible during the commit process) but it's also quite functional, uses popular tools, and the output is easy to understand.

There are also some other cool built-in hooks that could be useful if added in a future PR, such as:

            - id: end-of-file-fixer
            - id: mixed-line-ending
            - id: trailing-whitespace

I know I've had problems with these things in some PRs.

[hunhoffe]

Re: VS Code, I am able to import the python directly be using a .pth file instead of PYTHONPATH to point to $MLIR_INSTALL_DIR/python.

However, this does not give me all of the environment variables needed to compile from the python notebook. I need to find a way to run the env setup script in a way that percolates that environment to the environment seen by jupyter.

[jgmelber]

VSCode

Even with creating an ipykernel based on the mlir-aie venv, I have not successfully found out how to run the notebook in VS Code. It does work for me locally using a jupyter server started at the root of mlir-aie in a terminal where I've run the env setup script.

@STFleming Have you been able to get Jupyter notebooks working in VSCode?

[jgmelber]

I really like the notebook style for examples, this would be a huge improvement for the programming guide examples in particular.

The VSCode integration could be a WIP. It should be documented in the guide/examples the steps to set up the server and connect though.

[hunhoffe]

Tested in jupyter lab environment and seemed to work well! My current plan is to merge even without the notebook working in VS Code, once a few more people look at the PR.

[fifield]

Works for me! For vscode I was able to start jupyterlab from the command line then tell vscode to connect to the existing server. Then the notebook ran ok. Trying to use the ipython kernel directly in vscode I have some LD_LIBRARY_PATH issue. Standalone jupyter I had no issues (but without following directions or using quick_setup)

[jgmelber]

Could you comment the new additions to the setup scripts? (A follow on PR will document the existing.)

[hunhoffe]

@jgmelber Comments have been added to setup script!