From ce59b70c82a2b74ca70f0acb40b4e95c143207ed Mon Sep 17 00:00:00 2001 From: Dany Kaufman Date: Wed, 29 Mar 2023 14:00:19 +0300 Subject: [PATCH 1/2] update virtual-env.ipynb documentation for new jupy --- virtual-env.ipynb | 165 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 117 insertions(+), 48 deletions(-) diff --git a/virtual-env.ipynb b/virtual-env.ipynb index dcf650e3..3d7b1c90 100644 --- a/virtual-env.ipynb +++ b/virtual-env.ipynb @@ -1,23 +1,28 @@ { "cells": [ { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ - "# Creating Python Virtual Environments with conda" + "# Creating Python Virtual Environments with Conda" ] }, { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "- [Overview](#conda-virual-env-overview)\n", - "- [Setting Up a Virtual Environment Using conda](#setting-up-a-virtual-environment-using-conda)\n", - " - [Creating a conda Virtual Environment from a File](#creating-a-conda-environment-from-a-file)\n", - "- [Setting Up a RAPIDS conda Environment with cuDF and cuML](#setting-up-a-rapids-conda-environment)" + "- [Preconfigured Conda environments](#preconfigured-conda-environments)\n", + "- [Setting Up a Virtual Environment Using Conda](#setting-up-a-virtual-environment-using-conda)\n", + " - [Creating a Conda Virtual Environment from a File](#creating-a-conda-environment-from-a-file)\n", + " - [Cloning an existing Conda Virtual Environment](#cloning-an-existing-conda-environment)\n", + "- [Setting Up a RAPIDS Conda Environment with cuDF and cuML](#setting-up-a-rapids-conda-environment)" ] }, { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ @@ -26,78 +31,116 @@ "\n", "A Python virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects.\n", "Virtual environments make it easy to cleanly separate projects and avoid problems with different dependencies and version requirements across components.\n", - "The conda command-line interface (CLI) is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.\n", - "The Jupyter Notebook service of the Iguazio Data Science Platform (\"the platform\") comes pre-deployed with conda.\n", - "This tutorial explains how to use conda to create a Python virtual environment that will be available as a custom kernel in Jupyter Notebook.\n", - "For general information about using conda to create virtual environments, see the [conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)." + "The Conda command-line interface (CLI) is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.\n", + "The Jupyter service of the Iguazio Data Science Platform (\"the platform\") comes pre-deployed with Conda.\n", + "This tutorial explains how to use Conda to create a Python virtual environment that will be available as a custom kernel in JupyterLab.\n", + "For general information about using Conda to create virtual environments, see the [Conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)." ] }, { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## Preconfigured Conda environments\n", + "\n", + "Iguazio provides several preconfigured Conda environments that are included in the Jupyter pod:\n", + "- **base**: This is the default base environment that includes the Conda binaries. It should not be used for development purposes.\n", + "- **jupyter**: This environment incudes the JupyterLab server and all of its dependencies. It should not be used for development purposes.\n", + "- **mlrun-base**: This environment includes MLRun and all its dependencies preinstalled.\n", + "- **mlrun-extended**: This environment includes all the packages from mlrun-base, as well as additional packages such as TensorFlow, PyTorch, and scikit-learn, which are required for the demo notebooks.\n", + "\n", + "While Iguazio provides several preconfigured Conda environments that are included in the Jupyter pod, it's important to note that any Conda installations to these environments are not persistent. They will be reset when the Jupyter service is restarted.\n", + "\n", + "However, when these Conda environments are activated, the `PIP_PREFIX` and `PYTHONPATH` environment variables are automatically set to the data mount directory of the running user: `/User/.pythonlibs/`. This means that PIP installations will persist even after restarting the Jupyter pod.\n", + "\n", + "Any new Conda environment that is created or cloned will be fully located in the data mount directory of the running user, specifically at `/User/.conda/`. It is worth noting that these environments are fully persistent." + ] + }, + { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "\n", - "## Setting Up a Virtual Environment Using conda\n", + "## Setting Up a Virtual Environment Using Conda\n", "\n", - "Follow these steps from your Jupyter Notebook service to create a Python virtual environment using conda: \n", + "Follow these steps from your Jupyter service to create a Python virtual environment using Conda: \n", "\n", "1. Create a new terminal by selecting the **New Launcher** option (`+` icon) from the top action toolbar in the left sidebar, and then selecting **Terminal** from the main work area.
\n", - " The next steps should be executed from your new terminal, except where otherwise specified.\n", - "\n", + " The next steps should be executed from your new terminal, except where otherwise specified.
\n", "\n", "2. Create a new Python virtual environment by running the following command.\n", " Replace `` with your preferred virtual-environment name:\n", " ```sh\n", - " conda create -n ipykernel python=3.7\n", + " conda create -n \n", " ```\n", - "\n", - "
For example, the following command creates an environment named \"myenv\":
\n", + "
For example, the following command creates an environment named \"myenv\":\n", " ```sh\n", - " conda create -n myenv ipykernel python=3.7\n", + " conda create -n myenv\n", " ```\n", + "
\n", "\n", - "\n", - "3. Install the desired Python packages by running the following command.\n", - " Replace `` with the name of the environment that you created in the previous step and `` with the name of your selected package, optionally followed by `=`; you can specify multiple packages:\n", + "3. Activate the Conda environment by running the following command:\n", " ```sh\n", - " conda install -n [ ...]\n", + " conda activate \n", " ```\n", - "\n", - "
For example, the following command installs the SciPy, pandas version 0.25.0, and TensorFlow version 1.13.1 packages for a \"myenv\" environment:\n", + " \n", + "
For example, the following command activates an environment named \"myenv\":\n", " ```sh\n", - " conda install -n myenv scipy pandas=0.25.0 tensorflow=1.13.1 python=3.7\n", - " ```\n", + " conda activate myenv\n", + " ``` \n", + "
\n", "\n", + "4. Once you activate your environment, you can use either PIP or Conda to install the necessary Python packages.
To do this, replace `` with the name of the package you want to install, and optionally add `==` for PIP or `=` for Conda. You can specify multiple packages in the same command.
\n", "\n", - "4. Export your new virtual environment to an environment file in a platform data container by running the following command.\n", - " Replace `` with the name of a platform data container, `` with an optional relative container-directories path, and `` with the name of the environment that you created:\n", - " ```sh\n", - " conda env export -n > /v3io/[/]/.yaml\n", - " ```\n", - "\n", - "
It's recommended that you save the environment to a virtual-environments directory in your running-user home directory (**users/<running user>**).\n", - " For example, the following command creates a **users/<running user>/virtual_env/myenv.yaml** file:\n", - " ```sh\n", - " conda env export -n myenv > /v3io/users/$V3IO_USERNAME/virtual_env/myenv.yaml\n", - " ```\n", - " You can use the `User` data mount to the running-user directory to shorten this command (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n", - " ```sh\n", - " conda env -n myenv export > /User/virtual_env/myenv.yaml\n", - " ```\n", + " Using PIP:\n", + " ```sh\n", + " pip install [ ...]\n", + " ```\n", + "
Alternatively, using Conda:\n", + " ```sh\n", + " conda install [ ...]\n", + " ```\n", + " \n", + "
For example, the following command uses PIP to install the SciPy, pandas version 1.4.4, and TensorFlow version 2.9.3 packages for the \"myenv\" environment that we activated in the previous step:\n", + " ```sh\n", + " pip install scipy pandas==1.4.4 tensorflow==2.9.3\n", + " ```\n", + "
\n", "\n", + "5. Export your new virtual environment to an environment file in a platform data container by running the following command.\n", + " Replace `` with the name of a platform data container, `` with an optional relative container-directories path, and `` with the name of the environment that you created:\n", + " ```sh\n", + " conda env export -n > /v3io/[/]/.yaml\n", + " ```\n", + "
It is recommended that you save the environment to a virtual-environments directory in your running-user home directory (**/v3io/users/<running user>**).\n", + " \n", + " For example, the following command creates a **/v3io/users/<running user>/virtual_env/myenv.yaml** file:\n", + " ```sh\n", + " conda env export -n myenv > /v3io/users/$V3IO_USERNAME/virtual_env/myenv.yaml\n", + " ```\n", + " \n", + "
To shorten this command, you can use the `/User` data mount to the running-user directory (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n", + " ```sh\n", + " conda env -n myenv export > /User/virtual_env/myenv.yaml\n", + " ```\n", + "
\n", "\n", - "5. Refresh the JupyterLab UI to apply your changes.\n", - " After refreshing the UI, you should be able to see your new environment in the list of available kernels in Jupyter Notebook." + "6. Refresh the JupyterLab UI to apply your changes.
\n", + " After refreshing the UI, you should be able to see your new environment in the list of available kernels in JupyterLab." ] }, { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "\n", - "### Creating a conda Virtual Environment from a File\n", + "### Creating a Conda Virtual Environment from a File\n", "\n", - "If, for any reason, your conda environment variable is removed from Jupyter Notebook, you can easily deploy it again by using the YAML environment file that you exported in [Step 4](#conda-virt-env-setup-step-export-env) of the setup procedure:\n", + "If, for any reason, your Conda environment is removed from JupyterLab, you can easily deploy it again by using the YAML environment file that you exported in [Step 5](#conda-virt-env-setup-step-export-env) of the setup procedure:\n", "\n", "1. Open a new Jupyter terminal.\n", "\n", @@ -107,24 +150,50 @@ " conda env create --file /v3io/[/]/.yaml\n", " ```\n", "\n", - " For example, the following command loads a **users/<running user>/virtual_env/myenv.yaml** environment file; the command uses the `User` running-user directory data mount to the running-user directory in the \"users\" container:\n", + "
For example, the following command loads a /v3io/users/<running user>/virtual_env/myenv.yaml environment file.
\n", + " The command uses the `/User` running-user directory data mount to the running-user directory in the \"users\" container:\n", " ```sh\n", " conda env create --file /User/virtual_env/myenv.yaml\n", " ```" ] }, { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Cloning an existing Conda Virtual Environment\n", + "\n", + "To clone an existing Conda environment into a new one, follow these steps:\n", + "\n", + "1. Open a new Jupyter terminal.\n", + "\n", + "2. Create a new Conda enviroment by cloning an existing one.
\n", + " Replace `` with the name of the new environmnent and `` with the environment to be cloned:\n", + " ```sh\n", + " conda create -n --clone \n", + " ```\n", + " \n", + "
For example, the following command clones \"mlrun-base\" environment into \"mlrun-clone\":\n", + " ```sh\n", + " conda create -n mlrun-clone --clone mlrun-base\n", + " ``` \n" + ] + }, + { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "\n", - "## Setting Up a RAPIDS conda Environment with cuDF and cuML\n", + "## Setting Up a RAPIDS Conda Environment with cuDF and cuML\n", "\n", - "To use the cuDF and cuML RAPIDS libraries, you need to create a RAPIDS conda environment.\n", + "To use the cuDF and cuML RAPIDS libraries, you need to create a RAPIDS Conda environment.\n", "Use the following command to create a RAPIDS conda environment named `rapids`:\n", "\n", "```sh\n", - "conda create -n rapids -c rapidsai -c nvidia -c anaconda -c conda-forge -c defaults ipykernel rapids=0.17 python=3.7 cudatoolkit=11.0\n", + "conda create -n rapids -c rapidsai -c conda-forge -c nvidia rapids=22.12 cudatoolkit=11.7.0\n", "```" ] } @@ -150,4 +219,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} From 2b20a5f56939942beeb41a8aed1f3c80fa6e19cf Mon Sep 17 00:00:00 2001 From: Dany Kaufman Date: Thu, 13 Apr 2023 09:53:03 +0300 Subject: [PATCH 2/2] virtual-env.ipynb: review fixes 1 --- virtual-env.ipynb | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/virtual-env.ipynb b/virtual-env.ipynb index 3d7b1c90..197b478f 100644 --- a/virtual-env.ipynb +++ b/virtual-env.ipynb @@ -45,17 +45,17 @@ "\n", "## Preconfigured Conda environments\n", "\n", - "Iguazio provides several preconfigured Conda environments that are included in the Jupyter pod:\n", - "- **base**: This is the default base environment that includes the Conda binaries. It should not be used for development purposes.\n", - "- **jupyter**: This environment incudes the JupyterLab server and all of its dependencies. It should not be used for development purposes.\n", + "The platform provides several preconfigured Conda environments that are included in the Jupyter pod:\n", + "- **base**: This is the default base environment that includes the Conda binaries. Do not use it for development purposes.\n", + "- **jupyter**: This environment incudes the JupyterLab server and all of its dependencies. Do not use it for development purposes.\n", "- **mlrun-base**: This environment includes MLRun and all its dependencies preinstalled.\n", "- **mlrun-extended**: This environment includes all the packages from mlrun-base, as well as additional packages such as TensorFlow, PyTorch, and scikit-learn, which are required for the demo notebooks.\n", "\n", - "While Iguazio provides several preconfigured Conda environments that are included in the Jupyter pod, it's important to note that any Conda installations to these environments are not persistent. They will be reset when the Jupyter service is restarted.\n", + "While the platform provides several preconfigured Conda environments that are included in the Jupyter pod, it's important to note that any Conda installations to these environments are not persistent. They are reset when the Jupyter service is restarted.\n", "\n", - "However, when these Conda environments are activated, the `PIP_PREFIX` and `PYTHONPATH` environment variables are automatically set to the data mount directory of the running user: `/User/.pythonlibs/`. This means that PIP installations will persist even after restarting the Jupyter pod.\n", + "However, when these Conda environments are activated, the `PIP_PREFIX` and `PYTHONPATH` environment variables are automatically set to the data mount directory of the running user: `/User/.pythonlibs/`. This means that PIP installations persist even after restarting the Jupyter pod.\n", "\n", - "Any new Conda environment that is created or cloned will be fully located in the data mount directory of the running user, specifically at `/User/.conda/`. It is worth noting that these environments are fully persistent." + "Any new Conda environment that is created or cloned will be fully located in the data mount directory of the running user, specifically at `/User/.conda/`. These environments are fully persistent." ] }, { @@ -93,7 +93,7 @@ " ``` \n", "
\n", "\n", - "4. Once you activate your environment, you can use either PIP or Conda to install the necessary Python packages.
To do this, replace `` with the name of the package you want to install, and optionally add `==` for PIP or `=` for Conda. You can specify multiple packages in the same command.
\n", + "4. Once you activate your environment, use either PIP or Conda to install the necessary Python packages.
To do this, replace `` with the name of the package you want to install, and optionally add `==` for PIP or `=` for Conda. You can specify multiple packages in the same command.
\n", "\n", " Using PIP:\n", " ```sh\n", @@ -104,7 +104,7 @@ " conda install [ ...]\n", " ```\n", " \n", - "
For example, the following command uses PIP to install the SciPy, pandas version 1.4.4, and TensorFlow version 2.9.3 packages for the \"myenv\" environment that we activated in the previous step:\n", + "
For example, the following command uses PIP to install the SciPy, pandas version 1.4.4, and TensorFlow version 2.9.3 packages for the \"myenv\" environment that you activated in the previous step:\n", " ```sh\n", " pip install scipy pandas==1.4.4 tensorflow==2.9.3\n", " ```\n", @@ -122,14 +122,14 @@ " conda env export -n myenv > /v3io/users/$V3IO_USERNAME/virtual_env/myenv.yaml\n", " ```\n", " \n", - "
To shorten this command, you can use the `/User` data mount to the running-user directory (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n", + "
To shorten this command, use the `/User` data mount to the running-user directory (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n", " ```sh\n", " conda env -n myenv export > /User/virtual_env/myenv.yaml\n", " ```\n", "
\n", "\n", "6. Refresh the JupyterLab UI to apply your changes.
\n", - " After refreshing the UI, you should be able to see your new environment in the list of available kernels in JupyterLab." + " After refreshing the UI, you should see your new environment in the list of available kernels in JupyterLab." ] }, {