notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic"
      },
      "source": [
        "# Vertex AI Pipelines: Metrics visualization and run comparison using the KFP SDK\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fpipelines%2Fmetrics_viz_run_compare_kfp.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> \n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\" target='_blank'>\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> \n",
        "      Open in Vertex AI Workbench\n",
        "     </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:pipelines,metrics"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to use [the Kubeflow Pipelines (KFP) SDK](https://www.kubeflow.org/docs/components/pipelines/) to build [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) that generate model metrics and metrics visualizations, and comparing pipeline runs.\n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:pipelines,metrics"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use the KFP SDK for Python to build pipelines that generate evaluation metrics.\n",
        "\n",
        "This tutorial uses the following Vertex AI services:\n",
        "\n",
        "- Vertex AI Pipelines\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create KFP components:\n",
        "    - Generate ROC curve and confusion matrix visualizations for classification results\n",
        "    - Write metrics\n",
        "- Create KFP pipelines.\n",
        "- Execute KFP pipelines\n",
        "- Compare metrics across pipeline runs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:wine,lcn,sklearn"
      },
      "source": [
        "### Datasets\n",
        "\n",
        "The dataset used for this tutorial is the [Wine dataset](https://archive.ics.uci.edu/ml/datasets/wine) from [Scikit-learn builtin datasets](https://scikit-learn.org/stable/datasets.html).\n",
        "\n",
        "The dataset predicts the origin of a wine."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:iris,lcn,sklearn"
      },
      "source": [
        "The dataset used for this tutorial is the [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) from [Scikit-learn builtin datasets](https://scikit-learn.org/stable/datasets.html).\n",
        "\n",
        "The dataset predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1ea81ac77f0"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
        "                                 google-cloud-storage \\\n",
        "                                 kfp \\\n",
        "                                 google-cloud-pipeline-components\n",
        "\n",
        "if os.getenv(\"IS_TESTING\"):\n",
        "    ! pip3 install --upgrade matplotlib $USER_FLAG -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b96b39fd4d7b"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc7251520a07"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### UUID\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of a specifed length(default=8)\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    if IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip\n",
        "from kfp import compiler, dsl\n",
        "from kfp.dsl import ClassificationMetrics, Metrics, Output, component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pipeline_constants"
      },
      "source": [
        "#### Vertex AI constants\n",
        "\n",
        "Setup up the following constants for Vertex AI pipelines:\n",
        "- `PIPELINE_NAME`: Set name for the pipeline.\n",
        "- `PIPELINE_ROOT`: Cloud Storage bucket path to store pipeline artifacts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pipeline_constants"
      },
      "outputs": [],
      "source": [
        "PIPELINE_NAME = \"metrics-pipeline-v2\"\n",
        "PIPELINE_ROOT = \"{}/pipeline_root/iris\".format(BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:wine_classification"
      },
      "source": [
        "### Define pipeline components using scikit-learn\n",
        "\n",
        "In this section, you define some Python function-based components that use scikit-learn to train some classifiers and produce evaluations that can be visualized.\n",
        "\n",
        "Note the use of the `@component()` decorator in the definitions below. Optionally, you can set a list of packages for the component to install. That is, list the base image to use (the default is a Python 3.7 image), and the name of a component YAML file to generate, so that the component definition can be shared and reused.\n",
        "\n",
        "#### Define wine_classification component\n",
        "\n",
        "The first component shows how to visualize an *ROC curve*.\n",
        "Note that the function definition includes an output called `wmetrics`, of type `Output[ClassificationMetrics]`. You can visualize the metrics in the pipeline's user interface in the Google Cloud console.\n",
        "\n",
        "To do so, this example uses the artifact's `log_roc_curve()` method. This method takes input arrays with the false positive rates, true positive rates, and thresholds, as [generated by the `sklearn.metrics.roc_curve` function](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html).\n",
        "\n",
        "When you evaluate the cell below, a task factory function called `wine_classification` is created that is used to construct the pipeline definition.  In addition, a component YAML file is created, which can be shared and loaded via file or URL to create the same task factory function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_component:wine_classification"
      },
      "outputs": [],
      "source": [
        "@component(\n",
        "    packages_to_install=[\"scikit-learn==1.2\", \"numpy==1.26.4\"], base_image=\"python:3.9\"\n",
        ")\n",
        "def wine_classification(wmetrics: Output[ClassificationMetrics]):\n",
        "    from sklearn.datasets import load_wine\n",
        "    from sklearn.ensemble import RandomForestClassifier\n",
        "    from sklearn.metrics import roc_curve\n",
        "    from sklearn.model_selection import cross_val_predict, train_test_split\n",
        "\n",
        "    X, y = load_wine(return_X_y=True)\n",
        "    # Binary classification problem for label 1.\n",
        "    y = y == 1\n",
        "\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
        "    rfc = RandomForestClassifier(n_estimators=10, random_state=42)\n",
        "    rfc.fit(X_train, y_train)\n",
        "    y_scores = cross_val_predict(rfc, X_train, y_train, cv=3, method=\"predict_proba\")\n",
        "    fpr, tpr, thresholds = roc_curve(\n",
        "        y_true=y_train, y_score=y_scores[:, 1], pos_label=True\n",
        "    )\n",
        "    wmetrics.log_roc_curve(fpr, tpr, thresholds)\n",
        "\n",
        "\n",
        "compiler.Compiler().compile(wine_classification, \"wine_classification_component.yaml\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:iris_sgdclassifier"
      },
      "source": [
        "#### Define iris_sgdclassifier component\n",
        "\n",
        "The second component shows how to visualize a *confusion matrix*, in this case for a model trained using `SGDClassifier`.\n",
        "\n",
        "As with the previous component, you create a `metricsc` output artifact of type `Output[ClassificationMetrics]`.  Then, use the artifact's `log_confusion_matrix` method to visualize the confusion matrix results, as generated by the [sklearn.metrics.confusion_matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_component:iris_sgdclassifier"
      },
      "outputs": [],
      "source": [
        "@component(\n",
        "    packages_to_install=[\"scikit-learn==1.2\", \"numpy==1.26.4\"], base_image=\"python:3.9\"\n",
        ")\n",
        "def iris_sgdclassifier(\n",
        "    test_samples_fraction: float,\n",
        "    metricsc: Output[ClassificationMetrics],\n",
        "):\n",
        "    from sklearn import datasets, model_selection\n",
        "    from sklearn.linear_model import SGDClassifier\n",
        "    from sklearn.metrics import confusion_matrix\n",
        "\n",
        "    iris_dataset = datasets.load_iris()\n",
        "    train_x, test_x, train_y, test_y = model_selection.train_test_split(\n",
        "        iris_dataset[\"data\"],\n",
        "        iris_dataset[\"target\"],\n",
        "        test_size=test_samples_fraction,\n",
        "    )\n",
        "\n",
        "    classifier = SGDClassifier()\n",
        "    classifier.fit(train_x, train_y)\n",
        "    predictions = model_selection.cross_val_predict(classifier, train_x, train_y, cv=3)\n",
        "    metricsc.log_confusion_matrix(\n",
        "        [\"Setosa\", \"Versicolour\", \"Virginica\"],\n",
        "        confusion_matrix(\n",
        "            train_y, predictions\n",
        "        ).tolist(),  # .tolist() to convert np array to list.\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:iris_logregression"
      },
      "source": [
        "#### Define iris_logregression component\n",
        "\n",
        "The third component also uses the \"iris\" dataset, but trains a `LogisticRegression` model.  It logs model `accuracy` in the `metrics` output artifact."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_component:iris_logregression"
      },
      "outputs": [],
      "source": [
        "@component(\n",
        "    packages_to_install=[\"scikit-learn==1.2\", \"numpy==1.26.4\"],\n",
        "    base_image=\"python:3.9\",\n",
        ")\n",
        "def iris_logregression(\n",
        "    input_seed: int,\n",
        "    split_count: int,\n",
        "    metrics: Output[Metrics],\n",
        "):\n",
        "    from sklearn import datasets, model_selection\n",
        "    from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "    # Load digits dataset\n",
        "    iris = datasets.load_iris()\n",
        "    # # Create feature matrix\n",
        "    X = iris.data\n",
        "    # Create target vector\n",
        "    y = iris.target\n",
        "    # test size\n",
        "    test_size = 0.20\n",
        "\n",
        "    # cross-validation settings\n",
        "    kfold = model_selection.KFold(\n",
        "        n_splits=split_count, random_state=input_seed, shuffle=True\n",
        "    )\n",
        "    # Model instance\n",
        "    model = LogisticRegression()\n",
        "    scoring = \"accuracy\"\n",
        "    results = model_selection.cross_val_score(model, X, y, cv=kfold, scoring=scoring)\n",
        "    print(f\"results: {results}\")\n",
        "\n",
        "    # split data\n",
        "    X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
        "        X, y, test_size=test_size, random_state=input_seed\n",
        "    )\n",
        "    # fit model\n",
        "    model.fit(X_train, y_train)\n",
        "\n",
        "    # accuracy on test set\n",
        "    result = model.score(X_test, y_test)\n",
        "    print(f\"result: {result}\")\n",
        "    metrics.log_metric(\"accuracy\", (result * 100.0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_pipeline:metrics"
      },
      "source": [
        "### Define the pipeline\n",
        "\n",
        "Next, define a simple pipeline that uses the above components.\n",
        "\n",
        "**Note:** In the `@dsl.pipeline` decorator, you define `PIPELINE_ROOT` as the Cloud Storage path that's used as root folder. You can choose to skip it, but you have to provide it when creating the pipeline run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_pipeline:metrics"
      },
      "outputs": [],
      "source": [
        "@dsl.pipeline(\n",
        "    # Default pipeline root. You can override it when submitting the pipeline.\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    # A name for the pipeline.\n",
        "    name=\"metrics-pipeline-v2\",\n",
        ")\n",
        "def pipeline(seed: int, splits: int):\n",
        "    wine_classification_op = wine_classification()  # noqa: F841\n",
        "    iris_logregression_op = iris_logregression(  # noqa: F841\n",
        "        input_seed=seed, split_count=splits\n",
        "    )\n",
        "    iris_sgdclassifier_op = iris_sgdclassifier(test_samples_fraction=0.3)  # noqa: F841"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compile_pipeline"
      },
      "source": [
        "## Compile the pipeline\n",
        "\n",
        "Next, compile the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "compile_pipeline"
      },
      "outputs": [],
      "source": [
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline,\n",
        "    package_path=\"tabular_classification_pipeline.yaml\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline:metrics"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "Next, run the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_pipeline:metrics"
      },
      "outputs": [],
      "source": [
        "DISPLAY_NAME = \"iris_\" + UUID\n",
        "\n",
        "job1 = aip.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=\"tabular_classification_pipeline.yaml\",\n",
        "    job_id=f\"tabular-classification-v2{UUID}-1\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\"seed\": 7, \"splits\": 10},\n",
        ")\n",
        "\n",
        "job1.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_run:metrics"
      },
      "source": [
        "Click on the generated link to see your run in the Google Cloud console.\n",
        "\n",
        "<!-- It should look something like this as it is running:\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" width=\"40%\"/></a> -->\n",
        "\n",
        "In the Google Cloud console, many of the pipeline DAG nodes expand or collapse when you click them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compare_pipeline_runs:ui"
      },
      "source": [
        "## Comparing pipeline runs in the Google Cloud console\n",
        "\n",
        "Next, generate another pipeline run that uses a different `seed` and `split` for the `iris_logregression` step.\n",
        "\n",
        "Pass the input parameters required for the pipeline and run it. The defined pipeline takes the following parameters:\n",
        "\n",
        "- `pipeline_root`: Specify a Cloud Storage URI that your pipelines service account can access. The artifacts of your pipeline runs are stored within the pipeline root. \n",
        "- `display_name`: The name of the pipeline, that shows up in the Google Cloud console. \n",
        "- `parameter_values`: The pipeline parameters to pass to this run. For example, create a dict() with the parameter names as the dictionary keys and the parameter values as the dictionary values. \n",
        "- `job_id`: A unique identifier for this pipeline run. If the job ID is not specified, Vertex AI Pipelines creates a job ID for you using the pipeline name and the timestamp of when the pipeline run was started. \n",
        "- `template_path`: Complete pipeline path"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "compare_pipeline_runs:ui"
      },
      "outputs": [],
      "source": [
        "job2 = aip.PipelineJob(\n",
        "    display_name=\"iris_\" + UUID,\n",
        "    template_path=\"tabular_classification_pipeline.yaml\",\n",
        "    job_id=f\"tabular-classification-pipeline-v2{UUID}-2\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\"seed\": 5, \"splits\": 7},\n",
        ")\n",
        "\n",
        "job2.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compare_pipeline_runs:ui"
      },
      "source": [
        "When both pipeline runs have finished, compare their results by navigating to the pipeline runs list in the Google Cloud console, selecting both of them, and clicking **COMPARE** at the top of the Console panel."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compare_pipeline_runs"
      },
      "source": [
        "## Compare the parameters and metrics of the pipelines run from their tracked metadata\n",
        "\n",
        "Next, you use the Vertex AI SDK for Python to compare the parameters and metrics of the pipeline runs. Wait until the pipeline runs have finished to run the next cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "compare_pipeline_runs"
      },
      "outputs": [],
      "source": [
        "pipeline_df = aip.get_pipeline_df(pipeline=PIPELINE_NAME)\n",
        "print(pipeline_df.head(2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plot_coord_runs"
      },
      "source": [
        "### Plot parallel coordinates of parameters and metrics\n",
        "\n",
        "With the metric and parameters in a dataframe, you can perform further analysis to extract useful information. The following example compares data from each run using a parallel coordinate plot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "plot_coord_runs"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "plt.rcParams[\"figure.figsize\"] = [15, 5]\n",
        "\n",
        "pipeline_df[\"param.input:seed\"] = pipeline_df[\"param.input:seed\"].astype(np.float16)\n",
        "pipeline_df[\"param.input:splits\"] = pipeline_df[\"param.input:splits\"].astype(np.float16)\n",
        "\n",
        "ax = pd.plotting.parallel_coordinates(\n",
        "    pipeline_df.reset_index(level=0),\n",
        "    \"run_name\",\n",
        "    cols=[\"param.input:seed\", \"param.input:splits\", \"metric.accuracy\"],\n",
        ")\n",
        "ax.set_yscale(\"symlog\")\n",
        "ax.legend(bbox_to_anchor=(1.0, 0.5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plot_roc_runs"
      },
      "source": [
        "### Plot ROC curve and calculate AUC number\n",
        "\n",
        "In addition to basic metrics, you can extract complex metrics and perform further analysis using the `get_pipeline_df` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "plot_roc_runs"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    df = pd.DataFrame(pipeline_df[\"metric.confidenceMetrics\"][0])\n",
        "    auc = np.trapz(df[\"recall\"], df[\"falsePositiveRate\"])\n",
        "    plt.plot(df[\"falsePositiveRate\"], df[\"recall\"], label=\"auc=\" + str(auc))\n",
        "    plt.legend(loc=4)\n",
        "    plt.show()\n",
        "except Exception as e:\n",
        "    print(e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "outputs": [],
      "source": [
        "delete_bucket = False\n",
        "\n",
        "job1.delete()\n",
        "job2.delete()\n",
        "\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI\n",
        "\n",
        "! rm -rf tabular_classification_pipeline.yaml"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "metrics_viz_run_compare_kfp.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}