gemini/evaluation/customize_model_based_metrics.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s6u1FiU7N3Jm"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kOCmYX5Rc3BZ"
      },
      "source": [
        "# Customize Model-based Metrics to Evaluate a Gen AI model | Gen AI Evaluation SDK Tutorial\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fcustomize_model_based_metrics.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/customize_model_based_metrics.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/customize_model_based_metrics.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kPgYxQc1OvFn"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Jason Dai](https://github.com/jsondai), [Naveksha Sood](https://github.com/navekshasood) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ITMM0euN84i"
      },
      "source": [
        "## Overview\n",
        "\n",
        "In this notebook, you'll learn how to use the *Vertex AI Python SDK for Gen AI Evaluation Service* to customize the model-based metrics and evaluate a generative AI model based on your criteria.\n",
        "\n",
        "This notebook demonstrates:\n",
        "* Templated customization using pre-defined fields for pointwise and pairwise model-based metrics\n",
        "\n",
        "* Fully customized pointwise and pairwise model-based metrics\n",
        "\n",
        "\n",
        "See also: \n",
        "\n",
        "- Learn more about [Vertex Gen AI Evaluation Service SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview).\n",
        "\n",
        "- Learn more about how to [define your evaluation metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBw0sVZesunq"
      },
      "source": [
        "## Get Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMllZtZ1OEfC"
      },
      "source": [
        "### Install Vertex AI Python SDK for Gen AI Evaluation Service"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l-lBvYgzzlkE"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-cloud-aiplatform[evaluation]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuT9TvcktTBV"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3AdDLYUSmN0T"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWokIb9kmRZ5"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yIYidy--mOmG"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8lBlSBaaznrG"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "en_HfAkdtcRn"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V8M4F84GthAr"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "EXPERIMENT_NAME = \"customize-metrics\"  # @param {type:\"string\"}\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    raise ValueError(\"Please set your PROJECT_ID\")\n",
        "\n",
        "\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWTezEusurJU"
      },
      "source": [
        "### Import libraries\n",
        "\n",
        "Import the Vertex AI Python SDK and other required Python libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cpoVotOuzqsb"
      },
      "outputs": [],
      "source": [
        "# General\n",
        "import inspect\n",
        "\n",
        "from IPython.display import HTML, Markdown, display\n",
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "\n",
        "# Main\n",
        "from vertexai.evaluation import (\n",
        "    EvalTask,\n",
        "    MetricPromptTemplateExamples,\n",
        "    PairwiseMetric,\n",
        "    PairwiseMetricPromptTemplate,\n",
        "    PointwiseMetric,\n",
        "    PointwiseMetricPromptTemplate,\n",
        ")\n",
        "from vertexai.generative_models import GenerativeModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42svczJqjZOw"
      },
      "source": [
        "### Helper functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bLbaS-67jZOw"
      },
      "outputs": [],
      "source": [
        "def print_doc(function):\n",
        "    print(f\"{function.__name__}:\\n{inspect.getdoc(function)}\\n\")\n",
        "\n",
        "\n",
        "def display_eval_result(eval_result, title=None, metrics=None):\n",
        "    \"\"\"Display the evaluation results.\"\"\"\n",
        "    summary_metrics, metrics_table = (\n",
        "        eval_result.summary_metrics,\n",
        "        eval_result.metrics_table,\n",
        "    )\n",
        "\n",
        "    metrics_df = pd.DataFrame.from_dict(summary_metrics, orient=\"index\").T\n",
        "    if metrics:\n",
        "        metrics_df = metrics_df.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in metrics_df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "        metrics_table = metrics_table.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in metrics_table.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    if title:\n",
        "        # Display the title with Markdown for emphasis\n",
        "        display(Markdown(f\"## {title}\"))\n",
        "    # Display the summary metrics DataFrame\n",
        "    display(Markdown(\"### Summary Metrics\"))\n",
        "    display(metrics_df)\n",
        "    # Display the metrics table DataFrame\n",
        "    display(Markdown(\"### Row-based Metrics\"))\n",
        "    display(metrics_table)\n",
        "\n",
        "\n",
        "def display_explanations(eval_result, metrics=None, n=1):\n",
        "    \"\"\"Display the explanations.\"\"\"\n",
        "    style = \"white-space: pre-wrap; width: 1500px; overflow-x: auto;\"\n",
        "    metrics_table = eval_result.metrics_table\n",
        "    df = metrics_table.sample(n=n)\n",
        "\n",
        "    if metrics:\n",
        "        df = df.filter(\n",
        "            [\"response\", \"baseline_model_response\"]\n",
        "            + [\n",
        "                metric\n",
        "                for metric in df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "    for index, row in df.iterrows():\n",
        "        for col in df.columns:\n",
        "            display(HTML(f\"<div style='{style}'><h4>{col}:</h4>{row[col]}</div>\"))\n",
        "        display(HTML(\"<hr>\"))\n",
        "\n",
        "\n",
        "def plot_radar_plot(eval_results, max_score=5, metrics=None):\n",
        "    fig = go.Figure()\n",
        "\n",
        "    for eval_result in eval_results:\n",
        "        title, summary_metrics, report_df = eval_result\n",
        "\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k: summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric in k for selected_metric in metrics)\n",
        "            }\n",
        "\n",
        "        fig.add_trace(\n",
        "            go.Scatterpolar(\n",
        "                r=list(summary_metrics.values()),\n",
        "                theta=list(summary_metrics.keys()),\n",
        "                fill=\"toself\",\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig.update_layout(\n",
        "        polar=dict(radialaxis=dict(visible=True, range=[0, max_score])), showlegend=True\n",
        "    )\n",
        "\n",
        "    fig.show()\n",
        "\n",
        "\n",
        "def display_radar_plot(eval_results, metrics=None):\n",
        "    \"\"\"Plot the radar plot.\"\"\"\n",
        "    fig = go.Figure()\n",
        "    for item in eval_results:\n",
        "        eval_result, title = item\n",
        "        summary_metrics = eval_result.summary_metrics\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k.replace(\"/mean\", \"\"): summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric + \"/mean\" in k for selected_metric in metrics)\n",
        "            }\n",
        "        fig.add_trace(\n",
        "            go.Scatterpolar(\n",
        "                r=list(summary_metrics.values()),\n",
        "                theta=list(summary_metrics.keys()),\n",
        "                fill=\"toself\",\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "    fig.update_layout(\n",
        "        polar=dict(radialaxis=dict(visible=True, range=[0, 5])), showlegend=True\n",
        "    )\n",
        "    fig.show()\n",
        "\n",
        "\n",
        "def display_bar_plot(eval_results_list, metrics=None):\n",
        "    fig = go.Figure()\n",
        "    data = []\n",
        "\n",
        "    for eval_results in eval_results_list:\n",
        "        eval_result, title = eval_results[0], eval_results[1]\n",
        "\n",
        "        summary_metrics = eval_result.summary_metrics\n",
        "        mean_summary_metrics = [f\"{metric}/mean\" for metric in metrics]\n",
        "        updated_summary_metrics = []\n",
        "        if metrics:\n",
        "            for k, v in summary_metrics.items():\n",
        "                if k in mean_summary_metrics:\n",
        "                    updated_summary_metrics.append((k, v))\n",
        "            summary_metrics = dict(updated_summary_metrics)\n",
        "            # summary_metrics = {k: summary_metrics[k] for k, v in summary_metrics.items() if any(selected_metric in k for selected_metric in metrics)}\n",
        "\n",
        "        data.append(\n",
        "            go.Bar(\n",
        "                x=list(summary_metrics.keys()),\n",
        "                y=list(summary_metrics.values()),\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig = go.Figure(data=data)\n",
        "\n",
        "    # Change the bar mode\n",
        "    fig.update_layout(barmode=\"group\", showlegend=True)\n",
        "    fig.show()\n",
        "\n",
        "\n",
        "def sample_pairwise_result(eval_result, n=1, metric=None):\n",
        "    \"\"\"Display a random row of pairwise metric result with model responses.\"\"\"\n",
        "    style = \"white-space: pre-wrap; width: 800px; overflow-x: auto;\"\n",
        "    df = eval_result.metrics_table\n",
        "    df = df.sample(n=n)\n",
        "    for index, row in df.iterrows():\n",
        "        display(HTML(f\"<h2>Prompt:</h2> <div style='{style}'>{row['prompt']}</div>\"))\n",
        "        display(\n",
        "            HTML(\n",
        "                f\"<h2>Baseline Model Response:</h2> <div style='{style}'>{row['baseline_model_response']}</div>\"\n",
        "            )\n",
        "        )\n",
        "        display(\n",
        "            HTML(\n",
        "                f\"<h2>Candidate Model Response:</h2> <div style='{style}'>{row['response']}</div>\"\n",
        "            )\n",
        "        )\n",
        "        display(\n",
        "            HTML(\n",
        "                f\"<h2>Explanation:</h2> <div style='{style}'>{row[f'{metric}/explanation']}</div>\"\n",
        "            )\n",
        "        )\n",
        "        display(\n",
        "            HTML(\n",
        "                f\"<h2>Winner:</h2> <div style='{style}'>{row[f'{metric}/pairwise_choice']}</div>\"\n",
        "            )\n",
        "        )\n",
        "        display(HTML(\"<hr>\"))\n",
        "\n",
        "\n",
        "def display_pairwise_win_rate(eval_result, metric=None):\n",
        "    \"\"\"Display pairwise aggregated metrics\"\"\"\n",
        "    summary_metrics = eval_result.summary_metrics\n",
        "    candidate_model_win_rate = round(\n",
        "        summary_metrics[f\"{metric}/candidate_model_win_rate\"] * 100\n",
        "    )\n",
        "    display(\n",
        "        HTML(\n",
        "            f\"<h3>Win rate: Autorater prefers Candidate Model over Baseline Model {candidate_model_win_rate}% of time.</h3>\"\n",
        "        )\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGXDM4vtv_nG"
      },
      "source": [
        "## Define an evaluation dataset\n",
        "\n",
        "\n",
        "In this tutorial, we use a few examples from the open-source [XSum dataset](https://huggingface.co/datasets/EdinburghNLP/xsum?row=3) for summarization.\n",
        "\n",
        " **Note: For best results, we recommend using at least 100 examples.**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bbScc2Ca0Gp2"
      },
      "outputs": [],
      "source": [
        "instruction = \"Summarize the following article: \"\n",
        "\n",
        "context = [\n",
        "    'The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed. Repair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water. Trains on the west coast mainline face disruption due to damage at the Lamington Viaduct. Many businesses and householders were affected by flooding in Newton Stewart after the River Cree overflowed into the town. First Minister Nicola Sturgeon visited the area to inspect the damage. The waters breached a retaining wall, flooding many commercial properties on Victoria Street - the main shopping thoroughfare. Jeanette Tate, who owns the Cinnamon Cafe which was badly affected, said she could not fault the multi-agency response once the flood hit. However, she said more preventative work could have been carried out to ensure the retaining wall did not fail. \"It is difficult but I do think there is so much publicity for Dumfries and the Nith - and I totally appreciate that - but it is almost like we\\'re neglected or forgotten,\" she said. \"That may not be true but it is perhaps my perspective over the last few days. \"Why were you not ready to help us a bit more when the warning and the alarm alerts had gone out?\" Meanwhile, a flood alert remains in place across the Borders because of the constant rain. Peebles was badly hit by problems, sparking calls to introduce more defences in the area. Scottish Borders Council has put a list on its website of the roads worst affected and drivers have been urged not to ignore closure signs. The Labour Party\\'s deputy Scottish leader Alex Rowley was in Hawick on Monday to see the situation first hand. He said it was important to get the flood protection plan right but backed calls to speed up the process. \"I was quite taken aback by the amount of damage that has been done,\" he said. \"Obviously it is heart-breaking for people who have been forced out of their homes and the impact on businesses.\" He said it was important that \"immediate steps\" were taken to protect the areas most vulnerable and a clear timetable put in place for flood prevention plans. Have you been affected by flooding in Dumfries and Galloway or the Borders? Tell us about your experience of the situation and how it was handled. Email us on selkirk.news@bbc.co.uk or dumfries@bbc.co.uk.',\n",
        "    'A fire alarm went off at the Holiday Inn in Hope Street at about 04:20 BST on Saturday and guests were asked to leave the hotel. As they gathered outside they saw the two buses, parked side-by-side in the car park, engulfed by flames. One of the tour groups is from Germany, the other from China and Taiwan. It was their first night in Northern Ireland. The driver of one of the buses said many of the passengers had left personal belongings on board and these had been destroyed. Both groups have organised replacement coaches and will begin their tour of the north coast later than they had planned. Police have appealed for information about the attack. Insp David Gibson said: \"It appears as though the fire started under one of the buses before spreading to the second. \"While the exact cause is still under investigation, it is thought that the fire was started deliberately.',\n",
        "    'Ferrari appeared in a position to challenge until the final laps, when the Mercedes stretched their legs to go half a second clear of the red cars. Sebastian Vettel will start third ahead of team-mate Kimi Raikkonen. The world champion subsequently escaped punishment for reversing in the pit lane, which could have seen him stripped of pole. But stewards only handed Hamilton a reprimand, after governing body the FIA said \"no clear instruction was given on where he should park\". Belgian Stoffel Vandoorne out-qualified McLaren team-mate Jenson Button on his Formula 1 debut. Vandoorne was 12th and Button 14th, complaining of a handling imbalance on his final lap but admitting the newcomer \"did a good job and I didn\\'t\". Mercedes were wary of Ferrari\\'s pace before qualifying after Vettel and Raikkonen finished one-two in final practice, and their concerns appeared to be well founded as the red cars mixed it with the silver through most of qualifying. After the first runs, Rosberg was ahead, with Vettel and Raikkonen splitting him from Hamilton, who made a mistake at the final corner on his first lap. But Hamilton saved his best for last, fastest in every sector of his final attempt, to beat Rosberg by just 0.077secs after the German had out-paced him throughout practice and in the first qualifying session. Vettel rued a mistake at the final corner on his last lap, but the truth is that with the gap at 0.517secs to Hamilton there was nothing he could have done. The gap suggests Mercedes are favourites for the race, even if Ferrari can be expected to push them. Vettel said: \"Last year we were very strong in the race and I think we are in good shape for tomorrow. We will try to give them a hard time.\" Vandoorne\\'s preparations for his grand prix debut were far from ideal - he only found out he was racing on Thursday when FIA doctors declared Fernando Alonso unfit because of a broken rib sustained in his huge crash at the first race of the season in Australia two weeks ago. The Belgian rookie had to fly overnight from Japan, where he had been testing in the Super Formula car he races there, and arrived in Bahrain only hours before first practice on Friday. He also had a difficult final practice, missing all but the final quarter of the session because of a water leak. Button was quicker in the first qualifying session, but Vandoorne pipped him by 0.064secs when it mattered. The 24-year-old said: \"I knew after yesterday I had quite similar pace to Jenson and I knew if I improved a little bit I could maybe challenge him and even out-qualify him and that is what has happened. \"Jenson is a very good benchmark for me because he is a world champion and he is well known to the team so I am very satisfied with the qualifying.\" Button, who was 0.5secs quicker than Vandoorne in the first session, complained of oversteer on his final run in the second: \"Q1 was what I was expecting. Q2 he did a good job and I didn\\'t. Very, very good job. We knew how quick he was.\" The controversial new elimination qualifying system was retained for this race despite teams voting at the first race in Australia to go back to the 2015 system. FIA president Jean Todt said earlier on Saturday that he \"felt it necessary to give new qualifying one more chance\", adding: \"We live in a world where there is too much over reaction.\" The system worked on the basis of mixing up the grid a little - Force India\\'s Sergio Perez ended up out of position in 18th place after the team miscalculated the timing of his final run, leaving him not enough time to complete it before the elimination clock timed him out. But it will come in for more criticism as a result of lack of track action at the end of each session. There were three minutes at the end of the first session with no cars on the circuit, and the end of the second session was a similar damp squib. Only one car - Nico Hulkenberg\\'s Force India - was out on the track with six minutes to go. The two Williams cars did go out in the final three minutes but were already through to Q3 and so nothing was at stake. The teams are meeting with Todt and F1 commercial boss Bernie Ecclestone on Sunday at noon local time to decide on what to do with qualifying for the rest of the season. Todt said he was \"optimistic\" they would be able to reach unanimous agreement on a change. \"We should listen to the people watching on TV,\" Rosberg said. \"If they are still unhappy, which I am sure they will be, we should change it.\" Red Bull\\'s Daniel Ricciardo was fifth on the grid, ahead of the Williams cars of Valtteri Bottas and Felipe Massa and Force India\\'s Nico Hulkenberg. Ricciardo\\'s team-mate Daniil Kvyat was eliminated during the second session - way below the team\\'s expectation - and the Renault of Brit Jolyon Palmer only managed 19th fastest. German Mercedes protege Pascal Wehrlein managed an excellent 16th in the Manor car. Bahrain GP qualifying results Bahrain GP coverage details',\n",
        "    'Gundogan, 26, told BBC Sport he \"can see the finishing line\" after tearing cruciate knee ligaments in December, but will not rush his return. The German missed the 2014 World Cup following back surgery that kept him out for a year, and sat out Euro 2016 because of a dislocated kneecap. He said: \"It is heavy mentally to accept that.\" Gundogan will not be fit for the start of the Premier League season at Brighton on 12 August but said his recovery time is now being measured in \"weeks\" rather than months. He told BBC Sport: \"It is really hard always to fall and fight your way back. You feel good and feel ready, then you get the next kick. \"The worst part is behind me now. I want to feel ready when I am fully back. I want to feel safe and confident. I don\\'t mind if it is two weeks or six.\" Gundogan made 15 appearances and scored five goals in his debut season for City following his £20m move from Borussia Dortmund. He is eager to get on the field again and was impressed at the club\\'s 4-1 win over Real Madrid in a pre-season game in Los Angeles on Wednesday. Manager Pep Guardiola has made five new signings already this summer and continues to have an interest in Arsenal forward Alexis Sanchez and Monaco\\'s Kylian Mbappe. Gundogan said: \"Optimism for the season is big. It is huge, definitely. \"We felt that last year as well but it was a completely new experience for all of us. We know the Premier League a bit more now and can\\'t wait for the season to start.\" City complete their three-match tour of the United States against Tottenham in Nashville on Saturday. Chelsea manager Antonio Conte said earlier this week he did not feel Tottenham were judged by the same standards as his own side, City and Manchester United. Spurs have had the advantage in their recent meetings with City, winning three and drawing one of their last four Premier League games. And Gundogan thinks they are a major threat. He said: \"Tottenham are a great team. They have the style of football. They have young English players. Our experience last season shows it is really tough to beat them. \"They are really uncomfortable to play against. \"I am pretty sure, even if they will not say it loud, the people who know the Premier League know Tottenham are definitely a competitor for the title.',\n",
        "]\n",
        "\n",
        "reference = [\n",
        "    \"Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank.\",\n",
        "    \"Two tourist buses have been destroyed by fire in a suspected arson attack in Belfast city centre.\",\n",
        "    \"Lewis Hamilton stormed to pole position at the Bahrain Grand Prix ahead of Mercedes team-mate Nico Rosberg.\",\n",
        "    \"Manchester City midfielder Ilkay Gundogan says it has been mentally tough to overcome a third major injury.\",\n",
        "]\n",
        "\n",
        "eval_dataset = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [instruction + item for item in context],\n",
        "        \"reference\": reference,\n",
        "    }\n",
        ")\n",
        "\n",
        "eval_dataset.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4zcgTKrXLhcl"
      },
      "source": [
        "## Define a Candidate Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qDcBkDaT--y5"
      },
      "outputs": [],
      "source": [
        "# Model to be evaluated\n",
        "generation_config = {\"temperature\": 0.5, \"max_output_tokens\": 256, \"top_k\": 1}\n",
        "\n",
        "model = GenerativeModel(\"gemini-1.5-pro\", generation_config=generation_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qnd0bIydkOYG"
      },
      "source": [
        "## Customize metrics with your criteria\n",
        "\n",
        "Gen AI Evaluation Service SDK empowers you to customize metrics to align precisely with your specific evaluation criteria. Whether you need to modify individual components, or the entire metric structure, or define your own\n",
        "metric computation logic, the SDK offers the flexibility to tailor the evaluation process to your unique requirements.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nxy-M_K51V6"
      },
      "source": [
        "### Templated Customization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wYAyxF_K6G0V"
      },
      "source": [
        "#### Create a Pointwise MetricPromptTemplate with pre-defined default fields\n",
        "\n",
        "The required fields for templated customization are the criteria for evaluation, the rating rubric and the input variables.\n",
        "\n",
        "\n",
        "* **Criteria** can be a single criterion or a combination of multiple sub-criteria.\n",
        "\n",
        "* **Rating rubric** contains the values for the score and what each score corresponds to.\n",
        "\n",
        "* **Input variables** are the input columns users need to provide in the evaluation dataset to complete the prompt for the autorater and get a response.\n",
        "\n",
        "\n",
        "Other parts of the template such as \"instruction\" and \"evaluation steps\" have default text that can be updated as well if needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-bgfamkD3r3n"
      },
      "outputs": [],
      "source": [
        "linguistic_acceptability_criteria = {\n",
        "    \"Proper Grammar\": \"The language's grammar rules are correctly followed, including but not limited to sentence structures, verb tenses, subject-verb agreement, proper punctuation, and capitalization.\",\n",
        "    \"Appropriate word choice\": \"Words chosen are appropriate and purposeful given their relative context and positioning in the text. Vocabulary demonstrates prompt understanding.\",\n",
        "    \"Reference Alignment\": \"The response is consistent and aligned with the reference.\",\n",
        "}\n",
        "\n",
        "linguistic_acceptability_pointwise_rubric = {\n",
        "    \"5\": \"Excellent: The writing is grammatically correct, uses appropriate vocabulary and aligns perfectly with the reference.\",\n",
        "    \"4\": \"Good: The writing is generally grammatically correct, uses appropriate vocabulary and aligns well with the reference.\",\n",
        "    \"3\": \"Satisfactory: The writing may have minor grammatical errors or use less-appropriate vocabulary, but it aligns reasonably well with the reference.\",\n",
        "    \"2\": \"Unsatisfactory: The writing has significant grammatical errors, uses inappropriate vocabulary, deviates significantly from the reference.\",\n",
        "    \"1\": \"Poor: The writing is riddled with grammatical errors, uses highly inappropriate vocabulary, is completely unrelated to the reference.\",\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v1W0cxFsSxmU"
      },
      "outputs": [],
      "source": [
        "# The metric prompt template contains default prompts pre-defined for unspecified components.\n",
        "linguistic_acceptability_metric_prompt_template = PointwiseMetricPromptTemplate(\n",
        "    criteria=linguistic_acceptability_criteria,\n",
        "    rating_rubric=linguistic_acceptability_pointwise_rubric,\n",
        "    input_variables=[\"prompt\", \"reference\"],\n",
        ")\n",
        "\n",
        "# Display the assembled prompt template that will be sent to Gen AI Eval Service\n",
        "# along with the input data for model-based evaluation.\n",
        "print(linguistic_acceptability_metric_prompt_template.prompt_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yFD5S9uV6x2t"
      },
      "outputs": [],
      "source": [
        "# Define this custom \"linguistic_acceptability\" model-based metric.\n",
        "linguistic_acceptability = PointwiseMetric(\n",
        "    metric=\"linguistic_acceptability\",\n",
        "    metric_prompt_template=linguistic_acceptability_metric_prompt_template,\n",
        ")\n",
        "\n",
        "linguistic_acceptability_eval_task = EvalTask(\n",
        "    dataset=eval_dataset, metrics=[linguistic_acceptability], experiment=EXPERIMENT_NAME\n",
        ")\n",
        "\n",
        "linguistic_acceptability_result = linguistic_acceptability_eval_task.evaluate(\n",
        "    model=model,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xXXR7nGi7JMz"
      },
      "outputs": [],
      "source": [
        "display_eval_result(linguistic_acceptability_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXqVhFgk7bkA"
      },
      "source": [
        "#### Create a Pairwise MetricPromptTemplate with pre-defined default fields"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pPPfnYGVVGlN"
      },
      "outputs": [],
      "source": [
        "# Define a baseline model for pairwise evaluation\n",
        "baseline_model = GenerativeModel(\n",
        "    \"gemini-1.0-pro\",\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5jOlzuGS7ZLp"
      },
      "outputs": [],
      "source": [
        "linguistic_acceptability_pairwise_rubric = {\n",
        "    \"A\": \"Response A answers the given question as per the criteria better than response B.\",\n",
        "    \"SAME\": \"Response A and B answers the given question equally well as per the criteria.\",\n",
        "    \"B\": \"Response B answers the given question as per the criteria better than response A.\",\n",
        "}\n",
        "\n",
        "pairwise_linguistic_acceptability_prompt_template = PairwiseMetricPromptTemplate(\n",
        "    criteria=linguistic_acceptability_criteria,\n",
        "    rating_rubric=linguistic_acceptability_pairwise_rubric,\n",
        "    input_variables=[\"prompt\", \"reference\"],\n",
        ")\n",
        "\n",
        "pairwise_linguistic_acceptability = PairwiseMetric(\n",
        "    metric=\"pairwise_linguistic_acceptability\",\n",
        "    metric_prompt_template=pairwise_linguistic_acceptability_prompt_template,\n",
        "    baseline_model=baseline_model,\n",
        ")\n",
        "\n",
        "print(pairwise_linguistic_acceptability_prompt_template.prompt_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "86mEq01o7o3w"
      },
      "outputs": [],
      "source": [
        "pairwise_linguistic_acceptability_eval_task = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=[pairwise_linguistic_acceptability],\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")\n",
        "\n",
        "pairwise_linguistic_acceptability_result = (\n",
        "    pairwise_linguistic_acceptability_eval_task.evaluate(\n",
        "        model=model,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tNmMddL7GCJq"
      },
      "outputs": [],
      "source": [
        "display_eval_result(pairwise_linguistic_acceptability_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w9A5Jdu68Lka"
      },
      "source": [
        "### Free-form customization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "koFXPMGQ8Z3U"
      },
      "source": [
        "#### Create a fully-customized PointwiseMetric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Dc7lESwWULn"
      },
      "outputs": [],
      "source": [
        "# Take a look at an example Pointwise metric prompt template\n",
        "print(MetricPromptTemplateExamples.get_prompt_template(\"verbosity\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cIB-IFjD8Pxk"
      },
      "outputs": [],
      "source": [
        "# Make changes to create your own metric prompt template\n",
        "free_form_pointwise_metric_prompt = \"\"\"\n",
        "# Instruction\n",
        "You are an expert evaluator. Your task is to evaluate the quality of the responses generated by AI models.\n",
        "We will provide you with the user input and an AI-generated response.\n",
        "You should first read the user input carefully for analyzing the task, and then evaluate the quality of the responses based on the Criteria provided in the Evaluation section below.\n",
        "You will assign the response a rating following the Rating Rubric and Evaluation Steps. Give step-by-step explanations for your rating, and only choose ratings from the Rating Rubric.\n",
        "\n",
        "\n",
        "# Evaluation\n",
        "## Metric Definition\n",
        "You will be assessing the verbosity of the model's response, which measures its conciseness and ability to provide sufficient detail without being overly wordy or excessively brief.\n",
        "\n",
        "## Criteria\n",
        "Verbosity: The response is appropriately concise, providing sufficient detail without using complex language to thoroughly address the prompt without being overly wordy or excessively brief.\n",
        "Reference Alignment: The response is consistent and aligned with the reference.\n",
        "\n",
        "## Rating Rubric\n",
        "2: (Excessive Verbosity) The response is overly verbose, filled with unnecessary words and repetition, lacks clarity and conciseness, and deviates significantly from the reference.\n",
        "1: (Somewhat Verbose) The response contains some unnecessary wordiness or repetition, but still provides all necessary information, is generally easy to understand, and aligns reasonably well with the reference.\n",
        "0: (Just Right) The response is perfectly concise, providing all necessary information in a clear and succinct manner without any unnecessary wordiness or repetition, and aligns closely with the reference.\n",
        "-1: (Somewhat Brief) The response is slightly brief and could benefit from additional details or explanations to fully address the prompt, but still provides the core information and is generally understandable, though it may deviate slightly from the reference.\n",
        "-2: (Too Short) The response is excessively brief and lacks crucial information or explanations needed to adequately address the prompt, leaving the reader with unanswered questions or a sense of incompleteness, and deviates significantly from the reference.\n",
        "\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Assess conciseness: Is the response free of unnecessary wordiness, repetition, or filler words? Could any sentences or phrases be shortened or simplified without losing meaning?\n",
        "STEP 2: Assess reference alignment: Does the response align with the reference provided?\n",
        "STEP 3: Assess overall balance: Does the response strike the right balance between providing sufficient detail and being concise? Is it aligned with the reference.\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "### Reference\n",
        "{reference}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PO-KKJZw8WuV"
      },
      "outputs": [],
      "source": [
        "free_form_pointwise_text_quality_metric = PointwiseMetric(\n",
        "    metric=\"free_form_pointwise_text_quality\",\n",
        "    metric_prompt_template=free_form_pointwise_metric_prompt,\n",
        ")\n",
        "\n",
        "\n",
        "free_form_pointwise_eval_task = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=[free_form_pointwise_text_quality_metric],\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")\n",
        "\n",
        "free_form_pointwise_result = free_form_pointwise_eval_task.evaluate(model=model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8JH70TVtGFux"
      },
      "outputs": [],
      "source": [
        "display_eval_result(free_form_pointwise_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IEWeD9W8pUr"
      },
      "source": [
        "#### Create a fully-customized PairwiseMetric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kko10mBM8yK0"
      },
      "outputs": [],
      "source": [
        "free_form_pairwise_metric_prompt = \"\"\"\n",
        "# Instruction\n",
        "You are an expert evaluator. Your task is to evaluate the quality of the responses generated by two AI models. We will provide you with the user input and a pair of AI-generated responses (Response A and Response B).\n",
        "You should first read the user input carefully for analyzing the task, and then evaluate the quality of the responses based on the Criteria provided in the Evaluation section below.\n",
        "You will first judge responses individually, following the Rating Rubric and Evaluation Steps.\n",
        "Then you will give step-by-step explanations for your judgement, compare results to declare the winner based on the Rating Rubric and Evaluation Steps.\n",
        "\n",
        "# Evaluation\n",
        "## Metric Definition\n",
        "You will be assessing the verbosity of each model's response, which measures its conciseness and ability to provide sufficient detail without being overly wordy or excessively brief.\n",
        "\n",
        "## Criteria\n",
        "Verbosity: The response is appropriately concise, providing sufficient detail without using complex language to thoroughly address the prompt without being overly wordy or excessively brief.\n",
        "Reference Alignment: The response is consistent and aligned with the reference.\n",
        "\n",
        "## Rating Rubric\n",
        "\"A\": Response A is more appropriately concise than Response B. Response A is better aligned with the reference.\n",
        "\"SAME\": Response A and B are equally concise. They both are equally aligned with the reference.\n",
        "\"B\": Response B is more appropriately concise than Response A. Response B is better aligned with the reference.\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Analyze Response A based on the criteria regarding completeness, conciseness, and alignment with reference.\n",
        "STEP 2: Analyze Response B based on the criteria regarding completeness, conciseness, and alignment with reference.\n",
        "STEP 3: Compare the overall performance of Response A and Response B based on your analyses and assessment.\n",
        "STEP 4: Output your preference of \"A\", \"SAME\" or \"B\" to the pairwise_choice field according to the Rating Rubric.\n",
        "STEP 5: Output your assessment reasoning in the explanation field, justifying your choice by highlighting the specific strengths and weaknesses of each response in terms of verbosity.\n",
        "\n",
        "# User Inputs and AI-generated Responses\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "### Reference\n",
        "{reference}\n",
        "\n",
        "# AI-generated Responses\n",
        "### Response A\n",
        "{baseline_model_response}\n",
        "\n",
        "### Response B\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f3bg8_WY812p"
      },
      "outputs": [],
      "source": [
        "free_form_pairwise_metric = PairwiseMetric(\n",
        "    metric=\"free_form_pairwise_verbosity_reference_alignment\",\n",
        "    metric_prompt_template=free_form_pairwise_metric_prompt,\n",
        "    baseline_model=baseline_model,\n",
        ")\n",
        "free_form_pairwise_eval_task = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=[free_form_pairwise_metric],\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")\n",
        "\n",
        "free_form_pairwise_result = free_form_pairwise_eval_task.evaluate(\n",
        "    model=model,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NB2WxObKGJ4-"
      },
      "outputs": [],
      "source": [
        "display_eval_result(free_form_pairwise_result)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "customize_model_based_metrics.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}