testing-examples/comparing-runs/comparing-qa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1a33c67",
   "metadata": {},
   "source": [
    "# Comparing Q&A System Outputs\n",
    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/testing-examples/comparing-runs/comparing-qa.ipynb)\n",
    "\n",
    "The most common way to compare two models is to benchmark them both on a dataset and compare the aggregate metrics.\n",
    "\n",
    "This approach is useful but it may filter out helpful information about the quality of the two system variants. In this case, it can be helpful to directly perform pairwise comparisons on the responses and take the resulting preference scores into consideration.\n",
    "\n",
    "In this tutorial, we will share one way to do this in code. We will use a retrieval Q&A system over LangSmith's docs as a motivating example.\n",
    "\n",
    "The main steps are:\n",
    "\n",
    "1. Setup\n",
    "   - Create a dataset of questions and answers.\n",
    "   - Define different versions of your chains to evaluate.\n",
    "   - Evaluate chains directly on a dataset using regular metrics (e.g. correctness).\n",
    "4. Evaluate the pairwise preferences over that dataset\n",
    "\n",
    "In this case, we will test the impact of chunk sizes on our result quality. Let's begin!\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "This tutorial uses OpenAI for the model, ChromaDB to store documents, and LangChain to compose the chain. To make sure the tracing and evals are set up for [LangSmith](https://smith.langchain.com), please configure your API Key appropriately.\n",
    "\n",
    "We will also use pandas to render the results in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c788783",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:21.905597Z",
     "start_time": "2023-09-20T04:50:21.886104Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\" # Update with your API URL if using a hosted instance of Langsmith.\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = \"YOUR API KEY\" # Update with your API key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe780fbd",
   "metadata": {},
   "source": [
    "Install the required packages. `lxml` and `html2text` are used by the document loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f9e7425",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:23.269252Z",
     "start_time": "2023-09-20T04:50:23.265495Z"
    }
   },
   "outputs": [],
   "source": [
    "# %pip install -U \"langchain[openai]\" --quiet\n",
    "# %pip install chromadb --quiet\n",
    "# %pip install lxml --quiet\n",
    "# %pip install html2text --quiet\n",
    "# %pip install pandas --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afac8079",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:25.057984Z",
     "start_time": "2023-09-20T04:50:25.050240Z"
    }
   },
   "outputs": [],
   "source": [
    "# %env OPENAI_API_KEY=<YOUR-API-KEY>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff80ab6e",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "\n",
    "#### a. Create a dataset\n",
    "\n",
    "No evaluation process is complete without a development dataset. We've hard-coded a few examples below to demonstrate the process. In general, you'll want a lot more (>100) pairs for statistically significant results. Drawing from actual user queries can be helpful to ensure better representation of the domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "55e98a80-bf37-457e-b31d-952292e76c51",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:27.908227Z",
     "start_time": "2023-09-20T04:50:27.903470Z"
    }
   },
   "outputs": [],
   "source": [
    "examples = [\n",
    "    (\"What is LangChain?\", \"LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.\"),\n",
    "    (\"How might I query for all runs in a project?\", \"client.list_runs(project_name='my-project-name'), or in TypeScript, client.ListRuns({projectName: 'my-project-anme'})\"),\n",
    "    (\"What's a langsmith dataset?\", \"A LangSmith dataset is a collection of examples. Each example contains inputs and optional expected outputs or references for that data point.\"),\n",
    "    (\"How do I use a traceable decorator?\", \"\"\"The traceable decorator is available in the langsmith python SDK. To use, configure your environment with your API key,\\\n",
    "import the required function, decorate your function, and then call the function. Below is an example:\n",
    "```python\n",
    "from langsmith.run_helpers import traceable\n",
    "@traceable(run_type=\"chain\") # or \"llm\", etc.\n",
    "def my_function(input_param):\n",
    "    # Function logic goes here\n",
    "    return output\n",
    "result = my_function(input_param)\n",
    "```\"\"\"),\n",
    "    (\"Can I trace my Llama V2 llm?\", \"So long as you are using one of LangChain's LLM implementations, all your calls can be traced\"),\n",
    "    (\"Why do I have to set environment variables?\", \"Environment variables can tell your LangChain application to perform tracing and contain the information necessary to authenticate to LangSmith.\"\n",
    "     \" While there are other ways to connect, environment variables tend to be the simplest way to configure your application.\"),\n",
    "    (\"How do I move my project between organizations?\", \"LangSmith doesn't directly support moving projects between organizations.\")\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5edb7824",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:28.911950Z",
     "start_time": "2023-09-20T04:50:28.903935Z"
    }
   },
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cbcd3690",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:31.532293Z",
     "start_time": "2023-09-20T04:50:29.899253Z"
    }
   },
   "outputs": [],
   "source": [
    "import uuid \n",
    "\n",
    "dataset_name = f\"Retrieval QA Questions {str(uuid.uuid4())}\"\n",
    "dataset = client.create_dataset(dataset_name=dataset_name)\n",
    "for q, a in examples:\n",
    "    client.create_example(inputs={\"question\": q}, outputs={\"answer\": a}, dataset_id=dataset.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2976437f",
   "metadata": {},
   "source": [
    "#### b. Define RAG Q&A system\n",
    "\n",
    "Our Q&A system uses a simple retriever and LLM response generator. To break that down further, the chain will be composed of:\n",
    "\n",
    "1. A [VectorStoreRetriever](https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.base.VectorStoreRetriever.html#langchain.vectorstores.base.VectorStoreRetriever) to retrieve documents. This uses:\n",
    "   - An embedding model to vectorize documents and user queries for retrieval. In this case, the [OpenAIEmbeddings](https://api.python.langchain.com/en/latest/embeddings/langchain.embeddings.openai.OpenAIEmbeddings.html) model.\n",
    "   - A vectorstore, in this case we will use [Chroma](https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.chroma.Chroma.html#langchain.vectorstores.chroma.Chroma).\n",
    "2. A response generator. This uses:\n",
    "   - A [ChatPromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain.prompts.chat.ChatPromptTemplate.html#langchain.prompts.chat.ChatPromptTemplate) to combine the query and documents. \n",
    "   - An LLM, in this case, the 16k token context window version of `gpt-3.5-turbo` via [ChatOpenAI](https://api.python.langchain.com/en/latest/chat_models/langchain.chat_models.openai.ChatOpenAI.html#langchain.chat_models.openai.ChatOpenAI).\n",
    "\n",
    "We will combine them using LangChain's [expression syntax](https://python.langchain.com/docs/guides/expression_language/cookbook).\n",
    "\n",
    "First, load the documents to populate the vectorstore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "95fab721",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:47.262606Z",
     "start_time": "2023-09-20T04:50:36.292760Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/wfh/.pyenv/versions/3.11.2/lib/python3.11/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features=\"xml\"` into the BeautifulSoup constructor.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import RecursiveUrlLoader\n",
    "from langchain.document_transformers import Html2TextTransformer\n",
    "from langchain.text_splitter import TokenTextSplitter\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.vectorstores import Chroma\n",
    "\n",
    "api_loader = RecursiveUrlLoader(\"https://docs.smith.langchain.com\")\n",
    "doc_transformer = Html2TextTransformer()\n",
    "raw_documents = api_loader.load()\n",
    "transformed = doc_transformer.transform_documents(raw_documents)\n",
    "\n",
    "def create_retriever(transformed_documents, text_splitter):\n",
    "    documents = text_splitter.split_documents(transformed_documents)\n",
    "    embeddings = OpenAIEmbeddings()\n",
    "    vectorstore = Chroma.from_documents(documents, embeddings)\n",
    "    return vectorstore.as_retriever(search_kwargs={\"k\": 4})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e35f884-bb93-427d-a0ad-3858c449a1ea",
   "metadata": {},
   "source": [
    "Next up, we'll define the chain. Since we are going to vary the retriever parameters, our constructor will\n",
    "take the retriever as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e2f266e9-e4de-42ad-b41e-99ace4dc5131",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:47.264432Z",
     "start_time": "2023-09-20T04:50:47.263828Z"
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from operator import itemgetter\n",
    "\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.schema.output_parser import StrOutputParser\n",
    "\n",
    "\n",
    "def create_chain(retriever):\n",
    "    prompt = ChatPromptTemplate.from_messages(\n",
    "            [\n",
    "                (\"system\", \"You are a helpful documentation Q&A assistant, trained to answer\"\n",
    "                \" questions from LangSmith's documentation.\"\n",
    "                \" LangChain is a framework for building applications using large language models.\"\n",
    "                \"\\nThe current time is {time}.\\n\\nRelevant documents will be retrieved in the following messages.\"),\n",
    "                (\"system\", \"{context}\"),\n",
    "                (\"human\",\"{question}\")\n",
    "            ]\n",
    "        ).partial(time=str(datetime.now()))\n",
    "\n",
    "    model = ChatOpenAI(model=\"gpt-3.5-turbo-16k\", temperature=0)\n",
    "    response_generator = (\n",
    "        prompt \n",
    "        | model \n",
    "        | StrOutputParser()\n",
    "    )\n",
    "    chain = (\n",
    "        # The runnable map here routes the original inputs to a context and a question dictionary to pass to the response generator\n",
    "        {\n",
    "            \"context\": itemgetter(\"question\") | retriever | (lambda docs: \"\\n\".join([doc.page_content for doc in docs])),\n",
    "            \"question\": itemgetter(\"question\")\n",
    "        }\n",
    "        | response_generator\n",
    "    )\n",
    "    return chain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d02a25-2863-4c5c-81e8-00887247516d",
   "metadata": {},
   "source": [
    "With the documents prepared, and the chain constructor ready, it's time to create and evaluate our chains.\n",
    "We will vary the split size and overlap to evaluate its impact on the response quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1028d963-d84b-4759-a5ca-087b27065485",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-09-20T04:50:47.265315Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "text_splitter = TokenTextSplitter(\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    "    chunk_size=2000,\n",
    "    chunk_overlap=200,\n",
    ")\n",
    "retriever = create_retriever(transformed, text_splitter)\n",
    "\n",
    "chain_1 = create_chain(retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5765c938-c5f6-4ee4-be6b-90b7b341b683",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-09-20T04:50:47.266475Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We will shrink both the chunk size and overlap\n",
    "text_splitter_2 = TokenTextSplitter(\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    "    chunk_size=500,\n",
    "    chunk_overlap=50,\n",
    ")\n",
    "retriever_2 = create_retriever(transformed, text_splitter_2)\n",
    "\n",
    "chain_2 = create_chain(retriever_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3dddd61",
   "metadata": {},
   "source": [
    "#### c. Evaluate the chains\n",
    "\n",
    "At this point, we are still going through the regular development -> evaluation process. We have two candidates and will evaluate them with a correctness evaluator from LangChain. By running `run_on_dataset`, we will generate predicted answers to each question in the dataset and log feedback from the evaluator for that data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "62de3aea-638e-4add-998a-22eb21f6feaf",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-09-20T04:50:47.267369Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.smith import RunEvalConfig\n",
    "\n",
    "eval_config = RunEvalConfig(\n",
    "    # We will use the chain-of-thought Q&A correctness evaluator\n",
    "    evaluators=[\"cot_qa\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1ae51a83-6ae1-4fa6-a9d8-4e27bf03047f",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-09-20T04:50:47.268056Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'test-new-goat-73' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/projects/p/8097fed2-ca97-42b6-b71a-a24fe5e7c9d6\n",
      "[------------------------------------------------->] 7/7"
     ]
    }
   ],
   "source": [
    "results = client.run_on_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    llm_or_chain_factory=chain_1,\n",
    "    evaluation=eval_config\n",
    ")\n",
    "project_name = results[\"project_name\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ec375e76-7261-433b-9f02-f4750acf6f93",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-09-20T04:50:47.268715Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'test-large-stone-98' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/projects/p/4f3b1676-741b-4fb1-bfc1-46ca630ac160\n",
      "[------------------------------------------------->] 7/7"
     ]
    }
   ],
   "source": [
    "results_2 = client.run_on_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    llm_or_chain_factory=chain_2,\n",
    "    evaluation=eval_config\n",
    ")\n",
    "project_name_2 = results_2[\"project_name\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a6d2019-d076-4f18-832e-634dbd31091c",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now you should have two test run projects over the same dataset. If you click on one, it should look something like the following:\n",
    "    \n",
    "![Original Feedback](img/original_eval.png)\n",
    "\n",
    "You can look at the aggregate results here and for the other project to compare them. You could also view them in a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f50f74cc-fc86-4c60-8431-70695eba7aac",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:52.710833Z",
     "start_time": "2023-09-20T04:50:51.723217Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "runs_1 = list(client.list_runs(project_name=project_name, execution_order=1))\n",
    "runs_2 = list(client.list_runs(project_name=project_name_2, execution_order=1))\n",
    "\n",
    "def get_project_df(runs):\n",
    "    return pd.DataFrame([{**run.outputs, **{k: v.get('avg') for k, v in run.feedback_stats.items()}} for run in runs], index = [run.reference_example_id for run in runs])\n",
    "\n",
    "runs_1_df = get_project_df(runs_1)\n",
    "runs_2_df = get_project_df(runs_2)\n",
    "joined_df = runs_1_df.join(runs_2_df, lsuffix='_1', rsuffix='_2')\n",
    "columns_1 = [col for col in joined_df.columns if col.endswith('_1')]\n",
    "columns_2 = [col for col in joined_df.columns if col.endswith('_2')]\n",
    "new_columns_order = [col for pair in zip(columns_1, columns_2) for col in pair]\n",
    "joined_df = joined_df[new_columns_order]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "45e3d81d-b16b-4ba2-bf4b-1118973e30d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:50:53.133194Z",
     "start_time": "2023-09-20T04:50:53.122750Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>output_1</th>\n",
       "      <th>output_2</th>\n",
       "      <th>COT Contextual Accuracy_1</th>\n",
       "      <th>COT Contextual Accuracy_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>04a95258-4999-4abd-b1c3-0c5214130579</th>\n",
       "      <td>LangChain is an open-source framework for buil...</td>\n",
       "      <td>LangChain is an open-source framework for buil...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198d7039-72bc-4376-907a-87066d85275b</th>\n",
       "      <td>To query for all runs in a project, you can us...</td>\n",
       "      <td>To query for all runs in a project using LangS...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ea7b3f78-020e-410b-8e7b-bfdfa681a386</th>\n",
       "      <td>A LangSmith dataset is a collection of input-o...</td>\n",
       "      <td>A LangSmith dataset refers to a collection of ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3cdd7b83-9a60-4c1b-b30b-dfa3fae69740</th>\n",
       "      <td>To use a traceable decorator in LangSmith, you...</td>\n",
       "      <td>To use the `traceable` decorator in LangSmith,...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5ec65a7d-70de-4494-b780-139550e301e5</th>\n",
       "      <td>Yes, you can trace your Llama V2 LLM using Lan...</td>\n",
       "      <td>Yes, you can trace your Llama V2 LLM using Lan...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7b74f055-3a2e-4b6f-a3d1-45eecfddcc63</th>\n",
       "      <td>To move a project between organizations in Lan...</td>\n",
       "      <td>To move a project between organizations in Lan...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f23253e5-c537-43f0-8059-38153090a884</th>\n",
       "      <td>At LangChain, setting environment variables is...</td>\n",
       "      <td>Setting environment variables is necessary in ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                               output_1  \\\n",
       "04a95258-4999-4abd-b1c3-0c5214130579  LangChain is an open-source framework for buil...   \n",
       "198d7039-72bc-4376-907a-87066d85275b  To query for all runs in a project, you can us...   \n",
       "ea7b3f78-020e-410b-8e7b-bfdfa681a386  A LangSmith dataset is a collection of input-o...   \n",
       "3cdd7b83-9a60-4c1b-b30b-dfa3fae69740  To use a traceable decorator in LangSmith, you...   \n",
       "5ec65a7d-70de-4494-b780-139550e301e5  Yes, you can trace your Llama V2 LLM using Lan...   \n",
       "7b74f055-3a2e-4b6f-a3d1-45eecfddcc63  To move a project between organizations in Lan...   \n",
       "f23253e5-c537-43f0-8059-38153090a884  At LangChain, setting environment variables is...   \n",
       "\n",
       "                                                                               output_2  \\\n",
       "04a95258-4999-4abd-b1c3-0c5214130579  LangChain is an open-source framework for buil...   \n",
       "198d7039-72bc-4376-907a-87066d85275b  To query for all runs in a project using LangS...   \n",
       "ea7b3f78-020e-410b-8e7b-bfdfa681a386  A LangSmith dataset refers to a collection of ...   \n",
       "3cdd7b83-9a60-4c1b-b30b-dfa3fae69740  To use the `traceable` decorator in LangSmith,...   \n",
       "5ec65a7d-70de-4494-b780-139550e301e5  Yes, you can trace your Llama V2 LLM using Lan...   \n",
       "7b74f055-3a2e-4b6f-a3d1-45eecfddcc63  To move a project between organizations in Lan...   \n",
       "f23253e5-c537-43f0-8059-38153090a884  Setting environment variables is necessary in ...   \n",
       "\n",
       "                                      COT Contextual Accuracy_1  \\\n",
       "04a95258-4999-4abd-b1c3-0c5214130579                        1.0   \n",
       "198d7039-72bc-4376-907a-87066d85275b                        0.0   \n",
       "ea7b3f78-020e-410b-8e7b-bfdfa681a386                        1.0   \n",
       "3cdd7b83-9a60-4c1b-b30b-dfa3fae69740                        0.0   \n",
       "5ec65a7d-70de-4494-b780-139550e301e5                        1.0   \n",
       "7b74f055-3a2e-4b6f-a3d1-45eecfddcc63                        0.0   \n",
       "f23253e5-c537-43f0-8059-38153090a884                        1.0   \n",
       "\n",
       "                                      COT Contextual Accuracy_2  \n",
       "04a95258-4999-4abd-b1c3-0c5214130579                        1.0  \n",
       "198d7039-72bc-4376-907a-87066d85275b                        0.0  \n",
       "ea7b3f78-020e-410b-8e7b-bfdfa681a386                        1.0  \n",
       "3cdd7b83-9a60-4c1b-b30b-dfa3fae69740                        0.0  \n",
       "5ec65a7d-70de-4494-b780-139550e301e5                        1.0  \n",
       "7b74f055-3a2e-4b6f-a3d1-45eecfddcc63                        0.0  \n",
       "f23253e5-c537-43f0-8059-38153090a884                        1.0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b16bf4d-383b-4f44-b5bd-b56c42b91919",
   "metadata": {
    "tags": []
   },
   "source": [
    "It looks like the benchmark performance is similar, so let's move on to the pairwise comparison.\n",
    "\n",
    "### Compare in LangSmith\n",
    "\n",
    "Navigate to the \"Retrieval QA Questions\" dataset in LangSmith, select the two tests you just completed, then click \"Compare.\"\n",
    "\n",
    "![Compare Tests](./img/compare_tests_select.png)\n",
    "\n",
    "From this view, you can manually review and compare the results. You can even filter by initial scores to select outputs where the grades differ.\n",
    "\n",
    "![Filter Results](./img/filter_results.png)\n",
    "\n",
    "Manual comparison is incredibly useful, but it's hard to do this on every test run. Below, we will show how to use an LLM to directly compare the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17cb19f4-1855-4e9c-a447-491625c66646",
   "metadata": {},
   "source": [
    "## 2. Pairwise Evaluation\n",
    "\n",
    "Suppose both approaches return similar scores when evaluated in isolation.\n",
    "\n",
    "We can run a pairwise evaluator to see how try to predict preferred outputs. We will first define a couple helper functions to run the evaluator\n",
    "on each prediction pair. Let's break this function down:\n",
    "\n",
    "- The function accepts a dataset example and loads each model's predictions on that data point.\n",
    "- It then randomizes the order of the predictions and calls the evaluator. This is done to aveage out the impact of any ordering bias in the evaluator LLM.\n",
    "- Once the evaluation result is returned, we check it to make sure it is valid and then log feedback for both models.\n",
    "\n",
    "Once this is complete, the values are all returned so we can display them in a table in the notebook below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bf7c70f3-9c8f-436d-85ca-5a398c3b1369",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import logging\n",
    "\n",
    "def _get_run_and_prediction(example_id, project_name):\n",
    "    run = next(client.list_runs(reference_example_id=example_id, project_name=project_name, execution_order=1))\n",
    "    prediction = next(iter(run.outputs.values()))\n",
    "    return run, prediction\n",
    "\n",
    "def _log_feedback(run_ids):\n",
    "    for score, run_id in enumerate(run_ids):\n",
    "        client.create_feedback(run_id, key=\"preference\", score=score)\n",
    "\n",
    "def predict_preference(example, project_a, project_b, eval_chain):\n",
    "    example_id = example.id\n",
    "    print(example)\n",
    "    run_a, pred_a = _get_run_and_prediction(example_id, project_a)\n",
    "    run_b, pred_b = _get_run_and_prediction(example_id, project_b)\n",
    "    input_, answer = example.inputs['question'], example.outputs['answer']\n",
    "    result = {\"input\": input_, \"answer\": answer, \"A\": pred_a, \"B\": pred_b}\n",
    "\n",
    "    # Flip a coin to average out persistent positional bias\n",
    "    if random.random() < 0.5:\n",
    "        result['A'], result['B'] = result['B'], result['A']\n",
    "        run_a, run_b = run_b, run_a\n",
    "    try:\n",
    "        eval_res = eval_chain.evaluate_string_pairs(\n",
    "            prediction=result['A'],\n",
    "            prediction_b=result['B'],\n",
    "            input=input_, \n",
    "            reference=answer\n",
    "        )\n",
    "    except Exception as e:\n",
    "        logging.warning(e)\n",
    "        return result\n",
    "\n",
    "    if eval_res[\"value\"] is None:\n",
    "        return result\n",
    "\n",
    "    preferred_run = (run_a.id, \"A\") if eval_res[\"value\"] == \"A\" else (run_b.id, \"B\")\n",
    "    runner_up_run = (run_b.id, \"B\") if eval_res[\"value\"] == \"A\" else (run_a.id, \"A\")\n",
    "    _log_feedback((runner_up_run[0], preferred_run[0]))\n",
    "    result[\"Preferred\"] = preferred_run[1]\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e6a51df-616f-4f3e-9106-a5f7669b9090",
   "metadata": {},
   "source": [
    "For this example, we will use the `labeled_pairwise_string` evaluator from LangChain off-the-shelf. By default, instructs the evaluation llm to choose the preference based on helpfulness, relevance, correctness, and depth of thought. In your case, you will likely want to customize the criteria used!\n",
    "\n",
    "For more information on how to configure it, check out the [Labeled Pairwise String Evaluator](https://python.langchain.com/docs/guides/evaluation/comparison/labeled_pairwise_string) documentation and inspect the resulting traces when running this notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c2e1f12e-ed28-46ce-b09b-37088c475bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.evaluation import load_evaluator\n",
    "\n",
    "pairwise_evaluator = load_evaluator(\"labeled_pairwise_string\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f65e8572-18a3-46b3-936a-38e566b77922",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import functools\n",
    "from langchain.schema.runnable import RunnableLambda\n",
    "\n",
    "\n",
    "eval_func = functools.partial(\n",
    "    predict_preference,\n",
    "    project_a=project_name,\n",
    "    project_b=project_name_2,\n",
    "    eval_chain=pairwise_evaluator,\n",
    ")\n",
    "\n",
    "\n",
    "# We will wrap in a lambda to take advantage of its default `batch` convenience method\n",
    "runnable = RunnableLambda(eval_func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "16969ec1-d27c-4f8e-9624-8c3f25c51f8d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-20T04:45:43.550214Z",
     "start_time": "2023-09-20T04:45:43.459644Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'How do I move my project between organizations?'} outputs={'answer': \"LangSmith doesn't directly support moving projects between organizations.\"} id=UUID('7b74f055-3a2e-4b6f-a3d1-45eecfddcc63') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 588671) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 645828) runs=[]dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'How do I use a traceable decorator?'} outputs={'answer': 'The traceable decorator is available in the langsmith python SDK. To use, configure your environment with your API key,import the required function, decorate your function, and then call the function. Below is an example:\\n```python\\nfrom langsmith.run_helpers import traceable\\n@traceable(run_type=\"chain\") # or \"llm\", etc.\\ndef my_function(input_param):\\n    # Function logic goes here\\n    return output\\nresult = my_function(input_param)\\n```'} id=UUID('3cdd7b83-9a60-4c1b-b30b-dfa3fae69740') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 266140) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 325103) runs=[]\n",
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'What is LangChain?'} outputs={'answer': 'LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.'} id=UUID('04a95258-4999-4abd-b1c3-0c5214130579') created_at=datetime.datetime(2023, 10, 23, 6, 12, 26, 903170) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 26, 963093) runs=[]\n",
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'Can I trace my Llama V2 llm?'} outputs={'answer': \"So long as you are using one of LangChain's LLM implementations, all your calls can be traced\"} id=UUID('5ec65a7d-70de-4494-b780-139550e301e5') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 363825) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 435359) runs=[]\n",
      "\n",
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'Why do I have to set environment variables?'} outputs={'answer': 'Environment variables can tell your LangChain application to perform tracing and contain the information necessary to authenticate to LangSmith. While there are other ways to connect, environment variables tend to be the simplest way to configure your application.'} id=UUID('f23253e5-c537-43f0-8059-38153090a884') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 485675) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 547965) runs=[]\n",
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': \"What's a langsmith dataset?\"} outputs={'answer': 'A LangSmith dataset is a collection of examples. Each example contains inputs and optional expected outputs or references for that data point.'} id=UUID('ea7b3f78-020e-410b-8e7b-bfdfa681a386') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 120041) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 213155) runs=[]\n",
      "dataset_id=UUID('29addcf7-2be5-4320-bae7-10f9635d29e3') inputs={'question': 'How might I query for all runs in a project?'} outputs={'answer': \"client.list_runs(project_name='my-project-name'), or in TypeScript, client.ListRuns({projectName: 'my-project-anme'})\"} id=UUID('198d7039-72bc-4376-907a-87066d85275b') created_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 26145) modified_at=datetime.datetime(2023, 10, 23, 6, 12, 27, 80568) runs=[]\n"
     ]
    }
   ],
   "source": [
    "examples = list(client.list_examples(dataset_id=dataset.id))\n",
    "values = runnable.batch(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11be2783-6c15-4fb9-8fdc-11eaa91197fa",
   "metadata": {},
   "source": [
    "By running the function above, the \"preference\" feedback was automatically logged to the test projects you created in step 3. Below is a view of the same test run as before with the preference scores added. This model seems to be less preferred than the other! \n",
    "\n",
    "![Preference Tags](img/with_preferences.png)\n",
    "\n",
    "The `predict_preference` function we wrote above is set up to not log feedback in the case of a tie, meaning some of the examples do not have a corresponding preference score. You can adjust this behavior as you see fit. \n",
    "\n",
    "You can also view the feedback results for the other test run in the app to see how well the evaluator's results match your expectations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "46c7d7f7-9e26-4439-8a2c-93f6d89741ef",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>answer</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>Preferred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How do I move my project between organizations?</td>\n",
       "      <td>LangSmith doesn't directly support moving proj...</td>\n",
       "      <td>To move a project between organizations in Lan...</td>\n",
       "      <td>To move a project between organizations in Lan...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Why do I have to set environment variables?</td>\n",
       "      <td>Environment variables can tell your LangChain ...</td>\n",
       "      <td>Setting environment variables is necessary in ...</td>\n",
       "      <td>At LangChain, setting environment variables is...</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Can I trace my Llama V2 llm?</td>\n",
       "      <td>So long as you are using one of LangChain's LL...</td>\n",
       "      <td>Yes, you can trace your Llama V2 LLM using Lan...</td>\n",
       "      <td>Yes, you can trace your Llama V2 LLM using Lan...</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How do I use a traceable decorator?</td>\n",
       "      <td>The traceable decorator is available in the la...</td>\n",
       "      <td>To use the `traceable` decorator in LangSmith,...</td>\n",
       "      <td>To use a traceable decorator in LangSmith, you...</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What's a langsmith dataset?</td>\n",
       "      <td>A LangSmith dataset is a collection of example...</td>\n",
       "      <td>A LangSmith dataset is a collection of input-o...</td>\n",
       "      <td>A LangSmith dataset refers to a collection of ...</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How might I query for all runs in a project?</td>\n",
       "      <td>client.list_runs(project_name='my-project-name...</td>\n",
       "      <td>To query for all runs in a project, you can us...</td>\n",
       "      <td>To query for all runs in a project using LangS...</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What is LangChain?</td>\n",
       "      <td>LangChain is an open-source framework for buil...</td>\n",
       "      <td>LangChain is an open-source framework for buil...</td>\n",
       "      <td>LangChain is an open-source framework for buil...</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             input  \\\n",
       "0  How do I move my project between organizations?   \n",
       "1      Why do I have to set environment variables?   \n",
       "2                     Can I trace my Llama V2 llm?   \n",
       "3              How do I use a traceable decorator?   \n",
       "4                      What's a langsmith dataset?   \n",
       "5     How might I query for all runs in a project?   \n",
       "6                               What is LangChain?   \n",
       "\n",
       "                                              answer  \\\n",
       "0  LangSmith doesn't directly support moving proj...   \n",
       "1  Environment variables can tell your LangChain ...   \n",
       "2  So long as you are using one of LangChain's LL...   \n",
       "3  The traceable decorator is available in the la...   \n",
       "4  A LangSmith dataset is a collection of example...   \n",
       "5  client.list_runs(project_name='my-project-name...   \n",
       "6  LangChain is an open-source framework for buil...   \n",
       "\n",
       "                                                   A  \\\n",
       "0  To move a project between organizations in Lan...   \n",
       "1  Setting environment variables is necessary in ...   \n",
       "2  Yes, you can trace your Llama V2 LLM using Lan...   \n",
       "3  To use the `traceable` decorator in LangSmith,...   \n",
       "4  A LangSmith dataset is a collection of input-o...   \n",
       "5  To query for all runs in a project, you can us...   \n",
       "6  LangChain is an open-source framework for buil...   \n",
       "\n",
       "                                                   B Preferred  \n",
       "0  To move a project between organizations in Lan...       NaN  \n",
       "1  At LangChain, setting environment variables is...         A  \n",
       "2  Yes, you can trace your Llama V2 LLM using Lan...         B  \n",
       "3  To use a traceable decorator in LangSmith, you...         A  \n",
       "4  A LangSmith dataset refers to a collection of ...         A  \n",
       "5  To query for all runs in a project using LangS...         A  \n",
       "6  LangChain is an open-source framework for buil...         A  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd \n",
    "df = pd.DataFrame(values)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2ec77a-67b3-48e8-9cde-c9022641f245",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this walkthrough, you compared two versions of a RAG Q&A chain by predicting preference scores for each pair of predictions.\n",
    "This approach is one way to automatically compare two versions of a chain that can give additional context beyond regular benchmarking.\n",
    "\n",
    "There are many related ways to evaluate preferences! Here, we used binary choices to compare the two models and only evaluated once, but you may get better results by trying one of the following approaches:\n",
    "\n",
    "- Evaluate multiple times in each position and returning a win rate\n",
    "- Ensemble evaluators\n",
    "- Instruct the model to output continuous scores\n",
    "- Instruct the model to use a different prompt strategy than chain of thought\n",
    "\n",
    "For more information on measuring the reliability of this and other approaches, you can check out the [evaluations examples](https://python.langchain.com/docs/guides/evaluation/examples) in the LangChain repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d73b5ac9-1fd1-4eaa-bc6d-53263cd575bc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}