feedback-examples/algorithmic-feedback/algorithmic_feedback.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bddb3342-ae12-4d6b-98d1-21cb0674c1d0",
   "metadata": {},
   "source": [
    "# Creating an Automated Feedback Pipeline with LangSmith\n",
    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/algorithmic-feedback/algorithmic_feedback.ipynb)\n",
    "\n",
    "Manually analyzing text processed by your language model is useful, but not scalable. Automated metrics offer a solution. By adding these metrics to your LangSmith projects, you can track advanced metrics on your LLM's performance and user inputs directly from the dashboard.\n",
    "\n",
    "![model-based feedback monitoring charts](./img/feedback_charts.png)\n",
    "\n",
    "If the metrics reveal issues, you can isolate problematic runs for debugging or fine-tuning. This tutorial shows you how to set up an automated feedback pipeline for your language models.\n",
    "\n",
    "## Steps:\n",
    "\n",
    "1. **Filter Runs**: First, identify the runs you want to evaluate. For details, refer to the [Run Filtering Documentation](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local).\n",
    "  \n",
    "2. **Define Feedback Logic**: Create a chain or function to calculate the feedback metrics.\n",
    "  \n",
    "3. **Send Feedback to LangSmith**:\n",
    "    - Use the `client.create_feedback` method to send metrics.\n",
    "    - Alternatively, use `client.evaluate_run`, which both evaluates and logs metrics for you.\n",
    "  \n",
    "We'll be using LangSmith and the hub APIs, so make sure you have the necessary API keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7f496bc2-d9c6-46ea-be19-98ff2dd2800c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\" # Update with your API URL if using a hosted instance of Langsmith.\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = \"YOUR API KEY\" # Update with your API key\n",
    "os.environ[\"LANGCHAIN_HUB_API_URL\"] = \"https://api.hub.langchain.com\" # Update with your API URL if using a hosted instance of Langsmith.\n",
    "os.environ[\"LANGCHAIN_HUB_API_KEY\"] = \"YOUR API KEY\" # Update with your Hub API key\n",
    "project_name = \"YOUR PROJECT NAME\" # Change to your project name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b6a08dc-479e-4772-913d-5e4a9c057ff9",
   "metadata": {},
   "source": [
    "We will create some example runs in this project to get us off on a shared footing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "2edcc9b1-0c2c-4092-a354-0999ad1ef4c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "from datetime import datetime\n",
    "\n",
    "client = Client()\n",
    "example_data = [\n",
    "    ('Who trained Llama-v2?', \"I'm sorry, but I don't have that information.\"),\n",
    "     ('When did langchain first announce the hub?',\n",
    "      'LangChain first announced the LangChain Hub on September 5, 2023.'),\n",
    "     (\"What's LangSmith?\",\n",
    "      \"LangSmith is a platform developed by LangChain for building production-grade LLM (Language Model) applications. It allows you to debug, test, evaluate, and monitor chains and intelligent agents built on any LLM framework. LangSmith seamlessly integrates with LangChain's open-source framework called LangChain, which is widely used for building applications with LLMs.\\n\\nLangSmith provides full visibility into model inputs and outputs at every step in the chain of events, making it easier to debug and analyze the behavior of LLM applications. It has been tested with early design partners and on internal workflows, and it has been found to help teams in various ways.\\n\\nYou can find more information about LangSmith on the official LangSmith documentation [here](https://docs.smith.langchain.com/). Additionally, you can read about the announcement of LangSmith as a unified platform for debugging and testing LLM applications [here](https://blog.langchain.dev/announcing-langsmith/).\"),\n",
    "     ('What is the langsmith cookbook?',\n",
    "      'I\\'m sorry, but I couldn\\'t find any information about the \"Langsmith Cookbook\". It\\'s possible that it may not be a well-known cookbook or it may not exist. Could you provide more context or clarify the name?'),\n",
    "     ('What is LangChain?',\n",
    "      'I\\'m sorry, but I couldn\\'t find any information about \"LangChain\". Could you please provide more context or clarify your question?'),\n",
    "     ('When was Llama-v2 released?', 'Llama-v2 was released on July 18, 2023.'),\n",
    "]\n",
    "\n",
    "for input_, output_ in example_data:\n",
    "    client.create_run(name=\"ExampleRun\", run_type=\"chain\", inputs={\"input\": input_}, \n",
    "                          outputs={\"output\": output_}, project_name=project_name,\n",
    "                          end_time=datetime.utcnow())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "132bf771-6da1-4a98-840f-8800f07a8c9e",
   "metadata": {},
   "source": [
    "## 1. Select Runs\n",
    "\n",
    "In this example, we will be adding model-based feedback to the run traces within a single project. To find your project name or ID, you can go to the [Projects](https://smith.langchain.com/projects) page for your organization and then call the `list_runs()` method on the LangSmith client.\n",
    "\n",
    "```\n",
    "runs = client.list_runs(project_name=project_name)\n",
    "```\n",
    "\n",
    "If your project is capturing logs from a deployed chain or agent, you'll likely want to filter based on time so you can run the feedback pipeline on a schedul. The query below filters for runs since midnight, last-night UTC. You can also filter for other things, like runs without errors, runs with specific tags, etc. For more information on doing so, check out the [Run Filtering](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local) guide to learn more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "d9216ab8-4a0e-438b-80c5-50a2427df910",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "midnight = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "\n",
    "runs = list(client.list_runs(\n",
    "    project_name=project_name,\n",
    "    execution_order=1,\n",
    "    start_time=midnight,\n",
    "    errors=False\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec884ee8-c4af-4bc5-b9e7-e87fa4c4f97f",
   "metadata": {},
   "source": [
    "Once you've decided the runs you want to evaluate, it's time to define the feedback pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48efb84a-c84a-4903-95a8-e6c0d1e47fa4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2. Define Feedback Algorithm\n",
    "\n",
    "All feedback needs a key and should have a (nullable) numeric score. You can apply any algorithm to generate these\n",
    "scores, but you'll want to choose the one that makes the most sense for your use case.\n",
    "\n",
    "The following examples run on the \"input\" and \"output\" keys of runs. If your runs use different keys, you may have to update.\n",
    "\n",
    "#### Example A: Simple Text Statistics\n",
    "\n",
    "We will start out by adding some simple text statistics on the input text as feedback. We use this to illustrate that you can use any simple or custom algorithm to generate scores.\n",
    "\n",
    "Scores can be null, boolean, integesr, or float values.\n",
    "\n",
    "**Note:** We are measuring the 'input' key in this example, which is used by LangChain's AgentExecutor class. You will want to confirm the key(s) you want to measure within the run's inputs or outputs dictionaries when applying this example. Common run types (like 'chat' runs) have nested dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "78fadbbe-d641-4a37-abd8-e1a57dd1f99e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install textstat --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "97b2d53f-6a4c-4e08-bc8e-93ebe28574d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import textstat\n",
    "from langsmith.schemas import Run, Example\n",
    "from langchain.schema.runnable import RunnableLambda\n",
    "\n",
    "def compute_stats(run: Run) -> None:\n",
    "    # Note: your chain's runs may have different keys.\n",
    "    # Be sure to select the right field(s) to measure!\n",
    "    if \"input\" not in run.inputs:\n",
    "        return\n",
    "    if run.feedback_stats and \"smog_index\" in run.feedback_stats:\n",
    "        # If we are running this pipeline multiple times\n",
    "        return\n",
    "    text = run.inputs[\"input\"]\n",
    "    try:\n",
    "        fns = [\n",
    "            \"flesch_reading_ease\", \n",
    "            \"flesch_kincaid_grade\",\n",
    "            \"smog_index\", \n",
    "            \"coleman_liau_index\", \n",
    "            \"automated_readability_index\",\n",
    "        ]\n",
    "        metrics ={\n",
    "            fn: getattr(textstat, fn)(text) for fn in fns\n",
    "            for fn in fns\n",
    "        }\n",
    "        for key, value in metrics.items():\n",
    "            client.create_feedback(\n",
    "                run.id,\n",
    "                key=key,\n",
    "                score=value, # The numeric score is used in the monitoring charts\n",
    "                feedback_source_type=\"model\",\n",
    "            )\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "eadd5bf8-f91a-410c-b246-72b75d8d290e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Concurrently log feedback. You could also run this in a 'for' loop\n",
    "# And not use any langchain code\n",
    "_ = RunnableLambda(compute_stats).batch(\n",
    "    runs, \n",
    "    {\"max_concurrency\": 10},\n",
    "    return_errors=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74b85c62-df99-4b21-a2c3-61137a3c0627",
   "metadata": {},
   "source": [
    "#### Example B: AI-assisted feedback\n",
    "\n",
    "Text statistics are simple to generate but often not very informative. Let's make an example that scores each run's input using an LLM. This method lets you score runs based on targeted axes relevant to your application. You could apply this technique to select metrics as proxies for quality or to help curate data to fine-tune an LLM.\n",
    "\n",
    "In the example below, we will instruct an LLM to score user input queries along a number of simple axes.\n",
    "We will be using [this prompt](https://smith.langchain.com/hub/wfh/automated-feedback-example) to drive the chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5db72c58-ae92-47bc-9f0b-6e1688c91c94",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "\n",
    "prompt = hub.pull(\"wfh/automated-feedback-example\", api_url=\"https://api.hub.langchain.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ccbdd724-5b1d-410c-bc0d-124800d197df",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.callbacks import collect_runs\n",
    "from langchain.output_parsers.openai_functions import  JsonOutputFunctionsParser\n",
    "\n",
    "chain = (\n",
    "    prompt\n",
    "    | ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=1).bind(\n",
    "        functions=[{\n",
    "              \"name\": \"submit_scores\",\n",
    "              \"description\": \"Submit the graded scores for a user question and bot response.\",\n",
    "              \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {\n",
    "                  \"relevance\": {\n",
    "                    \"type\": \"integer\",\n",
    "                    \"minimum\": 0,\n",
    "                    \"maximum\": 5,\n",
    "                    \"description\": \"Score indicating the relevance of the question to LangChain/LangSmith.\"\n",
    "                  },\n",
    "                  \"difficulty\": {\n",
    "                    \"type\": \"integer\",\n",
    "                    \"minimum\": 0,\n",
    "                    \"maximum\": 5,\n",
    "                    \"description\": \"Score indicating the complexity or difficulty of the question.\"\n",
    "                  },\n",
    "                  \"verbosity\": {\n",
    "                    \"type\": \"integer\",\n",
    "                    \"minimum\": 0,\n",
    "                    \"maximum\": 5,\n",
    "                    \"description\": \"Score indicating how verbose the question is.\"\n",
    "                  },\n",
    "                  \"specificity\": {\n",
    "                    \"type\": \"integer\",\n",
    "                    \"minimum\": 0,\n",
    "                    \"maximum\": 5,\n",
    "                    \"description\": \"Score indicating how specific the question is.\"\n",
    "                  }\n",
    "                },\n",
    "                \"required\": [\"relevance\", \"difficulty\", \"verbosity\", \"specificity\"]\n",
    "              }\n",
    "            }\n",
    "        ]\n",
    "    )\n",
    "    | JsonOutputFunctionsParser()\n",
    ")\n",
    "\n",
    "def evaluate_run(run: Run) -> None:\n",
    "    try:\n",
    "        # Note: your chain's runs may have different keys.\n",
    "        # Be sure to select the right field(s) to measure!\n",
    "        if \"input\" not in run.inputs or not run.outputs or 'output' not in run.outputs:\n",
    "            return\n",
    "        if run.feedback_stats and 'specificity' in run.feedback_stats:\n",
    "            # We have already scored this run\n",
    "            # (if you're running this pipeline multiple times)\n",
    "            return\n",
    "        with collect_runs() as cb:\n",
    "            result = chain.invoke(\n",
    "                {\"question\": run.inputs[\"input\"][:3000], # lazy truncation\n",
    "                 \"prediction\": run.outputs[\"output\"][:3000]},\n",
    "            )\n",
    "            for feedback_key, value in result.items():\n",
    "                score = int(value) / 5\n",
    "                client.create_feedback(\n",
    "                    run.id,\n",
    "                    key=feedback_key,\n",
    "                    score=score,\n",
    "                    source_run_id=cb.traced_runs[0].id,\n",
    "                    feedback_source_type=\"model\",\n",
    "                )\n",
    "    except Exception as e:\n",
    "        pass\n",
    "\n",
    "wrapped_function = RunnableLambda(evaluate_run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "55e061a9-8d0e-4474-872a-2729785027f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concurrently log feedback\n",
    "_ = wrapped_function.batch(runs, {\"max_concurrency\": 10}, return_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "088e262f-6f82-4e0b-afee-e414cc9bd38b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'smog_index': {'n': 6, 'avg': 0.0, 'mode': 0, 'is_all_model': True},\n",
       " 'coleman_liau_index': {'n': 6,\n",
       "  'avg': 7.825,\n",
       "  'mode': 3.43,\n",
       "  'is_all_model': True},\n",
       " 'flesch_reading_ease': {'n': 6,\n",
       "  'avg': 92.79666666666667,\n",
       "  'mode': 75.88,\n",
       "  'is_all_model': True},\n",
       " 'flesch_kincaid_grade': {'n': 6,\n",
       "  'avg': 1.3,\n",
       "  'mode': 2.9,\n",
       "  'is_all_model': True},\n",
       " 'automated_readability_index': {'n': 6,\n",
       "  'avg': 9.0,\n",
       "  'mode': 5.2,\n",
       "  'is_all_model': True}}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Updating the aggregate stats is async, but after some time, the logged feedback stats\n",
    "client.read_project(project_name=project_name).feedback_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f59c106-7a98-41ef-a769-78005b6ab6c1",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Example C: LangChain Evaluators\n",
    "\n",
    "LangChain has a number of reference-free evaluators you can use off-the-shelf or configure to your needs. You can apply these directly to your runs to log the evaluation results as feedback. For more information on available LangChain evaluators, check out the [open source documentation](https://python.langchain.com/docs/guides/evaluation).\n",
    "\n",
    "Below, we will demonstrate this by using the criteria evaluator, which instructs an LLM to check that the prediction against the described criteria. The criterion we specify will be \"completeness\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7a8e56-ad86-4e7e-809b-f86704f0cd2f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from typing import Optional\n",
    "from langchain import evaluation, callbacks\n",
    "from langsmith import evaluation as ls_evaluation\n",
    "\n",
    "class CompletenessEvaluator(ls_evaluation.RunEvaluator):\n",
    "    \n",
    "    def __init__(self):\n",
    "        criteria_description=(\n",
    "            \"Does the answer provide sufficient and complete information\"\n",
    "            \"to fully address all aspects of the question (Y)?\"\n",
    "            \" Or does it lack important details (N)?\"\n",
    "        )\n",
    "        self.evaluator = evaluation.load_evaluator(\"criteria\", \n",
    "                                      criteria={\n",
    "                                          \"completeness\": criteria_description\n",
    "                                      })\n",
    "    def evaluate_run(\n",
    "        self, run: Run, example: Optional[Example] = None\n",
    "    ) -> ls_evaluation.EvaluationResult:\n",
    "        if not run.inputs or not run.inputs.get('input') or not run.outputs or not run.outputs.get(\"output\"):\n",
    "            return ls_evaluation.EvaluationResult(key=\"completeness\", score=None)\n",
    "        question = run.inputs['input']\n",
    "        prediction = run.outputs['output']\n",
    "        with callbacks.collect_runs() as cb:\n",
    "            result = self.evaluator.evaluate_strings(input=question, prediction=prediction)\n",
    "            run_id = cb.traced_runs[0].id\n",
    "        return ls_evaluation.EvaluationResult(\n",
    "            key=\"completeness\", evaluator_info={\"__run\": {\"run_id\": run_id}}, **result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bd061c7-039c-4799-8eeb-2acffc082545",
   "metadata": {},
   "source": [
    "Here, we are using the `collect_runs` callback handler to easily fetch the run ID from the evaluation run. By adding it to the evaluator_info, the feedback will retain a link from the evaluated run to the source run so you can see why the tag was generated. Below, we will log feedback to all the traces in our project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0fe87cd7-af8d-4e72-a994-fbf6c35a8672",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "evaluator = CompletenessEvaluator()\n",
    "\n",
    "# We can run as a regular for loop\n",
    "# for run in runs:\n",
    "#     client.evaluate_run(run, evaluator)\n",
    "\n",
    "# Or concurrently log feedback\n",
    "wrapped_function = RunnableLambda(lambda run: client.evaluate_run(run, evaluator))\n",
    "_ = wrapped_function.batch(runs, {\"max_concurrency\": 10}, return_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e66e144-141a-4d23-b7c3-3627d8ae6265",
   "metadata": {},
   "source": [
    "Check out the target project to see the feedback appear as the runs are evaluated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cdb41ef-3892-4385-8830-c6decfbf8f5c",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Congrats! You've set up an algorithmic feedback script to apply to your traced runs. This can improve the quality of your monitoring metrics, help to curate data for fine-tuning datasets, and let you better explore the usage of your app."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}