exploratory-data-analysis/exporting-llm-runs-and-feedback/llm_run_etl.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "id": "c4ae8806-be35-44e1-93c7-878eed7f14f3",
            "metadata": {},
            "source": [
                "# Exporting LLM Runs and Feedback\n",
                "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/exploratory-data-analysis/exporting-llm-runs-and-feedback/llm_run_etl.ipynb)\n",
                "\n",
                "Understanding how your LLM app interacts with users is crucial. LangSmith offers a number of useful ways to interact with and annotate trace data directly in the app. You can also easily query that trace data so you can process it with your tool of choice.\n",
                "\n",
                "This tutorial guides you through exporting LLM traces and associated feedback from LangSmith for further analysis. By the end, you'll be able to export a flat table of LLM run information that you can analyze, enrich, or use for model training.\n",
                "\n",
                "Before we start, ensure you have a LangChain project with some logged traces. You can generate some using almost any of the other recipes in this cookbook. The overall steps will be:\n",
                "\n",
                "1. Query runs, filtering by time, tags, or other attributes.\n",
                "2. Add in associated feedback metrics (if captured)\n",
                "3. Export to analysis tool.\n",
                "\n",
                "To make things easy, we will be loading the data into a pandas dataframe. We will be doing the ETL on LLM runs logged from LangChain, but you can modify the code below to handle whatever schema is used by your deployed model. Now let's set up our environment!\n",
                "\n",
                "#### Setup\n",
                "\n",
                "First, install langsmith and pandas and set your langsmith API key to connect to your project.\n",
                "We will also install LangChain to use one of its formatting utilities."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "id": "5a6b4974-1f59-4517-bce3-b4836783b3dc",
            "metadata": {},
            "outputs": [],
            "source": [
                "# %pip install -U langchain langsmith pandas seaborn --quiet"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "id": "4c37e579-d044-4d95-8dc5-1763f956f481",
            "metadata": {},
            "outputs": [],
            "source": [
                "# %env LANGCHAIN_API_KEY=\"\""
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "428695ac-6ca5-4967-8e17-7b0fc6a4df51",
            "metadata": {},
            "outputs": [],
            "source": [
                "from langsmith import Client\n",
                "\n",
                "client = Client()"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "d3143554-6a2a-4f42-b6a3-0df9cb521314",
            "metadata": {},
            "source": [
                "## 1. Query Runs\n",
                "\n",
                "Now that the environment is ready, we will load the run data from LangSmith. Let's try loading all our LLM runs from the past week. To do so, we will filter for runs with the \"llm\" `run_type` from the past week.\n",
                "\n",
                "Please reference the [docs](https://docs.smith.langchain.com/tracing/use-cases/export-runs/local) for guidance on more complex filters (using metadata, tags, and other attributes).\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "d85e089d-d7c1-4f61-bb4f-d6728da98b4e",
            "metadata": {},
            "outputs": [],
            "source": [
                "from datetime import datetime, timedelta\n",
                "\n",
                "start_time = datetime.utcnow() - timedelta(days=7)\n",
                "\n",
                "runs = list(client.list_runs(\n",
                "    project_name=\"YOUR PROJECT NAME\",\n",
                "    run_type=\"llm\",\n",
                "    start_time=start_time,\n",
                "))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "dbe677d3-be2d-430c-8609-6e3688d2e8ea",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>name</th>\n",
                            "      <th>model</th>\n",
                            "      <th>messages</th>\n",
                            "      <th>run</th>\n",
                            "      <th>llm_output</th>\n",
                            "      <th>generations</th>\n",
                            "      <th>error</th>\n",
                            "      <th>latency</th>\n",
                            "      <th>prompt_tokens</th>\n",
                            "      <th>completion_tokens</th>\n",
                            "      <th>total_tokens</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>4ddeed83-f438-4d77-b6bb-ec26ea0824af</th>\n",
                            "      <td>ChatOpenAI</td>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>[{'id': ['langchain', 'schema', 'messages', 'S...</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>[{'text': 'Coffee can have both positive and n...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>14.133700</td>\n",
                            "      <td>2028.0</td>\n",
                            "      <td>350.0</td>\n",
                            "      <td>2378.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>51d22986-33d2-416b-911f-1083c719b768</th>\n",
                            "      <td>ChatOpenAI</td>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>[{'id': ['langchain', 'schema', 'messages', 'H...</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>[{'text': 'Is coffee beneficial for our health...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.907260</td>\n",
                            "      <td>191.0</td>\n",
                            "      <td>7.0</td>\n",
                            "      <td>198.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fa5e83ff-497d-411c-97d4-6cd878ba80f6</th>\n",
                            "      <td>ChatOpenAI</td>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>[{'id': ['langchain', 'schema', 'messages', 'H...</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>[{'text': 'Is coffee beneficial for our health...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.588772</td>\n",
                            "      <td>180.0</td>\n",
                            "      <td>7.0</td>\n",
                            "      <td>187.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>54861172-33ec-4414-b907-0e75b4918125</th>\n",
                            "      <td>ChatOpenAI</td>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>[{'id': ['langchain', 'schema', 'messages', 'S...</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>[{'text': 'Hello! How can I assist you today?'...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.584520</td>\n",
                            "      <td>1202.0</td>\n",
                            "      <td>9.0</td>\n",
                            "      <td>1211.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a1cf4e28-d0e5-45f2-8faf-f314858380ca</th>\n",
                            "      <td>ChatOpenAI</td>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>[{'id': ['langchain', 'schema', 'messages', 'S...</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>NaN</td>\n",
                            "      <td>[{'text': 'You're welcome! If you have any mor...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.072372</td>\n",
                            "      <td>2488.0</td>\n",
                            "      <td>22.0</td>\n",
                            "      <td>2510.0</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                            name              model  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  ChatOpenAI  gpt-3.5-turbo-16k   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  ChatOpenAI  gpt-3.5-turbo-16k   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  ChatOpenAI  gpt-3.5-turbo-16k   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  ChatOpenAI  gpt-3.5-turbo-16k   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  ChatOpenAI  gpt-3.5-turbo-16k   \n",
                            "\n",
                            "                                                                               messages  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  [{'id': ['langchain', 'schema', 'messages', 'S...   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  [{'id': ['langchain', 'schema', 'messages', 'H...   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  [{'id': ['langchain', 'schema', 'messages', 'H...   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  [{'id': ['langchain', 'schema', 'messages', 'S...   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  [{'id': ['langchain', 'schema', 'messages', 'S...   \n",
                            "\n",
                            "                                      run  llm_output  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  NaN         NaN   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  NaN         NaN   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  NaN         NaN   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  NaN         NaN   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  NaN         NaN   \n",
                            "\n",
                            "                                                                            generations  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  [{'text': 'Coffee can have both positive and n...   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  [{'text': 'Is coffee beneficial for our health...   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  [{'text': 'Is coffee beneficial for our health...   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  [{'text': 'Hello! How can I assist you today?'...   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  [{'text': 'You're welcome! If you have any mor...   \n",
                            "\n",
                            "                                     error    latency  prompt_tokens  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  None  14.133700         2028.0   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  None   0.907260          191.0   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  None   0.588772          180.0   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  None   0.584520         1202.0   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  None   1.072372         2488.0   \n",
                            "\n",
                            "                                      completion_tokens  total_tokens  \n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af              350.0        2378.0  \n",
                            "51d22986-33d2-416b-911f-1083c719b768                7.0         198.0  \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6                7.0         187.0  \n",
                            "54861172-33ec-4414-b907-0e75b4918125                9.0        1211.0  \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca               22.0        2510.0  "
                        ]
                    },
                    "execution_count": 3,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "import pandas as pd\n",
                "\n",
                "df = pd.DataFrame(\n",
                "    [{\n",
                "        \"name\": run.name,\n",
                "        \"model\": run.extra['invocation_params']['model'], # The parameters used when invoking the model are nested in the extra info\n",
                "        **run.inputs,\n",
                "        **(run.outputs or {}),\n",
                "        \"error\":run.error,\n",
                "        \"latency\": (run.end_time - run.start_time).total_seconds() if run.end_time else None, # Pending runs have no end time\n",
                "        \"prompt_tokens\": run.prompt_tokens,\n",
                "        \"completion_tokens\": run.completion_tokens,\n",
                "        \"total_tokens\": run.total_tokens,\n",
                "    }\n",
                "          for run in runs\n",
                "    ],\n",
                "    index=[run.id for run in runs]\n",
                ")\n",
                "\n",
                "df.head(5)"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "a3711bfc-316b-4e85-85bf-e457d6679afc",
            "metadata": {},
            "source": [
                "#### Stringify\n",
                "\n",
                "If you are using a regular \"Completion\" style model that expects string input and returns a single string output, you can easily view the text data without further formatting.\n",
                "\n",
                "For chat models, the message dictionaries contain a lot of information that can be hard to read. If you want to _just_ see the string, you may consider parsing the message content into a human-readable format. You can use LangChain's `get_buffer_string` helper to do so, as shown below."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "id": "f8503857-ac53-44ee-8e01-c1a996e9d028",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>model</th>\n",
                            "      <th>messages</th>\n",
                            "      <th>generated_message</th>\n",
                            "      <th>error</th>\n",
                            "      <th>latency</th>\n",
                            "      <th>prompt_tokens</th>\n",
                            "      <th>completion_tokens</th>\n",
                            "      <th>total_tokens</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>4ddeed83-f438-4d77-b6bb-ec26ea0824af</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: \\n    You are an expert programmer and...</td>\n",
                            "      <td>AI: Coffee can have both positive and negative...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>14.133700</td>\n",
                            "      <td>2028.0</td>\n",
                            "      <td>350.0</td>\n",
                            "      <td>2378.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>51d22986-33d2-416b-911f-1083c719b768</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>Human: Given the following conversation and a ...</td>\n",
                            "      <td>AI: Is coffee beneficial for our health?</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.907260</td>\n",
                            "      <td>191.0</td>\n",
                            "      <td>7.0</td>\n",
                            "      <td>198.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>fa5e83ff-497d-411c-97d4-6cd878ba80f6</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>Human: Given the following conversation and a ...</td>\n",
                            "      <td>AI: Is coffee beneficial for our health?</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.588772</td>\n",
                            "      <td>180.0</td>\n",
                            "      <td>7.0</td>\n",
                            "      <td>187.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>54861172-33ec-4414-b907-0e75b4918125</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: \\n    You are an expert programmer and...</td>\n",
                            "      <td>AI: Hello! How can I assist you today?</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.584520</td>\n",
                            "      <td>1202.0</td>\n",
                            "      <td>9.0</td>\n",
                            "      <td>1211.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a1cf4e28-d0e5-45f2-8faf-f314858380ca</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: \\n    You are an expert programmer and...</td>\n",
                            "      <td>AI: You're welcome! If you have any more quest...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.072372</td>\n",
                            "      <td>2488.0</td>\n",
                            "      <td>22.0</td>\n",
                            "      <td>2510.0</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                  model  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  gpt-3.5-turbo-16k   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  gpt-3.5-turbo-16k   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  gpt-3.5-turbo-16k   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  gpt-3.5-turbo-16k   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  gpt-3.5-turbo-16k   \n",
                            "\n",
                            "                                                                               messages  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  System: \\n    You are an expert programmer and...   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  Human: Given the following conversation and a ...   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  Human: Given the following conversation and a ...   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  System: \\n    You are an expert programmer and...   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  System: \\n    You are an expert programmer and...   \n",
                            "\n",
                            "                                                                      generated_message  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  AI: Coffee can have both positive and negative...   \n",
                            "51d22986-33d2-416b-911f-1083c719b768           AI: Is coffee beneficial for our health?   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6           AI: Is coffee beneficial for our health?   \n",
                            "54861172-33ec-4414-b907-0e75b4918125             AI: Hello! How can I assist you today?   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  AI: You're welcome! If you have any more quest...   \n",
                            "\n",
                            "                                     error    latency  prompt_tokens  \\\n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af  None  14.133700         2028.0   \n",
                            "51d22986-33d2-416b-911f-1083c719b768  None   0.907260          191.0   \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6  None   0.588772          180.0   \n",
                            "54861172-33ec-4414-b907-0e75b4918125  None   0.584520         1202.0   \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca  None   1.072372         2488.0   \n",
                            "\n",
                            "                                      completion_tokens  total_tokens  \n",
                            "4ddeed83-f438-4d77-b6bb-ec26ea0824af              350.0        2378.0  \n",
                            "51d22986-33d2-416b-911f-1083c719b768                7.0         198.0  \n",
                            "fa5e83ff-497d-411c-97d4-6cd878ba80f6                7.0         187.0  \n",
                            "54861172-33ec-4414-b907-0e75b4918125                9.0        1211.0  \n",
                            "a1cf4e28-d0e5-45f2-8faf-f314858380ca               22.0        2510.0  "
                        ]
                    },
                    "execution_count": 4,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "from typing import Optional\n",
                "\n",
                "from langchain.load.load import load\n",
                "from langchain.schema import get_buffer_string\n",
                "\n",
                "def stringify_inputs(inputs: dict) -> dict:\n",
                "    return {\n",
                "        \"messages\": get_buffer_string(load(inputs['messages']))\n",
                "    }\n",
                "\n",
                "def stringify_outputs(outputs: Optional[dict]) -> dict:\n",
                "    if not outputs:\n",
                "        return {}\n",
                "    if isinstance(outputs['generations'], dict):\n",
                "        # Function Message\n",
                "        return {\n",
                "            \"generated_message\": get_buffer_string([load(outputs['generations']['message'])])\n",
                "        }\n",
                "    else:\n",
                "        return {\n",
                "            \"generated_message\": get_buffer_string([load(outputs['generations'][0]['message'])])\n",
                "        }\n",
                "\n",
                "df = pd.DataFrame(\n",
                "    [{\n",
                "        \"model\": run.extra['invocation_params']['model'], # The parameters used when invoking the model are nested in the extra info    \n",
                "        **stringify_inputs(run.inputs),\n",
                "        **stringify_outputs(run.outputs),\n",
                "        \"error\":run.error,\n",
                "        \"latency\": (run.end_time - run.start_time).total_seconds() if run.end_time else None, # Pending runs have no end time\n",
                "        \"prompt_tokens\": run.prompt_tokens,\n",
                "        \"completion_tokens\": run.completion_tokens,\n",
                "        \"total_tokens\": run.total_tokens,\n",
                "    }\n",
                "          for run in runs\n",
                "    ],\n",
                "    index=[run.id for run in runs]\n",
                ")\n",
                "\n",
                "df.head(5)"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "5b189858-f4b0-477b-8ec3-a6dad6390f14",
            "metadata": {},
            "source": [
                "## 2. Load Feedback\n",
                "\n",
                "Improving your model hinges on understanding user signals. In LangSmith, you can access both explicit (like thumbs up/down) and implicit signals (like click-through rates) through the feedback API. Feedback can be structured (contains a numeric score) or unstructured (comments/corrections). You'll find quantitative statistics on the run object's `run.feedback_stats` attribute if available, and you can directly fetch the full feedback entries by calling `list_feedback()` by run_id.\n",
                "\n",
                "**Note**, LLM runs are often nested in chains, and feedback is likely assigned to the root run of a trace. To find associated feedback, consider querying the feedback for the root run. This way the feedback is evenly assigned to any LLM calls within a trace."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "id": "3d68c161-8261-43e2-b8ef-f880c80dbf5a",
            "metadata": {},
            "outputs": [],
            "source": [
                "from functools import lru_cache\n",
                "from langsmith.schemas import Run\n",
                "\n",
                "@lru_cache(maxsize=1000)\n",
                "def _fetch_run(run_id: str) -> Run:\n",
                "    return client.read_run(run_id)\n",
                "\n",
                "def get_root(run: Run) -> Run:\n",
                "    if run.execution_order == 1:\n",
                "        return run\n",
                "    return _fetch_run(str(run.parent_run_ids[-1]))\n",
                "\n",
                "def get_feedback(r: Run) -> dict:\n",
                "    if not r.feedback_stats:\n",
                "        return {}\n",
                "    return {k: v['avg'] for k, v in r.feedback_stats.items()}\n",
                "\n",
                "def get_root_feedback(run: Run) -> dict:\n",
                "    root_run = get_root(run)\n",
                "    return get_feedback(root_run)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "id": "4d65008b-6b0e-46f2-ad80-1f78c5ff68fa",
            "metadata": {},
            "outputs": [],
            "source": [
                "df = pd.DataFrame(\n",
                "    [{\n",
                "        \"model\": run.extra['invocation_params']['model'],\n",
                "        **stringify_inputs(run.inputs),\n",
                "        **stringify_outputs(run.outputs),\n",
                "        **get_root_feedback(run),\n",
                "        \"error\":run.error,\n",
                "        \"latency\": (run.end_time - run.start_time).total_seconds() if run.end_time else None, # Pending runs have no end time\n",
                "        \"prompt_tokens\": run.prompt_tokens,\n",
                "        \"completion_tokens\": run.completion_tokens,\n",
                "        \"total_tokens\": run.total_tokens,\n",
                "    }\n",
                "          for run in runs\n",
                "    ],\n",
                "    index=[run.id for run in runs]\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "id": "f039f1bb-b360-449f-b24d-80603148c893",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>model</th>\n",
                            "      <th>messages</th>\n",
                            "      <th>generated_message</th>\n",
                            "      <th>error</th>\n",
                            "      <th>latency</th>\n",
                            "      <th>prompt_tokens</th>\n",
                            "      <th>completion_tokens</th>\n",
                            "      <th>total_tokens</th>\n",
                            "      <th>user_score</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>c1f718f7-0466-4234-8eda-4c8dd06f08a7</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>Human: Given the following conversation and a ...</td>\n",
                            "      <td>AI: What is the meaning of pressing \"f\" in onl...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.358865</td>\n",
                            "      <td>588.0</td>\n",
                            "      <td>13.0</td>\n",
                            "      <td>601.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>a34c03ad-f24e-4113-a32e-f53417fd999d</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>Human: Given the following conversation and a ...</td>\n",
                            "      <td>AI: Can the code responses be defaulted to Jav...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.033514</td>\n",
                            "      <td>1884.0</td>\n",
                            "      <td>9.0</td>\n",
                            "      <td>1893.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>f06b745b-10c5-42b2-87fd-f08472a997b0</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>Human: Given the following conversation and a ...</td>\n",
                            "      <td>AI: Can you provide an example of a chat funct...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.885441</td>\n",
                            "      <td>264.0</td>\n",
                            "      <td>16.0</td>\n",
                            "      <td>280.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2161535b-f2aa-4f68-8815-9679e9bc42f7</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: The LangChain Expression Language is a lan...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>16.265123</td>\n",
                            "      <td>1654.0</td>\n",
                            "      <td>298.0</td>\n",
                            "      <td>1952.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>ec79f4c8-9957-412b-84ee-4373b0d00340</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: {'name': 'Documentation', 'arguments': '{\\...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.104964</td>\n",
                            "      <td>356.0</td>\n",
                            "      <td>19.0</td>\n",
                            "      <td>375.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2ef17693-7a1e-45b1-82e1-b49423a408ea</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: LangChain is a Python package that provide...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>16.196424</td>\n",
                            "      <td>1648.0</td>\n",
                            "      <td>353.0</td>\n",
                            "      <td>2001.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>7b3b9603-8a2f-433f-8b4e-0a5f7947f562</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: {'name': 'Documentation', 'arguments': '{\\...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.077405</td>\n",
                            "      <td>738.0</td>\n",
                            "      <td>17.0</td>\n",
                            "      <td>755.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>92aa38b5-b9ee-4803-8257-45c7c6f9e1ec</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: {'name': 'Documentation', 'arguments': '{\\...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>1.089527</td>\n",
                            "      <td>694.0</td>\n",
                            "      <td>17.0</td>\n",
                            "      <td>711.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>74bac7f6-1ac9-4288-9e33-900659ae74a6</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: Retrieval Augmented Generation (RAG) is a ...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>14.711333</td>\n",
                            "      <td>3311.0</td>\n",
                            "      <td>777.0</td>\n",
                            "      <td>4088.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>6e363d67-35e7-4a12-b728-4e1959b02b0e</th>\n",
                            "      <td>gpt-3.5-turbo-16k</td>\n",
                            "      <td>System: You are an expert developer tasked ans...</td>\n",
                            "      <td>AI: {'name': 'Documentation', 'arguments': '{\\...</td>\n",
                            "      <td>None</td>\n",
                            "      <td>0.888599</td>\n",
                            "      <td>359.0</td>\n",
                            "      <td>20.0</td>\n",
                            "      <td>379.0</td>\n",
                            "      <td>1.0</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                                  model  \\\n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7  gpt-3.5-turbo-16k   \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d  gpt-3.5-turbo-16k   \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0  gpt-3.5-turbo-16k   \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7  gpt-3.5-turbo-16k   \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340  gpt-3.5-turbo-16k   \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea  gpt-3.5-turbo-16k   \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562  gpt-3.5-turbo-16k   \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec  gpt-3.5-turbo-16k   \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6  gpt-3.5-turbo-16k   \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e  gpt-3.5-turbo-16k   \n",
                            "\n",
                            "                                                                               messages  \\\n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7  Human: Given the following conversation and a ...   \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d  Human: Given the following conversation and a ...   \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0  Human: Given the following conversation and a ...   \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7  System: You are an expert developer tasked ans...   \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340  System: You are an expert developer tasked ans...   \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea  System: You are an expert developer tasked ans...   \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562  System: You are an expert developer tasked ans...   \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec  System: You are an expert developer tasked ans...   \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6  System: You are an expert developer tasked ans...   \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e  System: You are an expert developer tasked ans...   \n",
                            "\n",
                            "                                                                      generated_message  \\\n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7  AI: What is the meaning of pressing \"f\" in onl...   \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d  AI: Can the code responses be defaulted to Jav...   \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0  AI: Can you provide an example of a chat funct...   \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7  AI: The LangChain Expression Language is a lan...   \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340  AI: {'name': 'Documentation', 'arguments': '{\\...   \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea  AI: LangChain is a Python package that provide...   \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562  AI: {'name': 'Documentation', 'arguments': '{\\...   \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec  AI: {'name': 'Documentation', 'arguments': '{\\...   \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6  AI: Retrieval Augmented Generation (RAG) is a ...   \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e  AI: {'name': 'Documentation', 'arguments': '{\\...   \n",
                            "\n",
                            "                                     error    latency  prompt_tokens  \\\n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7  None   1.358865          588.0   \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d  None   1.033514         1884.0   \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0  None   0.885441          264.0   \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7  None  16.265123         1654.0   \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340  None   1.104964          356.0   \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea  None  16.196424         1648.0   \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562  None   1.077405          738.0   \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec  None   1.089527          694.0   \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6  None  14.711333         3311.0   \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e  None   0.888599          359.0   \n",
                            "\n",
                            "                                      completion_tokens  total_tokens  \\\n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7               13.0         601.0   \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d                9.0        1893.0   \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0               16.0         280.0   \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7              298.0        1952.0   \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340               19.0         375.0   \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea              353.0        2001.0   \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562               17.0         755.0   \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec               17.0         711.0   \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6              777.0        4088.0   \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e               20.0         379.0   \n",
                            "\n",
                            "                                      user_score  \n",
                            "c1f718f7-0466-4234-8eda-4c8dd06f08a7         1.0  \n",
                            "a34c03ad-f24e-4113-a32e-f53417fd999d         1.0  \n",
                            "f06b745b-10c5-42b2-87fd-f08472a997b0         1.0  \n",
                            "2161535b-f2aa-4f68-8815-9679e9bc42f7         1.0  \n",
                            "ec79f4c8-9957-412b-84ee-4373b0d00340         1.0  \n",
                            "2ef17693-7a1e-45b1-82e1-b49423a408ea         1.0  \n",
                            "7b3b9603-8a2f-433f-8b4e-0a5f7947f562         1.0  \n",
                            "92aa38b5-b9ee-4803-8257-45c7c6f9e1ec         1.0  \n",
                            "74bac7f6-1ac9-4288-9e33-900659ae74a6         1.0  \n",
                            "6e363d67-35e7-4a12-b728-4e1959b02b0e         1.0  "
                        ]
                    },
                    "execution_count": 7,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "df[~pd.isna(df['user_score'])].head(10)"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "ae814138-304d-4355-8710-0a3fdf04fb6b",
            "metadata": {},
            "source": [
                "## 3. Analyze Data\n",
                "\n",
                "Once you have the data in flat form, you can export to many compatible tools, from tabular storage like Airtable and Excel, to annotation tools like [LabelBox](https://docs.labelbox.com/reference/text-file) to other text analysis tools like [Lilac](../lilac/lilac.ipynb) or [Nomic](https://atlas.nomic.ai/).\n",
                "\n",
                "For the purpose of this tutorial, we will wrap things up with a simple token plot. Check out our other recipes for more involved analysis."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "id": "7b60594a-a3b2-4fdb-8a0a-686bba755dcb",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAABKQAAAJOCAYAAACJLN8OAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmG0lEQVR4nO3dd1yVdf/H8fcBZCjuvTKtxAWIGpojd5mjzG3lLs3QhpYjy8xxmyu9c+RMM82VA7XcM80VJuLKbebElBwgCFy/P/xxbo+ADDnXAXw9Hw8fd+ean+s6H84F7/v6XsdiGIYhAAAAAAAAwCROji4AAAAAAAAATxYCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoXRxcAAIC97NmzRx07dkxyuTJlyigwMNCEivCwAQMGaPny5TbTLBaL3NzclCdPHnl7e6tNmzaqWbNmvHU7dOigvXv3av369SpRokSK933+/Hn9/vvvev3115O1fL169XThwgUdPnxYLi4u1v5q1qyZxo4dm+L9J8f69ev11FNPqUyZMpJkyj7tZdmyZZo6daouXryo7Nmza+7cuXruuedslpk4caImTZqU7G2OHDlSLVq0SPbyKX3PH5bc8+/l5ZXsbRYtWlSbN29O1rJ///236tevr0qVKmnBggXJ3gcAAOkRgRQAINMrWrToI/8AzZcvn4nVICH169dX2bJlJUmGYejOnTs6f/68tm/frnXr1qlr167q37+/zTqvv/66/P39lTNnzhTv79ixY2rdurVefvnlZIcTHTt21K1bt+TkZM4N5mPHjtWMGTM0e/Zs67SiRYuqV69eKQo80oOTJ09q0KBBcnd3V7t27ZQlSxY99dRT8Zbz9/dXr169bKZt3LhRx44ds+mROA+/fpTUvOep9fAx3Lx5U3PnzlX27NnVqVMnm3nZs2e3ay0AAKRXBFIAgEyvaNGi6t27t6PLwCM0aNAgwTtd/vrrL/Xo0UPfffedihUrpjfffNM6LyV3xjzs33//VVRUVIrW6dy5c6r3lxrXrl2LN61YsWIZspcPHTqk2NhYvf766/rss88SXa5q1aqqWrWqzbQLFy7o2LFjifZIcqXmPU+th9+jv//+W3PnzlWOHDky5PsHAIA98AwpAACQbj311FOaNGmSsmTJookTJ+rOnTuOLgmpcO/ePUlS7ty5HVwJAABILwikAAB4QL169dS0aVNt2rRJ9erVk7e3t9q3by/DMCRJV65c0ZAhQ1S3bl1VqFBBNWvW1MCBA/X3338nuL2FCxfqtddek6+vr+rUqaNJkyYpJCREXl5eGj9+vHW5AQMGyMvLS7/99lu8bbRv315eXl7x9nHmzBl98sknqlmzpipUqKB69epp+PDhun79us1yy5Ytk5eXl9auXaulS5eqefPm8vHxUdWqVdW3b1+dP38+3j7v3bunmTNn6rXXXlPFihVVs2ZN9ejRQwcPHpQkRUZGyt/fX35+foqIiIi3/m+//SYvLy/95z//SeKMJ+2ZZ55Rw4YNdePGDe3YscM6vUOHDvLy8tK5c+es086fP69+/fqpQYMGqlChgmrUqKHevXvr0KFD1mUGDBhgfbbYqlWr5OXlpWXLlkl69Ptfr149eXl5KTo6Ol6Na9eu1auvvipvb2/VqVNHX331lW7evGmzTEreYy8vL+uztbp06WIdordnzx55eXnp448/tln/7t27mjx5spo0aSJvb29VqVJFnTt31tatW+Pty8vLSz169NDJkyf13nvv6fnnn5evr6/atGmjdevWJf5GPCQmJkY//PCDXn/9dfn6+srPz0/t2rXTihUr4u0v7q6oSZMmycvLSxMnTkz2fpKybds2denSRVWqVJG3t7caN26sSZMm2fTlo95zSdq9e7d69epl/VmqXLmy2rZta7OMPYWFhWnUqFFq2LChKlSooKpVq+rdd9/V/v37k7X+0qVLVaZMGTVs2FCXLl2yTo+MjNS0adPUtGlT+fj46Pnnn9fbb7+tffv2xdtGSvvi6NGj6tWrl/Wz8MUXX1T//v119uzZVJ8HAMCThSF7AAA85MqVK+rTp48aNGigXLlyKWfOnLJYLDp16pQ6duyof/75R7Vr11bjxo31999/KzAwUJs3b9acOXNsnmkzePBgLVq0SIULF1bLli119+5dTZ8+3SZUSa19+/ape/fuioqKUv369fXUU0/pzz//1A8//KBNmzZpwYIFKlSokM06M2fO1NGjR1W/fn3VqFFDe/bs0erVq/X777/rl19+UbZs2SRJUVFR6tSpk/bv36+SJUtaa//555+1c+dOzZw5U9WqVVOTJk30448/av369Xrttdds9hUXpjzOEKsH+fv765dfftG+ffv08ssvJ7jM9evX1a5dO926dUsvvfSSihQpogsXLmjdunXatm2bFi9erDJlyqhBgwbWGkuXLq2XXnrJ5n1L7P1PzO7du7V69WrVqVNH1atX1549ezR79mz99ttvWrRokTw8PFJ8vL169bI+O+m1115T8eLFE1321q1b6tChg44eParnnntObdu21b///qvNmzerR48eev/99xUQEGCzzrlz59S2bVuVKFFCLVu21LVr17RmzRq9//77mjx5svUcJSYqKko9e/bUjh07VLRoUTVv3lz37t3Ttm3b1L9/f+3du9caRvbq1UuHDx/Wli1b5O/vb/2XFiZNmqSJEyfK09NTdevWVa5cubR7925NnDhRGzdu1Lx58+Tp6fnI9/ynn37SZ599pnz58qlevXrKkSOH/vrrL23evFkDBw5URESEzVDRtHbp0iW98cYbunjxonx8fFS3bl1dvXpVmzZt0vbt2zV8+PBH/hytXLlSn332mUqUKKG5c+eqYMGCkqSIiAh17txZBw4cUPny5dW+fXuFh4dr3bp16tixo0aMGBFvu8nti9OnT6t9+/ZycXHRSy+9pLx58+rUqVNauXKltm7dqlWrVqlAgQJ2O2cAgMyBQAoAkOlduHAh0TsyihYtGu+Psps3b6p79+7q27evzfRPPvlE169f19SpU1WnTh3r9N27d6tLly7q16+fVq5cKYvFor1792rRokUqV66c5syZY33w9htvvPHYf9xGRUWpT58+io2N1aJFi1ShQgXrvOXLl2vAgAEaPHiwpk+fbrPesWPHNH/+fFWsWFGSFBsbq06dOmnv3r3atGmTXn31VUnS7NmztX//fjVt2lQjR46Uq6urJKldu3Zq27atRo4cqcDAQLVs2VI//vijVq5caRNI3b59Wxs2bFD58uWt3w73uIoUKSLpfliUmDVr1ujatWsaPny4WrdubZ3+4osvql+/fpo/f76GDRumBg0aKHv27Fq+fLm8vLziPdMnsfc/MaGhoRo8eLD1fY2Ojtann36qwMBAzZgxQ++//35KD1e9e/e2PjupefPmql69eqLLjh07VkePHlXr1q01ZMgQubjc//Xu/Pnzeuutt/TNN9+oWrVqqly5snWdM2fOqEOHDho0aJA1bHvhhRf06aefav78+UkGUnPmzNGOHTtUp04djR8/XlmzZpV0PxTs0qWLli5dqhdeeEHNmjVT7969tWTJEmsglVbPUAoODtbEiRNVrFgxzZkzxxraRUdH67PPPtPy5cs1atSoR77n9+7d05gxY5Q7d24FBgYqb9681u1v375d77zzjpYvX27XQOrzzz/XxYsX1bt3bwUEBFjfj8OHD+utt97S4MGD5e/vr2LFisVbd82aNRowYIBKlCih77//3hpGSdJ///tfHThwQO+884769u1r3W6vXr3Upk0bffHFF6pevbpNcJ3cvliyZIkiIiI0Z84cvfDCC9b1p0yZov/+979atmyZ3n333bQ/WQCATIUhewCATO/ChQuaNGlSgv/i7uR5WOPGjW1eHzx4UIcPH1b9+vVtwihJqlatmurXr6/jx4/rwIEDku7ftSDdDxYe/Ba4ChUqqH379o91PJs2bdLVq1fVrl07mzBKuv/Nc+XKldO2bdvihTc1atSwhlGS5OTkpPr160uSzXDAFStWyMXFRYMGDbKGUZLk7e2tfv36qUWLFrp3754qVKggLy8v7dq1S1evXrUut3btWkVERKTZ3VGSlCVLFkn3w67ExMbGSrofVMQ9s0i6/15u3LhRX3zxRbL39/D7/yjPPfec3njjDetrFxcXDRw4UFmyZIk3fC2tRUVFaeXKlcqRI4c+++wzaxglScWLF9eHH34oSVq8eHG8dd977z2bO7/iwobEhp8+aOnSpXJyctLQoUOtYZQk5cmTR4MGDZIkLVq0KFXHlFw//fSTJOmjjz6yuYPMxcVFn376qXLkyKEVK1YoMjIy0W3ExMToyy+/1OjRo23CKOn+z7WkeENg09KVK1f066+/qmTJkurVq5fN+1G+fHl169ZN9+7dS3Do4MaNG/Xxxx/r6aef1g8//GATRsXExGjJkiXKkyePPvroI5vtFixYUN26dVNUVJQCAwPjbTc5fRE3hDkoKMj635Ksw0S7d++emtMBAHjCcIcUACDT8/f31w8//JCidR4eIhUSEiLp/h+nCd1t9e+//0qSjhw5Ij8/Px09elSS5OvrG2/ZF154QbNnz05RPQnVcu7cuQRriQtmjh49avNHaqlSpeItG/eV83HfPhYZGanTp0/rmWeeUZ48eeIt//A3zbVs2VL/+c9/tGrVKnXr1k3S/bu0XF1d1bRp01QcXcLiHmYeN6wwIY0aNdKUKVO0ZMkSrV+/XtWqVVONGjX04osvPnLIW0JSsnylSpXiDenLnTu3SpYsqePHjyssLEy5cuVK0f6T6+zZswoPD1fNmjXl7u4eb36VKlUk3e/LB+XKlSve+5sjRw5JSvKb6O7cuaOzZ8+qZMmSNv0Vp1KlSnJ2do63z7R2+PBhSYr3rXzS/WPx8vLSvn37dPLkSZUvXz7Bbbi7u6tRo0aS7gfXJ0+e1N9//60zZ87ojz/+kHQ/3LGXuHOU2BDGxN6/M2fO6MMPP1R0dLR8fHyUP3/+ePNv376tQoUKacqUKfG2e+HCBUn/O4dxktsXLVq00MKFCzVx4kQtWLBA1atXt/6sFS5cOMnjBgBAIpACACBBD/9xH/eA6qCgIAUFBSW6XlhYmKT7z/WREg5QHvebxuJq2bJli7Zs2ZJkLXEevNspTlyQEneXQ9w6cUFVUl599VWNGTNGgYGB6tatm86fP6+goCC9/PLLaRrCxN2d8aigKH/+/Fq2bJmmTZumjRs3at26ddaHMVerVk1DhgxRyZIlk7W/hMKdR+03IXHvfXh4uN0Cqbg+S+z9evB5Qg9yc3OLt+zDvZCYuLvUEtuni4uL8uTJo3/++eeR23lcSdWR2LE/7I8//tDo0aOtDxC3WCwqUaKEqlatqkOHDiV5Ph5Hat+/Gzdu6JlnnpF0PwBu0qSJatWqZZ0fF5BfvnxZkyZNSnT/ccvFSW5flC5dWkuWLNGMGTO0detWrVy5UitXrpSzs7Pq1aunIUOGKF++fInuFwAAiUAKAIBkiQsX+vbtm6zhKHHD9K5evaqnnnrKZl5CQ4Di/uiLu7vpQQ//MRpXy/jx41M0tCw54oZfxf2hnFAt7u7u1npz586tevXqad26dTpx4oQ2btwowzDSdLiedP/b5aT/3TGSmMKFC2vIkCH64osvdPz4ce3atUurV6/W7t279e6772rt2rWPfEB5aoSHhyc4PW7IZNwdJil5j5PL09PTZl8Piwsv0zIQi+u/xPYZGxur27dv2y2Ei/PgsZcoUSLe/Liw5VF1XLp0SV27dpVhGOrfv7+qV6+up59+Wu7u7oqMjLT7sMPUvn+FChXSDz/8oLNnz+rNN9/U4MGDtWrVKuv24t6jWrVqaebMmXap/bnnntPo0aMVExOjw4cP67ffflNgYKA2bNigO3fuPNZdoACAJwPPkAIAIBnKlSsn6f6zpBKyZMkSffPNNzp9+rQkycfHR5ISvJsqoW3EPSMpbmhanJiYGJ0/f95mWty3gwUHBydYy8yZMzV58mSb5zolV/bs2VW0aFGdO3cu3h1WktS/f3/5+Pjo3Llz1mktW7aUJK1bt06bNm1SgQIFVLNmzRTvOzFnzpzR9u3blSdPHpu7QB72yy+/6IsvvtCtW7dksVjk5eWlzp07a9GiRXr66ad19uxZ6zlJy1Dq4WFP0v07ui5fvqxSpUpZQ4KUvMfJrbFUqVLy8PDQn3/+Ge9uF+l/QV7p0qWTPpBk8vT0VIkSJXT16lWdPXs23vyDBw8qIiIiTfeZkLifyb1798abFxkZqeDgYGXNmtX6MPCEzuf69esVHh6unj17qmvXripTpoz17rgTJ05ISvqOsccRdwx//PGHzXPP4iT2/hUpUkR58+ZV5cqV1a5dO128eFFjxoyxzi9VqpTc3d117NixBIdg7tu3T2PHjtVvv/2WqroXLlyoYcOGyTAMOTs7y8fHR++++66WLl2qrFmzat++fanaLgDgyUIgBQBAMlSqVEmlSpXShg0btHbtWpt5ISEhGjZsmGbOnGm9k6F169ZycnLS5MmTbe5+OHv2rObOnRtv+3HPd3p4CN6cOXPiPci7QYMGypUrl3788UfrMKM4mzdv1pgxY7RgwYJU36HSvHlzRUdHa9SoUYqOjrZOP3z4sLZs2aJixYrZ3JFSq1YtFSxYUMuWLdOhQ4f02muvydnZOVX7ftjly5f1/vvvKyYmRn369Elw2GGcY8eOaeHChZo3b57N9Js3byosLEzZsmWzDpeMe/h3QiFASu3du1ebN2+2vo6KitKIESMUGxurtm3bWqen5D1Obo1ZsmTRq6++qjt37ug///mPzfv1999/a/z48ZLuP+w+LbVs2VKGYWjo0KE2d4jduHFDI0aMsMs+E6pBkiZNmmQT6EVHR2vEiBG6efOmXnnlFWvAlND5jJsX90ylOP/++6+GDx9u3Z69FCpUSDVr1tTff/8db2jdsWPHNGvWLGXJkkXNmjVLdBsff/yxChQooEWLFlkDLFdXV7366qsKDQ3VmDFjbO7Ku3Hjhj7//HPNmDHjkQ98f5R9+/Zp3rx5WrVqlc30a9euKTIyMsFvBAQA4GEM2QMAIBmcnJw0ZswYdenSRR988IFq1KghLy8vXb16VevXr9e9e/c0cuRI6wOBy5Qpo759+2rMmDFq3ry5GjRooJiYGK1bt87mm9DivPbaa5o4caKWL1+ua9euqUyZMjp8+LB+//13VaxY0frtfdL9O1RGjx6t3r1766233lKdOnVUsmRJnT17Vlu2bJGrq6tGjRr1yPDmUXr06KEdO3Zo2bJlOnz4sKpWrapbt27pl19+kcVi0ejRo+Odm9dff11Tp06VlLogYuPGjdZQwDAMRURE6MyZM/rtt98UGRmpbt26qXXr1o/cRqdOnbR69WpNmDBBe/bsUbly5RQREaENGzYoLCxMn3/+ufWcxD14efv27Ro1apTq16+f5HDAxJQoUUK9evVSo0aNlD9/fu3cuVMnTpxQzZo11aFDB+tyKXmPH6zxv//9r37//XcFBAQkuP9PPvlEf/zxh1asWKHDhw+rWrVqunnzpjZv3qxbt26pV69eev7551N1bInp2rWrdu3apZ07d6pp06Z68cUXFR0dra1btyo0NFQtWrRQ8+bN03SfD/Pz89N7772nKVOmqHnz5qpXr55y5cql3bt36/jx4ypTpowGDhxoXT6h97xu3brKlSuXFi1apMuXL8vLy0vXrl3T5s2bFRERIU9PT926dUvR0dEJ/tymhaFDh+rNN9/U1KlT9dtvv6lSpUq6evWqNm7caP0WwIeH/T7I09NTX3zxhQICAvTZZ59p5cqV8vDw0CeffKIDBw5o7ty52rNnj/z9/RUdHa3169frn3/+UatWreJ9Y2hyBQQEaPv27erfv7/WrFmjZ555RmFhYVq3bp0Mw9DHH3+cyrMBAHiSEEgBAJBMFSpU0PLlyzVt2jT9+uuv2rt3r3Lnzq0aNWro7bffjhdovP3223r66ac1Y8YMBQYGKkeOHGrbtq2KFy+uIUOG2CybO3duzZ8/X+PHj9e+ffsUFBQkPz8/zZ8/X6tWrYoXVtSuXVtLlizR9OnTtWfPHm3fvl358+dXo0aN1L17d5UpUybVx+nm5qa5c+dq1qxZWr16tRYsWCB3d3dVr15dvXv3TvAby5o2baqpU6fKz8/P+rDllNi0aZM2bdpkU0OBAgXUsGFDtWnTJsFvUntY3rx59eOPP2r69OnauXOn/vjjD7m6uqp8+fIaOnSo6tWrZ122cOHC6tu3r+bMmaN58+bJw8Mj1YFU06ZNVahQIc2cOVMXLlxQ4cKF9eGHH6pbt242d4ql9D1+44039Mcff2jv3r3666+/9NprryW4/+zZs2vhwoWaNWuW1qxZo0WLFilr1qzy8/NTx44dHznMMbWyZMmiGTNmaP78+VqxYoWWL1+uLFmyqEyZMhowYECafsPio3zwwQeqUKGC5s6dq82bNys6OlpPPfWU+vTpo86dO9s8pDux93zu3LkaP368Dh48qL1796pgwYJ68cUX1aNHD82ePVs//fSTdu7cqdq1a9vlGIoWLaply5Zp6tSp2rRpk+bPn6+cOXOqbt266tatW4Lf1PmwBg0a6KWXXtL69es1fvx4ffrpp8qRI4cWLlyo2bNn2/RFyZIl1a9fP7366qupHrpaqlQpLVy4UNOnT9fvv/+uHTt2KGvWrKpcubLeeecdVa5cOVXbBQA8WSyGPQfGAwCAeJYtW6aBAwfq3Xff1UcffeToctJEYGCg+vXrp+HDhyd5JxMAAADAM6QAAMBjuX37tmbNmqWcOXOadmcMAAAAMjaG7AEAgFTZuHGjpkyZokuXLun69evq37+/PDw8HF0WAAAAMgDukAIAAKlSqFAhXb58WbGxserZs6e6dOni6JIAAACQQfAMKQAAAAAAAJiKO6QAAAAAAABgKgIpAAAAAAAAmOqJfqh5bGysoqOj5eTkJIvF4uhyAAAAAAAA0jXDMBQbGysXFxc5OaX+PqcnOpCKjo5WSEiIo8sAAAAAAADIULy9veXq6prq9Z/oQCouyfP29pazs7ODq0m+mJgYhYSEZLi6kb7RV7AH+gr2QF8hrdFTsAf6CvZAX8EeUtpXccs/zt1R0hMeSMUN03N2ds6QP8wZtW6kb/QV7IG+gj3QV0hr9BTsgb6CPdBXsIeU9tXjPvqIh5oDAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSmUSsYTi6hHRRAwAAAAAASP9cHF0A0oaTxaLJe3frwq2bDtl/0ew5FOBfzSH7BgAAAAAAGQuBVCZy4dZNnQ0Lc3QZAAAAAAAAj8SQPQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCqHB1KXLl1Sjx49VKlSJdWrV09z5syxzjty5Ihat24tX19ftWzZUocOHbJZd/Xq1WrQoIF8fX0VEBCg69evm1w9AAAAAAAAUsrhgdSHH36orFmzatmyZfr00081YcIEbdiwQeHh4erevbuqVKmiZcuWyc/PTz169FB4eLgk6eDBgxo0aJB69eqlRYsW6ebNmxo4cKCDjwYAAAAAAABJcWgg9e+//+rAgQPq2bOnnn76aTVo0EC1atXSrl279Msvv8jNzU39+vXTM888o0GDBilbtmxau3atJGnevHl65ZVX1Lx5c5UpU0ajR4/Wtm3bdP78eUceEgAAAAAAAJLg0EDK3d1dHh4eWrZsme7du6fTp09r//79Klu2rIKDg1W5cmVZLBZJksViUaVKlXTgwAFJUnBwsKpUqWLdVuHChVWkSBEFBwc74lAAAAAAAACQTA4NpNzc3DR48GAtWrRIvr6+euWVV/Tiiy+qdevWCg0NVYECBWyWz5s3ry5fvixJunr16iPnAwAAAAAAIH1ycXQBp06dUt26ddWlSxedOHFCw4YN0wsvvKCIiAi5urraLOvq6qqoqChJ0t27dx85PyViYmJSfwAOEFfvg3U7OzvLkGTIcEhNcXvNaOcS/5NQXwGPi76CPdBXSGv0FOyBvoI90Fewh5T2VVr1n0MDqV27dumnn37Stm3b5O7uLm9vb125ckXffvutihcvHi9cioqKkru7u6T7d1clNN/DwyPFdYSEhKT+IBworm4PDw+VK1dOEXfCdfvWbYfUEpHFTZL0559/KiIiwiE1IG1k1J8HpG/0FeyBvkJao6dgD/QV7IG+gj2Y3VcODaQOHTqkEiVKWEMmSSpXrpymTp2qKlWq6Nq1azbLX7t2zTpMr2DBggnOz58/f4rr8Pb2lrOzcyqOwDFiYmIUEhISr26PbFnleS/SITV5ZMsqSfLy8nLI/vH4Eusr4HHQV7AH+gppjZ6CPdBXsAf6CvaQ0r6KW/5xOTSQKlCggM6dO6eoqCjr8LvTp0+rWLFi8vX11YwZM2QYhiwWiwzD0P79+/Xuu+9Kknx9fRUUFKQWLVpIki5duqRLly7J19c3xXU4OztnyB/mh+u2SLLI4pBa4vaaEc8jbGXUnwekb/QV7IG+Qlqjp2AP9BXsgb6CPZjdVw59qHm9evWUJUsWffbZZzpz5ow2b96sqVOnqkOHDmrUqJFu3rypESNG6OTJkxoxYoQiIiL0yiuvSJLat2+vwMBALVmyRMeOHVO/fv1Up04dFS9e3JGHBAAAAAAAgCQ4NJDKnj275syZo9DQULVq1UojR45Uz5491bZtW3l6emratGnWu6CCg4M1ffp0Zc16f2iYn5+fhg4dqsmTJ6t9+/bKmTOnRo4c6cjDAQAAAAAAQDI4/Fv2nn32Wc2ePTvBeT4+Plq+fHmi67Zo0cI6ZA8AAAAAAAAZg0PvkAIAAAAAAMCTh0AKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqhwdSUVFR+vLLL/X888+revXq+vrrr2UYhiTpyJEjat26tXx9fdWyZUsdOnTIZt3Vq1erQYMG8vX1VUBAgK5fv+6IQwAAAAAAAEAKODyQGj58uH777TfNmjVL48aN0+LFi7Vo0SKFh4ere/fuqlKlipYtWyY/Pz/16NFD4eHhkqSDBw9q0KBB6tWrlxYtWqSbN29q4MCBDj4aAAAAAAAAJMXFkTsPCwvT0qVLNXv2bPn4+EiSunbtquDgYLm4uMjNzU39+vWTxWLRoEGDtH37dq1du1YtWrTQvHnz9Morr6h58+aSpNGjR6tu3bo6f/68ihcv7sCjAgAAAAAAwKM49A6poKAgeXp6yt/f3zqte/fuGjlypIKDg1W5cmVZLBZJksViUaVKlXTgwAFJUnBwsKpUqWJdr3DhwipSpIiCg4NNPQYAAAAAAACkjEPvkDp//ryKFi2qFStWaOrUqbp3755atGihnj17KjQ0VM8++6zN8nnz5tWJEyckSVevXlWBAgXizb98+XKK64iJiUn9QThAXL0P1u3s7CxDkiHDITXF7TWjnUv8T0J9BTwu+gr2QF8hrdFTsAf6CvZAX8EeUtpXadV/Dg2kwsPDde7cOS1cuFAjR45UaGioBg8eLA8PD0VERMjV1dVmeVdXV0VFRUmS7t69+8j5KRESEpL6g3CguLo9PDxUrlw5RdwJ1+1btx1SS0QWN0nSn3/+qYiICIfUgLSRUX8ekL7RV7AH+gppjZ6CPdBXsAf6CvZgdl85NJBycXHR7du3NW7cOBUtWlSSdPHiRS1YsEAlSpSIFy5FRUXJ3d1dkuTm5pbgfA8PjxTX4e3tLWdn51QehfliYmIUEhISr26PbFnleS/SITV5ZMsqSfLy8nLI/vH4Eusr4HHQV7AH+gppjZ6CPdBXsAf6CvaQ0r6KW/5xOTSQyp8/v9zc3KxhlCSVLFlSly5dkr+/v65du2az/LVr16zD9AoWLJjg/Pz586e4Dmdn5wz5w/xw3RZJFlkcUkvcXjPieYStjPrzgPSNvoI90FdIa/QU7IG+gj3QV7AHs/vKoQ819/X1VWRkpM6cOWOddvr0aRUtWlS+vr76448/ZBj3n05kGIb2798vX19f67pBQUHW9S5duqRLly5Z5wMAAAAAACB9cmggVapUKdWpU0cDBw7UsWPH9Ouvv2r69Olq3769GjVqpJs3b2rEiBE6efKkRowYoYiICL3yyiuSpPbt2yswMFBLlizRsWPH1K9fP9WpU0fFixd35CEBAAAAAAAgCQ4NpCRp7Nixeuqpp9S+fXv1799fb775pjp06CBPT09NmzZNQUFBatGihYKDgzV9+nRlzXr/WUV+fn4aOnSoJk+erPbt2ytnzpwaOXKkg48GAAAAAAAASXHoM6QkKXv27Bo9enSC83x8fLR8+fJE123RooVatGhhr9IAAAAAAABgBw6/QwoAAAAAAABPFgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpCKQAAAAAAABgKgIpAAAAAAAAmIpACgAAAAAAAKYikAIAAAAAAICpUhVIrV69WlFRUWldCwAAAAAAAJ4AqQqk+vXrpxo1amjIkCE6ePBgWtcEAAAAAACATCxVgdTmzZvVtWtX7d69W23btlXjxo01a9YshYaGpnV9AAAAAAAAyGRSFUgVKlRIPXv21Nq1azV//nxVqVJFM2bMUN26dfXuu+9q/fr1io6OTutaAQAAAAAAkAm4PO4GKlWqpEqVKql169YaPXq0tm7dqq1btypfvnzq1KmTunbtKmdn57SoFQAAAAAAAJnAYwVSFy5cUGBgoAIDA/XXX3/pqaeeUp8+fVSnTh1t3bpVkydP1smTJzVq1Ki0qhcAAAAAAAAZXKoCqSVLligwMFD79++Xm5ubGjVqpBEjRqhKlSrWZUqXLq0bN25o4cKFBFIAAAAAAACwSlUg9fnnn8vX11dDhgxR48aN5enpmeByXl5eatu27WMVCAAAAAAAgMwlVYHU6tWr9eyzzyomJsb6fKi7d+/q3r17yp49u3W55s2bp0mRAAAAAAAAyDxS9S17Tz/9tL744gu1adPGOm3//v164YUXNGrUKMXGxqZZgQAAAAAAAMhcUhVIffPNN1q5cqWaNm1qnVauXDl9/PHHWrx4sWbOnJlmBQIAAAAAACBzSdWQvVWrVql///5q166ddVquXLnUuXNnubi4aO7cuerevXuaFQkAAAAAAIDMI1V3SN24cUPFixdPcF6pUqV0+fLlxyoKAAAAAAAAmVeqAqlSpUpp3bp1Cc7bvHmzSpQo8VhFAQAAAAAAIPNK1ZC9jh07asCAAQoLC1ODBg2UN29eXb9+XVu2bNGaNWs0cuTItK4TAAAAAAAAmUSqAqnmzZvrzp07mjJlitavX2+dnjt3bn3++edq3rx5WtUHAAAAAACATCZVgZQkvfnmm3rjjTd05swZhYWFKUeOHCpVqpScnFI1ChAAAAAAAABPiFQHUpJksVhUqlSptKoFAAAAAAAAT4BUBVLXr1/XiBEjtHXrVkVERMgwDJv5FotFR44cSZMCAQAAAAAAkLmkKpAaOnSotmzZoiZNmqhQoUIM0wMAAAAAAECypSqQ2r59uz799FO1bds2resBAAAAAABAJpeqW5uyZMmi4sWLp3UtAAAAAAAAeAKkKpBq2LChVq9enda1AAAAAAAA4AmQqiF75cqV04QJE3T+/Hn5+vrK3d3dZr7FYlFAQECaFAgAAAAAAIDMJdUPNZekffv2ad++ffHmE0gBAAAAAAAgMakKpI4dO5bWdQAAAAAAAOAJkapnSD3o1q1bOnXqlKKiohQTE5MWNQEAAAAAACATS3UgtWfPHrVu3Vr+/v5q1qyZTpw4ob59++qrr75Ky/oAAAAAAACQyaQqkNq1a5e6desmd3d3ffzxxzIMQ5JUpkwZzZ07V7Nnz07TIgEAAAAAAJB5pCqQmjBhgurXr68ffvhBnTp1sgZS7777rt5++20tWbIkTYsEAAAAAABA5pGqQOro0aNq2bKlpPvfqPegGjVq6MKFC49fGQAAAAAAADKlVAVS2bNnV2hoaILzLl26pOzZsz9WUQAAAAAAAMi8UhVI1a9fX+PHj1dISIh1msVi0eXLlzV16lTVqVMnreoDAAAAAABAJuOSmpX69u2r4OBgtWnTRvny5ZMk9enTR5cvX1bhwoXVp0+fNC0SAAAAAAAAmUeqAqmcOXNqyZIlWrFihXbv3q2wsDBlz55dHTp0UIsWLeTh4ZHWdQIAAAAAACCTSFUgJUmurq5q06aN2rRpk5b1AAAAAAAAIJNLVSC1YsWKJJdp3rx5ajYNAAAAAACATC5VgdSAAQMSnG6xWOTs7CxnZ2cCKQAAAAAAACQoVYHUpk2b4k0LDw/X77//rhkzZmjy5MmPXRgAAAAAAAAyp1QFUkWLFk1w+nPPPad79+5p2LBh+vHHHx+rMAAAAAAAAGROTmm9QS8vLx0+fDitNwsAAAAAAIBMIk0DqaioKP3000/KmzdvWm4WAAAAAAAAmUiqhuzVq1dPFovFZlpsbKxu3LihyMhI9e/fP02KAwAAAAAAQOaTqkDK398/XiAlSZ6enqpbt66qV6/+2IUBAAAAAAAgc0pVIPXVV1+ldR0AAAAAAAB4QqQqkLp48WKKli9SpEhqdgMAAAAAAIBMKM2eIfUoR48eTc1uAAAAAAAAkAmlKpCaMGGCvvjiC5UvX16vvvqqChYsqBs3bmjz5s1as2aNevbsqaJFi6Z1rQAAAAAAAMgEUhVIBQYGqm7duvGeJdW4cWPlzZtX+/fvV69evdKkQAAAAAAAAGQuTqlZadeuXWratGmC81588UUFBQU9VlEAAAAAAADIvFIVSOXOnVvBwcEJztu1a5cKFiz4WEUBAAAAAAAg80rVkL1WrVrp22+/VUREhOrVq6c8efLo2rVrWrt2rRYsWKDPP/88resEAAAAAABAJpGqQOq9997TrVu3NGfOHM2aNUuSZBiGPDw89NFHH6ldu3ZpWiQAAAAAAAAyj1QFUhaLRQMGDNB7772nAwcO6N9//1Xu3LlVsWJFeXp6pnWNAAAAAAAAyERSFUjF8fT0VIECBSRJFStWVHR0dJoUBQAAAAAAgMwr1YFUYGCgxo0bp9DQUFksFi1ZskQTJ05UlixZNG7cOLm6uqZlnQAAAAAAAMgkUvUte7/88ov69++vatWq6euvv1ZsbKwkqWHDhtq2bZumTJmSpkUCAAAAAAAg80jVHVJTp05Vu3btNGTIEMXExFint2zZUtevX9fixYv14YcfplWNAAAAAAAAyERSdYfUmTNn1LBhwwTn+fr66sqVK49VFAAAAAAAADKvVAVSefPm1alTpxKcd+rUKeXNmzdVxXTv3l0DBgywvj5y5Ihat24tX19ftWzZUocOHbJZfvXq1WrQoIF8fX0VEBCg69evp2q/AAAAAAAAME+qAqnGjRvrm2++0dq1axUVFSVJslgsOnTokKZMmaJGjRqleJs///yztm3bZn0dHh6u7t27q0qVKlq2bJn8/PzUo0cPhYeHS5IOHjyoQYMGqVevXlq0aJFu3rypgQMHpuZwAAAAAAAAYKJUPUPqww8/1PHjx/Xhhx/Kyel+ptWhQweFh4erSpUq+uCDD1K0vbCwMI0ePVre3t7Wab/88ovc3NzUr18/WSwWDRo0SNu3b9fatWvVokULzZs3T6+88oqaN28uSRo9erTq1q2r8+fPq3jx4qk5LAAAAAAAAJggVYGUq6urZs6cqZ07d2r37t0KCwtT9uzZ5e/vr9q1a8tisaRoe6NGjdJrr72mq1evWqcFBwercuXK1m1ZLBZVqlRJBw4cUIsWLRQcHKx33nnHunzhwoVVpEgRBQcHE0gBAAAAAACkY6kKpLp166a3335bNWrUUI0aNR6rgF27dun333/XqlWrNGTIEOv00NBQPfvsszbL5s2bVydOnJAkXb16VQUKFIg3//Lly49VDwAAAAAAAOwrVYHU/v37U3wXVEIiIyP1xRdfaPDgwXJ3d7eZFxERIVdXV5tprq6u1mdW3b1795HzUyImJibF6zhSXL0P1u3s7CxDkiHDITXF7TWjnUv8T0J9BTwu+gr2QF8hrdFTsAf6CvZAX8EeUtpXadV/qQqkatWqpZUrV6py5crKkiVLqnc+adIkVahQQbVq1Yo3z83NLV64FBUVZQ2uEpvv4eGR4jpCQkJSvE56EFe3h4eHypUrp4g74bp967ZDaonI4iZJ+vPPPxUREeGQGpA2MurPA9I3+gr2QF8hrdFTsAf6CvZAX8EezO6rVAVSbm5uWrlypdasWaNnnnlGWbNmtZlvsVj0/fffJ7mdn3/+WdeuXZOfn58kWQOmdevWqWnTprp27ZrN8teuXbMO0ytYsGCC8/Pnz5/i4/H29pazs3OK13OUmJgYhYSExKvbI1tWed6LdEhNHtnu94CXl5dD9o/Hl1hfAY+DvoI90FdIa/QU7IG+gj3QV7CHlPZV3PKPK1WB1OXLl60hkiQZhu0wsYdfJ+aHH35QdHS09fXYsWMlSR9//LH27dunGTNmyDAMWSwWGYah/fv3691335Uk+fr6KigoSC1atJAkXbp0SZcuXZKvr2+Kj8fZ2TlD/jA/XLdFkkWPP5QyNeL2mhHPI2xl1J8HpG/0FeyBvkJao6dgD/QV7IG+gj2Y3VfJDqTWr1+vatWqKUeOHPrhhx/SZOdFixa1eZ0tWzZJUokSJZQ3b16NGzdOI0aMULt27bRw4UJFRETolVdekSS1b99eHTp0UMWKFeXt7a0RI0aoTp06fMMeAAAAAABAOueU3AU/+OADnT171mbajBkz9M8//6R1TZIkT09PTZs2zXoXVHBwsKZPn24dHujn56ehQ4dq8uTJat++vXLmzKmRI0fapRYAAAAAAACknWTfIfXwMLyYmBh9/fXXql69uvLmzZsmxXz11Vc2r318fLR8+fJEl2/RooV1yB4cK6ebu2INQ05p8O2Ljyu91AEAAAAAABKWqmdIxUnus6KQ+WVzzSIni0WT9+7WhVs3HVZH0ew5FOBfzWH7BwAAAAAASXusQAp42IVbN3U2LMzRZQAAAAAAgHQs2c+QAgAAAAAAANLCYwdSFp7VAwAAAAAAgBRI0ZC9gIAAubq62kx79913lSVLFptpFotFGzdufPzqAAAAAAAAkOkkO5B6/fXX7VkHAAAAAAAAnhDJDqRGjhxpzzoAAAAAAADwhOCh5gAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUoCdxBqGo0tIFzUAAAAAAPAwF0cXAGRWThaLJu/drQu3bjpk/0Wz51CAfzWH7BsAAAAAgEchkALs6MKtmzobFuboMgAAAAAASFcYsgcAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSyFRyurkr1jAcXQYAAAAAAHgEF0cXAKSlbK5Z5GSxaPLe3bpw66bD6vAtWEhtK/g4bP8AAAAAAKRnBFLIlC7cuqmzYWEO23+R7Nkdtm8AAAAAANI7huwBAAAAAADAVARSAKw8PDwcXQIAAAAA4AnAkD0gk4p7wLuTxZKs5Z2dnVWuXDm71JKSOgAAAAAAmR+BFJBJpfQB74akiDvh8siWVWkZHRXNnkMB/tXScIsAAAAAgIyOQArI5JL7gHdDhm7fui3Pe5GypGkkBQAAAACALZ4hBQAAAAAAAFMRSAEAAAAAAMBUBFIAAAAAAAAwFYEUAAAAAAAATEUgBQAAAAAAAFMRSAEAAAAAAMBUBFIA7Cqnm7tiDcPRZaSLGgAAAAAA97k4ugAAmVs21yxyslg0ee9uXbh10yE1FM2eQwH+1RyybwAAAABAfARSAExx4dZNnQ0Lc3QZAAAAAIB0gCF7AAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTOTyQunLlit5//335+/urVq1aGjlypCIjIyVJ58+fV+fOnVWxYkU1btxYO3bssFn3t99+U9OmTeXr66uOHTvq/PnzjjgEAAAAAAAApIBDAynDMPT+++8rIiJC8+fP1/jx47VlyxZNmDBBhmEoICBA+fLl09KlS/Xaa6+pV69eunjxoiTp4sWLCggIUIsWLfTTTz8pT548eu+992QYhiMPCQAAAAAAAElwceTOT58+rQMHDmjnzp3Kly+fJOn999/XqFGj9OKLL+r8+fNauHChsmbNqmeeeUa7du3S0qVL1bt3by1ZskQVKlRQ165dJUkjR45UjRo1tHfvXlWtWtWRhwUAAAAAAIBHcOgdUvnz59fMmTOtYVSc27dvKzg4WOXKlVPWrFmt0ytXrqwDBw5IkoKDg1WlShXrPA8PD5UvX946HwAAAAAAAOmTQ++QypEjh2rVqmV9HRsbq3nz5qlatWoKDQ1VgQIFbJbPmzevLl++LElJzk+JmJiYVFTvOHH1Pli3s7OzDEmGHDNk0Xjgfx1VA3U8Xg1xw10Nw5AsjqvDHuL2mtF+1jODhD6vgMdFXyGt0VOwB/oK9kBfwR5S2ldp1X8ODaQeNmbMGB05ckQ//fST5syZI1dXV5v5rq6uioqKkiRFREQ8cn5KhISEpL5oB4qr28PDQ+XKlVPEnXDdvnXbIbVERtyVJN29e9dhNVBH2tRw5/addFFHWorI4iZJ+vPPPxUREeGQGp50GfVzFukbfYW0Rk/BHugr2AN9BXswu6/STSA1ZswYff/99xo/frxKly4tNzc3hYWF2SwTFRUld3d3SZKbm1u88CkqKko5cuRI8b69vb3l7Oyc6trNFhMTo5CQkHh1e2TLKs97kQ6pyc3j/vvi7u4uz+yeDqmBOh6vBsMwdOf2HWXzzCaLJe1ukUoP58Ij2/2hv15eXg7Z/5Mssc8r4HHQV0hr9BTsgb6CPdBXsIeU9lXc8o8rXQRSw4YN04IFCzRmzBi9/PLLkqSCBQvq5MmTNstdu3bNOkyvYMGCunbtWrz5ZcuWTfH+nZ2dM+QP88N1WyRZ0nKsVQpYHvhfR9VAHY9Zw/8vYrFY0rTm9HQuMuLPeWaRUT9nkb7RV0hr9BTsgb6CPdBXsAez+8qhDzWXpEmTJmnhwoX6+uuv1aRJE+t0X19fHT58WHfv3rVOCwoKkq+vr3V+UFCQdV5ERISOHDlinQ8AAAAAAID0yaGB1KlTpzRlyhS98847qly5skJDQ63//P39VbhwYQ0cOFAnTpzQ9OnTdfDgQbVq1UqS1LJlS+3fv1/Tp0/XiRMnNHDgQBUrVkxVq1Z15CEBAAAAAAAgCQ4NpDZt2qSYmBh9++23qlmzps0/Z2dnTZkyRaGhoWrRooVWrlypyZMnq0iRIpKkYsWKaeLEiVq6dKlatWqlsLAwTZ48OU2ffQMAAAAAAIC059BnSHXv3l3du3dPdH6JEiU0b968ROfXrl1btWvXtkdpAAAAAAAAsBOHP0MKAAAAAAAATxYCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAAAAAAAAYCoCKQAAAAAAAJiKQAoAAAAAAACmIpACAAAAAACAqQikAGR6Od3cFWsYji5DktJNHQAAAADgSC6OLgAA7C2baxY5WSyavHe3Lty66bA6imbPoQD/ag7bPwAAAACkFwRSAJ4YF27d1NmwMEeXAQAAAABPPIbsAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgAAAAAAADAVgRQAAAAAAABMRSAFAAAAAAAAUxFIAQAAAAAAwFQEUgBgkpxu7oo1DEeXkS5qAAAAAPBkc3F0AQDwpMjmmkVOFosm792tC7duOqSGotlzKMC/mkP2DQAAAABxCKQAwGQXbt3U2bAwR5cBAAAAAA7DkD0AAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgDYlYeHh6NLAAAAAJDOuDi6AACAeXK6uSvWMORksZiyP2dnZ5UrVy7BeWbWAQAAACB9IZACgCdINtcscrJYNHnvbl24ddPu+zMkRdwJl0e2rHoweiqaPYcC/KvZff8AAAAA0icCKQB4Al24dVNnw8Lsvh9Dhm7fui3Pe5GyiLuhAAAAANzHM6QAAAAAAABgKgIpAIDp4p5l5WjpoQYAAADgScSQPQCA6cx+llVCeI4VAAAA4DgEUgAAhzHrWVYAAAAA0heG7AEAAAAAAMBUBFIAAAAAAAAwFYEUAAAAAAAATEUgBQAAAAAAAFMRSAEAAAAAAMBUBFIAgCdSTjd3xRqGo8uQJOpIZzUAAADA/lwcXQAAAI6QzTWLnCwWTd67Wxdu3XRYHUWz51CAfzWH7f9Bjj4f6elcAAAAwL4IpAAAT7QLt27qbFiYo8tINzgfAAAAMAND9gAAAAAAAGAqAikAABwoPT3LKiPx8PBwdAkAAAB4DBl+yF5kZKS+/PJLrV+/Xu7u7uratau6du3q6LIAAEiW9PIsK9+ChdS2go/D9i/9L5xzslgeuZyzs7PKlStn11qSUwcAAABSL8MHUqNHj9ahQ4f0/fff6+LFi+rfv7+KFCmiRo0aObo0AACSzdHPbiqSPbvD9h0nueGcISniTrg8smWVPSIjr7z51MHXzw5bThlCMXNx1x0AAObK0IFUeHi4lixZohkzZqh8+fIqX768Tpw4ofnz5xNIAQCQQSUVzhkydPvWbXnei5TFDpFUkezZHX7XWnoJxaT0EYzZu4bk3nWXHs5FeqkjPdQAAMjYMnQgdezYMUVHR8vP73+/sFWuXFlTp05VbGysnJx4RBYAAEgdR961lh5CMel/QzkdWYcZNSTnrrv0cC7SSx0EpumvBthKD++JPWvIiHd0Zvb3JCPWkR5k6EAqNDRUuXPnlqurq3Vavnz5FBkZqbCwMOXJk8eB1QEAADye9DKU09HhnL1rSM5dd+nhXKSXOghM/6do9hwK8K+W6Hwzg4P08EdueqhBksP7056hbUqfo8h7cl96+LyQkv7MeNJk6EAqIiLCJoySZH0dFRWV5PrG/3+rUVRUlJydndO+QDuJiYmRZFu3s7OzimfPIWe7PE0jaQU8siomJsahNVDH49VgGIYiXd3l5uEhSxpetDLiucjMdZhdQ2J99SSeC+pIuxrs9XmV0jrsKT3UkF7qMKOG5PRUejgX6aWOuBqcJIeeC4thOLyO7Fmy6F50dKJ/8Ht5eUn63+/v9rb8z6P6JyLClH09rFj2HKpf6hkZ6SD8kOTQvvBwdpYRG2uX98MwDN2LjFQWN7ckr4G8J/+THj4vpPvnICYmxrTPhORKKGNIzvLGY35TtMV43C040Jo1azR8+HDt3LnTOu3UqVNq3Lix9uzZo1y5cj1y/aioKIWEhNi5SgAAAAAAgMzF29s73k1CKZGh75AqWLCgbty4oejoaLm43D+U0NBQubu7K0eOHEmu7+LiIm9vbzk5Odnl/2EFAAAAAADITAzDUGxsrDWHSa0MHUiVLVtWLi4uOnDggKpUqSJJCgoKsoZMSXFycnqsNA8AAAAAAAApl6G/hs7Dw0PNmzfXkCFDdPDgQW3cuFHfffedOnbs6OjSAAAAAAAAkIgM/Qwp6f6DzYcMGaL169fL09NT3bp1U+fOnR1dFgAAAAAAABKR4QMpAAAAAAAAZCwZesgeAAAAAAAAMh4CKQAAAAAAAJiKQAoAAAAAAACmIpDKYCIjI/Xpp5+qSpUqqlmzpr777jtHl4R06MqVK3r//ffl7++vWrVqaeTIkYqMjJQkDR8+XF5eXjb/5s2bZ1139erVatCggXx9fRUQEKDr169b5xmGobFjx6patWry9/fX6NGjFRsba/rxwTE2bNgQr3fef/99SdKRI0fUunVr+fr6qmXLljp06JDNuvQVErJs2bJ4PeXl5aUyZcpIknr27Blv3pYtW6zrz5kzR7Vq1ZKfn58+/fRTRUREWOdxvXwyRUVFqWnTptqzZ4912vnz59W5c2dVrFhRjRs31o4dO2zW+e2339S0aVP5+vqqY8eOOn/+vM18+uzJllBPHThwQO3atZOfn59efvllLVmyxGadV199Nd5n1/HjxyUlfc27ceOGevfuLT8/P9WrV0+BgYHmHChMlVBf2fN3dPrqyfBwXw0YMCDB37M6duxoXadKlSrx5t+5c0dS0te4pK6vyWIgQxk6dKjRrFkz49ChQ8b69esNPz8/Y82aNY4uC+lIbGys0aZNG+Ptt982jh8/buzbt89o2LCh8dVXXxmGYRidO3c2pk2bZly9etX6Lzw83DAMwwgODjZ8fHyM5cuXG0ePHjXeeusto3v37tZtz5o1y6hdu7axb98+Y9euXUbNmjWNmTNnOuQ4Yb4pU6YYPXr0sOmdf//917hz545Ro0YN46uvvjJOnjxpDBs2zKhevbpx584dwzDoKyQuIiLCpp8uXrxoNGzY0BgxYoRhGIbRsGFDIzAw0GaZyMhIwzAMY+3atUblypWNzZs3G8HBwUbjxo2NL7/80rptrpdPnrt37xoBAQFG6dKljd27dxuGcf+a2KxZM6Nv377GyZMnjalTpxq+vr7GhQsXDMMwjAsXLhgVK1Y0Zs2aZRw/ftz44IMPjKZNmxqxsbGGYdBnT7qEeurq1atGlSpVjHHjxhlnzpwxVq9ebXh7extbtmwxDMMwoqOjDW9vb2Pv3r02n1337t0zDCPpa16PHj2MTp06GX/++aexePFio0KFCkZwcLDpxw77SaivDMO+v6PTV5lfQn118+ZNm376448/jAoVKhgbNmwwDMMwLl++bJQuXdr466+/bJaLuwY+6hqX1PU1uQikMpA7d+4Y3t7eNh9ckydPNt566y0HVoX05uTJk0bp0qWN0NBQ67RVq1YZNWvWNAzDMGrVqmX8+uuvCa77ySefGP3797e+vnjxouHl5WX89ddfhmEYRu3atY2lS5da569YscKoW7euPQ4D6VDfvn2NcePGxZu+ZMkSo169etaLV2xsrNGwYUNrr9BXSK6pU6caDRo0MCIjI43IyEijbNmyxunTpxNc9o033jC++eYb6+t9+/YZPj4+Rnh4ONfLJ9CJEyeMV1991WjWrJnNL+O//fabUbFiRWtAbhiG0alTJ2vvTJgwwaYvwsPDDT8/P+v69NmTK7Ge+vHHH41GjRrZLPv5558bffr0MQzDMM6ePWuUKVPGuHv3boLbfdQ179y5c0bp0qWN8+fPW+d/+umnNtdQZGyJ9ZVh2O93dPoq83tUXz2oa9euxscff2x9vXPnTqNGjRoJLpvUNS6p62tyMWQvAzl27Jiio6Pl5+dnnVa5cmUFBwczvAVW+fPn18yZM5UvXz6b6bdv39bt27d15coVPf300wmuGxwcrCpVqlhfFy5cWEWKFFFwcLCuXLmiS5cu6fnnn7fOr1y5si5cuKCrV6/a5ViQvpw6dSrB3gkODlblypVlsVgkSRaLRZUqVdKBAwes8+krJCUsLEwzZsxQ37595erqqtOnT8tisah48eLxlo2JiVFISIhNX1WsWFH37t3TsWPHuF4+gfbu3auqVatq0aJFNtODg4NVrlw5Zc2a1TqtcuXKiX4+eXh4qHz58jpw4AB99oRLrKfiHoXwsNu3b0uSTp48qcKFC8vNzS3eMkld84KDg1W4cGEVK1bMZv4ff/yRVocFB0usr+z5Ozp9lfkl1lcP2rVrl/bt26c+ffpYp508eVIlS5ZMcPmkrnFJXV+TyyVFS8OhQkNDlTt3brm6ulqn5cuXT5GRkQoLC1OePHkcWB3Sixw5cqhWrVrW17GxsZo3b56qVaumU6dOyWKxaOrUqdq+fbty5cqlLl266PXXX5ckXb16VQUKFLDZXt68eXX58mWFhoZKks38uNDr8uXL8dZD5mIYhs6cOaMdO3Zo2rRpiomJUaNGjfT+++8rNDRUzz77rM3yefPm1YkTJyTRV0ieBQsWqECBAmrUqJEk6fTp0/L09FS/fv20d+9eFSpUSL1791bt2rV18+ZNRUZG2vSHi4uLcuXKpcuXL8vJyYnr5RPmjTfeSHB6aGhoop8/Sc2nz55sifVUsWLFbP6w/+eff/Tzzz+rd+/eku7/nzdZsmRRjx49dOjQIZUsWVL9+vWTj49Pkte8xPrxypUraXpscJzE+sqev6PTV5lfYn31oOnTp+v1119X4cKFrdNOnTqliIgIdejQQWfOnFHZsmX16aefqmTJkklmD0ldX5OLO6QykIiICJuGkGR9HRUV5YiSkAGMGTNGR44c0UcffWS946BUqVKaPn26Wrdurc8//1wbNmyQJN29ezfBHouKitLdu3etrx+cJ9F/T4KLFy9aP4MmTJig/v37a9WqVRo9enSin01xfUFfISmGYWjJkiV66623rNNOnz6tu3fvqmbNmpo5c6Zq166tnj17KiQkJMG+iXsdFRXF9RJWSX0+PWo+fYak3L17V71791a+fPnUtm1bSdKZM2f077//qnXr1po+fbqeeeYZderUSZcuXUrympdUvyLzsufv6PQVzp8/r927d6tDhw4200+fPq1///1XPXv21JQpU+Tu7q7OnTvr9u3bSV7j0qqvuEMqA3Fzc4v3Bse9dnd3d0RJSOfGjBmj77//XuPHj1fp0qX13HPPqW7dusqVK5ckqUyZMjp79qwWLFighg0bJtpjHh4eNh9Acbehxy3r4eFh3kHBIYoWLao9e/YoZ86cslgsKlu2rGJjY/XJJ5/I398/wb6J+1yir5CUkJAQXblyRU2aNLFOe++999ShQwflzJlT0v3Pq8OHD2vx4sX66KOPJMX/oz+ur2JiYrheQtL9z5+wsDCbacn5fMqRI0e8z6QH59NnuHPnjt577z2dPXtWP/74o/WaNWzYMN29e1eenp6SpCFDhmj//v0KDAxU9erVJSV+zUusH+mnzK958+Z2+x2dvsK6detUtmzZeCMaZs2apXv37ilbtmySpLFjx6p27drasmVLktlDUtfX5OIOqQykYMGCunHjhqKjo63TQkND5e7urhw5cjiwMqRHw4YN0+zZszVmzBi9/PLLku4/2yfuQhenVKlS1lt2CxYsqGvXrtnMv3btmvLnz6+CBQtKkvW24Af/O3/+/PY6DKQjuXLlsj4nSpKeeeYZRUZGKn/+/An2TdxtvPQVkvLrr7+qSpUq1vBJkpycnGxeS//7vMqVK5fc3Nxs+io6OlphYWHWvuJ6CSnxz5/kfD7RZ0jM7du31a1bN504cULff/+9zXN/XFxcrGGUJOtdL1euXEnymveofkTmZs/f0ekr/Prrr6pfv3686a6urtYwSrr/f9IUK1bM+nn1qGtcUtfX5CKQykDKli0rFxcXmweFBQUFydvbW05OvJX4n0mTJmnhwoX6+uuvbe44+O9//6vOnTvbLHvs2DGVKlVKkuTr66ugoCDrvEuXLunSpUvy9fVVwYIFVaRIEZv5QUFBKlKkCM/5eQL8+uuvqlq1qiIiIqzTjh49qly5clkfjGkYhqT7w6/2798vX19fSfQVknbw4EFVqlTJZtqAAQM0cOBAm2lxn1dOTk7y9va26ZsDBw7IxcVFZcqU4XoJK19fXx0+fNg6pEW63wuJfT5FREToyJEj8vX1pc+QoNjYWPXq1Ut///23fvjhBz333HM28zt06KBJkybZLP/nn3+qVKlSSV7zKlasqAsXLtg8gyUoKEgVK1a0+3HBsez5Ozp99WQzDEMhISHxfs8yDEMNGjTQsmXLrNPCw8N17tw5lSpVKslrXFLX15QUiAzk888/N5o0aWIEBwcbGzZsMCpVqmSsW7fO0WUhHTl58qRRtmxZY/z48cbVq1dt/gUHBxvlypUzZs6caZw7d86YP3++UaFCBWP//v2GYRjG/v37jfLlyxuLFy82jh49arz11ltGjx49rNueNm2aUbNmTWP37t3G7t27jZo1axrfffedow4VJrp165ZRq1Yto0+fPsapU6eMrVu3GjVr1jSmT59u3Lp1y6hWrZoxbNgw48SJE8awYcOMGjVqWL8Glr5CUurWrWusXr3aZtq6deuM8uXLG8uXLzfOnj1rTJw40fDx8bF+bfXq1auNSpUqGRs2bDCCg4ONJk2aGMOGDbOuz/XyyfXgV15HR0cbjRs3Nj788EPj+PHjxrRp04yKFSsaFy5cMAzDMM6fP294e3sb06ZNM44fP2588MEHRrNmzYzY2FjDMOgz3PdgTy1atMgoU6aMsWXLFpvfsW7cuGEYhmF89913RuXKlY2NGzcap06dMr744gujevXqxq1btwzDSPqa17VrV+Ott94yjh49aixevNjw9vY2goODTT9m2N+DfWXv39HpqyfHg31lGPevc6VLlzauXr0ab9lhw4YZderUMXbv3m0cP37cCAgIMJo2bWpER0cbhvHoa1xS19fkIpDKYMLDw41+/foZFStWNGrWrGnMnj3b0SUhnZk2bZpRunTpBP8ZhmFs2LDBaNasmeHt7W00atQo3i/OS5cuNWrXrm1UrFjRCAgIMK5fv26dFx0dbfznP/8xqlSpYlStWtUYM2aM9Zd2ZH7Hjx83OnfubFSsWNGoUaOGMXHiROv7HxwcbDRv3tzw9vY2WrVqZRw+fNhmXfoKj+Lt7W1s37493vTFixcbL730klGhQgXj9ddfN/bu3Wszf9q0acYLL7xgVK5c2Rg4cKBx9+5d6zyul0+uh38ZP3v2rPHmm28aFSpUMJo0aWLs3LnTZvmtW7caL730kuHj42N06tTJ+Ouvv2zm02d4sKe6du2a4O9Yb731lmEYhhEbG2t8++23Rp06dYwKFSoYb775pvHnn39at5XUNe/atWtGjx49DG9vb6NevXrGqlWrzD1YmObhzyp7/o5OXz05Hu6rAwcOGKVLlzYiIyPjLXv37l1j5MiRRo0aNQxfX1+jR48exsWLF63zk7rGJXV9TQ6LYfz/GAsAAAAAAADABAxwBwAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAAAAAAAApiKQAgAAAAAAgKkIpAAAAAAAAGAqAikAAAAAAACYikAKAABkeIZhOLoEAAAApACBFAAAMN2AAQPk5eX1yH8dOnRIcjs3b95Uv3799Pvvv6d4//Xq1Utw3p49e5KszcvLS3///XeS+/Hy8tLEiRNTVJsZLl26pLJly+rLL79MdJlDhw7Jy8tLP/30U5Lbiztne/bsScsyAQBAJubi6AIAAMCT57333lO7du2sr6dMmaIjR45o0qRJ1mmenp5Jbufo0aMKDAxUy5Yt06y28uXLa9GiRdbXhw8f1tChQzV48GCVL1/eOr1AgQJptk+zFS5cWNWrV9eaNWs0aNAgubjE/5VwxYoVypYtmxo3buyACgEAQGZHIAUAAEz31FNP6amnnrK+zpMnj1xdXVWxYkXHFfX/PD09beqIjIyUJD377LPpor600rJlS+3YsUM7duxQnTp1bObdu3dPq1evVuPGjZU1a1bHFAgAADI1huwBAIB0a+fOnXrjjTdUuXJlVa1aVX379tWlS5ck3R8m1rFjR0lSx44drUP8YmJiNH36dDVt2lQ+Pj6qWLGi2rVrp927d6dpbVevXtXAgQNVu3Zt+fj4qFWrVtq0adMj1/nmm29UtmxZLV++3Drt999/11tvvSVfX1/5+/urf//+un79unX+smXLVK5cOQUHB6tt27by9vZW3bp1NWvWLJttr169Wq+++qp8fHxUrVo1ffzxx7py5UqitTRo0EC5cuXSqlWr4s3btm2bbty4oVatWkmS9u3bp27duun5559XhQoVVK9ePU2cOFGxsbEJbjuhIZF///23vLy8tGzZMuu0sLAwDR48WNWrV5e3t7fatGmjXbt22ay3c+dOtWnTRn5+fnr++efVs2dPnTp1KtHjAgAAGQOBFAAASJdWrFihrl27qnDhwvr66681cOBA/fHHH2rbtq3++ecflS9fXoMHD5YkDR48WF988YUkaezYsZoyZYratm2rmTNnatiwYQoLC9MHH3ygiIiINKnt2rVratWqlX7//Xd99NFHmjhxoooWLaqAgACtXLkywXVmzZqlKVOmaNiwYXr99dcl3Q96OnfuLHd3d02YMEGffvqp9u7dq44dO+ru3bvWdWNjY/Xhhx+qcePGmj59uipVqqTRo0fr119/lSQFBQWpX79+eumllzRjxgwNHDhQu3fvVt++fRM9BldXVzVr1kybNm3SnTt3bOatWLFCzz33nCpWrKhjx46pc+fOypUrl8aPH69vv/1WVapU0aRJk7RmzZpUn8PIyEh16tRJmzZt0kcffaRJkyapUKFCevvtt62h1Pnz5/Xee++pQoUK+vbbbzVixAidOXNG3bt3TzQMAwAAGQND9gAAQLoTGxursWPHqmbNmho3bpx1eqVKldS4cWPNmjVL/fr107PPPivp/nC6uP++evWqPvroI5uHoru5ual37976888/02TY3ezZs3X9+nWtW7dORYsWlSTVrl1bnTt31ujRo9W0aVM5Of3v//dbsGCBxowZo6FDh1rvOpKkcePGqWTJkpo2bZqcnZ0lSb6+vmrSpImWLl2qN998U9L9bxF877331Lp1a0lS5cqVtWHDBm3dulW1atVSUFCQ3N3d1b17d7m6ukqScuXKpZCQEBmGIYvFkuBxtGrVSj/88IM2btyo1157TZJ048YNbd26VR9//LEk6dixY6pevbrGjBljPaYaNWpo8+bN2rNnj5o0aZKqcxgYGKhjx45p8eLF8vX1lSS9+OKL6tChg8aOHaulS5fq4MGDunv3rnr06KGCBQtKkgoVKqRNmzYpPDw8Wc8ZAwAA6ROBFAAASHfOnDmj0NDQeHf4PPXUU/Lz89PevXsTXTcuwLp+/bpOnz6tc+fOacuWLZKkqKioNKlv79698vPzs4ZRcV599VUNHDhQp0+ftgZkW7Zs0ZEjR1SlShW1adPGumxERISCg4PVrVs3GYah6OhoSVLx4sX1zDPPaOfOndZASpL8/Pys/+3q6qo8efIoPDxckvT8889r/Pjxatq0qV5++WXVrl1bNWvWVO3atR95HGXKlFH58uW1atUqayD1888/W49Fkpo3b67mzZsrMjJSZ86c0blz53T06FHFxMTo3r17qTp/krRr1y7lz59f5cuXtx67JNWtW1ejR4/Wv//+K19fX7m5ualVq1Zq1KiRXnzxRVWtWlU+Pj6p3i8AAEgfCKQAAEC6ExYWJknKly9fvHn58uXTkSNHEl03JCREX375pUJCQuTh4aFnn31WRYoUkXT/TqO08O+//6p48eIJ1iZJN2/etE47fPiw6tSpo61bt2rz5s3WZyvdvHlTsbGxmjFjhmbMmBFvW25ubjav3d3dbV47OTlZj8fPz0/Tp0/XnDlzNHv2bE2fPl358uXTu+++a3OnWEJatmyp//znP/rnn3+UN29erVixQvXr11eePHkkSXfv3tWwYcMUGBio6OhoFStWTH5+fnJxcXms8xkWFqbQ0FCbby58UGhoqJ599lnNmzdP06dP108//aS5c+cqR44ceuONN/Thhx8meucXAABI/wikAABAupMrVy5J95/V9LDQ0FDlzp07wfVu376tt99+W15eXvr5559VqlQpOTk5adu2bVq3bl2a1ZczZ06FhoYmWJskm/ratm2rL7/8Uu3bt9eXX34pf39/eXp6Klu2bLJYLOrcuXOCw948PDxSVFOtWrVUq1YtRUREaPfu3Zo7d66GDx8uX1/fR95R1KxZM40aNUpr1qzRCy+8oJCQEH3wwQfW+SNGjNC6des0YcIEVa9e3fqtey+88EKi27RYLIqJibGZFnc3V5zs2bPr6aef1tixYxPcRrFixSRJPj4+mjRpkqKiohQUFKRFixZp6tSpKlOmjF555ZVHnxQAAJBu8VBzAACQ7pQsWVL58+fX6tWrbaafP39eBw4cUKVKlSTJ+tylOKdPn1ZYWJg6duyoZ5991vrMo+3bt0tSmj0I+/nnn9cff/yhCxcu2ExfuXKl8ufPrxIlSlin5c+fXxaLRUOGDNG1a9esQwo9PT1Vrlw5nT59Wt7e3tZ/zz33nCZOnKg9e/Yku55Ro0apZcuWMgxDHh4eqlu3rvr37y9Junjx4iPXzZEjhxo2bKh169ZpzZo1KlKkiGrUqGGdHxQUpKpVq6pBgwbWMOrQoUO6fv16ouczW7ZsunHjhiIjI2228yB/f39dunRJefPmtTn+nTt3aubMmXJ2dtacOXNUt25dRUVFydXVVS+88IKGDRuWrOMCAADpG4EUAABId5ycnNSnTx/t2LFDffv21bZt27RixQp16dJFOXPmVJcuXSTdv8tGkrZu3apjx46pZMmS8vT01NSpU7V161bt2LFDn3/+uX788UdJSrNv2evSpYty5cqlzp07KzAwUNu2bdNHH32k3bt366OPPrJ5oHmcMmXKqFOnTlqwYIH2798vSfGOcfPmzdZvmUtsKFtCqlWrpsOHD2vAgAHauXOntm7dquHDhytXrlyqVq1akuu3bNlSQUFBWrFihVq0aGFTv4+Pj3bs2KEFCxZo7969mjt3rt555x1ZLJZEz2fdunUVGRmpQYMGWe/Wmj59uk2A2KJFCxUpUkRdunTR8uXLtXv3bn399df673//qwIFCihLliyqVq2aQkNDFRAQoG3btmnHjh0aOHCgXF1dVbdu3WSfHwAAkP4wZA8AAKRLLVq0ULZs2TRt2jQFBATI09NTtWrVUp8+fZQ/f35J0nPPPaemTZtq/vz5+vXXX7V69WpNmTJFo0eP1gcffKBs2bKpbNmymjdvnt555x39/vvv1mc4PY78+fNrwYIFGjdunIYPH6579+6pTJkymjJliurXr5/oer1799batWv12WefacWKFapZs6ZmzZqlSZMm6f3331eWLFlUvnx5zZ49O0XfBli7dm2NHTtW3333nXr16iWLxaLKlStr7ty51uGPj/LCCy+oUKFC+vvvv9WiRQubeQMGDNC9e/c0YcIERUVFqVixYurZs6dOnjypzZs3xxuaJ93/Fr7+/fvrhx9+0Lp161S+fHlNmjRJ7dq1sy6TNWtWzZ8/X+PGjdOYMWN069YtFS1aVH379lXXrl0l3Q/xpk6dqsmTJ6tPnz6KiYlRhQoV9N1336lUqVLJPj8AACD9sRhp9XRPAAAAAAAAIBkYsgcAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAExFIAUAAAAAAABTEUgBAAAAAADAVARSAAAAAAAAMBWBFAAAAAAAAEz1f9u86O3xk66dAAAAAElFTkSuQmCC",
                        "text/plain": [
                            "<Figure size 1200x600 with 1 Axes>"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "import pandas as pd\n",
                "import seaborn as sns\n",
                "import matplotlib.pyplot as plt\n",
                "\n",
                "# Set the style\n",
                "sns.set_style(\"whitegrid\")\n",
                "\n",
                "plt.figure(figsize=(12, 6))\n",
                "sns.histplot(df['total_tokens'], kde=False, bins=30, color=sns.color_palette(\"viridis\", as_cmap=True)(0.5))\n",
                "\n",
                "plt.title('Frequency Distribution of Total Tokens', fontsize=15)\n",
                "plt.xlabel('Total Tokens Values', fontsize=12)\n",
                "plt.ylabel('Frequency', fontsize=12)\n",
                "plt.tight_layout()\n",
                "plt.show()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "b633e1e9-52df-40e1-af10-dc2216625b19",
            "metadata": {},
            "source": [
                "## Conclusion\n",
                "\n",
                "In this walkthrough, you queried LLM runs and loaded associated trace feedback all into a simple tabular format. This makes it easy to export to other analytical tools and data stores."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.11.2"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}