summarization_llm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6b515611",
   "metadata": {},
   "source": [
    "# LLM Text Summarization\n",
    "\n",
    "Concepts:\n",
    "- HuggingFace and Azure OpenAI APIs\n",
    "- Text summarization of 10-K Disclosures About Market Risk\n",
    "- Role prompting\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f256c97c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terence/env3.11/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas import DataFrame, Series\n",
    "import textwrap\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
    "from transformers.utils import logging\n",
    "from openai import AzureOpenAI\n",
    "from rouge_score import rouge_scorer\n",
    "from tqdm import tqdm\n",
    "from finds.database import SQL, RedisDB\n",
    "from finds.unstructured import Edgar\n",
    "from finds.structured import BusDay, CRSP, PSTAT\n",
    "from finds.readers import Sectoring\n",
    "from secret import paths, credentials\n",
    "logging.set_verbosity_error() # logging.set_verbosity_info() #logging.set_verbosity_warning()\n",
    "VERBOSE = 0\n",
    "SAVED = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c326579e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last FamaFrench Date 2024-04-30 00:00:00\n"
     ]
    }
   ],
   "source": [
    "sql = SQL(**credentials['sql'], verbose=VERBOSE)\n",
    "user = SQL(**credentials['user'], verbose=VERBOSE)\n",
    "bd = BusDay(sql)\n",
    "rdb = RedisDB(**credentials['redis'])\n",
    "crsp = CRSP(sql, bd, rdb, verbose=VERBOSE)\n",
    "pstat = PSTAT(sql, bd, verbose=VERBOSE)\n",
    "ed = Edgar(paths['10X'], zipped=False, verbose=VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7a206818",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve universe of stocks\n",
    "univ = crsp.get_universe(bd.endmo(20231231))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c021184f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lookup company names\n",
    "comnam = crsp.build_lookup(source='permno', target='comnam', fillna=\"\")\n",
    "univ['comnam'] = comnam(univ.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42f447d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lookup sic codes from Compustat, and map to FF 10-sector code\n",
    "sic = pstat.build_lookup(source='lpermno', target='sic', fillna=0)\n",
    "industry = Series(sic[univ.index], index=univ.index)\n",
    "industry = industry.where(industry > 0, univ['siccd'])\n",
    "sectors = Sectoring(sql, scheme='codes10', fillna='')   # supplement from crosswalk\n",
    "univ['sector'] = sectors[industry]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6cec1c9",
   "metadata": {},
   "source": [
    "Load Disclosure about Market Risk text from 10-K's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "95b20d57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve from 10K's in 1Q 2024\n",
    "item, form = 'qqr10K', '10-K'\n",
    "rows = DataFrame(ed.open(form=form, item=item))\n",
    "found = rows[rows['date'].between(20240101, 20240331)]\\\n",
    "    .drop_duplicates(subset=['permno'], keep='last')\\\n",
    "    .set_index('permno')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7fbeb48e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep largest decile of stocks\n",
    "found = found.loc[found.index.intersection(univ.index[univ['decile'] == 1])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f747cca6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep minimum length\n",
    "docs = {permno: ed[found.loc[permno, 'pathname']].lower()\n",
    "        for permno in found.index}\n",
    "permnos = [permno for permno, doc in docs.items() if len(doc)>2000]\n",
    "found = found.join(Series(docs, name='item').reindex(permnos), how='inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef14de3b",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "## HuggingFace APIs\n",
    "\n",
    "Command line interface\n",
    "   \n",
    "`pip install huggingface_hub[\"cli\"]`\n",
    "\n",
    "- To empty cache (in ~/.cache/huggingface/)\n",
    "    \n",
    "  `huggingface-cli delete-cache`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb850b5e",
   "metadata": {},
   "source": [
    "\n",
    "### Transformers modules\n",
    "\n",
    "- AutoTokenizers\n",
    "- AutoModels\n",
    "  - `generate` method: \n",
    "\n",
    "\n",
    "  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ca43e85d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n",
    "save_name = paths['scratch'] / \"Llama-3-8B-Instruct\"\n",
    "model_name = save_name if SAVED else model_id   # load from folder if saved locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0ac1b3",
   "metadata": {},
   "source": [
    "### Quantization\n",
    "\n",
    "https://medium.com/@manuelescobar-dev/implementing-and-running-llama-3-with-hugging-faces-transformers-library-40e9754d8c80\n",
    "\n",
    "Quantization reduces the hardware requirements by loading the model weights with lower precision. Instead of loading them in 16 bits (float16), they are loaded in 4 bits, significantly reducing memory usage from ~20GB to ~8GB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "025ecbd9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BitsAndBytesConfig {\n",
       "  \"_load_in_4bit\": true,\n",
       "  \"_load_in_8bit\": false,\n",
       "  \"bnb_4bit_compute_dtype\": \"float16\",\n",
       "  \"bnb_4bit_quant_storage\": \"uint8\",\n",
       "  \"bnb_4bit_quant_type\": \"nf4\",\n",
       "  \"bnb_4bit_use_double_quant\": false,\n",
       "  \"llm_int8_enable_fp32_cpu_offload\": false,\n",
       "  \"llm_int8_has_fp16_weight\": false,\n",
       "  \"llm_int8_skip_modules\": null,\n",
       "  \"llm_int8_threshold\": 6.0,\n",
       "  \"load_in_4bit\": true,\n",
       "  \"load_in_8bit\": false,\n",
       "  \"quant_method\": \"bitsandbytes\"\n",
       "}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_dtype = getattr(torch, \"float16\")\n",
    "bnb_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_use_double_quant=False,\n",
    "    bnb_4bit_quant_type=\"nf4\",\n",
    "    bnb_4bit_compute_dtype=compute_dtype,\n",
    ")\n",
    "bnb_config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b278870",
   "metadata": {},
   "source": [
    "Load model and tokenizer from Huggingface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "16e4ebb7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading checkpoint shards: 100%|██████████| 4/4 [00:08<00:00,  2.01s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LlamaForCausalLM(\n",
       "  (model): LlamaModel(\n",
       "    (embed_tokens): Embedding(128256, 4096)\n",
       "    (layers): ModuleList(\n",
       "      (0-31): 32 x LlamaDecoderLayer(\n",
       "        (self_attn): LlamaSdpaAttention(\n",
       "          (q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
       "          (k_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
       "          (v_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
       "          (o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
       "          (rotary_emb): LlamaRotaryEmbedding()\n",
       "        )\n",
       "        (mlp): LlamaMLP(\n",
       "          (gate_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
       "          (up_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
       "          (down_proj): Linear4bit(in_features=14336, out_features=4096, bias=False)\n",
       "          (act_fn): SiLU()\n",
       "        )\n",
       "        (input_layernorm): LlamaRMSNorm()\n",
       "        (post_attention_layernorm): LlamaRMSNorm()\n",
       "      )\n",
       "    )\n",
       "    (norm): LlamaRMSNorm()\n",
       "  )\n",
       "  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name,\n",
    "    quantization_config=bnb_config,\n",
    "    #torch_dtype=torch.bfloat16,\n",
    "    device_map=\"cuda\",  # \"auto\",  'cuda'\n",
    ")\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad610e9b",
   "metadata": {},
   "source": [
    "Show model's context length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0a016f25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max context length 8192\n"
     ]
    }
   ],
   "source": [
    "# Maximum context length\n",
    "print('max context length', model.config.max_position_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f7c15586",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CUDA memory: 6.06 GB\n"
     ]
    }
   ],
   "source": [
    "# save the model to local disk\n",
    "if not SAVED:\n",
    "    model.save_pretrained(save_name)\n",
    "print(f\"CUDA memory: {torch.cuda.memory_allocated()/1e9:.2f} GB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc97c4d2",
   "metadata": {},
   "source": [
    "### Temperature\n",
    "\n",
    "Temperature is a parameter used to control the randomness of the model's output during text generation. Low Temperature (value close to 0) decreases randomness, leading to more focused and predictable outputs. High Temperature increases randomness: the model is more likely to choose less probable words, leading to more diverse and creative outputs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0dc89e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Template to prompt and generate response\n",
    "MAX_CHAR = 20000\n",
    "def generate_response(text, max_char=MAX_CHAR,\n",
    "                      role=\"You are a helpful AI assistant\",\n",
    "                      prompt=\"Write a concise summary of the text.\"):\n",
    "    content = f\"\"\"\n",
    "{prompt}\n",
    "\n",
    "Text in triple quotes: '''{(text+' ')[:max_char]}'''\n",
    "\n",
    "Summary:\"\"\".strip()\n",
    "    \n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": role},\n",
    "        {\"role\": \"user\", \"content\": content},\n",
    "    ]\n",
    "\n",
    "    input_ids = tokenizer.apply_chat_template(\n",
    "        messages,\n",
    "        add_generation_prompt=True,\n",
    "        return_tensors=\"pt\"\n",
    "    ).to(model.device)\n",
    "\n",
    "    if VERBOSE:\n",
    "        print('tokens:', np.prod(input_ids.shape),\n",
    "              model.config.max_position_embeddings)\n",
    "\n",
    "    terminators = [\n",
    "        tokenizer.eos_token_id,\n",
    "        tokenizer.convert_tokens_to_ids(\"<|eot_id|>\")\n",
    "    ]\n",
    "\n",
    "    outputs = model.generate(\n",
    "        input_ids,\n",
    "        max_new_tokens=256,  \n",
    "        eos_token_id=terminators,\n",
    "        #do_sample=True,\n",
    "        temperature=0.01, #0.6,\n",
    "        #top_p=0.9,\n",
    "    )\n",
    "    response = outputs[0][input_ids.shape[-1]:]\n",
    "    return tokenizer.decode(response, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d5b8b1",
   "metadata": {},
   "source": [
    "## Azure AI API\n",
    "\n",
    "Azure, Microsoft's cloud-based platform for AI and other subscription services, provides API's and endpoints for OpenAI's models.\n",
    "\n",
    "- https://oai.azure.com/portal\n",
    "- https://openai.com/api/\n",
    "\n",
    "\n",
    "### OpenAI API\n",
    "\n",
    "Alternatively, GPT and other AI models, such as Whisper for automatic speech recognization (ASR), can be accessed directly from OpenAI:\n",
    "\n",
    "- https://platform.openai.com/docs/overview\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "22706dca",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "# create client object\n",
    "LLM_MODEL = \"gpt-35-turbo-16k\"   # name of model to use\n",
    "client = AzureOpenAI(api_version=\"2024-02-01\", **credentials['azure'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "abe03e79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# template to prompt GPT and generate response\n",
    "def get_gpt_response(prompt: str, system_prompt: str,\n",
    "                     model: str = LLM_MODEL) -> str:\n",
    "    \"\"\"Creating message required by API: \"prompt\" will be the query sent to GPT\"\"\"\n",
    "    try:\n",
    "        messages = [{\"role\": \"system\", \"content\": system_prompt},\n",
    "                    {\"role\": \"user\", \"content\": prompt}]\n",
    "        completion = client.chat.completions.create(\n",
    "            model=model,  # must match the custom deployment name you chose\n",
    "            messages=messages,\n",
    "            temperature=0.0,\n",
    "        )\n",
    "        return completion.choices[0].message.content\n",
    "    except Exception as e:\n",
    "        print(f\"ERROR!\\n\\n {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32c29f68",
   "metadata": {},
   "source": [
    "## Text summarization\n",
    "\n",
    "Text summarization is the task of condensing a piece of text into a shorter version while retaining its key information and meaning. It aims to generate a concise summary that captures the main points of the original document or passage.\n",
    "\n",
    "Sample one company from each industry, and generate summary of risk disclosure text with quantized Llama-3-8B model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "35bd4824",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample companies from each industry sector\n",
    "docs = univ.loc[found.index].groupby('sector').sample(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5a708ffc",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===== BOOKING HOLDINGS INC =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The company is exposed to market risk due to changes in interest\n",
      "rates, foreign currency exchange rates, and equity prices. To manage\n",
      "this risk, the company uses internal policies and procedures, as well\n",
      "as derivative financial instruments, such as foreign currency exchange\n",
      "contracts. The company evaluates its exposure to market risk by\n",
      "assessing near-term and long-term fluctuations in interest rates and\n",
      "foreign currency exchange rates. The company also faces exposure to\n",
      "movements in foreign currency exchange rates due to its businesses\n",
      "outside the US, which are translated from local currencies into US\n",
      "dollars. Additionally, the company is exposed to equity price risk due\n",
      "to changes in the fair values of its investments in equity securities\n",
      "of publicly-traded companies and private companies.\n",
      "\n",
      "===== PIONEER NATURAL RESOURCES CO =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The company, Pioneer Natural Resources Company, discloses its market\n",
      "risk exposure in its financial statements. The company's primary\n",
      "market risk exposure is related to commodity price risk, which is\n",
      "mitigated by maintaining financial flexibility and diversifying oil\n",
      "and gas sales. The company also uses marketing derivatives to\n",
      "diversify oil pricing to Gulf Coast and international markets.\n",
      "Additionally, the company is exposed to share price risk when holders\n",
      "of convertible notes exercise their conversion option, and interest\n",
      "rate risk due to fixed-rate debt. The company's credit risk is\n",
      "primarily associated with the collection of receivables from the sale\n",
      "of oil and gas production and purchased commodities, and the risk of a\n",
      "counterparty's failure to meet its obligations under derivative\n",
      "contracts.\n",
      "\n",
      "===== PAYPAL HOLDINGS INC =====\n",
      "The text discusses the market risk, interest rate risk, foreign\n",
      "currency exchange rate risk, and equity investment risk faced by the\n",
      "company. Market risk refers to the potential for economic losses due\n",
      "to adverse changes in market factors such as interest rates, foreign\n",
      "currency exchange rates, and equity investment risk. The company has\n",
      "established policies to mitigate these risks and monitors risk\n",
      "exposures on an ongoing basis.\n",
      "\n",
      "Interest rate risk is managed by maintaining a diversified portfolio\n",
      "of cash, cash equivalents, and available-for-sale debt securities, and\n",
      "by using interest rate swaps and other derivatives to hedge against\n",
      "interest rate changes. The company also has foreign currency exchange\n",
      "contracts to manage foreign currency exchange rate risk and to reduce\n",
      "the impact of currency fluctuations on its consolidated cash flows and\n",
      "results of operations.\n",
      "\n",
      "Equity investment risk is managed by diversifying the portfolio of\n",
      "strategic investments and by regularly reviewing the fair value of\n",
      "these investments. The company also has a foreign currency exchange\n",
      "exposure management program to identify material foreign currency\n",
      "exposures and to manage these exposures through the execution of\n",
      "foreign currency exchange contracts.\n",
      "\n",
      "The company's risk management policies and procedures are designed to\n",
      "identify, assess, and mitigate risks that could have a material impact\n",
      "on its financial performance and position.\n",
      "\n",
      "===== BOSTON SCIENTIFIC CORP =====\n",
      "Boston Scientific Corporation, a medical device company, reports its\n",
      "market risk exposure and risk management practices. The company's\n",
      "earnings and cash flows are exposed to market risk from changes in\n",
      "currency exchange rates and interest rates. To manage this risk, the\n",
      "company uses derivative financial instruments, such as currency and\n",
      "interest rate swaps, to hedge its exposure. The company's currency\n",
      "risk consists primarily of foreign currency-denominated firm\n",
      "commitments, forecasted foreign currency-denominated intercompany and\n",
      "third-party transactions, and net investments in certain subsidiaries.\n",
      "The company's interest rate risk relates primarily to US dollar and\n",
      "euro-denominated borrowings partially offset by US dollar cash\n",
      "investments. The company's risk management program is designed to\n",
      "minimize the impact of market risk on its earnings and cash flows.\n",
      "\n",
      "===== R T X CORP =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The company is exposed to market risk due to fluctuations in foreign\n",
      "currency exchange rates and interest rates. To manage this risk, they\n",
      "use foreign currency forward contracts to hedge foreign-denominated\n",
      "payments and receipts. The company also hedges foreign currency\n",
      "exposures associated with committed foreign currency purchases and\n",
      "sales. The aggregate notional amount of outstanding foreign currency\n",
      "hedges was $15.8 billion and $11.2 billion at December 31, 2023 and\n",
      "2022, respectively. The company's sales are typically denominated in\n",
      "US dollars, but they also transact business in various foreign\n",
      "currencies, exposing them to changes in foreign currency exchange\n",
      "rates. To minimize this exposure, they hedge a certain portion of\n",
      "sales to secure the rates at which US dollars will be converted. The\n",
      "company also has financial instruments subject to interest rate risk,\n",
      "primarily fixed-rate debt obligations, but they believe their exposure\n",
      "to interest rate risk is not material.\n",
      "\n",
      "===== MONSTER BEVERAGE CORP NEW =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The company is exposed to various market risks, including fluctuations\n",
      "in commodity and energy prices, foreign currency exchange rates, and\n",
      "limitations in raw material availability. To manage foreign exchange\n",
      "risk, the company enters into forward currency exchange contracts with\n",
      "financial institutions, but does not designate them as hedge\n",
      "transactions. The company's investments are also subject to various\n",
      "risks, including credit, liquidity, market, and interest rate risks.\n",
      "\n",
      "===== HILTON WORLDWIDE HOLDINGS INC =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The company is exposed to market risk primarily from changes in\n",
      "interest rates and foreign currency exchange rates. To manage this\n",
      "risk, they use derivative financial instruments, such as interest rate\n",
      "swaps and forward contracts, to hedge against a portion of the risks.\n",
      "The company's exposure to interest rate risk is primarily related to\n",
      "variable-rate indebtedness, while foreign currency exchange rate risk\n",
      "is associated with international revenues and expenses. The company\n",
      "uses hedging instruments to reduce volatility in their results of\n",
      "operations and cash flows.\n",
      "\n",
      "===== TARGET CORP =====\n",
      "The text is a section from a company's Form 10-K filing with the US\n",
      "Securities and Exchange Commission. It provides a summary of the\n",
      "company's market risk exposure, including:\n",
      "\n",
      "* Interest rate risk: The company's floating rate debt obligations and\n",
      "short-term investments expose it to changes in interest rates. A 1%\n",
      "increase in interest rates would increase earnings before income taxes\n",
      "by $5 million.\n",
      "* Pension risk: The company's qualified defined benefit pension plan\n",
      "is exposed to changes in interest rates, which could increase annual\n",
      "expense by $36 million if interest rates decrease.\n",
      "* General liability and workers' compensation risk: The company's\n",
      "liabilities for general liability and workers' compensation are\n",
      "recorded at net present value and fluctuate with changes in interest\n",
      "rates.\n",
      "* Nonqualified deferred compensation risk: The company's nonqualified,\n",
      "unfunded deferred compensation plans expose it to market returns on\n",
      "accumulated team member balances.\n",
      "\n",
      "The company maintains insurance coverage to limit its exposure to\n",
      "certain events, including network security matters. It also maintains\n",
      "a funded qualified defined benefit pension plan and nonqualified and\n",
      "international pension plans that are generally unfunded.\n",
      "\n",
      "===== A T & T INC =====\n",
      "Here is a concise summary of the text:\n",
      "\n",
      "The report discusses the financial statements of AT&T Inc. (AT&T) for\n",
      "the year ended December 31, 2023. The company is exposed to market\n",
      "risks, primarily from changes in interest rates and foreign currency\n",
      "exchange rates. To manage these risks, AT&T employs derivatives, such\n",
      "as interest rate swaps and foreign currency exchange contracts.\n",
      "\n",
      "The report also discusses the company's internal control over\n",
      "financial reporting, which is designed to provide reasonable assurance\n",
      "regarding the reliability of financial reporting and the preparation\n",
      "of financial statements. The company's management is responsible for\n",
      "maintaining effective internal control over financial reporting, and\n",
      "the independent registered public accounting firm, Ernst & Young LLP,\n",
      "has issued an unqualified opinion on the effectiveness of the\n",
      "company's internal control over financial reporting.\n",
      "\n",
      "The report also highlights the critical audit matters, which are\n",
      "matters arising from the current period audit of the financial\n",
      "statements that were communicated or required to be communicated to\n",
      "the audit committee and that relate to accounts or disclosures that\n",
      "are material to the financial statements. The critical audit matters\n",
      "discussed in the report include the discount rates used in determining\n",
      "pension and postretirement benefit obligations and the evaluation of\n",
      "goodwill for impairment.\n",
      "\n",
      "===== WASTE MANAGEMENT INC DEL =====\n",
      "The text discusses the market risk exposure of a company, specifically\n",
      "in the areas of interest rates, commodity prices, and Canadian\n",
      "currency rates. The company has exposure to changes in interest rates\n",
      "through its debt obligations, including short-term borrowings and tax-\n",
      "exempt bonds. The company also has exposure to commodity price risks\n",
      "through its operating agreements and investments in renewable energy\n",
      "production. The company has invested in facilities to capture methane\n",
      "from landfills and convert it into renewable natural gas, which is\n",
      "used to produce transportation fuels. The company is also exposed to\n",
      "currency rate risks due to its operations in Canada and India.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "summary = {}\n",
    "for permno in docs.index:\n",
    "    print('=====', univ.loc[permno, 'comnam'], '=====')\n",
    "    summary[permno] = generate_response(found.loc[permno, 'item'])\n",
    "    print(\"\\n\".join([textwrap.fill(s) for s in summary[permno].split('\\n')]))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1207478",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "__ROUGE__\n",
    "\n",
    "Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics used to evaluate the quality of summaries by comparing them to reference summaries or human-generated summaries.\n",
    "\n",
    "- ROUGE-N measures the overlap of n-grams (contiguous sequences of n words) between the system-generated and the reference summaries\n",
    "- ROUGE-L measures the longest common subsequence (LCS).\n",
    "\n",
    "Recall, precision and F1-measure versions of these metrics can be computed.\n",
    "\n",
    "__BLEU__\n",
    "\n",
    "Bilingual Evaluation Understudy (BLEU) is a metric originally designed for evaluating machine translation, but it is also adapted for text summarization evaluation.\n",
    "- N-gram Precision measures the overlap of n-grams (typically up to 4-grams) between the system-generated summary and the reference summary.\n",
    "- Brevity Penalty penalizes overly short summaries that do not capture enough information from the reference summaries.\n",
    "- Cumulative BLEU calculates the geometric mean of BLEU scores for 1-gram to n-gram, rewarding systems that produce more accurate translations across longer phrases.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "21af7391",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To compute rouge-1 and rouge-2 scores\n",
    "rouge_types = ['rouge1', 'rouge2']\n",
    "scorer = rouge_scorer.RougeScorer(rouge_types, use_stemmer=True)\n",
    "scores = {rouge_type: [] for rouge_type in rouge_types}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f5b63c4",
   "metadata": {},
   "source": [
    "Generete reference summaries from GPT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e7234200",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===== BOOKING HOLDINGS INC =====\n",
      "The text discusses the market risks that a company is exposed to,\n",
      "including changes in interest rates, foreign currency exchange rates,\n",
      "and equity prices. The company manages these risks through established\n",
      "policies and procedures, as well as the use of derivative financial\n",
      "instruments. The objective is to mitigate potential adverse\n",
      "fluctuations in income statement, cash flow, and fair value. The\n",
      "company evaluates its exposure by assessing near-term and long-term\n",
      "fluctuations, utilizing market indicators and projections. The company\n",
      "does not engage in speculative trading or leveraged derivatives. The\n",
      "text also mentions the impact of interest rate changes on the fair\n",
      "value of debt and the sensitivity of convertible senior notes to\n",
      "equity market price volatility. Additionally, the company faces\n",
      "exposure to foreign currency exchange rates due to the translation of\n",
      "financial results from local currencies to US dollars. The impact of\n",
      "currency exchange rates on gross bookings and revenues is discussed.\n",
      "The company uses foreign currency exchange derivatives to hedge its\n",
      "exposure. Lastly, the text mentions the equity price risk associated\n",
      "with investments in publicly-traded and private companies, with\n",
      "potential losses in fair value.\n",
      "\n",
      "===== PIONEER NATURAL RESOURCES CO =====\n",
      "The text provides quantitative and qualitative disclosures about\n",
      "market risk for a company. The company faces risks related to changes\n",
      "in commodity prices, changes in its share price, interest rate\n",
      "movements, and credit risks. The company mitigates commodity price\n",
      "risk through financial flexibility and diversification of sales. It\n",
      "also enters into pipeline capacity commitments and purchase\n",
      "transactions to diversify its sales. The company uses marketing\n",
      "derivatives to diversify its oil pricing. It has long-term marketing\n",
      "contracts that are accounted for as derivative instruments.\n",
      "Fluctuations in the fair value of marketing derivatives could occur\n",
      "based on changes in price differentials. The company is also exposed\n",
      "to share price risk and interest rate risk. It manages credit risk\n",
      "through credit risk policies and procedures.\n",
      "\n",
      "===== PAYPAL HOLDINGS INC =====\n",
      "The text discusses the market risk faced by a company due to adverse\n",
      "changes in market factors such as interest rates, foreign currency\n",
      "exchange rates, and equity investment risk. The company has policies\n",
      "in place to mitigate these risks and monitors its risk exposures. It\n",
      "is exposed to interest rate risk in its investment portfolio and\n",
      "customer balances. Changes in interest rates can affect the interest\n",
      "income earned and the fair value of securities. The company also faces\n",
      "foreign currency exchange rate risk due to its international\n",
      "operations and uses hedging contracts to manage this risk. Adverse\n",
      "changes in exchange rates can impact the company's cash flows and\n",
      "results of operations. Additionally, the company has strategic\n",
      "investments that are subject to market-related risks, and changes in\n",
      "the value of these investments can impact the company's net income.\n",
      "\n",
      "===== BOSTON SCIENTIFIC CORP =====\n",
      "The text discusses the market risks faced by a medical device company\n",
      "due to changes in currency exchange rates and interest rates. The\n",
      "company manages these risks through a risk management program that\n",
      "includes the use of derivative financial instruments. The company\n",
      "primarily uses nonderivative and derivative instruments to manage its\n",
      "exposure to currency exchange rate fluctuations. It also uses interest\n",
      "rate derivative instruments to manage its exposure to changes in\n",
      "interest rates. The company has a diversified group of major financial\n",
      "institutions as counterparties for its derivative instruments. The\n",
      "text also includes an audit opinion on the company's financial\n",
      "statements, highlighting a critical audit matter related to the\n",
      "accounting for business combinations. The auditor evaluated the\n",
      "company's controls and tested the fair value estimates of intangible\n",
      "assets and contingent consideration liabilities related to the\n",
      "acquisitions.\n",
      "\n",
      "===== R T X CORP =====\n",
      "The text discusses the market risk exposures of a company, primarily\n",
      "related to fluctuations in foreign currency exchange rates and\n",
      "interest rates. The company uses sensitivity analysis to quantify its\n",
      "market risk exposure and employs foreign currency forward contracts to\n",
      "hedge price risk associated with foreign denominated payments and\n",
      "receipts. The company actively manages foreign currency exposures and\n",
      "hedges significant exposures with foreign currency derivatives. The\n",
      "aggregate notional amount of outstanding foreign currency hedges is\n",
      "provided. The company also faces foreign currency exchange impact and\n",
      "risk to operational results due to costs incurred in local currencies\n",
      "and transacting business in various foreign currencies. To minimize\n",
      "exposure, the company hedges a portion of sales and forecasts of\n",
      "costs. Hedging gains and losses are partially offset by foreign\n",
      "currency translation impacts. The company also has fixed-rate and\n",
      "variable-rate debt obligations subject to interest rate risk, but the\n",
      "exposure is considered not material. The company may consider using\n",
      "interest rate swaps in the future to hedge floating rates.\n",
      "\n",
      "===== MONSTER BEVERAGE CORP NEW =====\n",
      "The text discusses the market risks faced by a company in its normal\n",
      "business operations. The main market risks include fluctuations in\n",
      "commodity and input prices, energy and fuel prices, and limitations in\n",
      "the availability of raw materials and packaging materials. The company\n",
      "does not use hedging agreements or alternative instruments to manage\n",
      "these risks. Additionally, the company is exposed to risks from\n",
      "changes in foreign currency exchange rates due to its international\n",
      "business expansion strategy. The company has entered into forward\n",
      "currency exchange contracts to manage a portion of the foreign\n",
      "exchange risk exposure. However, these contracts are not designated as\n",
      "hedge transactions. The gains and losses on these contracts are\n",
      "recognized in the consolidated statements of income. The company does\n",
      "not consider the potential loss resulting from a hypothetical adverse\n",
      "change in foreign currency exchange rates to be significant. The\n",
      "company also discloses its cash and investment holdings, which are\n",
      "subject to general credit, liquidity, market, and interest rate risks.\n",
      "\n",
      "===== HILTON WORLDWIDE HOLDINGS INC =====\n",
      "The company is exposed to market risk primarily from changes in\n",
      "interest rates and foreign currency exchange rates. They use\n",
      "derivative financial instruments to hedge against these risks and\n",
      "reduce volatility in their results of operations and cash flows. They\n",
      "are exposed to interest rate risk on their variable-rate indebtedness,\n",
      "but use an interest rate swap to maintain an appropriate level of\n",
      "exposure. They also have exposure to foreign currency exchange rate\n",
      "risk due to conducting business in various currencies. They use\n",
      "forward contracts as cash flow hedges and not designated as hedging\n",
      "instruments to offset this risk. The fair value and earnings effect of\n",
      "these contracts are not considered material to their financial\n",
      "statements.\n",
      "\n",
      "===== TARGET CORP =====\n",
      "The text provides various disclosures related to market risk, income\n",
      "taxes, pension accounting, legal and other contingencies, new\n",
      "accounting pronouncements, and forward-looking statements. It also\n",
      "includes quantitative and qualitative disclosures about market risk,\n",
      "specifically related to interest rate changes, debt obligations,\n",
      "short-term investments, liabilities, and pension plan exposures. The\n",
      "text concludes with an index to the financial statements.\n",
      "\n",
      "===== A T & T INC =====\n",
      "The text discusses AT&T Inc.'s exposure to market risks, primarily\n",
      "from changes in interest rates and foreign currency exchange rates.\n",
      "The company manages these risks through the use of derivatives, such\n",
      "as interest rate swaps and foreign currency exchange contracts. They\n",
      "do not use derivatives for trading or speculative purposes. The text\n",
      "also mentions the significant assumption of the weighted-average\n",
      "discount rate used in estimating postretirement benefit obligations.\n",
      "Lower discount rates result in higher obligations, while future\n",
      "increases in rates could lead to lower obligations. The text further\n",
      "highlights the interest rate risk associated with the company's\n",
      "financial instruments, such as fixed-rate notes and debentures. AT&T\n",
      "manages interest expense by controlling the mix of fixed- and\n",
      "floating-rate debt through various instruments. The company also uses\n",
      "cross-currency swaps to hedge foreign currency exchange risk. The text\n",
      "concludes with the report of management and the independent registered\n",
      "public accounting firm, which express opinions on the company's\n",
      "financial statements and internal control over financial reporting.\n",
      "\n",
      "===== WASTE MANAGEMENT INC DEL =====\n",
      "The text discusses the market risks that a company is exposed to,\n",
      "including changes in interest rates, commodity prices, and currency\n",
      "rates. The company uses derivatives to manage these risks but had no\n",
      "derivatives outstanding as of December 31, 2023. The company's\n",
      "exposure to interest rate changes primarily relates to its financing\n",
      "activities, and a 100-basis point increase in interest rates would\n",
      "increase its 2024 interest expense by $18 million. The company is also\n",
      "exposed to commodity price risks, particularly in relation to diesel\n",
      "fuel, electricity, and recycled materials. The decline in commodity\n",
      "prices has put pressure on the company's margins. Additionally, the\n",
      "company is exposed to currency rate risks due to its operations in\n",
      "Canada and support functions in India, but the impact of foreign\n",
      "currency has not materially affected its results of operations.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "reference = {}\n",
    "for permno in docs.index:\n",
    "    print('=====', univ.loc[permno, 'comnam'], '=====')\n",
    "    system_prompt = \"You are an AI assistant.\"\n",
    "    prompt_template = f\"\"\"\n",
    "\"Write a concise summary of the text.\n",
    "\n",
    "Text in triple quotes: '''{(found.loc[permno, 'item'] + ' ')[:MAX_CHAR]}'''\n",
    "\n",
    "Summary:\"\"\".strip()    \n",
    "    reference[permno] = get_gpt_response(prompt_template, system_prompt)\n",
    "    print(\"\\n\".join([textwrap.fill(s) for s in reference[permno].split('\\n')]))\n",
    "    print()\n",
    "    score = scorer.score(target=reference[permno], prediction=summary[permno])\n",
    "    for rouge_type in rouge_types:\n",
    "        scores[rouge_type].append(Series(score[rouge_type]._asdict(),\n",
    "                                         name=univ.loc[permno, 'comnam']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03719339",
   "metadata": {},
   "source": [
    "Compare rouge-1 metric\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bd571d38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROUGE1 metric:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>fmeasure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BOOKING HOLDINGS INC</th>\n",
       "      <td>0.825758</td>\n",
       "      <td>0.586022</td>\n",
       "      <td>0.685535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PIONEER NATURAL RESOURCES CO</th>\n",
       "      <td>0.511450</td>\n",
       "      <td>0.544715</td>\n",
       "      <td>0.527559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PAYPAL HOLDINGS INC</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.764286</td>\n",
       "      <td>0.604520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BOSTON SCIENTIFIC CORP</th>\n",
       "      <td>0.488372</td>\n",
       "      <td>0.456522</td>\n",
       "      <td>0.471910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R T X CORP</th>\n",
       "      <td>0.675000</td>\n",
       "      <td>0.646707</td>\n",
       "      <td>0.660550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MONSTER BEVERAGE CORP NEW</th>\n",
       "      <td>0.826667</td>\n",
       "      <td>0.382716</td>\n",
       "      <td>0.523207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HILTON WORLDWIDE HOLDINGS INC</th>\n",
       "      <td>0.726316</td>\n",
       "      <td>0.594828</td>\n",
       "      <td>0.654028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TARGET CORP</th>\n",
       "      <td>0.190217</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>0.286885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A T &amp; T INC</th>\n",
       "      <td>0.473684</td>\n",
       "      <td>0.578947</td>\n",
       "      <td>0.521053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WASTE MANAGEMENT INC DEL</th>\n",
       "      <td>0.638095</td>\n",
       "      <td>0.485507</td>\n",
       "      <td>0.551440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average</th>\n",
       "      <td>0.585556</td>\n",
       "      <td>0.562358</td>\n",
       "      <td>0.548669</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               precision    recall  fmeasure\n",
       "BOOKING HOLDINGS INC            0.825758  0.586022  0.685535\n",
       "PIONEER NATURAL RESOURCES CO    0.511450  0.544715  0.527559\n",
       "PAYPAL HOLDINGS INC             0.500000  0.764286  0.604520\n",
       "BOSTON SCIENTIFIC CORP          0.488372  0.456522  0.471910\n",
       "R T X CORP                      0.675000  0.646707  0.660550\n",
       "MONSTER BEVERAGE CORP NEW       0.826667  0.382716  0.523207\n",
       "HILTON WORLDWIDE HOLDINGS INC   0.726316  0.594828  0.654028\n",
       "TARGET CORP                     0.190217  0.583333  0.286885\n",
       "A T & T INC                     0.473684  0.578947  0.521053\n",
       "WASTE MANAGEMENT INC DEL        0.638095  0.485507  0.551440\n",
       "  average                       0.585556  0.562358  0.548669"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rouge_type = rouge_types[0]\n",
    "df = pd.concat(scores[rouge_type], axis=1)\n",
    "print(f\"{rouge_type.upper()} metric:\")\n",
    "pd.concat([df, df.T.mean().rename('  average')], axis=1).T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e498380",
   "metadata": {},
   "source": [
    "Compare rouge-2 metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fafac66b",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROUGE2 metric:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>fmeasure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BOOKING HOLDINGS INC</th>\n",
       "      <td>0.496183</td>\n",
       "      <td>0.351351</td>\n",
       "      <td>0.411392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PIONEER NATURAL RESOURCES CO</th>\n",
       "      <td>0.215385</td>\n",
       "      <td>0.229508</td>\n",
       "      <td>0.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PAYPAL HOLDINGS INC</th>\n",
       "      <td>0.291080</td>\n",
       "      <td>0.446043</td>\n",
       "      <td>0.352273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BOSTON SCIENTIFIC CORP</th>\n",
       "      <td>0.203125</td>\n",
       "      <td>0.189781</td>\n",
       "      <td>0.196226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R T X CORP</th>\n",
       "      <td>0.383648</td>\n",
       "      <td>0.367470</td>\n",
       "      <td>0.375385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MONSTER BEVERAGE CORP NEW</th>\n",
       "      <td>0.540541</td>\n",
       "      <td>0.248447</td>\n",
       "      <td>0.340426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HILTON WORLDWIDE HOLDINGS INC</th>\n",
       "      <td>0.521277</td>\n",
       "      <td>0.426087</td>\n",
       "      <td>0.468900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TARGET CORP</th>\n",
       "      <td>0.060109</td>\n",
       "      <td>0.186441</td>\n",
       "      <td>0.090909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A T &amp; T INC</th>\n",
       "      <td>0.269231</td>\n",
       "      <td>0.329412</td>\n",
       "      <td>0.296296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WASTE MANAGEMENT INC DEL</th>\n",
       "      <td>0.355769</td>\n",
       "      <td>0.270073</td>\n",
       "      <td>0.307054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average</th>\n",
       "      <td>0.333635</td>\n",
       "      <td>0.304461</td>\n",
       "      <td>0.306108</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               precision    recall  fmeasure\n",
       "BOOKING HOLDINGS INC            0.496183  0.351351  0.411392\n",
       "PIONEER NATURAL RESOURCES CO    0.215385  0.229508  0.222222\n",
       "PAYPAL HOLDINGS INC             0.291080  0.446043  0.352273\n",
       "BOSTON SCIENTIFIC CORP          0.203125  0.189781  0.196226\n",
       "R T X CORP                      0.383648  0.367470  0.375385\n",
       "MONSTER BEVERAGE CORP NEW       0.540541  0.248447  0.340426\n",
       "HILTON WORLDWIDE HOLDINGS INC   0.521277  0.426087  0.468900\n",
       "TARGET CORP                     0.060109  0.186441  0.090909\n",
       "A T & T INC                     0.269231  0.329412  0.296296\n",
       "WASTE MANAGEMENT INC DEL        0.355769  0.270073  0.307054\n",
       "  average                       0.333635  0.304461  0.306108"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rouge_type = rouge_types[1]\n",
    "df = pd.concat(scores[rouge_type], axis=1)\n",
    "print(f\"{rouge_type.upper()} metric:\")\n",
    "pd.concat([df, df.T.mean().rename('  average')], axis=1).T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc71daf",
   "metadata": {},
   "source": [
    "## Role prompting\n",
    "\n",
    "Creating prompts based on the role or perspective of the person can be a useful technique for generating more relevant and engaging responses from language models.\n",
    "\n",
    "The __system prompt__ is an initial set of instructions that serve as the starting point when starting a new chat session. This defines things for the model and helps to focus its capabilities.\n",
    "The model will assume that role or person including their style.\n",
    "\n",
    "\n",
    "For example, we can generate a more user-friendly summary to help explain the risk disclosures in 10-K's.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fa5c021a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===== BOOKING HOLDINGS INC =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "Imagine you have some money in a special account. Sometimes, the value\n",
      "of that money can go up or down. This can happen because of things\n",
      "like changes in interest rates, like when you borrow money from a\n",
      "bank, or changes in the value of other countries' money, like when you\n",
      "buy something from a friend who lives in another country.\n",
      "\n",
      "We have to be careful with our money, so we make plans to keep it\n",
      "safe. We use special tools, like special contracts, to help keep our\n",
      "money safe. We also look at what's happening in the world, like what's\n",
      "happening with interest rates and other countries' money, to make sure\n",
      "we're making good choices.\n",
      "\n",
      "Sometimes, we might have to make some changes to our plans, like if\n",
      "the value of our money goes up or down. But we're always trying to\n",
      "keep our money safe and make good choices.\n",
      "\n",
      "===== PIONEER NATURAL RESOURCES CO =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "The company that makes oil and gas is like a big store that sells\n",
      "things. They have to worry about some risks, like if the price of oil\n",
      "goes up or down. This can affect how much money they have to use for\n",
      "important things like building new things or paying bills.\n",
      "\n",
      "The company tries to be smart about this by making deals with other\n",
      "companies to sell their oil and gas to them. This helps them get a\n",
      "better price for their oil and gas. They also have to be careful about\n",
      "who they sell to, so they don't get stuck with bad debts.\n",
      "\n",
      "The company also has to worry about the price of their own shares,\n",
      "like if someone buys or sells them. This can affect how much money\n",
      "they have.\n",
      "\n",
      "The company is like a big team that works together to make sure\n",
      "they're safe and smart about their business.\n",
      "\n",
      "===== PAYPAL HOLDINGS INC =====\n",
      "Hi there! So, you know how sometimes we buy things like toys or candy,\n",
      "and we have to pay for them? Well, grown-ups have to pay for things\n",
      "too, like money they borrow from banks. And just like how we might get\n",
      "a little worried if we can't find our favorite toy, grown-ups get\n",
      "worried if they can't pay back the money they borrowed.\n",
      "\n",
      "This is because there are things called \"risks\" that can happen, like\n",
      "if the price of something goes up or down. It's like if you bought a\n",
      "toy for $10, but then the price went up to $15. You would have to pay\n",
      "more money to get the same toy!\n",
      "\n",
      "Grown-ups have to worry about these risks so they can make sure they\n",
      "have enough money to pay back what they borrowed. They do this by\n",
      "using special tools, like contracts, to help them manage these risks.\n",
      "It's like having a special insurance policy for your toys, so you can\n",
      "feel safe and not worry about losing them.\n",
      "\n",
      "So, that's what this big text is talking about! It's like a big report\n",
      "card for grown-ups, showing how they're doing with managing these\n",
      "risks and making sure they have enough money.\n",
      "\n",
      "===== BOSTON SCIENTIFIC CORP =====\n",
      "Hi there! So, you know how we use money to buy things we need and\n",
      "want? Like toys and candy? Well, grown-ups use money to buy things\n",
      "too, like buildings and machines. And just like how we have to be\n",
      "careful with our money, grown-ups have to be careful with the money\n",
      "they use to buy things too.\n",
      "\n",
      "Sometimes, grown-ups use special kinds of money, called \"derivatives\",\n",
      "to help them make sure they have enough money to buy the things they\n",
      "need. It's like having a special kind of insurance that helps them\n",
      "feel safe and secure.\n",
      "\n",
      "The company I'm talking about, Boston Scientific, is a big company\n",
      "that makes medical devices, like machines that help people feel\n",
      "better. They use derivatives to help them make sure they have enough\n",
      "money to buy the things they need to make those machines.\n",
      "\n",
      "They also have to be careful with the money they use to buy things,\n",
      "because sometimes the value of that money can go up and down. It's\n",
      "like how the value of a toy can go up and down depending on how many\n",
      "people want it. So, they have to be careful and make sure they're\n",
      "making good choices with their money.\n",
      "\n",
      "That's it!\n",
      "\n",
      "===== R T X CORP =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "Imagine you have some money in a special account, and you're worried\n",
      "that the value of that money might change because of something called\n",
      "\"market risk\". This means that if the value of money changes, it could\n",
      "affect how much money you have.\n",
      "\n",
      "The company I work for has to deal with this kind of risk because we\n",
      "do business with people in different countries. This means that we\n",
      "have to worry about changes in the value of money between those\n",
      "countries.\n",
      "\n",
      "To help us deal with this risk, we use special tools called \"hedges\".\n",
      "These hedges are like insurance policies that help us keep the value\n",
      "of our money stable, even if the value of money changes.\n",
      "\n",
      "We also have to worry about something called \"interest rate risk\".\n",
      "This is when the interest rate on our loans or debts changes, and it\n",
      "could affect how much money we have.\n",
      "\n",
      "But don't worry, we have people who help us manage these risks and\n",
      "make sure we have enough money to do our work.\n",
      "\n",
      "===== MONSTER BEVERAGE CORP NEW =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "Our company makes yummy drinks and foods, and we need to buy special\n",
      "things like sugar, milk, and aluminum cans to make them. Sometimes,\n",
      "the prices of these things go up and down, and that can affect how\n",
      "much money we make. We try to be careful and not get too much money\n",
      "from these price changes. We also buy things from other countries, and\n",
      "the prices of those things can change too. We have special agreements\n",
      "with banks to help us with these changes. We have a lot of money in\n",
      "the bank, and we try to be smart about how we use it.\n",
      "\n",
      "===== HILTON WORLDWIDE HOLDINGS INC =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "Imagine you have some money in your piggy bank, and you're worried\n",
      "that it might change value because of something called \"interest\n",
      "rates\" or \"foreign currency exchange rates\". These are like big waves\n",
      "that can make your money go up or down.\n",
      "\n",
      "The company I work for has to deal with these waves too. We have to be\n",
      "careful so that our money doesn't get too big or too small. To do\n",
      "this, we use special tools called \"derivatives\" that help us keep our\n",
      "money safe.\n",
      "\n",
      "For example, if we have some money that's tied to a special interest\n",
      "rate, we might use a special contract to make sure our money doesn't\n",
      "get too big or too small. It's like having a special insurance policy\n",
      "to keep our money safe!\n",
      "\n",
      "We also have to deal with foreign currencies, like when you go on a\n",
      "trip and need to exchange your money for the local currency. We have\n",
      "to be careful so that our money doesn't get too big or too small\n",
      "because of these changes.\n",
      "\n",
      "That's what this text is talking about!\n",
      "\n",
      "===== TARGET CORP =====\n",
      "Here is a summary of the text for a first-grader:\n",
      "\n",
      "Our company is like a big team that works together. We have to make\n",
      "sure we're safe and sound, so we have insurance to protect us from bad\n",
      "things that might happen. We also have to pay taxes, like when you buy\n",
      "something and have to give some money to the government. We have to\n",
      "make sure we're being fair and honest with our money.\n",
      "\n",
      "We also have to take care of our team members, like you and your\n",
      "friends. We have special plans to help them when they get old or sick.\n",
      "We have to make sure we're being kind and helping each other.\n",
      "\n",
      "We also have to be careful with our money and make sure we're not\n",
      "taking too many risks. We have to make smart choices so we can be safe\n",
      "and happy.\n",
      "\n",
      "===== A T & T INC =====\n",
      "Hi there! So, you know how sometimes we talk about money and how it's\n",
      "used? Like, how much we have, how we spend it, and how we save it?\n",
      "Well, this text is about a big company called AT&T, and it's talking\n",
      "about how they use their money.\n",
      "\n",
      "They're saying that they have some risks, like changes in interest\n",
      "rates and foreign currencies, that can affect how much money they\n",
      "have. But they're also saying that they have ways to manage those\n",
      "risks, like using special contracts to protect themselves.\n",
      "\n",
      "They're also talking about how they use something called \"discount\n",
      "rates\" to figure out how much money they owe to people who are\n",
      "retired. It's kind of like a big math problem, and they have to make\n",
      "some guesses about what will happen in the future.\n",
      "\n",
      "The company is also saying that they have a special team that checks\n",
      "their books to make sure everything is correct. It's like a big audit,\n",
      "and they're making sure that everything is fair and honest.\n",
      "\n",
      "So, that's what this text is about! It's like a big report card for\n",
      "AT&T, and it's showing how they're doing with their money.\n",
      "\n",
      "===== WASTE MANAGEMENT INC DEL =====\n",
      "Here's a summary of the text for a first-grader:\n",
      "\n",
      "A big company that helps keep our planet clean has to deal with some\n",
      "risks. Just like how you might worry about losing your favorite toy,\n",
      "this company worries about things like interest rates, prices of\n",
      "things like diesel fuel and electricity, and how much money they get\n",
      "from other countries.\n",
      "\n",
      "They use special tools to try to make sure they're not too worried\n",
      "about these things. They also try to make sure they're doing good\n",
      "things for the planet, like making special gas for cars and trucks.\n",
      "\n",
      "The company is like a big team working together to make sure they're\n",
      "safe and doing good things. They're like a superhero, but instead of\n",
      "having superpowers, they have smart people working together to make\n",
      "the world a better place!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "role = \"You are a helpful first-grade teacher.\",\n",
    "prompt = \"Write a simple summary of the text for a first-grader.\"\n",
    "simple = {}\n",
    "for permno in docs.index:\n",
    "    print('=====', univ.loc[permno, 'comnam'], '=====')\n",
    "    simple[permno] = generate_response(found.loc[permno, 'item'],\n",
    "                                       role=role, prompt=prompt)\n",
    "    print(\"\\n\".join([textwrap.fill(s) for s in simple[permno].split('\\n')]))\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "env3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}