Summarization With Map Reduce Using Custom Template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "53c524b4",
   "metadata": {},
   "source": [
    "### Summarization with LangChain and OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "966ba0d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -r ./requirements.txt -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f87112a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip show langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aeb28c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install langchain --upgrade -q\n",
    "# For working with unstructured pdf load, install the below dependencies\n",
    "# !pip install unstructured\n",
    "# !pip install pdf2image\n",
    "# !pip install pdfminer\n",
    "# !pip install pdfminer.six "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "884d9093",
   "metadata": {},
   "source": [
    "### Environment Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "311e6e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv, find_dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "eaa9e516",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_dotenv(find_dotenv(), override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4215ef61",
   "metadata": {},
   "source": [
    "### Summarization with LangChain & OpenAI using Map Reduce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "daa1bab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import PromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.chains.summarize import load_summarize_chain\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b75766c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature=0, model_name='gpt-3.5-turbo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0c5008d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./CNNs.txt', encoding='utf-8') as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1de785fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2022"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm.get_num_tokens(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "bad32527",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=5000, chunk_overlap=50)\n",
    "chunks = text_splitter.create_documents([text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "594ba857",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "712da434",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = load_summarize_chain(\n",
    "    llm,\n",
    "    chain_type='map_reduce',\n",
    "    verbose=False\n",
    ")\n",
    "summary = chain.run(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1dbd13c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Convolutional Neural Networks (CNNs) are widely used in image recognition tasks due to their ability to encode image-specific features and simplify the architecture of artificial neural networks. They are important in computer vision applications such as self-driving cars and face recognition, as they accurately classify and detect objects in images. CNNs are designed to handle image data and use the convolution operation as a fundamental building block. They can detect patterns in images by searching for specific pixel values and exploit knowledge about the specific type of input to simplify the network architecture.\n"
     ]
    }
   ],
   "source": [
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2e5fc7f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Write a concise summary of the following:\\n\\n\\n\"{text}\"\\n\\n\\nCONCISE SUMMARY:'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.llm_chain.prompt.template"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "36b14960",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Write a concise summary of the following:\\n\\n\\n\"{text}\"\\n\\n\\nCONCISE SUMMARY:'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.combine_document_chain.llm_chain.prompt.template"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751533cb",
   "metadata": {},
   "source": [
    "### Summarization with LangChain & OpenAI using Map Reduce & Custom Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a9ef22f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "map_custom_prompt = '''\n",
    "Summarize the following text in a clear and concise way:\n",
    "TEXT:`{text}`\n",
    "Brief Summary:\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3f619816",
   "metadata": {},
   "outputs": [],
   "source": [
    "map_prompt_template = PromptTemplate(\n",
    "    input_variables=['text'],\n",
    "    template = map_custom_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "86ccf35b",
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_custom_prompt='''\n",
    "Generate a summary of the following text that includes the following elements:\n",
    "\n",
    "* A title that accurately reflects the content of the text.\n",
    "* An introduction paragraph that provides an overview of the topic.\n",
    "* Bullet points that list the key points of the text.\n",
    "* A conclusion paragraph that summarizes the main points of the text.\n",
    "\n",
    "Text:`{text}`\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0a9178c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_prompt_template = PromptTemplate(\n",
    "    input_variables=['text'],\n",
    "    template = combine_custom_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "527a286b",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_chain = load_summarize_chain (\n",
    "    llm=llm,\n",
    "    chain_type='map_reduce',\n",
    "    map_prompt = map_prompt_template,\n",
    "    combine_prompt=combine_prompt_template,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1bed060d",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary = summary_chain.run(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f1a09c67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Understanding Convolutional Neural Networks (CNNs) for Image Recognition\n",
      "\n",
      "Introduction:\n",
      "This text provides an overview of convolutional neural networks (CNNs) and their significance in computer vision applications. It highlights the challenges of working with large images and the need for CNNs to handle these images effectively.\n",
      "\n",
      "Key Points:\n",
      "- CNNs are commonly used for image recognition tasks in computer vision.\n",
      "- They mimic the brain's recognition process by detecting and categorizing features.\n",
      "- CNNs are crucial in applications like self-driving cars and face recognition.\n",
      "- They can be used for image classification and object detection.\n",
      "- CNNs are designed to handle large images and overcome the challenges associated with them.\n",
      "- The structure of CNNs includes an input layer, the convolutional neural network itself, and an output layer.\n",
      "- CNNs can be used for tasks beyond image classification, such as detecting human emotions in images.\n",
      "- CNNs scan images by representing black and white images as 2-dimensional arrays and colored images as 3-dimensional arrays.\n",
      "- CNNs exploit knowledge about the specific input type to simplify the network architecture.\n",
      "\n",
      "Conclusion:\n",
      "In conclusion, convolutional neural networks (CNNs) are essential tools in computer vision for image recognition tasks. They mimic the brain's recognition process and can handle large images effectively. CNNs have various applications and can be used for tasks beyond image classification. Understanding the structure and operation of CNNs is crucial for leveraging their capabilities in computer vision.\n"
     ]
    }
   ],
   "source": [
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a8ab7e",
   "metadata": {},
   "source": [
    "### Another Use Case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "279dc69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "map_custom_prompt='''\n",
    "Extracts the Important keywords from each chunk:\n",
    "TEXT:`{text}`\n",
    "List of Keywords:\n",
    "'''\n",
    "\n",
    "map_prompt_template = PromptTemplate (\n",
    "    input_variables=['text'],\n",
    "    template=map_custom_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6c8bb204",
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_custom_prompt='''\n",
    "Counts the number of times each important keyword appears in the document & Find the totoal number of keywords:\n",
    "\n",
    "Text:`{text}`\n",
    "'''\n",
    "\n",
    "combine_prompt_template = PromptTemplate(\n",
    "    template=combine_custom_prompt, \n",
    "    input_variables=['text']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "91622532",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_chain = load_summarize_chain (\n",
    "    llm=llm,\n",
    "    chain_type='map_reduce',\n",
    "    map_prompt=map_prompt_template,\n",
    "    combine_prompt=combine_prompt_template,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "42679227",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The important keywords and their respective counts are as follows:\n",
      "\n",
      "- Convolutional Neural Networks: 4\n",
      "- images: 4\n",
      "- neural network applications: 1\n",
      "- pattern recognition tasks: 1\n",
      "- image-specific features: 1\n",
      "- brain: 1\n",
      "- objects: 3\n",
      "- people: 1\n",
      "- recognition: 2\n",
      "- artificial vision systems: 1\n",
      "- optical illusions: 1\n",
      "- features: 2\n",
      "- categorization: 1\n",
      "- angle: 1\n",
      "- object detection: 2\n",
      "- computer vision: 3\n",
      "- deep learning: 1\n",
      "- self-driving cars: 1\n",
      "- pedestrians: 1\n",
      "- face recognition: 1\n",
      "- phone: 1\n",
      "- apps: 1\n",
      "- image classifications: 1\n",
      "- image recognition: 1\n",
      "- position: 1\n",
      "- cars: 1\n",
      "- boxes: 1\n",
      "- multiple cars: 1\n",
      "- distance: 1\n",
      "- computer vision problems: 1\n",
      "- inputs: 1\n",
      "- 64 by 64 images: 1\n",
      "- 64 by 64 by three: 1\n",
      "- 12288: 1\n",
      "- X: 1\n",
      "- input features: 1\n",
      "- 1000-pixel by 1000-pixel image: 1\n",
      "- one megapixel: 1\n",
      "- three RGB channels: 1\n",
      "- three million: 1\n",
      "- first hidden layer: 1\n",
      "- 1000 hidden units: 1\n",
      "- three billion parameters: 1\n",
      "- overfitting: 1\n",
      "- computational requirements: 1\n",
      "- memory requirements: 1\n",
      "- convolution operation: 1\n",
      "- fundamental building blocks: 1\n",
      "- input image: 1\n",
      "- output layer: 1\n",
      "- information flow: 1\n",
      "- categorize images: 1\n",
      "- human emotions: 1\n",
      "- classification: 1\n",
      "- probability: 1\n",
      "- facial expressions: 1\n",
      "- training: 1\n",
      "- scanning images: 1\n",
      "- black and white image: 1\n",
      "- colored image: 1\n",
      "- two-dimensional array: 1\n",
      "- RGB layers: 1\n",
      "- intensity: 1\n",
      "- red layer: 1\n",
      "- blue layer: 1\n",
      "- green layer: 1\n",
      "- color pixel: 1\n",
      "- maroon pixel: 1\n",
      "- smiley face image: 1\n",
      "- white pixel: 1\n",
      "- black pixel: 1\n",
      "- detecting smiles: 1\n",
      "- patterns: 1\n",
      "- CNNs: 1\n",
      "- problem domain: 1\n",
      "- network architecture: 1\n",
      "- end: 1\n",
      "- story: 1\n",
      "- next: 1\n",
      "- talking: 1\n",
      "- detail: 1\n",
      "- operation: 1\n",
      "- stay tuned: 1\n",
      "\n",
      "The total number of keywords is 87.\n"
     ]
    }
   ],
   "source": [
    "summary=summary_chain.run(chunks)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e158d74",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9357cf6e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa3aed2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc69fd80",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adca7604",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48dc2184",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d3b1ad3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c489fee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3208947",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0378907c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be97b536",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}