exercises/Extracting_Insights_Medical_Papers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14c14aa3",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/chris-lovejoy/CodingForMedicine/blob/main/exercises/Extracting_Insights_Medical_Papers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5316200d",
   "metadata": {},
   "source": [
    "# Extracting Insights from Medical Research Papers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325f8c8e",
   "metadata": {},
   "source": [
    "**In this exercise, we will use natural language processing (NLP) to extract key insights from academic medical papers.**\n",
    "\n",
    "We will use the latest GPT model from OpenAI and [a dataset of nearly 200,000 PubMed Articles](https://huggingface.co/datasets/ccdv/pubmed-summarization). We'll first generate a summary of the full article text, and then we'll use that summary to generate an abstract and answer questions about the paper.\n",
    "\n",
    "In this exercise, we'll learn how to:\n",
    "\n",
    "- **Perform various NLP tasks**, including abstractive summarisation, extractive summarisation, and question-answering\n",
    "- **Prepare large amounts of text data** for use by NLP models\n",
    "- Use OpenAI's GPT **large language model (LLM)**\n",
    "- **Query an API** (Application Program Interface)\n",
    "\n",
    "*Note: Here are some helpful resources that relate to this exercise:*\n",
    "- [Summarizing Papers With Python and GPT-3 (Article)](https://medium.com/geekculture/a-paper-summarizer-with-python-and-gpt-3-2c718bc3bc88)\n",
    "- [Recursively summarize text of any length with GPT-3 (Video)](https://www.youtube.com/watch?v=fflkFtIwQXo)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ea8ffd8",
   "metadata": {},
   "source": [
    "## Part 0: Installations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f410a657",
   "metadata": {},
   "source": [
    "Before we get started, there's going to be a few packages that we need to download. Many of these packages may be new because we're working with natural language processing now, which has lots of specialised packages.\n",
    "\n",
    "If you're running this on Google Colab, just run the cell below. If you're running this on a local computer, either run the cell below or from your terminal (inside your virtual environment)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed9432f2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! pip install datasets\n",
    "! pip install openai\n",
    "! pip install nltk\n",
    "! pip install textwrap3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0e09066",
   "metadata": {},
   "source": [
    "## Part 1: Loading and understanding our data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed8a8fa",
   "metadata": {},
   "source": [
    "For this exercise, we're going to use full-text versions of academic articles. \n",
    "\n",
    "Because large language models like GPT are trained on ['plain text'](https://en.wikipedia.org/wiki/Plain_text) from the internet, we'll need to ensure that the data we're working with is in that format. For example, we can't directly feed in PDF files. And some of the formattings that programs like Word might include will also be unhelpful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77961d7f",
   "metadata": {},
   "source": [
    "To make our lives easier, in this exercise, we're going to use text that's already been processed into a plain text format. We'll use the [Hugging Face library](https://huggingface.co/), which makes it easy to load data and models with only a few lines of code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdfe1a37",
   "metadata": {},
   "source": [
    "After [installing the Hugging Face library](https://huggingface.co/docs/transformers/installation), we can call the following to load a dataset of PubMed articles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f9197b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "dataset = load_dataset(\"ccdv/pubmed-summarization\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eff39a7",
   "metadata": {},
   "source": [
    "This is a relatively small dataset, but it's still 1.5GB and may take some time to load, depending on your computer and internet connection speed. In my case, it took just under 5 minutes. (Note: after the first time, your computer will cache it and won't need to re-download it the next time you come back to the exercise)\n",
    "\n",
    "You can browse other datasets accessible via Hugging Face [here](https://huggingface.co/datasets), and to load those, you just need to change the parameter in the ```load_dataset()``` function call above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc34b23",
   "metadata": {},
   "source": [
    "Let's have a look at our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "632ba1b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4945b758",
   "metadata": {},
   "source": [
    "We can see that it's divided into training, validation, and test sets. This is because it's designed for 'fine-tuning' NLP models. But we won't worry about that; we will just use some of the data in the 'training' portion. We can view it as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b8d48c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['train']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9103273",
   "metadata": {},
   "source": [
    "But now we want to find the actual article texts. It's not *obvious* from just looking at the Dataset object we have exactly how to do that. \n",
    "\n",
    "This will be the case sometimes, and we must figure out how to get the data we want. There may be documentation on how to do so. But sometimes, you'll need to experiment and figure it out.\n",
    "\n",
    "One helpful command for this is ```.__dict__```. This returns a dictionary object with all the information that our dataset contains. (Can't remember what a dictionary is? Check out the [Python Principles exercises](https://github.com/chris-lovejoy/CodingForMedicine/blob/main/exercises/Python_Principles.ipynb).) \n",
    "\n",
    "Let's use ```.__dict__```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b351b02e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset['train'].__dict__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a63a2ec",
   "metadata": {},
   "source": [
    "This may look like a bit of a jumble, but from within it, we can see the structure of the dataset. To access the data, we first index to the data point of interest, and then we select either 'article' or 'abstract'. Let's look at the 10th article:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a82712",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset['train'][10]['article']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e06f88d0",
   "metadata": {},
   "source": [
    "Great! It looks like an article about Alzheimer's Disease and blood-based markers for diagnosis. Now that we can access the full articles let's prepare our data for our large language model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc0bf95",
   "metadata": {},
   "source": [
    "**TASK**: In the cell below, use the ```.__dict__``` method to look at dataset, and then find the 20th article in the test dataset. What is it about?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca9efb34",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9f013cbd",
   "metadata": {},
   "source": [
    "## Step 2: Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bcc926e",
   "metadata": {},
   "source": [
    "Let's continue with our Alzheimer's Disease article to help us understand the data.\n",
    "\n",
    "If we look at the text above, we can notice that there are a lot of extra spaces and certain symbols that we don't normally have, like '\\n'. This reflects that text is divided into 'tokens' to perform NLP.\n",
    "\n",
    "Tokens are a set of characters that represents a \"unit of meaning\" in a text. Tokens are typically individual words but can also be phrases or other meaningful sequences of characters, such as numbers, symbols, or punctuation marks.\n",
    "\n",
    "If we were working with a new dataset, we might need to do the tokenisation ourselves. In this case, it's already done for us.\n",
    "\n",
    "A space separates each token. The '\\n' token denotes a new line.\n",
    "\n",
    "Using the Python ``` print()``` function will convert the '\\n' symbols into new lines, makings it easier to follow. However, the additional spacing will still be present.\n",
    "\n",
    "*(NOTE: we'll index the text to only look at the first 5000 characters, rather than print out the entire article below)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3831df91",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(dataset['train'][10]['article'][:5000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24919274",
   "metadata": {},
   "source": [
    "We're going to be feeding this text data into our language models. However, an important consideration here is the length of our text. Language models have limits on the length of text that they can take in at one point in time. \n",
    "\n",
    "For example, GPT-derived models can typically take a maximum of 1024-2048 tokens. This needs to include *both* the text we're providing and the accompanying command we will provide.\n",
    "\n",
    "We can look at the number of characters using ```len()```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05ce398",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "len(dataset['train'][10]['article'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d2cb31",
   "metadata": {},
   "source": [
    "However, there's no direct conversion of characters to 'tokens' given that the token length can vary.\n",
    "\n",
    "We can use the handy NLTK library for this. NLTK is the 'natural language toolkit' and contains various helpful functions, including tokenisation, text tagging, and more. \n",
    "\n",
    "Let's now do three things:\n",
    "1. Import the nltk library \n",
    "2. Download the 'punkt' resource. This is a popular 'tokenizer' model, which converts full text into tokens.\n",
    "3. Use the ```word_tokenize()``` function, which *does* the tokenisation using the punkt tokenizer that we just loaded. So now, rather than the full text, we can divide up into *tokens*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17a167ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "print(len(nltk.word_tokenize(dataset['train'][10]['article'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8eaaca1",
   "metadata": {},
   "source": [
    "**TASK**: So what does it *mean* that we've converted our text into tokens? In the cell below, use the ```ntlk.word tokenize()``` function on an article to see what the *tokenized* article looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "573e676e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "803d077e",
   "metadata": {},
   "source": [
    "Now - back to the tokenized length of training article 10:\n",
    "\n",
    "Nearly 4000 tokens... That will be a problem as it's far beyond our limit.\n",
    "\n",
    "**What can we do?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76a5822b",
   "metadata": {},
   "source": [
    "There are a few different approaches; the most appropriate approach depends on our end goal.\n",
    "\n",
    "One option is to use a tool like [GPT Index](https://gpt-index.readthedocs.io/en/latest/index.html). This tool divides the text into parts, which it calls \"indices\". Then, when you ask a question, it will identify which of the indices (ie. which segments of the original text) are the best for answering that particular question, and it will use that section to generate an answer.\n",
    "\n",
    "An alternative is to create a *summary* of the original text and then use that summary as the basis for future questions to the language model.\n",
    "\n",
    "The GPT Index approach is necessary if the text is very long. For example, if there were 10,000+ tokens in the original text, it wouldn't be possible to make a summary without losing key information.\n",
    "\n",
    "We need to condense to around 20-25% (from ~4000 tokens to ~1000 tokens), which is quite reasonable. Therefore, we'll go with summarisation, which is also easier to implement.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "925e383c",
   "metadata": {},
   "source": [
    "**TASK**: Before we move on to AI-based summarisation, let's do a little calculation. \n",
    "\n",
    "It's helpful to understand what the ratio of text characters to tokens is. So in the cell below, pick three articles from the training set, and calculate both the number of characters and the number of tokens. The functions for both of these are above.\n",
    "\n",
    "Then divide the characters by the tokens, and you'll see *roughly* how many characters there are per token.\n",
    "\n",
    "(As an optional extra, you could do the same for the ratio of *words* to tokens.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f74fc0f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "94f24624",
   "metadata": {},
   "source": [
    "## Part 3: Using AI to create an initial summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f69c1b41",
   "metadata": {},
   "source": [
    "### Extractive and Abstractive summarisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b3f7e32",
   "metadata": {},
   "source": [
    "There are broadly two types of summarisation: **extractive** and **abstractive** summarisation.\n",
    "\n",
    "In **extractive** summarisation, the model highlights the most important sentences in the text and cuts out all the rest. So the final summary has no *new* words and comprises all the important sentences.\n",
    "\n",
    "In **abstractive** summarisation, the model *creates* a new summary in its own words.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e067b67",
   "metadata": {},
   "source": [
    "In this exercise, we're going to use **abstractive** summarisation. \n",
    "\n",
    "Given the token limitations, we can't just ask the model to write an overall summary. So let's divide it into chunks, create a short summary of each chunk, and then combine the chunks to make the overall summary. The result won't be *perfect*, but hopefully, it contains all the information we need. Let's use the **textwrap3** library, which has a bunch of functions for manipulating lain text. The one we're going to use is called ```textwrap3.wrap()```, which converts some given text into a list of strings, where each string represents a line of text with a maximum width specified by width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06fee59f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import textwrap3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9161ebd",
   "metadata": {},
   "source": [
    "We can write a function that splits up our article into separate 'chunks'. Let's set out chunk length to 2000 characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ecdf8ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "chunk_length = 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "517b9603",
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunk_paper(paper):    \n",
    "    chunks = textwrap3.wrap(paper, chunk_length)    \n",
    "    return chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5792db56",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_paper = dataset['train'][10]['article']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aec64379",
   "metadata": {},
   "source": [
    "Let's look at first first 'chunk' as a sanity check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88952443",
   "metadata": {},
   "outputs": [],
   "source": [
    "chunk_paper(test_paper)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8a6be5",
   "metadata": {},
   "source": [
    "### Prompt Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf9a70d",
   "metadata": {},
   "source": [
    "For each of those chunks, we now want GPT to generate a summary.\n",
    "\n",
    "So **how do we do that?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f62fa1f2",
   "metadata": {},
   "source": [
    "The way that GPT-3 and similar models work is that you **ask them in 'natural language'** (i.e. using words). If you want to understand who is mentioned in a text, you could say something like, \"Read this text and list all the people it contains\". If you want to translate a text into another language, you could say, \"Re-write this paragraph into French\".\n",
    "\n",
    "This is great because it means the same model can perform many different types of tasks. Before GPT, you would often use models that specifically performed one thing.\n",
    "\n",
    "However, writing the \"prompts\" that give the response you want is something of an art. The model's responses can vary a lot depending on small changes in the instructions. (You can read more about this [here](https://gwern.net/gpt-3#prompts-as-programming).)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b987b27d",
   "metadata": {},
   "source": [
    "Here's a simple prompt that works quite well for our purposes:\n",
    "\n",
    "(Note: we're using *f* before the string here to make it a *formatted* string - check out [this](https://github.com/chris-lovejoy/CodingForMedicine/blob/main/exercises/Python_Principles_3.ipynb) Python Principles notebook if you're not confident on this.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad9b5048",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_prompt(chunk):\n",
    "    prompt = f\"Write a concise summary of the following: \\n \\n {chunk} \\n \\n CONCISE SUMMARY:\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46e60f5",
   "metadata": {},
   "source": [
    "Play around with your own prompts and compare how the outputs of the model vary.\n",
    "\n",
    "You can do that within this Jupyter Notebook, plus OpenAI has a \"playground\" to experiment in: https://platform.openai.com/playground\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a43915",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_prompt(chunk):\n",
    "    prompt = # TODO: write your own prompts here, and test them out below.\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801af7ba",
   "metadata": {},
   "source": [
    "### Using the GPT API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0d82c7",
   "metadata": {},
   "source": [
    "Now we're going to interact with our model. One option would be to load a model into our Jupyter Notebook and interact with it. An easier option, though, is to send our text to a language model 'API'.\n",
    "\n",
    "An 'API' is an \"Application Programming Interface\", which basically means it's somewhere that you can send information and receive information back.\n",
    "\n",
    "OpenAI has an API for the GPT models, so we can send our text there directly from within this notebook. To do that, we'll import the openai library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d119bd67",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ad55dcf",
   "metadata": {},
   "source": [
    "And we'll need to define our 'API key'. This is a long string of text which tells the OpenAI API *who* is sending the request. To get this, you'll need to make an account with OpenAI and generate a new API key, which you can copy into the cell below.\n",
    "\n",
    "One reason for the API key is to stop people from attacking their service with too many requests. But another reason is that this service isn't *free*, so they use the API key to know which account to charge.\n",
    "\n",
    "It's not *free*, but the cost for personal use is very low. A whole day of playing with the model and sending requests will cost less than a coffee (usually $0.50 or less - I'd say it's worth it for the educational experience).\n",
    "\n",
    "But suppose you're absolutely against spending money here. In that case, an alternative is to load a model from [Hugging Face](https://huggingface.co/models?pipeline_tag=summarization&sort=downloads) and use it in the notebook - see the documentation on their website for how to do so (it's a fair amount more work than calling OpenAI's API)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1adb526",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_key = \"\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d693ed3",
   "metadata": {},
   "source": [
    "To send the API request, there are a number of standard variables we need to provide. Have a look at the OpenAI API documentation [here](https://platform.openai.com/docs/api-reference/completions) and [here](https://platform.openai.com/docs/guides/chat) and fill out the function below.\n",
    "\n",
    "\"**Completion**\" here refers to the AI model *completing* the text. So if you start a sentence, it will complete it. Or if you write a question, it's *completion* will be the answer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cd56fd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gpt_completion(chunk):\n",
    "    result = openai.Completion.create(\n",
    "    model=\"\", # TODO: add model name here\n",
    "    prompt=generate_prompt(chunk),\n",
    "    max_tokens = 1000,\n",
    "    temperature = , # TODO: add an appropriate temperature here\n",
    "    n = 1\n",
    "    )\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4862c9bb",
   "metadata": {},
   "source": [
    "### Bringing it together"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e33005",
   "metadata": {},
   "source": [
    "We now have functions for (1) dividing our long text into chunks, (2) generating a prompt to summarise the chunk, and (3) asking OpenAI's GPT to perform that task.\n",
    "\n",
    "The final step is to run all of those functions over our full text to generate our summary.\n",
    "\n",
    "Fill out the gaps in the cell below to generate a summary for the Alzheimer's paper we've been looking at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22861206",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_summary(paper):\n",
    "    chunks = # TODO: use the relevant function to create the chunks\n",
    "    results = []\n",
    "    for chunk in chunks:\n",
    "        result = gpt_completion() # TODO: add the relevant argument here for the gpt_completion function call\n",
    "        chunk_summary = # TODO: look at what 'result' includes and index into the appropriate text we want\n",
    "        results.append(chunk_summary)\n",
    "    summary = ' '.join(results)\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "382b8bff",
   "metadata": {},
   "outputs": [],
   "source": [
    "alzheimers_paper = dataset['train'][10]['article']\n",
    "\n",
    "summary = create_summary(alzheimers_paper)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29e131ac",
   "metadata": {},
   "source": [
    "### Looking at our summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b817d08",
   "metadata": {},
   "source": [
    "We can look at the summary that our model created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7936d3fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fa9abc",
   "metadata": {},
   "source": [
    "Let's compare it's length with the original text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81260d3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The original text was {len(alzheimers_paper)} characters.\")\n",
    "print(f\"The summary text is {len(summary)} characters.\")\n",
    "\n",
    "import numpy as np\n",
    "print(f\"\\nThis is {np.round(len(summary)/len(alzheimers_paper) * 100,2)} percent of the original length.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eff6c28",
   "metadata": {},
   "source": [
    "That's a pretty decent compression."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3056e349",
   "metadata": {},
   "source": [
    "We should also do a visual inspection of the summary text. Does it seem like a reasonable representation?\n",
    "\n",
    "This is one of the challenges with NLP: numerical, objective performance measures can be harder because text is so varied. So it's always worth visually inspecting the text ourselves and seeing if it looks reasonable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c933b3e",
   "metadata": {},
   "source": [
    "If our text is still too long, one option is to do a second round of summarisation. We could also experiment with changing the prompt. We asked for a \"concise summary\", but could change it - for example, to \"very concise summary\", or specify a maximum number of sentences, etc. It's ultimately about trial-and-error, to see what gives the desired output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffec7d00",
   "metadata": {},
   "source": [
    "## Part 4: Using our summary to answer questions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44dc97f2",
   "metadata": {},
   "source": [
    "Now that we've generated an initial summary, we can use it to answer questions about the text and generate other summaries. For example, we could generate an Abstract using a set format (Background, Methods, Results, Conclusion) and compare this to the *true* abstract of the paper.\n",
    "\n",
    "To do this, we can continue using GPT and create new prompts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a31703a2",
   "metadata": {},
   "source": [
    "Let's first define a general function which can take in both our summary and our prompt function, and return the generated text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7609174d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gpt_complete_custom_prompt(summary, prompt_function):\n",
    "    result = openai.Completion.create(\n",
    "    model=\"text-davinci-003\",\n",
    "    prompt=prompt_function(summary),\n",
    "    max_tokens = 1000,\n",
    "    temperature = 0.25,\n",
    "    n = 1\n",
    "    )\n",
    "    return result['choices'][0]['text']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcea",
   "metadata": {},
   "source": [
    "### Creating an abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e3a8f77",
   "metadata": {},
   "source": [
    "Here is an example prompt for creating an abstract:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c528d15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt for creating full abstract\n",
    "def make_abstract_prompt(summary):\n",
    "    prompt = f\"I want you to act as an academic researcher writing an abstract for an academic article you wrote.\\\n",
    "    I will share a summary of the article and it will be your job to write the abstract. The abstract should have four sections: \\\n",
    "    background, materials and methods, results and conclusion. PAPER: \\n \\ ${summary} \\n \\ ABSTRACT: \\n BACKGROUND:\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbdc9a8b",
   "metadata": {},
   "source": [
    "Let's test it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d65f6c30",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"BACKGROUND:\" + gpt_complete_custom_prompt(summary, make_abstract_prompt)\n",
    "# NOTE: We've added \"BACKGROUND:\" to the output text, as it is used as part of the input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d8cc2b",
   "metadata": {},
   "source": [
    "**Looks pretty good!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceb964b6",
   "metadata": {},
   "source": [
    "**TASK**: Wrte your own prompt for making an abstract from the summary, and see if you can get something better than the above result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bebd0ad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt for creating full abstract\n",
    "def alternative_abstract_prompt(summary):\n",
    "    prompt = # TODO: write your alternative prompt here\n",
    "    return prompt\n",
    "\n",
    "\"BACKGROUND:\" + gpt_complete_custom_prompt(summary, alternative_abstract_prompt)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066e6653",
   "metadata": {},
   "source": [
    "### Question-answering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1725772e",
   "metadata": {},
   "source": [
    "We can also ask specific questions about the text. Below are two custom prompts - the first for identifying the medical conditions and the second for identifying the main findings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6b5d853",
   "metadata": {},
   "outputs": [],
   "source": [
    "def med_conds_prompt(summary):\n",
    "    prompt = f\"Look at the following text and identify what medical conditions are mentioned. \\n \\n {summary} \\n \\n MEDICAL CONDITIONS:\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ebb78fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "def main_findings_prompt(summary):\n",
    "    prompt = f\"Look at the following summary of a research study and identify what the main findings were. \\n \\n {summary} \\n \\n MAIN FINDINGS:\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8571415",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The medical conditions mentioned in the paper are:\")\n",
    "gpt_complete_custom_prompt(summary, med_conds_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "623419aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The main findings of the paper were:\")\n",
    "gpt_complete_custom_prompt(summary, main_findings_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c28c40a",
   "metadata": {},
   "source": [
    "This seems to be working pretty well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc3c4abe",
   "metadata": {},
   "source": [
    "**TASK**: play around with other prompts for asking other questions. Come up with at least three other prompts for different aspects of the paper.\n",
    "\n",
    "What seems to work well and what not so well?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9ab8763",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: write and test further prompts here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0eb9ee5",
   "metadata": {},
   "source": [
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c568eb7",
   "metadata": {},
   "source": [
    "1. **Play around with different prompts for obtaining different information**. Check out [this course](https://learnprompting.org/docs/intro) if you want more guidance on how to generate prompts.\n",
    "\n",
    "2. **Try different models and compare them to the GPT model we used**. For this, you can:\n",
    "    - Modify the \"model\" parameter when calling the \"openai.Completion.create()\" function\n",
    "    - Use a different API. For example, the [HuggingFace API](https://huggingface.co) is another popular API for large language models.\n",
    "    - Load in specific models and see how they perform. We used a general model here (GPT), but there are models fine-tuned for biomedical text, such as [biomedLM](https://github.com/stanford-crfm/BioMedLM) and [bio-clinical-BERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT). Read the documentation, implement them, and compare their performance against more general models like the above.\n",
    "\n",
    "3. **Look at performance metrics for NLP model performance**. How might we compare the generated abstracts with the real ones?\n",
    "4. **Try fine-tuning your model** for \"abstract generation\" on the whole dataset to see if its performance improves. Use the real paper abstracts as the \"ground truth\" training data. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd6d7509",
   "metadata": {},
   "source": [
    "Fill out the form below and we'll provide feedback on your code.\n",
    "\n",
    "**Any feedback on the exercise? Any questions? Want feedback on your code? Please fill out the form [here](https://docs.google.com/forms/d/e/1FAIpQLSdoOjVom8YKf11LxJ_bWN40afFMsWcoJ-xOrKhMbfBzgxTS9A/viewform).**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "data_science",
   "language": "python",
   "name": "data_science"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}