Week1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week1",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PkgodRARslFI",
        "colab_type": "text"
      },
      "source": [
        "Preprocessing with NLTK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwZT94Dze0iC",
        "colab_type": "text"
      },
      "source": [
        "**Tokenization **"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v7_Rmefbe30m",
        "colab_type": "text"
      },
      "source": [
        "Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens , perhaps at the same time throwing away certain characters, such as punctuation. Here is an example of tokenization:\n",
        "\n",
        "Input: The best thing in the world!\n",
        "Output: The, best, thing, in, the, world"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bk7pRJEUiIbI",
        "colab_type": "code",
        "outputId": "326ec317-8d89-497b-d2c1-78cc1eefde27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Id1G_Kgxg89h",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BT1XNoKNhPB0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.text import Text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-A6_8WDmhT3o",
        "colab_type": "code",
        "outputId": "4004ad9d-692d-43e3-e444-bd96f5f1ba95",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "my_string = \"Brown skin girl Your skin just like pearls The best thing in the world Never trade guy anybody you for anybody else Singin' brown skin girl Your skin just like pearls The best thing in the world I never trade you for anybody else, singin'\"\n",
        "tokens = word_tokenize(my_string)\n",
        "\n",
        "tokens = [word.lower() for word in tokens]\n",
        "tokens[:8]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['brown', 'skin', 'girl', 'your', 'skin', 'just', 'like', 'pearls']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VLdJBmGM8uK9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jkSUJRwc5koW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pip install nltk "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BCYiRYljHx1",
        "colab_type": "text"
      },
      "source": [
        "Use the Text() method to convert back to text"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "He3wJBdAit1a",
        "colab_type": "code",
        "outputId": "2ee95af9-c4ac-453f-92b9-c258e3ea823b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "back_to_text = Text(tokens)\n",
        "back_to_text"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Text: brown skin girl your skin just like pearls...>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_2LcuKQUjZDf",
        "colab_type": "code",
        "outputId": "9dd6a515-7e2e-476a-804d-c95ec4e97b34",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "\n",
        "back_to_text.concordance('pearls')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Displaying 2 of 2 matches:\n",
            "                                    pearls the best thing in the world never t\n",
            "brown skin girl your skin just like pearls the best thing in the world i never\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eT3qjCbbAYeh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "string_2 = \"now now Many of Shakespeare's plays were published in editions of varying quality and accuracy in his lifetime. However, in 1623, two fellow actors and friends of Shakespeare's, John Heminges and Henry Condell, published a more definitive text known as the First Folio, a posthumous collected edition of Shakespeare's dramatic works that included all but two of his plays.Throughout the 20th and 21st centuries, Shakespeare's works have been continually adapted and rediscovered by new movements in scholarship and performance. His plays remain popular and are studied, performed, and reinterpreted through various cultural and political contexts around the world, Few records of Shakespeare's private life survive; this has stimulated considerable speculation about such matters as his physical appearance, his sexuality, his religious beliefs, and whether the works attributed to him were written by others\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EXSwuOLvAw_Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "t_lower = string_2.split(' ')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFMHyveGBez0",
        "colab_type": "code",
        "outputId": "38e5a4be-b54d-400a-ae70-019f7e1d0401",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "t_lower = Text(t_lower)\n",
        "print(t_lower)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Text: now now Many of Shakespeare's plays were published...>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SD8v1IqQBolw",
        "colab_type": "code",
        "outputId": "383da03c-30e0-477d-a2f6-027346f854f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "v = t_lower.concordance(\"of\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Displaying 6 of 6 matches:\n",
            "                                      of Shakespeare's plays were published in\n",
            "                                      of varying quality and accuracy in his l\n",
            "n 1623, two fellow actors and friends of Shakespeare's, John Heminges and Henr\n",
            "Folio, a posthumous collected edition of Shakespeare's dramatic works that inc\n",
            "matic works that included all but two of his plays.Throughout the 20th and 21s\n",
            "ontexts around the world, Few records of Shakespeare's private life survive; t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pz-sdlu0EH6o",
        "colab_type": "code",
        "outputId": "a2d70137-a001-4b8d-fe0e-ed9292744359",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "t_lower"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Text:  now now Many of Shakespeare's plays were...>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4rfCVyKjnmx3",
        "colab_type": "code",
        "outputId": "69f02c47-efc2-48c2-9eb1-12ca9a3e1f12",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "back_to_text.count('girl')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "21ihJ6WInzLc",
        "colab_type": "code",
        "outputId": "172f1bde-1031-466e-d758-fe6dc3b03fda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "t_lower.index('of')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qCQHBX4nscQW",
        "colab_type": "code",
        "outputId": "a94bc821-2a46-4fad-f5bd-7f184c11acbb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "t_lower.vocab()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "FreqDist({'1623,': 1,\n",
              "          '20th': 1,\n",
              "          '21st': 1,\n",
              "          'Condell,': 1,\n",
              "          'Few': 1,\n",
              "          'First': 1,\n",
              "          'Folio,': 1,\n",
              "          'Heminges': 1,\n",
              "          'Henry': 1,\n",
              "          'His': 1,\n",
              "          'However,': 1,\n",
              "          'John': 1,\n",
              "          'Many': 1,\n",
              "          \"Shakespeare's\": 4,\n",
              "          \"Shakespeare's,\": 1,\n",
              "          'a': 2,\n",
              "          'about': 1,\n",
              "          'accuracy': 1,\n",
              "          'actors': 1,\n",
              "          'adapted': 1,\n",
              "          'all': 1,\n",
              "          'and': 10,\n",
              "          'appearance,': 1,\n",
              "          'are': 1,\n",
              "          'around': 1,\n",
              "          'as': 2,\n",
              "          'attributed': 1,\n",
              "          'been': 1,\n",
              "          'beliefs,': 1,\n",
              "          'but': 1,\n",
              "          'by': 2,\n",
              "          'centuries,': 1,\n",
              "          'collected': 1,\n",
              "          'considerable': 1,\n",
              "          'contexts': 1,\n",
              "          'continually': 1,\n",
              "          'cultural': 1,\n",
              "          'definitive': 1,\n",
              "          'dramatic': 1,\n",
              "          'edition': 1,\n",
              "          'editions': 1,\n",
              "          'fellow': 1,\n",
              "          'friends': 1,\n",
              "          'has': 1,\n",
              "          'have': 1,\n",
              "          'him': 1,\n",
              "          'his': 5,\n",
              "          'in': 4,\n",
              "          'included': 1,\n",
              "          'known': 1,\n",
              "          'life': 1,\n",
              "          'lifetime.': 1,\n",
              "          'matters': 1,\n",
              "          'more': 1,\n",
              "          'movements': 1,\n",
              "          'new': 1,\n",
              "          'now': 2,\n",
              "          'of': 6,\n",
              "          'others': 1,\n",
              "          'performance.': 1,\n",
              "          'performed,': 1,\n",
              "          'physical': 1,\n",
              "          'plays': 2,\n",
              "          'plays.Throughout': 1,\n",
              "          'political': 1,\n",
              "          'popular': 1,\n",
              "          'posthumous': 1,\n",
              "          'private': 1,\n",
              "          'published': 2,\n",
              "          'quality': 1,\n",
              "          'records': 1,\n",
              "          'rediscovered': 1,\n",
              "          'reinterpreted': 1,\n",
              "          'religious': 1,\n",
              "          'remain': 1,\n",
              "          'scholarship': 1,\n",
              "          'sexuality,': 1,\n",
              "          'speculation': 1,\n",
              "          'stimulated': 1,\n",
              "          'studied,': 1,\n",
              "          'such': 1,\n",
              "          'survive;': 1,\n",
              "          'text': 1,\n",
              "          'that': 1,\n",
              "          'the': 4,\n",
              "          'this': 1,\n",
              "          'through': 1,\n",
              "          'to': 1,\n",
              "          'two': 2,\n",
              "          'various': 1,\n",
              "          'varying': 1,\n",
              "          'were': 2,\n",
              "          'whether': 1,\n",
              "          'works': 3,\n",
              "          'world,': 1,\n",
              "          'written': 1})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3VZBo9avmke",
        "colab_type": "text"
      },
      "source": [
        "**n-gram** is a contiguous sequence of n items from a given sample of text or speech."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H-SUGCOGvot0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "s = \"The best thing in the world\"\n",
        "s = s.lower()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6WikZniGwBdB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0_szyS0wKs7",
        "colab_type": "text"
      },
      "source": [
        "# RegEx"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MuSwC1K-wOew",
        "colab_type": "text"
      },
      "source": [
        "RegEx is used to check if a string contains the specified search pattern."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KUoi8Vl8w_sI",
        "colab_type": "text"
      },
      "source": [
        "Assignment, study and practise RegEx with python "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cAYkT2DyzJUg",
        "colab_type": "text"
      },
      "source": [
        "Next Class: N-gram, Stemming, lemmatizing, Parts of speech "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ayk1y77VKBo7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3mDjjXgKd6d",
        "colab_type": "code",
        "outputId": "75d4ed18-d277-4392-feaf-3094dc4755d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "txt = \"The rain in Spain the book\"\n",
        "x = re.search(\"^The.*book$\", txt)\n",
        "x"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<_sre.SRE_Match object; span=(0, 26), match='The rain in Spain the book'>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YAIO5-QqLel8",
        "colab_type": "code",
        "outputId": "b98079dd-fe93-4bee-c24a-482628c42bf1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "re.findall(\"in \", new_data)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['in ', 'in ']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OUZQ084OMqzm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "new_data = (\"The rainy in Spain the book\")"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}