Neurons_with_recurrence.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyOE21I5M+dY9GI+JlgP3LsV",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Antony-gitau/machine_learning_playground/blob/main/Neurons_with_recurrence.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "I am following the [MIT 6.S191 lecture on recurrent neural network](https://youtu.be/ySEx_Bqxvvo) and taking some notes and here I document them.\n",
        "\n",
        "Sequence modelling applications:\n",
        "- machine translation\n",
        "- image captioning\n",
        "- semantic classification\n",
        "\n",
        "\n",
        "Neuron with recurrence\n",
        "- RNN\n",
        "\n",
        "pseudocode of an RNN\n",
        "\n",
        "1. Define the rnn;\n",
        "my_rnn = RNN()\n",
        "2. iterate through all the inputs\n",
        "3. calculate and update the hidden state using an activation function\n",
        "4. generate a predicted output.\n",
        "\n",
        "design criteria for developing networks for sequence modelling:\n",
        "- handle variable lengths\n",
        "- track long dependencies\n",
        "- maintain information about the order of the sequence\n",
        "- share parameters across the sequence\n",
        "\n",
        "example:\n",
        "predicting the next word.\n",
        "\n",
        "1. represent language to a neural network\n",
        "- represent words as numerical representation.\n",
        "\n",
        "one way to represent words as input vectors of a neural network, we use a one hot encoding technique. By one hot encoding we mean, taking a count of every word in a single vector and identifying the word with a 1 and 0 everywhere else. e.g [0,1,0,0] is a one hot vector of a word in the second index (that is appearing second on the count of words in the sequence)\n",
        "\n",
        "2. Training and learning through neural networks\n",
        "\n",
        "- backpropagation through time.\n",
        "\n",
        "challenges:\n",
        "1. exploding gradients\n",
        "- the gradient gets bigger and bigger until its unfeaseble to calculate it, and by extension, training a model becomes unstable.\n",
        "2. vanishing gradients\n",
        "- the gradient on the other hand gets smaller and smaller, until it becomes insignificant.\n",
        "\n",
        "tricks to overcome the challenges:\n",
        "1. changing activation functions\n",
        "e.g ReLU is an a function that prevents the gradient from shrinking\n",
        "2. parameter initialization\n",
        "3. introducing gated cells.\n",
        "select flow of information in the neural network. like the LSTMs\n",
        "\n",
        "applications and limitations of RNN\n",
        "\n",
        "Music generation\n",
        "- Design an RNN that can predict the next musical note.\n",
        "\n",
        "limitation\n",
        "- encoding bottleneck\n",
        "- no easy parallelization techniques\n",
        "- not that long memory for quite long sequences, like the 10,000s of words\n",
        "\n",
        "Attention is all you need:\n",
        "- attend to the most import part of an input example.\n",
        "- extract the features deserve the highest attention.\n",
        "\n",
        "\n",
        "Let jump into a practical section drawing inspiration from [music generation with RNN lab](https://github.com/aamini/introtodeeplearning/blob/master/lab1/Part2_Music_Generation.ipynb) by MIT Introduction to Deep learning course.\n",
        "\n",
        "The goal is to train a model to generate new music from learning the patterns in raw sheet music.\n",
        "\n"
      ],
      "metadata": {
        "id": "cpJyUI9IDMQF"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "VuvW5pEkDE0K"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "%tensorflow_version 2.x #ensuring we are using any tensorflow 2. something version\n",
        "import tensorflow as tf\n",
        "\n",
        "# the data we are using lives in mitdeeplearning package\n",
        "!pip install mitdeeplearning\n",
        "import mitdeeplearning as mdl\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data:\n",
        "- the mitdeeplearning package has an irish folk song data set that has 817 songs.\n"
      ],
      "metadata": {
        "id": "hwReI4pgjox3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "songs = mdl.lab1.load_training_data()\n",
        "first_song = songs[0]\n",
        "print(\"This is just an example\\n \", first_song)\n",
        "second_song = songs[1]\n",
        "print(\"second song: \", second_song)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nipusg5rj3BH",
        "outputId": "2fee3c38-be63-4d82-fce3-23e6fcebd73a"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 817 songs in text\n",
            "This is just an example\n",
            "  X:1\n",
            "T:Alexander's\n",
            "Z: id:dc-hornpipe-1\n",
            "M:C|\n",
            "L:1/8\n",
            "K:D Major\n",
            "(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\n",
            "dAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\n",
            "AG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\n",
            "FAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\n",
            "second song:  X:2\n",
            "T:An Buachaill Dreoite\n",
            "Z: id:dc-hornpipe-2\n",
            "M:C|\n",
            "L:1/8\n",
            "K:G Major\n",
            "GF|DGGB d2GB|d2GF Gc (3AGF|DGGB d2GB|dBcA F2GF|!\n",
            "DGGB d2GF|DGGF G2Ge|fgaf gbag|fdcA G2:|!\n",
            "GA|B2BG c2cA|d2GF G2GA|B2BG c2cA|d2DE F2GA|!\n",
            "B2BG c2cA|d^cde f2 (3def|g2gf gbag|fdcA G2:|!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#converting the abc notation of the songs to audio file\n",
        "play_first_song = mdl.lab1.play_song(first_song)\n",
        "play_first_song"
      ],
      "metadata": {
        "id": "ZhEuJzWzkPo2"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Important questions:\n",
        "\n",
        "how does the number of different characters present in the text file impact the complexity of the learning problem?\n",
        "\n"
      ],
      "metadata": {
        "id": "BqOjn2B_lSBT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#join the songs leaving a blank line between them\n",
        "joined_songs = \"\\n\\n\".join(songs)\n",
        "\n",
        "# lets get unique characters from the list of songs we just joined\n",
        "vocab = sorted(set(joined_songs))\n",
        "print(\"we have \", len(vocab), \"unique characters in the irish folk songs dataset\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNd-e3jbkpsZ",
        "outputId": "8e77dc7b-a579-4e30-d121-68ca91c45e47"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "we have  83 unique characters in the irish folk songs dataset\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "vocab"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hWgRn-PJnNWi",
        "outputId": "3133413c-2d69-418d-8414-56cc3f9244e3"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['\\n',\n",
              " ' ',\n",
              " '!',\n",
              " '\"',\n",
              " '#',\n",
              " \"'\",\n",
              " '(',\n",
              " ')',\n",
              " ',',\n",
              " '-',\n",
              " '.',\n",
              " '/',\n",
              " '0',\n",
              " '1',\n",
              " '2',\n",
              " '3',\n",
              " '4',\n",
              " '5',\n",
              " '6',\n",
              " '7',\n",
              " '8',\n",
              " '9',\n",
              " ':',\n",
              " '<',\n",
              " '=',\n",
              " '>',\n",
              " 'A',\n",
              " 'B',\n",
              " 'C',\n",
              " 'D',\n",
              " 'E',\n",
              " 'F',\n",
              " 'G',\n",
              " 'H',\n",
              " 'I',\n",
              " 'J',\n",
              " 'K',\n",
              " 'L',\n",
              " 'M',\n",
              " 'N',\n",
              " 'O',\n",
              " 'P',\n",
              " 'Q',\n",
              " 'R',\n",
              " 'S',\n",
              " 'T',\n",
              " 'U',\n",
              " 'V',\n",
              " 'W',\n",
              " 'X',\n",
              " 'Y',\n",
              " 'Z',\n",
              " '[',\n",
              " ']',\n",
              " '^',\n",
              " '_',\n",
              " 'a',\n",
              " 'b',\n",
              " 'c',\n",
              " 'd',\n",
              " 'e',\n",
              " 'f',\n",
              " 'g',\n",
              " 'h',\n",
              " 'i',\n",
              " 'j',\n",
              " 'k',\n",
              " 'l',\n",
              " 'm',\n",
              " 'n',\n",
              " 'o',\n",
              " 'p',\n",
              " 'q',\n",
              " 'r',\n",
              " 's',\n",
              " 't',\n",
              " 'u',\n",
              " 'v',\n",
              " 'w',\n",
              " 'x',\n",
              " 'y',\n",
              " 'z',\n",
              " '|']"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "preprocessing:\n",
        "\n",
        "- we are asking the model: given a sequence of characters, what is the most probable next one? This is the goal of this model development.\n",
        "\n",
        "- the data we have is ABC notation, and we want the RNN to learn the pattern.\n",
        "\n",
        "\n",
        "so,\n",
        "\n",
        "1. we need to vectorize the text.\n",
        "- creating a numerical representation of the musical text.\n",
        "- we will therefore generate two lookup tables: one to map the characters to numbers and the other will map numbers back to characters.\n",
        "\n",
        "\n",
        "notes on the dictionary comprehension below:\n",
        "\n",
        "its equivalent for loop\n",
        "\n",
        "      char2indx = {}\n",
        "      for ind, ch in enumerate(vocab):\n",
        "         char2indx[ch] = ind\n"
      ],
      "metadata": {
        "id": "CYJmOwaLnpWu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "# mapping characters to unique index\n",
        "char2indx = {ch:indx for indx, ch in enumerate(vocab)}\n",
        "\n",
        "# now we move from the unique index to the characters in vocab list\n",
        "indx2char = np.array(vocab)"
      ],
      "metadata": {
        "id": "Tb0H1YUjncHJ"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "char2indx"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W3UrOAKGum2N",
        "outputId": "28fd5854-e633-4739-ac6f-87a276df0f30"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'\\n': 0,\n",
              " ' ': 1,\n",
              " '!': 2,\n",
              " '\"': 3,\n",
              " '#': 4,\n",
              " \"'\": 5,\n",
              " '(': 6,\n",
              " ')': 7,\n",
              " ',': 8,\n",
              " '-': 9,\n",
              " '.': 10,\n",
              " '/': 11,\n",
              " '0': 12,\n",
              " '1': 13,\n",
              " '2': 14,\n",
              " '3': 15,\n",
              " '4': 16,\n",
              " '5': 17,\n",
              " '6': 18,\n",
              " '7': 19,\n",
              " '8': 20,\n",
              " '9': 21,\n",
              " ':': 22,\n",
              " '<': 23,\n",
              " '=': 24,\n",
              " '>': 25,\n",
              " 'A': 26,\n",
              " 'B': 27,\n",
              " 'C': 28,\n",
              " 'D': 29,\n",
              " 'E': 30,\n",
              " 'F': 31,\n",
              " 'G': 32,\n",
              " 'H': 33,\n",
              " 'I': 34,\n",
              " 'J': 35,\n",
              " 'K': 36,\n",
              " 'L': 37,\n",
              " 'M': 38,\n",
              " 'N': 39,\n",
              " 'O': 40,\n",
              " 'P': 41,\n",
              " 'Q': 42,\n",
              " 'R': 43,\n",
              " 'S': 44,\n",
              " 'T': 45,\n",
              " 'U': 46,\n",
              " 'V': 47,\n",
              " 'W': 48,\n",
              " 'X': 49,\n",
              " 'Y': 50,\n",
              " 'Z': 51,\n",
              " '[': 52,\n",
              " ']': 53,\n",
              " '^': 54,\n",
              " '_': 55,\n",
              " 'a': 56,\n",
              " 'b': 57,\n",
              " 'c': 58,\n",
              " 'd': 59,\n",
              " 'e': 60,\n",
              " 'f': 61,\n",
              " 'g': 62,\n",
              " 'h': 63,\n",
              " 'i': 64,\n",
              " 'j': 65,\n",
              " 'k': 66,\n",
              " 'l': 67,\n",
              " 'm': 68,\n",
              " 'n': 69,\n",
              " 'o': 70,\n",
              " 'p': 71,\n",
              " 'q': 72,\n",
              " 'r': 73,\n",
              " 's': 74,\n",
              " 't': 75,\n",
              " 'u': 76,\n",
              " 'v': 77,\n",
              " 'w': 78,\n",
              " 'x': 79,\n",
              " 'y': 80,\n",
              " 'z': 81,\n",
              " '|': 82}"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "indx2char"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SSBnak_2uoed",
        "outputId": "f1d4aca9-ab6a-4d06-ab4c-060b95e4c4b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['\\n', ' ', '!', '\"', '#', \"'\", '(', ')', ',', '-', '.', '/', '0',\n",
              "       '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', '<', '=', '>',\n",
              "       'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',\n",
              "       'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',\n",
              "       '[', ']', '^', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i',\n",
              "       'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v',\n",
              "       'w', 'x', 'y', 'z', '|'], dtype='<U1')"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "comment on dtype = '<UI'\n",
        "\n",
        "seen above from running the indx2char.\n",
        "\n",
        "that datatype specifies that the array elements are unsigned 1-byte integers."
      ],
      "metadata": {
        "id": "tEfIPw-FwvDr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# vetorize the song strings\n",
        "def vectorize_string(string):\n",
        "  '''\n",
        "  we pass a string ie the song\n",
        "  we convert the string to characters\n",
        "  then convert the characters to indices\n",
        "  then the indices to numpy arrays for easier storage and manipulation\n",
        "  '''\n",
        "  #convert the strings of the song to characters\n",
        "  characters = list(string)\n",
        "\n",
        "  #vocab characters to indices\n",
        "  char2indx = {indx:ch for ch, indx in enumerate(vocab)}\n",
        "\n",
        "  #map each character in the input string to its corresponding index\n",
        "  vectorize = [char2indx[char] for char in characters]\n",
        "\n",
        "  # convert the list to a numpy array\n",
        "  vectorize_array = np.array(vectorize)\n",
        "\n",
        "  return vectorize_array\n"
      ],
      "metadata": {
        "id": "m0ufOZqdwGRK"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vectorized_songs = vectorize_string(joined_songs)"
      ],
      "metadata": {
        "id": "VpDBuVeZWMkV"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(vectorized_songs) == len(joined_songs)"
      ],
      "metadata": {
        "id": "PB4ooRZ44Ffq",
        "outputId": "864db7e8-c0ec-4292-dc4c-db3b9d7017b2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " #test example of the character to indices line in vectorize_string function above\n",
        "char2indx = {indx:ch for ch, indx in enumerate(vocab)}\n",
        "char2indx['r']"
      ],
      "metadata": {
        "id": "jOsTEcafTpZC",
        "outputId": "7e738221-b1af-4225-94b4-c93091f1e5cc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "73"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('{} ->  character mapped to int -> {}'.format(repr(joined_songs[:30]),vectorized_songs[:10] ))"
      ],
      "metadata": {
        "id": "7Wl6wrqRSB9T",
        "outputId": "737ecdc7-4815-4bab-dbe7-854e13be8eb6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"X:1\\nT:Alexander's\\nZ: id:dc-hor\" ->  character mapped to int -> [49 22 13  0 45 22 26 67 60 79]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "from the output of the print function above, you notice that we the individual characters from the string of the song were mapped to their unique indexes (what we are referring to as vectorized).\n",
        "\n",
        "\n",
        "Now that we have vectorized the songs, we can now split the training set from the testing set.\n",
        "\n",
        "notes:\n",
        "- random.choice() returns a randomly selected element from the provided sequence. Each time the code line runs, a new selection is made.\n",
        "\n",
        "an illustration to clearly explain the `get_batch()` function.\n",
        "\n",
        "suppose\n",
        "\n",
        "  the length of vectorized songs is n, the sequence length that we will feed the RNN is defined as 10\n",
        "  and the batch size is the chunks of examples that will be shown to the model. We will then randomly choose the starting point from the entire array of the input. The code snippet shows how.\n",
        "\n",
        "    n = vectorized_songs.shape[0] - 1\n",
        "    seq_length = 10\n",
        "    batch_size = 10`\n",
        "    rand_idx = np.random.choice(n - seq_length, batch_size)`\n",
        "\n",
        "  An example of an output of the `rand_idx` varible above will be\n",
        "  \n",
        "    `array([ 91766,  74645,  79707, 100050, 199214, 170350,  79151, 143346,\n",
        "            17091,  97731])`\n",
        "\n",
        "  To define an `input_batch` that will be fed to the RNN, we can loop through the indices produced randomly above (From `rand_idx`)\n",
        "\n",
        "    input_batch = [vectorized_songs[idx:idx + seq_length] for idx in rand_idx]\n",
        "    input_batch\n",
        "\n",
        "A sample input to the model would look as follows:\n",
        "\n",
        "    [array([ 1, 59, 32, 27, 59, 82, 60, 62, 62, 14]),\n",
        "    array([30,  1, 30, 26, 82,  2,  0, 27, 59,  1]),\n",
        "    array([82, 59, 15,  1, 26, 82,  2,  0, 27, 59]),\n",
        "    array([14,  1, 61, 58, 26, 58, 82, 62, 59, 59]),\n",
        "    array([ 0,  0, 49, 22, 18,  0, 45, 22, 34, 69]),\n",
        "    array([60, 60, 61,  1, 60, 14, 59, 60, 82, 61]),\n",
        "    array([14, 11, 16,  0, 37, 22, 13, 11, 20,  0]),\n",
        "    array([73, 60, 60, 67,  9, 14, 14, 20,  0, 38]),\n",
        "    array([58, 60,  1, 56, 60, 58, 26, 82,  6, 15]),\n",
        "    array([26,  8, 82, 26,  8, 14, 32, 30,  1, 29])]\n",
        "\n",
        "The equivalent of this vectorized input could be viewed by slicing using the same indices but from the non_vectorized input ( i.e the joined_songs)\n",
        "\n",
        "\n",
        "    input_batch_nonvectorized = [joined_songs[idx:idx + seq_length] for idx in rand_idx]\n",
        "    input_batch_nonvectorized\n",
        "\n",
        "\n",
        "The equivalent of the arrays we saw earlier is shown below. That is a section of the song in abc notation.\n",
        "\n",
        "    [' dGBd|egg2',\n",
        "    'E EA|!\\nBd ',\n",
        "    '|d3 A|!\\nBd',\n",
        "    '2 fcAc|gdd',\n",
        "    '\\n\\nX:6\\nT:In',\n",
        "    'eef e2de|f',\n",
        "    '2/4\\nL:1/8\\n',\n",
        "    'reel-228\\nM',\n",
        "    'ce aecA|(3',\n",
        "    'A,|A,2GE D']"
      ],
      "metadata": {
        "id": "ufPFVTDVWkPO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_batch(vectorized_songs, seq_length, batch_size):\n",
        "  '''\n",
        "  we initialize the function that will allow us to define a batch for training\n",
        "  vectorized_songs are the training data\n",
        "  seq_length will help us break the text into chunks\n",
        "\n",
        "  '''\n",
        "  # the length of vectorized songs\n",
        "  n = vectorized_songs.shape[0] - 1\n",
        "\n",
        "  #start at a random indice for examples in training batch\n",
        "  rand_idx = np.random.choice(n - seq_length, batch_size)\n",
        "\n",
        "  # list of input sequences\n",
        "  input_batch = [vectorized_songs[idx:idx + seq_length] for idx in rand_idx]\n",
        "\n",
        "  # list of output sequences\n",
        "  output_batch = [vectorized_songs[idx+1:idx + seq_length + 1] for idx in rand_idx]\n",
        "\n",
        "  x_batch = np.reshape(input_batch, [batch_size, seq_length])\n",
        "  y_batch = np.reshape(output_batch, [batch_size, seq_length])\n",
        "\n",
        "  return x_batch, y_batch\n"
      ],
      "metadata": {
        "id": "XxpIATAXXCnN"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(vectorized_songs) == vectorized_songs.shape[0]"
      ],
      "metadata": {
        "id": "k_xX_3ic6HjH",
        "outputId": "2517a2a2-5db4-4bd6-f499-a2def79f2e5d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that our training data is ready, we can start working on developing the RNN model.\n",
        "\n",
        "The goal is to use this model to generate new songs.\n",
        "\n",
        "notes on the LSTM function below:\n",
        "- return sequences indicates that the model will return sequences and not just the final output.\n",
        "- glorot uniform initializer is used to sample the weights from the uniform distribution based on the size of the weight tensor.\n",
        "- sigmoid function squaches the values between 0 and 1 controlling the flow of infomation through the LSTM cell.\n",
        "- stateful = true is useful to preserve the layer internal state between batches allowing the model to learn long-term dependencies within the sequence."
      ],
      "metadata": {
        "id": "bHhBaP7reEQb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define RNN model\n",
        "def LSTM(rnn_units):\n",
        "  return tf.keras.layers.LSTM(\n",
        "      rnn_units,\n",
        "      return_sequences=True,\n",
        "      recurrent_initializer='glorot_uniform',\n",
        "      recurrent_activation='sigmoid',\n",
        "      stateful=True)"
      ],
      "metadata": {
        "id": "Zi8D5pYPcnnC"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
        "  model = tf.keras.Sequential([\n",
        "      #transform indices into dense vectors - layer 1\n",
        "      tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[batch_size, None]),\n",
        "\n",
        "      #LSTM is layer 2\n",
        "      tf.keras.layers.LSTM(rnn_units),\n",
        "\n",
        "      #transform the LSTM output into a vocabulary size\n",
        "      tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "  ])\n",
        "\n",
        "  return model"
      ],
      "metadata": {
        "id": "TQ2LHRpBRLaX"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = build_model(len(vocab), embedding_dim=256, rnn_units=1024, batch_size=32)"
      ],
      "metadata": {
        "id": "1aKCI7pXrRXo"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "we do some sanity checks on our simple model."
      ],
      "metadata": {
        "id": "9_bxesTErklt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "id": "5Gx1fbCzrjYt",
        "outputId": "a5be32b7-a7e8-4c31-9403-2f1a7f130fda",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (32, None, 256)           21248     \n",
            "                                                                 \n",
            " lstm (LSTM)                 (32, 1024)                5246976   \n",
            "                                                                 \n",
            " dense (Dense)               (32, 83)                  85075     \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 5,353,299\n",
            "Trainable params: 5,353,299\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x, y = get_batch(vectorized_songs, seq_length=100, batch_size=32)"
      ],
      "metadata": {
        "id": "HO-JHCwJrs0U"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(x))\n",
        "x.shape"
      ],
      "metadata": {
        "id": "mtgUXaatsKPE",
        "outputId": "8f3a1040-b3b5-4b9e-c24b-f7c6e68804fd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "32\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(32, 100)"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(y))\n",
        "y.shape"
      ],
      "metadata": {
        "id": "h7IlTb5Qsuxp",
        "outputId": "76885151-1bc8-4938-9c1d-f747c73eafd1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "32\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(32, 100)"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pred = model(x)"
      ],
      "metadata": {
        "id": "inZQs4lxsynr"
      },
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pred"
      ],
      "metadata": {
        "id": "dXfjeOjB7SuB",
        "outputId": "5fc28a38-0181-4461-d22c-53ec02bc5d68",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(32, 83), dtype=float32, numpy=\n",
              "array([[-0.00319604,  0.00211253, -0.01241458, ..., -0.00012345,\n",
              "         0.00466482, -0.01294524],\n",
              "       [ 0.01571455,  0.00087369, -0.00417113, ...,  0.0032012 ,\n",
              "        -0.00440836, -0.00078404],\n",
              "       [ 0.01027013,  0.00438162, -0.00413364, ..., -0.01213961,\n",
              "         0.00628384, -0.00120205],\n",
              "       ...,\n",
              "       [-0.00533564,  0.00173857, -0.01452293, ..., -0.00029971,\n",
              "        -0.00322834, -0.00576325],\n",
              "       [ 0.00014208,  0.00317596, -0.00713918, ...,  0.00645009,\n",
              "        -0.00173611, -0.00363189],\n",
              "       [-0.00363292, -0.0065942 , -0.00263345, ..., -0.01786411,\n",
              "        -0.00663792, -0.00523666]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notes on categorical function:\n",
        "- we are using this function to select the next word to generate based on the predicted probabilities of each word in the vocabulary.\n",
        "\n",
        "- pred[0] is the predicted probabilities for each word in the vocabulary."
      ],
      "metadata": {
        "id": "1BqZ7jfWt9Q5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(vocab)"
      ],
      "metadata": {
        "id": "Oyq9V34gyefO",
        "outputId": "62582ede-9a39-44b9-ac2a-ccac9be7722d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "83"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Shape of pred[0]:\", pred[0].shape)\n",
        "print(\"Data type of pred[0]:\", pred[0].dtype)\n",
        "print(\"Min value of pred[0]:\", tf.reduce_min(pred[0]))\n",
        "print(\"Max value of pred[0]:\", tf.reduce_max(pred[0]))\n"
      ],
      "metadata": {
        "id": "-Sf0RRbfyRnl",
        "outputId": "8f88fd1f-6e6e-4b43-e115-5f8d58309046",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of pred[0]: (83,)\n",
            "Data type of pred[0]: <dtype: 'float32'>\n",
            "Min value of pred[0]: tf.Tensor(-0.013302604, shape=(), dtype=float32)\n",
            "Max value of pred[0]: tf.Tensor(0.022281915, shape=(), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Negative probabilities:\", tf.reduce_any(pred[0] < 0))\n",
        "print(\"NaN values:\", tf.reduce_any(tf.math.is_nan(pred[0])))\n"
      ],
      "metadata": {
        "id": "cRKwYPOLy6qF",
        "outputId": "a672e117-25fc-41bb-8942-a680c0483c6c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Negative probabilities: tf.Tensor(True, shape=(), dtype=bool)\n",
            "NaN values: tf.Tensor(False, shape=(), dtype=bool)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "GIlLTcapyEqZ"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}