3_Multiclass_Classification_Reuters_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiclass Classification\n",
    "\n",
    "In this problem we will design a neural network to classify newsreels from Reuters dataset into 46 mutually exclusive classes using the Python library, Keras. Thus, this problem is more specifically a _single-label, multiclass classification_ problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Reuters dataset\n",
    "\n",
    "The _Reuters dataset_ is a set of short newswires sorted into 46 _mutually exclusive_ topics. It was published by _Reuters_ in 1986. This dataset is widely used for text classification. There are 46 topics, where some topics are represented more than others. However, each topic contains atleast 10 examples in the training set.\n",
    "\n",
    "The _reuters dataset_ comes pleloaded with keras and contains 8,982 training examples and 2,246 testing examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data from pre-packaged module in Keras. We will limit the data to 10,000 of the most frequently occuring words. To do thiss we pass `num_words=10000` argument to the `load_data` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import reuters\n",
    "\n",
    "# Load \n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8982"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of training Samples\n",
    "len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2246"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of testing samples\n",
    "len(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 2,\n",
       " 2,\n",
       " 8,\n",
       " 43,\n",
       " 10,\n",
       " 447,\n",
       " 5,\n",
       " 25,\n",
       " 207,\n",
       " 270,\n",
       " 5,\n",
       " 3095,\n",
       " 111,\n",
       " 16,\n",
       " 369,\n",
       " 186,\n",
       " 90,\n",
       " 67,\n",
       " 7,\n",
       " 89,\n",
       " 5,\n",
       " 19,\n",
       " 102,\n",
       " 6,\n",
       " 19,\n",
       " 124,\n",
       " 15,\n",
       " 90,\n",
       " 67,\n",
       " 84,\n",
       " 22,\n",
       " 482,\n",
       " 26,\n",
       " 7,\n",
       " 48,\n",
       " 4,\n",
       " 49,\n",
       " 8,\n",
       " 864,\n",
       " 39,\n",
       " 209,\n",
       " 154,\n",
       " 6,\n",
       " 151,\n",
       " 6,\n",
       " 83,\n",
       " 11,\n",
       " 15,\n",
       " 22,\n",
       " 155,\n",
       " 11,\n",
       " 15,\n",
       " 7,\n",
       " 48,\n",
       " 9,\n",
       " 4579,\n",
       " 1005,\n",
       " 504,\n",
       " 6,\n",
       " 258,\n",
       " 6,\n",
       " 272,\n",
       " 11,\n",
       " 15,\n",
       " 22,\n",
       " 134,\n",
       " 44,\n",
       " 11,\n",
       " 15,\n",
       " 16,\n",
       " 8,\n",
       " 197,\n",
       " 1245,\n",
       " 90,\n",
       " 67,\n",
       " 52,\n",
       " 29,\n",
       " 209,\n",
       " 30,\n",
       " 32,\n",
       " 132,\n",
       " 6,\n",
       " 109,\n",
       " 15,\n",
       " 17,\n",
       " 12]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out the first sample\n",
    "train_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding a Review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load mapping of words to numbers\n",
    "word_index = reuters.get_word_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate integer to word mappings\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Decode the first newswire\n",
    "#\n",
    "# Indices are offset by 3 because, 0,1, and 2 are reserved for 'paddind', 'start of response', and 'unknown'\n",
    "decoded_newswire = ' '.join([reverse_word_index.get(i-3, '?') for i in train_data[0]])\n",
    "decoded_newswire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Label associated with the decoded newswire\n",
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we cannot feed integer sequences to the neural network, we will vectorize each sequence and convert them into _tensors_. We will do this by One-Hot Encoding each sequence.\n",
    "\n",
    "Since we have 10,000 unique elements in our training and testing data. Vectorizing our input data will result in two 2D tensors. A _training Input tensor_ of shape (8982, 10000) and a _test input tensor_ of shape (2246, 10000). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), 10000))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = vectorize_sequences(train_data)\n",
    "X_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Labels for this problem include 46 different classes. The labels are represented as integers in the range 1 to 46. To vecorize the labels we could either, \n",
    "* cast the labels as integer tensors\n",
    "* One-Hot encode the label data\n",
    "\n",
    "We will go ahead with One-Hot Encoding of the label data. This will give us tensors whose 2nd axis has 46 dimensions. This can be done easily using the `to_categorical` function in Keras. For greater clarity, we will do it manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Categorical Encoding of the Labels\n",
    "def to_one_hot(labels, dimension=46):\n",
    "    results = np.zeros((len(labels), dimension))\n",
    "    for i,label in enumerate(labels):\n",
    "        results[i,label] = 1\n",
    "    return results\n",
    "\n",
    "Y_train = to_one_hot(train_labels)\n",
    "Y_test = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other way to perform _One-Hot categorical encoding_ of the labels is to use the built-in function.\n",
    "\n",
    "```python\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "Y_train = to_categorical(train_labels)\n",
    "Y_test  = to_categorical(test_labels)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8982, 10000)\n",
      "(8982, 46)\n",
      "\n",
      "(2246, 10000)\n",
      "(2246, 46)\n"
     ]
    }
   ],
   "source": [
    "# Checkinf if the tensors we created, bear the predicted shapes\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print()\n",
    "print(X_test.shape)\n",
    "print(Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem of topic classification (_single-label multiclass classification_) is similar to _Binary Classification_ of text fields. Both problems follow a similar approach towards handling and pre processing of data. The pedogogy of implementing neural network remains unchanged. However, there is a new constraint: the number of classes just went from 2 to 46. The _dimensionality_ of the _output space_ is much higher.\n",
    "\n",
    "This means that each layer will have to deal with more data, which presents the very real possibility of _information bottleneck_.\n",
    "\n",
    "### Information Bottleneck\n",
    "\n",
    "Given a _multiclass classification_ problem; the amount of processing that needs to be performed on the data increses dramatically as compared to a _Bianry Classification_ problem. \n",
    "\n",
    "In a _stack_ of _Dense_ layers, like the ones we use, each layer can only access information present in the output of the previous layer. If a layer drops relevant information, that information is permanently inaccessable for all subsequent layers. Information once lost, can never be recovered by later layers. In cases, like _multiclass classification_, where information is limited and crucial, every layer could become an _information bottleneck_.\n",
    "\n",
    "If a layer drops relevent information, it is potentilly an information bottleneck.\n",
    "\n",
    "Such layers could easily bottleneck the performance of our network. To ensure that crucial data is not discarded, we will use layers with greater number of _hidden units_, i.e. _Larger layers_. For comparision, we use layers with _16 hidden units_, `Dense(16)` in the _two-class_ classification example of Seniment analysis of IMDB Reviws. In this case, where the output dimension is 46, we will use layers with 64 hidden units, `Dense(64)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Definition\n",
    "\n",
    "We will define our network with 2 _fully-connected_ ReLU activated layers with 64 hidden units each. The third and last layer will be Dense layer of size 46. This layer will use a _softmax_ activation and will output a 46-dimentional vector. Every dimension will be the probability of the input belonging to that class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape = (10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the model\n",
    "\n",
    "For our network, we will use the _optimizer_ `rmsprop`, _loss function_, `categorical_crossentropy`, and will monitor the `accuracy` (_metrics_) of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "             loss='categorical_crossentropy',\n",
    "             metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Validation Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validation samples\n",
    "X_val = X_train[:1000]\n",
    "partial_X_train = X_train[1000:]\n",
    "\n",
    "# Validation labels\n",
    "Y_val = Y_train[:1000]\n",
    "partial_Y_train = Y_train[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially, we will train our models for 20 epochs in mini-batches of 512 samples. We will aslo pass our _validation set_ to the `fit` method. \n",
    "\n",
    "Calling the `fit` method returns a `History` object. This object contains a member `history` which stores all data  about the training process including the values of observable or monitored quantaties as the epochs proceed. We will save this object to better determine the fine tuning to be applied to the training step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "16/16 [==============================] - 1s 60ms/step - loss: 2.5975 - accuracy: 0.5083 - val_loss: 1.6964 - val_accuracy: 0.6380\n",
      "Epoch 2/20\n",
      "16/16 [==============================] - 1s 39ms/step - loss: 1.4027 - accuracy: 0.7022 - val_loss: 1.3103 - val_accuracy: 0.7060\n",
      "Epoch 3/20\n",
      "16/16 [==============================] - 1s 40ms/step - loss: 1.0592 - accuracy: 0.7689 - val_loss: 1.1499 - val_accuracy: 0.7460\n",
      "Epoch 4/20\n",
      "16/16 [==============================] - 1s 37ms/step - loss: 0.8425 - accuracy: 0.8183 - val_loss: 1.0576 - val_accuracy: 0.7730\n",
      "Epoch 5/20\n",
      "16/16 [==============================] - 1s 36ms/step - loss: 0.6730 - accuracy: 0.8572 - val_loss: 0.9827 - val_accuracy: 0.7910\n",
      "Epoch 6/20\n",
      "16/16 [==============================] - 1s 39ms/step - loss: 0.5379 - accuracy: 0.8859 - val_loss: 0.9520 - val_accuracy: 0.8090\n",
      "Epoch 7/20\n",
      "16/16 [==============================] - 1s 44ms/step - loss: 0.4302 - accuracy: 0.9060 - val_loss: 0.9187 - val_accuracy: 0.8040\n",
      "Epoch 8/20\n",
      "16/16 [==============================] - 1s 46ms/step - loss: 0.3452 - accuracy: 0.9291 - val_loss: 0.9476 - val_accuracy: 0.8120\n",
      "Epoch 9/20\n",
      "16/16 [==============================] - 1s 35ms/step - loss: 0.2881 - accuracy: 0.9387 - val_loss: 0.9324 - val_accuracy: 0.8110\n",
      "Epoch 10/20\n",
      "16/16 [==============================] - 1s 38ms/step - loss: 0.2414 - accuracy: 0.9450 - val_loss: 0.9179 - val_accuracy: 0.8170\n",
      "Epoch 11/20\n",
      "16/16 [==============================] - 1s 35ms/step - loss: 0.2133 - accuracy: 0.9478 - val_loss: 0.9402 - val_accuracy: 0.8130\n",
      "Epoch 12/20\n",
      "16/16 [==============================] - 1s 39ms/step - loss: 0.1818 - accuracy: 0.9508 - val_loss: 0.9359 - val_accuracy: 0.8150\n",
      "Epoch 13/20\n",
      "16/16 [==============================] - 1s 36ms/step - loss: 0.1661 - accuracy: 0.9525 - val_loss: 0.9874 - val_accuracy: 0.8030\n",
      "Epoch 14/20\n",
      "16/16 [==============================] - 1s 31ms/step - loss: 0.1526 - accuracy: 0.9564 - val_loss: 0.9802 - val_accuracy: 0.8190\n",
      "Epoch 15/20\n",
      "16/16 [==============================] - 1s 40ms/step - loss: 0.1375 - accuracy: 0.9572 - val_loss: 0.9750 - val_accuracy: 0.8200\n",
      "Epoch 16/20\n",
      "16/16 [==============================] - 1s 39ms/step - loss: 0.1291 - accuracy: 0.9550 - val_loss: 0.9935 - val_accuracy: 0.8130\n",
      "Epoch 17/20\n",
      "16/16 [==============================] - 1s 44ms/step - loss: 0.1259 - accuracy: 0.9568 - val_loss: 1.0508 - val_accuracy: 0.8070\n",
      "Epoch 18/20\n",
      "16/16 [==============================] - 1s 38ms/step - loss: 0.1184 - accuracy: 0.9587 - val_loss: 1.0544 - val_accuracy: 0.7990\n",
      "Epoch 19/20\n",
      "16/16 [==============================] - 1s 45ms/step - loss: 0.1161 - accuracy: 0.9572 - val_loss: 1.0632 - val_accuracy: 0.8110\n",
      "Epoch 20/20\n",
      "16/16 [==============================] - 1s 41ms/step - loss: 0.1135 - accuracy: 0.9565 - val_loss: 1.1270 - val_accuracy: 0.7820\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_X_train,\n",
    "                   partial_Y_train,\n",
    "                   epochs=20,\n",
    "                   batch_size=512,\n",
    "                   validation_data=(X_val, Y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of training we have attained a training accuracy of 95% and validation accuracy of 80.9%\n",
    "\n",
    "Now that we have trained our network, we will observe its performance metrics stored in the `History` object.\n",
    "\n",
    "Calling the `fit` method returns a `History` object. This object has an attribute `history` which is a dictionary containing four enteries: one per monitored metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`history_dict` contains values of \n",
    "* Training loss\n",
    "* Trainining Accuracy\n",
    "* Validation Loss\n",
    "* Validation Accuracy\n",
    "\n",
    "at the end of each epoch.\n",
    "\n",
    "Let's use Matplotlib to plot Training and validation losses and Traing and Validation Accuracy side by side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9b3/8dcnEMCwKmAlIATFqoARQ8SlqHhVKti6/1Qu4lKVaq1LW73S2lqv1V5bW8XtqvTWrVLQVrFWcalKi9ZWDRSQpRRKASOIgJVFUAh8fn98T2AIM8lkOTOTzPv5eJzHzJz1k5PkfOa7nO8xd0dERPJXQbYDEBGR7FIiEBHJc0oEIiJ5TolARCTPKRGIiOQ5JQIRkTynRCBNwsxamdlGM+vdlOtmk5n1M7NY+lfX3LeZvWJmo+OIw8x+YGYPNnR7afmUCPJUdCGunrab2eaEz0kvSLVx923u3sHdlzflurnKzF4zs5uSzD/LzD4ws3r9b7n7cHef2ARxnWhmS2vs+0fufnlj953kWJea2R+ber+SeUoEeSq6EHdw9w7AcuCrCfN2uyCZWevMR5nTHgXGJJk/BnjC3bdnNhyRhlMikKTM7FYze9LMJpnZBuB8MzvKzP5qZp+Y2Uozu8fMCqP1W5uZm1lJ9PmJaPmLZrbBzP5iZn3ru260fISZ/cPM1pnZvWb2ZzO7KEXc6cT4dTNbbGb/NrN7ErZtZWZ3mdlaM/sncHItp+gZYB8zOzph+67ASODx6POpZjYr+pmWm9kPajnfb1b/THXFEX0TXxDt959mdmk0vzPwe6B3Qulu7+h3+WjC9qeb2bzoHL1uZgcmLKs0s2+b2XvR+Z5kZm1rOQ+pfp5eZva8mX1sZovM7GsJy440s5lmtt7MVpnZHdH8IjP7dfRzf2Jm75hZt/oeW+pPiUBqcwbwa6Az8CRQBVwDdAO+RLhAfb2W7f8T+AGwF6HU8aP6rmtmewNPAddHx/0XMKSW/aQT40hgMHAYIcGdGM2/AhgOHBod45xUB3H3T4HfAhckzD4PmOPu86LPG4HzCefvq8A1ZvaVWmKvVlccq4BTgE7AZcC9Zlbq7uui4yxPKN19lLihmR0MPAFcBXQHXgV+X50sI+cAJwH7Ec5TspJPXZ4k/K6KgXOBn5rZcdGye4E73L0T0I9wHgEuBoqAXkBX4BvAZw04ttSTEoHU5k13/727b3f3ze7+rru/7e5V7r4EmAAcV8v2v3X3CnffCkwEBjVg3a8As9z9d9Gyu4A1qXaSZoz/4+7r3H0p8MeEY50D3OXule6+Fri9lngBHgPOSfjGfEE0rzqW1919bnT+ZgOTk8SSTK1xRL+TJR68DrwGHJPGfiEkq+ei2LZG++4EHJGwznh3/zA69vPU/nvbTVSaGwKMc/fP3H0m8Ag7E8pW4AAz6+ruG9z97YT53YB+UTtShbtvrM+xpWGUCKQ27yd+MLODzOwFM/vQzNYDtxD+cVP5MOH9JqBDA9YtTozDwyiJlal2kmaMaR0LWFZLvAB/AtYBXzWzLxJKGJMSYjnKzP5oZqvNbB1waZJYkqk1DjP7ipm9HVW7fEIoPaRbhVKcuL+oLaMS6JmwTn1+b6mOsSYqNVVblnCMi4H+wMKo+mdkNP9RQgnlKQsN7reb2qYyQolAalOzy+JDwFzCN7ZOwE2AxRzDSkJVAQBmZux60aqpMTGuBPZN+Fxr99YoKf2KUBIYA0x198TSymTgaWBfd+8M/F+asaSMw8z2IFSl/A/wBXfvArySsN+6upmuAPok7K+AcH4/SCOudK0AuplZ+4R5vauP4e4L3f08YG/g58DTZtbO3be4+83ufjAwlFA1We8ebFJ/SgRSHx0J34A/jeqaa2sfaCrPA2Vm9tXo2+E1hLrtOGJ8CrjWzHpGDb83pLHNY4R2iK+RUC2UEMvH7v6ZmR1JqJZpbBxtgTbAamBb1OZwQsLyVYSLcMda9n2qmQ2L2gWuBzYAb6dYvy4FZtYucXL3fwEVwI/NrK2ZDSKUAiYCmNkYM+sWlUbWEZLXdjP7DzMbGCWn9YSqom0NjEvqQYlA6uM7wIWEC8dDhAbBWLn7KkJj453AWmB/4G/A5zHE+AChvv094F12NmLWFt8/gXeAdsALNRZfAfyPhV5X3yNchBsVh7t/AnwLmAJ8DJxNSJbVy+cSSiFLo543e9eIdx7h/DxASCYnA6dG7QUNcQywucYE4Xd2AKGa6bfA99x9WrRsJLAgOi8/A8519y2EKqVnCElgHqGaaEdVm8TH9GAaaU7MrBWh6uFsd38j2/GItAQqEUjOM7OTzaxz1DvnB4Quou9kOSyRFkOJQJqDocASQrfRk4HT3T1V1ZCI1JOqhkRE8pxKBCIiea7Z3azRrVs3LykpyXYYIiLNyowZM9a4e9Ku180uEZSUlFBRUZHtMEREmhUzS3mnvKqGRETynBKBiEieUyIQEclzza6NQEQyY+vWrVRWVvLZZ3okQHPSrl07evXqRWFhYd0rR5QIRCSpyspKOnbsSElJCWHQV8l17s7atWuprKykb9++dW8QyYuqoYkToaQECgrC68RGPyJcpOX77LPP6Nq1q5JAM2JmdO3atd6luBZfIpg4EcaOhU2bwudly8JngNEa6VykVkoCzU9DfmctvkRw4407k0C1TZvCfBERyYNEsHx5/eaLSPatXbuWQYMGMWjQIPbZZx969uy54/OWLVvS2sfFF1/MwoULa13n/vvvZ2IT1RUPHTqUWbNmNcm+Mq3FVw317h2qg5LNF5GmM3FiKGkvXx7+v267reHVr127dt1xUb355pvp0KED11133S7ruDvuTkFB8u+zjzzySJ3HufLKKxsWYAvT4ksEt90GRUW7zisqCvNFpGlUt8UtWwbuO9vimrpjxuLFixk4cCCXX345ZWVlrFy5krFjx1JeXs6AAQO45ZZbdqxb/Q29qqqKLl26MG7cOA499FCOOuooPvroIwC+//3vM378+B3rjxs3jiFDhnDggQfy1ltvAfDpp59y1llnceihhzJq1CjKy8vT/ua/efNmLrzwQg455BDKysqYPn06AO+99x6HH344gwYNorS0lCVLlrBhwwZGjBjBoYceysCBA/ntb+t8QF6TafGJYPRomDAB+vQBs/A6YYIaikWaUibb4ubPn88ll1zC3/72N3r27Mntt99ORUUFs2fP5g9/+APz58/fbZt169Zx3HHHMXv2bI466igefvjhpPt2d9555x3uuOOOHUnl3nvvZZ999mH27NmMGzeOv/3tb2nHes8999CmTRvee+89fvWrXzFmzBi2bNnC//7v/3Ldddcxa9Ys3n33XYqLi5k6dSolJSXMnj2buXPnctJJJzXsBDVAi08EEC76S5fC9u3hVUlApGllsi1u//335/DDD9/xedKkSZSVlVFWVsaCBQuSJoI99tiDESNGADB48GCWLl2adN9nnnnmbuu8+eabnHfeeQAceuihDBgwIO1Y33zzTcaMGQPAgAEDKC4uZvHixRx99NHceuut/PSnP+X999+nXbt2lJaW8tJLLzFu3Dj+/Oc/07lz57SP01ixJQIz29fMppnZAjObZ2bXJFlnmJmtM7NZ0XRTXPGISHxStbnF0RbXvn37He8XLVrE3Xffzeuvv86cOXM4+eSTk/ahb9OmzY73rVq1oqqqKum+27Ztu9s6jXl4V6ptx4wZw5QpU2jbti0nnXQS06dP5+CDD6aiooIBAwZw/fXX8+Mf/7jBx62vOEsEVcB33P1g4EjgSjPrn2S9N9x9UDTdkmS5iOS4bLXFrV+/no4dO9KpUydWrlzJyy+/3OTHGDp0KE899RQQ6vaTlThSOfbYY3f0SlqwYAErV66kX79+LFmyhH79+nHNNddwyimnMGfOHD744AM6dOjAmDFj+Pa3v83MmTOb/GdJJbZeQ+6+ElgZvd9gZguAnkD6Z1FEmoXq6tam6jWUrrKyMvr378/AgQPZb7/9+NKXvtTkx7jqqqu44IILKC0tpaysjIEDB6astvnyl7+8Y4yfY445hocffpivf/3rHHLIIRQWFvL444/Tpk0bfv3rXzNp0iQKCwspLi7m1ltv5a233mLcuHEUFBTQpk0bHnzwwSb/WVLJyDOLzawEmA4MdPf1CfOHAU8DlcAK4Dp3n5dk+7HAWIDevXsPXpasP6iINKkFCxZw8MEHZzuMrKuqqqKqqop27dqxaNEihg8fzqJFi2jdOnd73yf73ZnZDHcvT7Z+7D+JmXUgXOyvTUwCkZlAH3ffaGYjgWeBA2ruw90nABMAysvL489cIiKRjRs3csIJJ1BVVYW789BDD+V0EmiIWH8aMyskJIGJ7v5MzeWJicHdp5rZ/5pZN3dfE2dcIiLp6tKlCzNmzMh2GLGKs9eQAb8EFrj7nSnW2SdaDzMbEsWzNq6YRERkd3GWCL4EjAHeM7Pq2/C+B/QGcPcHgbOBK8ysCtgMnOeZaLQQEZEd4uw19CZQ63io7n4fcF9cMYiISN3y4s5iERFJTYlARHLOsGHDdrs5bPz48XzjG9+odbsOHToAsGLFCs4+++yU+66oqKh1P+PHj2dTwuBJI0eO5JNPPkkn9FrdfPPN/OxnP2v0fpqaEoGI5JxRo0YxefLkXeZNnjyZUaNGpbV9cXFxo0bvrJkIpk6dSpcuXRq8v1ynRCAiOefss8/m+eef5/PPPwdg6dKlrFixgqFDh+7o119WVsYhhxzC7373u922X7p0KQMHDgTCUNDnnXcepaWlnHvuuWzevHnHeldcccWOIax/+MMfAmHE0BUrVnD88cdz/PHHA1BSUsKaNaFX+5133snAgQMZOHDgjiGsly5dysEHH8xll13GgAEDGD58+C7HqUuyfX766aeccsopO4alfvLJJwEYN24c/fv3p7S0dLdnNDRUy7orQkRice210NQP3xo0CKJr3m66du3KkCFDeOmllzjttNOYPHky5557LmZGu3btmDJlCp06dWLNmjUceeSRnHrqqSmf1fvAAw9QVFTEnDlzmDNnDmVlZTuW3Xbbbey1115s27aNE044gTlz5nD11Vdz5513Mm3aNLp167bLvmbMmMEjjzzC22+/jbtzxBFHcNxxx7HnnnuyaNEiJk2axC9+8QvOOeccnn76ac4///w6z0OqfS5ZsoTi4mJeeOEFIAyl/fHHHzNlyhT+/ve/Y2ZNUl0FKhGISI5KrB5KrBZyd773ve9RWlrKiSeeyAcffMCqVatS7mf69Ok7LsilpaWUlpbuWPbUU09RVlbGYYcdxrx58+ocUO7NN9/kjDPOoH379nTo0IEzzzyTN954A4C+ffsyaNAgoPahrtPd5yGHHMKrr77KDTfcwBtvvEHnzp3p1KkT7dq149JLL+WZZ56hqOZIfw2kEoGI1CnVN/c4nX766TtG4dy8efOOb/ITJ05k9erVzJgxg8LCQkpKSpIOPZ0oWWnhX//6Fz/72c9499132XPPPbnooovq3E9ttzlVD2ENYRjrdKuGUu3zi1/8IjNmzGDq1Kl897vfZfjw4dx000288847vPbaa0yePJn77ruP119/Pa3j1EYlAhHJSR06dGDYsGF87Wtf26WReN26dey9994UFhYybdo06hqEMnEo6Llz5zJnzhwgDGHdvn17OnfuzKpVq3jxxRd3bNOxY0c2bNiQdF/PPvssmzZt4tNPP2XKlCkcc8wxjfo5U+1zxYoVFBUVcf7553Pdddcxc+ZMNm7cyLp16xg5ciTjx49P+5GZdVGJQERy1qhRozjzzDN36UE0evRovvrVr1JeXs6gQYM46KCDat3HFVdcwcUXX0xpaSmDBg1iyJAhQHja2GGHHcaAAQN2G8J67NixjBgxgh49ejBt2rQd88vKyrjooot27OPSSy/lsMMOS7saCODWW2/d0SAMUFlZmXSfL7/8Mtdffz0FBQUUFhbywAMPsGHDBk477TQ+++wz3J277ror7ePWJiPDUDel8vJyr6sPsIg0noahbr7qOwy1qoZERPKcEoGISJ5TIhCRlJpb1bE07HemRCAiSbVr1461a9cqGTQj7s7atWtp165dvbZTryERSapXr15UVlayevXqbIci9dCuXTt69epVr22UCEQkqcLCQvr27ZvtMCQDVDUkIpLnlAhERPKcEoGISJ5TIhARyXNKBCIieU6JQEQkzykRiIjkOSUCEZE8p0QgIpLnlAhERPKcEoGISJ5TIhARyXNKBCIieU6JQEQkzykRiIjkudgSgZnta2bTzGyBmc0zs2uSrGNmdo+ZLTazOWZWFlc8IiKSXJwPpqkCvuPuM82sIzDDzP7g7vMT1hkBHBBNRwAPRK8iIpIhsZUI3H2lu8+M3m8AFgA9a6x2GvC4B38FuphZj7hiEhGR3WWkjcDMSoDDgLdrLOoJvJ/wuZLdkwVmNtbMKsysQs9PFRFpWrEnAjPrADwNXOvu62suTrKJ7zbDfYK7l7t7effu3eMIU0Qkb8WaCMyskJAEJrr7M0lWqQT2TfjcC1gRZ0wiIrKrOHsNGfBLYIG735liteeAC6LeQ0cC69x9ZVwxiYjI7uLsNfQlYAzwnpnNiuZ9D+gN4O4PAlOBkcBiYBNwcYzxiIhIErElAnd/k+RtAInrOHBlXDGIiEjddGexiEieUyIQEclzSgQiInlOiUBEJM8pEYiI5DklAhGRPKdEICKS55QIRETynBKBiEieUyIQEclzSgQiInlOiUBEJM8pEYiI5DklAhGRPKdEICKS55QIRETyXF4lAvdsRyAiknvyJhG88gocfDCsXp3tSEREckveJIJevWDRIvjJT7IdiYhIbsmbRNC/P4wZA/fdBx98kO1oRERyR94kAoAf/hC2b4dbb812JCIiuaPORGBmXzSz18xsbvS51My+H39oTa9vX7jsMvi//4MlS7IdjYhIbkinRPAL4LvAVgB3nwOcF2dQcbrxRmjdGm6+OduRiIjkhnQSQZG7v1NjXlUcwWRCcTFcdRU88QTMm5ftaEREsi+dRLDGzPYHHMDMzgZWxhpVzG64ATp0gJtuynYkIiLZl04iuBJ4CDjIzD4ArgWuiDWqmHXtCt/5DjzzDFRUZDsaEZHsqjMRuPsSdz8R6A4c5O5D3X1p7JHF7FvfCgnh+82y2VtEpOm0rmsFM7upxmcA3P2WmGLKiE6dYNw4uP56mD4djj022xGJiGRHOlVDnyZM24ARQEmMMWXMlVdCjx6hJ5HGIRKRfFVnicDdf5742cx+BjwXW0QZtMce8IMfwDe+AS+/DCefnO2IREQyryF3FhcB+9W1kpk9bGYfVd+IlmT5MDNbZ2azoikrfXguuQRKSlQqEJH8lc6dxe+Z2ZxomgcsBO5OY9+PAnV9x37D3QdFU1baHNq0gf/+b5g5M/QiEhHJN+mUCL4CfDWahgPF7n5fXRu5+3Tg48aFlxmjR4chqn/wA9i2LdvRiIhkVspEYGZ7mdlewIaEaTPQKZrfFI4ys9lm9qKZDWiifdZbq1bwox/BggUwcWK2ohARyQ7zFBXjZvYvwt3ElmSxu3s67QQlwPPuPjDJsk7AdnffaGYjgbvd/YAU+xkLjAXo3bv34GXLltV16Hpzh/Jy+PhjWLgwVBmJiLQUZjbD3cuTLUtZInD3vu6+X/Rac6ozCdTF3de7+8bo/VSg0My6pVh3gruXu3t59+7dG3vopMzgtttg6VL45S9jOYSISE5Kq9eQme1pZkPM7NjqqbEHNrN9LLo7zcyGRLGsbex+G+PLX4ahQ0M10ebN2YxERCRz0uk1dCkwHXgZ+O/o9eY0tpsE/AU40MwqzewSM7vczC6PVjkbmGtms4F7gPM8VT1VhpjBj38MK1fC/fdnMxIRkcxJp0RwDXA4sMzdjwcOA+p8BLy7j3L3Hu5e6O693P2X7v6guz8YLb/P3Qe4+6HufqS7v9Won6SJHHNMKBncfjusXx/mTZwY7jUoKAivalAWkZakzjuLgc/c/TMzw8zauvvfzezA2CPLottuCw3Hd90F/frB2LGwaVNYtmxZ+Ayh26mISHOXTiKoNLMuwLPAH8zs38CKeMPKrsGD4cwz4ec/h86ddyaBaps2hTuRlQhEpCVI2X006cpmxwGdgZfcfUtsUdWivLzcKzLwEIH582HgwNTDTpjB9u2xhyEi0iQa1H3UzF4ws9Fm1r56nrv/yd2fy1YSyKT+/eH888MFP5nevTMbj4hIXGprLJ5AGF5iqZk9aWanm1le3WZ1882hgbh1jQq0oqLQjiAi0hLUdkPZ79x9FNAbeAa4EFgejSp6UqYCzKb99oPLLgvVQ8XFoXTQpw9MmKD2ARFpOdJ5VOVmd3/S3c8gDDp3GPBS7JHliO9/HwoL4aSTQpvA0qVKAiLSsqRzQ9kXzOwqM/szoefQK8Dg2CPLET17hieZ/epXYVA6EZGWprbG4svM7HVgJvBF4L+isYducPdZGYswB4wbF9oFfvjDbEciItL0aisRHA3cDuzr7le5+58zFFPO6dYNvv1t+M1v4O679SQzEWlZUt5Q5u4XZzKQXHfDDTB7Nlx7LVRUwEMPhVKCiEhz15BnFueloqLwKMtbbgljDQ0dGhqORUSaOyWCeigoCI+z/P3vYcmSMB7Ra69lOyoRkcZJp9fQ/mbWNno/zMyujsYeylunnALvvgtf+AIMHx7GJFK7gYg0V+mUCJ4GtplZP+CXQF/g17FG1QwccAD89a9wxhlw3XXwn/8Jn36a7ahEROovnUSw3d2rgDOA8e7+LaBHvGE1Dx07hp5EP/4xPPkkHH10qDISEWlO0kkEW81sFGGIieejeYXxhdS8mMF3vwtTp8Ly5aHd4JVXsh2ViEj60kkEFwNHAbe5+7/MrC/wRLxhNT8nnxy6lfbqBSNGwE9+onYDEWke0hlraL67X+3uk8xsT6Cju9+egdianf33h7/8Bc4+O9yNfO65sHFjtqMSEaldOr2G/mhmncxsL2A28IiZ3Rl/aM1T+/YweTL89Kfw9NNw5JGweHG2oxIRSS2dqqHO7r4eOBN4xN0HAyfGG1bzZgbXXw8vvwwrV8Lhh8OLL2Y7KhGR5NJJBK3NrAdwDjsbiyUNJ54Y2g1KSsK9B7feClu3ZjsqEZFdpZMIbgFeBv7p7u+a2X7AonjDajn69oU//xlGjQp3Je+3H9x5J2zYkO3IRESCdBqLf+Pupe5+RfR5ibufFX9oLUdRETzxBLzwAvTrB9/5Duy7b2hQXrky29GJSK5btgzGj4c//Sme/afTWNzLzKaY2UdmtsrMnjazXvGE03KZwciRMG0avPNOGJrijjtCtdEll+ihNyKykzvMmxeqkwcPDteJb30rfJmMQzpVQ48AzwHFQE/g99E8aaDDD4ennoJ//AMuvRQmTYL+/eHUU+GNN3T/gUg+2r4d3n471BQceCAMHBiqk9u2Db0QFy0Kr3Ewr+OqY2az3H1QXfMypby83CsqKrJx6NisWQP33w/33gtr14Yup9dfD6edBq1aZTs6EYnL1q2humfKFHj2WVixAlq3huOPD+OYnXYaFBc3zbHMbIa7lydblk6JYI2ZnW9mraLpfGBt04SWHyZODEW7goLwOnHirsu7dQuPwVy+PCSEjz6Cs86Cgw8OD8DZvDkbUYtIHDZtChf9Cy8MIxifdBI88ggccQQ8/nj4/3/lFbjiiqZLAnVJp0TQG7iPMMyEA28BV7v78vjD211zKxFMnAhjx4ZffrWiIpgwAUaPTr7Ntm3hITh33BGGu+7eHa66Cr7xDejaNTNxi0j6tm8Pow/XNn3yCbz6Krz0Uvhy16VLqA4+44zQZhj3Ew9rKxHUmQhS7PBadx/f6MgaoLklgpKS0OJfU58+dT/hzD0UG++4Iwxqt8ceYfiKCy8MRccCPVZIJC0ffACvvw4ffxyqY7Zuhaqq3d8nm5f4ftOm5Bf5dEvtxcVw+unh4n/ccVCYweE740gEy929d6Mja4DmlggKCpI3/pqFbxHpmjs3tCE8+SSsWxe6n44ZE5LCF7/YdPGKtARbt8Jbb4U7+l98EebMSb1u69bhglxYWPf7PfYIw8i0bw8dOux8X9uUuF5xcfa+wMWRCN53933rWOdh4CvAR+4+MMlyA+4GRgKbgIvcfWZdx25uiaAxJYJkNm+G556DRx8N9Yjbt8NRR4WEcO65obgpko8qK3de+F99Ndy02bp1eL74iBFhhOBevXa9uLdqFb6U5YOslAjM7FhgI/B4ikQwEriKkAiOAO529yPqOnZzSwQNaSNI14oVYf+PPgrz54duZqefHpLCSSeFP3SRlmrLlnDXfvXFf+7cML96KPgRI+CEE6BTp+zGmSsalAjMbAOhcXi3RcAe7l7nZcbMSoDnUySCh4A/uvuk6PNCYJi713qvbXNLBBAu1jfeGHoF9e4Nt93W+CSQyB1mzIDHHoNf/zrUg/boAeefH5LCgAFNdyyRbFq+fOeF/7XXwjDvhYVwzDE7L/79++fPt/z6aPISQT0OXELqRPA8cLu7vxl9fg24wd13u8qb2VhgLEDv3r0HL0tW1yIAfP55uPvwscdCA3NVVbgz8aKLwnhH6nWUnyorQ535tm2hnjtxKirafV6bNpm9mG7ZErpNrloFH364++uHH4afofpRsH367LzwH398eGys1C5XE8ELwP/USAT/5e4zattncywRZMtHH4USwmOPwaxZoZGqb99w12LNaZ999C2qJVm1KgxnUj0tqucwkQUFuyeHPfaAdu1CkmjbNkyp3idbVlgI//538ov92hR3JnXuHP42v/CF8HrEEeHif9BB+nutr9oSQTZrkSuBxAbnXsCKLMXSIu29N1x7bZjmzAn3JsyfDwsXhotDYpe3Tp1C76OaCeKAA+Lv3yyNt3Zt6Go8bVroJjl/fpjfqRMce2y4OenYY0PPlc2bd06bNu36OdW86vmffx6+vX/6aaiC3LIlzKuen/i+qip5rB067LywH3QQDBu268U+8bVdu4ydwryWzUTwHPBNM5tMaCxeV1f7gDRcaWmYqm3fDu+/H5LCwoVh3KOFC8NYRzXvfO7de2di6N8/tDn07x/uiM6UTz4Jva/atw/H7dw5v2Y2bE4AAA3nSURBVL8Rrl8P06fvvPDPnh3aioqKQi+ZCy4IVSZlZdnrNLBtW0gI1Qliy5bQq61Dh+zEI6nFVjVkZpOAYUA3YBXwQ6AQwN0fjLqP3gecTOg+enGy9oGaVDUUv02bQlVCdZJInBKfo7D33rsmhurX7t0bdtx168JjPRct2nVavDiMx5SoVavQ3lE9detW9/u99srM2E3u4WeprAzJtrIyTCui8m5h4c6qkvq8Vg9K9vrroXPAtm2h2uWoo+A//iNc+IcMCeuK1JS1NoI4KBFkj3u4oM2fH4bITXxdv37net27J08Qe+8dEknNi3z1+9Wrdz1er16haqp66tMnJKm1a8O0Zk3y91u2pP4ZunTZmRRqviab17VrKH1U3wSU6iJf/b76dePGXY9rFqo6CgpCfFu37vy2vG1b+r+D1q1DPfnxx4eL/5FHhrp7kbooEUis3MO33ZrJYd68cNGs1r59qFtO1LNnuMj367frRX+//RrWNuEeLsKpksXHH4dp7dpdXz/5JPU+Cwpgzz1Dffvq1ckv8j16hMS1777JX3v0SD2cwPbtO4cySEwSNV+3bw8DEapqRRoiVxuLpYUwCxf0nj3D4FnV3MMT2KoTw5Il4Rb76ov+/vuH5NDUsXTsGKaSkvS327Yt9GhJTA41E8a6daGqqeaFvraLfDoKCnb2rhHJBiUCiY1ZuPAXF4c7nXNZq1bhIp/JBnCRXKHxK0VE8pwSQTNQ14NtREQaQ1VDOa7moHXLloXP0LTjFYlI/lKJIMfdeOOuI5dC+HzjjdmJR0RaHiWCHLc8xQNBU80XEakvJYIc1zvFUx9SzRcRqS8lghx3222731hVVBTmi4g0BSWCHDd6dHiaWZ8+oV9+nz5N83QzEZFq6jXUDIwerQu/iMRHJQIRkTynRCAikueUCERE8pwSgYhInlMiEBHJc0oEeUCD1olIbdR9tIXToHUiUheVCFo4DVonInVRImjhNGidiNRFiaCF06B1IlIXJYIWToPWiUhdlAhaOA1aJyJ1Ua+hPKBB60SkNioRiIjkOSUCEZE8p0QgIpLnlAgkLRqmQqTlUmOx1EnDVIi0bCoRSJ00TIVIyxZrIjCzk81soZktNrNxSZZfZGarzWxWNF0aZzzSMBqmQqRli61qyMxaAfcDJwGVwLtm9py7z6+x6pPu/s244pDG6907VAclmy8izV+cJYIhwGJ3X+LuW4DJwGkxHk9iomEqRFq2OBNBT+D9hM+V0byazjKzOWb2WzPbN9mOzGysmVWYWcXq1avjiFVqoWEqRFq2OBOBJZnnNT7/Hihx91LgVeCxZDty9wnuXu7u5d27d2/iMCUdo0fD0qWwfXt4VRIQaTniTASVQOI3/F7AisQV3H2tu38effwFMDjGeEREJIk4E8G7wAFm1tfM2gDnAc8lrmBmPRI+ngosiDEeySLdkCaSu2LrNeTuVWb2TeBloBXwsLvPM7NbgAp3fw642sxOBaqAj4GL4opHskc3pInkNnOvWW2f28rLy72ioiLbYUg9lJQk737ap09obxCR+JnZDHcvT7ZMdxZL7HRDmkhuUyKQ2Om5ySK5TYlAYqcb0kRymxKBxK4pbkhTryOR+GgYasmIxjw3Wb2OROKlEoHkPA2DLRIvJQLJeep1JBIvJQLJeep1JBIvJQLJeU3R60iNzSKpKRFIzmtsr6PqxuZly8B9Z2OzkoFIoCEmpMXTEBciGmJC8pwam0Vqp0QgLV5TNDarjUFaMiUCafEa29isNgZp6ZQIpMVrbGNzU9zQphKF5DI1FovUoaAglARqMgvPcK5LzSEyIJRI6jvekkhjqLFYpBEa28agEoXkOiUCkTo0to2hsb2W1EYhcVMiEKlDY9sYVKKQXKdEIJKG0aPDzWfbt4fX+tTtt4QShRJJy6ZEIBKz5l6iUCLJA+7erKbBgwe7SD554gn3oiL3cBkOU1FRmJ8Os123rZ7M0tu+T5/k2/fpk5n4q/fRp0+IuU+f+m3bFNu3BECFp7iuZv3CXt9JiUDyUWMuZI29kDf3RNISElFTJDIlApE81tgLYXNPJM09ETVFInNXIhDJe435RtncE0lzT0SN3b5abYlAjcUieaAxvZ4a29jd2F5TjW0sb+z2je21le3t06FEICJ1as6JpLknoow8qjVVUSFXJ1UNieSfbDa2ZruOX20ESgQikgOy3esn7l5DGn1URCQPaPRRERFJKdZEYGYnm9lCM1tsZuOSLG9rZk9Gy982s5I44xERkd3FlgjMrBVwPzAC6A+MMrP+NVa7BPi3u/cD7gJ+Elc8IiKSXJwlgiHAYndf4u5bgMnAaTXWOQ14LHr/W+AEM7MYYxIRkRriTAQ9gfcTPldG85Ku4+5VwDqga80dmdlYM6sws4rVq1fHFK6ISH5qHeO+k32zr9lFKZ11cPcJwAQAM1ttZssaH14sugFrsh1ELXI9Psj9GBVf4yi+xmlMfH1SLYgzEVQC+yZ87gWsSLFOpZm1BjoDH9e2U3fv3pRBNiUzq0jVPSsX5Hp8kPsxKr7GUXyNE1d8cVYNvQscYGZ9zawNcB7wXI11ngMujN6fDbzuze3GBhGRZi62EoG7V5nZN4GXgVbAw+4+z8xuIdzh9hzwS+BXZraYUBI4L654REQkuTirhnD3qcDUGvNuSnj/GfD/4owhwyZkO4A65Hp8kPsxKr7GUXyNE0t8zW6ICRERaVoaYkJEJM8pEYiI5Dklgnoys33NbJqZLTCzeWZ2TZJ1hpnZOjObFU03JdtXjDEuNbP3omPvNlSrBfdEYzzNMbOyDMZ2YMJ5mWVm683s2hrrZPz8mdnDZvaRmc1NmLeXmf3BzBZFr3um2PbCaJ1FZnZhsnViiu8OM/t79DucYmZdUmxb699DjPHdbGYfJPweR6bYttYxyWKM78mE2Jaa2awU28Z6/lJdUzL695dqfGpNKR7gAD2Asuh9R+AfQP8a6wwDns9ijEuBbrUsHwm8SLih70jg7SzF2Qr4EOiT7fMHHAuUAXMT5v0UGBe9Hwf8JMl2ewFLotc9o/d7Zii+4UDr6P1PksWXzt9DjPHdDFyXxt/AP4H9gDbA7Jr/T3HFV2P5z4GbsnH+Ul1TMvn3pxJBPbn7SnefGb3fACxg96Ezct1pwOMe/BXoYmY9shDHCcA/3T3rd4q7+3R2v5kxcSysx4DTk2z6ZeAP7v6xu/8b+ANwcibic/dXPAzNAvBXwk2bWZHi/KUjnTHJGq22+KLxzc4BJjX1cdNRyzUlY39/SgSNEA2bfRjwdpLFR5nZbDN70cwGZDSwMEzHK2Y2w8zGJlmezjhQmXAeqf/5snn+qn3B3VdC+GcF9k6yTq6cy68RSnnJ1PX3EKdvRlVXD6eo2siF83cMsMrdF6VYnrHzV+OakrG/PyWCBjKzDsDTwLXuvr7G4pmE6o5DgXuBZzMc3pfcvYwwBPiVZnZsjeVpjfEUp+hu81OB3yRZnO3zVx+5cC5vBKqAiSlWqevvIS4PAPsDg4CVhOqXmrJ+/oBR1F4ayMj5q+OaknKzJPPqff6UCBrAzAoJv7CJ7v5MzeXuvt7dN0bvpwKFZtYtU/G5+4ro9SNgCqH4nSidcaDiNgKY6e6rai7I9vlLsKq6yix6/SjJOlk9l1Hj4FeA0R5VGteUxt9DLNx9lbtvc/ftwC9SHDfb5681cCbwZKp1MnH+UlxTMvb3p0RQT1F94i+BBe5+Z4p19onWw8yGEM7z2gzF197MOla/JzQozq2x2nPABVHvoSOBddVF0AxK+S0sm+evhsSxsC4EfpdknZeB4Wa2Z1T1MTyaFzszOxm4ATjV3TelWCedv4e44ktsdzojxXHTGZMsTicCf3f3ymQLM3H+armmZO7vL66W8JY6AUMJRa85wKxoGglcDlwerfNNYB6hB8RfgaMzGN9+0XFnRzHcGM1PjM8IT4/7J/AeUJ7hc1hEuLB3TpiX1fNHSEorga2Eb1mXEJ6N8RqwKHrdK1q3HPi/hG2/BiyOposzGN9iQv1w9d/hg9G6xcDU2v4eMhTfr6K/rzmEi1qPmvFFn0cSesr8M5PxRfMfrf67S1g3o+evlmtKxv7+NMSEiEieU9WQiEieUyIQEclzSgQiInlOiUBEJM8pEYiI5DklApGImW2zXUdGbbKRMM2sJHHkS5FcEuujKkWamc3uPijbQYhkmkoEInWIxqP/iZm9E039ovl9zOy1aFC118ysdzT/CxaeDzA7mo6OdtXKzH4RjTn/ipntEa1/tZnNj/YzOUs/puQxJQKRnfaoUTV0bsKy9e4+BLgPGB/Nu48wnHcpYcC3e6L59wB/8jBoXhnhjlSAA4D73X0A8AlwVjR/HHBYtJ/L4/rhRFLRncUiETPb6O4dksxfCvyHuy+JBgf70N27mtkawrAJW6P5K929m5mtBnq5++cJ+yghjBt/QPT5BqDQ3W81s5eAjYRRVp/1aMA9kUxRiUAkPZ7ifap1kvk84f02drbRnUIY+2kwMCMaEVMkY5QIRNJzbsLrX6L3bxFGywQYDbwZvX8NuALAzFqZWadUOzWzAmBfd58G/BfQBditVCISJ33zENlpD9v1AeYvuXt1F9K2ZvY24cvTqGje1cDDZnY9sBq4OJp/DTDBzC4hfPO/gjDyZTKtgCfMrDNhVNi73P2TJvuJRNKgNgKROkRtBOXuvibbsYjEQVVDIiJ5TiUCEZE8pxKBiEieUyIQEclzSgQiInlOiUBEJM8pEYiI5Ln/D96S+vb7X18YAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting losses\n",
    "loss_values = history_dict['loss']\n",
    "val_loss_values = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss_values) + 1)\n",
    "\n",
    "plt.plot(epochs, loss_values, 'bo', label=\"Training Loss\")\n",
    "plt.plot(epochs, val_loss_values, 'b', label=\"Validation Loss\")\n",
    "\n",
    "plt.title('Training and Validation Loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss Value')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1f3/8deHACKCsioWhKC1VokBMcUFq1AVsa24gALiSpVqRapWv1LR2vordam1rkWxRa0iiFoQq7jbgq0bWEABlcWArLIvomLg8/vj3IRJmEkmJDOTZN7Px2MembnrJzeT87nn3HvPMXdHRESyV71MByAiIpmlRCAikuWUCEREspwSgYhIllMiEBHJckoEIiJZTokgy5lZjpltMbP21blsJpnZd80sJfdFl922mb1iZoNSEYeZ3WRmD+7u+iLJUiKoZaKCuPi1w8y+ivkct0Aqj7tvd/cm7r6kOpetqczsdTP7TZzpfc1smZlV6n/C3Xu5+9hqiOskMysss+3/5+6XVXXbFezTzeyaVO1DagclglomKoibuHsTYAlwWsy0XQokM6uf/ihrtEeB8+NMPx94wt13pDecjLoQWBf9TCt9L2sWJYI6xsx+b2ZPmdk4M9sMnGdmx5jZO2a2wcxWmNm9ZtYgWr5+dFaYG31+Ipo/xcw2m9nbZtaxsstG8081s0/NbKOZ3Wdm/zGzixLEnUyMPzezBWa23szujVk3x8z+bGZrzWwh0LucQ/QPoI2ZHRuzfkvgx8Dfo899zGxm9DstMbObyjnebxX/ThXFYWaXmNm8aLsLzeySaPo+wPNA+5ja3b7R3/LRmPXPMLM50TF6w8wOiZm31MyuMbMPo+M9zsz2KCfuJsBZwOXAYWbWpcz846O/x0Yz+9zMzo+mN45+xyXRvKlmtke8Gk0UU4/ofaW+l9E6h5vZa2a2zsxWmtn/RdPrW2g2W2hmm8xsupl9x8weMrPby8QwxcyGJjoOEnF3vWrpCygETioz7ffANuA0QqLfE/gBcBRQHzgQ+BQYGi1fH3AgN/r8BLAGKAAaAE8RzpQru+y+wGbg9GjeNcC3wEUJfpdkYnwO2AfIJZzJnhTNHwrMAdoBLYGp4aud8Lg9AjwY8/kKYHrM5x8BedHx6xz9jj+N5n03dtvAW8W/U0VxRH+TAwGL9vEVkB/NOwkojPO3fDR6fyiwJVqvAXBDdIwaRPOXAu8AbaJ9fwpcUs4xuDhapx4wBbgrZl7H6G93TnTsWwFdonkPAa8D+wM5wHFRPPHiXwr02M3v5T7AKuCXwB7A3kC3aN6vgVnAwdG2ugAtgGOBz4F60XL7AVuBVpn+X63pr4wHoFcV/niJE8EbFax3LfB09D5e4R5bSPYBPtqNZQcD02LmGbCCBIkgyRiPjpn/D+Da6P3U2EKPcHbv5Wy7ByGR7BF9fhe4spzl7wf+GL0vLxFUNo5/AldE7ytKBL8DnoyZVw9YCRwXfV4KDIiZfxdwfzn7/hdwZ/T+/KjQrR99vqn42JdZJwf4BugUZ14yiaAy38vziUnOZZZbCPwkwbxPgZ7R+6uAyVX5H8uWl5qG6qbPYz+Y2ffN7IWoer0JuIVwlpfIypj3W4Emu7Hsd2Lj8PCfuTTRRpKMMal9AYvLiRfg38BG4DQz+x5wBDAuJpZjzOxfZrbazDYCl8SJJZ5y4zCzn5rZu1FTxwagV5LbLd52yfY8XMtYCrSNWSapv1vUtHc8UHxNaWK0bHFT1gGEwras/YCGCeYlozLfywOABQm2kyg+CM1750XvzwMe381Ys4oSQd1U9pbFh4CPgO+6+97Abwhn6Km0gtBEAoCZGaULrbKqEuMKQuFQrNzbW6Ok9DhwAeHM80V3XxOzyHjgWeAAd98H+GuSsSSMw8z2BJ4BbgX2c/dmwCsx263oNtPlQIeY7dUjHN9lScRV1gXRfqeY2UpCgdswmg6hwD4oznqrCM078eZ9CTSOia8+oYkqVmW+l4liqGje48BZZnZEtMzzCZaTGEoE2aEp4Qz4SzM7FPh5Gvb5T6CrmZ0WFQq/BFqnKMYJwFVm1ja68Ht9Eus8RjgDHhy9LxvLOnf/2syOBgZUQxx7EArb1cB2M/spcGLM/FVAKzNrWs62+5hZj+iC6nWEdvx3k4wt1gWEQrdLzKt/tP3mhCa/3hZuqa1vZq3MrLO7byfcdXW3mbWJLo53j+L5GGhqZqdEn28mXDsoT3l/88mEi+dDzayhme1tZt2ieX8Ffm9mB1nQxcxaALj7YmAm4W/6tLt/vRvHJ+soEWSHXxFuEdxMOAt7KtU7dPdVhMLlLmAt4ezsf4Q25uqOcRThAuaHwPuEM++K4lsIvAc0Al4oM/ty4Nbo7pYbCIVwleJw9w3A1YRmmHVAP0KyLJ7/EaEWUhjdRbNvmXjnEI7PKEIy6Q30cfdvk4wNADM7jtDM9IC7ryx+RXEVAv3d/TPCRd3ro1g/AA6PNnE1MA+YEc37A2Duvh64klAAL4vmxTZVxZPwb+7uG4GTgb7AF4S2/xOi2X8EJhGO9SZgNOHvWOyxKF41CyXJoosqIillZjmE5o1+7j4t0/FI3WVmPwL+BhzoKuCSohqBpIyZ9TazfaL72W8Cighn4SIpYWYNCc2QDysJJE+JQFLpOGAR4T783sAZ7p6oaUikSszscGA94ZmCeytYXGKoaUhEJMupRiAikuVqXcdPrVq18tzc3EyHISJSq8yYMWONu8e9hbvWJYLc3FymT5+e6TBERGoVM0v4xL2ahkREspwSgYhIllMiEBHJckoEIiJZTolARCTLKRGIiFRk7FjIzYV69cLPsbsMD16rKRGISMWqWhBmuiCtyv7HjoUhQ2DxYnAPP4cMqfw2avLxy/QQaZV9HXnkkS4iafTEE+6NG7uHYjC8GjcO09OxfvE2OnRwNws/K7tuVfbfoUPpdYtfHTqkZ//VcfzcnQRDf7rXwjGLlQhEdkNVCtKqFoSZLkirun+z+OubpWf/VV0/Ul4iUNOQSG2QyaaNJUsqN7261x8xArZuLT1t69YwPR37b59g5NNE06t7/1VdPwlKBCKpVh3tw1UpyKtakFa1IMx0QVrV/Y8cCY0bl57WuHGYno79V3X9ZCSqKtTUl5qGJO0y2T7tnvmmjUy3cWe6aal4G5n6DugagRKBZFimCzH3zLdRu1etIKzq+pkuyKtDJo9fpLxEUOsGpikoKHD1Pippk5sbmmLK6tABCgsrXr9evVB0lWUGO3akJ4bipqXY5qHGjWH0aBg0KLkYMm3s2NCUtWRJaBIZObL2xF5DmNkMdy+IN0/XCKTuq0obfabbp6HqbdSDBoVCv0OHkIA6dKhdSQBCrIWFIXkWFtau2GsBJQKp26p6oTXTFxqhegpyFaRSDiUCqduqesdMTTkbV0EuKaRrBFK3VUcbvdqnpQ7QNQKp3arSxl8dbfQ6G5c6TolAaraqtvFXRxu9SB2nRCA1W1Xb+OvCHTMiKaZrBFKzVUcbv4joGoHUYunoZ0UkyykRSM2mNn6RlFMikJpNbfwiKVc/0wGIVGjQIBX8IimkGoGISJZTIhARyXJKBJJ6VR2hS0RSStcIJLXK9oVf/GQwqN1fpIZQjUBSq6pPBotIyqU0EZhZbzP7xMwWmNnwOPM7mNnrZjbbzP5lZu1SGY9kQFUHdhGRlEtZIjCzHOAB4FTgMGCgmR1WZrE7gb+7ez5wC3BrquKRDNGTwSI1XiprBN2ABe6+yN23AeOB08sscxjwevT+zTjzpbbTk8EiNV4qE0Fb4POYz0ujabFmAX2j92cCTc2sZdkNmdkQM5tuZtNXr16dkmAlRfRksEiNl8pEYHGmle1G8lrgBDP7H3ACsAwo2mUl99HuXuDuBa1bt67+SCW1NLCLSI2WyttHlwIHxHxuByyPXcDdlwNnAZhZE6Cvu29MYUwiIlJGKmsE7wMHm1lHM2sIDAAmxy5gZq3MrDiGXwNjUhiP7C49ECZSp6UsEbh7ETAUeBmYB0xw9zlmdouZ9YkW6wF8YmafAvsBuoJY01R1qEgRqfE0QpmULzc3FP5ldegQ2vtFpFbQCGWy+/RAmEidp0Qg5dMDYSJ1nhKBlE8PhInUeUoEUj49ECZS56kbaqmYhooUqdNUIxARyXJKBCIiWU6JQEQkyykRiIhkOSUCEZEsp0SQDdRpnIiUQ7eP1nXFncYVDyBf3Gkc6JZQEQFUI6j7RozYmQSKbd0apouIoERQ96nTOBGpgBJBXadO40SkAkoEdZ06jRORCigR1HXqNE5EKqC7hrKBOo0TkXKoRiAikuWUCEREspwSgYhIllMiEBHJckoEIiJZTolARCTLKRGIiGQ5JQIRkSynRCAikuWUCGoDDSwjIimkLiZqOg0sIzWBO3z4IWzYAN98s3uvHTvg4IPhiCOga1fYf//Q/5VknLl7pmOolIKCAp8+fXqmw0if3NxQ+JfVoQMUFqY7muz01VfwySfh79C6NbRrFwqxBg0yHVnqffUVPPkk3H03fPRR8uvVqwd77FH6BaXHwdh335AUYl8HHRTWlWpnZjPcvSDePNUIajoNLJM+W7bAxx/D3LmlX4sWhTPiWGbQpk1IColebdvuLACTsW0bbN4MmzbFf23bBj/6ERx2WOrPpFesgL/8BR58ENasgc6dQ6+1Bx64awEf71U/QdGyeTPMmgX/+9/O1513QlFRmN+0adhXbHI47DBo2DC1v2+WU42gplONINi+HT7/PP6ZZmXPIDdtgnnzShf2c+aUPs4NGsAhh4RCqPiVmwtr18LSpaVfn38efm7atOu+imsQ7drBfvvB118nLui//jq5+L//fejXL7zy86s3KcyYAffcA+PHh8K5Tx+46io44YTUJZ9vvgnHPzY5zJoFX34Z5jdsCHl5ISnk5e38e7Rtq6alSiivRqBEUNOVvUYAYWCZuj6mwOrV8M47O1/vvRfO2ONp0CC5s9QdO+DTT0OhXWyPPULBWly4dOoUfh54YOWbfjZtgmXLdk0Uxa9Vq8Lfbu+9E7/22SfxvKIi+Oc/4Zln4F//Cr/Pd7+7Myl07bp7BeP27fDcc6H5Z9o0aNIEBg+GYcNCU00mbN8OCxaUTg7/+1+onRRr2rR0oi5+tW9fs5qX1q+HyZPDsb3gAjj++IyEoURQ240dGwabX7IkfMlHjkxvEti0CR54IBQW++8fCoeDDgqF0EEHhZgSNQUkY9s2mD27dMG/cGGYl5MTmgqOPhq6dAkF3e5erHQPFytjC42OHcM+apvVq2HixJAU3ngjFJwdO+5MCj/4QcVJYeNG+Nvf4L77Qu0yNzcU/oMHh4RUE61evWvT3dy5sHLlzmUaN4ZDD901QaTzb712bfh/efppeO21kMQbNgx/p7vugiuvTHttRolAds+GDaGQ+POfw1lNt26hjXfRolCwFqtfPzRVFSeI2NeBB8Jee5Xe7tKlpQv9GTN2Novsvz8cc0wo+I8+Go48ctehNqW04kLnmWfg1VdDodO+PfTtC2efDUcdVfoMecGC8HcdMybUso4/PjT/9OlTO5MiwLp18RPEsmU7l2nUKNSair9bxxwTmuyqyxdfwKRJiZPzIYfAhReGv9V558FDD6X1u52xRGBmvYF7gBzgr+5+W5n57YHHgGbRMsPd/cXytqlEkAbr14d24rvvDmeNp50Gv/kNFETfoR07wj/YwoXxXxs2lN5emzYhKbRoAR98sPOfs2HDUNAX/1MefXT4x1S77+4rboZ45hl45ZVQ22rbNiSFY48NdwA9/3xI3gMGwC9/Gf4GddXGjTuvB334YWhinDFj54lM27Y7E0Pxiceeeya//ZUrd9bMYpvrzj47FP5HHFH6+7xjR6jR33xzqOlOnBhqYmmQkURgZjnAp8DJwFLgfWCgu8+NWWY08D93H2VmhwEvuntuedtVIkihtWvD2f+994Yz/zPPhJtuCl/myli3Ln6CKL77pPifrnPnyt1VI5WzcePOawpTpoTCr1UruPzy8Np//0xHmBnbtoWL0bG10kWLwrz69Xd+R4tPTg48sHRhvmwZ/OMf4bhOmxaaHA85ZGfhn8wF/BdeCM27OTnhwvzJJ6fu941kKhEcA/zW3U+JPv8awN1vjVnmIWCRu98eLf8ndz+2vO0qEaTA6tWh3fL++0NTQb9+cOON4R9C6obNm8PF1h/8oHJnvNniiy/g3Xfh7bd33pxQfNdSq1YhIXTqBG+9Bf/5T5iel7ez2Wd3bumdPz+cbM2bB7feCtddl9LacKYSQT+gt7tfEn0+HzjK3YfGLLM/8ArQHNgLOMndZ8TZ1hBgCED79u2PXBzvdkqpvFWrwj3cf/lLeHCof/9wUTovL9ORiWTW9u3hltbYWsO8eeHkqLjw//73q76fLVvCxfmnn4ZzzgkX75s0qfp248jUA2XxUlvZrDMQeNTd/xTVCB43szx331FqJffRwGgINYKURJtNVqyAP/4xPCz0zTdw7rlwww3hTgsRCU02+fnhVdyly7Zt1f9gW5Mm8NRToaY2fHi4ljFxYrjOkEapvNl2KXBAzOd2wPIyy/wMmADg7m8DjYBWKYwpuy1bFi4OHnhguA5wzjnhLOfxx5UERCqSqqebzUKz0EsvwfLlISm8WO49M9UulYngfeBgM+toZg2BAcDkMsssAU4EMLNDCYlgdQpjyk5LlsAVV4QE8Je/hItUn3wCjz4K3/tepqMTEQgXjKdPD3cR/fSn8P/+X7jLKA1SlgjcvQgYCrwMzAMmuPscM7vFzPpEi/0KuNTMZgHjgIu8tj3YUJMVFsLPfx6qmQ8/HO5h/vRT+OtfM/fEqIgk1rFjuBh97rnhlu2zzorfdUk10wNlddGiRfCHP8Bjj4UHiX72s9D+2L59piMTkWS4h+bbX/0qnMhNmlTli9PlXSyuQR1ySJXNnw8XXxyae554ItwrvnBhaA5SEhCpPczC9bzXXgvP5XTrFpJBiigR1AUffwznnx/OGMaPD/2YfPZZOKOozkfoRSS9evQIT0Ifckh45uChh1KyG41HUJvNmRMeVx8/PjwkdM01oSrZpk2mIxOR6nLAAeEJ5uHDoXfvlOyiwkRgZkOBse6+PiURSOXNng2//314xH2vveD660MSaN0605GJSCo0ahT6/kqRZJqG2gDvm9kEM+ttph7BMmbmzHAXQefO4Z7jG24IdwbdequSgIjstgoTgbvfCBwM/A24CJhvZn8wM91/mKyxY8O9wfXqhZ9jx1Zu/a+/hqFDQ+dvb7wRei5cvDjUClq2TEXEIpJFkrpG4O5uZiuBlUARoW+gZ8zsVXf/v1QGWOuVHWFs8eKdj6wnM7jMp5+GPoBmzgx9xt98MzRrlrp4RSTrVFgjMLNhZjYDuAP4D3C4u18OHAn0TXF8td+IEaWHmYTwecSIitd9/PEwkMbnn4fuhP/8ZyUBEal2ydQIWgFnuXupLj/dfYeZ/TQ1YdUhS5ZUbjqE7m+HDg1dQPzwh2EwEd0GKiIpkszF4heBdcUfzKypmR0F4O7zUhVYnZHoQa5E02fPDiOBPfZYGBTmjTeUBEQkpZJJBKOALTGfv4ymSTJGjtx1XNLGjcP0WO7hYZGjjgpDPb72GtxyS9UGhRcRSUIyicBiO4KLxgpQ6ZSsQYNg9OgwuLtZ+Dl6dOkLxRs3hvFjL7ssDCQ+axb86EeZi1lEskoyBfoiMxvGzlrAL4BFqQupDho0KPEdQu+/H5LA4sVw222hX/J66vlDRNInmRLnMuBYYBlhsJmjiIaNlCpwD+MEd+8ORUUwdWp4QlhJQETSrMIagbt/QRhURqrLmjVw0UXwwgtwxhlhnNIWLTIdlYhkqWT6GmpEGFKyE2EEMQDcfXAK46q7pk2DgQNh9erQO+jQoeHagYhIhiTTDvE4ob+hU4B/E8Ye3pzKoOqk7dtDlxA9eoSeQt9+O3QXrSQgIhmWTCL4rrvfBHzp7o8BPwEOT21YdYx76FbippvCheEPPghPDIuI1ADJJIJvo58bzCwP2AfITVlEddGvfw1jxoRE8MQT0LRppiMSESmRzO2jo82sOXAjMBloAtyU0qjqkrvugttvD88I/O53agoSkRqn3ERgZvWATdGgNFOBA9MSVV3x+ONhxLB+/eD++5UERKRGKrdpKHqKeGiaYqlbXnwRBg8OTwg/8QTk5GQ6IhGRuJK5RvCqmV1rZgeYWYviV8ojq83efjvUAvLzYeJE2GOPTEckIpJQMtcIip8XuCJmmqNmovjmzIGf/ATatg21gr33znREIiLlSubJ4o7pCKROWLIETjkl1ABeeQX22y/TEYmIVCiZJ4sviDfd3f9e/eHUYmvWQK9esGVL6Deoo/KniNQOyTQN/SDmfSPgROADQImg2JYtoTlo8eJQE8jPz3REIiJJS6Zp6MrYz2a2D6HbCQHYtg369oXp08OF4R/+MNMRiYhUyu4MMLMVOLi6A6mVduwIvYi+8kroQbRPn0xHJCJSaclcI3iecJcQhNtNDwMmpDKoWsEdrr4axo2DW28NzwyIiNRCydQI7ox5XwQsdvelKYqn9rj11tCN9NVXhwFlRERqqWQSwRJghbt/DWBme5pZrrsXpjSymuzhh2HECDjvPLjzTnUdISK1WjJPFj8N7Ij5vD2alp0mTgwdyJ16auhRVENLikgtl0wpVt/dtxV/iN43TF1INdi//hVGF+vWDZ5+Gho0yHREIiJVlkwiWG1mJbfDmNnpwJrUhVQDjR0L3/kO9OwZ7hS68ELYa69MRyUiUi2SuUZwGTDWzO6PPi8F4j5tXJaZ9QbuAXKAv7r7bWXm/xnoGX1sDOzr7s2S2XbajB0Ll14KX30VPn/7behaumlTGDQos7GJiFQDc/eKlwLMrEm0fFLjFZtZDvApcDIhebwPDHT3uQmWvxI4wt3LvQ+zoKDAp0+fnlTM1SI3NzwxXFaHDlBYmL44RESqwMxmuHtBvHkVNg2Z2R/MrJm7b3H3zWbW3Mx+n8R+uwEL3H1RdF1hPHB6OcsPBMYlsd30ipcEIHQwJyJSByRzjeBUd99Q/CEarezHSazXFvg85vPSaNouzKwD0BF4I8H8IWY23cymr169OoldV6NWreJPb98+vXGIiKRIMokgx8xKRlYxsz2BZEZaiXdzfaJ2qAHAM+6+Pd5Mdx/t7gXuXtC6deskdl1Ntm0Lt4eWfU6gcWMYOTJ9cYiIpFAyieAJ4HUz+5mZ/Qx4FXgsifWWAgfEfG4HLE+w7ABqYrPQ6NHwxRdw7bXhmoBZ+Dl6tC4Ui0idkdTF4ujun5MIZ/nrgf3d/YoK1qlPuFh8IrCMcLH4XHefU2a5Q4CXgY6eRDBpu1i8eTMcdBB06gRvvKGnh0WkVqvSxeLISsLTxX0JBfu8ilZw9yLCwPcvR8tPcPc5ZnZL7HMJhIvE45NJAml1112wejXcfruSgIjUaQlrBGb2PUKTzUBgLfAUcK27d0hfeLtKS43giy9CbeCUU+CZZ1K7LxGRNCivRlDeA2UfA9OA09x9QbShq1MQX83z+9+HB8h0QVhEskB5TUN9CU1Cb5rZw2Z2IvHvBKpbFi2CBx+En/0MDjkk09GIiKRcwkTg7hPdvT/wfeBfwNXAfmY2ysx6pSm+9LvpJqhfH26+OdORiIikRYUXi939S3cf6+4/JdwCOhMYnvLIMmHmTHjySbjqqtDJnIhIFqhUZ/ruvs7dH3L3H6UqoIz69a+heXP4v//LdCQiImmzO4PX101vvgkvvRRGHGtWszpAFRFJJQ2vBWEg+uuvhwMOgCvKfU5ORKTOUY0A4Nln4f33w9CTjRplOhoRkbRSjaCoKAxE36kTXJDUeDsiInWKagRjxsCnn8Jzz0FOTqajERFJu+yuEWzdCr/9LXTvDqedluloREQyIrtrBPfcAytWwNNPq2M5Ecla2VsjWLsWbrsN+vQJNQIRkSyVvYng1lthyxb4wx8yHYmISEZlZyJYsgTuvx8uvDDcLSQiksWyMxEUdyj3u99lNg4RkRog+xLBRx/B3/8OQ4eGJ4lFRLJc9iWCG26Apk1DB3MiIpJlieCtt+D550O/Qi1bZjoaEZEaIXsSQXHHcvvvD7/8ZaajERGpMbLngbLnn4f//hceeggaN850NCIiNUb21Ai+/RZ69IDBgzMdiYhIjZI9iaBv3zD4TP3sqQSJiCQjexKBiIjEpUQgIpLllAhERLKcEoGISJZTIhARyXJKBCIiWU6JQEQkyykRiIhkOSUCEZEsp0QgIpLllAhERLJcShOBmfU2s0/MbIGZDU+wzDlmNtfM5pjZk6mMR0REdpWyHtjMLAd4ADgZWAq8b2aT3X1uzDIHA78Gurv7ejPbN1XxiIhIfKmsEXQDFrj7InffBowHTi+zzKXAA+6+HsDdv0hhPCIiEkcqE0Fb4POYz0ujabG+B3zPzP5jZu+YWe8UxiMiInGksnN+izPN4+z/YKAH0A6YZmZ57r6h1IbMhgBDANq3b1/9kYqIZLFU1giWAgfEfG4HLI+zzHPu/q27fwZ8QkgMpbj7aHcvcPeC1q1bpyxgEZFslMpE8D5wsJl1NLOGwABgcpllJgE9AcysFaGpaFEKYxIRkTJSlgjcvQgYCrwMzAMmuPscM7vFzPpEi70MrDWzucCbwHXuvjZVMYmIyK7MvWyzfc1WUFDg06dPz3QYIiK1ipnNcPeCePP0ZLGISJZTIhARyXJKBCIiWU6JQEQkyykRiIhkOSUCEZEsp0QgIpLlUtnXkIik0LfffsvSpUv5+uuvMx2K1CCNGjWiXbt2NGjQIOl1lAhEaqmlS5fStGlTcnNzMYvXx6NkG3dn7dq1LF26lI4dOya9npqGRGqpr7/+mpYtWyoJSAkzo2XLlpWuJSoRiNRiSgJS1u58J5QIRESynBKBSLYYOxZyc6FevfBz7Njd3tTatWvp0qULXbp0oU2bNrRt27bk87Zt25LaxsUXX8wnn3xS7jIPPPAAY6sQZ1mrVq2ifv36/O1vf6u2bdYF6n1UpJaaN28ehx56aHILjx0LQ4bA1q07pzVuDKNHw6BBVYrjt7/9LU2aNOHaa68tNd3dcXfq1as555v33nsvTz/9NHvssQevvfZayvZTVMjBuXYAABM7SURBVFRE/fqZuxcn3ndDvY+KZLsRI0onAQifR4yo1t0sWLCAvLw8LrvsMrp27cqKFSsYMmQIBQUFdOrUiVtuuaVk2eOOO46ZM2dSVFREs2bNGD58OJ07d+aYY47hiy++AODGG2/k7rvvLll++PDhdOvWjUMOOYT//ve/AHz55Zf07duXzp07M3DgQAoKCpg5c2bc+MaNG8fdd9/NokWLWLlyZcn0F154ga5du9K5c2d69eoFwObNm7nwwgs5/PDDyc/PZ9KkSSWxFhs/fjyXXHIJAOeddx6/+tWv6NmzJzfccAPvvPMOxxxzDEcccQTdu3dn/vz5QEgSV199NXl5eeTn5/OXv/yFl19+mbPPPrtku1OmTOGcc86p8t8jWbp9VCQbLFlSuelVMHfuXB555BEefPBBAG677TZatGhBUVERPXv2pF+/fhx22GGl1tm4cSMnnHACt912G9dccw1jxoxh+PDhu2zb3XnvvfeYPHkyt9xyCy+99BL33Xcfbdq04dlnn2XWrFl07do1blyFhYWsX7+eI488kn79+jFhwgSGDRvGypUrufzyy5k2bRodOnRg3bp1QKjptG7dmg8//BB3Z8OGDXG3G2vhwoW8/vrr1KtXj40bN/LWW2+Rk5PDSy+9xI033shTTz3FqFGjWL58ObNmzSInJ4d169bRrFkzhg0bxtq1a2nZsiWPPPIIF198cWUP/W5TjUAkG7RvX7npVXDQQQfxgx/8oOTzuHHj6Nq1K127dmXevHnMnTt3l3X23HNPTj31VACOPPJICgsL4277rLPO2mWZt956iwEDBgDQuXNnOnXqFHfdcePG0b9/fwAGDBjAuHHjAHj77bfp2bMnHTp0AKBFixYAvPbaa1xxxRVAuBOnefPmFf7uZ599dklT2IYNGzjrrLPIy8vj2muvZc6cOSXbveyyy8jJySnZX7169Tj33HN58sknWbduHTNmzCipmaSDagQi2WDkyPjXCEaOrPZd7bXXXiXv58+fzz333MN7771Hs2bNOO+88+Le496wYcOS9zk5ORQVFcXd9h577LHLMsle5xw3bhxr167lscceA2D58uV89tlnuHvcWy7jTa9Xr16p/ZX9XWJ/9xEjRnDKKafwi1/8ggULFtC7d++E2wUYPHgwffv2BaB///4liSIdVCMQyQaDBoULwx06gFn4WQ0XiiuyadMmmjZtyt57782KFSt4+eWXq30fxx13HBMmTADgww8/jFvjmDt3Ltu3b2fZsmUUFhZSWFjIddddx/jx4+nevTtvvPEGixcvBihpGurVqxf3338/EArv9evXU69ePZo3b878+fPZsWMHEydOTBjXxo0badu2LQCPPvpoyfRevXoxatQotm/fXmp/BxxwAK1ateK2227joosuqtpBqSQlApFsMWgQFBbCjh3hZ4qTAEDXrl057LDDyMvL49JLL6V79+7Vvo8rr7ySZcuWkZ+fz5/+9Cfy8vLYZ599Si3z5JNPcuaZZ5aa1rdvX5588kn2228/Ro0axemnn07nzp0ZFB2Xm2++mVWrVpGXl0eXLl2YNm0aALfffju9e/fmxBNPpF27dgnjuv7667nuuut2+Z1//vOf06ZNG/Lz8+ncuXNJEgM499xz6dixI9/73veqdEwqS7ePitRSlbp9tA4rKiqiqKiIRo0aMX/+fHr16sX8+fMzevvm7rrssss45phjuPDCC6u0ncrePlr7jpSISIwtW7Zw4oknUlRUhLvz0EMP1cok0KVLF5o3b869996b9n3XvqMlIhKjWbNmzJgxI9NhVFmiZx/SQdcIRESynBKBiEiWUyIQEclySgQiIllOiUBEKq1Hjx67PBx2991384tf/KLc9Zo0aQKEp3r79euXcNsV3SJ+9913szXmKekf//jHSfUFlKziDuyyhRKBiFTawIEDGT9+fKlp48ePT7rw/M53vsMzzzyz2/svmwhefPHFUr2CVsW8efPYsWMHU6dO5csvv6yWbcaTqBuNTFAiEKkLrroKevSo3tdVVyXcXb9+/fjnP//JN998A4SePZcvX85xxx1Xcl9/165dOfzww3nuued2Wb+wsJC8vDwAvvrqKwYMGEB+fj79+/fnq6++Klnu8ssvL+nC+uabbwbCmALLly+nZ8+e9OzZE4Dc3FzWrFkDwF133UVeXh55eXklXVgXFhZy6KGHcumll9KpUyd69epVaj+xnnzySc4//3x69erF5MmTS6YvWLCAk046ic6dO9O1a1cWLlwIwB133MHhhx9O586dS3pMja3VrFmzhtzcXCB0NXH22Wdz2mmn0atXr3KP1d///veSp4/PP/98Nm/eTMeOHfn222+B0H1Hbm5uyeeq0HMEIlJpLVu2pFu3brz00kucfvrpjB8/nv79+2NmNGrUiIkTJ7L33nuzZs0ajj76aPr06ZNwLN1Ro0bRuHFjZs+ezezZs0t1Iz1y5EhatGjB9u3bOfHEE5k9ezbDhg3jrrvu4s0336RVq1altjVjxgweeeQR3n33Xdydo446ihNOOKGkf6Bx48bx8MMPc8455/Dss89y3nnn7RLPU089xauvvsonn3zC/fffX1LLGTRoEMOHD+fMM8/k66+/ZseOHUyZMoVJkybx7rvv0rhx45J+g8rz9ttvM3v27JKuueMdq7lz5zJy5Ej+85//0KpVK9atW0fTpk3p0aMHL7zwAmeccQbjx4+nb9++NGjQoDJ/uriUCETqgujMN52Km4eKE8GYMWOA0EHbDTfcwNSpU6lXrx7Lli1j1apVtGnTJu52pk6dyrBhwwDIz88nPz+/ZN6ECRMYPXo0RUVFrFixgrlz55aaX9Zbb73FmWeeWdIL6FlnncW0adPo06cPHTt2pEuXLkDirq7ff/99WrduTYcOHWjXrh2DBw9m/fr11K9fn2XLlpX0V9SoUSMgdCl98cUX07hxY2BnF9blOfnkk0uWS3Ss3njjDfr161eS6IqXv+SSS7jjjjs444wzeOSRR3j44Ycr3F8ysqNpqBrHahWR4IwzzuD111/ngw8+4Kuvvio5kx87diyrV69mxowZzJw5k/322y9u19Ox4tUWPvvsM+68805ef/11Zs+ezU9+8pMKt1Ne32nFXVhD4q6ux40bx8cff0xubi4HHXQQmzZt4tlnn0243URdStevX58dO3YA5XdVnehYJdpu9+7dKSws5N///jfbt28vaV6rqrqfCIrHal28GNzDzyFDlAxEqqhJkyb06NGDwYMHl7pIvHHjRvbdd18aNGjAm2++WdK9cyLHH398yQD1H330EbNnzwZCG/hee+3FPvvsw6pVq5gyZUrJOk2bNmXz5s1xtzVp0iS2bt3Kl19+ycSJE/nhD3+Y1O+zY8cOnn76aWbPnl3SVfVzzz3HuHHj2HvvvWnXrh2TJk0C4JtvvmHr1q306tWLMWPGlFy4Lm4ays3NLen2oryL4omO1YknnsiECRNYu3Ztqe0CXHDBBQwcOLBaRzCr+4kgTWO1imSjgQMHMmvWrJIRwiC0pU+fPp2CggLGjh3L97///XK3cfnll7Nlyxby8/O544476NatGxBu4TziiCPo1KkTgwcPLtWd85AhQzj11FNLLhYX69q1KxdddBHdunXjqKOO4pJLLuGII45I6neZOnUqbdu2LRlDAEJimTt3LitWrODxxx/n3nvvJT8/n2OPPZaVK1fSu3dv+vTpQ0FBAV26dOHOO+8E4Nprr2XUqFEce+yxJRex40l0rDp16sSIESM44YQT6Ny5M9dcc02pddavX1+tt7emtBtqM+sN3APkAH9199vKzL8I+COwLJp0v7v/tbxtVrob6nr1Qk1g1+BCv+witZS6oc5OzzzzDM899xyPP/54wmVqTDfUZpYDPACcDCwF3jezye5edvigp9x9aKrioH370BwUb7qISC1y5ZVXMmXKFF588cVq3W4qm4a6AQvcfZG7bwPGA6encH/xjRwZxmaNlaKxWkVEUum+++5jwYIF1T6CWSoTQVvg85jPS6NpZfU1s9lm9oyZHVDtUWRorFaRdKhtIwxK6u3OdyKViSDe0yNlI3weyHX3fOA14LG4GzIbYmbTzWz66tWrKx9JBsZqFUm1Ro0asXbtWiUDKeHurF27tuQ5h2Sl8oGypUDsGX47YHnsAu6+Nubjw8Dt8Tbk7qOB0RAuFldvmCK1U7t27Vi6dCm7dXIkdVajRo1o165dpdZJZSJ4HzjYzDoS7goaAJwbu4CZ7e/uK6KPfYB5KYxHpE5p0KABHTt2zHQYUgekLBG4e5GZDQVeJtw+Osbd55jZLcB0d58MDDOzPkARsA64KFXxiIhIfCl9jiAVKv0cgYiIlPscQd1/slhERMpV62oEZrYaKL/zksxpBSR+njzzFF/V1PT4oObHqPiqpirxdXD31vFm1LpEUJOZ2fREVa+aQPFVTU2PD2p+jIqvalIVn5qGRESynBKBiEiWUyKoXqMzHUAFFF/V1PT4oObHqPiqJiXx6RqBiEiWU41ARCTLKRGIiGQ5JYJKMrMDzOxNM5tnZnPM7JdxlulhZhvNbGb0+k2aYyw0sw+jfe/yGLYF95rZgqgL8K5pjO2QmOMy08w2mdlVZZZJ+/EzszFm9oWZfRQzrYWZvWpm86OfzROse2G0zHwzuzBNsf3RzD6O/n4TzaxZgnXL/S6kOMbfmtmymL/jjxOs29vMPom+j8PTGN9TMbEVmtnMBOum9BgmKlPS+v1zd70q8QL2B7pG75sCnwKHlVmmB/DPDMZYCLQqZ/6PgSmErsKPBt7NUJw5wErCgy4ZPX7A8UBX4KOYaXcAw6P3w4Hb46zXAlgU/WwevW+ehth6AfWj97fHiy2Z70KKY/wtcG0S34GFwIFAQ2BW2f+nVMVXZv6fgN9k4hgmKlPS+f1TjaCS3H2Fu38Qvd9M6DE13oA7NdnpwN89eAdoZmb7ZyCOE4GF7p7xJ8XdfSqh48NYp7NzjIzHgDPirHoK8Kq7r3P39cCrQO9Ux+bur7h7UfTxHUI37xmT4PglIy0jGZYXn5kZcA4wrrr3m4xyypS0ff+UCKrAzHKBI4B348w+xsxmmdkUM+uU1sDCAECvmNkMMxsSZ36yo8el2gAS//Nl8vgV28+jbtKjn/vGWaYmHMvBhBpePBV9F1JtaNR8NSZB00ZNOH4/BFa5+/wE89N2DMuUKWn7/ikR7CYzawI8C1zl7pvKzP6A0NzRGbgPmJTm8Lq7e1fgVOAKMzu+zPxkRo9LKTNrSBiD4uk4szN9/Cojo8fSzEYQunEfm2CRir4LqTQKOAjoAqwgNL+UlfHvIjCQ8msDaTmGFZQpCVeLM63Sx0+JYDeYWQPCH2ysu/+j7Hx33+TuW6L3LwINzKxVuuJz9+XRzy+AiYTqd6wKR49Lg1OBD9x9VdkZmT5+MVYVN5lFP7+Is0zGjmV0YfCnwCCPGozLSuK7kDLuvsrdt7v7DsIIhPH2ndHvopnVB84Cnkq0TDqOYYIyJW3fPyWCSoraE/8GzHP3uxIs0yZaDjPrRjjOa+Mtm4L49jKzpsXvCRcVPyqz2GTggujuoaOBjb5zpLh0SXgWlsnjV8ZkoPgujAuB5+Is8zLQy8yaR00fvaJpKWVmvYHrgT7uvjXBMsl8F1IZY+x1pzMT7LtkJMOoljiAcNzT5STgY3dfGm9mOo5hOWVK+r5/qboSXldfwHGEqtdsYGb0+jFwGXBZtMxQYA7hDoh3gGPTGN+B0X5nRTGMiKbHxmfAA4S7NT4ECtJ8DBsTCvZ9YqZl9PgRktIK4FvCWdbPgJbA68D86GeLaNkC4K8x6w4GFkSvi9MU2wJC23Dxd/DBaNnvAC+W911I4/F7PPp+zSYUavuXjTH6/GPCnTILUxVjvPii6Y8Wf+9ilk3rMSynTEnb909dTIiIZDk1DYmIZDklAhGRLKdEICKS5ZQIRESynBKBiEiWUyIQiZjZdivdM2q19YRpZrmxPV+K1CT1Mx2ASA3ylbt3yXQQIummGoFIBaL+6G83s/ei13ej6R3M7PWoU7XXzax9NH0/C2MEzIpex0abyjGzh6M+518xsz2j5YeZ2dxoO+Mz9GtKFlMiENlpzzJNQ/1j5m1y927A/cDd0bT7Cd155xM6fbs3mn4v8G8PneZ1JTyRCnAw8IC7dwI2AH2j6cOBI6LtXJaqX04kET1ZLBIxsy3u3iTO9ELgR+6+KOocbKW7tzSzNYRuE76Npq9w91Zmthpo5+7fxGwjl9Bv/MHR5+uBBu7+ezN7CdhC6GV1kkcd7omki2oEIsnxBO8TLRPPNzHvt7PzGt1PCH0/HQnMiHrEFEkbJQKR5PSP+fl29P6/hN4yAQYBb0XvXwcuBzCzHDPbO9FGzawecIC7vwn8H9AM2KVWIpJKOvMQ2WlPKz2A+UvuXnwL6R5m9i7h5GlgNG0YMMbMrgNWAxdH038JjDaznxHO/C8n9HwZTw7whJntQ+gV9s/uvqHafiORJOgagUgFomsEBe6+JtOxiKSCmoZERLKcagQiIllONQIRkSynRCAikuWUCEREspwSgYhIllMiEBHJcv8fKGe3UhVtc7gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training and Validation Accuracy\n",
    "\n",
    "acc_values = history_dict['accuracy']\n",
    "val_acc_values = history_dict['val_accuracy']\n",
    "\n",
    "epochs = range(1, len(loss_values) + 1)\n",
    "\n",
    "plt.plot(epochs, acc_values, 'ro', label=\"Training Accuracy\")\n",
    "plt.plot(epochs, val_acc_values, 'r', label=\"Validation Accuracy\")\n",
    "\n",
    "plt.title('Training and Validation Accuraccy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that _minimum validation loss_ and _maximum validation Accuracy_ is achived at around 9-10 epochs. After that we observe 2 trends:\n",
    "* increase in validation loss and decrese in training loss\n",
    "* decrease in validation accuracy and increase in training accuracy\n",
    "\n",
    "This implies that the model is getting better at classifying the sentiment of the training data, but making consistently worse predictions when it encounters new, previously unseed data. This is the hallmark of _Overfitting_. After the 10th epoch the model begins to fit too closely to the trainning data.\n",
    "\n",
    "To address overfitting, we will reduce the number of epochs to 9. These results may vary depending on your machine and due to the very nature of the random assignment of weights that may vary from model to mode.\n",
    "\n",
    "In our case we will stop training after 9 epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retraining the model from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/9\n",
      "16/16 [==============================] - 1s 50ms/step - loss: 2.6164 - accuracy: 0.5335 - val_loss: 1.7389 - val_accuracy: 0.6350\n",
      "Epoch 2/9\n",
      "16/16 [==============================] - 1s 46ms/step - loss: 1.4390 - accuracy: 0.7020 - val_loss: 1.3235 - val_accuracy: 0.7120\n",
      "Epoch 3/9\n",
      "16/16 [==============================] - 1s 44ms/step - loss: 1.0720 - accuracy: 0.7686 - val_loss: 1.1556 - val_accuracy: 0.7500\n",
      "Epoch 4/9\n",
      "16/16 [==============================] - 1s 40ms/step - loss: 0.8502 - accuracy: 0.8192 - val_loss: 1.0815 - val_accuracy: 0.7570\n",
      "Epoch 5/9\n",
      "16/16 [==============================] - 1s 41ms/step - loss: 0.6842 - accuracy: 0.8505 - val_loss: 0.9838 - val_accuracy: 0.7920\n",
      "Epoch 6/9\n",
      "16/16 [==============================] - 1s 40ms/step - loss: 0.5511 - accuracy: 0.8796 - val_loss: 0.9427 - val_accuracy: 0.8040\n",
      "Epoch 7/9\n",
      "16/16 [==============================] - 1s 40ms/step - loss: 0.4401 - accuracy: 0.9069 - val_loss: 0.9202 - val_accuracy: 0.8090\n",
      "Epoch 8/9\n",
      "16/16 [==============================] - 1s 36ms/step - loss: 0.3571 - accuracy: 0.9253 - val_loss: 0.9246 - val_accuracy: 0.8060\n",
      "Epoch 9/9\n",
      "16/16 [==============================] - 1s 35ms/step - loss: 0.2973 - accuracy: 0.9364 - val_loss: 0.8962 - val_accuracy: 0.8250\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb0fc58b250>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Model definition\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000, )))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "# Model Compilation\n",
    "model.compile(optimizer = 'rmsprop',\n",
    "             loss = 'categorical_crossentropy',\n",
    "             metrics = ['accuracy'])\n",
    "\n",
    "# Training the model\n",
    "model.fit(partial_X_train,\n",
    "         partial_Y_train,\n",
    "         epochs=9,\n",
    "         batch_size=512,\n",
    "         validation_data=(X_val, Y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting results and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71/71 [==============================] - 1s 13ms/step - loss: 0.9830 - accuracy: 0.7845\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, Y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9829719066619873, 0.7845057845115662]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our approaches reaches an efficieny of ~80%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this were a balanced dataset, a random attribution of labels would, using simple probability, result in 50% accuracy. But since, this dataset is not balanced the accuracy of a random classifier might be on the lesser side.\n",
    "\n",
    "Let's determine this random baseline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18432769367764915"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "\n",
    "#  Number of accurate predictions, matches, or hits\n",
    "hits = np.array(test_labels) == np.array(test_labels_copy)\n",
    "\n",
    "random_baseline = float(np.sum(hits)) / len(test_labels)\n",
    "random_baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering that the random baseline is about 19%, our model with an accuracy of ~80% performs quite well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model with information bottleneck"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "63/63 [==============================] - 1s 22ms/step - loss: 2.6864 - accuracy: 0.4622 - val_loss: 2.0200 - val_accuracy: 0.5700\n",
      "Epoch 2/20\n",
      "63/63 [==============================] - 1s 16ms/step - loss: 1.7919 - accuracy: 0.5821 - val_loss: 1.6724 - val_accuracy: 0.5800\n",
      "Epoch 3/20\n",
      "63/63 [==============================] - 1s 18ms/step - loss: 1.5013 - accuracy: 0.6012 - val_loss: 1.5276 - val_accuracy: 0.6070\n",
      "Epoch 4/20\n",
      "63/63 [==============================] - 1s 17ms/step - loss: 1.3307 - accuracy: 0.6426 - val_loss: 1.4420 - val_accuracy: 0.6450\n",
      "Epoch 5/20\n",
      "63/63 [==============================] - 1s 23ms/step - loss: 1.1706 - accuracy: 0.6867 - val_loss: 1.3445 - val_accuracy: 0.6620\n",
      "Epoch 6/20\n",
      "63/63 [==============================] - 1s 18ms/step - loss: 1.0484 - accuracy: 0.7045 - val_loss: 1.3438 - val_accuracy: 0.6590\n",
      "Epoch 7/20\n",
      "63/63 [==============================] - 1s 23ms/step - loss: 0.9539 - accuracy: 0.7293 - val_loss: 1.3161 - val_accuracy: 0.6830\n",
      "Epoch 8/20\n",
      "63/63 [==============================] - 1s 18ms/step - loss: 0.8736 - accuracy: 0.7529 - val_loss: 1.3249 - val_accuracy: 0.6950\n",
      "Epoch 9/20\n",
      "63/63 [==============================] - 1s 17ms/step - loss: 0.8032 - accuracy: 0.7775 - val_loss: 1.3231 - val_accuracy: 0.7050\n",
      "Epoch 10/20\n",
      "63/63 [==============================] - 1s 16ms/step - loss: 0.7414 - accuracy: 0.8008 - val_loss: 1.3608 - val_accuracy: 0.7100\n",
      "Epoch 11/20\n",
      "63/63 [==============================] - 1s 17ms/step - loss: 0.6835 - accuracy: 0.8138 - val_loss: 1.3755 - val_accuracy: 0.7180\n",
      "Epoch 12/20\n",
      "63/63 [==============================] - 1s 19ms/step - loss: 0.6422 - accuracy: 0.8218 - val_loss: 1.4331 - val_accuracy: 0.7150\n",
      "Epoch 13/20\n",
      "63/63 [==============================] - 1s 18ms/step - loss: 0.6028 - accuracy: 0.8316 - val_loss: 1.4485 - val_accuracy: 0.7170\n",
      "Epoch 14/20\n",
      "63/63 [==============================] - 1s 19ms/step - loss: 0.5693 - accuracy: 0.8426 - val_loss: 1.4870 - val_accuracy: 0.7180\n",
      "Epoch 15/20\n",
      "63/63 [==============================] - 1s 19ms/step - loss: 0.5359 - accuracy: 0.8555 - val_loss: 1.5613 - val_accuracy: 0.7140\n",
      "Epoch 16/20\n",
      "63/63 [==============================] - 1s 21ms/step - loss: 0.5124 - accuracy: 0.8628 - val_loss: 1.6172 - val_accuracy: 0.7120\n",
      "Epoch 17/20\n",
      "63/63 [==============================] - 1s 21ms/step - loss: 0.4848 - accuracy: 0.8708 - val_loss: 1.6723 - val_accuracy: 0.7070\n",
      "Epoch 18/20\n",
      "63/63 [==============================] - 2s 26ms/step - loss: 0.4673 - accuracy: 0.8760 - val_loss: 1.7235 - val_accuracy: 0.7070\n",
      "Epoch 19/20\n",
      "63/63 [==============================] - 1s 18ms/step - loss: 0.4470 - accuracy: 0.8801 - val_loss: 1.7827 - val_accuracy: 0.7090\n",
      "Epoch 20/20\n",
      "63/63 [==============================] - 1s 20ms/step - loss: 0.4294 - accuracy: 0.8865 - val_loss: 1.8424 - val_accuracy: 0.7060\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb0a8389a50>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(4, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(partial_X_train,\n",
    "          partial_Y_train,\n",
    "          epochs=20,\n",
    "          batch_size=128,\n",
    "          validation_data=(X_val, Y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71/71 [==============================] - 1s 12ms/step - loss: 2.2367 - accuracy: 0.6834\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the presence of a bottleneck, the testing accuracy drops 10%"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}