RNN_LSTM_hands-on_2021.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9h-pNS_KaIbZ"
   },
   "source": [
    "# Lab 3: Deep Sequence Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_qr6VAorz7am"
   },
   "source": [
    "The third lab session is about data that have a sequential structure that must be taken into account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2919,
     "status": "ok",
     "timestamp": 1589487094276,
     "user": {
      "displayName": "Nicoletta Noceti",
      "photoUrl": "",
      "userId": "00427744574778931345"
     },
     "user_tz": -120
    },
    "id": "4TNAZGt4aIbe",
    "outputId": "480e8957-6ac5-4eec-8fe9-7e0c29010da6"
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import tensorflow as tf\n",
    "import os, json, re\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KnmA6CbEaIbv"
   },
   "source": [
    "# 3.1 Deal with sequential data\n",
    "<img src=\"https://drive.google.com/uc?id=1k6HwtWlMTkVJFuyBpeBGF2sApqxR-KRc\" width=\"600px\" align=\"right\"><br>\n",
    "In this lab we see Deep Learning models that can process sequential data (text, timeseries,..).<br>\n",
    "These models don’t take as input raw text: they only work with numeric tensors; **vectorizing** text is the process of transforming text into numeric tensors.<br><br><br>\n",
    "The different units into which you can break down text (words, characters) are called tokens; then if you apply a tokenization scheme, you associate numeric vectors with the generated tokens.<br>\n",
    "These vectors, packed into sequence tensors, are fed into Deep Neural Network.<br>\n",
    "There are multiple ways to associate a vector with a token: we will see One-Hot Encoding and Token Embedding.<br>\n",
    "In this section we are going to deal with:\n",
    "* 3.1.1 One-Hot Encoding\n",
    "* 3.1.2 Word embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qhRTCGvGaIbz"
   },
   "source": [
    "## 3.1.1 One-Hot Encoding\n",
    "One-Hot Encoding consists of associating a unique integer index with every word and then turning this integer index $i$ into a binary vector of size $N$ (the size of the vocabulary); the vector is all zeros except for the $i$-th entry, which is 1.\n",
    "<img src=\"https://drive.google.com/uc?id=1OzK9t_WXQsaDuZoOTQSksLuNMubXm0gc\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u8hfjPNJaIb1"
   },
   "source": [
    "#### Try to perform One-Hot Encoding using Tokenizer\n",
    "Keras provides the Tokenizer class for preparing text documents for DL.<br>\n",
    "The Tokenizer must be constructed and then fit on either raw text documents or integer encoded text documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1982,
     "status": "ok",
     "timestamp": 1589487126170,
     "user": {
      "displayName": "Nicoletta Noceti",
      "photoUrl": "",
      "userId": "00427744574778931345"
     },
     "user_tz": -120
    },
    "id": "0EPyke25aIb3",
    "outputId": "e7ff7621-5ef6-4c1a-c6ce-192ce93dc1a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 1. 1. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 1. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 1. 1. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "# define 4 documents\n",
    "docs = ['Well done!','Good work','Great effort','nice work']\n",
    "\n",
    "# create the tokenizer\n",
    "tokenizer = Tokenizer()\n",
    "\n",
    "# fit the tokenizer on the documents\n",
    "tokenizer.fit_on_texts(docs)\n",
    "\n",
    "\n",
    "encoded_docs = tokenizer.texts_to_matrix(docs, mode='count')\n",
    "print(encoded_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eDycrywyaIb-"
   },
   "source": [
    "Some problems related to this kind of encoding are sparsity of the solution and the high dimensionality of the vector encoding of the tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YHeXmpPfaIcB"
   },
   "source": [
    "## 3.1.2 Word embedding\n",
    "<img src=\"https://drive.google.com/uc?id=1YRcQ1_5n8Qay0GFoSLVrNcEKWeik5G3S\" width=\"400px\" align=\"right\"><br>\n",
    "The vector obtained from word embedding is dense and has lower dimensionality w.r.t One-Hot Encoding vector; the dimensionality of embedding space vector is an hyperparameter.<br>\n",
    "There are two ways to obtain word embeddings:<br>\n",
    "* May be learned jointly with the network\n",
    "* May use pre-trained word vectors (Word2Vec, GloVe,..)\n",
    "\n",
    "\n",
    "Word embeddings maps human language into a geometric space; in a reasonable embedding space synonyms are embedded into similar word vectors and the geometric distance between any two word vectors reflects the semantic distance between the associated words (words meaning different things are embedded at points far away from each other, whereas related words are closer).<br>\n",
    "How good is a word-embedding space depends on the specific task.<br>\n",
    "It is reasonable to learn a new embedding space with every new task: with backpropagation and Keras it reduces to learn the weights of the Embedding layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GFHFL8wSaIcF"
   },
   "source": [
    "### Learning Word Embeddings with the embedding layer\n",
    "#### Load imdb dataset\n",
    "This dataset contains movies reviews from IMDB, labeled by sentiment(positive/negative); reviews have been preprocessed, and each review is encoded as a sequence of word indexes(integers).<br>\n",
    "https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7763,
     "status": "ok",
     "timestamp": 1589487141822,
     "user": {
      "displayName": "Nicoletta Noceti",
      "photoUrl": "",
      "userId": "00427744574778931345"
     },
     "user_tz": -120
    },
    "id": "9KD1r3Dkz7bh",
    "outputId": "aefaa77f-a127-4e4a-f014-63aadb8fb730"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\vitop\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\datasets\\imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "C:\\Users\\vitop\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\datasets\\imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    }
   ],
   "source": [
    "max_features = 10000\n",
    "maxlen = 50\n",
    "\n",
    "imdb = tf.keras.datasets.imdb\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = tf.keras.preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a5muY3QYaIcK"
   },
   "source": [
    "#### Show the size of vocabulary and the most frequent words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1454,
     "status": "ok",
     "timestamp": 1589487223292,
     "user": {
      "displayName": "Nicoletta Noceti",
      "photoUrl": "",
      "userId": "00427744574778931345"
     },
     "user_tz": -120
    },
    "id": "JXB3wghhaIcL",
    "outputId": "5a3762a9-712f-4468-8764-933dd036d484"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size :  88584\n",
      "50 most common words: \n",
      "\n",
      "[('the', 1), ('and', 2), ('a', 3), ('of', 4), ('to', 5), ('is', 6), ('br', 7), ('in', 8), ('it', 9), ('i', 10), ('this', 11), ('that', 12), ('was', 13), ('as', 14), ('for', 15), ('with', 16), ('movie', 17), ('but', 18), ('film', 19), ('on', 20), ('not', 21), ('you', 22), ('are', 23), ('his', 24), ('have', 25), ('he', 26), ('be', 27), ('one', 28), ('all', 29), ('at', 30), ('by', 31), ('an', 32), ('they', 33), ('who', 34), ('so', 35), ('from', 36), ('like', 37), ('her', 38), ('or', 39), ('just', 40), ('about', 41), (\"it's\", 42), ('out', 43), ('has', 44), ('if', 45), ('some', 46), ('there', 47), ('what', 48), ('good', 49), ('more', 50)]\n"
     ]
    }
   ],
   "source": [
    "word_to_index = imdb.get_word_index()\n",
    "\n",
    "vocab_size = len(word_to_index)\n",
    "print('Vocab size : ', vocab_size)\n",
    "\n",
    "\n",
    "words_freq_list =words_freq_list = []\n",
    "[]\n",
    "for (k,v) in imdb.get_word_index().items():\n",
    "    words_freq_list.append((k,v))\n",
    "\n",
    "sorted_list = sorted(words_freq_list, key=lambda x: x[1])\n",
    "\n",
    "print(\"50 most common words: \\n\")\n",
    "print(sorted_list[0:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "27uI4g4saIcQ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "897"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_to_index['otherwise']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jXjwll2WaIda"
   },
   "source": [
    "# 3.2 Recurrent Neural Network\n",
    "Here https://colah.github.io/posts/2015-08-Understanding-LSTMs/ you can find a clear explanation about RNNs and LSTMs; the following is a summary of the main concepts.\n",
    "\n",
    "\n",
    "A major characteristic of some neural networks, as ConvNet, is that they have no memory: each input is processed independently, with no state kept in between inputs. Biological intelligence processes information incrementally while maintaining an internal model of what it’s processing, built from past information and constantly updated as new information comes in.<br>\n",
    "A recurrent neural network (RNN) adopts the same principle but in an extremely simplified version: it processes sequences by iterating through the sequence elements and maintaining a state containing information relative to what it has seen so far.\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=1RaDXXygc0HJv6YyIAjU4_Nbw1bXzAhAJ\" width=\"650px\"><br>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Each input $x_{i=t-1, t, t+1, ..}$ is combined with the internal state and then is applied an activation function (e.g. $tanh$); then the output is computed $h_{i=t-1, t, t+1, ..}$ and the internal state is updated.<br>\n",
    "In many cases, you just need the last output ($h_{i=last t}$ at the end of the loop), because it already contains information\n",
    "about the entire sequence.\n",
    "<img src=\"https://drive.google.com/uc?id=1RtulDLKQnzZTSbBsD2n7TIlRVEaESB8o\" width=\"550px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vL10YVWTaIdi"
   },
   "source": [
    "#### Create the model\n",
    "In the following sections we will develop different models. Be careful to the fact that we are dealing with a binary classification problem!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3PCby93EaIdi"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, 16))\n",
    "\n",
    "# Complete the model, it should be made by at least:\n",
    "# 1 SimpleRNN layer\n",
    "# 1 Dense layer\n",
    "model.add(tf.keras.layers.SimpleRNN(32))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4JTBSi5TaIdp"
   },
   "source": [
    "#### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DamvGVOpaIdq"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "157/157 [==============================] - 6s 39ms/step - loss: 0.6046 - acc: 0.6564 - val_loss: 0.4899 - val_acc: 0.7680\n",
      "Epoch 2/10\n",
      "157/157 [==============================] - 6s 39ms/step - loss: 0.4073 - acc: 0.8212 - val_loss: 0.4409 - val_acc: 0.7980\n",
      "Epoch 3/10\n",
      "157/157 [==============================] - 6s 40ms/step - loss: 0.3222 - acc: 0.8669 - val_loss: 0.4495 - val_acc: 0.7804\n",
      "Epoch 4/10\n",
      "157/157 [==============================] - 6s 40ms/step - loss: 0.2624 - acc: 0.8960 - val_loss: 0.4968 - val_acc: 0.7950\n",
      "Epoch 5/10\n",
      "157/157 [==============================] - 7s 44ms/step - loss: 0.2079 - acc: 0.9204 - val_loss: 0.5084 - val_acc: 0.7784\n",
      "Epoch 6/10\n",
      "157/157 [==============================] - 7s 42ms/step - loss: 0.1539 - acc: 0.9441 - val_loss: 0.5642 - val_acc: 0.7910\n",
      "Epoch 7/10\n",
      "157/157 [==============================] - 7s 43ms/step - loss: 0.1031 - acc: 0.9664 - val_loss: 0.6794 - val_acc: 0.7330\n",
      "Epoch 8/10\n",
      "157/157 [==============================] - 6s 41ms/step - loss: 0.0664 - acc: 0.9801 - val_loss: 0.7320 - val_acc: 0.7776\n",
      "Epoch 9/10\n",
      "157/157 [==============================] - 6s 41ms/step - loss: 0.0378 - acc: 0.9900 - val_loss: 0.8180 - val_acc: 0.7264\n",
      "Epoch 10/10\n",
      "157/157 [==============================] - 6s 40ms/step - loss: 0.0222 - acc: 0.9948 - val_loss: 0.8930 - val_acc: 0.7474\n"
     ]
    }
   ],
   "source": [
    "# Train your model here\n",
    "history = model.fit(x_train, y_train,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H5vtqdoL8gQl"
   },
   "outputs": [],
   "source": [
    "def plot_history(history):\n",
    "    # Plot training & validation accuracy values\n",
    "    plt.plot(history.history['acc'])\n",
    "    plt.plot(history.history['val_acc'])\n",
    "    plt.title('Model accuracy')\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.legend(['Train', 'Val'], loc='upper left')\n",
    "    plt.show()\n",
    "\n",
    "    # Plot training & validation loss values\n",
    "    plt.plot(history.history['loss'])\n",
    "    plt.plot(history.history['val_loss'])\n",
    "    plt.title('Model loss')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.legend(['Train', 'Val'], loc='upper left')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pt-M_A7OaIds"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA6aUlEQVR4nO3dd3yUVdr/8c+VhCQkQEihB5IAoUoRIqjYAQUbdsV17bq6+qhbH/W3+2x/1t1199Fd27IuKopiRbBhWxEbQpBQpRPSKIEACSmkXb8/zgSGOIEBMplk5nq/Xnll5i4z10S5v3Ofc9/niKpijDHGNBYR7AKMMca0ThYQxhhjfLKAMMYY45MFhDHGGJ8sIIwxxvhkAWGMMcYnCwgT9kQkXURURKL82PZGEfm8JeoyJtgsIEybIiK5IlItIimNlud4DvLpQSrNmJBjAWHaos3A1IYnIjIMaB+8cloHf86AjDkaFhCmLXoeuN7r+Q3ADO8NRCRBRGaISLGIbBGRX4hIhGddpIg8LCI7RWQTcIGPff8tIltFpFBEfi8ikf4UJiKvisg2EdkrIgtEZKjXuvYi8ldPPXtF5HMRae9Zd5qIfCkie0QkX0Ru9CyfLyK3er3GIU1cnrOmu0RkPbDes+xRz2uUisgSETnda/tIEXlQRDaKSJlnfW8ReVxE/tros7wlIvf587lNaLKAMG3RQqCTiAz2HLivBl5otM0/gASgL3AmLlBu8qy7DbgQOBHIAq5otO9zQC3Q37PNucCt+Oc9IBPoCnwDzPRa9zAwGjgVSAJ+DtSLSB/Pfv8AugAjgRw/3w/gEmAsMMTzfLHnNZKAF4FXRSTWs+7HuLOv84FOwM1ABe4zT/UK0RRgPPDSUdRhQo2q2o/9tJkfIBeYAPwC+CMwCfgQiAIUSAcigf3AEK/9fgDM9zz+D3CH17pzPftGAd08+7b3Wj8V+MTz+Ebgcz9r7ex53QTcl7FKYISP7R4AZjfxGvOBW72eH/L+ntc/5wh17G54X2AtMKWJ7b4FJnoe3w28G+z/3vYT3B9rszRt1fPAAiCDRs1LQAoQDWzxWrYF6OV53BPIb7SuQRrQDtgqIg3LIhpt75PnbOYPwJW4M4F6r3pigFhgo49dezex3F+H1CYiP8Gd8fTEBUgnTw1Heq/ngOtwgXsd8Ohx1GRCgDUxmTZJVbfgOqvPB95otHonUIM72DfoAxR6Hm/FHSi91zXIx51BpKhqZ89PJ1UdypFdC0zBneEk4M5mAMRTUxXQz8d++U0sBygH4ryed/exzYEhmT39Df8NXAUkqmpnYK+nhiO91wvAFBEZAQwG3mxiOxMmLCBMW3YLrnml3HuhqtYBrwB/EJGOIpKGa3tv6Kd4BbhHRFJFJBG432vfrcAHwF9FpJOIRIhIPxE50496OuLCZRfuoP6/Xq9bD0wH/iYiPT2dxaeISAyun2KCiFwlIlEikiwiIz275gCXiUiciPT3fOYj1VALFANRIvI/uDOIBk8DvxORTHGGi0iyp8YCXP/F88Drqlrpx2c2IcwCwrRZqrpRVbObWP1fuG/fm4DPcZ210z3r/gW8DyzDdSQ3PgO5HtdEtRrXfv8a0MOPkmbgmqsKPfsubLT+p8AK3EG4BPgTEKGqebgzoZ94lucAIzz7/B9QDWzHNQHN5PDex3V4r/PUUsWhTVB/wwXkB0Ap8G8OvUT4OWAYLiRMmBNVmzDIGOOIyBm4M610z1mPCWN2BmGMAUBE2gH3Ak9bOBiwgDDGACIyGNiDa0p7JKjFmFbDmpiMMcb4ZGcQxhhjfAqpG+VSUlI0PT092GUYY0ybsWTJkp2q2sXXupAKiPT0dLKzm7rq0RhjTGMisqWpddbEZIwxxicLCGOMMT4FLCBEZLqI7BCRlU2sFxH5u4hsEJHlIjLKa90kEVnrWXe/r/2NMcYEViD7IJ4FHuO7I202mIwbNz8TN5b9k8BYz4iYjwMTgQJgsYjMVdXVx1JETU0NBQUFVFVVHcvubUpsbCypqam0a9cu2KUYY0JAwAJCVRccYX7gKcAMdTdiLBSRziLSAzcC5gZV3QQgIrM82x5TQBQUFNCxY0fS09PxGr455Kgqu3btoqCggIyMjGCXY4wJAcHsg+jFoYOIFXiWNbXcJxG5XUSyRSS7uLj4O+urqqpITk4O6XAAEBGSk5PD4kzJGNMyghkQvo7YepjlPqnqNFXNUtWsLl18Xsob8uHQIFw+pzGmZQTzPogCDp20JRUowg2z7Gu5McaELVWlvLqOkn3V7CrfT0l5NbvKqykpr0YV7jyrqXmgjl0wA2IucLenj2EssFdVt4pIMZApIhm4cfWvwc3U1ebs2rWL8ePHA7Bt2zYiIyNpOMtZtGgR0dHRTe6bnZ3NjBkz+Pvf/94itRpjWpaqUlpZ+52DfUl5Nbv2VVNSvv/QZeXVVNf6HmS3a8eYthUQIvIScBaQIiIFwK9wc/2iqk8B7+ImSdkAVAA3edbVisjduIlPIoHpqroqUHUGUnJyMjk5OQD8+te/pkOHDvz0pz89sL62tpaoKN//CbKyssjKymqJMo0xzaC+XtlTWeMO7PsOHtSbOtjvLq+mtt5363lcdCRJ8dEkx0fTtWMMg7p3IrlDNEnx0QeWu98xJHWIJj46MiCfKZBXMU09wnoF7mpi3bu4AAk5N954I0lJSSxdupRRo0Zx9dVXc99991FZWUn79u155plnGDhwIPPnz+fhhx/m7bff5te//jV5eXls2rSJvLw87rvvPu65555gfxRjws7OfftZt62MtdvLWLe9jM07yw+Ewe6Kapo43tMxNurAQT01MY4RqZ1J6nDwQO99sE+Ojya2XWAO+EcrpMZiOpLfvLWK1UWlzfqaQ3p24lcX+TOf/UHr1q3jo48+IjIyktLSUhYsWEBUVBQfffQRDz74IK+//vp39lmzZg2ffPIJZWVlDBw4kDvvvNPudzAmQPZW1rB+uycIDgTCPkrKqw9s0zmuHf26dKBvl3iy0pNIjo8+8C0/OT7G/e4QTWJcNNFRbXPQirAKiNbiyiuvJDLSfUPYu3cvN9xwA+vXr0dEqKmp8bnPBRdcQExMDDExMXTt2pXt27eTmprakmUbE3IqqmtZv30f6zxnBGu372PdtjK2lR68XDw+OpIB3TsycXA3BnTvyMBuHRnQvQNdOsSE/JWDYRUQR/tNP1Di4+MPPP7lL3/J2WefzezZs8nNzeWss87yuU9MTMyBx5GRkdTW1ga6TGNCxv7aOjYVlx8Mgm0uFPJ3V9AwZ1p0VASZXTtwar9kBnTvyIBuHRjQrSO9OrcP+SBoSlgFRGu0d+9eevVy9wE+++yzwS3GmDautq6eLSUVB5qF1m/fx1pPX0Gdp4MgKkLISIlnWGoCV4xOZUA3FwZpyfFERoRnEDTFAiLIfv7zn3PDDTfwt7/9jXPOOSfY5RjTJtTXK4V7Kj3NQq6fYN32fWwo3nfgUlARSEuKI7NbRyYN7X6geSgjJb7N9gm0tJCakzorK0sbTxj07bffMnjw4CBV1PLC7fOa8FBZXUdO/h4W55awOLeEpXl72Lf/YDNrz4TYAwGQ2c397t+1A+0DdPlnKBGRJarq85p6O4MwxrQ6eyqqyc7dzeLcEhbllrCycC81dYoIDOzWkUtP7MWQnp0Y0K0jmd060CnWrugLBAsIY0zQFe2pdGGw2Z0hrNu+D4DoyAiGpyZw6+l9OSk9kdF9kkiIszBoKRYQxpgWpaps2LGPRbklLN5cwuLc3RTuqQSgQ0wUo9MSuXhET05KT2JE786t5qaxcGQBYYwJqJq6elYW7iU7dzeLckvIzi1hd4W73yelQwxjMhK59fQMTkpPYnCPTnYlUStiAWGMaVYV1bUszdtzoLload4eKmvqAEhPjmPC4G6clJHEmPQk0pLjwvYeg7bAAsIYc1xKyqvd1UWeQFhZVEpdvRIhMLhHJ64+qTdjMpLISkuka6fYYJdrjoIFRICdddZZPPDAA5x33nkHlj3yyCOsW7eOJ554wuf2Dz/8sI3kalolVaVgd+WBy00XbS5hY3E54O5EHtm7M3ee2Y+s9ERGpyXS0a4uatMsIAJs6tSpzJo165CAmDVrFn/5y1+CWJUxRye/pIJXs/N5Y2khBbtdh3Kn2Ciy0pO4fHQqY9KTGJaaQEyUdSiHEguIALviiiv4xS9+wf79+4mJiSE3N5eioiJefPFFfvSjH1FZWckVV1zBb37zm2CXaswhqmrqmLdyG69k5/Plxl2IwOmZXbj9jL6clJ7EwG4dibAO5ZAWXgHx3v2wbUXzvmb3YTD5oSZXJycnM2bMGObNm8eUKVOYNWsWV199NQ888ABJSUnU1dUxfvx4li9fzvDhw5u3NmOOkqqysrCUl7PzmJNTRFlVLb2T2vOTiQO4fHQqPTu3D3aJpgWFV0AESUMzU0NATJ8+nVdeeYVp06ZRW1vL1q1bWb16tQWECZrd5dXMXlrIK9n5rNlWRkxUBOcP68GVWamcnJFsZwphKrwC4jDf9APpkksu4cc//jHffPMNlZWVJCYm8vDDD7N48WISExO58cYbqaqqOvILGdOM6uqVzzfs5JXF+Xy4ejvVdfUMT03g95ecwEUjepLQ3jqYw114BUSQdOjQgbPOOoubb76ZqVOnUlpaSnx8PAkJCWzfvp333nuvyXkgjGluebsqeHVJPq8tKWDr3ioS49px3clpXJmVyuAenYJdnmlFLCBayNSpU7nsssuYNWsWgwYN4sQTT2To0KH07duXcePGBbs8E+IaOpxfXpzPV5tch/MZmV345YVDGD+4q119ZHwKaECIyCTgUSASeFpVH2q0PhGYDvQDqoCbVXWlZ10uUAbUAbVNDUfbVlx66aV4D63e1ORA8+fPb5mCTMhTVVYU7uXlxfnMXeY6nPskxfHTcwdw2SjrcDZHFrCAEJFI4HFgIlAALBaRuaq62muzB4EcVb1URAZ5th/vtf5sVd0ZqBqNCUUl5dW86aPD+aqs3ozNSLIOZ+O3QJ5BjAE2qOomABGZBUwBvANiCPBHAFVdIyLpItJNVbcHsC5jQk5dvfLZ+mJeyXYdzjV1yghPh/PFI3vafAnmmAQyIHoB+V7PC4CxjbZZBlwGfC4iY4A0IBXYDijwgYgo8E9VnebrTUTkduB2gD59+vgsRFXDYkCwUJod0PjHV4fz909O56qTUhnU3TqczfEJZED4OiI3PoI9BDwqIjnACmAp0DCP4DhVLRKRrsCHIrJGVRd85wVdcEwDN+Vo4/WxsbHs2rWL5OTkkA4JVWXXrl3ExtpgaKGusrqOeau28vLifBZuKiFC4IwB1uFsml8gA6IA6O31PBUo8t5AVUuBmwDEHb03e35Q1SLP7x0iMhvXZPWdgDiS1NRUCgoKKC4uPpbP0KbExsaSmpoa7DJMAKgqywv28kp2PnNziijbf7DD+fLRqfRIsA5n0/wCGRCLgUwRyQAKgWuAa703EJHOQIWqVgO3AgtUtVRE4oEIVS3zPD4X+O2xFNGuXTsyMjKO42MYEzz79tcye2khMxduYc22MmLbRXD+CT240jqcTQsIWECoaq2I3A28j7vMdbqqrhKROzzrnwIGAzNEpA7XeX2LZ/duwGxPk1AU8KKqzgtUrca0NquLSnnh6y3MWVpIeXUdQ3t2sg5n0+IklDo2s7KyNDs7O9hlGHNMqmrqeHfFVl5YuIVv8vYQExXBhcN7ct3JfRjZu3NI96GZ4BGRJU3dZ2Z3UhsTZLk7y3lxUR6vZuezu6KGvinx/OKCwVwxOpXOcdHBLs+EMQsIY4Kgtq6ej77dwcyvt/DZ+p1ERgjnDunGdSencWq/0L7izrQdFhDGtKBte6uYtTiPWYvy2VZaRY+EWH48cQBXn9SbbjZfs2llLCCMCbD6euWLjTuZuTCPD7/dTl29csaALvx2ylDOGdSVqMiIYJdojE8WEMYEyO7yal5bUsCLi/LYvLOcxLh23HpaBteO7UNacnywyzPmiCwgjGlGqsrS/D28sHALby/fSnVtPVlpidw7PpNJJ3Qntp3d5WzaDgsIY5pB+f5a3swpZObCPFZvLSU+OpKrslL53tg0m4THtFkWEMYch7Xbynhh4RZmLy1k3/5aBnXvyO8vOYFLTuxFhxj752XaNvs/2JijtL/Wzc72wsItLM7dTXRUBBcO68H3Tk5jVB+7oc2EDgsIY/yUt6uCmYu28Gp2ASXl1aQlx/Hg+YO4YnRvkuLthjYTeiwgjDmMunrlP2t28MLCLSxYX0yECBMGd+W6k9MY1y/FBsszIc0CwhgfSsqrmbU4j5kL8yjcU0m3TjHcc04mU8f0oXuC3dBmwoMFhDFeVhTs5bmvcpm7rIjq2npO7ZfMLy8czPjB3WhnN7SZMGMBYcLe/to63luxjee+ymVp3h7ioiO5Oqs315+SRma3jsEuz5igsYAwYWvr3kpe/DqPlxblsXNfNX1T4vn1RUO4bHSqzblgDBYQJsyoKos2lzDjqy3MW7WNelXGD+rK9aekc1p/63Q2xpsFhAkLFdW1vLm0iBlf5bJmWxkJ7d24SNednEbvpLhgl2dMq2QBYUJa7s5ynl+4hVey8ymrqmVwj0786fJhXDyiF+2jbVwkYw7HAsKEnPp65dP1xcz4Mpf564qJFGHysB7ccEoao9MS7U5nY/xkAWFCxt7KGl7Nzuf5hVvYsquCLh3dvQvfG9uHrjYZjzFHLaABISKTgEeBSOBpVX2o0fpEYDrQD6gCblbVlf7sa0yDNdtKee7LLby5tJDKmjqy0hL5ybkDmTS0O9FRdu+CMccqYAEhIpHA48BEoABYLCJzVXW112YPAjmqeqmIDPJsP97PfU0Yq6mr58PV23nuy1y+3lxCTFQEl4zsxfdPSeOEXgnBLs+YkBDIM4gxwAZV3QQgIrOAKYD3QX4I8EcAVV0jIuki0g3o68e+JgwVl+1n1qI8Zn6dx7bSKlIT2/PA5EFcldWbRBswz5hmFciA6AXkez0vAMY22mYZcBnwuYiMAdKAVD/3NWFCVcnJ38NzX+byzoqt1NQpp2em8PtLTuDsQV2JtHsXjAmIQAaEr3+12uj5Q8CjIpIDrACWArV+7uveROR24HaAPn36HGutphWqqqnj7eVbmfFVLssL9tIhJorvjU3j+6ek0a9Lh2CXZ0zIC2RAFAC9vZ6nAkXeG6hqKXATgLhrDzd7fuKOtK/Xa0wDpgFkZWX5DBHTtpRW1fDM57k8++VmdlfU0L9rB343ZSiXjkq1WdqMaUGB/Ne2GMgUkQygELgGuNZ7AxHpDFSoajVwK7BAVUtF5Ij7mtBTVlXDM1/k8vRnmyitqmXC4K7cPC6DU/ol270LxgRBwAJCVWtF5G7gfdylqtNVdZWI3OFZ/xQwGJghInW4DuhbDrdvoGo1wVVWVcOzX+Ty9Oeb2VtZw4TB3bhvQqZdjWRMkIlq6LTKZGVlaXZ2drDLMH7at7+W577M5V+fbWJPRQ0TBnfl3vEDGJZqwWBMSxGRJaqa5WudNeiaFtc4GM4Z1JX7JmQyPLVzsEszxnixgDAtpnx/LTO+2sK0BRvZXVHD2QO7cO+EAYzs3TnYpRljfLCAMAFXUd0QDJsoKa/mzAFduG9CJif2SQx2acaYw7CAMAFTUV3LCwu38M9PN7GrvJozPMEwyoLBmDbBAsI0u8rqOhcMCzayc181p2emcN+EAYxOs2Awpi2xgDDNprK6jplfb+GpTzexc99+Tuufwn0TMslKTwp2acaYY2ABYY5bVU0dM7/O46lPN1Jctp9x/ZN5csIoTrJgMKZNs4Awx6yqpo4Xv87jSU8wnNI3mcemnsjYvsnBLs0Y0wwsIMxRq6qpY9aiPJ6Yv5EdZfsZm5HEP6aeyMkWDMaEFAsI47eqmjpeXpzPE/M3sL10P2Myknj0mhM5pZ8FgzGhyALCHNH+2jpeWZzP459sZFtpFWPSk/i/q0dySl8bRM+YUGYBYZq0v7aOV7MLeOKTDRTtrSIrLZG/XjWCU210VWPCggWE+Y7q2npeXZLP4/9xwTA6LZE/XzGCcf0tGIwJJxYQ5hBfbNjJz19bTuGeSk7s05mHLh/O6ZkpFgzGhCELCAO4eZ//uWATf563hr5dOvDczWM4w4LBmLBmAWHYt7+Wn726jPdWbuOCYT348xXDibepPY0Je3YUCHMbi/fxg+eXsKl4Hw+eP4jbTu9rZw3GGMACIqy9v2obP3llGdFREbxwy1hO7Z8S7JKMMa3IEQNCRC4E3lXV+haox7SAunrlbx+u5fFPNjI8NYEnrxtNr87tg12WMaaVifBjm2uA9SLyZxEZHOiCTGDtLq/mpmcX8/gnG7k6qzev/OAUCwdjjE9HPINQ1etEpBMwFXhGRBR4BnhJVcsCXaBpPisL93LHC0vYUbqfP142jKlj+gS7JGNMK+bPGQSqWgq8DswCegCXAt+IyH8dbj8RmSQia0Vkg4jc72N9goi8JSLLRGSViNzktS5XRFaISI6IZB/VpzLf8cY3BVz+5JfU1ikv/+BkCwdjzBH50wdxEXAz0A94HhijqjtEJA74FvhHE/tFAo8DE4ECYLGIzFXV1V6b3QWsVtWLRKQLsFZEZqpqtWf92aq681g/nHF3Rf/hndU899UWxmYk8di1o+jSMSbYZRlj2gB/rmK6Evg/VV3gvVBVK0Tk5sPsNwbYoKqbAERkFjAF8A4IBTqKu66yA1AC1B5F/eYwdpRW8cOZ35C9ZTe3npbB/ZMHERXp10mjMcb4FRC/ArY2PBGR9kA3Vc1V1Y8Ps18vIN/reQEwttE2jwFzgSKgI3C119VSCnzg6fP4p6pO8/UmInI7cDtAnz7WbNIgO7eEO2d+w76qWv4+9UQuHtEz2CUZY9oYf75Ovgp4X+Ja51l2JL7uttJGz88DcoCewEjgMU+HOMA4VR0FTAbuEpEzfL2Jqk5T1SxVzerSpYsfZYU2VWXGV7lcM20hcdGRzL7rVAsHY8wx8Scgorz6BPA8jvZjvwKgt9fzVNyZgrebgDfU2QBsBgZ53qfI83sHMBvXZGUOo6qmjp+8uoz/mbOKMwZ0Ye7dpzGoe6cj72iMMT74ExDFInJxwxMRmQL403G8GMgUkQwRicbdTzG30TZ5wHjP63YDBgKbRCReRDp6lscD5wIr/XjPsJVfUsHlT37J7KWF/GjCAJ6+PouE9u2CXZYxpg3zpw/iDmCmiDyGazbKB64/0k6qWisidwPvA5HAdFVdJSJ3eNY/BfwOeFZEVnhe+79VdaeI9AVme8YEigJeVNV5R//xwsOCdcXcM2spdfXKv2/I4pxB3YJdkjEmBIhq426BJjYU6eDZvtXeHJeVlaXZ2eFzy4Sq8sT8jTz8wVoGduvIU9eNJj0lPthlGWPaEBFZoqpZvtb5NVifiFwADAViG0b6VNXfNluF5qiVVdXw01eX8f6q7Vw0oid/unwYcdE29qIxpvn4c6PcU0AccDbwNHAFsCjAdZnD2LCjjNufX8KWXRX88sIh3Dwu3YboNsY0O386qU9V1euB3ar6G+AUDr06ybSg91ZsZcpjX1BaWcPMW8dyy2kZFg7GmIDwp02iyvO7QkR6AruAjMCVZHypq1f+8v5anvp0IyN7d+bJ60bRI8FGYTXGBI4/AfGWiHQG/gJ8g7vZ7V+BLMocqqS8mnteWsrnG3Zy7dg+/OqiIcRERQa7LGNMiDtsQIhIBPCxqu4BXheRt4FYVd3bEsUZWFHghugu3refP18+nKtOstY9Y0zLOGwfhGdcpL96Pd9v4dByXs3O5/KnvkRVee2OUywcjDEtyp8mpg9E5HI8Q2IEuiDjhuj+7dureGFhHuP6J/P3a04kuYMN0W2MaVn+BMSPgXigVkSqcHc8q6raID8BsG1vFT+cuYRv8vbwgzP78rNzB9oQ3caYoPBnytGOLVGIgUWbS/jhzG+oqK7l8WtHccHwHsEuyRgTxvy5Ua6pYbYX+Fpujk1+SQXfe3ohvRPjeOm2sWR2s1w2xgSXP01MP/N6HIsbdnsJcE5AKgpTby4tpKZOmXHLGFIT44JdjjHG+NXEdJH3cxHpDfw5YBWFIVXlzZxCxmQkWTgYY1qNY+n9LABOaO5CwtnqraVsLC5nykib+c0Y03r40wfxDw5OFRqBmxp0WQBrCjtzc4qIihDOP8E6pY0xrYc/fRDeEyzUAi+p6hcBqifs1Ncrc5cVceaALiTG+zOTqzHGtAx/AuI1oEpV6wBEJFJE4lS1IrClhYfFuSVs3VvF/ZMHBbsUY4w5hD99EB8D3sOGtgc+Ckw54WfOsiLat4tk4hCbJtQY07r4ExCxqrqv4YnnsV1q0wyqa+t5d8VWJg7pZrPBGWNaHX8ColxERjU8EZHRQGXgSgofn60vZk9FjV29ZIxplfwJiPuAV0XkMxH5DHgZuNufFxeRSSKyVkQ2iMj9PtYniMhbIrJMRFaJyE3+7hsK5uQU0TmuHadndgl2KcYY8x3+3Ci3WEQGAQNxA/WtUdWaI+0nIpHA48BE3L0Ti0Vkrqqu9trsLmC1ql4kIl2AtSIyE6jzY982rXx/LR+u3s6lo3oRHWWD8RljWp8jHplE5C4gXlVXquoKoIOI/NCP1x4DbFDVTapaDcwCpjTaRoGO4iZV7gCU4C6l9WffNu2jb7dTWVPHlBHWvGSMaZ38+ep6m2dGOQBUdTdwmx/79QLyvZ4XeJZ5ewwYDBQBK4B7PZMU+bMvACJyu4hki0h2cXGxH2W1DnNziuiREMtJ6UluQelW+PwRWPMOVJcHtTZjjAH/7oOIEBFpmCzI03Tkzx1d4mNZ4wmHzgNycAP/9QM+9PRz+LOvW6g6DZgGkJWV1SYmNNpdXs2n64q55bQMIipL4PO/weKnobbKbRAVC33PggGT3E8nu8PaGNPy/AmI94FXROQp3EH6DuA9P/YrALznyEzFnSl4uwl4yBM+G0RkMzDIz33brHdXbiWufh+31L4Ej/4baipgxFQ47cdQWghr34O178C6eW6HnqNg4PkwcDJ0GwriKz9DxJ482PwZ5H4O+7ZBxhmQeS50HRLan9uYVkiONIuoiEQAtwMTcN/slwI9VPWuI+wXBawDxgOFwGLgWlVd5bXNk8B2Vf21iHQDvgFGAHuOtK8vWVlZmp2dfbhNgq+6nBf/8SAXlr1GJ/bB0EvhrAehy4BDt1OFHd/C2nddYBR6PldCHxg4yYVF2mkQ1caH59hbcDAQche4gABonwQdukLxGve8Uy/oPwEyJ0LGmRBrExoa0xxEZImqZvlc58800yIyErgWuBrYBLyuqo/5sd/5wCNAJDBdVf8gIncAqOpTItITeBbogQufh1T1hab2PdL7teqAqKmC7OnUffZXIit2sjnpdDKu/F/oMdy//cu2w/r3XVhs/ARqKyGmE/Qf784u+k+AuKTAfobmUFrkCQTPz+5ct7x9IqSNc2cM6adBl8EQEeG23/ARrP8ANs6H6jKIiII+p7iwyDwXugwK3bOL+joXkluXQepJkJIZ7IpMiDmmgBCRAcA1wFRgF+7+h5+qalqgCj1erTIg6mpg6Quw4C9QWkhB4hj+a9sFPPLT20hLjj+216yugM2furOLde/Dvu0gke6gOXCy+0nu17yf41iVbXNnB5sXuN8lG93y2AR3BpR+GmScDl2HukA4nLoayP/ahcX6j2CH54SyU6onLDxnFzEdAvuZAmlfMRQsdmeMBYuh8Buo9gxkEJMA358NqaODW6MJKccaEPXAZ8AtqrrBs2yTqvYNWKXHqVUFRH0drHgV5v/RfUtOHQPjf8n5c4XoqAjevGtcM71PPRQtPdgU1XDQTBnoCYvzITULIiKb5/2OZN8Od2bQ0Gy0a71bHpMAaaceDIRuJxx/TXsLPGcXH8Km+e5AGtHOvU/D2UXKgNZ7dlG7H7atcEFQ4AmEPVvcOomE7ie4s4bUkyAxA2bfDhW74frZ0MtCwjSPYw2IS3FnEKcC83D3IjytqhmBKvR4tYqAqK+Hb+fCJ/8LO9dC9+Fwzi8hcyLrd+xj4v8t4FcXDeGmcQH6M+7OhbXzYN177gBdXwtxye5qqIGToe/ZzfsNu3ynp7nocxcKO9e65dEdIe0USD/dBUL34YENqdpqyF948Oyi+Fu3PKGP19nFGRB9jGdtx0vV9a80hEFhtms2qqt26zv1cgf9hkDoMQKiGw15ticfnr0AKvdYSJhmc1x9ECISD1yCa2o6B3gOmK2qHzRzncctqAGh6r7J/ud3sG25+wZ/zv+DQRcdaDr56wdrefyTDSx8cDxdO8YGvqaqve4b9tr33IGzai9ExkDfM11YDJgEnY7yRr3yXbDl84OB0HAgbhd/MBDST3cHuMggDkC4Jx82fOjCYtN8qCmHyGjXz9FwdpHcP3BnF/vL3JndgbODbCjf4dZFtYeeJ7qmotSToFcWJPi8zcf357KQMM3ouDupvV4oCbgSuFpVz2mm+ppN0AJi8wL4z+9d+3hiOpz1AAy78pBvzKrKmX+ZT5+kOF64dWzL11hXA3lfeS6hffdg53CPkQcvoe0+7LsHzIoS2PLlwWajhiasdnHQ52TXZJR+BvQcCZHtWvADHYXa/e6zr//Q/TSc5XROc0GROdGFWuNv7P6qr4ed6zxh4AmE4m9B69365P6eM4MsFwbdhh7f3+qQkHgTeo060h7GNKnZAqK1a/GAyF/szhg2fwode8KZP4cTr/P5j39p3m4ufeJL/nzFcK7K6u3jxVqQKhSvPdhvUbAYUNfZO3CyO5htzXGhsG2lWxfVHvqM9QqEE9vuJba7txw8u9j8qbsPJTLGfbYDZxeH6eQv33WwE7mhI3l/qVsXm+BCoKGpqNeowFxdticPnr3QQsIcNwuI5rZ1OXzyB3cjW1wKnP4TyLoZ2jXdbPTruat4cVEe2b+YQKfYVvZNe98OdzXU2vdg0ycHD5i9xxy87LTXaIiKCXalza+mCvK+dGGx/oODneqJGQfPLuKSoGDJwUDYvdltI5HubKDh7CD1JEjqd+SrsZrLnjx3JlG1F66f40LbmKNkAdFcite5YFj9pvumOO5eGPODI3b61tbVc/If/0NWWiJPfb+VtxnXVLrmkpSBhw28kLU792BT1OYF7n6TBh26Q++TDvYb9BwZvE7vBuEYEnvyYfG/YMztkJAa7GraPAuI41WyGT79Myyf5dreT/4hnHIXtO/s1+6frS/m+/9exJPfG8XkYTauUptRUwVbvnCXz/Ya7a40ao2XzIZTSGz4CF6/DSpLoGMP+N5r7nJgc8wOFxA2EcHhlBbB2z+Cx7Jg1RsuGO5d5q5O8jMcwI3c2jEmirMHdQ1crab5tYt1d6oPmeK+qbbGcADo3AdueNvdazLjEijKCXZFza++Dj75I7xwhQuGa14CiYBnJrur1ExAWED4sq8Y5j0Ij46Eb56H0TfCPTlw3h8gPuWoXqqqpo55K7dx3gndiW3XQjermfCTmAY3vu2GX5kxJbRConwXzLwCPn0IRlwDt34Eg86HWz6EhN4uNJa/EuwqQ5IFhLfK3fDx7+DREfD1kzDsCvivJXDBX495yO35a3dQtr/W5p02gReKIVGwBP55hrvv5sJH4JInD16OnNALbn7PXW79xm3w2d/cFXqm2VhAAOzf58ZKenQEfPYwDDgPfvg1XPKE+0d3HObkFJHSIYZT+iY3U7HGHEaohIQqLPoXTD/PXRV2yweQddN3m/liE+C61919Rx//Bt75iWuOMs0iiLe6thKVe1wfQ3kxDJjs+he6D2uWly6tquHjNTu4dkwfoiIti00LaQiJZy9wIXHDXHdne1tRXQ5v3evGMss8Fy795+HvJYmKgUunuYsIvnjEDRB5+dPHfuOjOcCOWu07u87nWz+Ga2c1WzgAvL9yG9W19da8ZFregTOJji4kti4LdkX+KV4H/zoHVrwG5/wCpr7s342GEREw8Tdw/sPuBtAZF7u+C3NcLCAATv+xu9Gpmc1dVkSfpDhG9u7c7K9tzBElpruQiO7gCYnlwa7o8Fa+Af86253Nf382nPGzo7/pcMxtcPXzbpTcf0+Ekk2BqTVMWEAEyI6yKr7YsJMpI3sirfXySBP6EtPhhrc8IXFx6wyJ2mp473547SY3tewPPoN+Zx/76w2+CK6f6+6V+Pe5ULik+WoNMxYQAfLO8q3UK9a8ZIIvKcOFRLv41hcSewtdX8nXT8LYO+HGd/wf2fZw+ox1l8G2i3NjVq1rdYNPtwkWEAEyJ6eIwT060b9rx2CXYowLiRvfPhgS21YEuyJ3g9s/z4Adq+GKZ2DyQ807AGRKpguJlAHw0jWw5Lnme+0wYQERAFt2lZOTv8fOHkzrkpQBN3rOJJ67KHghUV/vLit//lJ34+ltn8AJlwXmvTp2c2cl/c6Bt+5xE3nZvRJ+s4AIgLk5RQBcNMICwrQySX09IREHzwXhTKKixH2b/8/vYehl7urBLgMC+54xHWDqS24o/k//BHPucvOjmCMKaECIyCQRWSsiG0Tkfh/rfyYiOZ6flSJS55mUCBHJFZEVnnWtZKLpI1NV3swpZEx6Er06tw92OcZ8V1JfT3NTe09IrGyZ9y1aCtPOhI3/cZejXv50805/eziR7eDix9xkXjkz4cWr3ax/5rACFhAiEgk8DkwGhgBTRWSI9zaq+hdVHamqI4EHgE9VtcRrk7M965v/GtQAWb21lI3F5VxszUumNTskJC4KbEioQvYz7oqi+nq4eZ67HLWlr+4TgbPud0GxaT48c767qc40KZBnEGOADaq6SVWrgVnAlMNsPxV4KYD1tIi5OUVERQjn27DeprVL6uu5uqm9p+M6ACFRXQFv/hDevs9NPPWDBQG55+iojPo+XPsy7NoIT090N+cZnwIZEL2AfK/nBZ5l3yEiccAk4HWvxQp8ICJLROT2pt5ERG4XkWwRyS4uLm6Gso9dfb3y1rIizhjQhaT4Njodpwkvyf1cSETGuJDYvqr5XnvXRnez2rKX4Mz73dwN8a1kTLLMiXDTO1Bb5Wrc8lWwKzo+VXsD8rKBDAhf549NXT5wEfBFo+alcao6CtdEdZeInOFrR1WdpqpZqprVpUuX46v4OGVv2U3R3iq7esm0Lcn9XHNTZIxrbmqOkFg9F6adBaWFLhjOfgAiWtlw9z1PhFs/dFdSzZgCq+cEu6Kjs68YFj7l/s7Tzg7I1VmBDIgCoLfX81SgqIltr6FR85KqFnl+7wBm45qsWrU5OYW0bxfJhMHdgl2KMUenuUKirgbe/3/wyvchub9rUsqc0Ly1NqfEdLj5AzeY4Ss3uANua1ZTCStfh5lXwV8Hwrz/hvpayLrZ/W5mgRzNdTGQKSIZQCEuBK5tvJGIJABnAtd5LYsHIlS1zPP4XOC3Aaz1uFXX1vPOiq1MHNKN+BgbJNe0QQ0h8ewFLiRueBu6DTnyfg3KtsGrN0Hel3DSrXDe/7qRVlu7+GQ34u3rt7oDbmkBTPjt0Y8DFSj19W7q2+Wz3JnZ/lLo2BNOvRuGX3N0/42OUsCOZKpaKyJ3A+8DkcB0VV0lInd41jdE9aXAB6pa7rV7N2C2ZwyjKOBFVZ0XqFqbw+cbitlTUWPNS6ZtS+7nbiw7EBJv+XcAyv3chUP1PrjsaRh+ZeBrbU7t2sNVM2De/fDlP9x0w5c8GdyA27EGlr/shj3fm+/G0xoyBYZfBemnt0iTnWgI3VWYlZWl2dnBuWXi3llL+XRdMYsenEB0VCv55mHMsdq10YVEXc3hQ0IVvngUPv6tuyrq6ueh6+CWrbU5qcKXf4cP/wfSToNrZh7V/PPHbd8ON9T58lluiHaJdHeBj7gGBp4fkDkuRGRJU7cSWFtIM6ioruWDVdu5dFQvCwcTGpL7uSam5y50ZxI3vv3dA3/lHncJ69p3YMglMOUxN/9EWyYC4+51TThv3gnTJ8F1r0FCauDes7rCzWGxbJa7iVDrXJ/IeX900x536Bq49z4CC4hm8OHq7VTW1DHFhtYwoSSlvwuJZy9wI6J6h8TW5fDK9a7pY9JDMPaOlr/xLZCGX+nGcZr1PXh6grsSq/sJzff69XWQ+xksf8X1K1SXQadUGHeP61foOqj53us4WEA0g7k5RfRIiOWkdD9mvjKmLUnp/90+icIlbu7n9olw47tuaO1QlHGGu+v7hSvgmcmu+azvWcf3mttXu+aj5a9CWRFEd4ShU1wopI1rPR3jHhYQx2l3eTWfrivm5tMyiIgIoW9QxjRI6e+5uulCd719baU7eF4+HToE996jgOs2FG79CGZe4YLikidcJ/HRKNt2sF9h2wrXr9B/Apz3e9ev0K71jtlmAXGc3l25ldp65WJrXjKhLCXThcSrN8LAyW7Qu9Z241ugJPSCm96Dl6+DN26DvQVw2o8O36RWXQ5r3nH9Cps+Aa13N+ZN+hOccHmbCVYLiOM0J6eIfl3iGdqzU7BLMSawUjLhzi+CXUVwtO8M173uOuU//o27Q3zynw8Nyfo62PwpLHsZvn0LasohobcLk+HXBH5Y8wCwgDgORXsqWbS5hB9PHGDzThsT6qJi4LJ/uTOKLx6F0q1uyPKSTa75aMVrULYVYhJg2OUuFPqc0ur6FY6GBcRxeGuZGznEmpeMCRMRETDxt+6Ko/d+Dn8b5AbKi4iC/hNh0h9hwGRoFxvsSpuFBcRxmJNTxIjenUlPiQ92KcaYljT2dncmsXSmu7LphMvcoH8hxgLiGG3YUcbqraX8z4WBGwfFGNOKDbrA/YSwtts4FmRzc4qIELhwuE0MZIwJTRYQx0BVmbOsiFP7pdC1U2i0NRpjTGMWEMdgWcFetuyqsHmnjTEhzQLiGMzJKSQ6KoJJJ3QPdinGGBMwFhBHqa5eeWvZVs4Z2JVOse2CXY4xxgSMBcRR+mrjLnbu228TAxljQp4FxFGak1NIh5gozh4UvDHajTGmJVhAHIWqmjrmrdzGeUO7E9suTAYqM8aELQuIozB/7Q7K9tda85IxJixYQByFOTlFpHSI5tR+ycEuxRhjAi6gASEik0RkrYhsEJH7faz/mYjkeH5WikidiCT5s29LK62q4eM1O7hweE+iIi1XjTGhL2BHOhGJBB4HJgNDgKkicsjARar6F1UdqaojgQeAT1W1xJ99W9r7K7dRXVtvN8cZY8JGIL8KjwE2qOomVa0GZgFTDrP9VOClY9w34OYuK6J3UntO7N05mGUYY0yLCWRA9ALyvZ4XeJZ9h4jEAZOA149h39tFJFtEsouLi4+7aF92lFXxxYadTBnRyyYGMsaEjUAGhK8jqTax7UXAF6pacrT7quo0Vc1S1awuXQIzz+u7y7dSr9jVS8aYsBLIgCgAens9TwWKmtj2Gg42Lx3tvgE3Z1kRg3t0IrNbx2CVYIwxLS6QAbEYyBSRDBGJxoXA3MYbiUgCcCYw52j3bQl5uypYmrfHzh6MMWEnYDPKqWqtiNwNvA9EAtNVdZWI3OFZ/5Rn00uBD1S1/Ej7BqrWw5m7rBCAi2zeaWNMmAnolKOq+i7wbqNlTzV6/izwrD/7tjRV5c2cIsakJ9Grc/tglmKMMS3O7vg6jG+3lrFhxz6798EYE5YsIA5jzrJCoiKE84fZvNPGmPBjAdGE+nrlrZwiTs9MISk+OtjlGGNMi7OAaEL2lt0U7a1iykif9+cZY0zIs4BowpycQmLbRTBxSLdgl2KMMUFhAeFDdW0976zYysQh3YmPCeiFXsYY02pZQPjw+YZi9lTUMMXufTDGhDELCB/m5BSR0L4dZwwIzNhOxhjTFlhANFJRXcsHq7Zz/rAeREfZn8cYE77sCNjIh6u3U1lTZ2MvGWPCngVEI28tK6J7p1jGpCcFuxRjjAkqCwgvu8urmb+2mItH9iQiwiYGMsaENwsIL++t3EZtvXKxXb1kjDEWEN7m5BTSr0s8Q3t2CnYpxhgTdBYQHkV7KlmUW8KUkTbvtDHGgAXEAW8vL0IVa14yxhgPCwiPOTlFjEhNID0lPtilGGNMq2ABAWzYUcaqolIutpFbjTHmAAsIYG5OESJw0XCbGMgYYxqEfUCoKnOWFXFqv2S6dooNdjnGGNNqhP1Y1pU1dZzSN5lT+6cEuxRjjGlVAhoQIjIJeBSIBJ5W1Yd8bHMW8AjQDtipqmd6lucCZUAdUKuqWYGoMS46iocuHx6IlzbGmDYtYAEhIpHA48BEoABYLCJzVXW11zadgSeASaqaJyJdG73M2aq6M1A1GmOMaVog+yDGABtUdZOqVgOzgCmNtrkWeENV8wBUdUcA6zHGGHMUAhkQvYB8r+cFnmXeBgCJIjJfRJaIyPVe6xT4wLP89qbeRERuF5FsEckuLi5utuKNMSbcBbIPwtd4Ferj/UcD44H2wFcislBV1wHjVLXI0+z0oYisUdUF33lB1WnANICsrKzGr2+MMeYYBfIMogDo7fU8FSjysc08VS339DUsAEYAqGqR5/cOYDauycoYY0wLCWRALAYyRSRDRKKBa4C5jbaZA5wuIlEiEgeMBb4VkXgR6QggIvHAucDKANZqjDGmkYA1MalqrYjcDbyPu8x1uqquEpE7POufUtVvRWQesByox10Ku1JE+gKzPaOqRgEvquq8QNVqjDHmu0Q1dJrts7KyNDs7O9hlGGNMmyEiS5q6zyykAkJEioEtx7h7CmD3XDj2tziU/T0OZX+Pg0Lhb5Gmql18rQipgDgeIpIdqLu12xr7WxzK/h6Hsr/HQaH+twj7wfqMMcb4ZgFhjDHGJwuIg6YFu4BWxP4Wh7K/x6Hs73FQSP8trA/CGGOMT3YGYYwxxicLCGOMMT6FfUCIyCQRWSsiG0Tk/mDXE0wi0ltEPhGRb0VklYjcG+yagk1EIkVkqYi8Hexagk1EOovIayKyxvP/yCnBrimYRORHnn8nK0XkJREJuTmLwzogvCY1mgwMAaaKyJDgVhVUtcBPVHUwcDJwV5j/PQDuBb4NdhGtxKO4wTUH4QbVDNu/i4j0Au4BslT1BNxwQtcEt6rmF9YBgX+TGoUNVd2qqt94HpfhDgCN5/AIGyKSClwAPB3sWoJNRDoBZwD/BlDValXdE9Sigi8KaC8iUUAc3x2tus0L94DwZ1KjsCQi6cCJwNdBLiWYHgF+jhtIMtz1BYqBZzxNbk97RloOS6paCDwM5AFbgb2q+kFwq2p+4R4Q/kxqFHZEpAPwOnCfqpYGu55gEJELgR2quiTYtbQSUcAo4ElVPREoB8K2z05EEnGtDRlATyBeRK4LblXNL9wDwp9JjcKKiLTDhcNMVX0j2PUE0TjgYhHJxTU9niMiLwS3pKAqAApUteGM8jVcYISrCcBmVS1W1RrgDeDUINfU7MI9IPyZ1ChsiJuA49/At6r6t2DXE0yq+oCqpqpqOu7/i/+oash9Q/SXqm4D8kVkoGfReGB1EEsKtjzgZBGJ8/y7GU8IdtoHck7qVq+pSY2CXFYwjQO+D6wQkRzPsgdV9d3glWRakf8CZnq+TG0CbgpyPUGjql+LyGvAN7ir/5YSgsNu2FAbxhhjfAr3JiZjjDFNsIAwxhjjkwWEMcYYnywgjDHG+GQBYYwxxicLCGOOgojUiUiO10+z3U0sIukisrK5Xs+Y4xXW90EYcwwqVXVksIswpiXYGYQxzUBEckXkTyKyyPPT37M8TUQ+FpHlnt99PMu7ichsEVnm+WkYpiFSRP7lmWfgAxFpH7QPZcKeBYQxR6d9oyamq73WlarqGOAx3EiweB7PUNXhwEzg757lfwc+VdURuDGNGu7gzwQeV9WhwB7g8oB+GmMOw+6kNuYoiMg+Ve3gY3kucI6qbvIMeLhNVZNFZCfQQ1VrPMu3qmqKiBQDqaq63+s10oEPVTXT8/y/gXaq+vsW+GjGfIedQRjTfLSJx01t48t+r8d1WD+hCSILCGOaz9Vev7/yPP6Sg1NRfg/43PP4Y+BOODDvdaeWKtIYf9m3E2OOTnuvkW7BzdHccKlrjIh8jfviNdWz7B5guoj8DDcjW8MIqPcC00TkFtyZwp24mcmMaTWsD8KYZuDpg8hS1Z3BrsWY5mJNTMYYY3yyMwhjjDE+2RmEMcYYnywgjDHG+GQBYYwxxicLCGOMMT5ZQBhjjPHp/wPpDL+S0BltqwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvIElEQVR4nO3deXxU1f3/8dcnOwkkkLAmIWwieyCIUHcWFzbLolbB1tr6/VqtVv3axdrl2/b76161aqu21ra2VYpWwQUVt4prRZGwg4iAIQsQEkgCZM/5/XEnGELAAJnczMz7+XjkwcydOzOfDHDfc8859xxzziEiIpEryu8CRETEXwoCEZEIpyAQEYlwCgIRkQinIBARiXAKAhGRCKcgEGkFM+tvZs7MYlqx79Vm9tbJvo5Ie1EQSNgxs+1mVmNm3ZttXxU4CPf3qTSRDklBIOFqGzCv8Y6ZjQI6+VeOSMelIJBw9Q/gqib3vwz8vekOZpZiZn83s2Iz+8TMfmBmUYHHos3sDjPbY2ZbgRktPPfPZlZkZgVm9lMziz7eIs0s3cyeMbNSM9tiZv/d5LHxZrbCzMrNbJeZ3RXYnmBmj5hZiZntM7P3zazX8b63SCMFgYSrd4FkMxsWOEBfDjzSbJ/fASnAQOA8vOD4SuCx/wZmAjnAOODSZs/9G1AHnBLY50Lgv06gzn8C+UB64D1+bmZTAo/dA9zjnEsGBgGPB7Z/OVB3XyANuA6oPIH3FgEUBBLeGs8KLgA2AQWNDzQJh9udcxXOue3AncCXArt8AbjbObfDOVcK/KLJc3sB04BbnHMHnHO7gd8CVxxPcWbWFzgbuM05V+WcWwU81KSGWuAUM+vunNvvnHu3yfY04BTnXL1z7gPnXPnxvLdIUwoCCWf/AOYDV9OsWQjoDsQBnzTZ9gmQEbidDuxo9lijfkAsUBRomtkH/BHoeZz1pQOlzrmKo9RwDXAqsCnQ/DOzye/1IrDQzArN7NdmFnuc7y1yiIJAwpZz7hO8TuPpwKJmD+/B+2bdr8m2LD49ayjCa3pp+lijHUA10N051zXwk+ycG3GcJRYCqWbWpaUanHMfOefm4QXMr4AnzCzJOVfrnPuJc244cCZeE9ZViJwgBYGEu2uAyc65A003Oufq8drcf2ZmXcysH3Arn/YjPA7cZGaZZtYN+G6T5xYBLwF3mlmymUWZ2SAzO+94CnPO7QDeAX4R6ADODtT7KICZfdHMejjnGoB9gafVm9kkMxsVaN4qxwu0+uN5b5GmFAQS1pxzHzvnVhzl4W8AB4CtwFvAAuAvgcf+hNf8shpYyZFnFFfhNS1tAPYCTwB9TqDEeUB/vLODxcCPnHMvBx6bCqw3s/14HcdXOOeqgN6B9ysHNgKvc2RHuEirmRamERGJbDojEBGJcAoCEZEIpyAQEYlwCgIRkQgXclPhdu/e3fXv39/vMkREQsoHH3ywxznXo6XHQi4I+vfvz4oVRxsNKCIiLTGzT472mJqGREQinIJARCTCKQhERCJcyPURtKS2tpb8/Hyqqqr8LiXoEhISyMzMJDZWk02KSNsIiyDIz8+nS5cu9O/fHzPzu5ygcc5RUlJCfn4+AwYM8LscEQkTYdE0VFVVRVpaWliHAICZkZaWFhFnPiLSfsIiCICwD4FGkfJ7ikj7CZsgEBEJWzUH4O17IO/dz973BCgI2kBJSQljxoxhzJgx9O7dm4yMjEP3a2pqjvncFStWcNNNN7VTpSISUmoOwju/g3tGw8v/C5uXBuVtwqKz2G9paWmsWrUKgB//+Md07tyZb33rW4cer6urIyam5Y963LhxjBs3rj3KFJFQUXMQVvwF3r4bDhTDwIkw8XuQNSEob6cgCJKrr76a1NRUcnNzGTt2LJdffjm33HILlZWVdOrUib/+9a8MGTKEZcuWcccdd7BkyRJ+/OMfk5eXx9atW8nLy+OWW27R2YJIJKmthBV/9QJg/y4YcB5MvB36nRHUtw27IPjJs+vZUFjepq85PD2ZH118vOuSw+bNm3nllVeIjo6mvLycN954g5iYGF555RW+973v8eSTTx7xnE2bNvHaa69RUVHBkCFDuP7663XNgEi4q62CDx6Gt34L+3dC/3Pg0r9C/7Pa5e3DLgg6kssuu4zo6GgAysrK+PKXv8xHH32EmVFbW9vic2bMmEF8fDzx8fH07NmTXbt2kZmZ2Z5li0h7qa2ClX+Ht+6CiiLodzZc8hAMOKddywi7IDiRb+7BkpSUdOj2D3/4QyZNmsTixYvZvn07EydObPE58fHxh25HR0dTV1cX7DJFpL3VVXsB8OZdUFEIWWfC3AdhwLm+lBN2QdBRlZWVkZGRAcDDDz/sbzEi4o+6ash9xAuA8nzo+zmY84DXF+DjNUIaPtpOvvOd73D77bdz1llnUV9f73c5ItKe6mq8UUD3joXnboXkdPjSYvjqUm9EkM8XippzztcCjte4ceNc84VpNm7cyLBhw3yqqP1F2u8rErLqa2HVo/DGnVCWBxnjYNLtMGhKux/8zewD51yLY9XVNCQi0tbqa2H1P+GN38C+PMg4DWb+Fk5p/wBoDQWBiEhbqa+FNY95AbB3O6TnwPQ7YfAFHTIAGikIREROVn1dkwDYBn3GwLzH4NSLOnQANFIQiIicqPo6WPsveOPXULoVemfDFf+EIdNCIgAaKQhERI5XQz2sfcILgJIt0HsUXLEAhkwPqQBopCAQEWmthnpYtwhe/xWUfAS9RsLlj8CQGRAVuqPxQ7fyDmTixIm8+OKLh227++67+frXv37U/ZsPgRWRDqzxDOD+M2DRf0F0LHzh7/C1N2HYxSEdAqAgaBPz5s1j4cKFh21buHAh8+bN86kiEWkTzsGGp+GBM+HJa8Ci4LKH4bq3YfiskA+ARuHxW/js0ksvZcmSJVRXVwOwfft2CgsLWbBgAePGjWPEiBH86Ec/8rlKETkuRWvg4Znw+FVeIFz6F7j+HRgxJ2wCoFFQ+wjMbCpwDxANPOSc+2Wzx1OAR4CsQC13OOf+elJv+sJ3Yefak3qJI/QeBdN+edSH09LSGD9+PEuXLmXWrFksXLiQyy+/nNtvv53U1FTq6+uZMmUKa9asITs7u21rE5G2dWAP/Pv/wQd/g07dYMZdMPbLEB2+XapBizUziwbuA6YBw4F5Zja82W43ABucc6OBicCdZhYXrJqCqWnzUGOz0OOPP87YsWPJyclh/fr1bNiwwecqReSo6mrgP/d58wHlPgKfux5uWgmnXxPWIQDBPSMYD2xxzm0FMLOFwCyg6dHQAV3MzIDOQClwcvMuH+ObezDNnj2bW2+9lZUrV1JZWUm3bt244447eP/99+nWrRtXX301VVVVvtQmIp9h80vw4u3eUNBBU2DqL6DHEL+rajfBbOjKAHY0uZ8f2NbU74FhQCGwFrjZOdfQ/IXM7FozW2FmK4qLi4NV70np3LkzEydO5Ktf/Srz5s2jvLycpKQkUlJS2LVrFy+88ILfJYpIc8Wb4ZFLYcFl3v35j8MXn4yoEIDgnhG0dFVF86lOLwJWAZOBQcDLZvamc+6wtSadcw8CD4I3+2jbl9o25s2bx9y5c1m4cCFDhw4lJyeHESNGMHDgQM46q32WnBORVqjcB6//Gt77I8QmwoU/g/HXQkxItkyftGAGQT7Qt8n9TLxv/k19Bfil8+bC3mJm24ChwHtBrCto5syZQ9NpvY+2AM2yZcvapyAROVxDPaz8G/z7p3CwFMZeBZN/CJ17+F2Zr4IZBO8Dg81sAFAAXAHMb7ZPHjAFeNPMegFDgK1BrElEItW2N2Hp7bBrLfQ7y+sH6DPa76o6hKAFgXOuzsxuBF7EGz76F+fcejO7LvD4H4D/BzxsZmvxmpJuc87tCVZNIhKB9n4CL//QuzAspa93Qdjw2SE5J1CwBHVMlHPueeD5Ztv+0OR2IXBhG70XFgF/saG2opyIb6r3w1u/hXd+B1HRMOkHcOaNENvJ78o6nLAYHJuQkEBJSQlpaWlhHQbOOUpKSkhISPC7FJGOq6HBmxr6lR9BRRGM+gKc/2NIaT5oURqFRRBkZmaSn59PRx1a2pYSEhLIzMz0uwyRjin/A1h6G+S/760OdtnfIGuC31V1eGERBLGxsQwYMMDvMkTEL+VF8OpPvHWCO/eC2Q9A9hVhNydQsIRFEIhIhKqtgnfvgzfuhIZaOPt/4JxvQnwXvysLKQoCEQk9zsGmJfDi92HfJzB0Jlz4/yB1oN+VhSQFgYiEll3rYel3Ydsb0HM4XPU0DJzod1UhTUEgIqHhQAm89jP44K+QkALT74DTvhL2M4O2B32CItKx1dfC+3+GZT/3rg0Yfy2cdxskpvpdWdhQEIhIx7XlFVj6PdjzIQyaDBf9AnoO9buqsKMgEJGOZX8xbHgK1jwO+e95HcDzHoNTL9K0EEGiIBAR/1VXwKbnvCuCP34NXD30GglTfwnjvgox8X5XGNYUBCLij7oa+PhV75v/hy9AXSWkZMFZN8Ooy6BX85VtJVgUBCLSfhoaIO8/sPZxWP8UVO2DxDTIudI7+GeO19XAPlAQiEhwOQe71nnNPmufhPJ8iE2CoTO8g/+gSRAd63eVEU1BICLBsXd74OD/BBRvgqgYOOV8uOAnMGQaxCX5XaEEKAhEpO00H/EDkHUGzLjLWwwmKc3P6uQoFAQicnKqK2DT8167f+OIn54jvDUARl4CXbP8rlA+g4JARI5fiyN++sJZNwVG/Izwu0I5DgoCEWmdlkb8dEqFMfMh+wsa8RPCFAQicnQtjvhJbDLiZ7JG/IQBBYGIHGnvdm+0z9p/fTriZ9AUr91/6HSN+AkzCgIR+VRdtbfYy/t/8u5nnQEz7oThczTiJ4wpCETEU5YPj38ZClbA+K/BGTdAt35+VyXtQEEgIvDxv+GJa7y5/y/7G4yY7XdF0o4UBCKRrKEB3rzTW/mr5zD4wj+g+yl+VyXtTEEgEqkOlsLir8FHL8GoL8DFd6sTOEIpCEQiUWEuPHYVVBR5ncHjrtGiLxFMQSASSZyDlX+D578NST3hq0shc5zfVYnPFAQikaLmIDz/LVj1KAycBJf8WUNCBVAQiESGko+9oaG71sJ5t3k/UdF+VyUdhIJAJNxteg4WX+/1Acz/F5x6od8VSQejIBAJV/V18NpP4a3fQp8x8IW/6wIxaZGCQCQc7d8NT3wVtr8Jp10NU38FsQl+VyUdlIJAJNzkvev1B1Ttg1n3ewvDixyDgkAkXDgH7z4AL//QWyTmi09A71F+VyUhQEEgEg6qK+DpG731godMh9kPQKeuflclIUJBIBLqdm+Cx74IpR976wWcebNWCpPjEtR/LWY21cw+NLMtZvbdo+wz0cxWmdl6M3s9mPXk7z0YzJcXaX9rn4A/Tfb6A656Gs7+H4WAHLeg/Ysxs2jgPmAaMByYZ2bDm+3TFbgf+LxzbgRwWbDqWZybz6Q7lrEyb2+w3kKk/dTVwPPfgSev8foBvvYmDDjX76okRAXzq8N4YItzbqtzrgZYCMxqts98YJFzLg/AObc7WMVMHtKL3ikJ3PjoSvYeqAnW24gEX1kBPDwd3vsjfO4GuHoJJPfxuyoJYcEMggxgR5P7+YFtTZ0KdDOzZWb2gZld1dILmdm1ZrbCzFYUFxefUDEpibHcN38se/bXcOvjq2hocCf0OiK++vg1+OM5sHsjXPYwTP25Fo+XkxbMIGhpTtvmR98Y4DRgBnAR8EMzO/WIJzn3oHNunHNuXI8ePU64oOzMrnx/xjBe+7CYP76x9YRfR6TdNTTAG7+Bf8yBpB7w36/BiDl+VyVhIpijhvKBvk3uZwKFLeyzxzl3ADhgZm8Ao4HNwSrqqjP68d62Uu546UPG9e/G6f1Tg/VWIm2jci8s+hp89CKMvBQuvgfiO/tdlYSRYJ4RvA8MNrMBZhYHXAE802yfp4FzzCzGzBKBCcDGINaEmfGLS0bRt1snblywkpL91cF8O5GTU7gK/niet6bwtN/AJQ8pBKTNBS0InHN1wI3Ai3gH98edc+vN7Dozuy6wz0ZgKbAGeA94yDm3Llg1NUpOiOW+K8ey92Attzym/gLpoFb+Hf58ITTUwVdegAnXahUxCQpzLrQOguPGjXMrVqxok9d6dPknfH/xOr55wal8Y8rgNnlNkZNWW+ktIJP7CAycGFhAprvfVUmIM7MPnHMtLkcX0VcWzx+fxfKtpfz2lc2c1r8bZw7SfzbxWek2ePxLsHMtnPttmHi7FpCRoIvoSxDNjJ/PHUX/7kncvHAVxRXqLxCf7FoPL37f6w/YlwfzH4fJP1AISLuI6CAA6Bwfw/1XjqWiqpabF+ZSr/4CaS8HS2H5g97B/4EzYfkfYeC58LU34NSL/K5OIkhENw01Gto7mf/7/Ei+8+Qa7nn1I2694IhLGUTaRn0tbHnVW0D+wxegoRZ6Z3sLx4y6TIvJiy8UBAGXjcvk3W0l/O7fH3F6/26cM/jEL1wTOcKu9bBqAax5HA7shsTuMP5aGDNPawaI7xQEAWbGT2ePZG1+GbcsXMXzN59Dr2Qt7Scn4WCpNzvoqkehaBVExcKQqTDmSjjlfE0NIR2GgqCJxDivv+Dzv3+bb/wzlwX/NYGY6IjvRpHjUV8LW14JNP0s9Zp++oyGab/2rgpW0490QAqCZgb36sLP5ozk1sdXc9fLm/nO1KF+lySh4FDTz2NwoNibD2jC12D0POg90u/qRI5JQdCCuWMzWb61lPuXfczpA1KZNKSn3yVJR3SgBNY1Nv2sVtOPhCwFwVH8ZNYIVufv49bHVvHcTeeQ3rWT3yVJR6CmHwlDCoKjSIiN5v4rx3Lx797iG//MZeG1nyNW/QWRS00/EsYUBMcwsEdnfnFJNjf9M5ffvPgh35s+zO+SpD2p6UcihILgM3x+dDrvbSvhwTe2Mr5/KucP7+V3SRJMavqRCKQgaIUfzBhObt4+vvmv1Tx309lkdkv0uyRpK7VVUFnqrQO84Sk1/UhEUhC0QmN/wcx73+KGBbn862tnEBej/oIOp67GO6gfLGn2s7eFbYH9ag98+nw1/UiEUhC0Ur+0JH59aTbXP7qSX76wif+9eLjfJYW3+lpvicYWD+AtHexLoabi6K8XnwyJqdAp1fu232MoJKZ52xLTvJ9+Z6npRyKSguA4TBvVh6vP7M9f3t7G+AHdmDqyj98lhS7nvJE4H74ApVuPPKhXlx39uXGdDz+Ap53y6e2m2xtvd0qFmLj2+91EQkzkBEF9LdRVQXyXk3qZ700fRm7eXr79xBqG90khK039Ba3W0AD578OmZ2Hjs7B3O2CQnOF9E09Mg9QB3oH7iIN62qff6GM1B5RIW2rVUpVmlgRUOucazOxUYCjwgnOuNtgFNnfCS1Vufslb+emU82H4LDh1KiQkn1ANO0oPMuPeN+mXlsQT159BfIwWDzmquhrY/iZsWgKbnoP9u7y2+IHnwdCZMHQGdNaV2yLB1hZLVb4BnGNm3YBXgRXA5cCVbVNiO+jWD077Cmx42jsoRcfBoCleKAyZBp26tvql+qYmcsdlo7n2Hx/ws+c28n+zNKrkMDUHvDn3Ny2BzUuhqgxik2Dw+TD0Yjj1QkhI8btKEQlobRCYc+6gmV0D/M4592szyw1mYW2uxxCY9ku46OdQsMILhA1Pw+YXAt9QJ8KI2TBkutcE8RkuHNGb/z5nAH96cxun90/l4tHpQf8VOrSDpbD5Re/gv+VVqKuETt0C3/pnwqBJEKtpOkQ6olYHgZmdgXcGcM1xPrdjiYqCvuO9nwt/CgUrvfHjG56Cp2+AqBgYcK53pjD04mOOIvnO1KF88Mlebl+0lpEZKQzontRuv0aHUF4UaPJZAtveBFcPXdJh7Je8g3+/syA6NP+ZiESS1vYRnAd8E3jbOfcrMxsI3OKcuynYBTZ3wn0En8U5b/GQDU/D+qdg7zawaOh/thcKwy5usS27cF8l0+99kz4pnVj89TNJiA3z/oKSj72O3k1LvI5fgLTBMGymF5zpOV7YikiHcqw+glYFQbMXiwI6O+fK26K44xW0IGjKOdi1zguEDU9ByRawKO8bbmModOl9aPfXNu3mKw+/z7zxWfxibpgtO+gc7Fz76cF/9wZve58x3sF/2Oe9ZjcR6dBOOgjMbAFwHVAPfACkAHc5537TloW2RrsEQVPOwe6Ngeajp6F4E2CQdcanoZCSwa+WbuKBZR9z9+VjmJ2T0X71BUNDPexYDhuXeEM99+V5QZh1ZuCb/wzomuV3lSJyHNoiCFY558aY2ZXAacBtwAfOuey2LfWztXsQNLd706cdzbvXe9syx1M/bBY3rurL67vieebGszmlZ2f/ajwRddWw7Q3vm/+Hz3vz7UTHwcBJ3sF/yHRI6u53lSJygtpi+GismcUCs4HfO+dqzez42pTCRc+h3s/E22DPR4FQeIrol7/PA8C6qFN47S/nkPnVm0joOdDvao+tej9sedn75v/RS1Bd7l21O/hC7+B/ygUnfK2FiISO1p4R3IR3FrAamAFkAY84584JbnlH8v2M4GhKPoaNz1Cx8km6lK71tvUZ4zUfDZ8FaYOC877OQc1+qK6AqnLvYF5V7k3R0Hj/iMfKvbH9xR9CfbV31e6Q6V4z14DzdOWuSBhq087iJi8a45yrO6nKTkCHDYImHnrm3+xa/i+u67GOtH1rvI29RwVCYTZ0H+xta6j3DtJND9CHDtplRx68WzrYV1eAazh2QRblTa0Rn+L9mZDsTcKWNshr7+/7OQ3zFAlzbdFHkAL8CDg3sOl14P+cc8eYGSw4QiEI6hscX3xoObk79vLcVQMYtOdVrwlpx3Jvh6Qe3jz4x5ots1FUjHfQbjx4N719tD+bb4vrDGbB/aVFpENriyB4ElgH/C2w6UvAaOfc3DarspVCIQgAdldUMf2et0jpFMMzN55NUnwMlBfChmdg11qI69K6A3lsJx3EReSktdmooc/a1h5CJQgA3tmyhy/+eTmzxmRw1xdGYzqgi4hPjhUErb0EtNLMzm7ygmcBlW1RXDg785Tu3HL+qSzOLeCx93f4XY6ISIta20N4HfD3QF8BwF7gy8EpKbzcMOkU3t9eyv8+s57szK4MT9dwTBHpWFp1RuCcW+2cGw1kA9nOuRxgclArCxPRUcZvLx9Dt8RYbliwkoqqdl/CQUTkmI5rdjDnXHmTOYZuDUI9Yal753juvSKHvNKD3L5oLSc6ZFdEJBhOZprIz+z5NLOpZvahmW0xs+8eY7/TzazezC49iXo6tAkD0/jmhaeyZE0Rj7z7id/liIgccjJBcMyvtWYWDdwHTAOGA/PMbPhR9vsV8OJJ1BISrjt3EJOG9OCHT6/n1sdWsbOsyu+SRESOHQRmVmFm5S38VACftSTXeGCLc26rc64GWAjMamG/bwBPArtP5BcIJVFRxn1XjuXrEwexZG0Rk+5Yxr2vfkRlTb3fpYlIBDtmEDjnujjnklv46eKc+6wRRxlA0zGT+YFth5hZBjAH+MOxXsjMrjWzFWa2ori4+DPetmNLjIvhO1OH8uqt5zFpaA/uenkzU+5cxtOrCtR3ICK+COZSUi31ITQ/0t0N3OacO+ZXYufcg865cc65cT169Gir+nzVNzWR+688jceu/RzdkuK4eeEqLnngHVbt2Od3aSISYYIZBPlA3yb3M4HCZvuMAxaa2XbgUuB+M5sdxJo6nAkD03jmxrP59aXZ7Nhbyez73lb/gYi0qxOeffQzX9gsBtgMTAEKgPeB+c659UfZ/2FgiXPuiWO9bihNMXG89lfX8cCyLfzpzW1Em3HdeYO49tyBdIoL83WQRSTo2mKKieMWmKL6RrzRQBuBx51z683sOjO7LljvG8o6x8fw7Yu8/oPJQ3vy21fUfyAiwRe0M4JgCeczgube21bK/y1Zz7qCcnKyuvK/M4eTk9XN77JEJAT5ckYgJ2/8gFSeucHrP8jfW8mc+9/hfx5bRVGZ5vsTkbajIOjgoqKML4zry2vfmsgNkwbx3NoiJt/xOve8ousPRKRtKAhCxGH9B8O8/oPJ6j8QkTagIAgxfVMTuW/+WB7/2hmkdfauP5j7wDvk5u31uzQRCVEKghDV2H/wG/UfiMhJUhCEsKgo47JA/8GNk07hucD8RXe/sln9ByLSagqCMNA5PoZvXTSEV289jynDenH3Kx8x+c5lPJVbQEOD+g9E5NgUBGGksf/gX9edQffO8dzymNd/sFL9ByJyDAqCMHR6/1SevuEsfnNpNgX7Kpl7/zvcsjCXwn3qPxCRIykIwlRj/8GyQP/B8+t2MvnOZfz25c0crKnzuzwR6UAUBGEuqUn/wfnDenHPqx8x+Y7X1X8gIocoCCJE39REfh/oP+jRxes/mH7vmzzy7idUVNX6XZ6I+EiTzkWghgbH4twC/vTmVjbtrCAxLppZY9KZNz6L7MyufpcnIkFwrEnnFAQRzDnHqh37WLA8j2fXFFJV28DIjGTmj+/H58ek0zn+s1YjFZFQoSCQz1RWWcvTqwpYsDyPTTsrSIqL5vNjMrhyQhYjM1L8Lk9ETpKCQFrNOcfKPO8sYcmaQqrrGsjOTGH++CwuHp1Oks4SREKSgkBOSNnBWhbn5rPgvTw279pP5/gYZo1JZ/6ELEak6yxBJJQoCOSkeGcJe3l0eR7PrSmiuq6B0X27cuX4LGaO7kNinM4SRDo6BYG0mX0Ha1i0soAF7+WxZfd+usTHMDsng/kTshjWJ9nv8kTkKBQE0uacc6z4ZC8Llufx3NoiauoayMnqyrzxWVycnU6nuGi/SxSRJhQEElT7Dtbw5MoCFiz/hI+LD9AlIYa5ORnMn9CPIb27+F2eiKAgkHbinOO9baX88708nl+3k5q6Bk7r141547OYmd2HhFidJYj4RUEg7W7vgRqeXOmNONpafIDkhBjmjs1k/oQsTu2lswSR9qYgEN8451i+rZQFy/NYum4nNfUNjOvXjfkTspg+SmcJIu1FQSAdQumBGp78IJ9/vpfH1j0HSOkUy9yxGVxxepb6EkSCTEEgHYpzjv9sLWHB8jxeXL+T2nrHiPRk5uRkMGtMBj26xPtdokjYURBIh1Wyv5ola4pYtDKf1fllREcZ5wzuztyxmVw4vJeajkTaiIJAQsKW3RUsWlnAU7kFFJZV0SU+hmmjejN3bCbj+6cSFWV+lygSshQEElIaGhzvbith8coCnl9bxIGaejK6dmJOTgZzxmYwqEdnv0sUCTkKAglZlTX1vLRhJ4tWFvDmR8U0OBjdtyuXjM1gZnY6qUlxfpcoEhIUBBIWdpdX8fSqQhblFrCxqJyYKGPS0J7Mzclg8rCexMeoP0HkaBQEEnY2FpWzOLeAxbkFFFdUk9IplpnZfZg7NoOxWd0wU3+CSFMKAglbdfUNvP1xCYtX5rN0/U6qahvol5bI3JxM5uRkkJWW6HeJIh2CgkAiwv7qOl5YW8Ti3AL+s7UE5+D0/t2Yk5PJjFF9SEmM9btEEd8oCCTiFO6r5KlVBSxaWcCW3fuJi4ni/GE9mZuTyXlDehAbHeV3iSLtSkEgEcs5x9qCMhatLOCZ1YWUHqghNSmOz49OZ+7YDEZlpKg/QSKCb0FgZlOBe4Bo4CHn3C+bPX4lcFvg7n7geufc6mO9poJATlRtfQNvbC5m0coCXt64i5q6Bgb1SGLu2Exm52SQ0bWT3yWKBI0vQWBm0cBm4AIgH3gfmOec29BknzOBjc65vWY2Dfixc27CsV5XQSBtoayylufXelNbvL99L2YwYUAqc3MymTaqN10S1J8g4cWvIDgD78B+UeD+7QDOuV8cZf9uwDrnXMaxXldBIG0tr+RgYChqPttLDhIfE8UFw3sxJyeDc09Vf4KEh2MFQUwQ3zcD2NHkfj5wrG/71wAvBLEekRZlpSVy8/mDuWnKKeTu2MdTuQU8u7qQJWuKSEuK4+LR6czJySA7U/0JEp6CGQQt/Y9p8fTDzCbhBcHZR3n8WuBagKysrLaqT+QwZsbYrG6MzerGD2YM5/XNxSzO9VZZe/id7QzskcScMRnMzsmgb6quT5Dw4XvTkJllA4uBac65zZ/1umoakvZWVlnLC2uLWJRbwHvbSgEY3z+V2TkZuj5BQoZffQQxeJ3FU4ACvM7i+c659U32yQL+DVzlnHunNa+rIBA/7Sg9yDOrC1m0Mp+Piw8QFx3FlGE9mZOTwcQhPYmLUX+CdEx+Dh+dDtyNN3z0L865n5nZdQDOuT+Y2UPAJcAngafUHa3QRgoC6Qgar09YHOhP2LO/hq6J3nxHc3IyGZvVVf0J0qHogjKRIKqtb+Ctj/awKLeAl9bvpLqugf5piczOyWBOTgb90pL8LlFEQSDSXiqqalm6budh8x2NzerKnLGZzBzVh25aP0F8oiAQ8UFRWSVPrypk8coCPtxVQWy0MXGI1k8QfygIRHzknGNDUTmLVxbw9OpCiiuqSU6IYUa2d33CuH7dtB6zBJ2CQKSDqG9wvL1lD4tzC1i6bieVtfVkdgusx5yTwUCtxyxBoiAQ6YAOVNcdWo/57S17Dq3HfHF2H2Zmp9M7JcHvEiWMKAhEOrjd5VU8s7qQxbkFrC8sxwxO75fKzNF9mDayDz26xPtdooQ4BYFICNlavJ/n1hTx7JpCNu/aT5TBGYPSmJmdztQRvTXySE6IgkAkRH24s4Ila7wJ8LbtOUBMlHH24O7MzE7nwhG9SNZ02dJKCgKREOecY31hOc+uKWTJ6iIK9lUSFx3FeUN6MDO7D+cP60VSfDDnkJRQpyAQCSPOOVbt2Mezq4t4bm0hu8qrSYiNYsrQXlw8ug8Th/QkIVbXKMjhFAQiYaqhwfH+9lKWrCnihXVF7NlfQ1JcNBcM78XM7HTOObW7LlwTQEEgEhHq6ht4d2spS9YU8sK6nZRV1pKcEMNFI3pz8eh0zhyURoxWW4tYCgKRCFNT18DbW/bw7JpCXlq/i/3VdaQmxTF1ZG8uzk5n/IBUonU1c0RREIhEsKrael7fXMySNUW8smEXlbX19OgSz4xRfbh4dB9y+mqKi0igIBARAA7W1PHvTbt5dnUhr31YTE1dA+kpCcwcnc7M7D6MytC6zOFKQSAiR6ioquWVjbt4dnURb35UTG29o19aIjNG9WH6qD4M75OsM4UwoiAQkWMqO1jLi+t38uyaQt75uIT6BkfXxFjG909lwsA0JgxIZVifZPUrhDAFgYi0Wsn+al77sJjlW0tYvq2UvNKDAHRJiAkEQyoTBqQxIj1Zo5BCiIJARE5Y4b5K3ttWyvJtJby7tZRtew4A0Dk+htP6dTsUDNmZKcQqGDosBYGItJnd5VW8u6300BnDlt37AegUG+0FwwCvOWl03xRdzNaBKAhEJGj27K/2zhgCwbBpZwUA8TFRjM369IwhJ6urpr7wkYJARNrN3gM1vLe9lOVbveakDUXlOAdx0VGM6dv1UDCM7deVxDhNlNdeFAQi4puyylpWbC9leeCsYV1hOfUNjpgoIzsz5dCopHH9U+msGVSDRkEgIh3G/uq6w4JhTX4ZdQ2O6ChjZHryYcGQ0knrLbQVBYGIdFgHa+pY+ck+lm8rYfnWUlbt2EdNfQNRBqf26sLIjBRGpCczIj2F4enJOms4QccKAn2iIuKrxLgYzh7cnbMHdwe8uZFy87xgyM3bx7IPd/PEB/mH9u+flsiIJuEwIj2Z7p21pvPJUBCISIeSEBvNGYPSOGNQGuAtxLO7opr1hWWsLyhnfWE5a/L38dyaokPP6ZUcfygUGgMis1snzZvUSgoCEenQzIxeyQn0Sk5g8tBeh7aXVdayobDcC4jAn8s+3E1DoLU7pVMsw/sEgiHDC4eB3ZN0NXQLFAQiEpJSOsUeduYAXrPSpp0VTcKhnH+8+wnVdQ0AJMRGMaR3MiObNCsN6d0l4q9vUBCISNhIiI1mTN+ujOnb9dC2uvoGPi4+cNiZwzOrC3l0eR4A0VHGKT06MyI9meHpyYzM8DqlkxMiZ8SSRg2JSMRxzpG/t5L1hWWsK/i0eWl3RfWhfbJSEw/1OQzs0ZleyQn0TkmgZ5f4kJxTSaOGRESaMDP6pibSNzWRqSP7HNpe3NgpXVh+qP/hhXU7mz0X0pLi6Z0ST68uCfRKSaB3svfTeLtXcjwpnWJDprNaQSAiEtCjSzwTh/Rk4pCeh7ZVVNWyo7SSXeVV7CyvYmdZFbvKvZ/Csipyd+yj9EDNEa+VEBt1qJO7MRwazyp6B7b3TI7vEBPzKQhERI6hS0Isw9NjGZ6efNR9quvq2V1ezc5AQDSGxc7yanaVVbE6fx87y6oOdVo3lZoUFwiLeHqnJDQLDy84uiUG9+xCQSAicpLiY6IPNTUdjXOOssraZmcVgfAo88421haUsWf/kWcXcdFR9EyO5+oz+/Nf5wxs8/oVBCIi7cDM6JoYR9fEOIb2PvrZRU1dA8X7qz89q2jSFNWjS3CuoFYQiIh0IHExUWR07URG107t9p5BHQNlZlPN7EMz22Jm323hcTOzewOPrzGzscGsR0REjhS0IDCzaOA+YBowHJhnZsOb7TYNGBz4uRZ4IFj1iIhIy4J5RjAe2OKc2+qcqwEWArOa7TML+LvzvAt0NbM+zV9IRESCJ5hBkAHsaHI/P7DtePfBzK41sxVmtqK4uLjNCxURiWTBDIKWBr02n8+iNfvgnHvQOTfOOTeuR48ebVKciIh4ghkE+UDfJvczgcIT2EdERIIomEHwPjDYzAaYWRxwBfBMs32eAa4KjB76HFDmnCtq/kIiIhI8QbuOwDlXZ2Y3Ai8C0cBfnHPrzey6wON/AJ4HpgNbgIPAV4JVj4iItCzkpqE2s2LgkxN8endgTxuWE+r0eRxOn8en9FkcLhw+j37OuRY7WUMuCE6Gma042nzckUifx+H0eXxKn8Xhwv3zCL3VFUREpE0pCEREIlykBcGDfhfQwejzOJw+j0/pszhcWH8eEdVHICIiR4q0MwIREWlGQSAiEuEiJgg+a22ESGJmfc3sNTPbaGbrzexmv2vym5lFm1mumS3xuxa/mVlXM3vCzDYF/o2c4XdNfjGz/wn8H1lnZv80swS/awqGiAiCVq6NEEnqgG8654YBnwNuiPDPA+BmYKPfRXQQ9wBLnXNDgdFE6OdiZhnATcA459xIvBkSrvC3quCIiCCgdWsjRAznXJFzbmXgdgXef/Qjpv+OFGaWCcwAHvK7Fr+ZWTJwLvBnAOdcjXNun69F+SsG6GRmMUAiYTopZqQEQavWPYhEZtYfyAGW+1yKn+4GvgM0+FxHRzAQKAb+Gmgqe8jMkvwuyg/OuQLgDiAPKMKbFPMlf6sKjkgJglatexBpzKwz8CRwi3Ou3O96/GBmM4HdzrkP/K6lg4gBxgIPOOdygANARPapmVk3vJaDAUA6kGRmX/S3quCIlCDQugfNmFksXgg86pxb5Hc9PjoL+LyZbcdrMpxsZo/4W5Kv8oF851zjGeITeMEQic4Htjnnip1ztcAi4EyfawqKSAmC1qyNEDHMzPDagDc65+7yux4/Oedud85lOuf64/27+LdzLiy/9bWGc24nsMPMhgQ2TQE2+FiSn/KAz5lZYuD/zBTCtOM8aOsRdCRHWxvB57L8dBbwJWCtma0KbPuec+55/0qSDuQbwKOBL01bidB1Qpxzy83sCWAl3ki7XMJ0qglNMSEiEuEipWlIRESOQkEgIhLhFAQiIhFOQSAiEuEUBCIiEU5BINKMmdWb2aomP212Za2Z9TezdW31eiJtISKuIxA5TpXOuTF+FyHSXnRGINJKZrbdzH5lZu8Ffk4JbO9nZq+a2ZrAn1mB7b3MbLGZrQ78NE5PEG1mfwrMc/+SmXXy7ZcSQUEg0pJOzZqGLm/yWLlzbjzwe7xZSwnc/rtzLht4FLg3sP1e4HXn3Gi8+Xoar2YfDNznnBsB7AMuCepvI/IZdGWxSDNmtt8517mF7duByc65rYFJ+3Y659LMbA/QxzlXG9he5JzrbmbFQKZzrrrJa/QHXnbODQ7cvw2Idc79tB1+NZEW6YxA5Pi4o9w+2j4tqW5yux711YnPFAQix+fyJn/+J3D7HT5dwvBK4K3A7VeB6+HQmsjJ7VWkyPHQNxGRI3VqMisreOv3Ng4hjTez5XhfouYFtt0E/MXMvo23ulfjbJ03Aw+a2TV43/yvx1vpSqRDUR+BSCsF+gjGOef2+F2LSFtS05CISITTGYGISITTGYGISIRTEIiIRDgFgYhIhFMQiIhEOAWBiEiE+/9MO9qP+OCotwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7pZ5tcAjaIdx"
   },
   "source": [
    "#### Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aDbsgw_5aIdy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 4s 5ms/step - loss: 0.8672 - acc: 0.7548\n",
      "Test accuracy: 0.755, Test loss: 0.867\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test accuracy: %.3f, Test loss: %.3f' % (test_acc,test_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "is6FCRVRaId0"
   },
   "source": [
    "#### Try to build a new model where you stack several recurrent layers.\n",
    "In such a setup, you have to get all of the intermediate layers to return full sequence of outputs. This is needed to return batch size, timesteps, hidden state. By doing this the output should contain all historical generated outputs along with time stamps (3D). This way the next layer can work further on the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eExHNhMraId1"
   },
   "outputs": [],
   "source": [
    "# Build the model. It should be made by at least:\n",
    "# 1 Embedding layer\n",
    "# More than 1 SimpleRNN layer, do not forget to put the return_sequences parameter to True \n",
    "# 1 Dense layer\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, 16))\n",
    "\n",
    "\n",
    "\n",
    "#add SimpleRNN layers here \n",
    "\n",
    "\n",
    "\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xJlV5HynaId_"
   },
   "source": [
    "#### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rWm__m8RaIeC"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "157/157 [==============================] - 22s 142ms/step - loss: 0.6658 - acc: 0.5706 - val_loss: 0.5934 - val_acc: 0.6752\n",
      "Epoch 2/10\n",
      "157/157 [==============================] - 22s 141ms/step - loss: 0.4478 - acc: 0.7955 - val_loss: 0.5051 - val_acc: 0.7882\n",
      "Epoch 3/10\n",
      "157/157 [==============================] - 23s 147ms/step - loss: 0.3519 - acc: 0.8483 - val_loss: 0.4247 - val_acc: 0.8054\n",
      "Epoch 4/10\n",
      "157/157 [==============================] - 23s 148ms/step - loss: 0.2926 - acc: 0.8790 - val_loss: 0.5127 - val_acc: 0.7502\n",
      "Epoch 5/10\n",
      "157/157 [==============================] - 23s 148ms/step - loss: 0.2330 - acc: 0.9065 - val_loss: 0.7823 - val_acc: 0.7684\n",
      "Epoch 6/10\n",
      "157/157 [==============================] - 34s 218ms/step - loss: 0.1723 - acc: 0.9347 - val_loss: 0.5585 - val_acc: 0.7648\n",
      "Epoch 7/10\n",
      "157/157 [==============================] - 35s 222ms/step - loss: 0.1133 - acc: 0.9600 - val_loss: 0.7262 - val_acc: 0.7702\n",
      "Epoch 8/10\n",
      "157/157 [==============================] - 35s 221ms/step - loss: 0.0666 - acc: 0.9768 - val_loss: 0.8503 - val_acc: 0.7504\n",
      "Epoch 9/10\n",
      "157/157 [==============================] - 35s 226ms/step - loss: 0.0370 - acc: 0.9883 - val_loss: 1.0752 - val_acc: 0.7750\n",
      "Epoch 10/10\n",
      "157/157 [==============================] - 81s 518ms/step - loss: 0.0257 - acc: 0.9914 - val_loss: 1.1771 - val_acc: 0.7638\n"
     ]
    }
   ],
   "source": [
    "# Train your model here\n",
    "history = model.fit(x_train, y_train,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SNdh2-0haIeI"
   },
   "outputs": [],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AEdVU-XYaIeL"
   },
   "source": [
    "#### Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RNerI-XraIeM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 36s 46ms/step - loss: 1.1467 - acc: 0.7674\n",
      "Test accuracy: 0.767, Test loss: 1.147\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test accuracy: %.3f, Test loss: %.3f' % (test_acc,test_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qjiFd_M7z7cW"
   },
   "source": [
    "**What can you say about the obtained results? What about the comparison between these results and the ones obtained in the single layer RNN?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XdJA3ttvaIeP"
   },
   "source": [
    "# 3.3 LSTM Network\n",
    "LSTMs are a special kind of recurrent neural network which works, for many tasks, much better than the standard RNNs.<br>\n",
    "These nets are capable of learning long-term dependencies (they are explicitly designed to avoid the long-term dependency problem); remembering information for long periods of time is practically their default behavior.<br><br>\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=1CQ0RkQrnIkr1dFqSU1qHfPh5sJe_A6pQ\" width=\"650px\"><br>\n",
    "\n",
    "RNNs have a very simple structure, such as a single $tanh$ layer.<br>\n",
    "LSTMs also have a chain like structure, but the repeating module has a different structure: instead of having a single neural network layer, there are four, interacting in a very special way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x0KJgSymaIeR"
   },
   "source": [
    "#### Create LSTM model in TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d15KAMKpaIeR"
   },
   "outputs": [],
   "source": [
    "# Build the model. It should be made by at least:\n",
    "# 1 Embedding layer\n",
    "# 1 LSTM layer\n",
    "# 1 Dense layer\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, 16))\n",
    "model.add(tf.keras.layers.LSTM(32))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JPz5KvjAaIeV"
   },
   "source": [
    "#### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "u6QTdfzZaIeW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "157/157 [==============================] - 2s 10ms/step - loss: 0.5461 - acc: 0.7138 - val_loss: 0.4534 - val_acc: 0.7892\n",
      "Epoch 2/10\n",
      "157/157 [==============================] - 1s 8ms/step - loss: 0.3746 - acc: 0.8345 - val_loss: 0.4084 - val_acc: 0.8100\n",
      "Epoch 3/10\n",
      "157/157 [==============================] - 1s 8ms/step - loss: 0.3204 - acc: 0.8648 - val_loss: 0.4127 - val_acc: 0.8064\n",
      "Epoch 4/10\n",
      "157/157 [==============================] - 1s 9ms/step - loss: 0.2910 - acc: 0.8804 - val_loss: 0.5320 - val_acc: 0.7924\n",
      "Epoch 5/10\n",
      "157/157 [==============================] - 1s 8ms/step - loss: 0.2692 - acc: 0.8909 - val_loss: 0.4663 - val_acc: 0.8012\n",
      "Epoch 6/10\n",
      "157/157 [==============================] - 1s 8ms/step - loss: 0.2543 - acc: 0.9006 - val_loss: 0.4709 - val_acc: 0.7896\n",
      "Epoch 7/10\n",
      "157/157 [==============================] - 1s 8ms/step - loss: 0.2425 - acc: 0.9054 - val_loss: 0.4692 - val_acc: 0.7964\n",
      "Epoch 8/10\n",
      "157/157 [==============================] - 2s 10ms/step - loss: 0.2326 - acc: 0.9090 - val_loss: 0.4651 - val_acc: 0.7972\n",
      "Epoch 9/10\n",
      "157/157 [==============================] - 2s 10ms/step - loss: 0.2236 - acc: 0.9136 - val_loss: 0.5206 - val_acc: 0.7882\n",
      "Epoch 10/10\n",
      "157/157 [==============================] - 1s 9ms/step - loss: 0.2158 - acc: 0.9187 - val_loss: 0.4794 - val_acc: 0.7968\n"
     ]
    }
   ],
   "source": [
    "# Train your model here\n",
    "history = model.fit(x_train, y_train,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XELeM5_KaIef"
   },
   "outputs": [],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WPSLRTciaIei"
   },
   "source": [
    "#### Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ovW6sEhNaIej"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 3s 3ms/step - loss: 0.4686 - acc: 0.7993\n",
      "Test accuracy: 0.799, Test loss: 0.469\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test accuracy: %.3f, Test loss: %.3f' % (test_acc,test_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try to stack more LSTM layers together**. Does the classification accuracy improve?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model. It should be made by at least:\n",
    "# 1 Embedding layer\n",
    "# 1 LSTM layer\n",
    "# 1 Dense layer\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, 16))\n",
    "model.add(tf.keras.layers.LSTM(32))\n",
    "\n",
    "\n",
    "#stack more LSTM layers here \n",
    "\n",
    "\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['acc'])\n",
    "\n",
    "# Train your model here\n",
    "history = model.fit(x_train, y_train,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)\n",
    "\n",
    "test_loss, test_acc = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test accuracy: %.3f, Test loss: %.3f' % (test_acc,test_loss))"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "TempNN_complete.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}