RNN_ActionClassify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence Classification using Recurrent Neural Networks(RNN)\n",
    "In this homework, you will learn how to train a recurrent neural network for human action classification. RNN is designed handle sequential data. The network can incorporate both past history and current input. [This](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is a very good tutorial. You should read it before you start."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "**Please make sure you have h5py and torchnet installed**\n",
    "> pip install h5py\n",
    "\n",
    "> pip install git+https://github.com/pytorch/tnt.git@master\n",
    "\n",
    "## Known Windows Issues:\n",
    "### In case you're getting an error [Read more](https://discuss.pytorch.org/t/brokenpipeerror-errno-32-broken-pipe-when-i-run-cifar10-tutorial-py/6224): \n",
    "```python\n",
    "BrokenPipeError: [Errno 32] Broken pipe\n",
    "```\n",
    "\n",
    ">In the dataloader block change Line 39, 42, and 45 num_workers=0 \n",
    "\n",
    "### In case of error (This should be a CUDA error [Read more](https://discuss.pytorch.org/t/asserterror-in-lstm-layer-on-gpu/8698)):\n",
    "\n",
    "```python\n",
    "--> 186             assert param_from.type() == param_to.type()\n",
    "AssertionError: \n",
    "```\n",
    "\n",
    "**Replace following lines:**\n",
    "```python\n",
    "def run_epoch(data_loader, model, criterion, epoch, is_training, optimizer=None):\n",
    "    ...\n",
    "    input_sequence_var = Variable(sequence).type(FloatTensor)\n",
    "    input_label_var = Variable(label).type(LongTensor)\n",
    "    ...\n",
    "```\n",
    "```python\n",
    "def predict_on_test(model, data_loader):\n",
    "    ...\n",
    "        input_sequence_var = Variable(sequence).type(FloatTensor)\n",
    "    ...\n",
    "```\n",
    "** With: **\n",
    "```python\n",
    "def run_epoch(data_loader, model, criterion, epoch, is_training, optimizer=None):\n",
    "    ...\n",
    "    input_sequence_var = Variable(sequence)\n",
    "    input_label_var = Variable(label)\n",
    "    ...\n",
    "```\n",
    "```python\n",
    "def predict_on_test(model, data_loader):\n",
    "    ...\n",
    "        input_sequence_var = Variable(sequence)\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "use cuda: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import h5py\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "import torch.utils.data as DD\n",
    "import torchnet as tnt\n",
    "\n",
    "\n",
    "use_cuda = torch.cuda.is_available()\n",
    "print('use cuda: %s'%(use_cuda))\n",
    "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n",
    "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n",
    "ByteTensor = torch.cuda.ByteTensor if use_cuda else torch.ByteTensor\n",
    "from keras.utils.np_utils import to_categorical\n",
    "import torchvision.models.inception as inception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "The data we are using is skeleton data, which indicates the 3D locations of body joints. In total, there are 25 body joints. It is collected by Kinect v2. To make it easier, each sequence have same number of frames. You need to classify 10 different actions. There are 4000 training sequences, 800 validation sequences, and 1000 test sequences. Each sequence has 15 frames, each frame is a 75-dimension vector (3*25).\n",
    "\n",
    "For your convenience, we provide the dataloader for you.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Dataset(DD.Dataset):\n",
    "    # subset can be: 'train', 'val', 'test'\n",
    "    def __init__(self, data_path, subset='train'):\n",
    "        super(Dataset, self).__init__()\n",
    "        self.data_path = os.path.join(data_path, '%s_data.h5'%subset)\n",
    "        self.subset = subset\n",
    "\n",
    "        with h5py.File(self.data_path) as f:\n",
    "            self.data = np.array(f['data'])\n",
    "\n",
    "        if subset != 'test':\n",
    "            self.label_path = os.path.join(data_path, '%s_label.h5'%subset)\n",
    "            with h5py.File(self.label_path) as f:\n",
    "                self.label = np.array(f['label'])\n",
    "\n",
    "        self.num_sequences = self.data.shape[0]\n",
    "        self.seq_len = self.data.shape[1]\n",
    "        self.n_dim = self.data.shape[2]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        seq = self.data[index]\n",
    "        if self.subset != 'test':\n",
    "            label = int(self.label[index])\n",
    "            sample = {'seq': seq, 'label': label}\n",
    "        else:\n",
    "            sample = {'seq': seq}\n",
    "        return sample\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.num_sequences\n",
    "\n",
    "trSet = Dataset('./data', subset='train')\n",
    "valSet = Dataset('./data', subset='val')\n",
    "tstSet = Dataset('./data', subset='test')\n",
    "\n",
    "batch_size = 100\n",
    "# trLD = DD.DataLoader(DD.ConcatDataset([trSet, valSet]), batch_size=batch_size,\n",
    "#        num_workers=2, pin_memory=False)\n",
    "trLD = DD.DataLoader(trSet, batch_size=batch_size,\n",
    "       sampler=DD.sampler.SequentialSampler(trSet),\n",
    "       num_workers=2, pin_memory=False)\n",
    "valLD = DD.DataLoader(valSet, batch_size=batch_size,\n",
    "       sampler=DD.sampler.SequentialSampler(valSet),\n",
    "       num_workers=1, pin_memory=False)\n",
    "tstLD = DD.DataLoader(tstSet, batch_size=batch_size,\n",
    "       sampler=DD.sampler.SequentialSampler(tstSet),\n",
    "       num_workers=1, pin_memory=False)\n",
    "\n",
    "#trLD = DD.ConcatDataset([trLD, valLD])\n",
    "\n",
    "input_dim = trSet.n_dim\n",
    "\n",
    "num_class = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40\n"
     ]
    }
   ],
   "source": [
    "print(len(trLD))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Flatten(nn.Module):\n",
    "    def forward(self, x):\n",
    "        N, C, H, W = x.size() # read in N, C, H, W\n",
    "        return x.view(N, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Pytorch has implemented different types of recurrent layers for you. For this homework, you can use any type of RNNs as you want:\n",
    "> torch.nn.RNN()\n",
    "\n",
    "> torch.nn.LSTM()\n",
    "\n",
    "> torch.nn.GRU()\n",
    "\n",
    "You can check details for different types of recurrent layers here: [RNN](http://pytorch.org/docs/master/nn.html#torch.nn.RNN), [LSTM]( http://pytorch.org/docs/master/nn.html#torch.nn.LSTM), [GRU](http://pytorch.org/docs/master/nn.html#torch.nn.GRU)\n",
    "\n",
    "\n",
    "### Implement a specific model\n",
    "In this section, you need to implement a model for sequence classification. The model has following layers:\n",
    "* 1 Layer LSTM layer with hidden size of 100, and input size of 75\n",
    "* A linear layer that goes from 100 to num_class (10). \n",
    "\n",
    "An LSTM layer takes an input of size of (batch_size, seq_len, fea_dim) and outputs a variable of shape (batch_size, seq_len, hidden_size). In this homework, the classification score for a sequence is the classification score for the last step of rnn_outputs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "After you have the dataloader and model, you can start training the model. Define a SGD optimizer with learning rate of 1e-2, and a cross-entropy loss function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sequence classification model\n",
    "class SequenceClassify(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SequenceClassify, self).__init__()\n",
    "        \n",
    "        ############## 1st To Do (20 points) ##############\n",
    "        ###################################################\n",
    "        self.recurrent_layer = nn.LSTM(75, 100, 2)\n",
    "        self.project_layer = nn.Linear(100, num_class)\n",
    "        ###################################################\n",
    "    \n",
    "    # the size of input is [batch_size, seq_len(15), input_dim(75)]\n",
    "    # the size of logits is [batch_size, num_class]\n",
    "    def forward(self, input, h_t_1=None, c_t_1=None):\n",
    "        # the size of rnn_outputs is [batch_size, seq_len, rnn_size]\n",
    "        rnn_outputs, (hn, cn) = self.recurrent_layer(input)\n",
    "        \n",
    "        # classify the last step of rnn_outpus\n",
    "        # the size of logits is [batch_size, num_class]\n",
    "        logits = self.project_layer(rnn_outputs[:,-1])\n",
    "        return logits\n",
    "\n",
    "model = SequenceClassify()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "################ 2nd To Do  (5 points)##################\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = 1e-2)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "global val_loss, val_accuracy, train_loss, train_accuracy\n",
    "val_loss = []\n",
    "val_accuracy = []\n",
    "train_loss = []\n",
    "train_accuracy = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train Epoch: 0  , Loss: 2.3041,  Accuracy: 9.95\n",
      "val Epoch: 0  , Loss: 2.3045,  Accuracy: 9.38\n"
     ]
    }
   ],
   "source": [
    "# run the model for one epoch\n",
    "# can be used for both training or validation model\n",
    "\n",
    "\n",
    "def run_epoch(data_loader, model, criterion, epoch, is_training, optimizer=None):\n",
    "    if is_training:\n",
    "        model.train()\n",
    "        logger_prefix = 'train'\n",
    "    else:\n",
    "        model.eval()\n",
    "        logger_prefix = 'val'\n",
    "\n",
    "    confusion_matrix = tnt.meter.ConfusionMeter(num_class)\n",
    "    acc = tnt.meter.ClassErrorMeter(accuracy=True)\n",
    "    meter_loss = tnt.meter.AverageValueMeter()\n",
    "\n",
    "    for batch_idx, sample in enumerate(data_loader):\n",
    "        sequence = sample['seq']\n",
    "        label = sample['label']\n",
    "        input_sequence_var = Variable(sequence).type(FloatTensor)\n",
    "        input_label_var = Variable(label).type(LongTensor)\n",
    "        \n",
    "\n",
    "        # compute output\n",
    "        # output_logits: [batch_size, num_class]\n",
    "        output_logits = model(input_sequence_var)\n",
    "        loss = criterion(output_logits, input_label_var)\n",
    "\n",
    "        if is_training:\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        meter_loss.add(loss.data[0])\n",
    "        acc.add(output_logits.data, input_label_var.data)\n",
    "        confusion_matrix.add(output_logits.data, input_label_var.data)\n",
    "        \n",
    "    global max_accu, val_loss, val_accuracy, train_loss, train_accuracy\n",
    "    \n",
    "    print('%s Epoch: %d  , Loss: %.4f,  Accuracy: %.2f'%(logger_prefix, epoch, meter_loss.value()[0], acc.value()[0]))\n",
    "    if logger_prefix == 'val':\n",
    "        val_loss.append(meter_loss.value()[0])\n",
    "        val_accuracy.append(acc.value()[0])\n",
    "    else:\n",
    "        train_loss.append(meter_loss.value()[0])\n",
    "        train_accuracy.append(acc.value()[0])\n",
    "    \n",
    "            \n",
    "    return acc.value()[0]\n",
    "\n",
    "num_epochs = 1\n",
    "evaluate_every_epoch = 1\n",
    "for e in range(num_epochs):\n",
    "    run_epoch(trLD, model, criterion, e, True, optimizer)\n",
    "    if e % evaluate_every_epoch == 0:\n",
    "        run_epoch(valLD, model, criterion, e, False, None)   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conv2D -> maxPool2d -> LeakyRelu -> BatchNorm2d -> inception -> conv2d -> maxPool2d -> leakyrely -> batchnorm2d -> inception -> flatten -> linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"model = nn.Sequential()\\nmodel.add(LSTM(256, input_shape=(75, 1), activation='relu'))\\nmodel.add(Dropout(0.02))\\nmodel.add(Dense(2, activation='sigmoid'))\\nmodel.compile(loss='Cross_Entropy', optimizer='adam', metrics=['accuracy'])\""
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sequence classification model\n",
    "import torchvision.models.inception as inception\n",
    "class SequenceClassify(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SequenceClassify, self).__init__()\n",
    "        \n",
    "        ############## 1st To Do (20 points) ##############\n",
    "        ###################################################\n",
    "\n",
    "        self.convolution = nn.Sequential(\n",
    "            nn.Conv2d(1, 250, kernel_size = (3, 5), stride = 1),\n",
    "            nn.MaxPool2d(2, stride = 2),\n",
    "            nn.LeakyReLU(negative_slope = 0.4),\n",
    "            nn.BatchNorm2d(250),\n",
    "            inception.InceptionA(250, 150),\n",
    "            nn.Conv2d(374, 150, stride = 1, kernel_size = 2),\n",
    "            nn.MaxPool2d(kernel_size = 2),\n",
    "            nn.LeakyReLU(negative_slope=0.3),\n",
    "            nn.BatchNorm2d(150),\n",
    "            \n",
    "            inception.InceptionA(150, 20),\n",
    "#             nn.Conv2d(150, 100, kernel_size = (2, 1), stride = 1),\n",
    "#             nn.LeakyReLU(negative_slope= 0.4),\n",
    "#             nn.BatchNorm1d(100),\n",
    "#             nn.MaxPool2d(kernel_size = 2, stride= 1),\n",
    "            Flatten(),\n",
    "            nn.Linear(8296, 100)\n",
    "        )\n",
    "        self.recurrent_layer = nn.LSTM(100, 50, 1)\n",
    "        self.project_layer = nn.Linear(50, 10)\n",
    "        ###################################################\n",
    "    \n",
    "    # the size of input is [batch_size, seq_len(15), input_dim(75)]\n",
    "    # the size of logits is [batch_size, num_class]\n",
    "    def forward(self, input1, h_t_1=None, c_t_1=None):\n",
    "        # the size of rnn_outputs is [batch_size, seq_len, rnn_size]\n",
    "        convolution_input = input1.view(100, 1, 15, 75)\n",
    "        convolution_output = self.convolution(convolution_input)\n",
    "        lstm_input = convolution_output.view(100, 1, 100)\n",
    "        rnn_outputs, (hn, cn) = self.recurrent_layer(lstm_input)        \n",
    "        # classify the last step of rnn_outpus\n",
    "        # the size of logits is [batch_size, num_class]\n",
    "        logits = self.project_layer(rnn_outputs[:,-1])\n",
    "        return logits\n",
    "    \n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "model = SequenceClassify()\n",
    "\n",
    "\"\"\"model = nn.Sequential()\n",
    "model.add(LSTM(256, input_shape=(75, 1), activation='relu'))\n",
    "model.add(Dropout(0.02))\n",
    "model.add(Dense(2, activation='sigmoid'))\n",
    "model.compile(loss='Cross_Entropy', optimizer='adam', metrics=['accuracy'])\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "################ 2nd To Do  (5 points)##################\n",
    "# optimizer = torch.optim.SGD(model.parameters(), lr = 1e-3)\n",
    "optimizer = torch.optim.Adadelta(model.parameters(), lr = 1e-1)\n",
    "# optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
    "criterion = nn.CrossEntropyLoss()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train Epoch: 0  , Loss: 2.1440,  Accuracy: 23.72\n",
      "val Epoch: 0  , Loss: 2.1733,  Accuracy: 20.25\n",
      "train Epoch: 1  , Loss: 1.6642,  Accuracy: 50.05\n",
      "val Epoch: 1  , Loss: 1.3901,  Accuracy: 59.88\n",
      "train Epoch: 2  , Loss: 1.2341,  Accuracy: 65.38\n",
      "val Epoch: 2  , Loss: 1.0593,  Accuracy: 69.88\n",
      "train Epoch: 3  , Loss: 0.9725,  Accuracy: 72.15\n",
      "val Epoch: 3  , Loss: 0.8932,  Accuracy: 73.88\n",
      "train Epoch: 4  , Loss: 0.8133,  Accuracy: 76.28\n",
      "val Epoch: 4  , Loss: 0.8001,  Accuracy: 75.38\n",
      "train Epoch: 5  , Loss: 0.7058,  Accuracy: 79.80\n",
      "val Epoch: 5  , Loss: 0.7534,  Accuracy: 76.75\n",
      "train Epoch: 6  , Loss: 0.6203,  Accuracy: 82.53\n",
      "val Epoch: 6  , Loss: 0.6688,  Accuracy: 78.25\n",
      "train Epoch: 7  , Loss: 0.5513,  Accuracy: 84.17\n",
      "val Epoch: 7  , Loss: 0.6119,  Accuracy: 80.62\n",
      "train Epoch: 8  , Loss: 0.4890,  Accuracy: 86.62\n",
      "val Epoch: 8  , Loss: 0.5912,  Accuracy: 81.25\n",
      "train Epoch: 9  , Loss: 0.4295,  Accuracy: 88.98\n",
      "val Epoch: 9  , Loss: 0.5655,  Accuracy: 82.00\n",
      "train Epoch: 10  , Loss: 0.3789,  Accuracy: 90.20\n",
      "val Epoch: 10  , Loss: 0.5306,  Accuracy: 83.00\n",
      "train Epoch: 11  , Loss: 0.3363,  Accuracy: 91.85\n",
      "val Epoch: 11  , Loss: 0.4905,  Accuracy: 84.50\n",
      "train Epoch: 12  , Loss: 0.2939,  Accuracy: 92.80\n",
      "val Epoch: 12  , Loss: 0.4821,  Accuracy: 84.88\n",
      "train Epoch: 13  , Loss: 0.2399,  Accuracy: 94.77\n",
      "val Epoch: 13  , Loss: 0.4988,  Accuracy: 84.12\n",
      "train Epoch: 14  , Loss: 0.2199,  Accuracy: 94.88\n",
      "val Epoch: 14  , Loss: 0.4417,  Accuracy: 86.62\n",
      "86.625_0.441693529487\n",
      "train Epoch: 15  , Loss: 0.1906,  Accuracy: 95.85\n",
      "val Epoch: 15  , Loss: 0.4499,  Accuracy: 84.88\n",
      "train Epoch: 16  , Loss: 0.1556,  Accuracy: 97.08\n",
      "val Epoch: 16  , Loss: 0.4561,  Accuracy: 85.75\n",
      "train Epoch: 17  , Loss: 0.1520,  Accuracy: 96.67\n",
      "val Epoch: 17  , Loss: 0.4637,  Accuracy: 84.25\n",
      "train Epoch: 18  , Loss: 0.1237,  Accuracy: 97.67\n",
      "val Epoch: 18  , Loss: 0.4804,  Accuracy: 85.38\n",
      "train Epoch: 19  , Loss: 0.1042,  Accuracy: 98.12\n",
      "val Epoch: 19  , Loss: 0.4443,  Accuracy: 87.12\n",
      "87.125_0.444309465587\n",
      "train Epoch: 20  , Loss: 0.0788,  Accuracy: 98.83\n",
      "val Epoch: 20  , Loss: 0.4421,  Accuracy: 86.38\n",
      "train Epoch: 21  , Loss: 0.0780,  Accuracy: 98.52\n",
      "val Epoch: 21  , Loss: 0.4935,  Accuracy: 85.38\n",
      "train Epoch: 22  , Loss: 0.0809,  Accuracy: 98.35\n",
      "val Epoch: 22  , Loss: 0.5347,  Accuracy: 83.38\n",
      "train Epoch: 23  , Loss: 0.0453,  Accuracy: 99.52\n",
      "val Epoch: 23  , Loss: 0.5324,  Accuracy: 84.88\n",
      "train Epoch: 24  , Loss: 0.0464,  Accuracy: 99.33\n",
      "val Epoch: 24  , Loss: 0.4856,  Accuracy: 85.75\n",
      "train Epoch: 25  , Loss: 0.0246,  Accuracy: 99.92\n",
      "val Epoch: 25  , Loss: 0.4878,  Accuracy: 86.50\n",
      "train Epoch: 26  , Loss: 0.0535,  Accuracy: 98.80\n",
      "val Epoch: 26  , Loss: 0.5391,  Accuracy: 84.00\n",
      "train Epoch: 27  , Loss: 0.0439,  Accuracy: 99.28\n",
      "val Epoch: 27  , Loss: 0.5311,  Accuracy: 85.62\n",
      "train Epoch: 28  , Loss: 0.0432,  Accuracy: 99.17\n",
      "val Epoch: 28  , Loss: 0.5950,  Accuracy: 83.75\n",
      "train Epoch: 29  , Loss: 0.0481,  Accuracy: 98.90\n",
      "val Epoch: 29  , Loss: 0.5262,  Accuracy: 85.62\n",
      "train Epoch: 30  , Loss: 0.0378,  Accuracy: 99.28\n",
      "val Epoch: 30  , Loss: 0.5309,  Accuracy: 85.75\n",
      "train Epoch: 31  , Loss: 0.0172,  Accuracy: 99.88\n",
      "val Epoch: 31  , Loss: 0.5044,  Accuracy: 87.38\n",
      "87.375_0.504404272884\n",
      "train Epoch: 32  , Loss: 0.0111,  Accuracy: 100.00\n",
      "val Epoch: 32  , Loss: 0.4993,  Accuracy: 86.75\n",
      "train Epoch: 33  , Loss: 0.0321,  Accuracy: 99.42\n",
      "val Epoch: 33  , Loss: 0.5290,  Accuracy: 86.38\n",
      "train Epoch: 34  , Loss: 0.0158,  Accuracy: 99.85\n",
      "val Epoch: 34  , Loss: 0.5401,  Accuracy: 86.38\n",
      "train Epoch: 35  , Loss: 0.0102,  Accuracy: 99.98\n",
      "val Epoch: 35  , Loss: 0.5406,  Accuracy: 86.00\n",
      "train Epoch: 36  , Loss: 0.0077,  Accuracy: 100.00\n",
      "val Epoch: 36  , Loss: 0.5394,  Accuracy: 86.50\n",
      "train Epoch: 37  , Loss: 0.0067,  Accuracy: 100.00\n",
      "val Epoch: 37  , Loss: 0.5379,  Accuracy: 86.50\n",
      "train Epoch: 38  , Loss: 0.0061,  Accuracy: 100.00\n",
      "val Epoch: 38  , Loss: 0.5400,  Accuracy: 86.62\n",
      "train Epoch: 39  , Loss: 0.0057,  Accuracy: 100.00\n",
      "val Epoch: 39  , Loss: 0.5421,  Accuracy: 87.12\n",
      "train Epoch: 40  , Loss: 0.0053,  Accuracy: 100.00\n",
      "val Epoch: 40  , Loss: 0.5439,  Accuracy: 87.00\n",
      "train Epoch: 41  , Loss: 0.0050,  Accuracy: 100.00\n",
      "val Epoch: 41  , Loss: 0.5462,  Accuracy: 87.12\n",
      "train Epoch: 42  , Loss: 0.0047,  Accuracy: 100.00\n",
      "val Epoch: 42  , Loss: 0.5470,  Accuracy: 87.00\n",
      "train Epoch: 43  , Loss: 0.0045,  Accuracy: 100.00\n",
      "val Epoch: 43  , Loss: 0.5484,  Accuracy: 87.12\n",
      "train Epoch: 44  , Loss: 0.0043,  Accuracy: 100.00\n",
      "val Epoch: 44  , Loss: 0.5503,  Accuracy: 87.38\n",
      "train Epoch: 45  , Loss: 0.0041,  Accuracy: 100.00\n",
      "val Epoch: 45  , Loss: 0.5513,  Accuracy: 87.38\n",
      "train Epoch: 46  , Loss: 0.0039,  Accuracy: 100.00\n",
      "val Epoch: 46  , Loss: 0.5526,  Accuracy: 87.25\n",
      "train Epoch: 47  , Loss: 0.0038,  Accuracy: 100.00\n",
      "val Epoch: 47  , Loss: 0.5558,  Accuracy: 87.00\n",
      "train Epoch: 48  , Loss: 0.0036,  Accuracy: 100.00\n",
      "val Epoch: 48  , Loss: 0.5572,  Accuracy: 87.00\n",
      "train Epoch: 49  , Loss: 0.0035,  Accuracy: 100.00\n",
      "val Epoch: 49  , Loss: 0.5591,  Accuracy: 87.00\n",
      "train Epoch: 50  , Loss: 0.0034,  Accuracy: 100.00\n",
      "val Epoch: 50  , Loss: 0.5605,  Accuracy: 87.00\n"
     ]
    }
   ],
   "source": [
    "# run the model for one epoch\n",
    "# can be used for both training or validation model\n",
    "\n",
    "\n",
    "def run_epoch(data_loader, model, criterion, epoch, is_training, optimizer=None):\n",
    "    if is_training:\n",
    "        model.train()\n",
    "        logger_prefix = 'train'\n",
    "    else:\n",
    "        model.eval()\n",
    "        logger_prefix = 'val'\n",
    "\n",
    "    confusion_matrix = tnt.meter.ConfusionMeter(num_class)\n",
    "    acc = tnt.meter.ClassErrorMeter(accuracy=True)\n",
    "    meter_loss = tnt.meter.AverageValueMeter()\n",
    "\n",
    "    for batch_idx, sample in enumerate(data_loader):\n",
    "        sequence = sample['seq']\n",
    "        label = sample['label']\n",
    "        input_sequence_var = Variable(sequence).type(FloatTensor)\n",
    "        input_label_var = Variable(label).type(LongTensor)\n",
    "        \n",
    "\n",
    "        # compute output\n",
    "        # output_logits: [batch_size, num_class]\n",
    "        output_logits = model(input_sequence_var)\n",
    "        loss = criterion(output_logits, input_label_var)\n",
    "\n",
    "        if is_training:\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        meter_loss.add(loss.data[0])\n",
    "        acc.add(output_logits.data, input_label_var.data)\n",
    "        confusion_matrix.add(output_logits.data, input_label_var.data)\n",
    "        \n",
    "    return acc.value()[0]\n",
    "\n",
    "num_epochs = 51\n",
    "evaluate_every_epoch = 1\n",
    "for e in range(num_epochs):\n",
    "    run_epoch(trLD, model, criterion, e, True, optimizer)\n",
    "    if e % evaluate_every_epoch == 0:\n",
    "        run_epoch(valLD, model, criterion, e, False, None)   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOW5wPHfM5ONLGQPELZAUCFsIUQEFRVXQCtFEUXc0JZqvdVrq1fqtdV69dbbWrXWpVWLGwp6VZRbUbRIFQRlk1VAICQQEsgGCUnINnnvH+dkGDCEQDI5Seb5fjyfmTlz5sxzyHie865HjDEopZRSAC6nA1BKKdV+aFJQSinlpUlBKaWUlyYFpZRSXpoUlFJKeWlSUEop5aVJQSmllJcmBaWOQ0SyReRip+NQqi1pUlBKKeWlSUGpkyQiPxWRHSJSIiILRCTZXi8i8pSIFIhImYhsFJEh9nsTReQ7ETkkIntF5F5nj0KpxmlSUOokiMiFwO+BqUAPIAeYZ799KXAecDoQbW9TbL/3d+BnxpgoYAjweRuGrVSzBTkdgFIdzHRgtjFmLYCI/Bo4ICIpQC0QBQwEVhpjtvh8rhZIE5H1xpgDwIE2jVqpZtKSglInJxmrdACAMaYcqzTQ0xjzOfAs8BxQICIvikhXe9OrgYlAjoh8ISJj2jhupZpFk4JSJycP6NvwQkQigHhgL4Ax5hljzEggDasa6T57/SpjzCQgCfgAeKeN41aqWTQpKNW0YBEJa1iAucAMEUkXkVDgv4FvjDHZInKmiJwlIsFABVAF1ItIiIhMF5FoY0wtUAbUO3ZESjVBk4JSTVsIHPZZLgB+A7wH5AOpwHX2tl2Bl7DaC3KwqpX+aL93I5AtImXA7VhtE0q1O6I32VFKKdVASwpKKaW8NCkopZTy0qSglFLKS5OCUkoprw43ojkhIcGkpKQ4HYZSSnUoa9asKTLGJJ5ouw6XFFJSUli9erXTYSilVIciIjkn3kqrj5RSSvnQpKCUUspLk4JSSimvDtemoJRqG7W1teTm5lJVVeV0KOokhIWF0atXL4KDg0/p85oUlFKNys3NJSoqipSUFETE6XBUMxhjKC4uJjc3l379+p3SPrT6SCnVqKqqKuLj4zUhdCAiQnx8fItKd5oUlFLHpQmh42np3yxwksL+zbD4EagscToSpZRqtwInKZRkwdI/QekepyNRSp1AcXEx6enppKen0717d3r27Ol9XVNT06x9zJgxg23btjW5zXPPPcebb77ZGiFz7rnnsm7dulbZl5MCp6E5wh7dXVHobBxKqROKj4/3nmAffvhhIiMjuffee4/axhiDMQaXq/Fr21deeeWE33PnnXe2PNhOJmBKCiv2W/VsJYV5DkeilDpVO3bsIC0tjenTpzN48GDy8/OZOXMmmZmZDB48mEceecS7bcOVe11dHTExMcyaNYvhw4czZswYCgoKAHjwwQd5+umnvdvPmjWLUaNGccYZZ7B8+XIAKioquPrqq0lLS2PKlClkZmY2u0Rw+PBhbr75ZoYOHUpGRgZffvklABs3buTMM88kPT2dYcOGkZWVxaFDh5gwYQLDhw9nyJAhvPvuu635T9dsAVNSkIgkAEoL84lzOBalOprf/d9mvssra9V9piV35aEfDT7pz23dupXXX3+dzMxMAB5//HHi4uKoq6tj3LhxTJkyhbS0tKM+U1payvnnn8/jjz/OL3/5S2bPns2sWbN+sG9jDCtXrmTBggU88sgjfPLJJ/zlL3+he/fuvPfee6xfv56MjIxmx/rMM88QGhrKxo0b2bx5MxMnTmT79u08//zz3HvvvVx77bVUV1djjOHDDz8kJSWFjz/+2BuzEwKmpNCrexI1xk3lwX1Oh6KUaoHU1FRvQgCYO3cuGRkZZGRksGXLFr777rsffKZLly5MmDABgJEjR5Kdnd3ovq+66qofbLNs2TKuu866Dffw4cMZPLj5iWzZsmXccMMNAAwePJjk5GR27NjB2WefzaOPPsof/vAH9uzZQ1hYGMOGDeOTTz5h1qxZfPXVV0RHRzf7e1pTwJQUesSEU0g0dYcKnA5FqQ7nVK7o/SUiIsL7fPv27fz5z39m5cqVxMTEcMMNNzTaRz8kJMT73O12U1dX1+i+Q0NDT7hNa7jxxhsZM2YMH330EePHj2f27Nmcd955rF69moULFzJr1iwmTJjAAw884LcYjidgSgpul3DIHYNUFDkdilKqlZSVlREVFUXXrl3Jz89n0aJFrf4d55xzDu+88w5gtQU0VhI5nrFjx3p7N23ZsoX8/HwGDBhAVlYWAwYM4O677+aKK65gw4YN7N27l8jISG688UZ+9atfsXbt2lY/luYImJICQFVIHKHVxU6HoZRqJRkZGaSlpTFw4ED69u3LOeec0+rf8Ytf/IKbbrqJtLQ073K8qp3LLrvMO+fQ2LFjmT17Nj/72c8YOnQowcHBvP7664SEhPDWW28xd+5cgoODSU5O5uGHH2b58uXMmjULl8tFSEgIf/3rX1v9WJpDjDGOfPGpyszMNKd6k531z1xLQvEakh/eriM1lTqBLVu2MGjQIKfDcFxdXR11dXWEhYWxfft2Lr30UrZv305QUPu9pm7sbycia4wxmcf5iFf7PSo/cEclEVtcRmF5NUlRYU6Ho5TqAMrLy7nooouoq6vDGMPf/va3dp0QWqrzHlkjusQkES7VbN1XRFJUL6fDUUp1ADExMaxZs8bpMNpMwDQ0A0TFJwOwb1+uw5EopVT7FFBJITbRSgoHCvY6HIlSSrVPfksKItJbRJaIyHcisllE7m5kGxGRZ0Rkh4hsEJHmDxU8BcFdrVHNFSU6gE0ppRrjzzaFOuBXxpi1IhIFrBGRz4wxvp18JwCn2ctZwAv2o3/Yk+JVl2pSUEqpxvitpGCMyTfGrLWfHwK2AD2P2WwS8LqxfA3EiEgPf8VEeAIA9eU6gE2p9mzcuHE/GIj29NNPc8cddzT5ucjISADy8vKYMmVKo9tccMEFnKhb+9NPP01lZaX39cSJEzl48GBzQm/Sww8/zBNPPNHi/fhTm7QpiEgKMAL45pi3egK+NzjI5YeJAxGZKSKrRWR1YWELpr4OCafGHU5E3QEOVjZvTnalVNubNm0a8+bNO2rdvHnzmDZtWrM+n5yc3KJZRo9NCgsXLiQmJuaU99eR+D0piEgk8B7w78aYU5pm0RjzojEm0xiTmZiY2KJ46sLiiJcycoorT7yxUsoRU6ZM4aOPPvLeUCc7O5u8vDzGjh3rHTeQkZHB0KFD+fDDD3/w+ezsbIYMGQJY01dfd911DBo0iMmTJ3P48GHvdnfccYd32u2HHnoIsGY2zcvLY9y4cYwbNw6AlJQUioqsGoYnn3ySIUOGMGTIEO+029nZ2QwaNIif/vSnDB48mEsvvfSo7zmRxvZZUVHB5Zdf7p1K++233wZg1qxZpKWlMWzYsB/cY6I1+HWcgogEYyWEN40x7zeyyV6gt8/rXvY6/8UUkUh8WRk5JZUM7x0YmV+pFvt4Fuzb2Lr77D4UJjze6FtxcXGMGjWKjz/+mEmTJjFv3jymTp2KiBAWFsb8+fPp2rUrRUVFjB49miuvvPK4sxS88MILhIeHs2XLFjZs2HDU1NePPfYYcXFxeDweLrroIjZs2MBdd93Fk08+yZIlS0hISDhqX2vWrOGVV17hm2++wRjDWWedxfnnn09sbCzbt29n7ty5vPTSS0ydOpX33nvPO0NqU463z6ysLJKTk/noo48Aayrt4uJi5s+fz9atWxGRVqnSOpY/ex8J8HdgizHmyeNstgC4ye6FNBooNcbk+ysmgJDobiRIGTlFFf78GqVUC/lWIflWHRljeOCBBxg2bBgXX3wxe/fuZf/+/cfdz5dffuk9OQ8bNoxhw4Z533vnnXfIyMhgxIgRbN68+YST3S1btozJkycTERFBZGQkV111FUuXLgWgX79+pKenA01Pz93cfQ4dOpTPPvuM+++/n6VLlxIdHU10dDRhYWHcdtttvP/++4SHhzfrO06GP0sK5wA3AhtFpOE2RQ8AfQCMMX8FFgITgR1AJTDDj/EA4I5MJNH1DdlafaRU8x3nit6fJk2axD333MPatWuprKxk5MiRALz55psUFhayZs0agoODSUlJaXS67BPZtWsXTzzxBKtWrSI2NpZbbrnllPbToGHabbCm3j6Z6qPGnH766axdu5aFCxfy4IMPctFFF/Hb3/6WlStXsnjxYt59912effZZPv/88xZ9z7H82ftomTFGjDHDjDHp9rLQGPNXOyFg9zq60xiTaowZaow5tZnuTkZEIrGUsbu43O9fpZQ6dZGRkYwbN45bb731qAbm0tJSkpKSCA4OZsmSJeTk5DS5n/POO4+33noLgE2bNrFhwwbAmnY7IiKC6Oho9u/f773jGUBUVBSHDh36wb7Gjh3LBx98QGVlJRUVFcyfP5+xY8e26DiPt8+8vDzCw8O54YYbuO+++1i7di3l5eWUlpYyceJEnnrqKdavX9+i725MQM19BEBEIkF4KC5uQS8mpVSbmDZtGpMnTz6qJ9L06dP50Y9+xNChQ8nMzGTgwIFN7uOOO+5gxowZDBo0iEGDBnlLHMOHD2fEiBEMHDiQ3r17HzXt9syZMxk/fjzJycksWbLEuz4jI4NbbrmFUaNGAfCTn/yEESNGNLuqCODRRx/1NiYD5ObmNrrPRYsWcd999+FyuQgODuaFF17g0KFDTJo0iaqqKowxPPnk8WrmT11ATZ0NwIZ34P2fcmH1E/zfwzOICA28vKhUc+jU2R1XS6bODqi5jwCIsHoTxFPG7hJtV1BKKV8BmBSscQ7WWAXtgaSUUr4CNikkSKn2QFLqBDpa9bJq+d8s8JJCeDwAvUIqdFSzUk0ICwujuLhYE0MHYoyhuLiYsLBTv7Nk4LWyuoOhSyx9TAVfavWRUsfVq1cvcnNzadF8Y6rNhYWF0avXqd9ZMvCSAkBEIj2qtaSgVFOCg4Pp16+f02GoNhZ41UcA4QkkSBl5pYeprvM4HY1SSrUbgZkUIhKIrj+IMbCnpGVD0ZVSqjMJ0KSQSHjtAQB2l2i7glJKNQjYpBBUfQA3HrKLtF1BKaUaBGhSsEY19wo9rAPYlFLKR4AmBWsA2+DoGnJ0qgullPIK0KRglRQGRlVpt1SllPIRoEnBKimkdKlkT0kldZ56hwNSSqn2IaCTQq+QCurqDfmlp363JaWU6kwCMymExYC46ea27qyUrY3NSikFBGpScLkgIoFYUwqg7QpKKWULzKQAEJFIl5oSQoNc2i1VKaVsgZsUwuORymL6xofrfRWUUsoWuEkhIhEqCukTF8FuTQpKKQUEfFIoIiU+nJySCurr9UYiSikVwEkhAWoO0T/GRVVtPQWHqp2OSCmlHBfAScEaq5AaYSUD7ZaqlFKaFEjpYiWDrEJNCkopFfBJIVHKCQt2sbOw3OGAlFLKeQGcFOIBcB0uon9CJFmaFJRSKpCTglVSoKKQ/okR7NTqI6WUCuCkEBIJQWFQUUhqYiR7DlRSVetxOiqllHJU4CYFEe9YhdSkSIzRHkhKKRW4SQGssQoVhaQmRgCws0CTglIqsAV4UrCmuuiXYCUFbWxWSgW6wE4K4QlQUUx4SBA9Y7pot1SlVMAL7KRgVx9hjPZAUkopAj4pJIKnGqoPkZpojVUwRifGU0oFLr8lBRGZLSIFIrLpOO9fICKlIrLOXn7rr1iOy2esQmpSJBU1HvaX6cR4SqnA5c+SwqvA+BNss9QYk24vj/gxlsZ5k0IRqXZjs7YrKKUCmd+SgjHmS6DEX/tvFREJ1qNdUgBNCkqpwOZ0m8IYEVkvIh+LyOA2//aGkkJlEUlRoUSGBrGzQJOCUipwBTn43WuBvsaYchGZCHwAnNbYhiIyE5gJ0KdPn9aLwKekICKkag8kpVSAc6ykYIwpM8aU288XAsEiknCcbV80xmQaYzITExNbL4igUAjtChVFAN4eSEopFagcSwoi0l1ExH4+yo6luM0DaRirAPRPjCCvtIqK6ro2D0MppdoDv1Ufichc4AIgQURygYeAYABjzF+BKcAdIlIHHAauM04MErCnugCrpACwq6iCIT2j2zwUpZRymt+SgjFm2gnefxZ41l/f32wRiVCSBXBUDyRNCkqpQOR07yPn+VQf9Y0PxyVoDySlVMDSpBCeAJXFUF9PaJCb3nHh7CzSHkhKqcCkSSEiEUw9HD4AWO0KWlJQSgUqTQo+YxUAUhMj2FVUgadeJ8ZTSgUeTQqRSdbjoTzAKilU19WTd/Cwg0EppZQzNCkkpVmP+6zJXHUOJKVUINOkEJEA0X0g71sA+ntnS9XGZqVU4NGkAJA83JsU4iJCiAkP1pKCUiogaVIASB4BB3bB4QP2xHjaA0kpFZg0KYCVFADy1wPobKlKqYClSQGgR7r12NCukBhJUXk1pYdrHQxKKaXaniYFgPA4iOkLeeuAIxPj6TTaSqlAo0mhQfIIb0khNVF7ICmlApMmhQbJI+BgDlSW0DsunGC3aA8kpVTA0aTQINluV8hfR7DbRd/4CK0+UkoFHE0KDXoMtx59BrFp9ZFSKtBoUmjQJRbi+h9pV0iKJKe4glpPvcOBKaVU29Gk4Ct5xFE9kGo9hj0llQ4HpZRSbUeTgq8e6VC6ByqKvD2QtuvIZqVUANGk4KthZHPeOgb16EqI28WanAPOxqSUUm1Ik4Ivn8bmsGA3I/rEsHxnkbMxKaVUG9Kk4CusK8SfBvlWu8KY1Hg255VRWqnTXSilAoMmhWMlp3t7IJ2dmoAx8M2uYoeDUkqptqFJ4VjJI6BsL5QXMLx3NGHBLpbv1KSglAoMzUoKIpIqIqH28wtE5C4RifFvaA7xaWwODXJzZkocX2dpUlBKBYbmlhTeAzwiMgB4EegNvOW3qJzUfRgg3iqk0f3j2brvEMXl1c7GpZRSbaC5SaHeGFMHTAb+Yoy5D+jhv7AcFBoJCad7k8KY1HgAvs4qcTIqpZRqE81NCrUiMg24GfiHvS7YPyG1Az7TaA/rGU1kaJB2TVVKBYTmJoUZwBjgMWPMLhHpB7zhv7AcljwCyvdBWT5BbhdnpsSyQtsVlFIBoFlJwRjznTHmLmPMXBGJBaKMMf/j59ic4zONNlhVSFmFFewvq3IwKKWU8r/m9j76l4h0FZE4YC3wkog86d/QHNR9KIjrqPEKACu0a6pSqpNrbvVRtDGmDLgKeN0YcxZwsf/CclhIBCQO9M6YOqhHV7qGBWlSUEp1es1NCkEi0gOYypGG5s6tobHZGNwuYXT/eJZnaWOzUqpza25SeARYBOw0xqwSkf7Adv+F1Q70SIeKAijLA6x2hT0lh8k9oPdXUEp1Xs1taP5fY8wwY8wd9ussY8zV/g3NYQ0jm/d8AxwZr6BVSEqpzqy5Dc29RGS+iBTYy3si0svfwTkqeQREJcO6NwE4PSmK+IgQTQpKqU6tudVHrwALgGR7+T97XeflDoKMG2HHYjiQg8tuV1iRVYwxxunolFLKL5qbFBKNMa8YY+rs5VUgsakPiMhsu1Sx6Tjvi4g8IyI7RGSDiGScZOz+l3ETiMDa1wAYnRpPfmkV2cXarqCU6pyamxSKReQGEXHbyw3AiepRXgXGN/H+BOA0e5kJvNDMWNpOdC847VL4dg54ajlb2xWUUp1cc5PCrVjdUfcB+cAU4JamPmCM+RJoaha5SVhjHowx5msgxu722r6MnAHl+2HbQvonRJAUFapTXiilOq3m9j7KMcZcaYxJNMYkGWN+DLS091FPYI/P61x73Q+IyEwRWS0iqwsLC1v4tSfptEugay9Y/QoiwpjUeFbs1HYFpVTn1JI7r/2y1aI4AWPMi8aYTGNMZmJik00Zrc/lttoWspZASRZnp8ZTVF7NjoLyto1DKaXaQEuSgrTwu/di3aynQS97XfuTcSOIG9a85p0Hacm2AoeDUkqp1teSpNDS+pMFwE12L6TRQKkxJr+F+/SPrslw+nhY9ya9uwaR0SeGt77ZTX29ViEppTqXJpOCiBwSkbJGlkNY4xWa+uxcYAVwhojkishtInK7iNxub7IQyAJ2AC8BP2/54fhR5gyoKISt/+CmMSlkF1eydIfOhaSU6lyCmnrTGBN1qjs2xkw7wfsGuPNU99/mUi+E6D6w5hUmTJ/Eox+F8PrybM4/vY3bOJRSyo9aUn0UWFxuGHkT7PqS0NJspo3qw+fbCthTogPZlFKdhyaFkzHiRnAFwZpXuP6sPrhEmPN1jtNRKaVUq9GkcDKiusMZE2DdW/SIcHFpWjfeXr2HqlqP05EppVSr0KRwskbOgMpiWD+PG8f05WBlLQvW5zkdlVJKtQpNCier/zjoPRo+fZAxsYc4LSmS11dk6whnpVSnoEnhZLlccNWLAMj7M7n5rJ5s2lvGt3sOOhyYUkq1nCaFUxHbF654CnJXMrVyHpGhQbyxQhuclVIdnyaFUzV0CgyfRsjyP3HP6UV8tCGfovJqp6NSSqkW0aTQEhP/CDF9uCn/McI8h3h71Z4Tf0YppdoxTQotERoFV/+d4Ip9/C12DnNWZFPnqXc6KqWUOmWaFFqqVyaMe4Axh7/g7PLP+Oy7/U5HpJRSp0yTQms49x5M33P4r5BXmfvxYh3MppTqsDQptAaXG7nqRYJDu/Bcxb0seecZ0HELSqkOSJNCa4nuRfDtX7A//DQmbH+Y8jnXQ2VTt6hWSqn2R5NCa4rtS/Tti3iK6YTt/BTz/BjY/k+no1JKqWbTpNDKEqPDSZpwP1dWP0KZRMKbV8NH90KNTrGtlGr/NCn4wbQz+xDWO53LKh6hauTtsOolmDcN6rUBWinVvmlS8AOXS3hs8lAKq4SHqqfDlX+BrH/B5//ldGhKKdUkTQp+MqhHV35ybj/eXr2HlbFXQOatsOwp+O5Dp0NTSqnj0qTgR3dffBo9Y7rwn/M3UnPxf0PPTPjg51C4zenQlFKqUZoU/Cg8JIhHJg1me0E5zy/bA1Nfh+AuMG86VJU5HZ5SSv2AJgU/u2hQN36cnswzi7eztCAErnkVSrLggzugXudJUkq1L5oU2sBjk4cyICmSu+Z+y96YkXDpo7D1H/DVU06HppRSR9Gk0AYiQoP46w0jqfMY7pizhqqRM2HIFFj8X7D9M6fDU0opL00KbaR/YiR/mjqcDbml/O4fW+DKZ6DbEJh3Pax/2+nwlFIK0KTQpi4d3J2fX5DK3JW7eWd9Cdzyf9D7LJg/Ez5/TCfRU0o5TpNCG/vVpWdw7oAEHvxwExuLXXDD+zDiBvjyD/DebVBb5XSISqkApkmhjbldwp+vSychIoTb56zhQDVw5bNw8cOw6T147UdQXuhwlEqpQKVJwQHxkaE8f8NICg9Vc/ucNRyurYdz77HGMezbAC9fCHnfOh2mUioAaVJwSHrvGP54zTBWZpcw843V1t3a0ibBLQutKqQXL4C/jIRPfwO7v9bJ9JRSbUJMB2vczMzMNKtXr3Y6jFbzzqo9/Md7G7h4UBLPTx9JSJALKopg83zYthB2LYX6WohIhNPHW4mj/zhwBzkduvKHuhpY/DvoPgyGX+t0NKoTEZE1xpjME22nZxaHTT2zN9Ween7zwSbunvctf5k2gqCIBBj1U2upKrXGMmxbaE2m9+0bENUD0q+H9OkQn+r0IajWUlsF79wE2xdZrz01kHGjszGpgKMlhXbi5aVZPPrRFialJ/Pk1HTcLvnhRnU18P0n8O0c2PEZmHroew6MuBHSroSQiLYPXLWOmgqYOw12fQHjH4ftn8LOJTD5b1piUK2iuSUFTQrtyHNLdvDHRduYmtmLx68ahquxxNCgLB/Wv2UliJIsCIuxGqtHzYSQ8LYLWrVcVSm8ORVyV8Kk56xSYE0lvDUVcr6CKbNh8GSno1Qnw1MLNeVWsq+psEp9nlprqW94rIO6KqirthZPtc/zhu1qjmzrqYHUC2HQj04pJK0+6oDuHDeA6rp6nlm8HbfLxWM/HnL8xNC1B4z9FZz7S8hZDl89Df98CL5+Hs67DzJuhqCQtj0AdfIqS+CNybB/09En/5BwmDYP5lwN7/0E3CEw8HJnY20PPHVHnzzrDlvVbr6PnjowHqtzhvfRHDmxNpyUPT4n3boq67nvydlTY3223mN91nfxnrTrrMeGdQ2JwFPTOsfrCrb+9u4g6zEq+ZSTQnNpSaGdMcbwx0XbeP5fO5mUnswT1wwn2N3MTmK7v4bFj1hXlzF94IJfw7BrweX2b9AdWV0N5K8HlwuCukBw2JHH4IiWJ9bDB62bK9XXWX+T6F720ts6ibzxYyjeaXVHPmP8Dz9fVWYljfz1cN1bcPqlLYunOYyB8v1QsAUKt1rLgRzrGMCqtvQuBlxB1knLFWw/93l0h9jPQ46sP+oKubHHxtbVWCdr46eZhd0h4A61/t7uUAgKteMNtv7/cQX5LG6f93yO3R1iVeGGREBIJIRGWs+Dw48cf8PnGh6DQn2WMGu7oNAj/27SRG3BSdLqow7u+X/t4A+fbOPCgUk8d30GXUKaeWI3BnZ+biWH/HWQOBAm/A/0v8Cf4XZM2cvgH7+EouPc9CgoDC78DYz+uZU0Tlb+eqvh+OAe6yRQd8xodXFZ3zFtbtN/n8MH4fUroWArnDXTuiKuOgiHD9jLQavzQcaNVu+04C4nH2tVmVUVuWWBlQyqDh55LyzG6tDgDrFiFpd1shL736Tec6SKw/fq2bfaw7fqpOHEFxR29MkwuEvj64PCjjlZh/icSI9J5EFdrJO0uK2Tt/fRZZ/MQxpJVMGn9vftYNpFUhCR8cCfATfwsjHm8WPevwX4I7DXXvWsMeblpvYZKEkBYM7XOfzmw02c2TeOl2/JpGtYcPM/bIz1P/hnD8GBXVa1xKWPQXRP/wXcUVQUWeM/1r9lXb2PexDCon9YFbHrC6thv9958OMXrCv85lr7Onx0L4THw9TXoNeZUFkMB3dDaS6U7oFD+6y/S8+ME++vssSqSsr71oq1Syx0ibEew2KsC4CSLOu9YdfCyFug2+AT7/fgbvjmb1a81WWQPMJaEgdB4hmQNMjqDt2KV6zKGY4nBRFxA98HXkYjAAAUGUlEQVQDlwC5wCpgmjHmO59tbgEyjTH/1tz9BlJSAFiwPo9fvr2OgT2ieG3GKOIjQ09uB7VVsPwZWPon66rp/P+wrnwDsb2hvt7q0vvZb61633PugrH3Hr9h3hhr+49nWVeflz8JQ6c0/R01lbDwPlg3xxpPcvXLEJHQOvEbY1WfNFYdaIxV8ln7mtV12VNj3f510BUQkQThcVaC6hJnPS/JghXPHbln+ODJMObn0HNk68Sq2p32kBTGAA8bYy6zX/8awBjze59tbkGTwgkt2VrA7XPW0DO2C3NuO4vkmFOoHjiQDZ88ANs+goTT4ZL/sq5ew+Mavwqsq4G8tZC91DrZFO2A8++zrkA7osLvYcG/wZ5vrG68lz8JSQOb99ninTD/Z5C7CoZeAxOfsK7SG9vunZtg/2Yr+Z5/vzPtOZUlsH6elSAKtx5/u9BoyLzF6rF2MqUg1SG1h6QwBRhvjPmJ/fpG4CzfBGAnhd8DhVilinuMMXsa2ddMYCZAnz59Rubk5Pgl5vZs5a4Sbnt1FVFhQbx66yhO7xZ1ajv6/lP4+D+sKiWwGsRi+hxZwqKtk9/ub6xqFLDu+xAUCnvXwNl3wcW/a5s62HoPFHxnVZG05KS18V1YcJd1DJc9BsOnnXx1iKcOlj0J/3rcuuLu2uNII2it3ThafQjCusJVL8NpF596vK3FGCumwyVWoqgssZ8XW42fQ662GkNVQOgoSSEeKDfGVIvIz4BrjTEXNrXfQCwpNNicV8otr6yiutbDyzefyah+cae2o9oqqzH6YI5Vp3zAfjyYY9UrdxsCKedaS5+zISLeOil+cj+sehkGXgFXvdT64yHqPbBvo1UyyV4Gu5dbffhdQTByhnXlHZnY/P3V1cCn/wkrX7TuW3HNq9A1uWUx5q6xquLq6+yGTZ8lNMoqScX0btl3KOUH7SEpnLD66Jjt3UCJMSa6qf0GclIA2FNSyc2vrCT3wGGevjadiUN7tO4X1NUcv73BGPj6BVj0ACSnW/3oo7o3f9/GWP24y/KshtbSvVC290jDa956qC61to1LtZJS33Ngz9ew5jWrd8rZv4Axd1on4KaU5sI7N8Pe1TD6Trjkd1ZPE6UCVHtICkFYVUIXYfUuWgVcb4zZ7LNND2NMvv18MnC/MWZ0U/sN9KQAcKCihtteW8W3ew7y0BVp3HJOv7YNYNvH8O5tVrXO9HesXi6eOvsEv8fqgtnQu6Z8P5QXQEWB9VhbeczOxOpOGd0TktKsnj59z7GqZ3wV7YDPH7EaRiMSrVLD8Qbo7VhsDfjy1MKkZ2Hwj/32T6FUR+F4UrCDmAg8jdUldbYx5jEReQRYbYxZICK/B64E6oAS4A5jTBMtY5oUGlTVerhr7rd8+t1+fnZ+f+6/bGDT02K0tvz18Na1Vv/28DgrIRw7sKhLHER2g8gke+lmndC79rSSQHQvKyGczBV87mqrm23OMkCO6dtu928v3GZ1pZz6BiQMaNXDVqqjahdJwR80KRzhqTc8tGATc77ezZXDk/mfq4c1f5BbayjLs07QItYI3Zg+Vn16tD1yNzjMP99rDOxcbI3gbhj1Wnv4SMNvbF9rNLdOEKiUlyaFAGGM4YUvdvLHRds4PSmK56ZnMCBJe5QopY7W3KTQ+cd2d3Iiws8vGMBrM0ZRWF7Nlc8u48N1e0/8QaWUaoQmhU7ivNMTWXjXWAYnd+Xueet4YP5G6xafSil1EjQpdCLdo8OY+9PR3H5+Km99s5urnl9OdlGF02EppToQTQqdTJDbxawJA/n7zZnsPXiYy59Zyusrsqmv71htR0opZ2hS6KQuGtSNj+46l4y+sfz2w81c87cVbN9/yOmwlFLtnCaFTqxXbDiv3zqKP10znJ2F5Ux8ZilPffY91XXa1qCUapwmhU5ORLh6ZC8W//J8Lh/agz8v3s7lzyxjdXaJ06EppdohTQoBIj4ylKevG8ErM87kcI2HKX9dwX/O30hZVa3ToSml2hFNCgFm3BlJfHrPedx2bj/mrtzNxX/6gk825dPRBjEqpfxDk0IAiggN4jdXpPHBneeQEBnK7XPWMvONNeSXHnY6NKWUwzQpBLBhvWJY8G/n8MDEgSzdXsglT37Ja8u1+6pSgUyTQoALcruYeV4qn91zPiP6xPDQgs3c8eYayqvrnA5NKeUATQoKgN5xVvfV31yRxj+3FHDV81/paGilApAmBeUlItx2bj9ev3UUBYesyfW+/L7Q6bCUUm1Ik4L6gXMGJLDgznNJjunCLa+s5MUvd2rvJKUChCYF1ag+8eG8d8fZjB/Snf9euJV/f3udVicpFQCCnA5AtV8RoUE8d30Gzy3ZwZ8++54P1+WRmhjBxYO6cdGgbmT0iSHIrdcVSnUmeuc11Sx7Sir555b9LN5SwDe7iqn1GGLCgxl3RhLXZPZiTP94RNrwHtFKqZOit+NUfnOoqpYvvy9i8db9fL61gIOVtQzq0ZVbz0nhR8OTCQtuw/tEK6WaRZOCahNVtR4+XLeX2cuy2bb/EAmRIUw/qy83jO5LYlSo0+EppWyaFFSbMsbw1Y5iZn+1i8+3FhDidnHtmb35xYUDSOoa5nR4SgW85iYFbWhWrUJEOPe0BM49LYGdheW8vHQXc1fu5n/X7OHms1O4/bxUYiNCnA5TKXUCWlJQfpNTXMHT/9zOB+v2EhkSxMzz+jPj3H5Ehuq1iFJtTauPVLuxbd8h/vTpNj79bj/xESGM7h9PRKibyNBgIkPdRIYFEREaRHrvGAYnRzsdrlKdklYfqXbjjO5RvHhTJt/uPsBzS3awdV8ZFdUeyqvrqKipw/e6JLNvLDedncL4wd0JCdIxEEq1NS0pKEfV1xsqaz0crKzhk037eOPrHHKKK0mKCuX6s/pw/ag+2lCtVCvQ6iPVIdXXG774vpDXVmTzr22FBLmEC85I4qJBSYw7I4nu0ZoglDoVWn2kOiSXSxg3MIlxA5PYVVTBnK9z+GTTPv65ZT8AaT26Mm5gIhcOTCK9dyxul46iVqo1aUlBtXvGGL7fX87nWwtYsq2ANTkH8NQbEqNCuSqjJ9eM7M2ApEinw1SqXdPqI9VplVbW8uX2Qj5cl8eSbQV46g0ZfWK4JrM3VwzrQVRYsNMhKtXuaFJQAaHgUBUffLuX/12dy/aCcsKCXZyZEkdokAuXCG6X4HIJbhEMcLimjsoaD4drPRyu8VBZ4yEs2MVPxvbnqhE9ddZX1WlpUlABxRjDuj0H+d81uWzIPYin3lrnqTd4jKG+3iAihAW7CQ+xli72852FFWzcW8qApEjuu+wMLk3rpjO+qk5HG5pVQBERRvSJZUSf2JP+rDGGRZv38YdF2/jZG2tI7x3D/eMHMiY1vsVxVdV6WLK1gNBgFynxEfSOCydYSyOqHdOSglK2Ok8976/dy1P//J780irOTIklPiKUuvp66uqtUkedx+BywdmpCUwY0p3+iY03cOeXHuaNFTnMXbmbA5W13vVul9AzpgspCRH0iw8nMiwIl4h3cbusBHe4xsOByhoOVtZyoLKGA5W1lFbWEB4aRGpiBP0TI0lNjPQ+j+6i7SiqaVp9pNQpqqr1MOfrHN5dk4sx1ok82G21TwS5XFTU1LE5rwyAgd2jmDCkBxOHdmdAUiRrdx9g9lfZfLJpH8YYLknrxo2jU+gS4mJXUSU5xRXsKqogu7iCnKJKKms91BvDsf8bugRiwkOICQ8mNjyE2PBgoruEcKiqlqyiCrKLKqirP/KhvvHhXDyoG5ekdSOzb+xJt4146g2rs0vYWVjB8N7RDOreFZd29+1U2kVSEJHxwJ8BN/CyMebxY94PBV4HRgLFwLXGmOym9qlJQbUH+aWH+WTTPj7euI9VOSUYAwmRIRSV1xAVFsR1Z/bmpjEp9I4Lb9b+jDHUG6i320FC3K4mT8q1nnr2lFSSVVjBjsJyvs4qZvmOYmo89cSGBzNuYBKXpnVjdP94orsEN9pGUlXr4asdRXy6eT//3LKf4ooa73tdw4I4MyWOs/rHMapfPEOSu2ojfAfneFIQETfwPXAJkAusAqYZY77z2ebnwDBjzO0ich0w2RhzbVP71aSg2puCsioWbd7HiqxixqQmcNWInkQ4MBNseXUdS78v5LPv9vP5NuuOeAAhbhcJkSEkRoWSEGktZVW1fPF9IZU1HqJCgxg3MInLBndnUI8o1uce5JusElbuKiGrqMLaR5CLxMhQ4iJCiI8MIT4ilPjIEGLDQ4gIdRMW5CYsxE1YkIsuIW7Cgt0E2SUrlwuCXC67pGVVk4lYJTCr2syqMjvqEUFcWNuC9zNiv9ewrWCv044BJ9QeksIY4GFjzGX2618DGGN+77PNInubFSISBOwDEk0TQWlSUOrE6jz1rM45wKa9pRSWV1N0qMZ+rKawvBq3CBcOshLBmP7xx518sOBQFSt3lbAxt5Si8hqKK6opLq+huLyaoooaaurq2/jImubySRDehIFg/+d9Ld7XVmLB97WdX4563/4s9quGz+P7HT9YfyRRefd5gm29nxCOfm1vc92ZvfnJ2P6n8k/TLnof9QT2+LzOBc463jbGmDoRKQXigSLfjURkJjAToE+fPv6KV6lOI8jtYnT/eEb3b1kPqqSoMK4YlswVw5J/8J4xxjvmo8q71Hsf6+rrrcb5hkb6ekOdpx5jwGPMUVVm9fUGA0ceTUOVmtXeYq2zn9ufs9YfeR97PRy9vmE7vK9/+F7DZWjD9ehRn/M+x/sc3/XHfKfv5xvWN6w8sg9zzP6Ofs/3te+LhEj/3+K2Q3RJNca8CLwIVknB4XCUUlhXrhGhQY5UlSn/8WfL0V6gt8/rXva6Rrexq4+isRqclVJKOcCfSWEVcJqI9BOREOA6YMEx2ywAbrafTwE+b6o9QSmllH/5rdxntxH8G7AIq0vqbGPMZhF5BFhtjFkA/B14Q0R2ACVYiUMppZRD/FoZaIxZCCw8Zt1vfZ5XAdf4MwallFLNp6NRlFJKeWlSUEop5aVJQSmllJcmBaWUUl4dbpZUESkEck6wWQLHjIoOAHrMgUGPOTD445j7GmMST7RRh0sKzSEiq5szx0dnosccGPSYA4OTx6zVR0oppbw0KSillPLqrEnhRacDcIAec2DQYw4Mjh1zp2xTUEopdWo6a0lBKaXUKdCkoJRSyqvTJQURGS8i20Rkh4jMcjoefxCR2SJSICKbfNbFichnIrLdfox1MsbWJiK9RWSJiHwnIptF5G57fac8bhEJE5GVIrLePt7f2ev7icg39u/7bXta+k5FRNwi8q2I/MN+3amPWUSyRWSjiKwTkdX2Osd+150qKYiIG3gOmACkAdNEJM3ZqPziVWD8MetmAYuNMacBi+3XnUkd8CtjTBowGrjT/tt21uOuBi40xgwH0oHxIjIa+B/gKWPMAOAAcJuDMfrL3cAWn9eBcMzjjDHpPmMTHPtdd6qkAIwCdhhjsowxNcA8YJLDMbU6Y8yXWPef8DUJeM1+/hrw4zYNys+MMfnGmLX280NYJ42edNLjNpZy+2WwvRjgQuBde32nOd4GItILuBx42X4tdPJjPg7HftedLSn0BPb4vM611wWCbsaYfPv5PqCbk8H4k4ikACOAb+jEx21Xo6wDCoDPgJ3AQWNMnb1JZ/x9Pw38B1Bvv46n8x+zAT4VkTUiMtNe59jvWu+43QkZY4yIdMq+xiISCbwH/Lsxpsy6kLR0tuM2xniAdBGJAeYDAx0Oya9E5AqgwBizRkQucDqeNnSuMWaviCQBn4nIVt832/p33dlKCnuB3j6ve9nrAsF+EekBYD8WOBxPqxORYKyE8KYx5n17dac/bmPMQWAJMAaIEZGGi7nO9vs+B7hSRLKxqn4vBP5M5z5mjDF77ccCrOQ/Cgd/150tKawCTrN7K4Rg3fN5gcMxtZUFwM3285uBDx2MpdXZdct/B7YYY570eatTHreIJNolBESkC3AJVjvKEmCKvVmnOV4AY8yvjTG9jDEpWP/vfm6MmU4nPmYRiRCRqIbnwKXAJhz8XXe6Ec0iMhGrXtINzDbGPOZwSK1OROYCF2BNr7sfeAj4AHgH6IM1tfhUY8yxjdEdloicCywFNnKkvvkBrHaFTnfcIjIMq4HRjXXx9o4x5hER6Y91FR0HfAvcYIypdi5S/7Crj+41xlzRmY/ZPrb59ssg4C1jzGMiEo9Dv+tOlxSUUkqdus5WfaSUUqoFNCkopZTy0qSglFLKS5OCUkopL00KSimlvDQpKGUTEY89U2XD0mqTkIlIiu+stkq1VzrNhVJHHDbGpDsdhFJO0pKCUidgz3f/B3vO+5UiMsBenyIin4vIBhFZLCJ97PXdRGS+fS+E9SJytr0rt4i8ZN8f4VN7pDIicpd9n4gNIjLPocNUCtCkoJSvLsdUH13r816pMWYo8CzWiHmAvwCvGWOGAW8Cz9jrnwG+sO+FkAFsttefBjxnjBkMHASuttfPAkbY+7ndXwenVHPoiGalbCJSboyJbGR9NtYNb7LsSfn2GWPiRaQI6GGMqbXX5xtjEkSkEOjlOxWDPd33Z/ZNUxCR+4FgY8yjIvIJUI41VckHPvdRUKrNaUlBqeYxx3l+Mnzn6/FwpE3vcqw7BmYAq3xmBFWqzWlSUKp5rvV5XGE/X441myfAdKwJ+8C6feId4L1RTvTxdioiLqC3MWYJcD8QDfygtKJUW9ErEqWO6GLf6azBJ8aYhm6psSKyAetqf5q97hfAKyJyH1AIzLDX3w28KCK3YZUI7gDyaZwbmGMnDgGese+foJQjtE1BqROw2xQyjTFFTseilL9p9ZFSSikvLSkopZTy0pKCUkopL00KSimlvDQpKKWU8tKkoJRSykuTglJKKa//ByHq9OQYuxX/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f75cbf49910>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs = []\n",
    "for i in range(1, 52):\n",
    "    epochs.append(i)\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plot1 = plt.plot(epochs, train_loss)\n",
    "plot2 = plt.plot(epochs, val_loss)\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.title(\"Loss\")\n",
    "plt.legend([\"Training Loss\", \"Validation Loss\"], loc = 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXZ+PHvnX0he8K+BAFlk02KC6AgorgUxBVxQdFStVXcWq3aWn3bvtiftS71tcUF0SqIC+KGiogVNxBQFgEbloBAgJCQfZ3M/fvjnISAExJCJkMy9+e65po5Z86cc58hnHue5TyPqCrGGGPMoUICHYAxxphjkyUIY4wxPlmCMMYY45MlCGOMMT5ZgjDGGOOTJQhjjDE+WYIwxhjjkyUIE5RE5FMR2S8ikYGOxZhjlSUIE3REJB0YCSgwvhmPG9ZcxzKmKViCMMHoGuBr4AVgSvVKEYkWkb+JyDYRyReRz0Uk2n1vhIh8KSJ5IvKjiFzrrv9URG6otY9rReTzWssqIr8SkQwgw133uLuPAhFZKSIja20fKiL3ishmESl03+8iIk+JyN9qn4SIvC0it/vjCzIGLEGY4HQN8LL7OEdE2rnrHwFOAk4DkoHfAl4R6QYsBJ4E0oBBwHdHcLwLgZOBvu7yN+4+koFXgNdEJMp97w7gCuA8IB6YCpQAs4ErRCQEQERSgbPczxvjF5YgTFARkRFAN2Ceqq4ENgOT3QvvVGC6qu5U1SpV/VJVy4HJwMeqOkdVK1U1R1WPJEH8r6rmqmopgKr+292HR1X/BkQCJ7jb3gDcr6o/qGO1u+1yIB8Y4243CfhUVfcc5VdiTJ0sQZhgMwX4SFX3ucuvuOtSgSichHGoLnWsb6gfay+IyF0issGtxsoDEtzj13es2cBV7uurgJeOIiZj6mWNZiZouO0JlwGhIrLbXR0JJAIdgDKgB7D6kI/+CAyrY7fFQEyt5fY+tqkZMtltb/gtTknge1X1ish+QGodqwewzsd+/g2sE5GBQB/grTpiMqZJWAnCBJMLgSqctoBB7qMPsBSnXeJ54FER6eg2Fp/qdoN9GThLRC4TkTARSRGRQe4+vwMuEpEYEekJXF9PDHGAB8gGwkTkDzhtDdWeBf5HRHqJY4CIpACo6g6c9ouXgDeqq6yM8RdLECaYTAFmqep2Vd1d/QD+AVwJ3AOsxbkI5wIPAyGquh2n0fhOd/13wEB3n38HKoA9OFVAL9cTw4fAB8B/gW04pZbaVVCPAvOAj4AC4Dkgutb7s4ETseol0wzEJgwypuUQkdNxqpq6qf3nNX5mJQhjWggRCQemA89acjDNwRKEMS2AiPQB8nAa0x8LcDgmSFgVkzHGGJ+sBGGMMcanFn0fRGpqqqanpwc6DGOMaVFWrly5T1XT6tuuRSeI9PR0VqxYEegwjDGmRRGRbQ3ZzqqYjDHG+GQJwhhjjE+WIIwxxvjUotsgjDGOyspKduzYQVlZWaBDMceQqKgoOnfuTHh4eKM+bwnCmFZgx44dxMXFkZ6ejojU/wHT6qkqOTk57Nixg+7duzdqH36rYhKR50Vkr4isq7UuWUQWiUiG+5zkrhcReUJENonIGhEZ4q+4jGmNysrKSElJseRgaogIKSkpR1Wq9GcbxAvAuEPW3QMsVtVewGJ3GeBcoJf7mAY87ce4jGmVLDmYQx3t34TfqphU9TMRST9k9QRglPt6NvApcLe7/kV3ALKvRSRRRDqoapa/4jPGX7LyS9mSXczQ9CQiw0IDHQ4AlVVeKjxevKp4vUqVUvM6JESIiQglOjy03guK16tUVnmpcj/rVQ56rWjN9EjVg/jYaD7+ER8dRkyEf1sJmrsNol2ti/5uoHqy+E4cPCb+DnfdTxKEiEzDKWXQtWtX/0VqjGt/cQUrt+0nPjqc5NgIUmIjSIgOJyREUFV27C/l6y05LNuay/KtuWzPLQEgLS6Sa09L56qTu5EQ07hGwsaq8FRRXF5FcbmH4goP5R5vvZ8JESdRxEaGERsRSlhoCOWVVZR5vJRVVlFW6aXCU8Wh1/u8/blMmzQBgH3ZewkJCSU5JQWAl99ZTHhERL3H/v0dv+L6X91Geo9edW4z94VniEtI4PyJl9W7v4bIyd7L2J/15f7/fZSLrrimSfbZnMJDo/2eIPw6WJ9bgnhXVfu7y3mqmljr/f2qmiQi7wIzVPVzd/1i4G5VPext0kOHDlW7k9r4y56CMp75bAuvLN9OSUXVQe+FhghJMeGICNmF5QAkxYQzrHsyw7qn0CkxipeXbWdpxj5iI0KZNKwrU0d0p1OiM/ePqrIrv4zvd+azblcB/91dCEB0RChR4aE1v+ijI0JJbRNBh4RoOiZG0SEhmtjIAxeFonIPW7OLqcjZTqfuvSivrKK0ooqKKm9NnLERYcRGhhEVHkKICCEihIY4CSEkRPBUKSUVHoornIRSVnnwuQJEhoUSFR5CZFgokWEhhIYIIQIhIUKou58QgQcffJC4Nm248667auZQFXESqaoSEnLs9Kx/8sknmTdvHhERESxevNhvx/F4PISFBa4/0IYNG+jTp89B60RkpaoOre+zzR31nuqqIxHpAOx11+/Emay9Wmd3nTFHTFX5flcB//lvNnklFXRKjKZTUgydk6LplBRNfNThf81vzynhn59t5vUVO6hSZcLAjlz+sy5UVHnJLa4gp6jCeS6uoNxTxeAuiQzrnkKvtm0ICTlQRTOufwe+35XPM59t4YUvM5n9ZSZn9WlHUbmH73fls7+kEoAQgfTUWMJDQiitrKKkooqyyipKKjx4ffx+i48Ko31CFPmllewpcJLTM+M7EFFQRnhYiJNUIiOJjXCSQn3VRhFhQkRYBInuzNpVXi8lFVV4qpTI8BCiwkIPOq/DCRFB3CS0adMmxo8fz+DBg/n2229ZtGgRDz74IKtWraK0tJTLL7+cP/zhDwCMGDGCf/zjH/Tv35/U1FRuvPFGFi5cSExMDAsWLKBt27bcf//9pKamcttttzFixAhGjBjBJ598Qn5+PrNmzeK0006juLiYa665hg0bNtC3b18yMzN59tlnGTRo0E9inTNnDk8++SSXXHIJWVlZdOjQAYD33nuP3//+91RVVdGuXTs++ugjCgsL+fWvf823334LwEMPPcQFF1xAamoqeXl5AMydO5ePP/6YZ599lquuuoq4uDhWrlzJqFGjuOiii7j99tspKysjJiaGF154gV69euHxePjNb37DokWLCAkJ4cYbb6Rnz57MnDmT119/HYCFCxfy/PPP89prrzXo36ApNXeCeBtn2scZ7vOCWut/LSJzgZOBfGt/MEciv6SSpZuy+fSHbP7z3+yaX/URYSFUHFK9EhcVRvv4KJJiIkiICScpJpzEGKfaKGNPIe+sySJUhEuHdubGM3rQJTmm0XH165jAY5MG85txvXn+860s+G4XHRKiOKdfe/p1SqBfx3j6tI8nOuKnbRWqSrnHS3ZhOVn5ZWTllzrPec5zXFQ4x6XF0iMtlnbk0q9TAiEiPPjO96zfVdDomH3p2zGeB37e74g/t3HjRl588UWGDnV+rM6YMYPk5GQ8Hg+jR4/mkksuoW/fvgd9Jj8/nzPOOIMZM2Zwxx138Pzzz3PPPff8ZN+qyvLly3n77bd56KGH+OCDD3jyySdp3749b7zxBqtXr2bIEN8dIjMzM8nNzeWkk07i0ksvZd68eUyfPp3du3dz0003sXTpUrp160Zubi4Af/zjH0lLS2PNmjWoak1SOJysrCy+/vprQkJCyM/PZ+nSpYSFhfHBBx9w//338+qrr/L000+za9cuVq9eTWhoKLm5uSQmJvLrX/+anJwcUlJSmDVrFlOnTj3Sr75J+C1BiMgcnAbpVBHZATyAkxjmicj1OPPxVlcmvo8z5+8moAS4zl9xmZYvv7SSjVkFbMgqYENWIRt2F/D9rgKqvEp8VBgjj09j1PFpnHFCGmltItlXVMHOvFJ27i9lZ14JO/aXsregnLzSCn7MLWHtjkrySisoq/QSExHK1OHp3DDyONrFRzVZzJ0So/n9BX35/QV969/YJSJEhYfSJTmm3iS1YUMeIcdgL6YePXrUJAdwfrU/99xzeDwedu3axfr163+SIKKjozn33HMBOOmkk1i6dKnPfV900UU122RmZgLw+eefc/fddwMwcOBA+vXzndTmzp3L5ZdfDsCkSZO4+eabmT59Ol999RWjR4+mW7duACQnJwPw8ccf89ZbbwHOv0tSUhIej+ew537ppZfWVKnl5eVxzTXXsHnz5oO2+fjjj7ntttsIDQ096HhXXnklr7zyCldeeSUrV65kzpw5hz2Wv/izF9MVdbw1xse2CvzKX7GYlk1V+XpLLnO/2c6KzP3szCuteS85NoI+HeK46YwejDohjUFdEgkLPbieOy0ukrS4SAZ1STx01weprnuPCj82eh41VmN+6ftLbGxszeuMjAwef/xxli9fTmJiIldddZXPPvoRtRq1Q0ND67wQR0ZG1rtNXebMmcO+ffuYPXs2ALt27WLLli1HtI+QkBBqt+Eeei61z/2+++7jnHPO4eabb2bTpk2MG3foHQAHmzp1KhdffDEAl19+eU0CaW52J7UJmB92FxIVHkLnpBhCfdRx55dU8vqqHby8bBtbsotJiA7n9OPTuPKUrvTpEE/fDvG0jYtssv7/LT0xHOsKCgqIi4sjPj6erKwsPvzww3ovlEdq+PDhzJs3j5EjR7J27VrWr1//k23Wr1+Px+Nh584DzZz33Xcfc+fO5frrr2f69Ols27atpoopOTmZsWPH8tRTT/HII4/UVDElJSWRlJRERkYGPXr0YP78+aSl+Z5iIT8/n06dOgHwwgsv1KwfO3Ys//znPzn99NNrqpiSk5Pp0qULqampzJgxgyVLljTpd3QkLEGYZqWqfLUlh8c+zmD5Vqd+NzIshB5pbejZtg292raha0oMSzP28c7qXZR7vAzumsjfLh3I+QM62EW8BRsyZAh9+/ald+/edOvWjeHDhzf5MW655RauueYa+vbtW/NISEg4aJs5c+YwceLEg9ZdfPHFTJkyhXvvvZenn36aCRMmoKp07NiRhQsX8sADD3DzzTfTv39/QkND+Z//+R/Gjx/Pww8/zDnnnEPbtm056aSTKC8v9xnX3XffzdSpU3nwwQdrqs8AfvnLX5KRkcGAAQMICwvjpptu4sYbbwRg8uTJFBQUcPzxxzfxt9RwLXpOauvm2nKoKl9tzuGxxU5iaBsXybTTjyM+KpyMvYVk7C0iY09RTfVRbEQoFw7uxOSTu9KvY0I9eze+ujIGI4/Hg8fjISoqioyMDM4++2wyMjIC2s20sW688UZOPfVUpkyZclT7aUndXE2QUVW+2JTDE4szWJ6ZS7v4SB4c34/Lf9bFZ2mguNzDtpwSuqbE0CbS/jzNkSkqKmLMmDF4PB5UlX/9618tMjkMGjSIpKQknnjiiYDG0fK+OdMi5BZX8OaqHcxZvp3N2cW0j4/ioQn9uGyo78RQLTYyjL4d45sxUtOaJCYmsnLlykCHcdS+++67QIcAWIIwTai6fWHO8h/5cN1uKqq8DOmayF8vGcD4gR2t/cCYFsYShDkiu/JKmf1VJgWllZRUOMM6lLrDO2Tll7Ezr5T4qDAmn9yVK4Z15YT2cYEO2RjTSJYgTIMt25LDzS+voqCskqSYCKJrjRcUHR7KiZ0SuPPs4znvROttZExrYAnC1EtVeenrbTz0znq6Jsfw6i9PpWfbNoEOyxjjZ8fO0IrmmFTuqeLuN9bwhwXfc/rxabz16+GWHMxPjB49mg8//PCgdY899hg33XTTYT/Xpo3zt7Rr1y4uueQSn9uMGjWK+rqzP/bYY5SUlNQsn3feeQ0aL6mhBg0axKRJk5psfy2FJQhTp935ZVz+r6+Zt2IHt5zZk2evGVrvSKgmOF1xxRXMnTv3oHVz587liivqGnHnYB07dqwZvbQxDk0Q77//PomJhx9apaE2bNhAVVUVS5cupbi4uEn26cuRDhfSHCxBmJ8oKvcwd/l2fv6Pz/nvnkL+edUQ7jz7hAYP+WyCzyWXXMJ7771HRUUF4IyWumvXLkaOHFlzb8KQIUM48cQTWbBgwU8+n5mZSf/+/QEoLS1l0qRJ9OnTh4kTJ1JaemDsrZtuuomhQ4fSr18/HnjgAQCeeOIJdu3axejRoxk9ejQA6enp7Nu3D4BHH32U/v37079/fx577LGa4/Xp04df/OIX9OvXj7PPPvug49Q2Z84crr76as4+++yDYt+0aRNnnXUWAwcOZMiQITUD8T388MOceOKJDBw4sGYU2tqloH379pGeng44w26MHz+eM888kzFjxhz2u3rxxRcZMGAAAwcO5Oqrr6awsJDu3btTWekMG19QUHDQclOwNggDOO0MK7ft59VvfuS9tVmUVFTRp0M8/77+ZOuJ1NIsvAd2r23afbY/Ec6dUefbycnJDBs2jIULFzJhwgTmzp3LZZdd5oxIGxXF/PnziY+PZ9++fZxyyimMHz++zjG0nn76aWJiYtiwYQNr1qw5aMjuP//5zyQnJ1NVVcWYMWNYs2YNt956K48++ihLliwhNTX1oH2tXLmSWbNmsWzZMlSVk08+mTPOOKNmDKU5c+bwzDPPcNlll/HGG29w1VVX/SSeV199lUWLFrFx40aefPJJJk+eDDgjrt5zzz1MnDiRsrIyvF4vCxcuZMGCBSxbtoyYmJia4cIPZ9WqVaxZs6ZmGHRf39X69ev505/+xJdffklqaiq5ubnExcUxatQo3nvvPS688ELmzp3LRRddRHh405XyLUEEufzSSuYu386rK35kS3YxsRGhjB/Ykct+1oXBXRKbbCA80/pVVzNVJ4jnnnsOcH583HvvvXz22WeEhISwc+dO9uzZQ/v27X3u57PPPuPWW28FYMCAAQwYMKDmvXnz5jFz5kw8Hg9ZWVmsX7/+oPcP9fnnnzNx4sSakVUvuugili5dyvjx4+nevXvNREK1hwyvbcWKFaSmptK1a1c6derE1KlTyc3NJTw8nJ07d9aM6RQV5QwN//HHH3PdddcRE+MMz149fPfhjB07tma7ur6rTz75hEsvvbQmAVZvf8MNN/DXv/6VCy+8kFmzZvHMM8/Ue7wjYQkiiC3ZuJd73lzDnoJyhnZL4sZLenD+iR0OmtLStECH+aXvTxMmTOD2229n1apVlJSUcNJJJwHw8ssvk52dzcqVKwkPDyc9Pd3nMN/12bp1K4888gjffPMNSUlJXHvttY3aT7Xq4cLBGTLcVxXTnDlz2LhxY02VUEFBAW+88cYRN1iHhYXh9ToTVx1uWPAj/a6GDx9OZmYmn376KVVVVTXVdE3F2iCCUEFZJb99fTXXvfANCdHhLPjVcF6/6TQuG9rFkoNptDZt2jB69GimTp16UON0fn4+bdu2JTw8nCVLlrBt27bD7uf000/nlVdeAWDdunWsWbMGcC7OsbGxJCQksGfPHhYuXFjzmbi4OAoLC3+yr5EjR/LWW29RUlJCcXEx8+fPZ+TIkQ06H6/Xy7x581i7di2ZmZlkZmayYMEC5syZQ1xcHJ07d66ZRKi8vJySkhLGjh3LrFmzahrMq6uY0tPTa4YAOVxjfF3f1Zlnnslrr71GTk7OQfsFuOaaa5g8eTLXXdf086wFJEGIyHQRWSci34vIbe66ZBFZJCIZ7nNSIGJr7ZZmZDPu75/x+sod3DyqB+/cMoKB9UykY0xDXXHFFaxevfqgBHHllVeyYsUKTjzxRF588UV69+592H3cdNNNFBUV0adPH/7whz/UlEQGDhzI4MGD6d27N5MnTz5ouPBp06Yxbty4mkbqakOGDOHaa69l2LBhnHzyydxwww0MHjy4QeeydOlSOnXqRMeOHWvWnX766axfv56srCxeeuklnnjiCQYMGMBpp53G7t27GTduHOPHj2fo0KEMGjSIRx55BIC77rqLp59+msGDB9c0nvtS13fVr18/7rvvPs444wwGDhzIHXfccdBn9u/f3+AeY0ei2Yf7FpH+wFxgGFABfADcCEwDclV1hojcAySp6t2H25cN991whWWV/O/CjbyybDs90mL522WD6p1hzbQcNtx38Hr99ddZsGABL730ks/3W9pw332AZapaAiAi/wEuAibgzGENMBv4FDhsgjD1Kyyr5MWvtvHM0i3kl1byy9OP4/axx9tQGMa0ArfccgsLFy7k/fff98v+A5Eg1gF/FpEUoBQ4D1gBtFPVLHeb3UA7Xx8WkWk4pQ26du3q/2hbqMKySmZ/mcmzn28lr6SSM3u35fazjufEzjb5jjGtxZNPPunX/Td7glDVDSLyMPARUAx8B1Qdso2KiM+6L1WdCcwEp4rJz+G2OEXlHl74YivPLN1KfmklY3q35dYxvaydIQioqnVLNgc52iaEgHRZUdXngOcAROQvwA5gj4h0UNUsEekA7A1EbC3Z7vwyrnpuGZv2FjGmd1umn9WLAZ0tMQSDqKgocnJySElJsSRhACc55OTk1Nyj0RgBSRAi0lZV94pIV5z2h1OA7sAUYIb7/NP78U2dfswt4cpnl5FTVM7LN5zM8J6p9X/ItBqdO3dmx44dZGdnBzoUcwyJioqic+fOjf58oDq9v+G2QVQCv1LVPBGZAcwTkeuBbcBlAYqtxdm6r5grn/maonIP/77hZAZ3tR7CwSY8PJzu3bsHOgzTygSqiuknd6qoag4wJgDhtGj/3VPIlc8uo8qrzJl2Cv06WiO08aE0DypLIL5j/dsahyqUF0Lhbigv8L2NhEBkHES0gcg2EB4LIa3n/mO7bbYFW7czn6ufW0Z4aAjzfnkKPdsG4aB6Xi989zKU5sLx4yD1eLA6+ANU4btX4MPfQVk+JHWH7iOh+xmQPhLifHYWPHZ4q2BfBuz61h2AUA9cjCPaHLg4eyuhvAgqipyLeXkRVBRDSKjv7dFa2xe6z0XO31HhbijMgoIsqGzE8N4RbSAqAdq0g7gOENf+wHNsmhNTU0jrDYldmmZfdWj2G+WaUjDfKLdy236unbWc+KhwXvnFyXRLia3/Q/7krYKtn0G34RAW0TzHLMiCt26CLUsOrEvuASecCyecB11OhtAA/QbyegP/SzJ/B7xzG2xaBF1Pg97nw7YvIPMLKM93tkk9HroMg5RezuvUXpCUDqFNOO9HZSmo96fr1etcxMuLoKLQeS4vhLI82L3OSQpZqw9cpMOinYtrRVH9xwwJd5KC1+vs29fxa5MQiIiD6ESnlFVzUXcfUQm+f3h4PQfHX51oyvLcROMmm9L6R3U9Yuc/Cj+7vlEfbeiNcpYgWqAVmblMeX45aXGRvPyLU+iUGB3YgLxV8NbNsGYudBgIFz0Lacf795gb3oW3bwFPGYz7X+h5Fvz3A9j4vpOovJUQnQR9fg4nXQedhtS/z6OhCns3wA/vww8LnV+7E5+G/hf797h1xbLqRfjofucCdtYf4We/OJCwvFWwe43zPW1dClnfQXGtxu2QMCdJxHVwLpyHCouClB6Q0tNJKKnHO7+WRZyqrKzvnIv7zlWw6zvI337k5xAWBe0HQMfBBx6pvZwE4fU6SaP6glxRdCAhRMQ5z2EHBuJD1UlS1aWF8kI3IcQeKFGER/u35FlZBkV7oHgf0ETX3MSu0KZtoz5qCaKV+iYzl2ufX067+CjmTDuFdvGN78LWJLxeePvXTjXPoKucC2RlqXPRPunaw/+nK9wDsalHVuSuKIYP74WVLzjJ6OLnnAtHbWUFsPkTJ5YN7zh17x0Hw9CpzgU7oolKW1Ue5xf5DwudY+W5g9B1Osm5MO9eB5c8D/0ubJrj1RlH5YFqkuJ9sPghp1SVPhLGPwnJDWi8Ls2DnE1OdU5OhvNcVEdP84oiyNkMnlqjn0bEOQm5djJISne+97b96ihVysEX6epqoKgESOwWuNJfELAE0Qp945Yc2h9LyeHd6c6v1TPugdG/O7ja54TznQtUbMqBz+TvhHWvw5rXYM9aZyKa8/8OXX5W//F2roQ3f+lcyIbfCqPvr786qywf1syDb56D7A0QmQADJ8GwaZDa88jPWRV2rXLiX/cGFO+F0Eg4bhT0Ps9pB4lr71yw/32xE/Ols6HPBXXvc9NiWPJnGDAJhv2i/l+yuVvhnVthz3rnYu05ZDjo8Fg4+yE4aar/qrm8XijY6SaTTc5zSQ606+ckhQ6DIKb+uRBMYFiCaGVqkkNCFHN/cQptmyM5qNZ9sVKF9+6EFc/ByLvgzPsPbOv1wrKn4eM/QnQyXPB3KNnnXKgzPwfU+ZXd8yxY9ZJTR3vSFBjzwE8vKqqQuRS+eMKpS4/rCBP/CcedceTn8uMyJ1Gsf8upYhj/JAxoYG/qnM2w9jXnkbMJQiPg+HPgxEud8/BVKikrgJcmOvXol/8bThh3yPv58OF98O1LEBnvNK72m+jEFVlHh4ON78P8G53X/Sc6nzv0F3iXkyGhU8O/GxN0LEG0Isu35nLtLD8lB1XYn3lw1UL167IC50J8wrlw/LkHeryowsK7Yfm/YPhtTh23r0SStQbeuAH2/eAsJ/dwLsgnXurUYYPzS/vTGfD1004D4diHYOBkp1Fxw9vwxeNOnXZsGgz7JQy7wanKOBqFe+D1qbDtczjtVif+uqq5CrKcKq3v3wQE0kc48fcd37A4yvLhxQmw53uY9Ar0Guusz1gE70x3kuNpt8Koe5zv4JP/cb6ny16Edn0P7KfKA5885HwfHQY6pZKGVB0Z44MliFZi2ZYcrnvhGzokRDGnqZPD3g1OKWDbFwfWRSe5PVp6OQ2FmxZBnluv3GmokywKs+CbZ+HUX8PZfzp8lUhFiXNxTevjNBTXte3udfDeHc6v/E5DnUbTvG3OxfK0W5xqofAmbIyvqoQPfgffPOOUAC5+zklQNe97YPlMWPIXqKqA4dOdNpXG/DIv3Q+zx0P2D3DRv5zk8N3LTjfFCf8HnU86sG3m507yKiuACx6FQZOdJPX6VNj+pdOOcs7/QniAqxdNi2YJooWrrPLy1JJN/OOTTXRLiWna5FBeBP95GL7+P6d6YuSd0PlnTmKo3V4Abu+c9Qd65+x0ZsUjQ+SLAAAc5UlEQVTi5Bth3Iym7fnh9cLqV5yLcnxH56J8wnlN12/cl5UvwHt3QVI3mDTH6X21fZmTrPasg55j4by/QvJxR3ecklwnSexZCxIKI26HM357cG+baoV74I3rnaq1fhOdpFFRDD9/vOFVYsYchiWIFmzj7gLunLea73cVcOGgjvxxfD8SY5rg3gJV2PguLLwHCnbA4KvhrAd/mhQOp3C3U/I4blTruSFt21fw6lVOSaHnGPh+PsR3chJgn5833XkW58DSR5yLfMd6ZjWr8sCnf4Glf4PUE5wqp7aHn4nNmIayBNECeaq8/OuzLTz28X9JiA7nTxeeyLj+7Ztm53s3wqI/QMaHTrfDCx6Frqc0zb5bg7wf4dUrnbaCU26GM+52Gn0DLfsHSOgCETGBjsS0IsfyjHLGh4w9hdz52mrW7MjnggEdeGhCf5Jjj7LU4Cl37gNYMctpkI1oA+f8xWnstT7mB0vsAtd/7NzxGtdESbkppJ0Q6AhMELOrxDFg7Y58Js38isjwUJ6aPITzB3Q4uh3mbnXq1r/9t9O9NCndqUoafJVzY5rxLSzi2EoOxgSYJYgAy9xXzLWzlpMYE8HrN51Kh4RG9tTJ2Xzgjt5tXzgNoSec6/R6OW504McFMsa0OJYgAii7sJxrnl+OV5UXrx92ZMnBW+X0KPrhfefmqep7Ddr2g1G/cxqg7WYpY8xRsAQRIEXlHq57YTl7C8t45Ren0COtngbR6sHgMpc6g6xt+8LpXx8S5oygOnSqc6duUnqzxG+Maf0CNeXo7cANOMMargWuAzoAc4EUYCVwtapWBCI+f6vweLnxpZVsyCrkmWtOYsjhZoDb+D6sedXpC1+yz1mX2NUZ56jHaKdb5tHeWWyMMT40e4IQkU7ArUBfVS0VkXnAJOA84O+qOldE/glcDzzd3PH5m9er3PXaaj7ftI//d8kAzux9mAlblv0LFv7WGXa55xjofrozQmdSt+YL2BgTtAJVxRQGRItIJRADZAFnApPd92cDf6QVJoi/vL+Bt1fv4jfnnMClQw8zG9QXjzv3LfS+wBky2tcdt8YY40fN3rVFVXcCjwDbcRJDPk6VUp6qetzNdgA+W1hFZJqIrBCRFdnZ2b42OWZ9sC6LZz/fypRTu3HzqB6+N1KF//zVSQ79LoJLX7DkYIwJiGZPECKSBEwAugMdgVhg3GE/VIuqzlTVoao6NC0tzU9RNr29BWX87s21nNgpgfsv6Iv4Gr5B1RnNc8mfYeAVcPGzTTv1ozHGHIFAVDGdBWxV1WwAEXkTGA4kikiYW4roDOwMQGx+oar89o01lFRU8ffLBxEe6iMvqzpzA3z9FAyZAhc8ZvcuGGMCKhBXoO3AKSISI87P6DHAemAJcIm7zRRgQQBi84uXl23n0x+yufe8PvRs66M7a0WxM0PY1085w2D8/HFLDsaYgGv2EoSqLhOR14FVgAf4FpgJvAfMFZE/ueuea+7Y/GFLdhF/fm8DI3ulcvUph/Q+UoWN78EH90D+jzDiDhjzh9YzSqoxpkULSC8mVX0AeOCQ1VuAYQEIx28qq7zc/up3RISF8MilAwkJqXXh35/pzMr23w+cu5+nfmijqxpjjil2J7UfPbVkE6t35PPU5CG0q57sx1MOXz4Bnz3i3AV99p/h5F9aY7Qx5phjCcJPvt2+nyc/2cTEwZ2c0VlVndLCR/c7k973neBMHWnjJRljjlGWIPygrLKKO+atpn18FA9O6AdZq50eSplLnWk9r3wDep0V6DCNMeawLEH4wdOfbmbrvmJem9yN+IW3wuo5znhJ5z3iTHxv1UnGmBbAEkQT255Twqz/rOf/On3Cz96eB1oFp90CI++E6MRAh2eMMQ1mCaKJzZ/3PO+HPkbnnGzoNxHO+qMNwW2MaZEsQTSVvB/Jfu12pu9ZRG5sd7h0FnQfGeiojDGm0SxBHK2qSvjqKfQ/D9OmsoqZEddw7fT/B5FRgY7MGGOOiiWIo1G0F16cAHvXszXlDK7eeREzrjyfCEsOxphWwBLE0fjwPtiXQc7PZ3PeW5GM7t+Wkb1azgizxhhzODYiXGNt+Q+snQcjbuP3G52Jf+47v0+AgzLGmKZjCaIxPOXw3p2QlM6XHa7l/bW7+dWonnROigl0ZMYY02SsiqkxvnwCcjLwXDGP37+7iW4pMfzi9OMCHZUxxjQpK0EcqdytzkB7fcbzUcUANmcXc+95fYgKDw10ZMYY06QsQRwJVVj4W2cU1nEzmP/tTtrFR3JWn3aBjswYY5pcIOakPkFEvqv1KBCR20QkWUQWiUiG+5zU3LHVa8M7kPERjPod+8PS+PSHvUwY1InQEJvgxxjT+jR7glDVH1R1kKoOAk4CSoD5wD3AYlXtBSx2l48d5YXOBD/tToSTb+TdNbuorFIuHGTDdRtjWqdAVzGNATar6jZgAjDbXT8buDBgUfny6Qwo3AUXPAqhYbz57U56t4+jb8f4QEdmjDF+EegEMQmY475up6pZ7uvdwLFTsb8vA75+GoZMgS7D2LqvmG+35zFxsJUejDGtV8AShIhEAOOB1w59T1UV0Do+N01EVojIiuzsbD9H6fphoTNs9yin1mv+tzsRgQlWvWSMacUCWYI4F1ilqnvc5T0i0gHAfd7r60OqOlNVh6rq0LS0ZhrWYtsXkNwD4juiqrz17U6G90ilfYKNuWSMab0CmSCu4ED1EsDbwBT39RRgQbNH5Iu3CrZ9BenDAVi1fT/bc0u40KqXjDGtXEAShIjEAmOBN2utngGMFZEM4Cx3OfD2rIPyfOg2AoA3V+0kKjyEcf3bBzgwY4zxr3qH2hCRW4B/q+r+pjqoqhYDKYesy8Hp1XRs2fal85w+nHJPFe+uyeKcfu1pE2mjlBhjWreGlCDaAd+IyDwRGSciwXVXWObnkNgNEjqzZGM2+aWV1nvJGBMU6k0Qqno/0At4DrgWyBCRv4hIDz/HFnher1OCSHeql+Z/u4PUNpGM6Jka4MCMMcb/GtQG4XY73e0+PEAS8LqI/NWPsQVe9kYozYVuw8krqWDJxmzGD+xIWGigbx8xxhj/a0gbxHTgGmAf8CzwG1WtFJEQIAP4rX9DDKBtXzjP3U7jvbVZVFR5uWiIVS8ZY4JDQ1pak4GL3OEwaqiqV0Qu8E9Yx4jMzyG+EySlM3/VV/Rq24Z+NrSGMSZINKSuZCGQW70gIvEicjKAqm7wV2ABp+qUILoN58f9pazYtp+JQzoRbG30xpjg1ZAE8TRQVGu5yF3Xuu3LgOJsSB/O8q1Ofhxr8z4YY4JIQxKEuI3UgFO1RDBMVVrT/jCCTdlFhIUI6amxgY3JGGOaUUMSxBYRuVVEwt3HdGCLvwMLuG1fQJt2kNKDTXuLSE+NJdx6LxljgkhDrng3AqcBO4EdwMnANH8GFXCqkOm0PyDC5r1F9ExrE+iojDGmWdVbVaSqe3HmbQge+7c6kwOlD6fC42Vbbgnnndgh0FEZY0yzash9EFHA9UA/oGZ8a1Wd6se4Aiuzuv1hOJk5xVR5lZ5trQRhjAkuDaliegloD5wD/AfoDBT6M6iA2/YFxKRAWm827XU6cFmCMMYEm4YkiJ6q+nugWFVnA+fjtEO0XplfQLfTQKQmQRyXZj2YjDHBpSEJotJ9zhOR/kAC0NZ/IQVY3nbI314z/8OmvUV0SowmJqL19+w1xpjaGnLVmykiScD9OLO+tQF+79eoAqnW/A/gJAirXjLGBKPDJgh3QL4Cd7Kgz4DjmuKgIpKIM/Bff0CBqcAPwKtAOpAJXNaUkxQ1WObnEJUIbfvh9Spb9hVxao+U+j9njDGtzGGrmNy7pv0xWuvjwAeq2hsYCGwA7gEWq2ovYLG73Py2ue0PISHszCulrNJrJQhjTFBqSBvExyJyl4h0EZHk6kdjDygiCcDpOBMQoaoVqpoHTABmu5vNBi5s7DEarSALcrc4N8iB9WAyxgS1hrRBXO4+/6rWOqXx1U3dgWxglogMBFYC04F2qprlbrMbZ6rTnxCRabh3cnft2rWRIdThx2XOc7dTgVoJwu6iNsYEoYZMOdrdx+No2iLCgCHA06o6GCjmkOokd3BA9fFZVHWmqg5V1aFpaWlHEYYPRXud58RugJMgUmIjSIqNaNrjGGNMC9CQO6mv8bVeVV9s5DF3ADtU1f25zus4CWKPiHRQ1SwR6QDsbeT+G68s33mOdCYF2pRdRA+rXjLGBKmGtEH8rNZjJPBHYHxjD6iqu4EfReQEd9UYYD1OF9op7ropwILGHqPRyvIgPAbCIlBV6+JqjAlqDRms75bay24X1blHedxbgJdFJAJn6PDrcJLVPBG5HtgGXHaUxzhyZfkQlQDAvqIK8ksrrf3BGBO0GnN7cDFOQ3Ojqep3wFAfb405mv0etfKCmgRhPZiMMcGuIW0Q73CgwTgE6AvM82dQAVOrBLEp2xKEMSa4NaQE8Uit1x5gm6ru8FM8gVWWD7FOz6jNe4uIjQilQ0JUPR8yxpjWqSEJYjuQpaplACISLSLpqprp18gCoSwfknsAThVTj7ZtEJEAB2WMMYHRkF5MrwHeWstV7rrWp3YVk00zaowJcg1JEGGqWlG94L5ufXeOqdYkiMKySnYXlNk9EMaYoNaQBJEtIjX3PYjIBGCf/0IKkMoS8HogKoHN2cWANVAbY4JbQ9ogbsS5Z+Ef7vIOwOfd1S1a9V3UUQnWxdUYY2jYjXKbgVNEpI27XOT3qAKhrMB5jkpg049FhIcK3ZJjAhuTMcYEUL1VTCLyFxFJVNUiVS0SkSQR+VNzBNesDilBpKfEEhbakBo4Y4xpnRpyBTzXna8BAHeWt/P8F1KA1EoQm7NtDCZjjGlIgggVkcjqBRGJBiIPs33L5CaIivA2bMsptgRhjAl6DWmkfhlYLCKzAAGu5cDMb61HmVNI2l4cgVetgdoYYxrSSP2wiKwGzsIZk+lDoJu/A2t2bgkiI98pVPWwm+SMMUGuoa2we3CSw6XAmcAGv0UUKGX5EBbFf3MqEbEEYYwxdZYgROR44Ar3sQ94FRBVHd1MsTUvd6jvTdlFdEqMJjoiNNARGWNMQB2uimkjsBS4QFU3AYjI7c0SVSC4w2zYLHLGGOM4XBXTRUAWsEREnhGRMTiN1EdNRDJFZK2IfCciK9x1ySKySEQy3OekpjhWg5Xlo5HxbMm2QfqMMQYOkyBU9S1VnQT0BpYAtwFtReRpETm7CY49WlUHqWr1zHL3AItVtRew2F1uPmX5lIW2odzjtRKEMcbQgEZqVS1W1VdU9edAZ+Bb4G4/xDKBA91nZwMX+uEYdSvLpwBnaA1LEMYY0/BeTIBzF7WqzlTVo507WoGPRGSliExz17VT1Sz39W6gna8Pisg0EVkhIiuys7OPMoxayvLJ8zoJomuKjcFkjDENuVHOH0ao6k4RaQssEpGNtd9UVRUR9fVBVZ0JzAQYOnSoz20apSyffI0hRCAltvXdKG6MMUcqIKPRqepO93kvMB8YBuwRkQ4A7vPeZguosgyqKsitiialTSShITbNqDHGNHuCEJFYEYmrfg2cDawD3gamuJtNARY0W1DuXdTZlVGktbHSgzHGQGCqmNoB80Wk+vivqOoHIvINME9Erge2AZc1W0RugthTEUlqkiUIY4yBACQIVd0CDPSxPgc42sbvxnETxK7SCNK6WIIwxhgIUBvEMcdNED+WhZMWZwnCGGPAEoTDHeo7tyrGEoQxxrgsQUBNCaJALUEYY0w1SxBwIEEQa72YjDHGZQkCoLyAqpBwyrE2CGOMqWYJAqAsn4rQOEAsQRhjjMsSBEBZPqWhsUSEhRAfFajRR4wx5thiCQKgLJ8it/3BvYHPGGOCniUIqBmoL9Wql4wxpoYlCICyfPZ7o60HkzHG1GIJAqAsn32eaGugNsaYWixBAFpW4IzkagnCGGNqWILwlCOeUvLtLmpjjDmIJYiyAsDuojbGmENZgnCH2ShUa4MwxpjaLEHUGoeprSUIY4ypEbAEISKhIvKtiLzrLncXkWUisklEXhWRiGYJxB3qu0BjSLUqJmOMqRHIEsR0YEOt5YeBv6tqT2A/cH2zROGWIDwR8URHhDbLIY0xpiUISIIQkc7A+cCz7rIAZwKvu5vMBi5slmDKnUbqiNikZjmcMca0FIEqQTwG/BbwusspQJ6qetzlHUAnXx8UkWkiskJEVmRnZx99JG4JIqpN8tHvyxhjWpFmTxAicgGwV1VXNubzqjpTVYeq6tC0tLSjD6gsHw+hxMfHH/2+jDGmFQnE2NbDgfEich4QBcQDjwOJIhLmliI6AzubJZqyfAqJIS0+qlkOZ4wxLUWzlyBU9Xeq2llV04FJwCeqeiWwBLjE3WwKsKA54qkqySPfa3dRG2PMoY6l+yDuBu4QkU04bRLPNcdBK4r3U0CM3UVtjDGHCOj0aar6KfCp+3oLMKy5Y6gqyaPAxmEyxpifOJZKEIFRlk8BsXaTnDHGHCLoE0RIeYGVIIwxxoegTxBhlYUUEEtKm+YZ2cMYY1qK4E4QVZVEeEvxhLchPDS4vwpjjDlUcF8V3bkgNDIhwIEYY8yxJ8gThDOSa0h0YoADMcaYY0+QJwhnHKbwWEsQxhhzqKBOEOomiMg4G6jPGGMOFdQJoqxwPwDRliCMMeYngjpBFOXnANAm0RKEMcYcKqgTRElBLgAJiU0wbLgxxrQyQZ0gyotyqVIhOclKEMYYc6igThCekjybC8IYY+oQ1AlCS/MoIJbE6PBAh2KMMcecoE4QUl5AaUgsISES6FCMMeaYE9QJIqyigLLQuECHYYwxx6RmTxAiEiUiy0VktYh8LyIPuuu7i8gyEdkkIq+KiN+HVw33FOGJiPf3YYwxpkUKRAmiHDhTVQcCg4BxInIK8DDwd1XtCewHrvd3INFVRXgtQRhjjE/NniDUUeQuhrsPBc4EXnfXzwYu9GccXq8Sq8VItCUIY4zxJSBtECISKiLfAXuBRcBmIE9VPe4mO4BOdXx2moisEJEV2dnZjY5hf1EJcVJKqI3kaowxPgUkQahqlaoOAjoDw4DeR/DZmao6VFWHpqU1/g7onNx9AES0sZvkjDHGl4D2YlLVPGAJcCqQKCJh7ludgZ3+PHa+myBsoD5jjPEtEL2Y0kQk0X0dDYwFNuAkikvczaYAC/wZR6E7UF9sgiUIY4zxJaz+TZpcB2C2iITiJKh5qvquiKwH5orIn4Bvgef8GUSxO1BfXFKqPw9jjDEtVrMnCFVdAwz2sX4LTntEsygvdBJEdJuk5jqkMca0KEF7J3VFkTMftVgvJmOM8SloE0RVqZMgiLT7IIwxxpegTRCU5eNFLEEYY0wdgjZBhJbnUx4SCyFB+xUYY8xhBeXVscLjJbKqiMpwG8nVGGPqEpQJIqe4nDhKqLKB+owxpk5BmSD2FVYQLyVoVEKgQzHGmGNWUCaI7KIy4imxgfqMMeYwgjNBFJYTL8WEx1oJwhhj6hK8CYISIuwuamOMqVNQJoibzziOOCklLMYShDHG1CUQg/UFXEhFIaBgjdTGGFOnoCxBUJbvPFuCMMaYOgVngigvcJ4tQRhjTJ2CM0FYCcIYY+oViBnluojIEhFZLyLfi8h0d32yiCwSkQz32X8tyNUJwgbqM8aYOgWiBOEB7lTVvsApwK9EpC9wD7BYVXsBi91l/7AShDHG1KvZE4SqZqnqKvd1Ic581J2ACcBsd7PZwIV+C8IShDHG1CugbRAiko4z/egyoJ2qZrlv7Qba+e3AiV2h9wVWxWSMMYcRsPsgRKQN8AZwm6oWiEjNe6qqIqJ1fG4aMA2ga9eujTt47/OdhzHGmDoFpAQhIuE4yeFlVX3TXb1HRDq473cA9vr6rKrOVNWhqjo0LS2teQI2xpggFIheTAI8B2xQ1UdrvfU2MMV9PQVY0NyxGWOMOSAQVUzDgauBtSLynbvuXmAGME9Erge2AZcFIDZjjDGuZk8Qqvo5IHW8PaY5YzHGGFO34LyT2hhjTL0sQRhjjPHJEoQxxhifLEEYY4zxSVR93o/WIohINk6Pp8NJBfY1QzjHEjvn4GDnHBz8cc7dVLXeG8ladIJoCBFZoapDAx1Hc7JzDg52zsEhkOdsVUzGGGN8sgRhjDHGp2BIEDMDHUAA2DkHBzvn4BCwc271bRDGGGMaJxhKEMYYYxrBEoQxxhifWnWCEJFxIvKDiGwSEf/NcR1AIvK8iOwVkXW11iWLyCIRyXCfkwIZY1MSkS4iskRE1ovI9yIy3V3fms85SkSWi8hq95wfdNd3F5Fl7t/3qyISEehYm5qIhIrItyLyrrvcqs9ZRDJFZK2IfCciK9x1AfvbbrUJQkRCgaeAc4G+wBUi0jewUfnFC8C4Q9bdAyxW1V7AYne5tfAAd6pqX+AU4Ffuv2trPudy4ExVHQgMAsaJyCnAw8DfVbUnsB+4PoAx+st0nHnrqwXDOY9W1UG17n0I2N92q00QwDBgk6puUdUKYC4wIcAxNTlV/QzIPWT1BGC2+3o2cGGzBuVHqpqlqqvc14U4F49OtO5zVlUtchfD3YcCZwKvu+tb1TkDiEhn4HzgWXdZaOXnXIeA/W235gTRCfix1vIOd10waKeqWe7r3UC7QAbjLyKSDgwGltHKz9mtavkOZyreRcBmIE9VPe4mrfHv+zHgt4DXXU6h9Z+zAh+JyEoRmeauC9jfdiBmlDPNSFVVRFpdX2YRaYMzr/ltqlrg/Lh0tMZzVtUqYJCIJALzgd4BDsmvROQCYK+qrhSRUYGOpxmNUNWdItIWWCQiG2u/2dx/2625BLET6FJrubO7LhjsEZEOAO7z3gDH06REJBwnObysqm+6q1v1OVdT1TxgCXAqkCgi1T/yWtvf93BgvIhk4lQPnwk8Tus+Z1R1p/u8F+eHwDAC+LfdmhPEN0Avt9dDBDAJeDvAMTWXt4Ep7uspwIIAxtKk3Hro54ANqvporbda8zmnuSUHRCQaGIvT9rIEuMTdrFWds6r+TlU7q2o6zv/dT1T1SlrxOYtIrIjEVb8GzgbWEcC/7VZ9J7WInIdTjxkKPK+qfw5wSE1OROYAo3CGBN4DPAC8BcwDuuIMh36Zqh7akN0iicgIYCmwlgN10/fitEO01nMegNM4GYrzo26eqj4kIsfh/LpOBr4FrlLV8sBF6h9uFdNdqnpBaz5n99zmu4thwCuq+mcRSSFAf9utOkEYY4xpvNZcxWSMMeYoWIIwxhjjkyUIY4wxPlmCMMYY45MlCGOMMT5ZgjDGBxGpckfUrH402QBpIpJee/RdY45VNtSGMb6VquqgQAdhTCBZCcKYI+CO1/9Xd8z+5SLS012fLiKfiMgaEVksIl3d9e1EZL47l8NqETnN3VWoiDzjzu/wkXuHNCJyqzvXxRoRmRug0zQGsARhTF2iD6liurzWe/mqeiLwD5w79QGeBGar6gDgZeAJd/0TwH/cuRyGAN+763sBT6lqPyAPuNhdfw8w2N3Pjf46OWMawu6kNsYHESlS1TY+1mfiTN6zxR00cLeqpojIPqCDqla667NUNVVEsoHOtYeDcIcpX+ROAIOI3A2Eq+qfROQDoAhnuJS3as0DYUyzsxKEMUdO63h9JGqPH1TFgfbA83FmQhwCfFNr5FJjmp0lCGOO3OW1nr9yX3+JM+oowJU4AwqCM0XkTVAz6U9CXTsVkRCgi6ouAe4GEoCflGKMaS7268QY36LdGdyqfaCq1V1dk0RkDU4p4Ap33S3ALBH5DZANXOeunw7MFJHrcUoKNwFZ+BYK/NtNIgI84c7/YExAWBuEMUfAbYMYqqr7Ah2LMf5mVUzGGGN8shKEMcYYn6wEYYwxxidLEMYYY3yyBGGMMcYnSxDGGGN8sgRhjDHGp/8PJrHSrDLNFwoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f75cbf956d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot3 = plt.plot(epochs, train_accuracy)\n",
    "plot4 = plt.plot(epochs, val_accuracy)\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.title(\"Accuracy\")\n",
    "plt.legend([\"Training Accuracy\", \"Validation Accuracy\"], loc = 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit your results on Kaggle\n",
    "\n",
    "### Train a better model for action recognition!\n",
    "Now it's your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves better accuracy on the action recognition validation set.\n",
    "\n",
    "\n",
    "### Testing the model and submit on Kaggle\n",
    "Testing the model on the testing set and save the results as a .csv file. \n",
    "Please submitted the results.csv file generated by predict_on_test() to Kaggle(https://www.kaggle.com/c/cse512springhw5) to see how well your network performs on the test set. \n",
    "################ 3rd To Do  (30 points, the highest 3 entries get extra 10 points) ###############\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n"
     ]
    }
   ],
   "source": [
    "# Use your best model to generate results on test set.\n",
    "\n",
    "# generate csv file for test set\n",
    "def predict_on_test(model, data_loader):\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    results=open(\"results.csv\",'w')\n",
    "    count=0\n",
    "    results.write('Id'+','+'Class'+'\\n')\n",
    "    for batch_idx, sample in enumerate(data_loader):\n",
    "        sequence = sample['seq']\n",
    "        input_sequence_var = Variable(sequence).type(FloatTensor)\n",
    "        scores = model(input_sequence_var)\n",
    "        _, preds = scores.data.max(1)\n",
    "        for i in range(len(preds)):\n",
    "            results.write(str(count)+','+str(preds[i])+'\\n')\n",
    "            count+=1\n",
    "    results.close()\n",
    "    return count\n",
    "\n",
    "count = predict_on_test(model, tstLD)\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Report the performance\n",
    "################ 4th To Do  (15 points)################## <br>\n",
    "Accuracy on Validation set achieved is 87%\n",
    "\n",
    "Accuracy on Training set achieved is 100%\n",
    "\n",
    "Loss on Validation set after 50th epoch is 0.5605\n",
    "\n",
    "Loss on Training set after 50th epoch is 0.0034\n",
    "### Documentation of what you did\n",
    "In this cell, you should write an explanation of what you did (network architecture, optimiziter, learning rate, epoches) and visualizations or graphs of loss/accuracy curve tin the process of training and evaluating.\n",
    "\n",
    "Network Architecture: 2 layers of Convolution Neural Network along with 2 layers of Inception, upon it a single layer of LSTM with 50 hidden states.\n",
    "\n",
    "Optimizer: Adadelta optimizer used because it reduces the learning rate dynamically with decaying rate 0.9\n",
    "\n",
    "Learning Rate: 1e-1\n",
    "Epochs used: 50\n",
    "\n",
    "### performance on Kaggle\n",
    "You should also report your Kaggle Performance here:\n",
    "Rank 2 with score 0.86900"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}