ULMFiT - Sentiment Classification - Twitter US Airlines Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Classification of US Airline Tweets <a name=\"top\"></a>\n",
    "\n",
    "The purpose of this notebook is to apply a [ULMFiT](https://arxiv.org/abs/1801.06146) model to Twitter US Airlines Sentiment data.\n",
    "\n",
    "In particular, I have chosen to focus on the sentiment classification task. We could instead investigate other subjects, such as classifying the negative reason or predicting whether a tweet will be retweeted, however I will limit my discussion of other models to the data exploration section.\n",
    "\n",
    "At a high level, after preparing the notebook and exploring the data, we fine-tune [fastai](https://www.fast.ai/about/)'s general [WikiText-103 language model](http://files.fast.ai/models/wt103/) on the tweet data. Then we transfer the fine-tuned word embedding layer (i.e. encoder) to a new language classification model which we train to predict whether the tweet expresses a *negative*, *neutral*, or *positive* sentiment. Finally, we check our work, investigate sources of error, and propose potential next steps to take for improving the model further.\n",
    "\n",
    "A baseline model for this task, which always predicts *negative* (the most common class), has an accuracy of 62.69%. The best two results I found in an online search for public sentiment classification models on this dataset are:\n",
    "\n",
    "* [Stanford 2016](http://cs229.stanford.edu/proj2016spr/report/042.pdf) - an SVM (linear kernel) with 80% accuracy\n",
    "* [jiashenliu 2017](https://www.kaggle.com/jiashenliu/how-can-we-predict-the-sentiment-by-tweets) - a random forest with 81.3% accuracy\n",
    "\n",
    "After training, our model has a final accuracy of 84.8%. This could likely be improved by continuing to optimize hyperparameters, including more data, including more features from the data, and/or fixing some of the mislabled data.\n",
    "\n",
    "To streamline this notebook, I've commented out some sections which list many lines of output. Wherever this is done, I've written about my observations immediately below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Contents:\n",
    "0. [Notebook Setup](#0)\n",
    "0. [Load, Analyze, and Prepare the Data](#1)\n",
    "   1. sentiment label exploration: [airline_sentiment](#airline_sentiment)\n",
    "   1. negative reason labels exploration: [negativereason](#negativereason)\n",
    "   1. language data exploration: [text](#text)\n",
    "   1. location data exploration: [tweet_coord, tweet_location, user_timezone](#tweet_coord)\n",
    "   1. time data exploration: [tweet_created, user_timezone](#tweet_created)\n",
    "   1. other data exploration: [retweet_count](#retweet_count)\n",
    "   1. prepare the [training, validation, and test sets](#prepare-datasets)\n",
    "0. [Language Model Fine-Tuning](#2)\n",
    "0. [Classification Model Training](#3)\n",
    "0. [Analysis and Discussion](#4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Notebook Setup \n",
    "<a name=\"0\">[(... back to the top)](#top)</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the command below setup your environment with Anaconda. If you're using a GPU, you may need to modify this depending on your CUDA Toolkit version - see [PyTorch](https://pytorch.org/get-started/locally/) for help with this.\n",
    "\n",
    "`conda create -n fastai-py3.7 -c pytorch -c fastai fastai jupyter notebook python=3.7`\n",
    "\n",
    "The following cell prepares the the notebook to be restarted from any major section (1, 2, 3, or 4). **Always run this cell:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reloads modules without having to restart the kernel\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# automatically displays plots\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai.text import *     # implementation of ULMFiT in PyTorch + other NLP tools\n",
    "import pytz                   # handle timezones\n",
    "from datetime import datetime # handle dates\n",
    "from pprint import pprint     # nicely formatted printing\n",
    "\n",
    "# prefix given to filenames for keeping track of various trials\n",
    "trial_name = ''\n",
    "\n",
    "# project path\n",
    "path = Path('.')\n",
    "path.mkdir(exist_ok=True)\n",
    "\n",
    "# columns used in classification tasks (the first column must be the label)\n",
    "model_input_cols = ['text_len', 'retweet_count', 'tweet_createdLocalHour', 'text']\n",
    "cols = ['airline_sentiment'] + model_input_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load, Analyze, and Prepare the Data <a name=\"1\"></a>\n",
    "<a name=\"0\">[(... back to the top)](#top)</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, a bit of context. This data was compiled during one week in February 2015 from public tweets that mention any of the following 6 US airlines: American, Delta, Southwest, United, US Airways, and Virgin America.\n",
    "\n",
    "To get started, download the Twitter US Airline Sentiment data `Tweets.csv` from [Kaggle](https://www.kaggle.com/crowdflower/twitter-airline-sentiment) and put it in your `path` directory.\n",
    "\n",
    "Load it into pandas to start exploring:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "N Rows: 14640\n"
     ]
    }
   ],
   "source": [
    "df_raw = pd.read_csv(path / 'Tweets.csv')\n",
    "print('N Rows:', len(df_raw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a few rows of data to learn what information this dataset contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>570306133677760513</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cairdin</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:35:52 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>570301130888122368</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:59 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>570301083672813571</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>yvonnalynn</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:48 -0800</td>\n",
       "      <td>Lets Play</td>\n",
       "      <td>Central Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>570301031407624196</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.7033</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:36 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             tweet_id airline_sentiment  airline_sentiment_confidence  \\\n",
       "0  570306133677760513           neutral                        1.0000   \n",
       "1  570301130888122368          positive                        0.3486   \n",
       "2  570301083672813571           neutral                        0.6837   \n",
       "3  570301031407624196          negative                        1.0000   \n",
       "\n",
       "  negativereason  negativereason_confidence         airline  \\\n",
       "0            NaN                        NaN  Virgin America   \n",
       "1            NaN                     0.0000  Virgin America   \n",
       "2            NaN                        NaN  Virgin America   \n",
       "3     Bad Flight                     0.7033  Virgin America   \n",
       "\n",
       "  airline_sentiment_gold        name negativereason_gold  retweet_count  \\\n",
       "0                    NaN     cairdin                 NaN              0   \n",
       "1                    NaN    jnardino                 NaN              0   \n",
       "2                    NaN  yvonnalynn                 NaN              0   \n",
       "3                    NaN    jnardino                 NaN              0   \n",
       "\n",
       "                                                text tweet_coord  \\\n",
       "0                @VirginAmerica What @dhepburn said.         NaN   \n",
       "1  @VirginAmerica plus you've added commercials t...         NaN   \n",
       "2  @VirginAmerica I didn't today... Must mean I n...         NaN   \n",
       "3  @VirginAmerica it's really aggressive to blast...         NaN   \n",
       "\n",
       "               tweet_created tweet_location               user_timezone  \n",
       "0  2015-02-24 11:35:52 -0800            NaN  Eastern Time (US & Canada)  \n",
       "1  2015-02-24 11:15:59 -0800            NaN  Pacific Time (US & Canada)  \n",
       "2  2015-02-24 11:15:48 -0800      Lets Play  Central Time (US & Canada)  \n",
       "3  2015-02-24 11:15:36 -0800            NaN  Pacific Time (US & Canada)  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw.iloc[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now begin classifying our columns:\n",
    "* natural language: \n",
    "  * 'text'\n",
    "* prediction targets: \n",
    "  * 'airline_sentiment', 'negativereason'\n",
    "* loss importance / confidence in the labels:\n",
    "  * 'airline_sentiment_confidence', 'negativereason_confidence'\n",
    "* user location: \n",
    "  * 'tweet_coord', 'tweet_location', 'user_timezone'\n",
    "* time:\n",
    "  * 'tweet_created'\n",
    "* numeric data: \n",
    "  * 'retweet_count'\n",
    "* unclear: \n",
    "  * 'airline_sentiment_gold', 'negativereason_gold'\n",
    "* irrelevant to future predictions:\n",
    "  * 'tweet_id', 'name', 'airline'\n",
    "  \n",
    "Most importantly, we need to make sure our language data and prediction targets are clean. The location and additional data columns warrant further investigation as they may be able to add additional predictive power to a model. The 'airline' column is listed as irrelevant because the tweets mention the airlines directly in the text.\n",
    "\n",
    "Let's start by looking at our labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>negativereason</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>14640</td>\n",
       "      <td>9178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>negative</td>\n",
       "      <td>Customer Service Issue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>9178</td>\n",
       "      <td>2910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       airline_sentiment          negativereason\n",
       "count              14640                    9178\n",
       "unique                 3                      10\n",
       "top             negative  Customer Service Issue\n",
       "freq                9178                    2910"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw[['airline_sentiment', 'negativereason']].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9178"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_raw[(df_raw.airline_sentiment == 'negative') & (df_raw.negativereason.notnull())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that all rows have sentiment and negative reasons only apply to negative sentiment labels.\n",
    "\n",
    "Let's take a closer look at our sentiment classification labels. <a name=\"airline_sentiment\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unique_cat_stats(df, colname):\n",
    "    \"\"\"For a categorical column, return the unique values as counts and as a percent of total rows.\"\"\"\n",
    "    series = df.groupby(colname).count().text\n",
    "    res = pd.concat([series, series / len(df) * 100], axis=1)\n",
    "    res.columns = ['raw counts', '% of rows']\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw counts</th>\n",
       "      <th>% of rows</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>9178</td>\n",
       "      <td>62.691257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>3099</td>\n",
       "      <td>21.168033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>2363</td>\n",
       "      <td>16.140710</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   raw counts  % of rows\n",
       "airline_sentiment                       \n",
       "negative                 9178  62.691257\n",
       "neutral                  3099  21.168033\n",
       "positive                 2363  16.140710"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_cat_stats(df_raw, 'airline_sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see here that there are 3 classes of airline sentiment, cleanly labeled. Now let's take a look at the 'airline_sentiment_confidence'. <a name=\"airline_sentiment_confidence\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "min: 0.335 max: 1.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAQSElEQVR4nO3df4xl5V3H8fdHtvQ3ZekODe6uDk23WkpiihPY2qTWboWFGpY/wGxjZUs2blKx1tqoVP9YAyUBf6EkLXUta5emFhAb2bRUsgGaqimUoVTKD8mOgDCCZeoua5X0x7Zf/7jP4u1yZ+fO3Jk7O7PvVzK55zznOed+n51hPnOec+4hVYUk6dj2Y4tdgCRp8RkGkiTDQJJkGEiSMAwkScCKxS5grlatWlWjo6OLXYYkLRn33Xfft6pqpNe2JRsGo6OjjI+PL3YZkrRkJPn36bY5TSRJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJJbwJ5Al6VgxetkXXlh+4qp3L8h7eGYgSTIMJEmGgSQJw0CSRB9hkGRnkmeTPNjVdlKSPUn2tteVrT1Jrk0ykeSBJGd07bOl9d+bZEtX+88m+Ubb59okme9BSpKOrJ8zg08BGw9ruwy4o6rWAXe0dYBzgXXtaxtwHXTCA9gOnAWcCWw/FCCtz7au/Q5/L0nSApsxDKrqy8C+w5o3Abva8i7ggq72G6rjbuDEJKcA5wB7qmpfVe0H9gAb27YTquorVVXADV3HkiQNyVyvGbyuqp4BaK8nt/bVwFNd/SZb25HaJ3u0S5KGaL4vIPea7685tPc+eLItyXiS8ampqTmWKEk63FzD4Jttiof2+mxrnwTWdvVbAzw9Q/uaHu09VdWOqhqrqrGRkZ7/T2dJ0hzMNQx2A4fuCNoC3NrVfnG7q2g9cKBNI90OnJ1kZbtwfDZwe9v27STr211EF3cdS5I0JDM+myjJZ4F3AKuSTNK5K+gq4OYkW4EngYta99uA84AJ4HngEoCq2pfkCuDe1u/yqjp0Ufr9dO5YejnwxfYlSRqiGcOgqt4zzaYNPfoWcOk0x9kJ7OzRPg6cPlMdkqSF4yeQJUmGgSTJMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEgOGQZIPJXkoyYNJPpvkZUlOTXJPkr1JbkpyfOv70rY+0baPdh3nI6390STnDDYkSdJszTkMkqwGfhMYq6rTgeOAzcDVwDVVtQ7YD2xtu2wF9lfVG4BrWj+SnNb2ezOwEfh4kuPmWpckafYGnSZaAbw8yQrgFcAzwDuBW9r2XcAFbXlTW6dt35Akrf3GqvpuVT0OTABnDliXJGkW5hwGVfUfwJ8AT9IJgQPAfcBzVXWwdZsEVrfl1cBTbd+Drf9ru9t77PMjkmxLMp5kfGpqaq6lS5IOM8g00Uo6f9WfCvw48Erg3B5d69Au02ybrv3FjVU7qmqsqsZGRkZmX7QkqadBponeBTxeVVNV9X3gc8DPASe2aSOANcDTbXkSWAvQtr8G2Nfd3mMfSdIQDBIGTwLrk7yizf1vAB4G7gIubH22ALe25d1tnbb9zqqq1r653W10KrAO+OoAdUmSZmnFzF16q6p7ktwCfA04CNwP7AC+ANyY5KOt7fq2y/XAp5NM0Dkj2NyO81CSm+kEyUHg0qr6wVzrkiTN3pzDAKCqtgPbD2t+jB53A1XVd4CLpjnOlcCVg9QiSZo7P4EsSTIMJEmGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSGDAMkpyY5JYk/5rkkSRvTXJSkj1J9rbXla1vklybZCLJA0nO6DrOltZ/b5Itgw5KkjQ7g54Z/AXwD1X108DPAI8AlwF3VNU64I62DnAusK59bQOuA0hyErAdOAs4E9h+KEAkScMx5zBIcgLwduB6gKr6XlU9B2wCdrVuu4AL2vIm4IbquBs4MckpwDnAnqraV1X7gT3AxrnWJUmavUHODF4PTAF/neT+JJ9M8krgdVX1DEB7Pbn1Xw081bX/ZGubrv1FkmxLMp5kfGpqaoDSJUndBgmDFcAZwHVV9Rbgf/n/KaFe0qOtjtD+4saqHVU1VlVjIyMjs61XkjSNQcJgEpisqnva+i10wuGbbfqH9vpsV/+1XfuvAZ4+QrskaUjmHAZV9Z/AU0l+qjVtAB4GdgOH7gjaAtzalncDF7e7itYDB9o00u3A2UlWtgvHZ7c2SdKQrBhw/w8An0lyPPAYcAmdgLk5yVbgSeCi1vc24DxgAni+9aWq9iW5Ari39bu8qvYNWJckaRYGCoOq+jow1mPThh59C7h0muPsBHYOUoskae78BLIkyTCQJBkGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAksQ8hEGS45Lcn+Tzbf3UJPck2ZvkpiTHt/aXtvWJtn206xgfae2PJjln0JokSbMzH2cGHwQe6Vq/GrimqtYB+4GtrX0rsL+q3gBc0/qR5DRgM/BmYCPw8STHzUNdkqQ+DRQGSdYA7wY+2dYDvBO4pXXZBVzQlje1ddr2Da3/JuDGqvpuVT0OTABnDlKXJGl2Bj0z+HPgd4EftvXXAs9V1cG2PgmsbsurgacA2vYDrf8L7T32+RFJtiUZTzI+NTU1YOmSpEPmHAZJfgl4tqru627u0bVm2HakfX60sWpHVY1V1djIyMis6pUkTW/FAPu+DTg/yXnAy4AT6JwpnJhkRfvrfw3wdOs/CawFJpOsAF4D7OtqP6R7H0nSEMz5zKCqPlJVa6pqlM4F4Dur6leAu4ALW7ctwK1teXdbp22/s6qqtW9udxudCqwDvjrXuiRJszfImcF0fg+4MclHgfuB61v79cCnk0zQOSPYDFBVDyW5GXgYOAhcWlU/WIC6JEnTmJcwqKovAV9qy4/R426gqvoOcNE0+18JXDkftUiSZs9PIEuSDANJkmEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQGCIMka5PcleSRJA8l+WBrPynJniR72+vK1p4k1yaZSPJAkjO6jrWl9d+bZMvgw5IkzcYgZwYHgQ9X1ZuA9cClSU4DLgPuqKp1wB1tHeBcYF372gZcB53wALYDZwFnAtsPBYgkaTjmHAZV9UxVfa0tfxt4BFgNbAJ2tW67gAva8ibghuq4GzgxySnAOcCeqtpXVfuBPcDGudYlSZq9eblmkGQUeAtwD/C6qnoGOoEBnNy6rQae6tptsrVN197rfbYlGU8yPjU1NR+lS5KYhzBI8irg74Dfqqr/PlLXHm11hPYXN1btqKqxqhobGRmZfbGSpJ4GCoMkL6ETBJ+pqs+15m+26R/a67OtfRJY27X7GuDpI7RLkoZkkLuJAlwPPFJVf9a1aTdw6I6gLcCtXe0Xt7uK1gMH2jTS7cDZSVa2C8dntzZJ0pCsGGDftwG/Cnwjyddb2+8DVwE3J9kKPAlc1LbdBpwHTADPA5cAVNW+JFcA97Z+l1fVvgHqkiTN0pzDoKr+id7z/QAbevQv4NJpjrUT2DnXWiRJg/ETyJIkw0CSZBhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkScCKxS5A0tFn9LIv9Gx/4qp3D7kSDYthIAmYPgB0bHCaSJLkmYGk/nWfPThltLx4ZiBJMgwkSU4TScc0LxrrEM8MJEmeGUiaGy8mLy+GgXSMcWpIvRgGkgbmWcLSd9RcM0iyMcmjSSaSXLbY9UjSseSoODNIchzwMeAXgUng3iS7q+rhxa1Mw+ZfmPNv2NNCfg+XpqMiDIAzgYmqegwgyY3AJsAwOAZM98uqn19i/rLp7Wi5LmAwLB1HSxisBp7qWp8Ezjq8U5JtwLa2+j9JHp3j+60CvjXHfZeqZTnmXH3EzctyzEdwVI93hu/VXB3VY14IuXqgMf/kdBuOljBIj7Z6UUPVDmDHwG+WjFfV2KDHWUoc8/J3rI0XHPN8OlouIE8Ca7vW1wBPL1ItknTMOVrC4F5gXZJTkxwPbAZ2L3JNknTMOCqmiarqYJLfAG4HjgN2VtVDC/iWA081LUGOefk71sYLjnnepOpFU/OSpGPM0TJNJElaRIaBJGl5h8FMj7hI8tIkN7Xt9yQZHX6V86eP8f52koeTPJDkjiTT3nO8VPT7GJMkFyapJEv+NsR+xpzkl9v3+qEkfzPsGudbHz/bP5HkriT3t5/v8xajzvmSZGeSZ5M8OM32JLm2/Xs8kOSMgd+0qpblF50L0f8GvB44HvgX4LTD+vw68Im2vBm4abHrXuDx/gLwirb8/qU83n7H3Pq9GvgycDcwtth1D+H7vA64H1jZ1k9e7LqHMOYdwPvb8mnAE4td94BjfjtwBvDgNNvPA75I5zNa64F7Bn3P5Xxm8MIjLqrqe8ChR1x02wTsasu3ABuS9PoA3FIw43ir6q6qer6t3k3n8xxLWT/fY4ArgD8CvjPM4hZIP2P+NeBjVbUfoKqeHXKN862fMRdwQlt+DUv8c0pV9WVg3xG6bAJuqI67gROTnDLIey7nMOj1iIvV0/WpqoPAAeC1Q6lu/vUz3m5b6fxlsZTNOOYkbwHWVtXnh1nYAurn+/xG4I1J/jnJ3Uk2Dq26hdHPmP8QeG+SSeA24APDKW3RzPa/9xkdFZ8zWCD9POKir8dgLBF9jyXJe4Ex4OcXtKKFd8QxJ/kx4BrgfcMqaAj6+T6voDNV9A46Z3//mOT0qnpugWtbKP2M+T3Ap6rqT5O8Ffh0G/MPF768RTHvv7uW85lBP4+4eKFPkhV0Ti+PdGp2NOvrkR5J3gX8AXB+VX13SLUtlJnG/GrgdOBLSZ6gM7e6e4lfRO735/rWqvp+VT0OPEonHJaqfsa8FbgZoKq+AryMzkPslqt5f4TPcg6Dfh5xsRvY0pYvBO6sdnVmCZpxvG3K5C/pBMFSn0eGGcZcVQeqalVVjVbVKJ3rJOdX1fjilDsv+vm5/ns6NwuQZBWdaaPHhlrl/OpnzE8CGwCSvIlOGEwNtcrh2g1c3O4qWg8cqKpnBjngsp0mqmkecZHkcmC8qnYD19M5nZygc0awefEqHkyf4/1j4FXA37br5E9W1fmLVvSA+hzzstLnmG8Hzk7yMPAD4Heq6r8Wr+rB9DnmDwN/leRDdKZL3reE/7AjyWfpTPOtatdBtgMvAaiqT9C5LnIeMAE8D1wy8Hsu4X8vSdI8Wc7TRJKkPhkGkiTDQJJkGEiSMAwkSRgGkiQMA0kS8H+Nz8a2D00+IQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('min:', df_raw.airline_sentiment_confidence.min(), 'max:', df_raw.airline_sentiment_confidence.max())\n",
    "plt.hist(df_raw.airline_sentiment_confidence, bins=np.linspace(0,1,101));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_low_confidence:    236 \n",
      "n_med_confidence:   3945 \n",
      "n_high_confidence: 10459\n"
     ]
    }
   ],
   "source": [
    "low_confidence  = df_raw[df_raw.airline_sentiment_confidence < 0.5]\n",
    "med_confidence  = df_raw[(0.5 < df_raw.airline_sentiment_confidence) & (df_raw.airline_sentiment_confidence < 0.8)]\n",
    "high_confidence = df_raw[0.8 < df_raw.airline_sentiment_confidence]\n",
    "print('n_low_confidence:   ', len(low_confidence), '\\nn_med_confidence:  ', len(med_confidence),\n",
    "      '\\nn_high_confidence:', len(high_confidence))\n",
    "#for r in low_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].airline_sentiment, r[1].text)\n",
    "#for r in med_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].airline_sentiment, r[1].text)\n",
    "#for r in high_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].airline_sentiment, r[1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three label confidence groups clearly stand out. Less than 2% of the labels have a confidence of ~33% and I disagree with the sentiment label on 12 out of 20 randomly selected low confidence labels. With such poor confidence, our predictions would likely be improved by removing these rows from our classification training set. As for the medium confidence labels, which make up about 1/3 of the data, I disagree with 4 out of 20 random labels. It might be helpful to reduce importance of these rows to the training loss. I disagree with 1 out of 20 random high confidence labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "airline_sentiment\n",
       "negative     63\n",
       "neutral     102\n",
       "positive     71\n",
       "Name: text, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_confidence.groupby('airline_sentiment').count().text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that by discarding low confidence rows, we would be removing fewer rows with negative sentiment than neutral and positive, thus slightly increasing our bias towards negative rows. This also indicates that it may be slightly harder for humans to correctly classify neutral and positive sentiments compared to negative sentiments.\n",
    "\n",
    "We now have enough information to describe our sentiment classification task:\n",
    "- This is a **3-way classification task**, with categories: *negative*, *neutral*, and *positive*.\n",
    "- This is an unbalanced dataset, biased towards the *negative* class.\n",
    "- Our naive baseline model, which always predicts *negative*, has an accuracy of 62.69%.\n",
    "\n",
    "Next, let's take a look at our negative reason classification task. <a name=\"negativereason\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw counts</th>\n",
       "      <th>% of rows</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>negativereason</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Bad Flight</th>\n",
       "      <td>580</td>\n",
       "      <td>6.319460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Can't Tell</th>\n",
       "      <td>1190</td>\n",
       "      <td>12.965788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cancelled Flight</th>\n",
       "      <td>847</td>\n",
       "      <td>9.228590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Customer Service Issue</th>\n",
       "      <td>2910</td>\n",
       "      <td>31.706254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Damaged Luggage</th>\n",
       "      <td>74</td>\n",
       "      <td>0.806276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flight Attendant Complaints</th>\n",
       "      <td>481</td>\n",
       "      <td>5.240793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flight Booking Problems</th>\n",
       "      <td>529</td>\n",
       "      <td>5.763783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Late Flight</th>\n",
       "      <td>1665</td>\n",
       "      <td>18.141207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lost Luggage</th>\n",
       "      <td>724</td>\n",
       "      <td>7.888429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>longlines</th>\n",
       "      <td>178</td>\n",
       "      <td>1.939420</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             raw counts  % of rows\n",
       "negativereason                                    \n",
       "Bad Flight                          580   6.319460\n",
       "Can't Tell                         1190  12.965788\n",
       "Cancelled Flight                    847   9.228590\n",
       "Customer Service Issue             2910  31.706254\n",
       "Damaged Luggage                      74   0.806276\n",
       "Flight Attendant Complaints         481   5.240793\n",
       "Flight Booking Problems             529   5.763783\n",
       "Late Flight                        1665  18.141207\n",
       "Lost Luggage                        724   7.888429\n",
       "longlines                           178   1.939420"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_cat_stats(df_raw[df_raw.airline_sentiment == 'negative'], 'negativereason')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 10 categories of cleanly-labelled negative reasons if we disregard null rows (i.e. tweets with neutral and positive sentiment).\n",
    "\n",
    "As with the sentiment labels, the negative reason labels also have a confidence associated with them. <a name=\"negativereason_confidence\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "min: 0.3122 max: 1.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAATLElEQVR4nO3cf4xl5X3f8ffHyw+ntRuWMFjb3W2XpmvV2FIWNAUqSy0xDixE8jqSXS1SwgahbppClbRWFJz+gWMXyW7rICE5pGuxNUSJMXF+sCKb0i3GchyVH0OM1ywEMQEKk12xkywmsVBood/+cZ+1rpf5cWdn5i6zz/slXd1zvuc59zwPO3zmzHPOPakqJEl9eMep7oAkaXwMfUnqiKEvSR0x9CWpI4a+JHXkjFPdgYWcd955tWXLllPdDUlaUx5//PG/rKqJuba9rUN/y5YtTE1NnepuSNKakuR/z7fN6R1J6oihL0kdWTT0k7wzyaNJvp3kUJJfbfUvJXk+yRPtta3Vk+T2JNNJDia5eOizdiV5tr12rd6wJElzGWVO/3XgQ1X1vSRnAt9M8kdt2y9V1VdPaH81sLW9LgXuAC5Nci5wCzAJFPB4kn1V9cpKDESStLhFz/Rr4Htt9cz2WuiBPTuAu9t+DwPnJNkAXAUcqKpjLegPANuX131J0lKMNKefZF2SJ4CjDIL7kbbp1jaFc1uSs1ttI/DS0O4zrTZf/cRj7U4ylWRqdnZ2icORJC1kpNCvqjerahuwCbgkyQeATwL/BPinwLnAL7fmmesjFqifeKw9VTVZVZMTE3PeZipJOklLununqr4LfB3YXlVH2hTO68B/Ay5pzWaAzUO7bQIOL1CXJI3JKHfvTCQ5py3/EPBh4M/aPD1JAnwUeLLtsg+4rt3FcxnwalUdAR4ArkyyPsl64MpWkySNySh372wA7kqyjsEviXur6v4kX0sywWDa5gngX7f2+4FrgGngNeB6gKo6luQzwGOt3aer6tjKDUWS1rYtN//h95df+OxPrsoxFg39qjoIXDRH/UPztC/gxnm27QX2LrGPkqQV4jdyJakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjqyaOgneWeSR5N8O8mhJL/a6hckeSTJs0m+kuSsVj+7rU+37VuGPuuTrf5MkqtWa1CSpLmNcqb/OvChqvoxYBuwPcllwOeA26pqK/AKcENrfwPwSlX9Y+C21o4kFwI7gfcD24FfT7JuJQcjSVrYoqFfA99rq2e2VwEfAr7a6ncBH23LO9o6bfsVSdLq91TV61X1PDANXLIio5AkjWSkOf0k65I8ARwFDgB/Dny3qt5oTWaAjW15I/ASQNv+KvAjw/U59hk+1u4kU0mmZmdnlz4iSdK8Rgr9qnqzqrYBmxicnb9vrmbtPfNsm69+4rH2VNVkVU1OTEyM0j1J0oiWdPdOVX0X+DpwGXBOkjPapk3A4bY8A2wGaNt/GDg2XJ9jH0nSGIxy985EknPa8g8BHwaeBh4CPtaa7QLua8v72jpt+9eqqlp9Z7u75wJgK/DoSg1EkrS4MxZvwgbgrnanzTuAe6vq/iRPAfck+Y/At4A7W/s7gd9MMs3gDH8nQFUdSnIv8BTwBnBjVb25ssORJC1k0dCvqoPARXPUn2OOu2+q6m+Bj8/zWbcCty69m5KkleA3ciWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6smjoJ9mc5KEkTyc5lOQXWv1TSf4iyRPtdc3QPp9MMp3kmSRXDdW3t9p0kptXZ0iSpPmcMUKbN4BPVNWfJnk38HiSA23bbVX1X4YbJ7kQ2Am8H/j7wP9M8t62+QvATwAzwGNJ9lXVUysxEEnS4hYN/ao6Ahxpy3+T5Glg4wK77ADuqarXgeeTTAOXtG3TVfUcQJJ7WltDX5LGZElz+km2ABcBj7TSTUkOJtmbZH2rbQReGtptptXmq594jN1JppJMzc7OLqV7kqRFjBz6Sd4F/C7wi1X118AdwI8C2xj8JfD5403n2L0WqP9goWpPVU1W1eTExMSo3ZMkjWCUOX2SnMkg8H+rqn4PoKpeHtr+ReD+tjoDbB7afRNwuC3PV5ckjcEod+8EuBN4uqp+bai+YajZTwFPtuV9wM4kZye5ANgKPAo8BmxNckGSsxhc7N23MsOQJI1ilDP9DwI/A3wnyROt9ivAtUm2MZiieQH4OYCqOpTkXgYXaN8AbqyqNwGS3AQ8AKwD9lbVoRUciyRpEaPcvfNN5p6P37/APrcCt85R37/QfpKk1eU3ciWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6smjoJ9mc5KEkTyc5lOQXWv3cJAeSPNve17d6ktyeZDrJwSQXD33Wrtb+2SS7Vm9YkqS5jHKm/wbwiap6H3AZcGOSC4GbgQeraivwYFsHuBrY2l67gTtg8EsCuAW4FLgEuOX4LwpJ0ngsGvpVdaSq/rQt/w3wNLAR2AHc1ZrdBXy0Le8A7q6Bh4FzkmwArgIOVNWxqnoFOABsX9HRSJIWtKQ5/SRbgIuAR4D3VNURGPxiAM5vzTYCLw3tNtNq89VPPMbuJFNJpmZnZ5fSPUnSIkYO/STvAn4X+MWq+uuFms5RqwXqP1io2lNVk1U1OTExMWr3JEkjGCn0k5zJIPB/q6p+r5VfbtM2tPejrT4DbB7afRNweIG6JGlMRrl7J8CdwNNV9WtDm/YBx+/A2QXcN1S/rt3Fcxnwapv+eQC4Msn6dgH3ylaTJI3JGSO0+SDwM8B3kjzRar8CfBa4N8kNwIvAx9u2/cA1wDTwGnA9QFUdS/IZ4LHW7tNVdWxFRiFJGsmioV9V32Tu+XiAK+ZoX8CN83zWXmDvUjooSVo5fiNXkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOLhn6SvUmOJnlyqPapJH+R5In2umZo2yeTTCd5JslVQ/XtrTad5OaVH4okaTGjnOl/Cdg+R/22qtrWXvsBklwI7ATe3/b59STrkqwDvgBcDVwIXNvaSpLG6IzFGlTVN5JsGfHzdgD3VNXrwPNJpoFL2rbpqnoOIMk9re1TS+6xJOmkLWdO/6YkB9v0z/pW2wi8NNRmptXmq79Fkt1JppJMzc7OLqN7kqQTnWzo3wH8KLANOAJ8vtUzR9taoP7WYtWeqpqsqsmJiYmT7J4kaS6LTu/MpapePr6c5IvA/W11Btg81HQTcLgtz1eXJI3JSZ3pJ9kwtPpTwPE7e/YBO5OcneQCYCvwKPAYsDXJBUnOYnCxd9/Jd1uSdDIWPdNP8mXgcuC8JDPALcDlSbYxmKJ5Afg5gKo6lOReBhdo3wBurKo32+fcBDwArAP2VtWhFR+NJGlBo9y9c+0c5TsXaH8rcOsc9f3A/iX1TpK0ovxGriR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHFg39JHuTHE3y5FDt3CQHkjzb3te3epLcnmQ6ycEkFw/ts6u1fzbJrtUZjiRpIaOc6X8J2H5C7WbgwaraCjzY1gGuBra2127gDhj8kgBuAS4FLgFuOf6LQpI0PouGflV9Azh2QnkHcFdbvgv46FD97hp4GDgnyQbgKuBAVR2rqleAA7z1F4kkaZWd7Jz+e6rqCEB7P7/VNwIvDbWbabX56pKkMVrpC7mZo1YL1N/6AcnuJFNJpmZnZ1e0c5LUu5MN/ZfbtA3t/WirzwCbh9ptAg4vUH+LqtpTVZNVNTkxMXGS3ZMkzeVkQ38fcPwOnF3AfUP169pdPJcBr7bpnweAK5Osbxdwr2w1SdIYnbFYgyRfBi4Hzksyw+AunM8C9ya5AXgR+Hhrvh+4BpgGXgOuB6iqY0k+AzzW2n26qk68OCxJWmWLhn5VXTvPpivmaFvAjfN8zl5g75J6J0laUX4jV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOLPmVT0ulry81/OGf9hc/+5Jh7onHxTF+SOuKZvtSZ+c7u1QfP9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOuLdO5LeYvgOH+/ZP70s60w/yQtJvpPkiSRTrXZukgNJnm3v61s9SW5PMp3kYJKLV2IAkqTRrcT0zo9X1baqmmzrNwMPVtVW4MG2DnA1sLW9dgN3rMCxJUlLsBrTOzuAy9vyXcDXgV9u9burqoCHk5yTZENVHVmFPuhtzukD6dRY7pl+Af8jyeNJdrfae44HeXs/v9U3Ai8N7TvTaj8gye4kU0mmZmdnl9k9SdKw5Z7pf7CqDic5HziQ5M8WaJs5avWWQtUeYA/A5OTkW7ZLkk7ess70q+pwez8K/D5wCfBykg0A7f1oaz4DbB7afRNweDnHlyQtzUmHfpK/m+Tdx5eBK4EngX3ArtZsF3BfW94HXNfu4rkMeNX5fMFgfv/4S9LqWs70znuA309y/HN+u6r+e5LHgHuT3AC8CHy8td8PXANMA68B1y/j2JKkk3DSoV9VzwE/Nkf9r4Ar5qgXcOPJHk+StHw+hkGSOuJjGKTTnNdKNMzQl7Qgv0h3enF6R5I6YuhLUkcMfUnqiHP6eltx/lhaXZ7pS1JHDH1J6oihL0kdMfQlqSNeyNXY+M1Q6dQz9KXT0Gr9gvXuqrXP6R1J6oihL0kdcXpHb1tOJUgrzzN9SeqIoS9JHTH0JakjzulLp4lxfw/Cay5rk6GvVbVSQWTASCtj7NM7SbYneSbJdJKbx318SerZWM/0k6wDvgD8BDADPJZkX1U9Nc5+SKeLt8ujLfxLbO0Y9/TOJcB0VT0HkOQeYAdg6J9GVjuIeg+Yt0vQa20ad+hvBF4aWp8BLh1ukGQ3sLutfi/JM8s43nnAXy5j/7WoqzHnc32Nt3lbjzmfW5WPfVuPeTUs82f7H863Ydyhnzlq9QMrVXuAPStysGSqqiZX4rPWit7G3Nt4wTH3YrXGPO4LuTPA5qH1TcDhMfdBkro17tB/DNia5IIkZwE7gX1j7oMkdWus0ztV9UaSm4AHgHXA3qo6tIqHXJFpojWmtzH3Nl5wzL1YlTGnqhZvJUk6LfjsHUnqiKEvSR1Z86G/2GMdkpyd5Ctt+yNJtoy/lytrhDH/+yRPJTmY5MEk896zu1aM+viOJB9LUknW/O19o4w5yb9s/9aHkvz2uPu40kb42f4HSR5K8q32833NqejnSkmyN8nRJE/Osz1Jbm//PQ4muXjZB62qNfticDH4z4F/BJwFfBu48IQ2/wb4jba8E/jKqe73GMb848Dfacs/38OYW7t3A98AHgYmT3W/x/DvvBX4FrC+rZ9/qvs9hjHvAX6+LV8IvHCq+73MMf9z4GLgyXm2XwP8EYPvOF0GPLLcY671M/3vP9ahqv4PcPyxDsN2AHe15a8CVySZ60tia8WiY66qh6rqtbb6MIPvQ6xlo/w7A3wG+E/A346zc6tklDH/K+ALVfUKQFUdHXMfV9ooYy7g77XlH2aNf8+nqr4BHFugyQ7g7hp4GDgnyYblHHOth/5cj3XYOF+bqnoDeBX4kbH0bnWMMuZhNzA4U1jLFh1zkouAzVV1/zg7topG+Xd+L/DeJH+S5OEk28fWu9Uxypg/Bfx0khlgP/Bvx9O1U2ap/78vaq0/T3/RxzqM2GYtGXk8SX4amAT+xar2aPUtOOYk7wBuA352XB0ag1H+nc9gMMVzOYO/5v44yQeq6rur3LfVMsqYrwW+VFWfT/LPgN9sY/5/q9+9U2LF82utn+mP8liH77dJcgaDPwkX+nPq7W6kR1kk+TDwH4CPVNXrY+rballszO8GPgB8PckLDOY+963xi7mj/mzfV1X/t6qeB55h8EtgrRplzDcA9wJU1f8C3sngYWynqxV/dM1aD/1RHuuwD9jVlj8GfK3aFZI1atExt6mO/8og8Nf6PC8sMuaqerWqzquqLVW1hcF1jI9U1dSp6e6KGOVn+w8YXLQnyXkMpnueG2svV9YoY34RuAIgyfsYhP7sWHs5XvuA69pdPJcBr1bVkeV84Jqe3ql5HuuQ5NPAVFXtA+5k8CfgNIMz/J2nrsfLN+KY/zPwLuB32jXrF6vqI6es08s04phPKyOO+QHgyiRPAW8Cv1RVf3Xqer08I475E8AXk/w7BtMcP7uWT+KSfJnB9Nx57TrFLcCZAFX1GwyuW1wDTAOvAdcv+5hr+L+XJGmJ1vr0jiRpCQx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JH/D+IInAUFPH0ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "negatives = df_raw[df_raw.airline_sentiment == 'negative']\n",
    "print('min:', negatives.negativereason_confidence.min(), 'max:', negatives.negativereason_confidence.max())\n",
    "plt.hist(negatives.negativereason_confidence, bins=np.linspace(0,1,101));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_low_confidence:  1781 \n",
      "n_med_confidence:  3952 \n",
      "n_high_confidence: 3445\n"
     ]
    }
   ],
   "source": [
    "low_confidence  = negatives[(0.2 < negatives.negativereason_confidence) & (negatives.negativereason_confidence < 0.5)]\n",
    "med_confidence  = negatives[(0.5 < negatives.negativereason_confidence) & (negatives.negativereason_confidence < 0.8)]\n",
    "high_confidence = negatives[(0.8 < negatives.negativereason_confidence)]\n",
    "print('n_low_confidence: ', len(low_confidence), '\\nn_med_confidence: ', len(med_confidence),\n",
    "      '\\nn_high_confidence:', len(high_confidence))\n",
    "#for r in low_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].negativereason, r[1].text)\n",
    "#for r in med_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].negativereason, r[1].text)\n",
    "#for r in high_confidence.sample(n=20, random_state=42).iterrows(): print(r[1].negativereason, r[1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of a sample of 20 low confidence negative reasons, I think 2 are misclassified. I think that 3 of these don't express negative sentiment. \n",
    "\n",
    "Out of a sample of 20 medium confidence negative reasons, I think 2 tweets mention more than one issue and only one of the issues is listed as the negative reason, otherwise I agree with the classifications. \n",
    "\n",
    "I agree with all of a sample of 20 high confidence negative reasons.\n",
    "\n",
    "Now we can describe our negative reason classification task:\n",
    "- This is a **10- or 11-way classification task**, with 10 categories plus 1 N/A class if we choose to include it.\n",
    "- This is an unbalanced dataset, biased significantly towards the *N/A* and *Customer Service Issue* classes and against the *Damaged Luggage* and *longlines* classes.\n",
    "- Our naive baseline model for the 10-way task always predicts *Customer Service Issue* with an accuracy of 31.71%.\n",
    "- Our naive baseline model for the 11-way task always predicts *N/A* with an accuracy of 37.31%.\n",
    "\n",
    "Next, let's take a look at the tweet text and identify what kinds of preprocessing might be helpful.  <a name=\"text\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count              14640\n",
       "unique             14427\n",
       "top       @united thanks\n",
       "freq                   6\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw['text'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All rows have tweet data. 213 rows contain duplicate text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     airline_sentiment  airline_sentiment_confidence\n",
      "504           positive                        1.0000\n",
      "515           positive                        0.6634\n",
      "2285          positive                        0.6809\n",
      "2859          positive                        1.0000\n",
      "3565           neutral                        0.7168\n",
      "3604          positive                        1.0000\n"
     ]
    }
   ],
   "source": [
    "print(df_raw[df_raw['text'] == '@united thanks'][['airline_sentiment', 'airline_sentiment_confidence']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, the most frequent text '@united thanks' receives different labels. This certainly reduces our confidence in the accuracy of the labels. When we review how well our initial model performs, we should consider how robust our model is to noisy labels and consider whether it would be worth our time to remove or relabel some samples.\n",
    "\n",
    "Let's examine the length of the tweets. We would expect a cap at 160 characters due to Twitter's character limit in 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    14640.000000\n",
      "mean       103.836066\n",
      "std         36.281706\n",
      "min         12.000000\n",
      "25%         77.000000\n",
      "50%        114.000000\n",
      "75%        136.000000\n",
      "max        186.000000\n",
      "Name: text_len, dtype: float64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD7CAYAAACG50QgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAUJUlEQVR4nO3df6zd9X3f8ecrhqRdkxUoF+QaZyapu5VUqkF3gJS1ykIGhnQ12ZYINjVuhuR2AinRuqmmmUqaFIlsS6JGolREuDFVEmBNIqzUHfFosih/8MOm5odxKDfghht72I0JIWJjM3vvj/O52cHc3773XOLP8yEdne95fz/f8/18v+fc1/3ez/l+z01VIUnqw+tWugOSpNEx9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjJn6Cf5iSQPJHk4yb4kv9/qn03ydJK97bah1ZPk00kmkjyS5IKh59qc5Ml227x8myVJms4p82jzEvDOqvphklOBbyb5izbv31fVnx3X/nJgfbtdBNwCXJTkDOAGYBwoYE+SHVX13FJsiCRpbnOGfg2u3vphe3hqu812Rdcm4Pa23H1JTkuyGngHsKuqjgIk2QVsBL4w0xOdeeaZtW7dunlshiRpyp49e/62qsammzefI32SrAL2AD8H3FxV9yf5N8CNSX4PuBfYWlUvAWuAZ4YWn2y1meozWrduHbt3755PFyVJTZK/mWnevD7IraqXq2oDcA5wYZJfBK4H/gHwD4EzgN+ZWt90TzFL/fjObkmyO8nuI0eOzKd7kqR5WtDZO1X1feDrwMaqOlQDLwF/AlzYmk0Ca4cWOwc4OEv9+HXcWlXjVTU+NjbtXyeSpEWaz9k7Y0lOa9M/CbwL+FYbpydJgCuBx9oiO4D3t7N4Lgaer6pDwD3ApUlOT3I6cGmrSZJGZD5j+quB7W1c/3XAXVX1lSR/mWSMwbDNXuC3WvudwBXABPAi8AGAqjqa5GPAg63dR6c+1JUkjUZey1+tPD4+Xn6QK0kLk2RPVY1PN88rciWpI4a+JHXE0Jekjhj6ktSReV2RK+nksW7rny962QM3vXsJe6KV4JG+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTO0E/yE0keSPJwkn1Jfr/Vz01yf5Ink9yZ5PWt/ob2eKLNXzf0XNe3+hNJLluujZIkTW8+R/ovAe+sql8CNgAbk1wMfBz4VFWtB54DrmntrwGeq6qfAz7V2pHkPOAq4G3ARuCPkqxayo2RJM1uztCvgR+2h6e2WwHvBP6s1bcDV7bpTe0xbf4lSdLqd1TVS1X1NDABXLgkWyFJmpd5jeknWZVkL3AY2AV8G/h+VR1rTSaBNW16DfAMQJv/PPAzw/VplpEkjcC8Qr+qXq6qDcA5DI7Of2G6Zu0+M8ybqf4KSbYk2Z1k95EjR+bTPUnSPC3o7J2q+j7wdeBi4LQkp7RZ5wAH2/QksBagzf9p4OhwfZplhtdxa1WNV9X42NjYQronSZrDfM7eGUtyWpv+SeBdwH7ga8C/aM02A3e36R3tMW3+X1ZVtfpV7eyec4H1wANLtSGSpLmdMncTVgPb25k2rwPuqqqvJHkcuCPJHwB/BdzW2t8G/GmSCQZH+FcBVNW+JHcBjwPHgGur6uWl3RxJ0mzmDP2qegQ4f5r6U0xz9k1V/S/gvTM8143AjQvvpiRpKXhFriR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6sicoZ9kbZKvJdmfZF+SD7b6R5J8N8nedrtiaJnrk0wkeSLJZUP1ja02kWTr8mySJGkmp8yjzTHgt6vqoSRvAvYk2dXmfaqq/vNw4yTnAVcBbwN+FvhvSX6+zb4Z+CfAJPBgkh1V9fhSbIgkaW5zhn5VHQIOtekXkuwH1syyyCbgjqp6CXg6yQRwYZs3UVVPASS5o7U19CVpRBY0pp9kHXA+cH8rXZfkkSTbkpzeamuAZ4YWm2y1merHr2NLkt1Jdh85cmQh3ZMkzWHeoZ/kjcAXgQ9V1Q+AW4C3AhsY/CXwiamm0yxes9RfWai6tarGq2p8bGxsvt2TJM3DfMb0SXIqg8D/XFV9CaCqnh2a/xngK+3hJLB2aPFzgINteqa6JGkE5nP2ToDbgP1V9cmh+uqhZu8BHmvTO4CrkrwhybnAeuAB4EFgfZJzk7yewYe9O5ZmMyRJ8zGfI/23A78OPJpkb6v9LnB1kg0MhmgOAL8JUFX7ktzF4APaY8C1VfUyQJLrgHuAVcC2qtq3hNsiSZrDfM7e+SbTj8fvnGWZG4Ebp6nvnG05SdLy8opcSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkTlDP8naJF9Lsj/JviQfbPUzkuxK8mS7P73Vk+TTSSaSPJLkgqHn2tzaP5lk8/JtliRpOvM50j8G/HZV/QJwMXBtkvOArcC9VbUeuLc9BrgcWN9uW4BbYPBLArgBuAi4ELhh6heFJGk05gz9qjpUVQ+16ReA/cAaYBOwvTXbDlzZpjcBt9fAfcBpSVYDlwG7qupoVT0H7AI2LunWSJJmtaAx/STrgPOB+4Gzq+oQDH4xAGe1ZmuAZ4YWm2y1merHr2NLkt1Jdh85cmQh3ZMkzWHeoZ/kjcAXgQ9V1Q9mazpNrWapv7JQdWtVjVfV+NjY2Hy7J0mah3mFfpJTGQT+56rqS638bBu2od0fbvVJYO3Q4ucAB2epS5JGZD5n7wS4DdhfVZ8cmrUDmDoDZzNw91D9/e0snouB59vwzz3ApUlObx/gXtpqkqQROWUebd4O/DrwaJK9rfa7wE3AXUmuAb4DvLfN2wlcAUwALwIfAKiqo0k+BjzY2n20qo4uyVZIkuZlztCvqm8y/Xg8wCXTtC/g2hmeaxuwbSEdlCQtHa/IlaSOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjoyn3+MLkkArNv654te9sBN717CnmixPNKXpI4Y+pLUkTlDP8m2JIeTPDZU+0iS7ybZ225XDM27PslEkieSXDZU39hqE0m2Lv2mSJLmMp8j/c8CG6epf6qqNrTbToAk5wFXAW9ry/xRklVJVgE3A5cD5wFXt7aSpBGa84PcqvpGknXzfL5NwB1V9RLwdJIJ4MI2b6KqngJIckdr+/iCeyxJWrQTGdO/Lskjbfjn9FZbAzwz1Gay1Waqv0qSLUl2J9l95MiRE+ieJOl4iw39W4C3AhuAQ8AnWj3TtK1Z6q8uVt1aVeNVNT42NrbI7kmSprOo8/Sr6tmp6SSfAb7SHk4Ca4eangMcbNMz1SVJI7Ko0E+yuqoOtYfvAabO7NkBfD7JJ4GfBdYDDzA40l+f5Fzguww+7P2XJ9JxqVcncoGUNGfoJ/kC8A7gzCSTwA3AO5JsYDBEcwD4TYCq2pfkLgYf0B4Drq2ql9vzXAfcA6wCtlXVviXfGknSrOZz9s7V05Rvm6X9jcCN09R3AjsX1DtJ0pLyilxJ6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR1Z1D9Gl3Ri/OfmWike6UtSRwx9SeqIoS9JHZlzTD/JNuBXgcNV9YutdgZwJ7AOOAC8r6qeSxLgD4ErgBeB36iqh9oym4H/0J72D6pq+9JuijRajsvrx9F8jvQ/C2w8rrYVuLeq1gP3tscAlwPr220LcAv86JfEDcBFwIXADUlOP9HOS5IWZs7Qr6pvAEePK28Cpo7UtwNXDtVvr4H7gNOSrAYuA3ZV1dGqeg7Yxat/kUiSltlix/TPrqpDAO3+rFZfAzwz1G6y1Waqv0qSLUl2J9l95MiRRXZPkjSdpf4gN9PUapb6q4tVt1bVeFWNj42NLWnnJKl3i70469kkq6vqUBu+Odzqk8DaoXbnAAdb/R3H1b++yHVLS8YPY9WbxR7p7wA2t+nNwN1D9fdn4GLg+Tb8cw9waZLT2we4l7aaJGmE5nPK5hcYHKWfmWSSwVk4NwF3JbkG+A7w3tZ8J4PTNScYnLL5AYCqOprkY8CDrd1Hq+r4D4clSctsztCvqqtnmHXJNG0LuHaG59kGbFtQ7yRJS8ovXNOPNcfkpYXxaxgkqSOGviR1xOEdrTiHaKTR8Uhfkjpi6EtSRwx9SeqIY/paEo7LSz8ePNKXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0JakjnqevH/Fce+nk55G+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHTujsnSQHgBeAl4FjVTWe5AzgTmAdcAB4X1U9lyTAHwJXAC8Cv1FVD53I+vVqnoEjaTZLcaT/j6tqQ1WNt8dbgXuraj1wb3sMcDmwvt22ALcswbolSQuwHMM7m4DtbXo7cOVQ/fYauA84LcnqZVi/JGkGJxr6BXw1yZ4kW1rt7Ko6BNDuz2r1NcAzQ8tOtpokaURO9Irct1fVwSRnAbuSfGuWtpmmVq9qNPjlsQXgzW9+8wl2T5I07ISO9KvqYLs/DHwZuBB4dmrYpt0fbs0ngbVDi58DHJzmOW+tqvGqGh8bGzuR7kmSjrPoI/0kPwW8rqpeaNOXAh8FdgCbgZva/d1tkR3AdUnuAC4Cnp8aBtL/59k3kpbTiQzvnA18eXAmJqcAn6+q/5rkQeCuJNcA3wHe29rvZHC65gSDUzY/cALrliQtwqJDv6qeAn5pmvr3gEumqRdw7WLXJ0k6cV6RK0kdMfQlqSP+E5Vl4Iexkl6rPNKXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0JakjnqcvaSRO5PqVAze9ewl70jeP9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOuLZOzPwmzIlnYw80pekjhj6ktQRQ1+SOmLoS1JHDH1J6sjIQz/JxiRPJJlIsnXU65ekno009JOsAm4GLgfOA65Oct4o+yBJPRv1efoXAhNV9RRAkjuATcDjI+6HpB8jfkPn0hn18M4a4Jmhx5OtJkkagVEf6WeaWr2iQbIF2NIe/jDJEyewvjOBvz2B5ZeL/VoY+7Uw9mtIPj5nk5Nxf/29mWaMOvQngbVDj88BDg43qKpbgVuXYmVJdlfV+FI811KyXwtjvxbGfi1Mb/0a9fDOg8D6JOcmeT1wFbBjxH2QpG6N9Ei/qo4luQ64B1gFbKuqfaPsgyT1bOTfsllVO4GdI1rdkgwTLQP7tTD2a2Hs18J01a9U1dytJEknBb+GQZI6clKG/mvlqx6SrE3ytST7k+xL8sFW/0iS7ybZ225XrEDfDiR5tK1/d6udkWRXkifb/ekj7tPfH9one5P8IMmHVmp/JdmW5HCSx4Zq0+6jDHy6veceSXLBCPv0n5J8q633y0lOa/V1Sf7n0H774+Xo0xx9m/G1S3J9219PJLlsxP26c6hPB5LsbfWR7LNZsmH5319VdVLdGHxA/G3gLcDrgYeB81aoL6uBC9r0m4C/ZvD1Ex8B/t0K76cDwJnH1f4jsLVNbwU+vsKv4/9gcL7xiuwv4FeAC4DH5tpHwBXAXzC4FuVi4P4R9ulS4JQ2/fGhPq0bbrdC+2va1679HDwMvAE4t/3MrhpVv46b/wng90a5z2bJhmV/f52MR/o/+qqHqvrfwNRXPYxcVR2qqofa9AvAfl7bVyBvAra36e3AlSvYl0uAb1fV36xUB6rqG8DR48oz7aNNwO01cB9wWpLVo+hTVX21qo61h/cxuP5l5GbYXzPZBNxRVS9V1dPABIOf3ZH2K0mA9wFfWI51z9KnmbJh2d9fJ2Povya/6iHJOuB84P5Wuq79mbZt1MMoTQFfTbIng6ugAc6uqkMweFMCZ61Av6ZcxSt/EFd6f02ZaR+9Vt53/5rBEeGUc5P8VZL/nuSXV6A/MP1r91rZX78MPFtVTw7VRrrPjsuGZX9/nYyhP+dXPYxakjcCXwQ+VFU/AG4B3gpsAA4x+PNy1N5eVRcw+MbTa5P8ygr0YVoZXLj3a8B/aaXXwv6ay4q/75J8GDgGfK6VDgFvrqrzgX8LfD7J3x1ln5j5tVvx/dVczSsPLka6z6bJhhmbTlNb1P46GUN/zq96GKUkpzJ4UT9XVV8CqKpnq+rlqvq/wGdYpj9rZ1NVB9v9YeDLrQ/PTv3J2O4Pj7pfzeXAQ1X1bOvjiu+vITPtoxV93yXZDPwq8K+qDQK3oZPvtek9DMbNf35UfWrrnem1W/Gf0ySnAP8MuHOqNsp9Nl02MIL318kY+q+Zr3po44W3Afur6pND9eGxuPcAjx2/7DL366eSvGlqmsEHgY8x2E+bW7PNwN2j7NeQVxx9rfT+Os5M+2gH8P52lsXFwPNTf6YvtyQbgd8Bfq2qXhyqj2XwPyxI8hZgPfDUKPo01IeZXrsdwFVJ3pDk3Na3B0bZN+BdwLeqanKqMKp9NlM2MIr313J/Sr0SNwafdP81g9/SH17BfvwjBn+CPQLsbbcrgD8FHm31HcDqEffrLQzOnHgY2De1j4CfAe4Fnmz3Z6zAPvs7wPeAnx6qrcj+YvCL5xDwfxgcaV0z0z5i8Of3ze099ygwPsI+TTAY7516j/1xa/vP2+v7MPAQ8E9XYH/N+NoBH2776wng8lH2q9U/C/zWcW1Hss9myYZlf395Ra4kdeRkHN6RJM3A0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSP/Dx5WMEex8AiGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_raw['text_len'] = df_raw.text.apply(len)\n",
    "print(df_raw.text_len.describe())\n",
    "plt.hist(df_raw.text.apply(len), bins=np.linspace(0, 200, 21));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tweet length appears follow an interesting distribution - in particular, we can see a spike in texts around the 160 limit. This is an interesting feature that we should consider preserving - Twitter users who have more to say may, for instance, be feeling stronger sentiment compared to users who aren't interested in writing very much. Since our ULMFiT model is based on words, we'll be degrading this potential signal. Adding a field for original text length in characters would preserve this signal.\n",
    "\n",
    "The shortest tweet is a reasonable 12 characters, however, some tweets appear to be exceeding Twitter's character limit. We should investigate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8359, '@JetBlue Hey guys! Your Flight Booking Problems system ran my first and middle names together at time of Flight Booking Problems. Is that going to be a #TSAnightmare ?')\n",
      "(10405, \"@USAirways Eyyyy! Cancelled Flightlations, Flight Booking Problemss, reFlight Booking Problemss, but y'all got me on the same flight out tonight (not tomorrow) &amp; the FC upgrade. Thx!\")\n",
      "(11665, '@USAirways My Flight Booking Problems C68LD9 just times out when I select it under Manage My Flight Booking Problems for months now. I have emailed but no response. Help?')\n",
      "(11783, \"@USAirways Flight Booking Problems a flight using credit from a previously Cancelled Flightled flight. Can't get thru on the phone and can't book online. Help please.\")\n",
      "(11807, '@USAirways Travelling from pwm to atl on Sunday That flight got Cancelled Flightled and my new flight is Cancelled Flightled and got disconnected when reFlight Booking Problems')\n",
      "(12208, \"@AmericanAir When Flight Booking Problems an int'l flight online, do I have to provide a passport number when Flight Booking Problems or just when I arrive at airport?\")\n"
     ]
    }
   ],
   "source": [
    "# change 165 to 160 or lower for more examples\n",
    "for t in df_raw.text[df_raw.text_len > 165].iteritems(): print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notably, all of these include negative reasons have similar spelling errors such as 'Cancelled Flighted', 'Cancelled Flightled', 'reFlight Booking Problems', 'Flight Booking Problems', 'Late Flightr'.\n",
    "\n",
    "While I had no luck finding exact methods used to preprocess this dataset, one person in a [Kaggle discussion](https://www.kaggle.com/crowdflower/twitter-airline-sentiment/discussion/18283) hypothesized that a find/replace operation was applied to particular strings in the tweets (e.g. 'late' -> 'Late Flight' and 'cancel' -> Cancelled Flight).\n",
    "\n",
    "Let's look into this theory, by checking whether potentially replaced words exist in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# negative reasons: ['Bad Flight', \"Can't Tell\", 'Late Flight', 'Customer Service Issue', 'Flight Booking Problems',\n",
    "#                    'Lost Luggage', 'Flight Attendant Complaints', 'Cancelled Flight', 'Damaged Luggage', 'longlines']\n",
    "for i, text in df_raw.text.iteritems():\n",
    "    if 'late' in text: print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For brevity, I haven't shown the full investigation, however it seems clear that the following replacements were made:\n",
    "* 'cancel' -> 'Cancelled Flight'\n",
    "* 'late' -> 'Late Flight'\n",
    "* 'booking' -> 'Flight Booking Problems'\n",
    "\n",
    "We could try undo-ing the find/replace operation, but we risk removing occurrences where the text was actually written as such. Let's leave this for now as a possibility to explore in the future.\n",
    "\n",
    "During the course of the text investigation, there were a few more noteworthy observations:\n",
    "* There is some HTML (e.g. \\\\& amp;) to be cleaned, but fastai takes care of this by default.\n",
    "* Some tweets contain shortened URLs. Short of following the URLs and scraping them for additional data, they don't have any meaning beyond their existence in the tweet. By replacing the URL with a token 'xxurl', we could identify that the user went to the effort of adding a URL.\n",
    "\n",
    "For ULMFiT classification, we only need the columns we've reviewed so far: the language inputs 'text' and the labels 'airline_sentiment' and/or 'negativereason'. However, we could also consider adding other columns as inputs and there aren't too many to explore, so let's look at them now, starting with the location data. <a name=\"tweet_coord\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1019</td>\n",
       "      <td>9907</td>\n",
       "      <td>9820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>832</td>\n",
       "      <td>3081</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>[0.0, 0.0]</td>\n",
       "      <td>Boston, MA</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>164</td>\n",
       "      <td>157</td>\n",
       "      <td>3744</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tweet_coord tweet_location               user_timezone\n",
       "count         1019           9907                        9820\n",
       "unique         832           3081                          85\n",
       "top     [0.0, 0.0]     Boston, MA  Eastern Time (US & Canada)\n",
       "freq           164            157                        3744"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw[['tweet_coord', 'tweet_location', 'user_timezone']].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "location_counts = df_raw.tweet_location.value_counts()\n",
    "#location_counts[location_counts > 20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tweet coordinates are infrequently listed compared to tweet location and user timezone. Tweet location could reasonably be predictive on its own, as specific airports and general regions could both be reasonably predictive in both tasks (e.g maybe NYC has generally more delays and/or cancellations and/or grumpier travellers). However, it looks like this column contains bogus values as well as values with similar meanings but different spellings and would require some effort to clean.\n",
    "\n",
    "Next let's check out the date and time columns. <a name=\"tweet_created\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('min:',\n",
       " '2015-02-16 23:36:05 -0800',\n",
       " 'max:',\n",
       " '2015-02-24 11:53:37 -0800',\n",
       " count                         14640\n",
       " unique                        14247\n",
       " top       2015-02-24 09:54:34 -0800\n",
       " freq                              5\n",
       " Name: tweet_created, dtype: object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'min:', df_raw.tweet_created.min(), 'max:', df_raw.tweet_created.max(), df_raw.tweet_created.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see confirmation that the tweets were collected in a span of 1 week between Feb 16, 2015 and Feb 24, 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unknown                        4820\n",
       "Eastern Time (US & Canada)     3744\n",
       "Central Time (US & Canada)     1931\n",
       "Pacific Time (US & Canada)     1208\n",
       "Quito                           738\n",
       "Atlantic Time (Canada)          497\n",
       "Mountain Time (US & Canada)     369\n",
       "Arizona                         229\n",
       "London                          195\n",
       "Alaska                          108\n",
       "Sydney                          107\n",
       "Hawaii                          104\n",
       "Amsterdam                        74\n",
       "America/Chicago                  37\n",
       "America/New_York                 26\n",
       "Indiana (East)                   26\n",
       "Paris                            25\n",
       "Brasilia                         23\n",
       "Abu Dhabi                        23\n",
       "Name: user_timezone, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timezone_freq = df_raw.user_timezone.fillna('Unknown').value_counts()\n",
    "timezone_freq[timezone_freq > 20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With only 1 week of data, we can ignore the date portion of the timestamp, however the time of day is more likely to be relevant. In particular, the combination of 'user_timezone', 'tweet_location', and 'tweet_created' could be useful for the model in detecting both sentiment and negative reason (e.g. 2am tweets might tend to be more negative and caused by delays).\n",
    "\n",
    "Finally, let's take a look at retweet_count. <a name=\"retweet_count\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    14640.000000\n",
      "mean         0.082650\n",
      "std          0.745778\n",
      "min          0.000000\n",
      "25%          0.000000\n",
      "50%          0.000000\n",
      "75%          0.000000\n",
      "max         44.000000\n",
      "Name: retweet_count, dtype: float64\n",
      "\n",
      "Frequency of counts:\n",
      " 0     13873\n",
      "1       640\n",
      "2        66\n",
      "3        22\n",
      "4        17\n",
      "5         5\n",
      "7         3\n",
      "6         3\n",
      "22        2\n",
      "8         1\n",
      "32        1\n",
      "9         1\n",
      "31        1\n",
      "18        1\n",
      "15        1\n",
      "28        1\n",
      "44        1\n",
      "11        1\n",
      "Name: retweet_count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "retweet_count_freqs = df_raw.retweet_count.value_counts() \n",
    "print(df_raw.retweet_count.describe())\n",
    "print('\\nFrequency of counts:\\n', retweet_count_freqs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There may be value in retweet counts, but could also be easy for the model to quickly memorize the training data for infrequently encountered tweet counts. A simple way to address this is to bin the counts (e.g. bins = 0, 1, 2, 3+)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare Training, Validation, and Test Sets <a name=\"prepare-datasets\"></a>\n",
    "\n",
    "First, we'll preprocess the data according to our earlier analysis, starting with dropping unused columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['airline_sentiment', 'airline_sentiment_confidence', 'negativereason',\n",
       "       'negativereason_confidence', 'retweet_count', 'text', 'tweet_created',\n",
       "       'user_timezone', 'text_len'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_raw.copy()\n",
    "df.drop(columns=['tweet_id', 'airline', 'airline_sentiment_gold', 'name', \n",
    "                 'negativereason_gold', 'tweet_coord', 'tweet_location'], \n",
    "        inplace=True)\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting the 'tweet_created' to a datetime object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datetime64[ns, pytz.FixedOffset(-480)]\n"
     ]
    }
   ],
   "source": [
    "df.tweet_created = pd.to_datetime(df.tweet_created, infer_datetime_format=True)\n",
    "print(df.tweet_created.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mapping non-standard timezones to standard timezones for all values of 'user_timezone' with 20 or more occurrences; then finding the local date and time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "tz_conversions = {'Eastern Time (US & Canada)': 'US/Eastern', 'Central Time (US & Canada)': 'US/Central',\n",
    "    'Pacific Time (US & Canada)': 'US/Pacific', 'Quito': 'Etc/GMT-5', 'Atlantic Time (Canada)': 'Canada/Atlantic',\n",
    "    'Mountain Time (US & Canada)': 'US/Mountain', 'Arizona': 'US/Arizona', 'London': 'Europe/London',\n",
    "    'Alaska': 'US/Alaska', 'Sydney': 'Australia/Sydney', 'Hawaii': 'US/Hawaii', 'Amsterdam': 'Europe/Amsterdam',\n",
    "    'Indiana (East)': 'US/East-Indiana', 'Paris': 'Europe/Paris', 'Abu Dhabi': 'Etc/GMT+4', 'Brasilia': 'Etc/GMT-3'}\n",
    "\n",
    "def toLocalTime(dt, tz, nonstd_tz_map=None):\n",
    "    \"\"\"Convert a date, `dt`, to the local time at timezone, `tz`, optionally correcting .\"\"\"\n",
    "    if tz in pytz.all_timezones: std_tz = tz\n",
    "    elif nonstd_tz_map and tz in nonstd_tz_map: std_tz = nonstd_tz_map[tz]\n",
    "    else: return None\n",
    "    return dt.astimezone(std_tz)\n",
    "\n",
    "local_dt = df.apply(lambda row: toLocalTime(row.tweet_created, row.user_timezone, tz_conversions), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating new hour column from the local datetime and dropping the remaining datetime columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>user_timezone</th>\n",
       "      <th>text_len</th>\n",
       "      <th>tweet_createdLocalHour</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "      <td>2015-02-24 11:35:52-08:00</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "      <td>35</td>\n",
       "      <td>xxtweet_createdLocalHour_14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
       "      <td>2015-02-24 11:15:59-08:00</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "      <td>72</td>\n",
       "      <td>xxtweet_createdLocalHour_11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  airline_sentiment  airline_sentiment_confidence negativereason  \\\n",
       "0           neutral                        1.0000            NaN   \n",
       "1          positive                        0.3486            NaN   \n",
       "\n",
       "   negativereason_confidence  retweet_count  \\\n",
       "0                        NaN              0   \n",
       "1                        0.0              0   \n",
       "\n",
       "                                                text  \\\n",
       "0                @VirginAmerica What @dhepburn said.   \n",
       "1  @VirginAmerica plus you've added commercials t...   \n",
       "\n",
       "              tweet_created               user_timezone  text_len  \\\n",
       "0 2015-02-24 11:35:52-08:00  Eastern Time (US & Canada)        35   \n",
       "1 2015-02-24 11:15:59-08:00  Pacific Time (US & Canada)        72   \n",
       "\n",
       "          tweet_createdLocalHour  \n",
       "0  xxtweet_createdLocalHour_14.0  \n",
       "1  xxtweet_createdLocalHour_11.0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def extractHour(dt): \n",
    "    if hasattr(dt, 'hour'): return dt.hour\n",
    "\n",
    "df['tweet_createdLocalHour'] = 'xxtweet_createdLocalHour_' + (local_dt.apply(extractHour)).astype(str)\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binning 'retweet_count' into buckets: 0, 1, 2, and 3+, and 'text_len' into buckets: <75, 75to130, >130."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xxtextlencount_75_130    5859\n",
       "xxtextlencount_>130      5244\n",
       "xxtextlencount_<=75      3537\n",
       "Name: text_len, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl_int = df.text_len.copy()\n",
    "df.loc[:, 'text_len'] = df.text_len.astype(str)\n",
    "df.loc[tl_int <= 75, 'text_len'] = '<=75'\n",
    "df.loc[(tl_int > 75) & (tl_int < 130), 'text_len'] = '75_130'\n",
    "df.loc[tl_int >= 130, 'text_len'] = '>130'\n",
    "df.loc[:, 'text_len'] = 'xxtextlencount_' + df.text_len\n",
    "df.text_len.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xxretweetcount_0     13873\n",
       "xxretweetcount_1       640\n",
       "xxretweetcount_2        66\n",
       "xxretweetcount_3+       61\n",
       "Name: retweet_count, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:, 'retweet_count'] = df.retweet_count.astype(str)\n",
    "df.loc[df.retweet_count.astype(np.int64) >= 3, 'retweet_count'] = '3+'\n",
    "df.loc[:, 'retweet_count'] = 'xxretweetcount_' + df.retweet_count\n",
    "df.retweet_count.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create test set\n",
    "\n",
    "To create the test set, we pull out 7% of the original data, randomly sampled, to create a test dataset of 1024 samples. This leaves 13616 tweets for training and validation. 7% split was chosen so that our test accuracy will have ~3 digits of precision (`3.01 = log10(1024)`, note: this is a very rough approximation). We'll create a similarly-sized validation set. This sample size would not be a good choice if we cared about small changes in prediction accuracy. However, it is sufficient for broadly determining the ability of a ULMFiT model to fit the data and to compare the model with others.\n",
    "\n",
    "It is important to mention that the test set is created from a sample of the full dataset, before removing any rows, regardless of labelling accuracy. This ensures that our test set best represents the real data distribution and does not overestimate our model's performance. Given a more specific goal, we should revisit this decision to ensure that the test set best represents the data distribution of the goal task.\n",
    "\n",
    "Thus far, we have considered modeling both sentiment and negative reason classification. From here on we'll focus on modeling sentiment only. The process for modelling negative reason would be similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_train: 13616\n",
      "n_test: 1024\n"
     ]
    }
   ],
   "source": [
    "test_pct = 0.07\n",
    "\n",
    "# get random indices for our train and test sets\n",
    "np.random.seed(9741) # set seed to ensure we always get the same test set\n",
    "random_idx = np.random.permutation(len(df))\n",
    "n_test = int(len(df) * test_pct)\n",
    "train_idx = random_idx[:-n_test]\n",
    "test_idx = random_idx[-n_test:]\n",
    "\n",
    "# split train and test sets\n",
    "df_train = df.iloc[train_idx].copy()\n",
    "df_test = df.iloc[test_idx].copy()\n",
    "\n",
    "print('n_train:', len(df_train))\n",
    "print('n_test:', len(df_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mark validation set\n",
    "\n",
    "To create the validation set, we'll add a new column to the training dataframe for identifying these rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_pct = 0.2\n",
    "\n",
    "n_valid = int(valid_pct * len(df_train))\n",
    "\n",
    "# get random indices for our validation set\n",
    "np.random.seed(8874) # make sure we always get the same random values\n",
    "valid_idx = np.array([0] * (len(df_train) - n_valid) + [1] * n_valid)\n",
    "valid_idx = np.random.permutation(valid_idx)\n",
    "df_train['is_valid'] = valid_idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finalize training set\n",
    "\n",
    "We'll remove rows which have low confidence labels from our training set and drop columns not used in sentiment classification. Note that we don't remove these rows from our test set as this would not reflect the real data and might lead to unrealistic expectations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train[df_train.airline_sentiment_confidence > 0.5]\n",
    "df_train = df_train[cols + ['is_valid']]\n",
    "\n",
    "df_test = df_test[cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One final look at the data to make sure we didn't make any obvious mistakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment classification - training set: 13392\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text_len</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>tweet_createdLocalHour</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5101</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_nan</td>\n",
       "      <td>@SouthwestAir How to find out if my parents lu...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12352</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_7.0</td>\n",
       "      <td>@AmericanAir don't merge with an airline that ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment             text_len     retweet_count  \\\n",
       "5101           negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "12352          negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "\n",
       "             tweet_createdLocalHour  \\\n",
       "5101   xxtweet_createdLocalHour_nan   \n",
       "12352  xxtweet_createdLocalHour_7.0   \n",
       "\n",
       "                                                    text  is_valid  \n",
       "5101   @SouthwestAir How to find out if my parents lu...         0  \n",
       "12352  @AmericanAir don't merge with an airline that ...         0  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Sentiment classification - training set:', len(df_train))\n",
    "df_train.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment classification - test set: 1024\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text_len</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>tweet_createdLocalHour</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5101</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_nan</td>\n",
       "      <td>@SouthwestAir How to find out if my parents lu...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12352</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_7.0</td>\n",
       "      <td>@AmericanAir don't merge with an airline that ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment             text_len     retweet_count  \\\n",
       "5101           negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "12352          negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "\n",
       "             tweet_createdLocalHour  \\\n",
       "5101   xxtweet_createdLocalHour_nan   \n",
       "12352  xxtweet_createdLocalHour_7.0   \n",
       "\n",
       "                                                    text  is_valid  \n",
       "5101   @SouthwestAir How to find out if my parents lu...         0  \n",
       "12352  @AmericanAir don't merge with an airline that ...         0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Sentiment classification - test set:', len(df_test))\n",
    "df_train.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save prepared datasets\n",
    "\n",
    "Save our work so that we can skip this preprocessing step when we return to this notebook later.\n",
    "\n",
    "As a convenience, we'll also prepare a data set which combines the training and test sets for training the language model in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "both: 14416 train: 13392 test: 1024\n"
     ]
    }
   ],
   "source": [
    "def save_df(df, filepath):\n",
    "    filepath.parent.mkdir(exist_ok=True)\n",
    "    df.to_csv(filepath, header=True, index=False)\n",
    "\n",
    "df_train_and_test = pd.concat([df_train.loc[:, cols], df_test])\n",
    "print('both:', len(df_train_and_test), 'train:', len(df_train), 'test:', len(df_test))\n",
    "\n",
    "save_df(df_train_and_test, path / 'train_and_test.csv')\n",
    "save_df(df_train, path / 'train.csv')\n",
    "save_df(df_test, path / 'test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Language Model Fine-Tuning <a name=\"2\"></a>\n",
    "<a name=\"0\">[(... back to the top)](#top)</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare language model data\n",
    "Since we aren't using our classification labels, we can use all of our data in this step, not just the training data. However, we do still need a validation set so we can tune our model. We create this by randomly sampling 10% of the original data (~1500 rows).\n",
    "\n",
    "This text processing step can take some time, so we save the language model data to disk for easy retrieval in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_csv(path, 'train_and_test.csv', cols=model_input_cols) # load data\n",
    "                   .split_by_rand_pct(0.1)    # create validation set from 10% of rows\n",
    "                   .label_for_lm()            # add dummy labels (labels are ignored by the language model)\n",
    "                   .databunch())              # prepare the data for use in fastai models\n",
    "data_lm.save('data_lm.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For quickly reloading the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load preprocessed language model data:\n",
    "Let's take a look at the first few rows of a batch of data to verify the data has loaded as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxretweetcount_0 xxtweet_createdlocalhour_23.0 @usairways i have a story for you . xxmaj it starts with a two hour delayed flight that makes me miss my connection in xxmaj miami . xxbos xxtextlencount_&lt;=75 xxretweetcount_0 xxtweet_createdlocalhour_nan @americanair 11 out of 11 delayed flights , you suck and getting worse xxbos xxtextlencount_75_130 xxretweetcount_0 xxtweet_createdlocalhour_nan @united great job losing one of my six bags on a direct flight . xxmaj do n't know how you</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>is charging me for the xxunk that was in the lost suitcase # help xxbos xxtextlencount_&lt;=75 xxretweetcount_0 xxtweet_createdlocalhour_9.0 @jetblue to allow payment by # xxmaj apple xxmaj pay - xxmaj xxunk http : / / t.co / xxunk xxbos xxtextlencount_75_130 xxretweetcount_0 xxtweet_createdlocalhour_nan @americanair i look forward to hearing back and hope that my next trip will not be like this xxbos xxtextlencount_75_130 xxretweetcount_0 xxtweet_createdlocalhour_nan @jetblue how much longer to xxup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_18.0 @jetblue if u want to be helpful save me the trip from xxmaj westchester xxmaj xxunk to xxup jfk , plus parking - my info is in the photo http : / / t.co / xxunk xxbos xxtextlencount_&lt;=75 xxretweetcount_0 xxtweet_createdlocalhour_19.0 @jetblue sounds great ! xxmaj thank you ! ! :) xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_nan @united ruined my day &amp; &amp; start trip xxup sfo to xxup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>paris yesterday and ca n’t get in touch with anyone xxunk . xxmaj what to do ? xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_nan @usairways @americanair on hold for 8 hours yesterday w / o speaking to an agent on hold again w / 2 phones xxunk mins let the xxunk begin ... xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_0.0 “ @southwestair : xxunk i 'm so sorry for your frustration . xxmaj did you get</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_13.0 xxunk @usairways we hate delays ! ! have you tried any of these ? ! http : / / t.co / xxunk ( although we 're not sure what a xxunk is ... ) xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_21.0 @united it was eventually explained that weather conditions too extreme to get luggage off planes . xxmaj will be sent on . xxmaj but i wo n't be</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here it is worth noting that the `fastai` library does some text preprocessing automatically for us. There are 4 text preprocessing steps:\n",
    "1. Preprocessing Transforms - list of functions that modify sentences. See next cell.\n",
    "2. Tokenization - [Spacy](https://spacy.io/) is the default tokenizer, and does some intelligent word processing (e.g. replacing \"haven't\" with \"have n't\" and \"it's\" with \"it 's\").\n",
    "3. Postprocessing Transforms - list of functions that modify tokens (i.e. words). See next cell.\n",
    "4. Numericalization - Tokens are replaced by numbers for indexing into the embedding layer. Words with < N occurrences are replaced by the token *xxunk*, where N defaults to 2.\n",
    "\n",
    "Here are the fastai methods being run for data processing and the list special tokens used. See the method source (Shift+Tab 3 times in Jupyter notebook) or the [fastai docs](https://docs.fast.ai/text.transform.html#text.transform) for more info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessing transforms : ['fix_html', 'replace_rep', 'replace_wrep', 'spec_add_spaces', 'rm_useless_spaces']\n",
      "Postprocessing transforms: ['replace_all_caps', 'deal_caps']\n",
      "Special Tokens           : ['xxunk', 'xxpad', 'xxbos', 'xxeos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep']\n"
     ]
    }
   ],
   "source": [
    "print('Preprocessing transforms :', [f.__name__ for f in defaults.text_pre_rules])\n",
    "print('Postprocessing transforms:', [f.__name__ for f in defaults.text_post_rules])\n",
    "print('Special Tokens           :', defaults.text_spec_tok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare language model learner\n",
    "Our first goal is to generate fine-tuned word embeddings for the vocabulary in our Tweets data from embeddings trained on a large, general English language dataset.\n",
    "\n",
    "Since a language model predicts the next token in a sentence, it captures useful information about how words are used and have been shown to be useful in pretraining a model for common NLP tasks. This model is uses the [AWD LSTM](https://github.com/salesforce/awd-lstm-lm) architecture pretrained on WikiText-103.\n",
    "\n",
    "In this case, tokens are all words in our vocabulary as well as any special tokens. For example, 'xxbos' is added to the beginning of every sample and 'xxunk' replaces all words that aren't in our vocabulary. The full list of special tokens is in the cell output above.\n",
    "\n",
    "The fastai library merges our vocabulary with the vocab used in used in Wikitext-103 so that the pretrained embeddings are preserved and our new words initialized to the mean of all pretrained embeddings.\n",
    "\n",
    "Here we set the remaining hyperparameters of our language model.\n",
    "* arch = AWD_LSTM : AWD_LSTM and Transformer are the two options for which the fastai library includes pretrained weights. In trials not shown, the Transformer architecture takes quite a lot more memory and computational power to train, so we'll stick with AWD_LSTM.\n",
    "* bs = 400        : Larger batch sizes speed up learning, so it is set to approximately the largest batch size that will fit on the GPU (I set this by trial and error).\n",
    "* drop_mult = 1.0 : Each architecture has its own default values for dropout. `drop_mult` allows us to scale all of the individual dropouts by a constant factor. This is the first hyperparameter for directly controlling the balance between overfitting and underfitting within a particular architecture.\n",
    "\n",
    "Additionally, there are several default hyperparameters. Except for `bptt`, we must keep these unchanged to easily take advantage of the pretrained weights. Increasing `bptt` increases the ability for the model to learn long term dependencies - in this case, the default value `bptt=70` is fine for now.\n",
    "* bptt = 70    : number of iterations to calculate back-propagation through time\n",
    "* emb_sz = 400 : size of the embedding weights\n",
    "* nh = 1150    : number of hidden units\n",
    "* nl = 3       : number of hidden layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded language model using data: data_lm.pkl \n",
      "Trial prefix: lm_AWD_LSTM_drop_mult_1.0_\n"
     ]
    }
   ],
   "source": [
    "arch = AWD_LSTM # AWD_LSTM, Transformer, or TransformerXL (no pretrained weights)\n",
    "bs = {'AWD_LSTM': 400, 'Transformer': 40}[arch.__name__]\n",
    "drop_mult = 1.0 # default is 1.0\n",
    "lm_data_name = 'data_lm.pkl'\n",
    "lm_trial_prefix = 'lm_' + arch.__name__ + '_drop_mult_' + str(drop_mult) + '_'\n",
    "\n",
    "data_lm = load_data(path, lm_data_name, bs=bs)\n",
    "lm_learn = language_model_learner(data_lm, arch=arch, drop_mult=drop_mult)\n",
    "print('Loaded language model using data:', lm_data_name, '\\nTrial prefix:', lm_trial_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "lm_learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXyV5Z3//9cn+0JWSCDsBBBERlYX3JW6TTta2/qtdrN1oXSx7cy32zz6/Xam7W/sdGynteO3MojaZWztuLS1ndatrSuCIiIqm2xCgJCQkITs2+f3xznRGAMEyH3us7yfj8d5eM59X+fc78SQT677vq/rMndHRERSV1rYAUREJFwqBCIiKU6FQEQkxakQiIikOBUCEZEUlxF2gGM1atQonzx5ctgxREQSyksvvXTA3csG25dwhWDy5MmsWbMm7BgiIgnFzN483D6dGhIRSXEqBCIiKU6FQEQkxakQiIikOBUCEZEUp0IgIpLiVAhERFJcwo0jOF77GttY+2YDLR3dNHd009rZTWZ6GhNL85gQfRTlZoYdU0Qk5lKmEKx9s4HP/XLtEdtUluVzZuVIFlWOZM74YnKz0snKSCMrPY2czDTMLEZpRURixxJtYZqFCxf68YwsbmzrorqxnfzsdEZkZ5CXlUFbVw+761upOtjK9gMtrNl5kBd21NPc0f2u92elpzFqRBZlBdmUF+awYFIJZ08dxayxhaSnDa1AdPf0srehnQ37Gtmwt4mttc2MGpHN9NEFnFQ+gill+ZTmZZGRrjN2IjK8zOwld1842L6U6REU5Wa+69RPVkYaReOKmD2u6K1t3T29vLa3ic3VTXR299LZ43R299LY1kXtoQ5qmzvYVtvM4xv2A1Ccl8ncCcVUFOUypjCH8sJsDrZ2sr22hR0HWqg62EprZw/tXT109bxddNMMJpbmcaC5812FpyAng5K8SNGpKMqJPnI5aXQBJ1cUMHJENgCNrV28treR1/c20t3rjMjOID8rg4KcDEYXRt43ckQ2ve7sbWhjV30rew620euQkW5kphtFuZksmFhKUV64p8UOtnSy/UAzNU0dzJlQzNji3FDziKSSlCkEQ5WRnsbcCcXMnVB8xHY1Te2s3FbHs1sPsGFvE69WNVLX0vnW/rKCbCpH5XP+SWXkZ2eQk5lObmY6ZQXZzKooZMaYAnIy03F39jW2s3n/IXbVtXKwtZOG1i4OtnZS09TB63ubeGLjftq7et/x2XlZ6bxZ13rUr6evt9LTe/ienxnMqijkzMqRTBmVT352OvlZGYzIyWBccS5ji3PJjPZS3J2m9m72NbbR1Ba51tLW2UNHdy/paZHikpGWRnq6kW5GepphRHpkdS2d1EcfDa2d1Ld2Ud/SQdXBNhpau96RacboAi6YWcbpk0sZW5xLRVEORbmZOj0nEoCUOTUUCx3dPdQe6qAoN5OCnOH7C9vdqWvpZHP1ITbua2LjvkO0dXVzytgiTh1fxOyxReRkptPS2U1LRzeNbV3sb+qguqmd6sY2DGPiyDwmluYxviSXjLQ0unp66e519je188KOelZtr+OlNw/S0d37ruOnGYwtziU3M519je2Dnjo7FgXZGZTkZ0UeeZmMLc6lclQ+lWX5lOZn88KOOv66qZYXd9bT3a+A5WSmMbYol3EluYwvyWNkfhb1rZ0ciPbU2rt6MSAtDdLNKCvIYVxxDuNKcqPFJJexxTmUjcjW6TdJOUc6NaRCIG/p7O6loa2Tlo4eWjq6aWrroqqhjd31reyqb6W9q4eKolzGFedSUZxDcW4WuVlp5GZmkJ2ZRk+vRwpMj9PjTm+v09PrOFCYk8nIEVmU5GWRlTG0X8KH2rvYsr+Z/U3t7GtsZ19DG3sb26g62Maeg20cbO2kJC+LUSOyGVWQRW5mBuC4Q1evU9PUzp6DbRwaULjS04wxhTmMK86NFokcSvKyKMzJpDA3clpt5phCcrPSh/+bLBKSUK4RmNkM4Nf9NlUC33T3Hw3S9jRgFfBhd38gqExyZFkZaZQX5EBB2EkiCnIyWTCp5LD73X1Ip4oa27rY29BGdWM7exvb2NfQzt6GNqoa2nhhRz3VTe3vOnWWZjCtfASnjC1i5pgCThpTwIzRBVQU5ej0lCSdwAqBu28G5gKYWTqwB/jNwHbRfd8DHg0qiySnof5C7rtR4OSKwkH39/Y6LZ3dNLVHekG76lt5fW8TG/Y28vy2On7z8p632hbmZDBnQjHzJpYwb2Ixp44reuvivUiiitXF4sXANncfbGGEm4EHgdNilEXkHdLSjIKcyHWdccW5nFxRyKWnjHlrf2NrF1tqDrG5+hCv721i3e4Gbv/LG/R1IsoKspk5poBZFYWcM30UZ0wZOeTTXyLxIFaF4BrgVwM3mtk44CrgIo5QCMxsCbAEYOLEiQFFFBlcUV4mp00u5bTJpW9ta+7oZv3uBjZEL95vqm7inud28p9Pb6cgJ4OLZpZz+ewK3nNyuS5MS9wL/GKxmWUBe4FT3H3/gH33Az9w91Vm9lPgD0e7RqCLxRKv2jp7eHbrAR57vZo/b6qhvqWTCaW53HhOJVcvHE9elu7WlvCEeteQmV0JfM7dLxlk3w6g70TvKKAVWOLuvz3c56kQSCLo6XUe37Cf5U9vY+2uBorzMvn4mZP4+KJJkQvyIjEWdiG4D3jU3e85Srufoh6BJKE1O+v5z6e388TG/WSkGVfMGccN50xh1tjBL16LBCG0KSbMLA+4GPh0v21LAdx9WZDHFokXCyeXsnByKTsPtHDPczv47zVVPLi2iotmlvOFxdOPOopdJGgaUCYSY42tXfxi1U5WPLuDhtYuzj+pjC++ZzrzJx5+zITIidLIYpE41NzRzc+f38mdT2/nYGsXl8wazVcvm8G08jgZ0SdJRYVAJI61dHRz17M7WP70dlo7u/nQgvF85dKZlBVooJoMnyMVAt3gLBKy/OwMvrB4Ok995QI+edYUfvvyXt7742d4YUd92NEkRagQiMSJkSOy+ebfzeJ3nz+bvKx0rr1zFcuf3kai9dol8agQiMSZkysKefjmc7hk1mhu+eMmlv7XS7Sc4NTfIkeiQiAShwpzMvnJR+fzf957Mk9srOHqZc9T3dgedixJUioEInHKzLjx3EpWXLeQN+tauOonz7Fhb1PYsSQJqRCIxLkLZ5Rz/9KzcIerl63kyc01YUeSJKNCIJIAZo0t5LefO5tJI/O54Wdr+O81u8OOJElEhUAkQYwpyuHXnz6Ts6aO5KsPrOe2J97QHUUyLFQIRBJIQU4md3/yND4wfxw/fGIL//jQq3T39IYdSxKcJkgXSTCZ6Wn84Oo5jC3K5fa/bqXmUAe3f2Se1juQ46YegUgCMjO+fOkM/r/3z+bJzTV85M7V1Ld0hh1LEpQKgUgC+9iZk7jjYwvYuK+JD92xkt31rWFHkgSkQiCS4C49ZQz33ngGdS2dfOCOlWytaQ47kiQYFQKRJLBwcin3L12EO1x75yoVAzkmKgQiSeKk0QX86qYzcIdrlq9ia82hsCNJglAhEEki00cXcN+SMwC4ZvlqFQMZEhUCkSQzrfztYnDd3S9Se6gj5EQS7wIrBGY2w8zW9Xs0mdmXBrT5qJmtjz5WmtmcoPKIpJJp5QXc/cmF1LV0cNPP19De1RN2JIljgRUCd9/s7nPdfS6wAGgFfjOg2Q7gfHc/FfgOsDyoPCKp5tTxxfzow/N4paqB/33/K/T2ajoKGVysTg0tBra5+5v9N7r7Snc/GH25ChgfozwiKeGy2WP4+mUz+Z/1+/jhE1vCjiNxKlZj0q8BfnWUNjcAfxpsh5ktAZYATJw4cXiTiSS5JedVsuNAC//xl62cNLqAv5szNuxIEmcC7xGYWRZwBXD/EdpcSKQQfG2w/e6+3N0XuvvCsrKyYIKKJCkz4zvvn82CSSV87cH1upNI3iUWp4YuB9a6+/7BdprZqcAK4Ep3r4tBHpGUk5mexv/7yHzystL59C9eollrIEs/sSgE13KY00JmNhF4CPi4u+sEpkiAxhTl8ONr57HjQAtff3C91jKQtwRaCMwsD7iYyC/7vm1LzWxp9OU3gZHAT6K3mK4JMo9Iqjtr6ii+culM/rB+H/c8tzPsOBInAr1Y7O6tRH7R99+2rN/zG4Ebg8wgIu+09PxK1u46yHf/tJH5k0qYO6E47EgSMo0sFkkxZsb3PzSH8oIcPv/LtTS2doUdSUKmQiCSgoryMrn9I/OobmznKw+8ousFKU6FQCRFzZtYwtcum8ljG/bz05U7w44jIVIhEElhN547hcUzy7nljxt5taox7DgSEhUCkRRmZnz/6jmU5mfx1QfX09XTG3YkCYEKgUiKK8nP4ltXzGbjvibuenZH2HEkBCoEIsJls8dw8azR/OiJLeyqaw07jsSYCoGIAPDtK08h3Yxv/PZV3UWUYlQIRASAiqJcvnrZTJ554wC/W7c37DgSQyoEIvKWj505ibkTivnOHzZwsKUz7DgSIyoEIvKW9DTjux/4GxrauvjeI5vCjiMxokIgIu9wckUhN5wzhfte3M2LO+vDjiMxoEIgIu/yxcXTGVuUwzd+8yqd3RpbkOxUCETkXfKzM/jWlbPZsr+ZFc9uDzuOBEyFQEQGdfGs0VwyazQ//vMb7K7X2IJkpkIgIof1z1ecQpoZ3/r9hrCjSIBUCETksMYW5/K5C6fxxMb9unCcxFQIROSIPnX2ZMoLsvnuHzdqxHGSUiEQkSPKy8rg7y8+ibW7Gnhsw/6w40gAAisEZjYjuiB936PJzL40oI2Z2Y/NbKuZrTez+UHlEZHjd/WC8Uwty+ffHtlEt6aqTjqBFQJ33+zuc919LrAAaAV+M6DZ5cD06GMJcEdQeUTk+GWkp/HVy2ayrbaF+1+qCjuODLNYnRpaDGxz9zcHbL8S+LlHrAKKzawiRplE5BhcMms08ycW88PHt9DW2RN2HBlGsSoE1wC/GmT7OGB3v9dV0W3vYGZLzGyNma2pra0NKKKIHImZ8Y9/ezI1hzr4+fM7w44jwyjwQmBmWcAVwP2D7R5k27tuS3D35e6+0N0XlpWVDXdEERmi0yaXcu70Udz5zHb1CpJILHoElwNr3X2w2w2qgAn9Xo8HNBG6SBz7wuLpHGju5Jcv7Ao7Skr58v2v8PArwfx6jEUhuJbBTwsBPAx8Inr30JlAo7vvi0EmETlOp00uZVHlSJY9tY32LvUKYqGn13lwbRXbapoD+fxAC4GZ5QEXAw/127bUzJZGX/4R2A5sBe4EPhtkHhEZHl9YPJ3aQx38+sXdR28sJ+xgayfuMHJEViCfnxHIp0a5eyswcsC2Zf2eO/C5IDOIyPA7s7KU0yeXcseT27jm9AlkZ6SHHSmp1UdXiyvND6YQaGSxiBwzM+MLi6dT3dTO/Ws0riBoB5o7ABUCEYkzZ08byfyJxdzx5Da6NNo4UH09gpH52YF8vgqBiBwXM+OzF0xjT0Mbf3qtOuw4SU2nhkQkbl00s5zKUfmseGa7ZiYNUF1zpBCU5GUG8vkqBCJy3NLSjOvPmcL6qkZe3Hkw7DhJq76lk+K8TDLSg/mVrUIgIifkg/PHU5KXyZ3PaG3joNS3dAZ2WghUCETkBOVmpfOxMyfxxMb97DjQEnacpFTX0sGogC4UgwqBiAyDjy+aRGZaGnc/uyPsKElJPQIRiXvlBTlcOXcs97+0m4PRO1xk+NQ1d1Ia0KhiUCEQkWFy47mVtHf1cu/qgcuOyIno7XUOtnYyUj0CEYl3M8YUcN5JZfx05Zt0dGsyuuHS0NZFrwc3hgBUCERkGC05t5IDzR387mXNJj9c6luCnV4CVAhEZBidPW0kJ1cUsvyZ7fT2aoDZcOgbTBbU9BKgQiAiw8jMWHLeFLbWNPPUFi0rOxyCnl4CVAhEZJi979SxjCnMYfnTGmA2HOr6JpzTXUMikigy09O4/pzJPL+9jlerGsOOk/D6egQleSoEIpJArjl9IiOyMzTtxDCob+mkICeDrIzgfl2rEIjIsCvMyeTa0yfwP6/uo+pga9hxEtqB5o5AxxCACoGIBORTZ0/BgHue2xl2lIRW39LJyBHB3TEEwS9eX2xmD5jZJjPbaGaLBuwvMrPfm9krZva6mX0qyDwiEjtji3N576kV3PfCLhrbusKOk7CCnmcIhlgIzGyqmWVHn19gZl8ws+IhvPU24BF3nwnMATYO2P85YIO7zwEuAH5gZsF+xSISMzedW0lLZw/3vbAr7CgJq64l2OklYOg9ggeBHjObBtwFTAF+eaQ3mFkhcF60Pe7e6e4NA5o5UGBmBowA6oHuoccXkXg2e1wRiypHcs9zO+ns1rrGx8rdORgvPQKg1927gauAH7n73wMVR3lPJVAL3GNmL5vZCjPLH9DmduBkYC/wKvBFd3/XT4uZLTGzNWa2prZWg1REEsmS8yqpbmrnf17VtBPHqqmtm+5ej5tC0GVm1wLXAX+Ibjva4pkZwHzgDnefB7QAXx/Q5lJgHTAWmAvcHu1JvIO7L3f3he6+sKysbIiRRSQenH9SGdPKR7D86R1a1/gY1UXnGQpyMBkMvRB8ClgE/Iu77zCzKcB/HeU9VUCVu6+Ovn6ASGEY+LkPecRWYAcwc4iZRCQBpKUZN507hY37mli5rS7sOAnl7ekl4uCuIXff4O5fcPdfmVkJUODu/3qU91QDu81sRnTTYmDDgGa7otsxs9HADEAjUESSzJVzxzFqRJamnThGB96acC4OegRm9qSZFZpZKfAKkfP+/z6Et94M3Gtm64mc+rnFzJaa2dLo/u8AZ5nZq8Cfga+5+4Fj/zJEJJ7lZKZz3aLJPLWlljf2Hwo7TsKIxYRzMPRTQ0Xu3gR8ALjH3RcA7znam9x9XfTc/qnu/n53P+juy9x9WXT/Xne/xN3/xt1nu/vRTjeJSIL66JmTyM5I4+7ntK7xUMViLQIYeiHIMLMK4H/x9sViEZEhK83P4gPzx/Pg2j3UNXeEHSch1LV0kp+VTk5meqDHGWoh+DbwKLDN3V80s0rgjeBiiUgyuuGcyXR29/JfqzTAbChiMb0EDP1i8f3R0zufib7e7u4fDDaaiCSbaeUFXDCjjF+s2kl7l9Y1PppYTC8BQ79YPN7MfmNmNWa238weNLPxQYcTkeRz4zmVHGju5OFXNMDsaOqag59eAoZ+auge4GEiA7/GAb+PbhMROSZnTxvJzDEF3P2sBpgdTVz1CIAyd7/H3bujj58CGuIrIsfMzLjhnClsqj7Es1t1t/jhuHukEAQ8qhiGXggOmNnHzCw9+vgYoCGCInJcrpg7llEjsrnrWd1KejiHOrrp7OmNq1ND1xO5dbQa2Ad8iMj0ECIixyw7I53rFk3iyc0aYHY49c2xmV4Chn7X0C53v8Ldy9y93N3fT2RwmYjIcekbYKZeweDqWmIzvQSc2Apl/zBsKUQk5ZTmZ/HBBeN56OU9HNAAs3eJ1fQScGKFwIYthYikpOvPnhIdYPZm2FHiTqyml4ATKwS670tETsi08hFcNLOcXzz/pgaYDfDWqaGw7xoys0Nm1jTI4xCRMQUiIifkxnOmUNfSye/W7Qk7Slypb+4kJzONvKyMwI91xELg7gXuXjjIo8Ddg08nIklv0dSRnFxRyIpnNMCsv9rmDsoKgr9jCE7s1JCIyAkzi6xg9kZNM09u0Zrkfaob2xlTmBOTY6kQiEjo3nfqWEYXZrPiGa1g1md/UzvlKgQikiqyMtL45FlTeG5rHa/vbQw7Tujcneom9QhEJMV85PSJ5GWls+IZDTBrauumvas3OQqBmRWb2QNmtsnMNprZokHaXGBm68zsdTN7Ksg8IhK/ivIy+fBpE/j9K3vZ19gWdpxQVTe1AzC6KAkKAXAb8Ii7zwTmABv77zSzYuAnwBXufgpwdcB5RCSOXX/2FHrd+enKnWFHCVVfIUj4HoGZFQLnAXcBuHunuzcMaPYR4CF33xVtUxNUHhGJfxNK87h8dgW/XL2L5o7usOOEZn9jkhQCoBKoBe4xs5fNbIWZ5Q9ocxJQYmZPmtlLZvaJAPOISAK48dwpHGrv5tcv7g47Smj2R3sE5YWJP44gA5gP3OHu84AW4OuDtFkAvBe4FPi/ZnbSwA8ysyVmtsbM1tTW6j5jkWQ2b2IJp00u4e5nd9Dd0xt2nFBUN7VTkpdJTmZ6TI4XZCGoAqrcfXX09QNECsPANo+4e4u7HwCeJnIt4R3cfbm7L3T3hWVlWhhNJNktOW8qexra+ONr1WFHCcX+pnZGx+i0EARYCNy9GthtZjOimxYDGwY0+x1wrpllmFkecAYDLiiLSOpZPLOcyrJ8lj+9LSWnnahuamdMjO4YguDvGroZuNfM1gNzgVvMbKmZLQVw943AI8B64AVghbu/FnAmEYlzaWnGTedW8tqeJp7flnqr4lY3dsTsQjFEztEHxt3XAQsHbF42oM2twK1B5hCRxHPVvHH84LHN/OfT2zlr2qiw48RMV08vdS0dyXFqSETkRORkpnPdosk8taWWzdWps65xzaEO3EmqU0MiIsftY2dOIjczneVPp85kdNUxHkMAKgQiEsdK8rP48GkTePiVPW/9gkx2fWMIdGpIRCTqhnOm0Otw93OpMRnd24UgNoPJQIVAROLchNI83ndqBfeuepPG1q6w4wSuuqmdrPS0mCxa30eFQETi3tLzp9LS2cMvVu0MO0rg9je2U16YjZnF7JgqBCIS906uKOTCGWXc89xO2jp7wo4TqFguSNNHhUBEEsJnLphGXUsn/70muSej29/UEbN1CPqoEIhIQjhtcgkLJpWw/OntdCXpZHTuHtNF6/uoEIhIQjAzPnN+ZDK6P6zfG3acQDS1d9PW1aNCICJyOBfNLGfG6AKWPbk9KSej2x/jJSr7qBCISMJISzM+fX4lm/cf4q+bk29BwzBGFYMKgYgkmL+bM5Zxxbnc8eS2sKMMu/0xXqu4jwqBiCSUzPQ0bjx3Ci/uPMianfVhxxlWsV6iso8KgYgknA+fNoGSvEyWPZVcvYJYL1HZR4VARBJOXlYG1501mSc21rBlf/JMUV3dGNt1CPqoEIhIQrpu0WRyM9OTqlewP8ZLVPZRIRCRhFSSn8U1p0/g4XV72dPQFnacYRHG9BKgQiAiCezGcysBuDMJFq7p6unlQHMH5clWCMys2MweMLNNZrbRzBYdpt1pZtZjZh8KMo+IJJdxxblcOXccv35xN/UtnWHHOSG1fUtUJlshAG4DHnH3mcAcYOPABmaWDnwPeDTgLCKShJaeX0lbVw8/W7kz7CgnZF9j5PRWRTJdIzCzQuA84C4Ad+9094ZBmt4MPAgk3zBBEQnc9NEFXDxrND97fictHd1hxzlu22paAJgyKj/mxw6yR1AJ1AL3mNnLZrbCzN7xFZrZOOAqYNmRPsjMlpjZGjNbU1tbG1xiEUlIn7lgKg2tXdz3YuJOUb21tpmsjDQmlObF/NhBFoIMYD5wh7vPA1qArw9o8yPga+5+xJUm3H25uy9094VlZWXBpBWRhDV/YglnTCllxTPb6exOzCmqt9Y0Uzkqn/S02K1M1ifIQlAFVLn76ujrB4gUhv4WAveZ2U7gQ8BPzOz9AWYSkST1mQumsq+xnd+t2xN2lOOyrbaZqeUjQjl2YIXA3auB3WY2I7ppMbBhQJsp7j7Z3ScTKRSfdfffBpVJRJLX+SeVcXJFIcue2kZvb2JNUd3e1cPu+lamlSVZIYi6GbjXzNYDc4FbzGypmS0N+LgikmLMjM9cMJVttS08+np12HGOyY4DLfQ6TAupR5AR5Ie7+zoip3/6G/TCsLt/MsgsIpL83vs3Ffzw8S3c/tetXDZ7DGaxP99+PLbWNAPhFQKNLBaRpJGeFukVvL63iSe3JM4dhltrmkmzcG4dBRUCEUkyV80bx7jiXG7/y9aEWc5ya20zE0rzYj79dB8VAhFJKpnpaSw9v5KX3jzIqu2JsXDNtprm0C4UgwqBiCShqxdOoKwgm9v/+kbYUY6qp9fZfqAltOsDoEIgIkkoJzOdJedW8tzWOtbuOhh2nCPaXd9KZ3dvaGMIQIVARJLUR86YSEleJrf/ZWvYUY4o7DuGQIVARJJUfnYGN55byV821fBqVWPYcQ5ra60KgYhIYD6xaBLFeZnc9uctYUc5rK01zZQXZFOYkxlaBhUCEUlaBTmZ3HjOFJ7YGL+9gq01zUwN8Y4hUCEQkSR33VmTKcqNz16Bu0duHQ3xtBCoEIhIkivIyeSmc+OzV1BzqINDHd0qBCIiQYvXXkE83DEEKgQikgLitVegQiAiEkPXnTWZ4rxMvv/Y5rCjvGVrTTMF2RmUF2SHmkOFQERSQkFOJp+9YCpPball1fa6sOMA0TuGykeEPl22CoGIpIxPLJrMmMIc/u2RTaHPTNrT67y+t5GTKwpCzQEqBCKSQnIy0/nie6azdlcDT2ysCTXLxn1NNLV3c/qU0lBzgAqBiKSYqxeMZ8qofG59dBM9Ia5tvHpHZIrsM6aMDC1DHxUCEUkpGelp/O9LTmLL/mZ+t25PaDlWb69jQmkuY4tzQ8vQJ9BCYGbFZvaAmW0ys41mtmjA/o+a2froY6WZzQkyj4gIwN/OruCUsYX8++NbaO/qifnxe3udF3bWx0VvAILvEdwGPOLuM4E5wMYB+3cA57v7qcB3gOUB5xERIS3N+MfLT6bqYBt3Pbsj5sffUnOIhtYuzoiD6wMQYCEws0LgPOAuAHfvdPeG/m3cfaW7960asQoYH1QeEZH+zpk+iktPGc3tf9nKvsa2mB57dXQJzTMrk79HUAnUAveY2ctmtsLM8o/Q/gbgT4PtMLMlZrbGzNbU1tYGkVVEUtD/ee8set255Y+bYnrc1TvqGFuUw/iS8K8PQLCFIAOYD9zh7vOAFuDrgzU0swuJFIKvDbbf3Ze7+0J3X1hWVhZUXhFJMRNK8/j0+VP5/St7WR2jQWbuzgs76jmjcmToA8n6BFkIqoAqd18dff0AkcLwDmZ2KrACuNLd42O4n4ikjM+cP5Vxxbn808Ov093TG/jxttU2c6C5M26uD0CAhcDdq4HdZjYjumkxsKF/GzObCDwEfNzd42taQBFJCZK5FFcAAAxqSURBVLlZ6XzjvSezqfoQv3xhV+DHWxVn1wcg+LuGbgbuNbP1wFzgFjNbamZLo/u/CYwEfmJm68xsTcB5RETe5fLZYzh72khufXQztYc6Aj3W6h31jC7MZtLIvECPcywCLQTuvi56bv9Ud3+/ux9092Xuviy6/0Z3L3H3udHHwiDziIgMxsz41hWzae/q4ZY/DrzLffi4O6u313HGlPi5PgAaWSwiAkTWBPj0eVP5zct7WLntQCDH2FnXSs2hDs6ojJ/rA6BCICLyls9fNI0Jpbn839++Rmf38F847pv+Ol5GFPdRIRARicrJTOfbV8xmW20Ldz6zfdg//w/r9zKxNI+pZUcaUhV7KgQiIv1cOLOcy04Zw4///AZv1rUM2+fua2xj5bY6rpo3Lq6uD4AKgYjIu/zTFbPIykjjS79eR9cwjS347ct7cYcPzB83LJ83nFQIREQGqCjK5Zar/oaXdzXwH39+44Q/z915aG0VCyaVMGlkfJ0WAhUCEZFB/d2csXxw/nhu/+tWXoguInO8Xt/bxBs1zVw1L/56A6BCICJyWN+68hQmlObx979eR2Nb13F/zkNr95CVnsb7Tq0YxnTDR4VAROQwRmRncNs189jf1M7XH1xP73Esbdnd08vDr+zhopnlFOdlBZDyxKkQiIgcwdwJxXz1shn86bVqvvrg+mNe5/iZNw5woLkzLi8S98kIO4CISLxbct5UWjt7+NETb9Db69x69RzS04Z2C+iDa6soycvkghnlAac8fioEIiJD8KX3nES6GT94fAu97nz/6jlkpB/5pMqm6iYe37CfD582gayM+D0Bo0IgIjJENy+eTlqaceujm2nv6uVH18wlJzN90LYb9zXx0RWrKc7L5KZzK2Oc9NjEb4kSEYlDn7twGt983ywe3VDNx+9aTUNr57vavL63kY/cuYrsjDR+vWQRE0rjZ8rpwagQiIgco+vPmcJ/XDuPV3Y38qFlz1N1sJXunl7erGvhkdf28dEVq8nNTOe+JWcyeVT8DSAbSKeGRESOw/tOHcvI/GyW/GINF//703T19NIdvaNoXHEuv7rpTCbG0eIzR6JCICJynBZNHckDS8/ipyt3UJqfxaSR+UwZlc+sikLysxPn12viJBURiUMzxhTw3Q+cGnaME6JrBCIiKS7QQmBmxWb2gJltMrONZrZowH4zsx+b2VYzW29m84PMIyIi7xb0qaHbgEfc/UNmlgUMvHJyOTA9+jgDuCP6XxERiZHAegRmVgicB9wF4O6d7t4woNmVwM89YhVQbGbxOT2fiEiSCvLUUCVQC9xjZi+b2QozG3hD7Thgd7/XVdFt72BmS8xsjZmtqa2tDS6xiEgKCrIQZADzgTvcfR7QAnx9QJvBZm1619R+7r7c3Re6+8KysrLhTyoiksKCLARVQJW7r46+foBIYRjYZkK/1+OBvQFmEhGRAQIrBO5eDew2sxnRTYuBDQOaPQx8Inr30JlAo7vvCyqTiIi8m7kf+4o7Q/5ws7nACiAL2A58CvgwgLsvMzMDbgcuA1qBT7n7mqN8Zi3QADQO2FV0lG1He97331HAgSF/kUc+/lD2D9x+pNcDs/bfdjy5Y5m5//Mwvtf6+dDPx5H2J+LPx7FkBpju7kWDfrq7J9wDWH6s2472vN9/1wxXpqHsH7j9SK8HZj3R3LHMHPb3Wj8f+vlItp+PY8l8tGMk6sji3x/HtqM9H+z9J5ppKPsHbj/S68GynkjuWGbu/zyM77V+Po6dfj6G/jzeMx/xGIGeGkpEZrbG3ReGneNYJWJuZY6dRMytzLGTqD2CIC0PO8BxSsTcyhw7iZhbmWNEPQIRkRSnHoGISIpTIRARSXFJXQjM7G4zqzGz147jvQvM7NXoFNk/jo556Nt3s5ltNrPXzezfhjd1MLnN7J/NbI+ZrYs+/jbeM/fb/2UzczMbNXyJA/s+fyc6pfo6M3vMzMYmQOZbo1PFrzez35hZ8XBmDjD31dF/g71mNmwXaE8k62E+7zozeyP6uK7f9iP+3MfU8dzzmigPIrOfzgdeO473vgAsIjIf0p+Ay6PbLwSeALKjr8sTJPc/A19OpO91dN8E4FHgTWBUvGcGCvu1+QKwLAEyXwJkRJ9/D/heIvx8ACcDM4AngYVhZ43mmDxgWymRwbSlQEn0ecmRvq4wHkndI3D3p4H6/tvMbKqZPWJmL5nZM2Y2c+D7olNhF7r78x75P/Zz4P3R3Z8B/tXdO6LHqEmQ3IEKMPMPga8yyGSE8ZjZ3Zv6Nc0f7twBZX7M3bujTVcRmfNrWAWUe6O7b46XrIdxKfC4u9e7+0HgceCyMP+tDiapC8FhLAdudvcFwJeBnwzSZhyRCfH69J8e+yTgXDNbbWZPmdlpgaZ924nmBvh8tPt/t5mVBBf1LSeU2cyuAPa4+ytBB+3nhL/PZvYvZrYb+CjwzQCz9hmOn40+1xP56zQWhjN30IaSdTCHm2o/Xr4uIMUWrzezEcBZwP39TsdlD9Z0kG19f9llEOninQmcBvy3mVVGq3oghin3HcB3oq+/A/yAyD/6QJxoZjPLA75B5LRFTAzT9xl3/wbwDTP7R+DzwD8Nc9S3gwxT5uhnfQPoBu4dzoyDGc7cQTtSVjP7FPDF6LZpwB/NrBPY4e5Xcfj8oX9d/aVUISDSA2pw97n9N5pZOvBS9OXDRH5p9u8e958euwp4KPqL/wUz6yUy0VSQK+accG5339/vfXcCfwgwL5x45qnAFOCV6D++8cBaMzvdIzPbxmPmgX4J/A8BFgKGKXP0Iub7gMVB/lHTz3B/r4M0aFYAd78HuAfAzJ4EPunuO/s1qQIu6Pd6PJFrCVWE/3W9LayLE7F6AJPpd9EHWAlcHX1uwJzDvO9FIn/1913I+dvo9qXAt6PPTyLS7bMEyF3Rr83fA/fFe+YBbXYyzBeLA/o+T+/X5mbggQTIfBmRKeLLhjtrLH4+GOaLxceblcNfLN5B5CxCSfR56VB/7mP1COWgMfvi4FfAPqCLSAW+gchfmY8Ar0R/+L95mPcuBF4DthGZKrtvFHYW8F/RfWuBixIk9y+AV4H1RP7Sqoj3zAPa7GT47xoK4vv8YHT7eiKTfI1LgMxbifxBsy76GNY7nQLMfVX0szqA/cCjYWZlkEIQ3X599Hu8lchU+0P+uY/VQ1NMiIikuFS8a0hERPpRIRARSXEqBCIiKU6FQEQkxakQiIikOBUCSQpm1hzj460ws1nD9Fk9Fpmt9DUz+/3RZv80s2Iz++xwHFsEtEKZJAkza3b3EcP4eRn+9kRsgeqf3cx+Bmxx9385QvvJwB/cfXYs8knyU49AkpaZlZnZg2b2YvRxdnT76Wa20sxejv53RnT7J83sfjP7PfCYmV1gZk+a2QMWma//3r4546PbF0afN0cnmnvFzFaZ2ejo9qnR1y+a2beH2Gt5nrcn3RthZn82s7UWmbf+ymibfwWmRnsRt0bbfiV6nPVm9q1h/DZKClAhkGR2G/BDdz8N+CCwIrp9E3Ceu88jMjvoLf3eswi4zt0vir6eB3wJmAVUAmcPcpx8YJW7zwGeBm7qd/zbosc/6jwy0Xl2FhMZ+Q3QDlzl7vOJrIPxg2gh+jqwzd3nuvtXzOwSYDpwOjAXWGBm5x3teCJ9Um3SOUkt7wFm9ZsxstDMCoAi4GdmNp3IjI+Z/d7zuLv3n4v+BXevAjCzdUTmoHl2wHE6eXsSv5eAi6PPF/H2HPO/BL5/mJy5/T77JSJz1kNkDppbor/Ue4n0FEYP8v5Loo+Xo69HECkMTx/meCLvoEIgySwNWOTubf03mtl/AH9196ui59uf7Le7ZcBndPR73sPg/2a6/O2LbYdrcyRt7j7XzIqIFJTPAT8msp5BGbDA3bvMbCeQM8j7Dfiuu//nMR5XBNCpIUlujxFZDwAAM+ubRrgI2BN9/skAj7+KyCkpgGuO1tjdG4ksb/llM8skkrMmWgQuBCZFmx4CCvq99VHg+ui8+ZjZODMrH6avQVKACoEkizwzq+r3+Aciv1QXRi+gbiAyhTjAvwHfNbPngPQAM30J+AczewGoABqP9gZ3f5nIDJfXEFkgZqGZrSHSO9gUbVMHPBe93fRWd3+MyKmn583sVeAB3lkoRI5It4+KBCS6ylqbu7uZXQNc6+5XHu19IrGmawQiwVkA3B6906eBAJcGFTkR6hGIiKQ4XSMQEUlxKgQiIilOhUBEJMWpEIiIpDgVAhGRFPf/A9XpJAoUufxSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lm_learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learning rate finder `lr_find` finds the training loss for a wide range of learning rates. From this plot, we choose a learning rate that where the loss is still decreasing. This has two key advantages: 1) it reduces training time and 2) it minimizes the number of times the model sees any particular example, which is a form of regularization, helping to keep the model from being able to memorize the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.339741</td>\n",
       "      <td>3.852715</td>\n",
       "      <td>0.270946</td>\n",
       "      <td>11:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.525032</td>\n",
       "      <td>3.725696</td>\n",
       "      <td>0.284786</td>\n",
       "      <td>10:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 0.1\n",
    "lm_learn.fit_one_cycle(2, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learn.save(lm_trial_prefix + 'fit_head')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have only changed the weights in the last layer of the model. Now we'll unfreeze the model and allow all weights to be updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learn.load(lm_trial_prefix + 'fit_head');\n",
    "lm_learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "lm_learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxcVZ338c+veu9O70u2TqezkEAMhECzico6DgpGcXtQmZc++nqYGUd0XIaRx3mBD46jyDg6izNjBrcZRxlFmEEUQYHIIgQ6CYmBJBCWTjpbr0kv1Xv9nj/qdtKE7qRD+lbd7vq+X696pereU/f+Tqq7fn3Oufccc3dERCRzxdIdgIiIpJcSgYhIhlMiEBHJcEoEIiIZTolARCTDZac7gBNVVVXl9fX16Q5DRGRa2bBhQ5u7V4+3b9olgvr6ehobG9MdhojItGJmTRPtU9eQiEiGUyIQEclwoScCM8sys01mdu8xyrzXzNzMGsKOR0REXi0VLYJPAdsm2mlmxcAngfUpiEVERI4SaiIws1rgSuD2YxT7EvA1oD/MWEREZHxhtwi+CdwAJMbbaWargQXuPmG3UVDuOjNrNLPG1tbWEMIUEclcoSUCM7sKaHH3DRPsjwHfAD57vGO5+1p3b3D3hurqcS+DFRGR1ynMFsGFwBozewW4A7jUzH44Zn8xsBJYF5Q5H7hHA8YiIq82NJLgK7/cxubdB0M5fmiJwN1vdPdad68HrgEecvdrx+w/5O5V7l4flHkSWOPuultMRGSMPZ19fPuRl9hxoDuU46f8PgIzu8XM1qT6vCIi01VTRxyAhRWFoRw/JVNMuPs6YF3w/KYJylycilhERKabXe29ACysLArl+LqzWEQk4l5pj5OfE6OmOC+U4ysRiIhEXFN7nLqKQmIxC+X4SgQiIhG3q6OXuopwuoVAiUBEJNISCWdXR5z6ynAGikGJQEQk0lq6B+gfSrBQiUBEJDM1BVcM1YV0xRAoEYiIRFrY9xCAEoGISKQ1tfeSFTPmlxeEdg4lAhGRCGtqjzO/rICcrPC+rpUIREQibFdHPNSBYlAiEBGJtKZ2JQIRkYx1MD7Iob4hFoZ4MxkoEYiIRFZTe/KKoTq1CEREMtPopaP1Id5DAEoEIiKR1dQW3EwW4j0EoEQgIhJZTR1xaorzKMjNCvU8SgQiIhG1KwVXDEEKEoGZZZnZJjO7d5x9f2JmvzezZ8zsMTNbEXY8IiLTRVNHb2irko2VihbBp4BtE+z7kbuf7u5nAl8D/i4F8YiIRF7f4AgHugZCnWNoVKiJwMxqgSuB28fb7+5dY14WAR5mPCIi08WujtRcOgrhL17/TeAGoHiiAmb2Z8BngFzg0pDjERGZFkannw770lEIsUVgZlcBLe6+4Vjl3P1b7r4E+EvgryY41nVm1mhmja2trSFEKyISLaM3k033weILgTVm9gpwB3Cpmf3wGOXvAN413g53X+vuDe7eUF1dPfWRiohETFNHLyX52ZQV5oZ+rtASgbvf6O617l4PXAM85O7Xji1jZqeMeXkl8EJY8YiITCfJyebC7xaC8McIXsPMbgEa3f0e4BNmdjkwBHQCH051PCIiUbSrI87p80tTcq6UJAJ3XwesC57fNGb7p1JxfhGR6WRoJEFzZx9XnTE3JefTncUiIhGz92AfIwkPffrpUUoEIiIRk8orhkCJQEQkckbvIUjVYLESgYhIxDS1x8nLjlFTnJeS8ykRiIhEzN5DfcwvLyAWs5ScT4lARCRi2noGqZqVmtYAKBGIiEROe88AVbPCv6N4lBKBiEjEtPcOUlmkFoGISEYaGklwMD5EpVoEIiKZqbN3EIBKjRGIiGSmtp5kIqgqUotARCQjtfcOAGoRiIhkrPae0a4htQhERDJSW0+yRVClq4ZERDJTR+8g2TGjpCB1y8UoEYiIREh7zyCVs3IxS830EqBEICISKe29Aym9mQyUCEREIqUtaBGkUuiJwMyyzGyTmd07zr7PmNlzZrbFzB40s4VhxyMiEmXtvQMpnXAOUtMi+BSwbYJ9m4AGdz8DuBP4WgriERGJrPaeQSpTeDMZhJwIzKwWuBK4fbz97v6wu8eDl08CtWHGIyISZfHBYeKDIym9mQzCbxF8E7gBSEyi7MeA+8bbYWbXmVmjmTW2trZOZXwiIpGRjpvJIMREYGZXAS3uvmESZa8FGoDbxtvv7mvdvcHdG6qrq6c4UhGRaGgPJpxL5VoEAGHesXAhsMbM3g7kAyVm9kN3v3ZsITO7HPgCcJG7D4QYj4hIpLUHdxXPmMtH3f1Gd69193rgGuChcZLAauDbwBp3bwkrFhGR6WDGdQ1NxMxuMbM1wcvbgFnAT83sGTO7J9XxiIhERVtveloEKZnMwt3XAeuC5zeN2X55Ks4vIjIdtPcMUpSbRUFuVkrPqzuLRUQior1nIOWXjoISgYhIZLT3pn56CVAiEBGJjLaewZSPD4ASgYhIZLT3DKT8HgJQIhARiYREwulQ15CISObq6h9iOOHqGhIRyVRtabqZDJQIREQiYXR6iVSvRQBKBCIikTA64ZxaBCIiGWq0RVCR4kVpQIlARCQSRscIKgqVCEREMlJ77wDlhTlkZ6X+a1mJQEQkAtp7BtMyzxAoEYiIREI6Fq0fpUQgIhIBbb0Dabl0FJQIREQiIdk1pBaBiEhGGhxOcKhvKC3TS4ASgYhI2nXG03czGaQgEZhZlpltMrN7x9n3FjPbaGbDZvbesGMREYmitsPTS8zQRAB8Ctg2wb5dwEeAH6UgDhGRSGo/POHcDOwaMrNa4Erg9vH2u/sr7r4FSIQZh4hIlLX3JlsEM/Xy0W8CN3CSX/Rmdp2ZNZpZY2tr69REJiISETO2RWBmVwEt7r7hZI/l7mvdvcHdG6qrq6cgOhGR6GjrGSQnyyjJz07L+cNsEVwIrDGzV4A7gEvN7Ichnk9EZFrq6B2gsigPM0vL+UNLBO5+o7vXuns9cA3wkLtfG9b5RESmq3TeTAZpuI/AzG4xszXB83PMrBl4H/BtM3s21fGIiKRbW2/6JpwDSEmHlLuvA9YFz28as/1poDYVMYiIRFV7zwBLqorSdn7dWSwikmYZ1zUkIiJHxAeH6RsaSWvXkBKBiEgaHb6HIE03k4ESgYhIWh2MDwFQloa1ikcpEYiIpFF8cBiAwtystMWgRCAikkbxoREACpQIREQyU99gMhGoRSAikqHio4kgJz3zDMEkE4GZLTGzvOD5xWb2STMrCzc0EZGZry8YI5gOXUM/A0bMbCnwHWARWkxGROSkxadR11DC3YeBq4FvuvungbnhhSUikhlGE0FBTvQTwZCZfQD4MDC69nBOOCGJiGSOvqER8nNixGLpmYIaJp8I/jdwAfBld3/ZzBYBWltAROQkxQeHKcxN30AxTHL2UXd/DvgkgJmVA8Xu/tUwAxMRyQTxwZG0dgvB5K8aWmdmJWZWAWwGvmdmfxduaCIiM1/f4EhaB4ph8l1Dpe7eBbwb+J67nw1cHl5YIiKZIT6NEkG2mc0F3s+RwWIRETlJfYMjab2HACafCG4B7gdedPenzWwx8EJ4YYmIZIb40PD0GCNw95+6+xnu/qfB65fc/T2Tea+ZZZnZJjN7TUvCzPLM7L/MbKeZrTez+hMJXkRkukt2DaX3qqHJDhbXmtndZtZiZgfM7GdmNtm1hj8FbJtg38eATndfCnwDuHWSxxQRmRGmU9fQ94B7gHnAfODnwbZjCpLFlcDtExR5J/CD4PmdwGVmlr67KkREUmw6DRZXu/v33H04eHwfqJ7E+74J3AAkJtg/H9gNEExhcQioPLqQmV1nZo1m1tja2jrJkEVEom86tQjazOzaoL8/y8yuBdqP9QYzuwpocfcNxyo2zjZ/zQb3te7e4O4N1dWTyT8iItE3PJJgcCSR1imoYfKJ4KMkLx3dD+wD3kty2oljuRBYY2avAHcAl5rZ0dNSNAMLAMwsGygFOiYZk4jItNY3lP6ZR2HyVw3tcvc17l7t7jXu/i6SN5cd6z03unutu9cD1wAPufu1RxW7h+REdpBMLg+5+2taBCIiM9Ho6mTTpWtoPJ95PW8ys1vMbE3w8jtApZntDI73+ZOIR0RkWonCWgQwyUnnJjDpq3vcfR2wLnh+05jt/cD7TiIGEZFpKyqJ4GRaBOrCERE5CX1Do8tURngaajPrZvwvfAMKQolIRCRDRKVFcMxE4O7FqQpERCTTRGGZSji5riERETkJfRFpESgRiIikyZGuoelxQ5mIiEyx+ODoYLFaBCIiGUldQyIiGS4+NEJOlpGTld6vYiUCEZE06RscSfsVQ6BEICKSNvHB4bQPFIMSgYhI2kRhURpQIhARSZsoLEoDSgQiImmjFoGISIaLD42kfcI5UCIQEUmbvsFhCnXVkIhI5lLXkIhIhpvxg8Vmlm9mT5nZZjN71sz+3zhlFprZg2a2xczWmVltWPGIiERNJrQIBoBL3X0VcCZwhZmdf1SZvwX+3d3PAG4BvhJiPCIikZFIOH0zfbDYk3qClznB4+jVzlYADwbPHwbeGVY8IiJR0j8cjQnnIOQxAjPLMrNngBbg1+6+/qgim4H3BM+vBorNrHKc41xnZo1m1tja2hpmyCIiKRGVZSoh5ETg7iPufiZQC5xrZiuPKvI54CIz2wRcBOwBhsc5zlp3b3D3hurq6jBDFhFJib6ILFMJx1mzeKq4+0EzWwdcAWwds30v8G4AM5sFvMfdD6UiJhGRdIrK6mQQ7lVD1WZWFjwvAC4Hth9VpsrMRmO4EfhuWPGIiETJ6OpkM71raC7wsJltAZ4mOUZwr5ndYmZrgjIXAzvM7HlgNvDlEOMREYmMw11DEUgEobVJ3H0LsHqc7TeNeX4ncGdYMYiIRFXGDBaLiMj44kNKBCIiGa0vGCOY0TeUiYjIxA53DUXg8lElAhGRNIhHaLBYiUBEJA36BkeIGeRlp/9rOP0RiIhkoOTMo9mYWbpDUSIQEUmHvqFh8iMwPgBKBCIiaRGVtQhAiUBEJC2UCEREMlxUlqkEJQIRkbSIDw6rRSAiksnigyMU5KT/rmJQIhARSYu+IY0RiIhktCgNFkejXSJp190/xPMHuukdGKE4P5vi/BxK8rOZlZ9NXnYWWbH03/QiMpNEabBYiWCGc3cefaGN7zz2Mm09A1QU5VJZlEtFUR75OTF2tvSwbX8Xuzv6jnmcrJiRmxUjNztGdXEeq2rLWLWglFW1ZZw6t5i87Gj8QItMB+4eqcFiJYKIe6m1h2f3drGzpYedrT282NJDe+8gs0vymFNSwLyyfOaU5rOwooilNbOoryokLzuLkYTzq637+Zff7mTrni5ml+SxYm4JHb2DvNzWS0fvIP1DIyyqKuKM2jKuOaeOU+cUU1qQQ3f/MF39Q3T3D9MzMMzAUILBkREGhxMMDifYc7CP3z7fws82NgOQHTPKi3Ipzs+mJD+H4vxsygqTCadqVi6Vs/KoLMplcXUR9ZVFZGepR1Iy28BwgoRHY71iyKBE0NLdT1ffMIuriohNopujZ2CYTbs62dh0kPjQMOWFuZQV5FBWmEtJQTZZZpgZZmBAbnaMorxsZuVlU5SXTWFOFoMjCfqHRugfStA3NELCnbzsGHnZWeTlxMjPziJ3nAmnWrr6+Z9n9nLXpj1s29cFgBnUVRSytHoWq2rLaOnup7kzztOvdHCob+jwe2NBuRF3dnf0saiqiFvfczrvWj3/NX+1JxI+qf+L8bg7ew/1s3n3QbbuOURnfIiu/iG6+pIJZHdHnPbeQbr7h1/1vtysGEtqZrF89iyWzylh2exZLJtdzPyygtcdi8h0c3iZyohMMRFaIjCzfOARIC84z53ufvNRZeqAHwBlQBbweXf/ZRjx3LVxD1+9bzvFedmcXlvKGbVlrKotpSA3i67+YQ71Jb/E9h/qZ0NTJ9v3d5Hw5BdwTizG4EgijLAozs9mdkk+c0ryqSnJo7V7gMd3tpFwWFVbys3vWMF5iypZXF004bwkvQPDvNzWy4tBi2Fnaw/d/cPc+LbT+MM3zJmwf/9kvnjNjPllBcwvK+Dtp8+dsNzA8AgdvYO0dA3wYmsPOw508/z+bp56uYP/fmbv4XJFuVksnV3MospCassLqS0vOPzvvLKCcROmyHQVpdXJINwWwQBwqbv3mFkO8JiZ3efuT44p81fAT9z9X8xsBfBLoD6MYK48fS4VRblsaT7I5t2H+M5jLzE04q8pV5SbxaoFZXzi0lNoWFjO6royZuVl0zc0Qmd8iIPxQQ71DeFO8oGTcBgcTtA7kOxK6R0YJj44Qm52jIKcLApysyjIycIs2SQcGE4wMDRC/9AIbT2DHOjq50BXP+tf6iUny/j4xUt51+r5LK2ZNam6FeVls3J+KSvnl071f9tJy8vOYm5pAXNLC1i1oOxV+7r6h3jhQDc79vfw/IFunj/QTWNTJz/fso+RxJHPJmYwpyQ/mRgqClhQXsj88gJqy5PP55Tmk6PuJplGjqxONsMTgbs70BO8zAkeR3/zOlASPC8F9hKSBRWFLKgo5P0NCwDoHxphx/5uhhMJSgtyKMnPoaQgZ8K/ugtzsynMzWZ+WUFYIWackvwczl5YwdkLK161fXgkwf6ufpo7+9jdEWd3Zx/NnXGaO/p44sV27u7ag4/5SYoZzCsroK6ikLrgc142u5iz6sqonJWX4lqJHF/fYLKHISPGCMwsC9gALAW+5e7rjyryReABM7seKAIun+A41wHXAdTV1U1JbPk5Wa/5C1WiITsrFnQLFXL+4srX7B8cTrDvUB97OvuSyaIzzu6OOLs64vxmWwttPQOHyy6sLOSsunJOn1/KrLxsYjEjO2bEgqugCnOPtNiK8rKpmpVLcX5OKqsrGSgetAgyoWsIdx8BzjSzMuBuM1vp7lvHFPkA8H13/7qZXQD8R1AmcdRx1gJrARoaGl7bnyMZJTc7xsLKIhZWFo27v3dgmOf2dbGxqZONuzp5bGcbd2/aM+njF+VmMTsYs5lbmrwya15ZcqxiftDyiMo88jI9jY4RzPiuobHc/aCZrQOuAMYmgo8F23D3J4IB5iqgJRVxycxUlJfNOfUVnFOf7HJyd9p7BxkcTjCScIYTzkgiOVbTNzhCPHj0DgzT1jPAga4BDnT3c+BQP0+93MH+rv5XjVmYwYLyQpZUF7G4ehZLa2axfE4xy2cXU5QXjaa+RNvoVUMzvkVgZtXAUJAECkh2+9x6VLFdwGXA983sNCAfaA0rJslMZkbVSYwVjCSclu7+w11Rh6/Qau3ldy+2MzB8pAG7oKKA5bNLWFVbylkLy1m1IHmxgchYowvXF0Zk0rkwo5gL/CAYJ4iRvDroXjO7BWh093uAzwL/ZmafJjlw/JFgkFkkMrJidvjKp4b6V+9LJJzmzj627+9ix/5uth/oZvu+Ln6z7QCQHMheNruY1XVlrJhXyoq5JZw6Ry2HTJdJVw1tAVaPs/2mMc+fAy4MKwaRsMViRl1lIXWVhbz1DXMObz/UN8Qzuw8eHqf45e/38+OndgPJrqX6yiJW1ZZydn0FDQvLWTa7WPM5ZZB4pnQNiWSy0oIcLlpWzUXLqoEjd2Jv29vFc/u6eHbvIR5/sf3wTXXF+dmcVVfOqtpSTg9udqwpyU9nFSRE8Uy5s1hEjhh7J/blK2YDyeSwu6OPxqYOGps62djUyT893MrouPTskjzOqivnvEUVnL+kkmU1xZqGY4boGxohPycWmc9TiUAkTcyOdCu9+6xaIHk1ybN7D7Gl+RBbmg/y9Cud3Ld1PwBlhTmcW1/B+YsrOW9xBafNKYnMF4mcmOTMo9H5+o1OJCJCQW4WDfUVNNQfudt6d0ec9S93sP6ldp58uZ0HnksORJfkZ3PuogrOXZS8O3vl/BJNBz5NJJepjM5npUQgEnGj06O89+xkq2HvwT7Wv9zO+pc6WP9yB7/ZlrztJjc7lrxsta6c2vICqovzko9Z+cwuzVOSiJC+CK1OBkoEItPOvLICrl5dy9Wrk4mhpbufjU2dbGjqpLGpk+8+/vJrJlTMz4lx3qJK3nxKFRctq2ZpzSzM1K2ULlFaphKUCESmvZrifK5YOZcrVianAx9JOB29g7R2D9DS3U9L9wDP7e3ikRda+etfbOOvf7GNOSX5vGVZFRctq+FNS6soLdT8SqkUpWUqQYlAZMbJitnhbqEVhyf3TWrujPPYC2389vlW7tu6n580NhMzOHNBGZedNpt3njmP2vLCNEWeOeJDw9QUR+fyYCUCkQxSW17INefWcc25dQyPJNjcfJDf7mhl3fOt3Hb/Dm67fwfnL67g3WfV8vbT52p6jJDE1SIQkSjIzoodXg/iM29dzu6OOHdv2sNdG5u54c4t3PQ/W7lkeQ1/+IY5XHJqDaUF6j6aKn2DIxTqqiERiZoFFYV88rJTuP7SpWzcdZC7NzXzwLMHuG/rfrJjxgVLKrn8tNlcuLSKJdVFGmw+CRosFpFIMzPOXljO2QvLuWXNSp5pPsgDzx7ggWf3c/M9zwLJu57fuKSKC5ZUcvHy6kj1d08HycHi6Hz9RicSEYmcWMw4q66cs+rK+fzbTqWpPTn19uM723jk+Vbu3rQHM2hYWB5cuTRHy7kex/BIgsGRhFoEIjI9ja4M94Fz63B3tu/vDrqP9vGle5/jS/c+xxm1pVy0rJo3La1idV05udmxdIcdKaOrkykRiMi0Z2acNreE0+aW8KnLT+Hltl7u27qP3zx3gH9e9yL/+NBOinKzOH9xJW86pYo3n1KtsQWOrE6mq4ZEZMZZVFXExy9eyscvXsqhviGeeLGdx3a28ugLbTy4PTkNxtzSfN4cJIVLTq3JyMtTo7YWASgRiEgISgtyuGLlHK5YmVysZ3dHnEdfaOOxna38KriRLTc7xiXLq7nyjHlcdmpNxqzaFh9dnSwiy1SCEoGIpMCCikI+eF4dHzyvjpGEs3FXJ7/Yso9f/n4f9z97gLzsGBcureL8xclptlfMLSE7a2aOLURt4XoId/H6fOARIC84z53ufvNRZb4BXBK8LARq3L0srJhEJP2yYsY59RWcU1/BTVetoLGpk19s2cujL7TxUNCFVJyXnGL7yjPm8odvmDOjWguZ1jU0AFzq7j1mlgM8Zmb3ufuTowXc/dOjz83sesZZ41hEZq5YzA6vqQDQ0tXPky938ORL7TzyfCsP/qSFwtytXLFyDu9eXcsFSyqn/drO8UwaLHZ3B3qClznBwyd+Bx8Abj7GfhGZ4WpK8lmzah5rVs3D3Wls6uSujc3cu2Ufd23cw/yyAj54Xh3vb1hAdXFeusN9XfqGkmMEGbNCmZllARuApcC33H39BOUWAouAhybYfx1wHUBdXV04wYpIpJgd6UK6+R1v4NfPHeCOp3dx2/07+OZvnueKlXP5o/MXck59+bS6JDVqC9dDyInA3UeAM82sDLjbzFa6+9Zxil5DcgxhZILjrAXWAjQ0NByrVSEiM1B+ThbvWDWPd6yax86WHv5zfRN3bmjm55v3sqiqiHevns/VZ82fFlNoR/E+gpQMy7v7QWAdcMUERa4BfpyKWERkeltaM4ub3/EG1v/fy/jae8+gpjiPr//6ed5068Ncs/YJ7trYzOBwIt1hTiiKg8WhJQIzqw5aAphZAXA5sH2ccsuBcuCJsGIRkZmnMDeb9zcs4L/++AIeveESPvsHy9h/qJ/P/GQzb7r1Ib718E4OxgfTHeZrxAdHyMkyciJ0eWyYkcwFHjazLcDTwK/d/V4zu8XM1owp9wHgjmBwWUTkhC2oKOT6y07h4c9dzA8+ei7L5xRz2/07uOArD3Hz/2zlpdae4x8kRfoGhyM1PgDhXjW0hXEuB3X3m456/cWwYhCRzGJmXLSsmouWVbNtXxe3P/oyP3pqFz94ook3n1LFH52/kEtPrUnrzWrJtQiic8UQ6M5iEZmhTptbwtffv4q/fNty/uup3fzoqV1c9x8bDl+Ceu15CyktTP2qa/GhaC1KAykaLBYRSZea4nyuv+wUHr3hEv712rOprypMdht99UG+eM+z7O6IpywWd6e1eyBSVwyBWgQikiGys2KHJ8Lbtq+Lf3v0JX74ZBP//sQrvP30uVz3lsWcURveDDdd/UN89iebeerlDv74osWhnef1sOk2RtvQ0OCNjY3pDkNEZoB9h/r4/uOv8KP1u+geGOaCxZVcd9FiLl5WPaU3qW3b18Wf/nADzZ193Pj20/johfUpvwnOzDa4e8O4+5QIRCTTdfcP8eOndvHdx15hf1c/y2cX87E3LeIPVsymvCj3dR/X3fnvZ/Zw412/pyQ/h2996CzOqa+YwsgnT4lARGQSBocT/HzzXtY+8hI7DnQTMzh7YTmXnjqby0+rYWnNrEn9Jd/eM8Ddm/bw08Zmdhzo5rxFFfzjB1dTU5yfglqMT4lAROQEuDubmw/x0LYDPLi9hWf3dgEwrzSfNy6t4sKllbxxSRWzS/JJJJyW7gGaO+Ps6ojzwLMHeHD7AYZGnFULynh/Qy3/q2FB2tdXUCIQETkJ+w718dD2Fh7f2cbvXmznYHwIgNkleXT0DjI0cuR7tLIol6tXz+d9DQtYPqc4XSG/xrESga4aEhE5jrmlBXzovIV86LyFJBLOc/u6+N2LbWzf101NST615QXBo5CFlYWRmj5iMpQIREROQCxmrJxfysr5pekOZcpMr7QlIiJTTolARCTDKRGIiGQ4JQIRkQynRCAikuGUCEREMpwSgYhIhlMiEBHJcNNuigkzawWaxmwqBQ6NU/To7cd6PdHzKqDtJEOeKL4TKTfevslsO14dp6J+E8VyouVeTx0nU98of4bjbZ9pP6fjbZ9pdZyK30UIv44L3b163D3uPq0fwNrJbD/W62M8bwwrvhMpN96+yWw7Xh2non7prONk6hvlz/B4dZjMZ6g6pr+OU/G7mOo6Hv2YCV1DP5/k9mO9nuj5VJjs8Y5Vbrx9k9k20+s42fqerLDqN972mfYZjrd9ptVxOv4uvsq06xpKJTNr9Alm65sJZnr9QHWcKVTHcM2EFkGY1qY7gJDN9PqB6jhTqI4hUotARCTDqUUgIpLhlAhERDJcRh/+538AAAZ8SURBVCQCM/uumbWY2dbX8d6zzez3ZrbTzP7BxqxcbWbXm9kOM3vWzL42tVGfcJxTXkcz+6KZ7TGzZ4LH26c+8hOKM5TPMdj/OTNzM6uauohPXEif45fMbEvwGT5gZvOmPvITijOMOt5mZtuDet5tZmVTH/mkYwyjfu8LvmcSZjb1A8one93qdHgAbwHOAra+jvc+BVwAGHAf8LZg+yXAb4C84HXNDKzjF4HPpfvzC7OOwb4FwP0kb1Ssmml1BErGlPkk8K8zsI5vBbKD57cCt86w+p0GLAfWAQ1THXNGtAjc/RGgY+w2M1tiZr8ysw1m9qiZnXr0+8xsLslfoic8+Wn8O/CuYPefAl9194HgHC3h1uLYQqpjpIRYx28ANwBpv3IijDq6e9eYokWkuZ4h1fEBdx8Oij4J1IZbi4mFVL9t7r4jrJgzIhFMYC1wvbufDXwO+OdxyswHmse8bg62ASwD3mxm683st2Z2TqjRvj4nW0eATwTN7e+aWXl4ob5uJ1VHM1sD7HH3zWEHehJO+nM0sy+b2W7gQ8BNIcb6ek3Fz+qoj5L8azpKprJ+Uy4jF683s1nAG4Gfjukqzhuv6DjbRv+aygbKgfOBc4CfmNniIJOn3RTV8V+ALwWvvwR8neQvWSScbB3NrBD4AsluhUiaos8Rd/8C8AUzuxH4BHDzFIf6uk1VHYNjfQEYBv5zKmM8GVNZv7BkZCIg2RI66O5njt1oZlnAhuDlPSS/CMc2MWuBvcHzZuCu4Iv/KTNLkJw0qjXMwE/ASdfR3Q+Med+/AfeGGfDrcLJ1XAIsAjYHv6C1wEYzO9fd94cc+2RNxc/qWD8CfkGEEgFTVEcz+zBwFXBZVP4gC0z1Zzj10jWgkuoHUM+YwRvgd8D7gucGrJrgfU+T/Kt/dPDm7cH2PwFuCZ4vA3YT3KA3g+o4d0yZTwN3zLTP8agyr5DmweKQPsdTxpS5HrhzBtbxCuA5oDrddQujfmP2ryOEweK0/4el6EP5MbAPGCL5l/zHSP4l+Ctgc/ADdNME720AtgIvAv80+mUP5AI/DPZtBC6dgXX8D+D3wBaSf7HMTVV9UlXHo8qkPRGE9Dn+LNi+heSkZPNnYB13kvxj7JngkbYro0Kq39XBsQaAA8D9UxmzppgQEclwmXzVkIiIoEQgIpLxlAhERDKcEoGISIZTIhARyXBKBDLtmVlPis93u5mtmKJjjQSzgm41s58fb9ZMMyszs49PxblFRunyUZn2zKzH3WdN4fGy/cgEZqEaG7uZ/QB43t2/fIzy9cC97r4yFfFJZlCLQGYkM6s2s5+Z2dPB48Jg+7lm9jsz2xT8uzzY/hEz+6mZ/Rx4wMwuNrN1ZnZnMM/9f46ZG37d6JzwZtYTTOi22cyeNLPZwfYlweunzeyWSbZanuDIZHizzOxBM9toyfnp3xmU+SqwJGhF3BaU/YvgPFvM7P9N4X+jZAglApmp/h74hrufA7wHuD3Yvh14i7uvJjkL59+Mec8FwIfd/dLg9Wrgz4EVwGLgwnHOUwQ86e6rgEeA/zPm/H8fnP+488UE885cRvIOboB+4Gp3P4vk2hdfDxLR54EX3f1Md/8LM3srcApwLnAmcLaZveV45xMZK1MnnZOZ73JgxZjZHkvMrBgoBX5gZqeQnNkxZ8x7fu3uY+eRf8rdmwHM7BmS88c8dtR5BjkyGd8G4A+C5xdwZM2DHwF/O0GcBWOOvQH4dbDdgL8JvtQTJFsKs8d5/1uDx6bg9SySieGRCc4n8hpKBDJTxYAL3L1v7EYz+0fgYXe/OuhvXzdmd+9RxxgY83yE8X9fhvzIQNtEZY6lz93PNLNSkgnlz4B/ILluQDVwtrsPmdkrQP447zfgK+7+7RM8r8hh6hqSmeoBkvPuA2Bmo1MAlwJ7gucfCfH8T5LskgK45niF3f0QyWUkP2dmOSTjbAmSwCXAwqBoN1A85q33Ax8N5rzHzOabWc0U1UEyhBKBzASFZtY85vEZkl+qDcEA6nMkpw0H+BrwFTN7HMgKMaY/Bz5jZk8Bc4FDx3uDu28iOTvlNSQXVmkws0aSrYPtQZl24PHgctPb3P0Bkl1PT5jZ74E7eXWiEDkuXT4qEoJg9bM+d3czuwb4gLu/83jvE0kHjRGIhONs4J+CK30OEqElPkWOphaBiEiG0xiBiEiGUyIQEclwSgQiIhlOiUBEJMMpEYiIZLj/D8pLWtEHVrZqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lm_learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we choose a learning rate as large as possible while still avoiding the catastrophic forgetting 'cliff'. This point is just before the steep increase in loss, which occurrs here at approximately `lr=0.6`).\n",
    "\n",
    "Note that if we get too close to the 'cliff', we'll see our loss and accuracy metrics start getting significantly worse. If this happens, resume from a previously saved model and try a lower learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.666841</td>\n",
       "      <td>3.563700</td>\n",
       "      <td>0.298643</td>\n",
       "      <td>15:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.531399</td>\n",
       "      <td>3.453411</td>\n",
       "      <td>0.315143</td>\n",
       "      <td>16:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.413248</td>\n",
       "      <td>3.420279</td>\n",
       "      <td>0.320375</td>\n",
       "      <td>15:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.300605</td>\n",
       "      <td>3.418946</td>\n",
       "      <td>0.323786</td>\n",
       "      <td>15:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 0.01\n",
    "lm_learn.fit_one_cycle(4, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice in epoch 4 that the training loss drops below the validation loss which means we are beginning to overfit. By adjusting the dropout with the `drop_mult` parameter during model initialization, we can directly trade-off our model's tendency to overfit with our model's capacity to learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learn.save(lm_trial_prefix + 'fine_tuned')\n",
    "lm_learn.save_encoder(lm_trial_prefix + 'fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fitting for more cycles may improve our model further, although the model may also begin overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learn.load(lm_trial_prefix + 'fine_tuned');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "lm_learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhc9X3v8fd3ZrRLlhfJeJGxANtAcMAGQVhCwuImBhKTjZS09JKEG57b22YpTfKEJ73chjxpk3B7SXvTJYQ2oSFLA5TgkBBCAwYSsI2MFwxmM7bxLlmWZO2zfe8fc2QrQpJlW2cWzef1PPP4zDm/mfOZ8Wi+c87vnN8xd0dERIpXJNcBREQkt1QIRESKnAqBiEiRUyEQESlyKgQiIkUulusAx6qurs4bGxtzHUNEpKCsW7fugLvXj7Ss4ApBY2Mjzc3NuY4hIlJQzGzHaMu0a0hEpMipEIiIFDkVAhGRIhd6ITCzqJmtN7OHR1n+UTN7ycxeNLMfhZ1HRER+XzY6iz8LbAGmDF9gZguBW4FL3L3dzGZmIY+IiAwR6haBmTUA1wB3j9LkU8A/uns7gLu3hJlHRETeKuxdQ98CvgikR1m+CFhkZr8zs9VmtnykRmZ2s5k1m1lza2trWFlFRIpSaIXAzN4HtLj7ujGaxYCFwGXAx4C7zWzq8Ebufpe7N7l7U339iOdDiIhMWum087VfvMSmXR2hPH+YWwSXACvMbDvwE+AKM7t3WJtdwEPunnD3bcArZAqDiIgEXt7XxXef3sbrLd2hPH9ohcDdb3X3BndvBK4HHnf3G4Y1+xlwOYCZ1ZHZVfRGWJlERArRM1sPAHDRaTNCef6sn0dgZreb2Yrg7qNAm5m9BDwBfMHd27KdSUQknz27tY1T6qqYXVsRyvNnZawhd18FrAqmbxsy34FbgpuIiAyTTKVZs+0gK5bMCW0dOrNYRCSPbd5ziO6BJBeHtFsIVAhERPLaYP/AhaeqEIiIFKVnt7Zx+kk11FWXhbYOFQIRkTw1kEzx3PaDoR0tNEiFQEQkT23c2Ul/Ih1q/wCoEIiI5K1nth7ADN5xigqBiEhRemZrG4vn1FJbWRLqelQIRETyUF88xfo320PfLQQqBCIieWndjnYSKQ+9oxhUCERE8tIzWw8QixjnN04PfV0qBCIieWjdjnbOmltLVVn4IwGpEIiI5KHXW7o5c1ZNVtalQiAikmfaugdo64mzYGZ1VtanQiAikmcGL0CjQiAiUqReb80UgoUnadeQiEhRem1/N5WlUebUlmdlfSoEIiJ5ZmtrNwtmVmNmWVlf6IXAzKJmtt7MHh6jzUfMzM2sKew8IiL57rX93Syoz07/AGRni+CzwJbRFppZDfAZYE0WsoiI5LWu/gT7DvWz4KRJUgjMrAG4Brh7jGZfBb4J9IeZRUSkEAweMbRwZnY6iiH8LYJvAV8E0iMtNLOlwDx3H3W3UdDuZjNrNrPm1tbWEGKKiOSH17J86CiEWAjM7H1Ai7uvG2V5BLgT+MujPZe73+XuTe7eVF9fP8FJRUTyx9aWbkpjEeZNq8jaOsPcIrgEWGFm24GfAFeY2b1DltcAi4FVQZsLgZXqMBaRYvZaSzen1lURi2bvoM7Q1uTut7p7g7s3AtcDj7v7DUOWd7p7nbs3Bm1WAyvcvTmsTCIi+e71lu6s7haCHJxHYGa3m9mKbK9XRCTf9SdS7GzvzWpHMUD445sC7r4KWBVM3zZKm8uykUVEJF9tbe3GPbsdxaAzi0VE8sbhQ0ezeA4BqBCIiOSN1/Z3E40YjTOqsrpeFQIRkTzxeks382dUUhrL7lezCoGISJ54raWLhVnuHwAVAhGRvBBPptnR1pv1jmJQIRARyQs72npIpj3rh46CCoGISF54ZX8XkP1DR0GFQEQkLzRvb6eiJMrps7RFICJSlNZuO8i586dSksUxhgapEIiI5Nih/gRb9h3igsYZOVm/CoGISI6t296OO5x/yrScrF+FQEQkx9ZsO0hJ1Fg6T4VARKQoPbf9IGc3TKWiNJqT9asQiIjkUF88xaZdHVxwyvScZVAhEBHJofU720mknAsaVQhERIrS2m0HMYPzGnPTPwAqBCIiOfXc9oO8bfYUppSX5CxD6IXAzKJmtt7MHh5h2S1m9pKZbTKz35jZ/LDziIjki3gyzbod7Zyfw91CkJ0tgs8CW0ZZth5ocvezgfuBb2Yhj4hIXti8p5P+RJp35LCjGEIuBGbWAFwD3D3Scnd/wt17g7urgYYw84iI5JPnth0E4PzJXAiAbwFfBNLjaHsT8MhIC8zsZjNrNrPm1tbWicwnIpIza7cd5LT6Kuqqy3KaI7RCYGbvA1rcfd042t4ANAF3jLTc3e9y9yZ3b6qvr5/gpCIi2ZdMpXlu+8Gcnj8wKBbic18CrDCzq4FyYIqZ3evuNwxtZGbLgC8D73b3gRDziIjkjY27OjnUn+SdC3L/4za0LQJ3v9XdG9y9EbgeeHyEIrAU+A6wwt1bwsoiIpJvnnylhYjBOxfU5TpK9s8jMLPbzWxFcPcOoBq4z8w2mNnKbOcREcmFJ19tZenJ06itzN35A4PC3DV0mLuvAlYF07cNmb8sG+sXEcknbd0DbNrdyS3LFuU6CqAzi0VEsu7p1w7gDu8+Pff9A6BCICKSdU++2sqMqlIWz6nNdRRAhUBEJKvSaeepV1t516J6IhHLdRxAhUBEJKs27+mkrSfOuxflx24hUCEQEcmqVa+0YgaXLsz9YaODVAhERLLoyVdbOXtuLTNyPKzEUCoEIiJZ0tEbZ/2b7bz79Jm5jvJ7VAhERLLkt68fIO3kVf8AqBCIiGTNb7a0MLWyhHMa8uOw0UEqBCIiWdAXT/HrF/ex/KxZxKL59dWbX2lERCapx19uoSee4v3nzMl1lLdQIRARyYKVG3dTX1PGhafOyHWUt1AhEBEJ2aH+BE+80so1b59NNE/OJh5KhUBEJGSPbt5HPJlmxZL82y0EKgQiIqFbuXEP86ZXsHTe1FxHGZEKgYhIiA50D/DM1jbef/YczPJvtxBkoRCYWdTM1pvZwyMsKzOz/zCz181sjZk1hp1HRCSbfvnCXlJpz9vdQpCdLYLPAltGWXYT0O7uC4A7gW9kIY+ISNas3LCHRSdVc8asKbmOMqpQC4GZNQDXAHeP0uRa4J5g+n7gSsvXbScRkWO0q72X5h3trMjDcweGCnuL4FvAF4H0KMvnAjsB3D0JdAL5d5CtiMhx+OlzOzGDa5fMzXWUMYVWCMzsfUCLu68bq9kI83yE57rZzJrNrLm1tXXCMoqIhCWRSvOT53Zy2aJ65k2vzHWcMYW5RXAJsMLMtgM/Aa4ws3uHtdkFzAMwsxhQCxwc/kTufpe7N7l7U319fo3aJyIykt9saaGla4A/esf8XEc5qtAKgbvf6u4N7t4IXA887u43DGu2ErgxmP5I0OYtWwQiIoXmh2t2MLu2nMtPz/8fr1k/j8DMbjezFcHdfwVmmNnrwC3Al7KdR0Rkou1o6+Hp1w5w/fkn591IoyOJZWMl7r4KWBVM3zZkfj9wXTYyiIhky4/X7iQaMf7w/Hm5jjIu+V+qREQKyEAyxX3NO7nyjJnMqi3PdZxxUSEQEZlAj764n7aeOH98Yf53Eg9SIRARmUA/WrODedMruHRBXa6jjJsKgYjIBNnd0cfqNw5y3XnziOThdQdGo0IgIjJBHtqwG4AP5PmZxMOpEIiITAB358Hnd9M0fxonz8jvM4mHUyEQEZkAL+09xGst3XxgaWFtDYAKgYjIhPjZ+t2URI1r3j4711GOmQqBiMgJSqWdhzbs4bLTZzKtqjTXcY6ZCoGIyAl6dmsbLV0DBddJPGhchcDMTjOzsmD6MjP7jJnl51WYRUSy7GcbdlNTFuPKM2fmOspxGe8WwQNAyswWkBko7hTgR6GlEhEpEH3xFL/avI+r3j6L8pJoruMcl/EWgnRwBbEPAt9y978ACq9HRERkgv3Xlv10DyQL8mihQeMtBAkz+xiZawc8HMwrCSeSiEjheGjDHmZNKecdpxTuVXbHWwg+AVwEfM3dt5nZKcDwq42JiBSVjt44T77awvvPmU20gIaUGG5c1yNw95eAzwCY2TSgxt2/HmYwEZF898jmfSRSnvcXpz+a8R41tMrMppjZdGAj8D0z+7/hRhMRyW8PbdjNqfVVnDVnSq6jnJDx7hqqdfdDwIeA77n7ecCysR5gZuVmttbMNprZi2b2lRHanGxmT5jZejPbZGZXH/tLEBHJvr2dfazZdpBrz5mLWeHuFoLxF4KYmc0GPsqRzuKjGQCucPdzgCXAcjO7cFibvwJ+6u5LyVzg/p/G+dwiIjn18Ma9uMOKJXNyHeWEjbcQ3A48Cmx19+fM7FTgtbEe4Bndwd2S4ObDmwGD21S1wJ5x5hERyamHNu7mnIZaTqmrynWUEzauQuDu97n72e7+p8H9N9z9w0d7nJlFzWwD0AI85u5rhjX5a+AGM9sF/BL49CjPc7OZNZtZc2tr63gii4iE5vWWbjbvPsSKAu8kHjTezuIGM3vQzFrMbL+ZPWBmDUd7nLun3H0J0ABcYGaLhzX5GPB9d28ArgZ+YGZvyeTud7l7k7s31dfXjyeyiEhoVm7cgxm8/+zJcV7teHcNfQ9YCcwB5gI/D+aNi7t3AKuA5cMW3QT8NGjzLFAOFM6FPkWk6Lg7Kzfs5qJTZzBzSnmu40yI8RaCenf/nrsng9v3gTF/mptZ/eDAdGZWQeYoo5eHNXsTuDJocyaZQqB9PyKStzbs7GB7Wy/XToJO4kHjLQQHzOyGYJ9/1MxuANqO8pjZwBNmtgl4jkwfwcNmdruZrQja/CXwKTPbCPwY+Li7D+9QFhHJG/ev20V5SYSrC/ACNKMZ15nFwCeBbwN3kjnS5xkyw06Myt03AUtHmH/bkOmXgEvGG1ZEJJf6Eyl+vnEPy8+aRU355BlubbxHDb3p7ivcvd7dZ7r7B8icXCYiUjT+a8t+DvUn+ch583IdZUKdyBXKbpmwFCIiBeCBdbuYXVvORacV7kijIzmRQlDY51SLiByDlkP9PPlqKx86d25BjzQ6khMpBOrUFZGi8eD63aQdPnTuUU+hKjhjdhabWRcjf+EbUBFKIhGRPOPuPPD8Ls49eSqn1VfnOs6EG7MQuHtNtoKIiOSrF3Z38ur+br72weGDI0wOJ7JrSESkKDywbhelsQjvO3vynEQ2lAqBiMgY0mnnl5v3ceUZM6mtmDznDgylQiAiMob1O9tp7Rpg+eJZuY4SGhUCEZEx/GrzPkqixuVnzMx1lNCoEIiIjMLdefTF/VyyoI4pk2hIieFUCERERrFlbxdvHuxl+VmTd7cQqBCIiIzqVy/uI2Kw7G0n5TpKqFQIRERG8ejmfTQ1TqeuuizXUUKlQiAiMoJtB3p4ZX/XpN8tBCoEIiIjevTFfQC8dxIfNjpIhUBEZAS/2ryPsxtqmTt18g+rFlohMLNyM1trZhvN7EUz+8oo7T5qZi8FbX4UVh4RkfHa19nPhp0dvLcIdgvB+C9VeTwGgCvcvdvMSoDfmtkj7r56sIGZLQRuBS5x93Yzm7xnbIhIwTi8W0iF4MQEF6HvDu6WBLfhQ1p/CvhHd28PHtMSVh4RkfF6ZPNeFs6sZsHMyTfk9EhC7SMws6iZbQBagMfcfc2wJouARWb2OzNbbWbLR3mem82s2cyaW1tbw4wsIkXuQPcAa7cd5Kq3z851lKwJtRC4e8rdlwANwAVmNnww7xiwELgM+Bhwt5lNHeF57nL3Jndvqq+vDzOyiBS5X7+4n7TDVUVwtNCgrBw15O4dwCpg+C/+XcBD7p5w923AK2QKg4hITjyyeS+n1FVxxqziuS5XmEcN1Q/+ujezCmAZ8PKwZj8DLg/a1JHZVfRGWJlERMbS3hPnma1tLF88C7PJdYH6sYR51NBs4B4zi5IpOD9194fN7Hag2d1XAo8C7zGzl4AU8AV3bwsxk4jIqB7bsp9U2rl6cfH0D0C4Rw1tApaOMP+2IdMO3BLcRERy6leb99EwrYLFc6fkOkpW6cxiERHgUH+Cp19rZflZxbVbCFQIREQAeHxLC4mUF9Vho4NUCEREyBwtdNKUMpbOe8sR7JOeCoGIFL3OvgSrXmnlqsWziUSKa7cQqBCIiPDzjXsYSKb50Llzcx0lJ1QIRKTo3bduF2fMquHtc2tzHSUnVAhEpKi9ur+LjTs7uK5pXtEdLTRIhUBEitp9zTuJRYwPLJmT6yg5o0IgIkUrkUrz4PrdLDvzJGZM8gvUj0WFQESK1hMvt3CgO851TQ25jpJTKgQiUrR+2ryL+poy3r2ouIe3VyEQkaLU0tXPE6+08OFzG4hFi/ursLhfvYgUrQef300q7UW/WwhUCESkCKXTzg/XvMn5jdM4rb44rks8FhUCESk6q15t4c2Dvdx4cWOuo+QFFQIRKTr3PLODmTVlvPes4rku8VhUCESkqGw70MOTr7byx++YT0mRdxIPCvOaxeVmttbMNprZi2b2lTHafsTM3MyawsojIgLwg2d3UBI1PvaOebmOkjfCvGbxAHCFu3ebWQnwWzN7xN1XD21kZjXAZ4A1IWYREaFnIMl963Zy1eLZzKwpz3WcvBHaFoFndAd3S4Kbj9D0q8A3gf6wsoiIADy4fjdd/UluvHh+rqPklVB3kJlZ1Mw2AC3AY+6+ZtjypcA8d3/4KM9zs5k1m1lza2triIlFZLJyd/792e2cNWcK5548Lddx8kqohcDdU+6+BGgALjCzxYPLzCwC3An85Tie5y53b3L3pvr64j4VXESOz7Nb23h1fzf/7aL5RTvc9Giy0mXu7h3AKmD5kNk1wGJglZltBy4EVqrDWETC8N2n32BGVSnXLinOq5CNJcyjhurNbGowXQEsA14eXO7une5e5+6N7t4IrAZWuHtzWJlEpDi93tLFE6+08icXzae8JJrrOHknzC2C2cATZrYJeI5MH8HDZna7ma0Icb0iIr/nX3+7jbJYhD+5UJ3EIwnt8FF33wQsHWH+baO0vyysLCJSvA50D/DA87v58LkNRX3xmbHotDoRmdR+8OwO4sk0N73zlFxHyVsqBCIyafUnUty7egdXnjGTBTM1yuhoVAhEZNJ6cP1u2nri3HSptgbGokIgIpNSOu3c/fQbnDVnChedOiPXcfKaCoGITEqrXm1ha2sPn7r0VJ1AdhQqBCIyKX33qW3Mri3nmrNn5zpK3lMhEJFJZ/PuTp59o42PX9yoaw6Mg94hEZl07n76DapKo1x/wcm5jlIQVAhEZFLZ29nHw5v28ofnn0xtRUmu4xQEFQIRmVS+/7vtpN35xCWNuY5SMFQIRGTS6B5I8qO1b3LV22czb3plruMUDBUCEZk0frzmTbr6k3zq0lNzHaWgqBCIyKTQF0/xnae2csmCGSyZNzXXcQpKmBevFxHJmntX7+BAd5x/XrYo11EKjrYIRKTg9caT/MuTW3nngjrOb5ye6zgFR4VARArevat30NYT5y/+YGGuoxQkFQIRKWi98STfefINLl1Yx3nztTVwPMK8ZnG5ma01s41m9qKZfWWENreY2UtmtsnMfmNmuo6ciByTHzyb2Rr4nPoGjluYWwQDwBXufg6wBFhuZhcOa7MeaHL3s4H7gW+GmEdEJpn2njjfeeoN3rWonvPmT8t1nIIVWiHwjO7gbklw82FtnnD33uDuaqAhrDwiMrm4O5+/byPd/Um+tPyMXMcpaKH2EZhZ1Mw2AC3AY+6+ZozmNwGPjPI8N5tZs5k1t7a2hhFVRArMv/1uO795uYVbrz6Dt82Zkus4BS3UQuDuKXdfQuaX/gVmtnikdmZ2A9AE3DHK89zl7k3u3lRfXx9eYBEpCJt2dfD1R7aw7MyT+PjFjbmOU/CyctSQu3cAq4Dlw5eZ2TLgy8AKdx/IRh4RKVxd/Qk+/eP11FWXccdHztbVxyZAmEcN1ZvZ1GC6AlgGvDyszVLgO2SKQEtYWURk8rjtoRfZ1d7HP3xsKdOqSnMdZ1IIc4iJ2cA9ZhYlU3B+6u4Pm9ntQLO7rySzK6gauC+o6m+6+4oQM4lIAfvFpr08uH43n1u2UGcQT6DQCoG7bwKWjjD/tiHTy8Jav4hMLi2H+vmrn73AOQ21/NnlC3IdZ1LRmcUikvfcnS/95wv0xlP83UeX6DrEE0zvpojkvf94biePv9zCl646gwUzq3MdZ9JRIRCRvPbink6++vBLXHzaDG68qDHXcSYlFQIRyVsPrt/Fh//5GWrKS7jjunOIRHSoaBh0YRoRyTvxZJqv/eIl7nl2B+84ZTrf/qNzqa8py3WsSUuFQETyygu7OvlfD21mw84O/vs7T+FLV51BTJ3DoVIhEJG8sP9QP3c8+goPPL+LGVWl/OMfncs1Z8/OdayiUDSFYPUbbfxmy34uPq2O80+ZTnXZxL70dNrp6k9ysDfO3o4+dnX0sbu9j86+BNVlMWrKY9SUlzCtsoTpVaXMqC5lRlUZtRUlWdnv6e60dg2wt7MfM4hFIpREjaqyGDNryo76iyuZSpNMO2WxyIin9Ls7ezr7WbejnQ1vdtAbT1JZGqOyNEpVWYyplSVMqyxlelUpUytLqC6LUVUWo7osRm88SXtPgvbeON0DScpiEcpLoocfW12WeR4NJTA5dfYl+NffbuPup98gmXJuftep/PnlC6gpL8l1tKJRNIVgy95D3PPMDr779DZiEeOceVOZP6OSkkiEaNSImpFIpRlIphlIpognnbQP3qAsFqEm+EIviUZo64nT0tXP/kMDtHUP0NmXIO2/v04zqC6L0TOQfMuyQdGIMb2qlLrqMipLo/TFU/QnUvQlUkTMKC+JUFEapaashDNm17B4Ti2L59ZSV11KbzwV3JJB7jQDiRRd/Un2dvaxp7OfPR197GrvY+fBXgaS6REzRAxOmlLO7NpyKkqjGJkv3GQ6zcGeOK1dA7T3Jg63L4tlMpVGI5TGMrfu/iQtXZmhospLIkwpLzmcbbTXfiwiBlVlseD/oITq8kyBGCwSVWUxykuilMYilAW3ytIYVWXRwwWpNBY5nDkaGXyVEItGqK0oYUp5TLsgsqh7IMn3f7eNu556g0P9Sa5aPIsvXXUG82dU5Tpa0TH3CfgrzaKmpiZvbm4+rsf2J1I0b2/nma0HeGZrG61dA6TSTjLtpNLpw19qZbEoJdEI0QhEzcDs8Bds90CSgWSKuuoyZtaUcdKUcmZUlzKtspSplaVMqyxhVm05DVMrmVVbTmksgrvTG09xqD9Be0+Cgz1x2noGONgT50D3AAe6Mv/2xlNUlkapKI1SURIl5c5AIk1fIsXBnjiv7OuiL5Ea9+udXlXK7NpyGqZVcPL0SuZNr2R2bQVG5ks+kfIjRaOjn72dfYeLhbsfLlL1NWXUVZdRGovQn8gUm75EingyTTwoQGWxCOfMm8p586dx+qyawyf8uDv9iTQdfXEO9sRp70nQ0RenZyB5+P2sLI0yvaqM6VUlVJXGiKfS9MZT9AWFrnsgQXd/kkNB++7+JF3BvJ54ip6B4P8lkSaeGrnYjVdNUGCOFJQo1WUxaitKqK0ooab8yJZMZVn08Os0MkW9przkcNuqsiixSIRY1IhFjOoyFRp35/k323lowx5WbtxDR2+CZWfO5HPLFrF4bm2u401qZrbO3ZtGXFZMhaDQpdLOtgPdbN59iM6+xOFfwhWl0cNfWplfwlFm11ZQURrNdeSsc3fiwZZd70CKnniS3oHMlkk8lT5cvFLB5949UxQ7exN09CXo6E0c3sKKJzNFOFOEEhzqy/zbGx9/MR6utiKza3BasKts8MfD9OrMVmF9dRn1NZnbjKrSoxaORCpNLGLHvNusZyDzGiuDz06Yu93cnc27D/GLF/by84172N3RR3lJhCvPPIlPXXoqS+ZNDW3dcsRYhaBodg1NBtGIsWBmDQtm1uQ6St4ys6AgRpkS0j7mVNrpjSfpGUiRTKcZ/C2VTDtd/Qk6+zK33oHU4a3NeMo51JfpBznYE6e9N87ezn627D1Ee29ixC09M5gR7DacUlHClPISplTE6OpPsqejj90dfXT0JohGjKrSaLCVktkNVhH0sVSURikvydxPO2w/0MMbB7rZf+jIiO+Dj6+vKWN2bQWzasuZNaWcuupS6msy/9YFhWlK+dH7tJKpNDsO9vLa/m6ef7OdRzbvZefBPmIR45IFdXz+vYv4g7fNmvB+Ojl++p8QOUaDu4AmsjOzN57M7CLsGaC1K3Nr6RqgtaufA91xDvUl2N3Rx5a9mYMP5kwtZ+nJU5lZU85AMkXPQIrugSS98eThvqMD3XH6EqnD/U4OzJ9RySUL6jitvpqq0ii9iRS9Aym6+hO0BAcT/Pa1A7R09Y/YtxOLGFMrSyiNRohFI8QiRiRiuDtO5qCJPR39h3fRDX75f/ryhbznrJOYWqlho/ORCoFIHqgsjXHyjBgnz6jMdRQgs9XT3nukD6utZ4AD3XHaujMHDiRS6cNHkqXdMwcYGETMeO/ichbOrGHhzGoWzKymSr/8857+h0TkLaIRo646c5AAs3KdRsJW3IcwiIiICoGISLEL85rF5Wa21sw2mtmLZvaVEdqUmdl/mNnrZrbGzBrDyiMiIiMLc4tgALjC3c8BlgDLzezCYW1uAtrdfQFwJ/CNEPOIiMgIQisEntEd3C0JbsMPSLsWuCeYvh+40jSgjIhIVoXaR2BmUTPbALQAj7n7mmFN5gI7Adw9CXQCM0Z4npvNrNnMmltbW8OMLCJSdEItBO6ecvclQANwgZktHtZkpF//bzmNxd3vcvcmd2+qr68PI6qISNHKylFD7t4BrAKWD1u0C5gHYGYxoBY4mI1MIiKSEdoJZWZWDyTcvcPMKoBlvLUzeCVwI/As8BHgcT/KKHjr1q07YGYdZHYjDVV7lHlHmx78tw44cNQX+FYjrX88y4fPH+v+8KxD5x1P7mxmHjqdi/e6ED8fx5J5pKxDl+f750Of6fA/0/NHfXZ3D+UGnA2sBzYBm4Hbgvm3AyuC6XLgPuB1YC1w6jif+65jnXe06SH/Nh/n633L+sezfPj8se4Pz3qiubOZOewmFBwAAAeXSURBVNfvdSF+Po4l8yhZh7bN68+HPtPZ+0yPdAtti8DdNwFLR5h/25DpfuC643j6nx/HvKNNj/T4E800nuXD5491f6SsJ5I7m5mHTufivS7Ez8exZB56X5/pY19eTJ/ptyi46xGEzcyafZQxu/NZIeZW5uwpxNzKnD0aYuKt7sp1gONUiLmVOXsKMbcyZ4m2CEREipy2CEREipwKgYhIkZvUhcDM/s3MWsxs83E89jwzeyEYGfUfho6BZGafNrNXglFVv5nvmc3sr81st5ltCG5XT2TmsHIPWf55M3Mzq5u4xKG91181s03B+/xrM5tTAJnvMLOXg9wPmtmEX00+pNzXBX+DaTObsA7aE8k6yvPdaGavBbcbh8wf83OfVcdzzGuh3IB3AecCm4/jsWuBi8gMg/EIcFUw/3Lgv4Cy4P7MAsj818DnC+29DpbNAx4FdgB1+Z4ZmDKkzWeAfymAzO8BYsH0N4BvFMLnAzgTOJ3MqAVNuc4a5GgcNm868Ebw77RgetpYrysXt0m9ReDuTzFsyAozO83MfmVm68zsaTM7Y/jjzGw2mT/oZz3zP/bvwAeCxX8KfN3dB4J1tBRA5tCFmPtO4IuMMAZVPmZ290NDmlZNdO6QMv/aM4M+AqwmMzbYhAop9xZ3fyVfso7ivWQG3Dzo7u3AY2SG5M/p3+twk7oQjOIu4NPufh7weeCfRmgzl8w4SIN2BfMAFgGXWuZCOk+a2fmhps040cwAfx5s+v+bmU0LL+rvOaHcZrYC2O3uG8MOOsQJv9dm9jUz2wn8MXAb4ZuIz8egT5L5dZoNE5k7bOPJOpLDIywHBvPny+sCiuzi9WZWDVwM3Ddkd1zZSE1HmDf4yy5GZhPvQuB84KdmdmpQ1SfcBGX+Z+Crwf2vAn9H5g8+NCea28wqgS+T2W2RFRP0XuPuXwa+bGa3An8O/O8JjnokyARlDp7ry0AS+OFEZhzJROYO21hZzewTwGeDeQuAX5pZHNjm7h9k9Pw5f11DFVUhILMF1OGZobEPM7MosC64u5LMF+fQzeMGYE8wvQv4z+CLf62ZpckMNBXWhRJOOLO77x/yuO8CD4eUdagTzX0acAqwMfjjawCeN7ML3H1fnmYe7kfALwixEDBBmYNOzPcBV4b1o2aYiX6vwzRiVgB3/x7wPQAzWwV83N23D2myC7hsyP0GMn0Ju8j96zoiV50T2boBjQzp9AGeAa4Lpg04Z5THPUfmV/9gR87Vwfz/AdweTC8is9lneZ559pA2fwH8pBDe62FttjPBncUhvdcLh7T5NHB/AWReDrwE1IfxuQj788EEdxYfb1ZG7yzeRmYvwrRgevp4P/fZuuVkpVl7cfBjYC+QIFOBbyLzK/NXwMbgw3/bKI9tIjNq6lbg2xw5C7sUuDdY9jyZ6zLne+YfAC+QGQl2JUMKQz7nHtZmOxN/1FAY7/UDwfxNZAb6mlsAmV8n84NmQ3Cb0COdQsz9weC5BoD9wKO5zMoIhSCY/8ngPX4d+MSxfO6zddMQEyIiRa4YjxoSEZEhVAhERIqcCoGISJFTIRARKXIqBCIiRU6FQAqemXVneX13m9nbJui5UpYZqXSzmf38aCN/mtlUM/ufE7FukUE6fFQKnpl1u3v1BD5fzI8MwhaqodnN7B7gVXf/2hjtG4GH3X1xNvJJcdAWgUxKZlZvZg+Y2XPB7ZJg/gVm9oyZrQ/+PT2Y/3Ezu8/Mfg782swuM7NVZna/Zcbq/+HgePHB/KZgujsYZG6jma02s5OC+acF958zs9vHudXyLEcG3Ks2s9+Y2fOWGbP+2qDN14HTgq2IO4K2XwjWs8nMvjKBb6MUCRUCmaz+HrjT3c8HPgzcHcx/GXiXuy8lMzLo3wx5zEXAje5+RXB/KfA54G3AqcAlI6ynCljt7ucATwGfGrL+vw/Wf9QxZIIxdq4kc+Y3QD/wQXc/l8w1MP4uKERfAra6+xJ3/4KZvQdYCFwALAHOM7N3HW19IkMV26BzUjyWAW8bMlrkFDOrAWqBe8xsIZnRHkuGPOYxdx86Dv1ad98FYGYbyIw/89th64lzZBC/dcAfBNMXcWR8+R8B/2eUnBVDnnsdmfHqITP+zN8EX+ppMlsKJ43w+PcEt/XB/WoyheGpUdYn8hYqBDJZRYCL3L1v6Ewz+3/AE+7+wWB/+6ohi3uGPcfAkOkUI/+9JPxIR9tobcbS5+5LzKyWTEH5M+AfyFzLoB44z90TZrYdKB/h8Qb8rbt/5xjXK3KYdg3JZPVrMtcCAMDMBocQrgV2B9MfD3H9q8nskgK4/miN3b2TzKUtP29mJWRytgRF4HJgftC0C6gZ8tBHgU8GY+ZjZnPNbOYEvQYpEioEMhlUmtmuIbdbyHypNgUdqC+RGT4c4JvA35rZ74BoiJk+B9xiZmuB2UDn0R7g7uvJjG55PZmLwzSZWTOZrYOXgzZtwO+Cw03vcPdfk9n19KyZvQDcz+8XCpGj0uGjIiEIrrDW5+5uZtcDH3P3a4/2OJFcUB+BSDjOA74dHOnTQciXBhU5EdoiEBEpcuojEBEpcioEIiJFToVARKTIqRCIiBQ5FQIRkSL3/wEh8PxpsadTRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lm_learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "Let's try backing away from the \"catastrophic forgetting cliff\" a bit more to see if we can continue improving the accuracy of our model's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.039267</td>\n",
       "      <td>3.405346</td>\n",
       "      <td>0.324714</td>\n",
       "      <td>15:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.997673</td>\n",
       "      <td>3.389007</td>\n",
       "      <td>0.328839</td>\n",
       "      <td>15:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.944736</td>\n",
       "      <td>3.419117</td>\n",
       "      <td>0.327929</td>\n",
       "      <td>15:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.887626</td>\n",
       "      <td>3.421754</td>\n",
       "      <td>0.328643</td>\n",
       "      <td>15:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 4e-3\n",
    "lm_learn.fit_one_cycle(4, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've improved accuracy and validation loss at the cost of a small amount of overfitting. Not shown, I tried continuing training a bit more, however the validation loss began to increase while the accuracy remained about the same. Let's save this model and move on to classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learn.save(lm_trial_prefix + 'fine_tuned_2')\n",
    "lm_learn.save_encoder(lm_trial_prefix + 'fine_tuned_enc_2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Classification Model <a name=\"3\"></a>\n",
    "<a name=\"0\">[(... back to the top)](#top)</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Classification Model Data\n",
    "Here, we create the data bunch using the training and test datasets we prepared in step 1. There are a couple of things worth noting here:\n",
    "* We must explicitly pass in the same language model vocabulary so that the classification model will correctly use the language model embeddings.\n",
    "* We create a validation set by randomly selecting 20% percent of the training set. So now our training set has 12527 samples, and our validation set has 1089 samples. This gives us roughly 3 digits of precision in our validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_data_fn = 'data_lm.pkl'\n",
    "train_fn = 'train.csv'\n",
    "test_fn = 'test.csv'\n",
    "clas_data_fn = 'data_clas.pkl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load datasets:\n",
    "\n",
    "Get the same vocabulary we used in the language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = load_data(path, lm_data_fn).vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load training and test datasets. Split validation from training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: 10724 valid: 2668 test: 1024\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text_len</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>tweet_createdLocalHour</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_nan</td>\n",
       "      <td>@SouthwestAir How to find out if my parents lu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>negative</td>\n",
       "      <td>xxtextlencount_&gt;130</td>\n",
       "      <td>xxretweetcount_0</td>\n",
       "      <td>xxtweet_createdLocalHour_7.0</td>\n",
       "      <td>@AmericanAir don't merge with an airline that ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  airline_sentiment             text_len     retweet_count  \\\n",
       "0          negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "1          negative  xxtextlencount_>130  xxretweetcount_0   \n",
       "\n",
       "         tweet_createdLocalHour  \\\n",
       "0  xxtweet_createdLocalHour_nan   \n",
       "1  xxtweet_createdLocalHour_7.0   \n",
       "\n",
       "                                                text  \n",
       "0  @SouthwestAir How to find out if my parents lu...  \n",
       "1  @AmericanAir don't merge with an airline that ...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv(path / 'train.csv')\n",
    "df_test  = pd.read_csv(path / 'test.csv')\n",
    "\n",
    "df_valid = df_train.loc[df_train.is_valid == 1, cols]\n",
    "df_train = df_train.loc[df_train.is_valid == 0, cols]\n",
    "\n",
    "print('train:', len(df_train), 'valid:', len(df_valid), 'test:', len(df_test))\n",
    "df_train.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the classification model data object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = TextClasDataBunch.from_df(path, df_train, df_valid, df_test, \n",
    "                                      vocab=vocab, label_cols=0, text_cols=model_input_cols)\n",
    "data_clas.save(clas_data_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_nan @united xxmaj hi have a question re future xxmaj flight xxmaj booking xxmaj problems . xxup dub - xxup jac 29 / 9 xxup jac - xxup lax 8 / 10 xxup lax - xxup dub 13 / 10 . i 'm * xxup xxunk xxmaj what is checked bag allowance for xxup jac - xxup lax ?</td>\n",
       "      <td>neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_22.0 @united xxup where xxup is xxup my xxup fucking xxup bag ? ! ? ! xxmaj where the fuck is my fucking bag ? ? ? xxup tell xxup me xxup now xxup or xxup give xxup me a xxup number xxup to xxup call a xxup human . xxup xxunk m</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_nan @southwestair xxmaj the xxmaj fact xxmaj that u xxmaj see xxmaj black xxmaj history xxmaj month 12 xxmaj months a xxmaj year xxmaj is xxmaj xxunk ! xxmaj we xxup will xxup be xxmaj an xxmaj xxunk xxmaj xxunk xxmaj for xxmaj xxunk xxmaj like u xxmaj in xxmaj future !</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_12.0 @united xxunk 7 xxup weeks xxmaj late flightr xxup and i xxup still xxup have xxup not xxup received xxup my xxup miles xxup from xxup the mileageplus xxmaj gift xxmaj card $ 150 xxup xxunk xxup card i xxup handed xxup over ! ! !</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_11.0 @americanair xxmaj aww xxmaj thanks xxup aa .. xxup dfw was on xxup xxunk up here this xxup am .. so i understand .. xxmaj btw xxup xxunk is my xxmaj airline when i m able to xxunk .. xxmaj love you guys . : )</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Classification Model Learner\n",
    "The goal of the classification model is to predict the class (sentiment = {*negative*, *neutral*, *positive*}) from a series of word embeddings which we've transferred from our language model.\n",
    "\n",
    "The primary difference between this model and the languange model is that we now predict a class rather than the next word. This can be seen by inspecting the final layer, which now has only 3 outputs (one per class) followed by a softmax activation. The model's prediction is the class with the largest value. Another important difference between the language model and the classification model is in the default levels of dropout in each section of the model. These defaults are tuned by the creators of fastai for generally performing well on classification tasks.\n",
    "\n",
    "As before, we have a variety of hyperparameters to set, and the justifications for each are the similar to the language model so I won't go through these again with the exception of `drop_mult`. I found that reducing the dropout multiplier from 1.0 to 0.7 both reduced the validation loss and increased the accuracy metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "arch = AWD_LSTM # AWD_LSTM, Transformer, or TransformerXL\n",
    "bs = {'AWD_LSTM': 512, 'Transformer': 32}[arch.__name__]\n",
    "drop_mult = 0.7\n",
    "lm_encoder_name = 'lm_AWD_LSTM_drop_mult_1.0_fine_tuned_enc_2'\n",
    "clas_data_name = 'data_clas.pkl'\n",
    "clas_trial_prefix = 'clas_' + arch.__name__ + '_drop_mult_' + str(drop_mult) + '_'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we reload the saved data and create the learner. Note how we intentionally set the language model encoder to the same encoder we just fine-tuned in the previous step. This is transfer learning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded classification model using data: data_clas.pkl \n",
      "Trial prefix: clas_AWD_LSTM_drop_mult_0.7_\n"
     ]
    }
   ],
   "source": [
    "data_clas = load_data(path, clas_data_name, bs=bs)\n",
    "learn = text_classifier_learner(data_clas, arch, drop_mult=drop_mult)\n",
    "learn.load_encoder(lm_encoder_name) # <-- transfer learning!\n",
    "print('Loaded classification model using data:', clas_data_name, '\\nTrial prefix:', clas_trial_prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did with the language model, we find a good learning rate and then we train our model. In this case, we'll first train with all but last layer group of weights frozen. Then we'll unfreeze weights one layer group at a time until we're training the whole model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3hc1b3u8e9v1K0uq7jIvRts3IGYZlogIQGHhB4M5IRD6OGSc0hIQi45aTckECCBGDAESOgxAQ6EbrqNbYwLxU1usrElW9XqZd0/ZmSEGdkCz549I72f55lHM7vMvKP2m7XX3muZcw4REZG9BfwOICIisUkFQkREwlKBEBGRsFQgREQkLBUIEREJK9HvAJGUn5/vhg4d6ncMEZG4sXTp0p3OuYJw63pUgRg6dChLlizxO4aISNwws01drdMhJhERCUsFQkREwlKBEBGRsFQgREQkLBUIEREJSwVCRETCUoEQEZGwVCBEROLYSx/u4K+vrceLqRtUIERE4tgzK7Zx/zubMLOIP7cKhIhIHNuws47hBemePLcKhIhInHLOUbKzjuH5KhAiItLJzt3N1Da2MkwFQkREOtuwsw6AYQUZnjy/CoSISJzasHM3gA4xiYjIZ5XsrCM5IcCAnDRPnt+zAmFm88yszMxWdbF+rJm9Y2ZNZnbtXus2mtlKM3vfzDTBg4hIGBvK6xjStw8Jgcif4gretiDuA07ax/oK4Ergpi7Wz3LOTXLOTYt0MBGRnqDEw1NcwcMC4Zx7nWAR6Gp9mXNuMdDiVQYRkZ6qrd2xaVcdw/K96aCG2O2DcMALZrbUzC7e14ZmdrGZLTGzJeXl5VGKJyLir62VDbS0Oc86qCF2C8RM59wU4GTgMjM7qqsNnXNznXPTnHPTCgrCzrstItLjlITOYBoWj4eYDoRzblvoaxkwH5jhbyIRkdiy5xqI3tSCMLN0M8vsuA+cCIQ9E0pEpLcqKa8jKzWRvunJnr1GoldPbGYPAccA+WZWCtwAJAE45+40s37AEiALaDezq4HxQD4wPzQyYSLwD+fcv73KKSISjzbsrGNYQYYno7h28KxAOOfO3s/67UBxmFU1wCGehBIR6SE27KxjxrA8T18j5g4xiYjIvjW2tLG1qsHT/gdQgRARiTsbd3nfQQ0qECIicaekXAVCRETCiMYprqACISISd0rK6+iXlUp6imfnGQEqECIicWfDzt2etx5ABUJEJO4Er4FQgRARkU4q65qprG/xdJC+DioQIiJxpCRKHdSgAiEiElc6zmAaXuDdPBAdVCBEROLI9uoGAPpnp3r+WioQIiJxpLqhhbSkBFKTEjx/LRUIEZE4UlXfQnZaUlReSwVCRCSOVDeoQIiISBjVDS1k94lOgfD2Om3Zo6q+mceWlPLY0i00trST0yeJ7LQkcvok0y8rhX7ZaQzITmVgbhqjizK7PL7Y0tYevLU6WtrbyUhJ/MLHIlva2lm9vZbNFfUMzuvDyMKMqBzPFJEDV93QwqC8PlF5LRUIDzU0t7FyazVPLC3lX8u30tjSzrQhuYzrn0Z1QwtV9S1srqjnxZpGGlva9+yXGDBGF2UysTibQXl92LyrnvXluynZWUdFXfNnXiMhtO0hxdlMLM4hLTlARV0LFXVNVNa30NbmCATAzGhrc3y8o5aPPqmhufXT1wsYDOmbzvD8dAoyU8hLT6ZvRgr5Gcn0z06jf3YqhVkppCSqiIj4rbqhhQlROsSkArGXRSW7+PE/V9La7ijKSqEwK5XCzBRy0pLJSkskKzWJPskJNLe109zaTnNbOy2t7bS0BT/Rt7Y5tlY2sLy0irVlu2lrd6QlJTB7cjHfPWwI4wdkfe41nXNU1bfwSXUjmyvqWLW1huWlVTy3ajvVDS3kZyQzvCCDrx5URP/sNJITAyQlBEhKMHbUNLKitJrnVm3n4cVb9jxnQsDI7ZNEYiBAu3O0u+DykYXpXPCVoRw8MJthfdPZUlnP6u21rC2rZcPOelZuraairpnWjh066ZueTFFWKv2yUynKSiE/I4WcPsnkpSeRk5ZMY0sbVaHCV1XfzK66Zirqgl9rGlpobW+nrc3R5hwpiQmMKEhnVFEmIwszOGhAFuP6ZREIeDd9okhPEM0+CBWITh5ZvJmfPrmK4tw+TBqUQ1ltIx9tq+G12iZ2N7V2+3ly+yQxoTiHE8YXMbE4hxnD8vb5AzUzctOTyU1PZvyALE46uD8QLBz1zW3dGrHROceWigZa29vpm55CZmpit/7ZTijO5msT+n/uuWoaWtlR28j26uBtW3UDO2qa2FETfLx8SxWV9c2EqSMAJCcG6JueTF7oNjivD0kBIxAwEsyoa25lfXkdb63ftac1k5+RwpGj8jlqdD7ThuRRnJvm6Xy7IvGmubWd+uY2FYhoam1r5zfPfcw9b27gyFH53H7OlM/9AFrb2tnd1EpNQyt1za0kJQRISQx85tN88GuAhAh9Cjazbg/na2YM7huZ45JmRnafJLL7JDG6KLPL7drbHTWNLVTUNVMVOjc7p0+wNZGaFOjWP/e2dsfminre21TJ62vLeW1NOfOXbQUgIyWRkYUZjCnKZOKgbKYNyWNUYYZaGdJrVTe0AJCjTuro2N3UymV/f4/X1pRzwVeG8tOvjyMx4fMndyUmBMjpk0xOn2QfUsamQMAO+HuSEDCG5aczLD+d06cW097u+GBbDSu2VrFmey1rduzmhQ+388iS4OGzrNREpg3NY+qQXKYOyeWQ4hzSktU3Ir1DdUOwDzJLLYjoSE4I0Nrezq9nT+CcQwf7HafXCwSMCcXZTCjO3rPMOcemXfUs2VTJko0VLN5YwSsflwHBDv2DBmZz+PC+zBzZl+lD83RGlvRYn7YgovNBVQUiMcADFx2qwxYxzMwYmp/O0Px0vj21GAgOefze5kqWbqpk8cYK7n6jhDtfW09yQoApQ3I4dFhfDh2Wx+TBuWphSI/RUSDUBxFFKg7xJzc9mePGFXHcuCIA6ppaeXdjBW+v28nb63dx6ytrcQ6SEoyDBmQzYWA2Bw/M4qAB2Yzpl0lSmMOIIrGuql4FQuQLS09JZNaYQmaNKQSgprGFpRsreXdjBUs3VTJ/2VYeWLgJgMzURL56UD++PrE/R4zMV7GQuLHnEJMKhMiXl5WaxKyxhcwaGywY7aGzpVZurea1NeU8/8F2Hl9aSk6fJM6eMZgrjx2lQ1ES8zpaEOqkFomgQODTfoxvHDKAX80+mDfW7GT+sq3csWA9z6zYxv+cNoGjRxf4HVWkS9UNLWSmJEbsVPr98axtbWbzzKzMzFZ1sX6smb1jZk1mdu1e604ys9Vmts7MrvMqo/ReKYkJHD++iD+fO4VHLj6MpIQAc+a9y5UPLWPX7ia/44mEVRPFgfrA29Fc7wNO2sf6CuBK4KbOC80sAfgzcDIwHjjbzMZ7lFGEQ4f35bmrjuTq40fx71Xb+cZtb7KytNrvWCKfUxXFYTbAwwLhnHudYBHoan2Zc24x0LLXqhnAOudciXOuGXgYONWrnCIQbFFcffxo/nnpVzAzvn3n2/zzvVK/Y4l8RnVDS9SuoobYnA9iILCl0+PS0LKwzOxiM1tiZkvKy8s9Dyc928EDs3nq8plMHpzDNY8u58anP6SlrX3/O4pEQTQH6oPYLBDhel+6GBIOnHNznXPTnHPTCgrUwSgHrm9GCg9871AunDmUeW9t4JRb32RRyS6/Y4lEdbpRiM0CUQoM6vS4GNjmUxbppZISAtzwjYOY+92p7G5q5cy5C/nhI+9TVtPodzTppYKjLLeQnRa98eBisUAsBkaZ2TAzSwbOAp7yOZP0Uice1I+XrjmaK44dyf+u+ITj/vAaTy/X5xWJvoaWNprb2qPagvDsOggzewg4Bsg3s1LgBiAJwDl3p5n1A5YAWUC7mV0NjHfO1ZjZ5cDzQAIwzzn3gVc5RfYnLTmB/3PiGL41pZhrH1vOFQ8t473Nlfz45HEkJ8biZyzpiaI9DhN4WCCcc2fvZ/12goePwq17FnjWi1wiX9aw/HQevvgwfvPsx8x7awPLt1Tx53On0D87ze9o0gtEey4IiM1DTCIxKykhwM+/MZ7bz5nM6u216sCWqIn2QH2gAiHypZwycQD/uvwIsvskce7di3gwNBCgiFf8OMSkAiHyJY0szODJy2Zy5Kh8fvrkKq6fv3LP/NoikVatFoRIfMlKTeLuOdO55OgR/H3RZs6ft4jaxr0HBxA5cHtaEOqDEIkfCQHjupPHcsuZk1iysZLz7l5EVX2z37Gkh6luaCEhYGSmRG8QbhUIkQg5bfJA7jxvKh99UstZcxeyU6PCSgRVNTSTlZqIWfRmwFSBEImg48cXcc8F09i4q44z/voO26t15bVERnVDKzl9oncVNahAiETckaMKuP+iQymraeLMuSoSEhnVDS1Rm0mugwqEiAdmDMvjge/NYGdtE+fercNNcuCq65ujegYTqECIeGby4FzuvXAG26oaOe/uRVTWqeNavrzqhhZyVCBEeo4Zw/K4e840SnbWcf68d/ecqijyRUV7NjlQgRDx3MyR+fz1vKl8vL2Gs+YupLSy3u9IEmfa2zuG+laBEOlxZo0t5J450ymtrOfU299i8cYuZ+MV+Zzdza20u+gO1AcqECJRc9ToAp68bCbZaUmcc9dCHlm82e9IEic6htnQWUwiPdiIggzmXzqTw4b35b+fWMnPnlxFU2ub37Ekxu0Z6lsFQqRny+6TxL0XTOfio4bzwMJNnPnXhWyravA7lsQwP4b6BhUIEV8kJgT4ydfGcce5U1hXtpuv3/oGb6wt9zuWxCg/BuoDFQgRX508oT9PXT6TwsxUzp/3Lk8u2+p3JIlBnx5i0lAbIr3K8IIM5l/2FWYMzeO/nljBss2VfkeSGFPVELzIUoeYRHqhPsmJ3HHeVIqyUvjPB5Zq/Cb5jOqGFpITA6QmRfdftgqESIzIS0/m7vOnU9fUysUPLKGxRWc3SVDHRXLRHOobVCBEYsqYfpncctZkVm6t5kePr8A553ckiQFV9dG/ihpUIERizgnji7j2xDE8vXwbN7+4xu84EgP8GKgPIHpz14lIt116zAg276rn1lfWUZSdyrmHDvE7kvioqr6F/tmpUX9dtSBEYpCZ8T+zD2bWmAJ+9uQqXvpwh9+RxEfVPgzUByoQIjErKSHA7edM4eCB2Vz+0Hs6/bUXq2loifpFcuBhgTCzeWZWZmarulhvZnarma0zsxVmNqXTujYzez90e8qrjCKxLj0lkXkXTKcoK5WL7lvM+1uq/I4kUdba1k5tU2uPa0HcB5y0j/UnA6NCt4uBOzqta3DOTQrdvuldRJHYl5+Rwv0XzSAzNYmz5y7k5Y90uKk3qWlsBaI/UB94WCCcc68D+xr0/lTgfhe0EMgxs/5e5RGJZ0P6pvPED77CyMIMvn//Eh56V0OF9xZ+jcME/vZBDAS2dHpcGloGkGpmS8xsoZmdtq8nMbOLQ9suKS/XYGfScxVkpvDwxYdx1OgCfvzPlfzxhdW6TqIXqKr3Z5gN8LdAhLsksOO3fbBzbhpwDnCLmY3o6kmcc3Odc9Occ9MKCgq8yCkSM9JTErnr/GmcMa2YW19Zx1UPv68rrnu4PS2IXlYgSoFBnR4XA9sAnHMdX0uABcDkaIcTiVVJCQF+d/pE/uukMTy1fBvn3LWQnbub/I4lHumYCyK3T3RHcgV/C8RTwPmhs5kOA6qdc5+YWa6ZpQCYWT4wE/jQx5wiMcfMuPSYkdxx7hQ+/KSG0/78Fmt31PodSzxQURc8xJSX3oMKhJk9BLwDjDGzUjP7npldYmaXhDZ5FigB1gF3AZeGlo8DlpjZcuBV4LfOORUIkTBOntCfRy4+nKbWdr57z7t75i6WnqOyvpmAQVZqDxpqwzl39n7WO+CyMMvfBiZ4lUukpzlkUA73zJnG7L+8zS+e/oCbz5zkdySJoIq6ZnL7JBMIRHckV9CV1CI9wsTiHK44diTzl23l2ZWf+B1HIqiyvpkcH05xBRUIkR7jslkjmViczfXzV1JWowmHeoqKumZf+h9ABUKkx0hKCPDHMyZR39zGdf9cqWskeojKuhZfzmACFQiRHmVkYQbXnTyWVz4u4+HFW/a/g8S8ivoYb0GY2YhOp54eY2ZXmlmOt9FE5MuYc/hQZo7sy41Pf8j68t1+x5ED4Jyjqr6Z3FguEMATQJuZjQTuAYYB//AslYh8aYGA8YfvTCI1KcCVDy2jqVVXWser3U2ttLQ58mL8EFO7c64VmA3c4pz7IaCB9URiVL/sVH53+kQ+2FbDTc+v9juOfEmVdaGrqGO8BdFiZmcDc4BnQsv8Oe9KRLrlxIP68d3DhnDXGxt4fY0GsoxHFfUdV1HH9mmuFwKHA79yzm0ws2HAg97FEpFIuP7r4xhVmME1jy7XeE1xqDI0zEZMn8XknPvQOXelc+4hM8sFMp1zv/U4m4gcoNSkBG47ZzK1jS2cNXchW6sa/I4kX0BFPBQIM1tgZllmlgcsB+41sz96G01EImFsvyzuu3AGO2oamf3nt/hgW7XfkaSbKkOHmGK9DyLbOVcDfAu41zk3FTjeu1giEkmHj+jLEz/4CgkB48y/LuSNteqTiAcVdc0kBIysVM+Gzdun7haIxNB0oGfwaSe1iMSR0UWZzL90JsW5aVx472Je/FBzW8e6yvrgQH1m0R+oD7pfIG4EngfWO+cWm9lwYK13sUTEC/2yU3n0ksMZPyCLKx9axspSHW6KZcFxmPw7YbS7ndSPOecmOud+EHpc4pw73dtoIuKFrNQk7p4zjbz0ZC7622J1XMewynr/xmGC7ndSF5vZfDMrM7MdZvaEmRV7HU5EvFGYmcq9F06nsbmN7923mNpGTTQUiyp9HMkVun+I6V6CU4QOAAYCT4eWiUicGl2UyR3nTWVd2W4u/ft7tLS1+x1J9lLp4zhM0P0CUeCcu9c51xq63QcUeJhLRKLgiFH5/Hr2BN5Yu5PrntAQ4bGkvd1RWd/i2zhM0P0CsdPMzjOzhNDtPGCXl8FEJDrOmD6IHx4/mifeK+X3GrcpZtQ2ttLW7nybTQ66Pyf1RcDtwM2AA94mOPyGiPQAVx43kh21jfxlwXoKM1O4YOYwvyP1ep+Ow+RfC6JbBcI5txn4ZudlZnY1cIsXoUQkusyMX556MDtrm/i/z3xIfmYKp0wc4HesXm3PMBtx0AcRzjURSyEivksIGLeePZmpg3O55tHlmmzIZx0D9cVDH0Q4/lzaJyKeSU1K4C/nTSE1McBPNK+1r2LhENOBFAj95oj0QIWZqfzka+NYtKGCx5aU+h2n16qM9UNMZlZrZjVhbrUEr4kQkR7ojGmDmDEsj189+xHltZpHwg+V9S0kJwRIT07wLcM+C4RzLtM5lxXmlumc82d4QRHxXCBg/Hr2BBqa2/jlMx/6HadXqqxrJjc9ybeB+uDADjGJSA82sjCDy2aN5Knl23h1dZnfcXqditBIrn7ytECY2bzQ+E2rulhvZnarma0zsxVmNqXTujlmtjZ0m+NlThEJ75JjhjOyMIOf/HOlDjVFWWVdDy8QwH3ASftYfzIwKnS7GLgDIDRz3Q3AocAM4IbQVKciEkUpiQnccuYkKuubueTBpTS1tvkdqdeoqPd3oD7wuEA4514HKvaxyanA/S5oIZATmpjoq8CLzrkK51wl8CL7LjQi4pGDB2bzh+9MYummSq6fv0qnvkZJRx+En/zugxgIbOn0uDS0rKvln2NmF5vZEjNbUl6uaRRFvPD1if256rhRPL60lLvf2OB3nB6vrd1R1eDvQH3gf4EI1z3v9rH88wudm+ucm+acm1ZQoAFmRbxy1XGj+NqEfvzmuY949WN1WnupuqEF5/y9BgL8LxClwKBOj4uBbftYLiI+CQSMm75zCOP6Z3Hlw8vYuLPO70g9Vsc4TD26D6IbngLOD53NdBhQ7Zz7hOD81yeaWW6oc/rE0DIR8VGf5ETuPG8qCQHjkgeX0tCsTmsvVIaG2ejRZzGZ2UPAO8AYMys1s++Z2SVmdklok2eBEmAdcBdwKYBzrgL4JbA4dLsxtExEfDYorw+3nDmJ1TtquX6+xmvyQmWMtCA8vRraOXf2ftY74LIu1s0D5nmRS0QOzDFjCrn6uNHc/NIaJg/J5buHDfE7Uo/S0YLwc7Ig8P8Qk4jEqSuOHcmsMQXc+PQHvLe50u84PUpFXQvgfwtCBUJEvpRAwLj5zEkUZaVyxT+WUd3Q4nekHqOyvpmUxABpSf4N1AcqECJyAHL6JHPb2ZPZUdOo/ogIqqgLXkXt50B9oAIhIgdo8uBcrjlxNM+s+ETzR0RILIzDBCoQIhIBlxw1gpkj+3LDUx+wrkxTlR6oWBiHCVQgRCQCAgHjj2dMIi05gSseWkZji66POBDBcZhUIESkhyjKSuWm70zko09quOv1Er/jxLWKumbyfD7FFVQgRCSCjh1bxPHjirjnrQ3sbmr1O05cam1rp6axVS0IEel5Lj92JFX1Lfx94Sa/o8SlHaGJmWKhk1rzSotIRE0alMORo/K5640Szj98KGnJ/p7LHw9a29p5Y91O5r+3lRc+3A7A4Lw+PqdSC0JEPHDFsaPYubuZhxdv9jtKzGtvd5x4y+tceO9iXltTzulTinn8ksOZNbbQ72hqQYhI5M0YlseMYXn89bUSzjl0MCmJakV0pbK+mZLyOv7jiGH86KQxMfW9UgtCRDxxxbEj2V7TyBNLt/odJaaVhfocpgzJjaniACoQIuKRI0bmc8igHP6yYB0tbe1+x4lZO2oaASjKSvE5yeepQIiIJ8yMK2aNpLSygSeXqRXRlY4WRGFmqs9JPk8FQkQ8c9y4Qg4akMXtr66jVa2IsMpCLYiCTLUgRKQXMTOuOm4Um3bV86/3Na18OGW1TWSnJZHq89De4ahAiIinThhfxPj+Wdz2ylq1IsLYUdMYk/0PoAIhIh4zM646fhQbd9Xz1PJ9tyKWbqrg6f1s09OU1TbFZP8DqECISBScOL6Icf2zuP2VrvsinHNc98RKrn7kfTbsrItyQv+U1TRRqBaEiPRWHX0RJTvreHpF+BbCsi1VrC3bTVu746YXVkc5oT+cc5TVNqoFISK924njixjbL5PbXl5HW/vnpyZ9dPEW0pISuHDmUP53xSesLK32IWV0Vda30NLm1AchIr1bIGBcffxoSnbW8diSLZ9ZV9fUytPLt3HKxP5cc8Jocvsk8bt/f+xT0ujpuEhOLQgR6fW+elAR04bkctMLaz4zX8T/rviEuuY2zpw+iMzUJC6bNZI31+3kzbU7fUzrvY6L5NSCEJFez8z46Snj2bm7iTsXrN+z/JElWxhekM7UIbkAnHfYEAbmpPG7f39Me5jDUT2FWhAiIp1MGpTDaZMGcNcbJWytamBdWS1LN1Vy1vRBmBkAqUkJ/PCE0azcWs2zqz7xObF3yjuG2eiNLQgzO8nMVpvZOjO7Lsz6IWb2spmtMLMFZlbcaV2bmb0fuj3lZU4Ria4fnTQWgN//+2MeWbyFxIDxrSnFn9lm9uSBjCnK5OYX14Tt1O4JdtQ0kpWaGJNXUYOHBcLMEoA/AycD44GzzWz8XpvdBNzvnJsI3Aj8ptO6BufcpNDtm17lFJHoG5iTxvePHM6T72/joXe3cNy4QvIzPvspOiFgXHncKNaX1/Hsyp7ZiiiraaIoKzYPL4G3LYgZwDrnXIlzrhl4GDh1r23GAy+H7r8aZr2I9FCXHDOC/IwUdje1cub0QWG3OfngfowqzOD2V9b1yL6IHbWNMXt4CbwtEAOBzueylYaWdbYcOD10fzaQaWZ9Q49TzWyJmS00s9O6ehEzuzi03ZLy8vJIZRcRj2WkJPLLUw/iuLGFHDWqIOw2gYBx+bEjWb2jds9czT1JWU0TRTHaQQ3eFggLs2zvjwDXAkeb2TLgaGAr0HHu22Dn3DTgHOAWMxsR7kWcc3Odc9Occ9MKCsL/kolIbDp5Qn/uuWA6iQld/ys6ZeIAhuen86eX1+Fcz2lFOOcor22ioJe2IEqBzu3GYuAz19g757Y5577lnJsMXB9aVt2xLvS1BFgATPYwq4jEqISAcdmskXz0SQ0vfVTmd5yIqapvobmtvde2IBYDo8xsmJklA2cBnzkbyczyzawjw4+BeaHluWaW0rENMBP40MOsIhLDTp00gMF5fbjtlbU9phWxozZ0DURvbEE451qBy4HngY+AR51zH5jZjWbWcVbSMcBqM1sDFAG/Ci0fBywxs+UEO69/65xTgRDppRITAlw2awQrSqtZsLpn9DWW1XRcRR27LYhEL5/cOfcs8Oxey37e6f7jwONh9nsbmOBlNhGJL7MnF3P7q+v4zXMfccSofJL20W8RDz69iroXtiBERCIpOTHAz74+njU7dvO3tzf6HeeAdYzDFKvDbIAKhIjEkRPGF3HMmAJueWktZaFP4PGqrKaRzNRE0pJj8ypqUIEQkThiZvziGwfR3NrOr5/9yO84B6SsNravogYVCBGJM0Pz0/nPo4PDdCwq2eV3nC9tR01jTPc/gAqEiMShS48ZycCcNH7+rw9o6WKO61i3I8bHYQIVCBGJQ2nJCfzslPGs3lHLI4u37H+HGNNxFXUsXwMBKhAiEqe+elARkwblMPf1krgbDrzjKupYPoMJVCBEJE6ZGZccPZzNFfU8F2eTCsX6VKMdVCBEJG6dML4fw/PT+etrJXE1BEesTzXaQQVCROJWQsD4/lHDWbm1mrfXx88ZTWpBiIhEwezJAynITOHO19b7HaXb1IIQEYmC1KQELpw5lDfW7mTV1mq/43RLeW1TzF9FDSoQItIDnHvoEDJSEpn7eonfUbolHi6SAxUIEekBstOSOOfQwTyzYhsl5bv9jrNf8TDMBqhAiEgP8R9HDCM9JZFrH1tOawxfXe2cY9Ouevpnp/kdZb9UIESkRyjMSuV/TjuY9zZXcceC2O2w3rSrnp27m5gyJMfvKPulAiEiPcapkwbyjUMG8KeX17KitMrvOGG9u6ECgEOH5fmcZP9UIESkR/mfUw+mIDOFqx95n4bmNr/jfM7CDbvom57MiIIMv6PslwqEiPQo2X2S+GKLaPoAAA1/SURBVMN3DqGkvC4m54x4d0MFM4blYWZ+R9kvFQgR6XG+MjKf/zhiGA8s3MRP5q+ksSU2WhJbqxoorWxgRhwcXgJI9DuAiIgXrjt5LEmJAe5YsJ73NlVy+zlTGFno72GdxaH+BxUIEREfJSYE+O+TxnLosDyueXQ537jtTS45egSJCUZtYys1jS1MGpTDGdMGRS3Tog27yExNZGy/rKi95oFQgRCRHu2YMYU8d9WRXPXwMm5+aQ0ASQlGalIC/1i0mabWdr572JCoZFm0oYIZQ/NICMR+/wOoQIhIL1CUlcpD3z+Mirpm0lMSSUkM0NbuuOTBpfz8X6soyEjmpIP7e5qhvLaJkvI6zoxii+VAqZNaRHoFM6NvRgqpSQmYGYkJAW47ewqTBuVw5cPv77k+wSuLN8ZX/wOoQIhIL5aWnMA9c6ZTnJPGf/xtMWt21Hr2WotKdtEnOYGDB2Z79hqRpgIhIr1aXnoyf7toBilJCcyZ9y5bqxo8eZ1FGyqYOiSXpIT4+bfraVIzO8nMVpvZOjO7Lsz6IWb2spmtMLMFZlbcad0cM1sbus3xMqeI9G6D8vrwtwtnsLuple/es4hdu5si+vxV9c2s3lHLjKHxc3gJPCwQZpYA/Bk4GRgPnG1m4/fa7CbgfufcROBG4DehffOAG4BDgRnADWaW61VWEZHxA7K4Z850tlY2cOF9i9nd1Bqx516ysRLn4qv/AbxtQcwA1jnnSpxzzcDDwKl7bTMeeDl0/9VO678KvOicq3DOVQIvAid5mFVEhBnD8vjLuVP4YFsNF9+/hKbWA78C2znHgjVlJCcGOGRQ7I/g2pmXBWIgsKXT49LQss6WA6eH7s8GMs2sbzf3BcDMLjazJWa2pLy8PCLBRaT3Om5cEb//9kTeXr+LHzz43gEN07FqazXn3r2IBxdu5tgxhaQmxfYUo3vzskCEuxLE7fX4WuBoM1sGHA1sBVq7uW9woXNznXPTnHPTCgoKDiSviAgA35pSzK9nT+DV1WV872+LqW/+YoebymubuOaR9znltjf56JMafvGN8dx69mSP0nrHywvlSoHOV4QUA9s6b+Cc2wZ8C8DMMoDTnXPVZlYKHLPXvgs8zCoi8hnnHDqY1KQA1z62nPPveZd5F04nKzVpv/s557j6kWUs3ljJJUeP4NJZI7q1XyzysgWxGBhlZsPMLBk4C3iq8wZmlm9mHRl+DMwL3X8eONHMckOd0yeGlomIRM23phRz+zlTeH9LFefetYjq+pb97vPyR2W8tW4XPzl5LNedPDZuiwN4WCCcc63A5QT/sX8EPOqc+8DMbjSzb4Y2OwZYbWZrgCLgV6F9K4BfEiwyi4EbQ8tERKLqaxP6M/f8qXy8vYYfz1+Bc2GPdgPQ0tbOr5/9iBEF6ZwbpfGdvOTpWEzOuWeBZ/da9vNO9x8HHu9i33l82qIQEfHNsWOLuOaEMfzu3x/zr/e3cdrksOfM8ODCTZTsrGPeBdPi6oK4rsT/OxARiYKLjxrO1CG5/Oxfq/ik+vNXW1fVN3PLS2s5YmQ+s8YU+pAw8lQgRES6ISFg/PGMQ2hrd/zosRW0t3/2UNOfXl5LbWMLPz1lXFxMJ9odKhAiIt00pG861399HG+u28mDizZR19TKe5sruf+djTzwzibOnD44biYD6g7NByEi8gWcM2MwL3ywg1889QE3PPUBHX3Wg/LSuOaE0f6GizAVCBGRL8DM+P23J/KHF9YwICeNcf0zGdc/i+LctB5zaKmDCoSIyBdUmJXK77490e8YnlMfhIiIhKUCISIiYalAiIhIWCoQIiISlgqEiIiEpQIhIiJhqUCIiEhYKhAiIhKW7Wts83hjZuXApr0WZwPV+1nW+XG4+x1f84GdXzJeuBzdWR/J/PDl38P+8u9rm33l3fvx/u4r/xffZn+/Q129n0jm31e+/a3X37C3+Yc458LP1+yc69E3YO7+lnV+HO5+p69LIpmjO+sjmf9A3sP+8n+R9/BF80fiZ6D8XS/r6v1EMn933kM0/gaU/4vt0xsOMT3djWVP7+d+uOeIRI7urI+X/PvaZl95937cnftfhvJ3vayr9xPJ/N15jnj/G4j3/J/Tow4xec3Mljjnpvmd40DE+3tQfn8pv7+inb83tCAiaa7fASIg3t+D8vtL+f0V1fxqQYiISFhqQYiISFgqECIiElavLRBmNs/Mysxs1ZfYd6qZrTSzdWZ2q3WaRsrMrjCz1Wb2gZn9v8im/kyGiOc3s1+Y2VYzez90+1rkk38mhyc/g9D6a83MmVl+5BJ/LoMXP4NfmtmK0Pf/BTMbEPnkezJ4kf/3ZvZx6D3MN7OcyCffk8GL/N8J/e22m5knncEHkruL55tjZmtDtzmdlu/zb6Rbvuw5tfF+A44CpgCrvsS+7wKHAwY8B5wcWj4LeAlICT0ujLP8vwCujeefQWjdIOB5ghdN5sdTfiCr0zZXAnfGWf4TgcTQ/d8Bv4uz/OOAMcACYFos5Q5lGrrXsjygJPQ1N3Q/d1/v8Yvcem0Lwjn3OlDReZmZjTCzf5vZUjN7w8zG7r2fmfUn+Ef8jgv+FO4HTgut/gHwW+dcU+g1yuIsf1R5+B5uBv4L8PQMDC/yO+dqOm2ajofvwaP8LzjnWkObLgSK4yz/R8651V5lPpDcXfgq8KJzrsI5Vwm8CJwUqb/zXlsgujAXuMI5NxW4FvhLmG0GAqWdHpeGlgGMBo40s0Vm9pqZTfc07ecdaH6Ay0OHB+aZWa53Ubt0QO/BzL4JbHXOLfc6aBcO+GdgZr8ysy3AucDPPcwaTiR+hzpcRPCTazRFMn80dSd3OAOBLZ0ed7yXiLzHxC+6Q09lZhnAV4DHOh2qSwm3aZhlHZ/yEgk28w4DpgOPmtnwUAX3VITy3wH8MvT4l8AfCP6RR8WBvgcz6wNcT/AwR9RF6GeAc+564Hoz+zFwOXBDhKOGFan8oee6HmgF/h7JjPsSyfzRtK/cZnYhcFVo2UjgWTNrBjY452bT9XuJyHtUgfhUAKhyzk3qvNDMEoCloYdPEfwn2rnZXAxsC90vBf4ZKgjvmlk7wcG1yr0MHnLA+Z1zOzrtdxfwjJeBwzjQ9zACGAYsD/2hFQPvmdkM59x2j7NDZH6HOvsH8L9EqUAQofyhjtJTgOOi8eGok0h//6MlbG4A59y9wL0AZrYAuMA5t7HTJqXAMZ0eFxPsqyglEu/Ri06YeLkBQ+nUUQS8DXwndN+AQ7rYbzHBVkJH58/XQssvAW4M3R9NsOlncZS/f6dtfgg8HG8/g7222YiHndQe/QxGddrmCuDxOMt/EvAhUOD1746Xvz942En9ZXPTdSf1BoJHLnJD9/O68x67lTMaP8RYvAEPAZ8ALQSr7fcIfvr8N7A89Ev+8y72nQasAtYDt/PpFenJwIOhde8Bx8ZZ/geAlcAKgp+0+nuV36v3sNc2G/H2LCYvfgZPhJavIDi42sA4y7+O4Aej90M3L8/C8iL/7NBzNQE7gOdjJTdhCkRo+UWh7/s64MIv8jeyv5uG2hARkbB0FpOIiISlAiEiImGpQIiISFgqECIiEpYKhIiIhKUCIT2ame2O8uvdbWbjI/RcbRYc1XWVmT29v5FRzSzHzC6NxGuLgGaUkx7OzHY75zIi+HyJ7tPB6DzVObuZ/Q1Y45z71T62Hwo845w7OBr5pOdTC0J6HTMrMLMnzGxx6DYztHyGmb1tZstCX8eEll9gZo+Z2dPAC2Z2jJktMLPHLTj3wd87xtoPLZ8Wur87NPDecjNbaGZFoeUjQo8Xm9mN3WzlvMOnAxJmmNnLZvaeBcf7PzW0zW+BEaFWx+9D2/4o9DorzOz/RvDbKL2ACoT0Rn8CbnbOTQdOB+4OLf8YOMo5N5ngKKq/7rTP4cAc59yxoceTgauB8cBwYGaY10kHFjrnDgFeB77f6fX/FHr9/Y6PExpL6DiCV7cDNAKznXNTCM5B8odQgboOWO+cm+Sc+5GZnQiMAmYAk4CpZnbU/l5PpIMG65Pe6HhgfKeRM7PMLBPIBv5mZqMIjnyZ1GmfF51zncfwf9c5VwpgZu8THFvnzb1ep5lPBzxcCpwQun84n47N/w/gpi5ypnV67qUEx/qH4Ng6vw79s28n2LIoCrP/iaHbstDjDIIF4/UuXk/kM1QgpDcKAIc75xo6LzSz24BXnXOzQ8fzF3RaXbfXczR1ut9G+L+lFvdpJ19X2+xLg3NukpllEyw0lwG3EpwnogCY6pxrMbONQGqY/Q34jXPur1/wdUUAHWKS3ukFgvMsAGBmHcMsZwNbQ/cv8PD1FxI8tAVw1v42ds5VE5x+9FozSyKYsyxUHGYBQ0Kb1gKZnXZ9HrgoNN8AZjbQzAoj9B6kF1CBkJ6uj5mVdrpdQ/Cf7bRQx+2HBIdpB/h/wG/M7C0gwcNMVwPXmNm7QH+gen87OOeWERzp8yyCk/BMM7MlBFsTH4e22QW8FTot9vfOuRcIHsJ6x8xWAo/z2QIisk86zVUkykIz3zU455yZnQWc7Zw7dX/7iUSb+iBEom8qcHvozKMqojitq8gXoRaEiIiEpT4IEREJSwVCRETCUoEQEZGwVCBERCQsFQgREQnr/wPy+UOiS8MY6QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_start=10, skip_end=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there is currently an issue with dataloaders in PyTorch, on which fastai is based, which causes them to run very slowly on Windows - this is why my training times are very slow. This is typically not an issue on a Linux machine. For reference, [here](https://github.com/pytorch/pytorch/issues/12831) is the issue on GitHub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.871165</td>\n",
       "      <td>2.121521</td>\n",
       "      <td>0.206897</td>\n",
       "      <td>05:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.750053</td>\n",
       "      <td>0.546243</td>\n",
       "      <td>0.787856</td>\n",
       "      <td>06:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 0.1\n",
    "learn.fit_one_cycle(2, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(clas_trial_prefix + 'first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load(clas_trial_prefix + 'first');\n",
    "learn.freeze_to(-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhU1fnA8e+bPSQhCwlbAiRAQJCdgCxKtW5oW3EXV9y1datWrf5srXWpVm2tVhQ3qEtdcUOlolWwspMgayASEgJhzwTIRvbz+2MmOIQJE5K5yZ3M+3meeZi5c+7Me5jMvPfcc885YoxBKaWUOpqg9g5AKaWU/WmyUEop5ZUmC6WUUl5pslBKKeWVJgullFJehbR3AL6SmJhoUlNT2zsMpZTyK1lZWUXGmCRv5TpMskhNTSUzM7O9w1BKKb8iIgXNKaenoZRSSnmlyUIppZRXmiyUUkp5pclCKaWUV5oslFJKeaXJQimllFeaLJRSSnmlyUIppfzYh1mFvLdiq+Xvo8lCKaX82BtLC/jkhx2Wv48mC6WU8lM1dfVs2FnCkOTOlr+XJgullPJTm/eWUV1bz5DkWMvfS5OFUkr5qXXbSwA4vqcmC6WUUk1Yt/0AUWHB9E2Msvy9NFkopZSfWrf9AIN7diYoSCx/L00WSinlh+rqDdk7S9rkFBRoslBKKb+UX1RORXVdm3RugyYLpZTyS+t3HABok8tmQZOFUkr5pbWFBwgPCaJ/UnSbvJ8mC6WU8kPrdhxgUI/OhAS3zc+4JgullPIz9fWG9dvbZuR2A00WSinlZ7btq6C0qpYhbXQlFGiyUEopv9MwcrutroQCTRZKKeV31m4/QGiwkN6tbTq3QZOFUkr5nfU7DjCwewzhIcFt9p6aLJRSyo8YY1i3/UCb9leAJgullPIrOw5Usq+ihuPbsL8CNFkopZRfWbfdNXK7Z9tdNgsWJwsRmSwiOSKSKyL3eXj+GRFZ5br9KCL73Z6rc3tujpVxKqWUv1i3/QDBQcKgHm2bLEKsemERCQamA6cDhcAKEZljjMluKGOMudOt/G3ASLeXOGiMGWFVfEop5Y9WbdtPetdoIkLbrnMbrG1ZjAVyjTF5xphq4F1gylHKXwq8Y2E8Sinl1ypr6lieX8z4fl3a/L2tTBbJwDa3x4WubUcQkT5AGvCt2+YIEckUkaUicm4T+93oKpO5d+9eX8WtlFK2lFWwj6raek5KT2zz97YyWXhausk0UXYqMNsYU+e2rbcxJgO4DPiHiPQ74sWMedkYk2GMyUhKSmp9xEopZWMLc4sICRJOSOtYLYtCoJfb4xRgRxNlp9LoFJQxZofr3zxgAYf3ZyilVMBZuKmIUb3jiQq3rLu5SVYmixVAuoikiUgYzoRwxFVNIjIQiAeWuG2LF5Fw1/1EYCKQ3XhfpZQKFPvKq1m34wAntsMpKLDwaihjTK2I3ArMA4KBmcaY9SLyMJBpjGlIHJcC7xpj3E9RDQJeEpF6nAntCferqJRSKtAs3uzAGJjYv4MlCwBjzFxgbqNtDzZ6/JCH/RYDQ62MTSml/MnC3L3EhIcwPKVtR2430BHcSinlBxbmFjGuX5c2WxmvMU0WSillcwWOcrYVH2yXS2YbaLJQSimbW5hbBLRffwVoslBKKdtbuKmInrER9E2MarcYNFkopZSN1dUbFm92cGJ6IiKexjq3DU0WSillY+u2H+DAwZp2PQUFmiyUUsrW7NBfAZoslFLK1n7cXUqvhEgSo8PbNQ5NFkopZWNllbXERoa2dxiaLJRSys5Kq2qJboeJAxvTZKGUUjZWVqnJQimllBdl2rJQSinlTVlVLdERmiyUUkodhfM0lHZwK6WUakJVbR3VdfXEaMtCKaVUU8oqawG0z0IppVTTyqo0WSillPKitKFloaehlFJKNaWhZRGjLQullFJNKdOWhVJKKW+0z0IppZRXpVXaslBKKeVFw2moGB2Up5RSqillVTUEBwkRoe3/U93+ESillPKoYcbZ9lx7u4EmC6WUsim7rGUBmiyUUsq2yiprbTEvFGiyUEop27LLWhagyUIppWzLLmtZgCYLpZSyLbssqQqaLJRSyrZKq7TPQimllBfaslBKKXVUtXX1HKyps8WSqqDJQimlbKm8qg6wx7xQYHGyEJHJIpIjIrkicp+H558RkVWu248ist/tuWkissl1m2ZlnEopZTelVTWAPdayALAsChEJBqYDpwOFwAoRmWOMyW4oY4y50638bcBI1/0E4E9ABmCALNe++6yKVyml7CSQWhZjgVxjTJ4xphp4F5hylPKXAu+47p8JfG2MKXYliK+ByRbGqpRStlLmalkEQgd3MrDN7XGha9sRRKQPkAZ8eyz7isiNIpIpIpl79+71SdBKKWUHdlp/G6xNFp6mSTRNlJ0KzDbG1B3LvsaYl40xGcaYjKSkpBaGqZRS9mOn9bfB2mRRCPRye5wC7Gii7FR+OgV1rPsqpVSHY6f1t8HaZLECSBeRNBEJw5kQ5jQuJCIDgXhgidvmecAZIhIvIvHAGa5tSikVEOy0/jZYeDWUMaZWRG7F+SMfDMw0xqwXkYeBTGNMQ+K4FHjXGGPc9i0WkUdwJhyAh40xxVbFqpRSdtPQZxEV1sGTBYAxZi4wt9G2Bxs9fqiJfWcCMy0LTimlbKxhevKgoPZfJQ90BLdSStmSneaFAk0WSillS3ZaywI0WSillC3Zaf1t0GShlFK2VFZZY5u1LECThVJK2ZKd1t8GTRZKKWVL2sGtlFLKq1Lt4FZKKXU0xhjKqmptMy8UaLJQSinbqaiuwxj7zAsFmiyUUsp2fpoXyh7rb4MmC6WUsh27rWUBmiyUUsp27LaWBWiyUEop27HbWhagyUIppWzHbutvgyYLpZSynUN9FposlFJKNeVQn4WehlJKKdWUhj6LKG1ZKKWUakpZVS0RoUGEBtvnJ9o+kSillALst5YFaLJQSinbsduMs9DMZCEi/UQk3HX/ZBG5XUTirA1NKaUCk92WVIXmtyw+BOpEpD/wGpAGvG1ZVEopFcD8tmUB1BtjaoHzgH8YY+4EelgXllJKBS5nn4V9JhGE5ieLGhG5FJgGfO7aZq+aKKVUB1FWZa/1t6H5yeIaYDzwmDEmX0TSgLesC0sppQKXHU9DNSsaY0w2cDuAiMQDMcaYJ6wMTCmlAlHDKnl+2cEtIgtEpLOIJACrgVki8ndrQ1NKqcBTVVtPTZ2xXcuiuaehYo0xJcD5wCxjzGjgNOvCUkqpwGTHeaGg+ckiRER6ABfzUwe3UkopHyuz4Yyz0Pxk8TAwD9hsjFkhIn2BTdaFpZRSgemn9bftlSya28H9AfCB2+M84AKrglJKqUBlx/W3ofkd3Cki8rGI7BGR3SLyoYikWB2cUkoFmp/W37bXULbmnoaaBcwBegLJwGeubUoppXzo0JKq/tiyAJKMMbOMMbWu27+AJAvjUkqpgOTvHdxFInKFiAS7blcADm87ichkEckRkVwRua+JMheLSLaIrBeRt92214nIKtdtTjPjVEopv1Zq00tnmxvNtcDzwDOAARbjnAKkSSISDEwHTgcKgRUiMsc1GryhTDpwPzDRGLNPRLq6vcRBY8yIZtdEKaU6gLLKWkKChPAQey031KxojDFbjTHnGGOSjDFdjTHn4hygdzRjgVxjTJ4xphp4F5jSqMwNwHRjzD7X++w5xviVUqpD2VfhnERQRNo7lMO0JnXd5eX5ZGCb2+NC1zZ3A4ABIrJIRJaKyGS35yJEJNO1/VxPbyAiN7rKZO7du/eYK6CUUnbz4+5S0rvGtHcYR2hNsvCW9jw9bxo9DgHSgZOBS4FX3Vbg622MyQAuA/4hIv2OeDFjXjbGZBhjMpKStL9dKeXf6usNG3aWMLhn5/YO5QitSRaNf/gbKwR6uT1OAXZ4KPOpMabGGJMP5OBMHhhjdrj+zQMWACNbEatSStleQXEFFdV1DO7hZ8lCREpFpMTDrRTnmIujWQGki0iaiIQBU3GO1XD3CXCK670ScZ6WyhOReLc1vxOBiUA2SinVgWXvKAGwZcviqFdDGWNafOLMGFMrIrfinFMqGJhpjFkvIg8DmcaYOa7nzhCRbKAOuMcY4xCRCcBLIlKPM6E94X4VlVJKdUTZOw8QEiT07xrd3qEcwdILeY0xc4G5jbY96Hbf4Owov6tRmcXAUCtjU0opu8neUUL/rtFEhAa3dyhHsNeFvEopFcCyd5bYsr8CNFkopZQtFJVVsbukypb9FaDJQimlbGHDTlfntrYslFJKNaXhSqhBmiyUUko1JXtnCT1jI4iPCmvvUDzSZKGUUjaQvcOeI7cbaLJQPmOM4f0V2/hxd2l7h6KUX6msqWPz3jLb9leAxeMsVGDJ2V3KvR+uISw4iDtPH8CNk/oSHGSvmTOVsqOcXaXUG3uO3G6gLQvlMws3FQEwoX8X/vrlRi6csZjNe8vaOSql7C/70JVQse0cSdM0WSifWZRbRN/EKGZdPYZnp44gb285Zz/7PQtydJkSpY4me0cJMeEhpMRHtncoTdJkoXyiuraeZfnFTOyfiIgwZUQyX905ibTEKO54dxXbiivaO0SlbCt7ZwmDenQmyManbTVZKJ9YtW0/FdV1TOyfeGhbt84RzLhiNPX1hlveXklVbV07RqiUPdl5DQt3miyUTyzKLSJIYHzfLodtT02M4umLh7Om8ACPfK4TByu1att+ps/PJWeX86pBO69h4U6vhlI+sSi3iKEpccR2Cj3iuTOP785Nk/ry0v/yyOiTwLkjG6+uq1THV1NXz3PfbGL6/FzqDTw1L4chyZ3pn+ScjlxbFqrDK62s4Ydt+5nYr0uTZe45cyBj0xK4/6O1rC08YEkce0ur+PeyAmrr6i15/fZUX2949fs8NukYFr+Uu6eM819YzD+/zeX8USl8d8/J/OlXgwH4ZNUOwkKCbLmGhTttWahWW55fTF294US3/orGQoKDeP7SkZz3wmIue3Upb1w7lpG9430ax4zvNvPawnwW5zr4x9QRhAZ3nGOhH7bt59EvNvD3r3/k7xcPZ/KQHu0dkmqGLUXlvLW0gDeXFtApLJgZV4w69NldMzGNayamkbOrlIrqWluuYeGu43ybVLtZlOsgPCSIUX2O/uPftXME7900jvhOYVz52nIytxT7NI6leQ7iOoXyxdqd3PLvlVTXdpwWxgrX/1XfpChufmslT8/Loa7etHNUyhNjDF9n7+aqmcs5+ekF/GvxFs48vjvz7pzkMckP7B7j8wMnK2jLQrXaotwixqYlNOvIKCW+E+/fNJ7LXlnKVTOXM/PqMYzr2/Tpq+Y6UFFD9s4SfnvqAGIjQ3jos2xufiuLFy4fZfsjtuZYkV9M36QoPvz1BB78ZD3Pz89lzfYDDE+JpaisGkdZFTV19dx5+gCGpcS1d7gB7dEvNvDawny6d47grtMHMHVML7p2jmjvsFpNWxaqVfaUVpKzu/SwS2a96R4bwbs3jqNnXCRXz1rO6m37Wx3HsnwHxsC4vglcPTGNR88dwrcb93Dd6yvYW1rV6tdvT/X1hhVbihmbmkB4SDBPXDCUR84dwtI8B9Pn5/J19i4KHBWs3X6AS19eypLNjma9bllVrcWRB573VmzltYX5TBvfh4W/P4XbT03vEIkCNFmoVlqc6/xhmtiv+ckCnKek3r1xHF2iwrn93R8oraxpVRxL84oJDwliRG/nUfUV4/rw1IXDWLFlH6c/8x2frtqOc8l3/5Ozu5SSylrGpCYAICJcOa4Pax86g02PnU3mH05n3p2T+Py2k+gZF8m0Wcv5Onv3UV/z38sKGPbQPGYuzG+LKgSE5fnF/OGTdUwakMQffzmYkA7UZwaaLFQrLcotIq5TaIsu+0uMDucfU0ewrbiCBz9df8Tz1bX1ZBXsa9aP/JI8B6P7xBMe8tMpp4syejH39hPp08U5ivzXb61sVSvjw6xCXv7f5jZPOg39FWPTEg7bHh4SfNhEjd1jI3j/pvEM6tGZm9/K4qOVhR5fb3dJJY/P3UinsBAe/jybRz7Ppl77P1plW3EFN7+VRa/4Tvzz0pEdLlGAJgvVCsYYFuUWMaFflxbPLjsmNYHbT03n4x+2H/bjVuAo58IZi7ngxcU8/VXOUV9jf0U1G3eVHDEgEKB/1xg+vHk89511HN/m7OFnT83n3tmrydxSfEw/+rV19Tw2dwN/mbuRJ77c2KYJY3l+Md07RzRr3qD4qDD+ff0JnJCWwF3vr+bDrCMTxsOfZVNTV8+cWydy9YRUXluYz23v/EBlTdMj7Asc5by/Ypt2qntQXlXLDW9kUltXz6vTMoiNPHKsUUegHdyqxYrKqtlxoJLr+iR4L3wUt57Sn8W5Dv74yTpG9Y5ndeF+Hvh4HUECpwxMYvr8zXSJCufaE9M87r8sv9jZX9HEOI+Q4CBu/lk/ThvUjZf/t5kv1uzk/cxC+iZGceOkvkwd29trjCu27KO4vJphKbG89F0e4cFB3HXGwFbVuzmMcfVXpHVBpHkJOTo8hJlXj+H61zO598M1xEaGctrgbgDM37iHL9bu5O4zBtA3KZo//WowyXGRPDZ3A3tKK3npygwSGq3UtqWonItfWsKe0iq+yt7Ns1NHEBWuPx0N5q3fxcZdpcy8OoO+SfYeK9Ea2rJQLVbgKAegb2JUq14nJDiIZ6aOICQ4iPNeWMQd765iYPcY5t5xEq9clcGZx3fj4c+z+XTVdo/7L9nsICI0iGEpR5/euX/XaJ68cDjLHziNpy4cRkxkKPd9tJa5a3d6jXHe+l2EhwTx9g3juCSjF899m8vz325qUX2PxdbiCnaXVB1xCsqbiNBgZlw5miE9O3PL2ytZlufgYHUdf/x0Hf27RnPjpH6As//jhkl9ef6ykawuPMCU6QsPTUMBsH3/QS5/dRk1dfXcekp/vt24mwtnLGH7/oOHymzeW8bjczfw0Jz1lLSy78kf5ReVExwknJSe1N6hWEqThWqxLQ7nTLKprUwWAMlxkfz1gmFUVNdx6yn9ee/GcaTEdyIkOIhnp47khLQEfvf+ao/TnS/Nc5DRJ+Gw/oqjiQoP4aKMXnxw03hG9Irj3tlryC8qb7K8MYZ563cxaUAS0eEh/OX8oZw/Mpmnv/qRV7/Pa3Gdm2N5vqu/IvXYW2/R4SHMumYsvRI6cf3rmdw9ezWF+w7y2LlDCAs5/Kv/y2E9ef+m8VTW1HP+C4v4ZsNu9pRUcvkrSymprOHN607g7jMHMuuasRQWVzDl+UW8tjCfi2cs4dS/fcdrC/N5c2kBv/rnQtbvsGaEvl3lF5WTEh/ZoQaBetKxa+cHsgr28adP11Hjh1NUFDicR1TJcb6Zg3/ykO6s//OZ3H3mwMM6CCNCg3llWgbp3WL49Vsr2eBaKAaguLyajbtKGdf32H9Mw0KCmH75KEKChV+/ldXkOfs1hQfYeaCSycd3ByA4SHjywmGcPbQ7j83dwH+9XHnUGiu2FBMbGUp6C6eCSIgK483rxtI5MpQv1uzk4owUTmhiXMuIXnHMuXUiaUlRXP9GJuc8v4g9pVX865qxDEl2ttp+NiCJj34zgU5hwTzyeTZ7Siv5/eTjWHz/z3nvxnFU1tRx/guLeX/FthbX2d8UOCro06X1B0x2p8miHZVW1nDb2yt5fUlBk1eu2Fl+UTnJcZFHHKW2RlNXkXSOCOX1a8YQExHCb/698tCltsvznZfutnRgX3JcJM9cMoKNu0p58NN1Hst8uX4XIUHCqYO6Hhbn3y4awZCesdzx7g9HrDteU1fPq9/nsSi3qEVxNVixZR9jUuNbtc5Bj9hI3rxuLNPG9+H/zh7ktewHN03g7KE9OHCwhtemjWF0o5H56d1i+OL2E/ni9hOZf/fJ/PrkfnSNiSAjNYEvbj+JjNR47v1wDVe+toyn5+Xw0cpCVm3bz8HqjjdFvTGGLY5y0rp0au9QLKfJoh399cuN7CyppHdCJ577JtfvpqcocFT45BRUc3XtHME/Lx3J1uIK7p29BmMMSzY7iAwNbtWo5VMGduW2n/fn/cxC3s88/IjYGMOX63Yxvl8X4jod3vEbGRbMK1dl0Ck8hOtfz6S4vBqATbtLOe+FRTz6xQaumbWCxS1MGHtKK8kvKj80vqI1+iZF8+cpQ46ogyeRYcFMv2wUPzx4OuObuGggJiKU43vGHtHpnhgdzhvXnsCdpw2gcN9BXvxuM3e9v5pzpy9i/BPf8Or3eUe96qpBUVkV2/cftP3YmOLyakorawOiZaGXNLSTJZsdvLV0K9efmMZJA5KYNnM572Vu48pxfdo7tGZpOKIa2bttp5Y4oW8X7j1zII//ZyMzF21haV4xGanxrW7d/Pa0AWQV7OOPn6xjcI/Oh067bNpTRn5ROdc1cSVW99gIXr5yNJe8vJRfv5XF6YO78eS8HKLDQ/j7xcN56bs8bngjk7dvGMfwXsf2f5W5ZR8AY46xc9tXWjpNSnCQcMdp6dxxWjrVtfVsLa4gd08Zby/fyqNfbGDWoi3cdfoAzh2ZfNgl18YYVm7dz6xF+Xy5bhe19YaY8BAGdI9hYPcYxvftwmmDuhEZZp/pW37qt+v4LQtNFu3gYHUd9320hj5dOvG7MwYSERpERp94pn+by0WjU/xiLqP2PKK6cVJfsgr28fjcDdTWG84Z0bPVrxkcJDx36Uh+9c+F3PRmFp/ddiIJUWF8uW4XInCG69JTT0b2jufJC4bx2/dWsSy/mNMGdePx84eSFBPOxP6JXDhjMdNmLeeDm8aT3i2m2TEtzy8mMjSYIT2PfpWXnTVMvd2/azSTh3RnUW4RT/xnI7/7YDUPfbaeXvGdSI6PJDkukpVb97Gm8AAxESFcMzGV3l2i+HFXKTm7Svls9Q7eXraVqLBgzhzSnXNHJDOubxefngJtiYYrAlO1ZaFaq77esGlPGV2iw+gSFYaI8PevcyhwVPDODeMOHSXddcYALntlGW8v29rkeAI7aTiiSmuHIyoR4amLhnPO8wspcFT4ZCJCcJ5CmXHFaC6asYTb3/mBf10zhnnrdzGqd7zX+X3OHZlMdW094aFBnDO856HTM906R/DWdSdw4YwlXPHaMmbfPIFeCc37P1ueX8zI3nHt/oPoSxP7J/LpLROZt34XS/IcFO47yFZHBYtzi+gRF8kjU47n/FEpR4zjqKs3LMtz8Mmq7fxn7S4+Wrmd0GAhvWsMg3t2ZnCPzmSkxnN8z9gWDxBtiS1F5QSJc4LMjk6ThcW+yt7FzW+tBKBTWDC9Ezrx4+5SLjuh92Hngyf0S2R83y68sGAzl47tbaumticNR1Ttda42NjKUV67KYHZWIcO9jK84FsN7xfHIucfz+w/XcvcHq1m/o4QHvHQKN7h4TC+P2/t0ieLN68Zy8YwlXPf6Cj76zUSivQxq21JUzoZdJdz+8/RjroPdBQUJZw3twVlDm78mR3CQMKF/IhP6J/LwlCF89+Nefti6n+ydJSzI2cNs10j1uE6hTOjXhYn9E/nF0B7N6qNpjS2OCpLjfXuRh11psrDY1mLnEfj9Zx3HrpJKthVX0DMukvvPOu6IsnedMYCLZizhjSVbuOln/do40mPTcETVqx2PqAZ0i/F6dU9LXDKmN6sLD/D2sq2Ac1nY1jque2deuHw0V81cxu/eX8WLl49u8gqn+nrDvR+uITo8hEubMbo80ESEBnPm8d0P+1x2l1SyNM/B95uKWLipiLlrd/HX/2zkN6f05+oJqZad2i1wlAfEKSjQZGE5R1k1YSFB3Dipr9fpGsakJnBSeiIvLNjM6YO72XrqgI5+RPWnXw0md08ZAvT20WWRJ6Yn8n9nD+LRLzbw/Pxcbj/Vc6vhzaUFLM8v5skLh9E9tmNMb221bp0jmDIimSkjkjHGsH5HCX/7Kocn/rOR1xdv4c7TB3DBqBSfn6La4qjgnOGt7zPzB5Z+00VksojkiEiuiNzXRJmLRSRbRNaLyNtu26eJyCbXbZqVcVrJUV59qK+iOR6eMoSQIOHK15azw21KBbvp6EdU4SHBvHfjON687gSfvu51J6Zx3shk/v71jx6nEd/qqOCvX25k0oAkLhqd4tP3DhQiwpDkWGZdM5Z3bhhH15hw7p29hnOeX+jT1Rn3lVdz4GANfQJgjAVYmCxEJBiYDpwFDAYuFZHBjcqkA/cDE40xxwO/dW1PAP4EnACMBf4kIvZfd9ADR1kVXaKbf940LTGK168dS8nBGq54bRmOMvst3GOMIb+ovMN/SUTE5y0nEeHx84cyLCWWO99bxXc/7j00k6vz9NNqgkR44vyhzT7AUE0b368Ln9wykWenjsBRVs2FM5Zw53ur2FNS2erX3hJAV0KBtS2LsUCuMSbPGFMNvAtMaVTmBmC6MWYfgDGmYeKfM4GvjTHFrue+BiZbGKtlisurSYgKP6Z9hiTHMvOaMezYf5Bps5bbbnK2/RU1lFTWBsyXxNciQoOZccVoosNDmDZzOeMe/4aH5qznqa9yWJpXzB9+MYiePppCRTkT9JQRyXzzu59xyyn9+GLNTk55egFvL9vaqkF/BT6cG80fWJkskgH34bCFrm3uBgADRGSRiCwVkcnHsC8icqOIZIpI5t69e30Yuu8UlVWTGHXsV2SMSU3gxctHs3FnKTe9kWWrxWkC7YjKCj3jIpl/98lMv2wUo3vH8/byrby4YDMnpSdySRNXVanWiQoP4Z4zj+OrOycxoncc//fxWm5+K4t9rpH3xyq/qBwR6JUQGIndymThqQ3d+BcvBEgHTgYuBV4Vkbhm7osx5mVjTIYxJiMpyZ7TAzvKj+00lLtTjuvKo+cOYUmeg9k2mjvqULIIgFGrVooMC+YXw3ow48rRZP3hNF68fBTPTR2pp58slpoYxZvXnsADZw/i2417OOvZ71s0JUuBo5yesZHNnu3Y31mZLAoB90OkFGCHhzKfGmNqjDH5QA7O5NGcfW2vorqWypr6Yz4N5e7ijF6M6h3Hk1/mtHqdal/ZUlThOqLSZOErMRGhnDW0B/EtaIWqYxcU5FzH4+PfTKRTWDCXv7aM6fNzj+m01BZHBWkBcgoKrE0WK4B0EUkTkTBgKjCnUZlPgFMARCQR52DskjoAABLMSURBVGmpPGAecIaIxLs6ts9wbfMrjjJn87alLQtw/lE/dM7xOMqr+Oe3ub4KrVUC7YhKdVxDkmP5/PYT+dWwnjw1L4fffbCaqtrmzY67xdHxL/JwZ1myMMbUArfi/JHfALxvjFkvIg+LyDmuYvMAh4hkA/OBe4wxDmNMMfAIzoSzAnjYtc2vOFznQru08mhxWEocF41OYebCfDbvLfNFaK2S76jQU1Cqw+gUFsKzU0dw52kD+Gjldq541ftViPsrqtlfURNQ/XaWjrMwxsw1xgwwxvQzxjzm2vagMWaO674xxtxljBlsjBlqjHnXbd+Zxpj+rtssK+O0SsMfXJfolp+GanDPmccRGepccKa9dfQxFirwiDhnyn3+spGsKTzAuS8sIndPaZPlG66E0paF8glftSwAkmLCuf3UdBbk7OXbjdatzOZNIB5RqcDxy2E9ee+m8Rysruf8FxazeLPnju+Gizy0z0L5hC/6LNxNm5BK36Qo/vjJep8MKmqJQDyiUoFlRK84Pv7NBLp2jmDazOV8/MORVyIG4kUemiws5CirIjI0mE5hvpmCKywkiL9fPIJ9FdVc/uqyQyuztaVAPKJSgadXQic+/PUEMvokcOd7q3num02HXSlV4CinR+cIv1h7xlc0WVjIOXrbt5dCjugVx6vTMthaXMFVM5e1+ejuQDyiUoEpNjKU168dy/muubz++Om6Q4NjtzjKA2bkdgNNFhYqKq8m0UenoNxN6JfIjCtGk7OrlGtnraCiurbZ+76zfCsPzVnP7KxCftxdemheouYKxCMqFbjCQoL428XDuWlSX95aupW73l9FTV09WxwVAbHutjudotxCxeVVJPngSihPTjmuK89OHcmtb6/kpjezmHX1GEKCj577V27dx/99vJYgkUNJIjI0mDtOS+fmZq6fkR+AR1QqsIkI9511HJ0jQ3lqXg6O8mqKy6tJDbB+O21ZWMhRVu2Ty2abcvbQHjx+/lC+31TE4//ZeNSylTV13Dt7DT1jI1n5x9P5+s5JPHPJcDJS43l6Xg65e7yP3zDGsKWoPOCOqJQSEW45pT+PTDme7zc5r5AKtO+BJguLGGNcycLa6RsuGdObqyek8trCfD7Manr+qH9+u4ncPWX85fyhxEaGkt4thvNGpvDMJSOIDA3m0S+8j99YuXU/+ypqGNkrzpdVUMpvXDk+lWcuGU5KfCTDe/luOV9/oMnCImVVtVTX1ftkjIU3D/xiEOP7duH+j9eyatv+I55ft/0AM77L46LRKfxswOETLiZG/zR+Y37OniP2dTc7axuRocGcPaz5aycr1dGcNzKFhb//OT1iA2O22QaaLCxyaIxFKyYRbK7Q4CCmXz6KrjHh3PxmFntKfxqDUV1bz90frKZLVBh/+MVgj/tPm5BKWmIUj36eTU1dvccyB6vr+Hz1Ts4a2p3ocO3qUirQ6LfeIg2jtxMsPg3VICEqjJevzOCCFxdz8lML6NY5gi5RYdQZw8ZdpbxyVQaxnUI97hsWEsQDZw/i+jcyeXNJAdeemHZEmXnrd1FaVctFo3WtBaUCkbYsLNIwL1RiG7QsGgzu2Zk3rhvLxRm9GJIcS1hIEBVVdfzm5H6cPrjbUfc9dVBXTkpP5B///dHjYL8PsrbRKyGSE9ISrApfKWVj2rKwSHEbtywajElNYEzqsf+giwh//OVgznr2e/782Xr+ccmIQ4vwFO6rYPFmB3ecmk5QkC7Mo1Qg0paFRXw5iWBbGdAtht+ems6nq3bw4nebD23/MGs7xsAFo1LaMTqlVHvSloVFisqqiA4P8buRzrf+vD+5e8t48ssc+iZGccbg7sxeuY0J/broFB9KBTBtWVjEinmh2oKI8NcLhjGqdxy/fW8VMxfls634IBdlaKtCqUCmycIibTEgzyoRocG8dGUGXaLCefSLDcSEhzD5eB1boVQg02RhEUd5tV/1VzSWFBPOa1dnEB0ewgWjU4gM86/TaUop39I+C4s4yqoYluzf0wEc170zi+77OVGaKJQKeJosLGCMobjcf09DuYuN9DyQTykVWPQ0lAVKDtZSW2/8soNbKaU80WRhgaJy1+htC6cnV0qptqTJAtjqqKCsqvmrzXlzaPS2tiyUUh1EwCeL/KJyfvb0fD7I3Oaz12yYF6oj9FkopRRosiAtMYqRveL41+Itx7wedVMapvrQ01BKqY4i4JMFwLUnplHgqODbjUdf/AecrYab3swkc0vxUco4k0V8J21ZKKU6Bk0WwOTju9MzNoKZC/O9lv1y/S7mrd/NFa8t49uNuz2WcZRV0TkihLAQ/e9VSnUM+msGhAQHcdWEVJbkOcjeUXLUsotzHXSNCSe9aww3vJHlcd1rR3k1XfQUlFKqA9Fk4TJ1TC8iQ4OZtajp1kV9vWHx5iJOSk/inRvHcUJaAr/7YDWvfp93WDlHmX9P9aGUUo1psnCJ6xTGBaOT+XTVDopcVzM1lr2zhH0VNZyY3oXo8BBmXTOGs4d259EvNvDpqu2HyvnrjLNKKdUUTRZurp6QRnVdPf9eutXj84tyiwCY0C8RgPCQYJ6bOpKRveN48NP17CmpBMBRXqWnoZRSHYomCzf9u0Zz8sAk3lxaQFVt3RHPL8wtIr1rNN06RxzaFhIcxNMXDaeypo77P1pLXb1zXqhEHWOhlOpANFk0cu3ENIrKqvh01Y7DtlfV1rFiSzET+ycesU+/pGjuOXMg32zcw8yF+dQbHb2tlOpYNFk0clJ6IoN6dGbGd5sPG6S3smA/lTX1HpMFOJPM2NQEnvhyI4CehlJKdSiWJgsRmSwiOSKSKyL3eXj+ahHZKyKrXLfr3Z6rc9s+x8o4G8XELaf0I29vOV+u23Vo++LNRQQJnNA3weN+QUHCUxcNIyzY+V+qV0MppToSy5KFiAQD04GzgMHApSIy2EPR94wxI1y3V922H3Tbfo5VcXpy1pAe9E2MYvr8XIxxti4W5hYxvFccnSOaXt+hT5coHvjFIIIEeid0aqtwlVLKcla2LMYCucaYPGNMNfAuMMXC9/OZ4CDh5pP7kb2zhAU5eymprGFN4QEm9vN8CsrdFeP68MODZ9BLk4VSqgOxMlkkA+5TuRa6tjV2gYisEZHZItLLbXuEiGSKyFIROdfTG4jIja4ymXv37vVh6HDeyGSS4yJ5fn4uSzc7qKs3TfZXNKaryymlOhork4V42NZ4WtfPgFRjzDDgv8Drbs/1NsZkAJcB/xCRfke8mDEvG2MyjDEZSUlJvoobgNDgIG76WV+yCvbx/PxcIkKDGNUnzqfvoZRS/sLKZFEIuLcUUoDDrkc1xjiMMQ3DpV8BRrs9t8P1bx6wABhpYaweXZzRi8TocNYUHmBMagLhIcFtHYJSStmClcliBZAuImkiEgZMBQ67qklEerg9PAfY4NoeLyLhrvuJwEQg28JYPYoIDeb6k9IAmn0KSimlOqIQq17YGFMrIrcC84BgYKYxZr2IPAxkGmPmALeLyDlALVAMXO3afRDwkojU40xoTxhj2jxZAFw5rg97S6s4f5Sn7hallAoM0nBpqL/LyMgwmZmZ7R2GUkr5FRHJcvUPH5WO4FZKKeWVJgullFJeabJQSinllSYLpZRSXmmyUEop5ZUmC6WUUl5pslBKKeWVJgullFJedZhBeSKyFyjw8FQscMDLNvfHnu67b0sEiloQoqc4mlvGF3Vwv9/SOhwtxuaUOVrM3h43/izsUgdP2+zyWRzt+ZZ+Fnb+e/K0Tb/b3vUxxnifidUY06FvwMvetrk/9nS/0bZMX8XR3DK+qEOj+rSoDr6ux7E8bvxZ2KUOdv4sjvZ8Sz8LO/89teSz0O9282+BcBrqs2Zs+8zLfU+v4Ys4mlvGF3Vobgze+LIex/JYP4vmxdLc51v6Wdj578nTNv1u+0iHOQ3VVkQk0zRjHhU70zrYR0eoR0eoA3SMelhZh0BoWfjay+0dgA9oHeyjI9SjI9QBOkY9LKuDtiyUUkp5pS0LpZRSXmmyUEop5VXAJgsRmSkie0RkXQv2HS0ia0UkV0SeExFxe+42EckRkfUi8qRvo/YYi8/rISIPich2EVnlup3t+8gPi8OSz8L1/N0iYlzL81rKos/iERFZ4/ocvhKRnr6P/LA4rKjDUyKy0VWPj0UkzveRHxaHFXW4yPWdrhcRyzrBWxN7E683TUQ2uW7T3LYf9XvjkVXX5Nr9BkwCRgHrWrDvcmA8IMB/gLNc208B/guEux539dN6PATc7c+fheu5XjiX9S0AEv2xHkBntzK3AzP8sA5nACGu+38F/uqHdRgEDAQWABl2i90VV2qjbQlAnuvfeNf9+KPV82i3gG1ZGGP+h3Pd70NEpJ+IfCkiWSLyvYgc13g/EemB8wu8xDj/198AznU9/Wuc64VXud5jj7W1sKwebcrCOjwD3Au0yVUcVtTDGFPiVjQKi+tiUR2+MsbUuoouBVL8sA4bjDE5VsbdmtibcCbwtTGm2BizD/gamNzS737AJosmvAzcZowZDdwNvOChTDJQ6Pa40LUNYABwkogsE5HvRGSMpdE2rbX1ALjVddpgpojEWxdqk1pVBxE5B9hujFltdaBetPqzEJHHRGQbcDnwoIWxNsUXf08NrsV5JNvWfFmHttac2D1JBra5PW6oT4vqGdLMN+3wRCQamAB84Hb6LtxTUQ/bGo72QnA298YBY4D3RaSvK3u3CR/V40XgEdfjR4C/4fySt4nW1kFEOgEP4Dz90W589FlgjHkAeEBE7gduBf7k41Cb5Ks6uF7rAaAW+LcvY/TGl3Voa0eLXUSuAe5wbesPzBWRaiDfGHMeTdenRfXUZPGTIGC/MWaE+0YRCQayXA/n4PwhdW9GpwA7XPcLgY9cyWG5iNTjnNhrr5WBN9Lqehhjdrvt9wrwuZUBe9DaOvQD0oDVri9YCrBSRMYaY3ZZHLs7X/xNuXsb+II2TBb4qA6uztVfAqe25cGTi68/h7bkMXYAY8wsYBaAiCwArjbGbHErUgic7PY4BWffRiEtqadVHTX+cANScetIAhYDF7nuCzC8if1W4Gw9NHQOne3afjPwsOv+AJxNQPHDevRwK3Mn8K6/1aFRmS20QQe3RZ9FuluZ24DZfliHyUA2kNQWn4GVf09Y3MHd0thpuoM7H+fZjnjX/YTm1NNjXG314dntBrwD7ARqcGba63AejX4JrHb9cT/YxL4ZwDpgM/A8P42EDwPecj23Evi5n9bjTWAtsAbnEVcPf6tDozJbaJuroaz4LD50bV+Dc7K4ZD+sQy7OA6dVrpvVV3RZUYfzXK9VBewG5tkpdjwkC9f2a13//7nANcfyvWl80+k+lFJKeaVXQymllPJKk4VSSimvNFkopZTySpOFUkoprzRZKKWU8kqTherQRKSsjd/vVREZ7KPXqhPnbLPrROQzb7O1ikiciPzGF++tVGN66azq0ESkzBgT7cPXCzE/TYpnKffYReR14EdjzGNHKZ8KfG6MGdIW8anAoi0LFXBEJElEPhSRFa7bRNf2sSKyWER+cP070LX9ahH5QEQ+A74SkZNFZIGIzBbnOg3/blgPwLU9w3W/zDUJ4GoRWSoi3Vzb+7kerxCRh5vZ+lnCT5MkRovINyKyUpxrEkxxlXkC6OdqjTzlKnuP633WiMifffjfqAKMJgsViJ4FnjHGjAEuAF51bd8ITDLGjMQ5u+tf3PYZD0wzxvzc9Xgk8FtgMNAXmOjhfaKApcaY4cD/gBvc3v9Z1/t7nZPHNYfRqThH0wNUAucZY0bhXEPlb65kdR+w2Rgzwhhzj4icAaQDY4ERwGgRmeTt/ZTyRCcSVIHoNGCw2yyenUUkBogFXheRdJyzcIa67fO1McZ9nYHlxphCABFZhXM+n4WN3qeanyZhzAJOd90fz0/rB7wNPN1EnJFur52Fcz0CcM7n8xfXD389zhZHNw/7n+G6/eB6HI0zefyvifdTqkmaLFQgCgLGG2MOum8UkX8C840x57nO/y9we7q80WtUud2vw/N3qcb81CnYVJmjOWiMGSEisTiTzi3AczjXtUgCRhtjakRkCxDhYX8BHjfGvHSM76vUEfQ0lApEX+FcFwIAEWmY/jkW2O66f7WF778U5+kvgKneChtjDuBcUvVuEQnFGeceV6I4BejjKloKxLjtOg+41rUmAiKSLCJdfVQHFWA0WaiOrpOIFLrd7sL5w5vh6vTNxjm1PMCTwOMisggItjCm3wJ3ichyoAdwwNsOxpgfcM46OhXn4kEZIpKJs5Wx0VXGASxyXWr7lDHmK5ynuZaIyFpgNocnE6WaTS+dVaqNuVbyO2iMMSIyFbjUGDPF235KtSfts1Cq7Y0GnnddwbSfNlyyVqmW0paFUkopr7TPQimllFeaLJRSSnmlyUIppZRXmiyUUkp5pclCKaWUV/8PmvR5MrXejIoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we're training multiple layer groups, we can use a technique fastai calls \"differential learning rates\". This means that layer groups learn at different rates. Training seems to improve when earlier layer groups update slower than later groups. Over the past couple of years, fastai has used a variety of multipliers for reducing the learning rate (I've noticed 2x, 3x, 10x, and 45x). This is another hyperparameter which could be tuned, but I will stick with the latest default of 2x according to fastai's text model tutorial [here](https://docs.fast.ai/text.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.607337</td>\n",
       "      <td>0.460494</td>\n",
       "      <td>0.821214</td>\n",
       "      <td>07:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.580150</td>\n",
       "      <td>0.462130</td>\n",
       "      <td>0.820090</td>\n",
       "      <td>06:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.560132</td>\n",
       "      <td>0.441579</td>\n",
       "      <td>0.826837</td>\n",
       "      <td>06:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.531817</td>\n",
       "      <td>0.431517</td>\n",
       "      <td>0.835082</td>\n",
       "      <td>06:17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 1e-2\n",
    "lr_reduction_factor = 2.\n",
    "learn.fit_one_cycle(4, slice(lr/lr_reduction_factor, lr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(clas_trial_prefix + 'second')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load(clas_trial_prefix + 'second');\n",
    "learn.freeze_to(-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEGCAYAAABGnrPVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3xUVdrA8d+ThNBbSKihhd5bQMGGDVFXwLogq+JaVn1R17aLr6674uuuq7uLDXVRsa0KiqtiRSwgIi1A6BBCQgktk4SWAKnP+8fc6BBTJslMZiZ5vp/PfJh7bpnnZIZ55tx77jmiqhhjjDG+EhboAIwxxtQulliMMcb4lCUWY4wxPmWJxRhjjE9ZYjHGGONTEYEOoCZER0drly5dAh2GMcaElNWrV2eoakxl96sTiaVLly4kJCQEOgxjjAkpIrKrKvvZqTBjjDE+ZYnFGGOMT1liMcYY41OWWIwxxviUJRZjjDE+ZYnFGGOMT1liMcYY41OWWIwxphZKcWXzz6+2kX70ZI2/tiUWY4yphb7b5uK5b5MpKKr5ObcssRhjTC20PCWTzq0a0b5Fwxp/bUssxhhTyxQWKStSMjm9a6uAvL4lFmOMqWW27D/K0ZMFjOxmicUYY4wPLNuRCWCJxRhjjG8sT8kkLroxbZo1CMjrW2IxxphapKCwiJWpWZwWF5jWClhiMcaYWmXTvqMcyw3c9RXwc2IRkbEisk1EkkVkWhnbXCMim0Vkk4i845QNFpFlTtl6Efm1x/avi0iqiCQ6j8H+rIMxxoSSZSnu6yunx0UFLAa/zSApIuHATOBCIA1YJSLzVXWzxzY9gAeBM1T1kIi0dlYdB65X1e0i0h5YLSILVPWws/4BVZ3nr9iNMSZULU/JpHvrJrRuGpjrK+DfFssIIFlVU1Q1D5gDjC+xzS3ATFU9BKCq6c6/Saq63Xm+D0gHKj3vsjHG1CX5hUWsSs0KaGsF/JtYOgB7PJbTnDJPPYGeIrJURJaLyNiSBxGREUAksMOj+HHnFNkMEalf2ouLyK0ikiAiCS6Xq3o1McaYELBh7xFy8goZGRcd0Dj8mViklLKSg9ZEAD2A0cAk4BURafHTAUTaAW8BN6pqkVP8INAbGA5EAX8s7cVVdZaqxqtqfEyMNXaMMbVf8f0rtbnFkgZ09FiOBfaVss3HqpqvqqnANtyJBhFpBnwGPKyqy4t3UNX96pYLvIb7lJsxxtR5y1My6dWmKa2alHoip8b4M7GsAnqISFcRiQQmAvNLbPMRcC6AiETjPjWW4mz/IfCmqr7vuYPTikFEBJgAbPRjHYwxJiTkFRSRsPNQwFsr4MdeYapaICJTgQVAODBbVTeJyHQgQVXnO+vGiMhmoBB3b69MEfkNcDbQSkSmOIecoqqJwNsiEoP7VFsicJu/6mCMMaFifdphTuQXBvT+lWJ+SywAqvo58HmJskc8nitwr/Pw3OY/wH/KOOZ5vo/UGGNC25LtGQCcFqARjT3ZnffGGBPivtp0gBcWJXNG91a0bBwZ6HAssRhjTCj7YsN+7nh7DX3bN+eFycMCHQ5gicUYY0LW/HX7mPruWgZ1bMFbN42gecN6gQ4J8PM1FmOMMf7x0dq93PteIvFdopg9ZThN6gfP13nwRGKMMcZr0z/dzKCOLXj9xuE0igyur3I7FWaMMSHmZH4hWTl5nN+7ddAlFbDEYowxISczJw+A6ADfYV8WSyzGGBNiXMdyAYhpaonFGGOMDxQnFmuxGGOM8YmMbGuxGGOM8aHiFkurJoG/y740lliMMSbEZGTn0rxhPepHhAc6lFJZYjHGmBDjOpYbtKfBwBKLMcaEHNexXKKD9DQYWGIxxpiQk5GdS0zTBoEOo0yWWIwxJsS4juUSE6RdjcESizHGhJTjeQXk5BUS3dROhRljjPGBjGPu4VysxWKMMcYnXNknAYiuq73CRGSsiGwTkWQRmVbGNteIyGYR2SQi73iU3yAi253HDR7lw0Rkg3PMZ0VE/FkHY4wJJq4QaLH4bbxlEQkHZgIXAmnAKhGZr6qbPbbpATwInKGqh0SktVMeBfwZiAcUWO3sewh4EbgVWA58DowFvvBXPYwxJpi4gnw4F/Bvi2UEkKyqKaqaB8wBxpfY5hZgppMwUNV0p/wiYKGqZjnrFgJjRaQd0ExVl6mqAm8CE/xYB2OMCSoZx3IRgajGdfPifQdgj8dymlPmqSfQU0SWishyERlbwb4dnOflHRMAEblVRBJEJMHlclWjGsYYEzxc2blENYqkXnjwXiL3Z2SlXfvQEssRQA9gNDAJeEVEWpSzrzfHdBeqzlLVeFWNj4mJ8TpoY4wJZhnHcoN2uPxi/kwsaUBHj+VYYF8p23ysqvmqmgpsw51oyto3zXle3jGNMabWcmUH9zhh4N/EsgroISJdRSQSmAjML7HNR8C5ACISjfvUWAqwABgjIi1FpCUwBligqvuBYyJyutMb7HrgYz/WwRhjgkqwjxMGfuwVpqoFIjIVd5IIB2ar6iYRmQ4kqOp8fk4gm4FC4AFVzQQQkcdwJyeA6aqa5Ty/HXgdaIi7N5j1CDPG1Amq6owTFtwtFr8lFgBV/Rx3l2DPskc8nitwr/Moue9sYHYp5QlAf58Ha4wxQS47t4CT+UVBn1iCt1uBMcaYU2Rku2+OrMsX740xxvhQ8ZTE1mIxxhjjE8WJxVosxhhjfCIjBIZzAUssxhgTMlzHcgkPE1o2Cu7uxpZYjDEmRGRk5xLVOJLwsOAe1N0SizHGhIhgn5K4mCUWY4wJEa7s3KCe4KuYJRZjjAkRGdZiMcYY4yvu4VzyiG4a3BfuwRKLMcaEhKMnCsgrLLIWizHGGN9wZZ8Egv8eFrDEYowxIcF1zD1OmLVYjDHG+IQrRO66B0ssxhgTEkJlnDCwxGKMMSEhIzuXeuFC84b1Ah1KhSyxGGNMCHBPSVyfsCAfzgUssRhjTEjIyM4NidNgYInFGGNCgutY8M91X8yviUVExorINhFJFpFppayfIiIuEUl0Hjc75ed6lCWKyEkRmeCse11EUj3WDfZnHYwxJhi4T4UF/133ABH+OrCIhAMzgQuBNGCViMxX1c0lNp2rqlM9C1T1O2Cwc5woIBn4ymOTB1R1nr9iN8aYYFJUpGTm5FmLBRgBJKtqiqrmAXOA8VU4zlXAF6p63KfRGWNMiDh0PI/CIrVrLEAHYI/HcppTVtKVIrJeROaJSMdS1k8E3i1R9rizzwwRKfUvLSK3ikiCiCS4XK4qVcAYY4JBegjdwwL+TSyl9YnTEsufAF1UdSDwNfDGKQcQaQcMABZ4FD8I9AaGA1HAH0t7cVWdparxqhofExNTtRoYY0wQWL3rEAB92jUNcCTe8WdiSQM8WyCxwD7PDVQ1U1VzncWXgWEljnEN8KGq5nvss1/dcoHXcJ9yM8aYWmvRNhcdWjSkW0yTQIfiFX8mllVADxHpKiKRuE9pzffcwGmRFBsHbClxjEmUOA1WvI+ICDAB2OjjuI0xJmjkFhTy444MRveKwf21F/z81itMVQtEZCru01jhwGxV3SQi04EEVZ0P3CUi44ACIAuYUry/iHTB3eJZXOLQb4tIDO5TbYnAbf6qgzHGBFrCzkMczytkdK/WgQ7Fa35LLACq+jnweYmyRzyeP4j7mklp++6klIv9qnqeb6M0xpjgtWhbOpHhYYzq1irQoXjN7rw3xpggtmibixFdo2hc36/tAJ+yxGKMMUFq7+ETbE/PZnSv0OrZaonFGGOC1KJt6QCWWIwxxvhGqHUzLmaJxRhjglBeQRE/JodWN+NilliMMSYIJezMIifEuhkXs8RijDFBaFGSK+S6GRezxGKMMUFo0bZ0hndtGVLdjItZYjHGmCCz7/AJkg5mM7pn6J0GA0ssxhgTdL7dGprdjItZYjHGmCBSVKS8uWwnvdo0pXvr0OpmXMwSizHGBJFFSekkHczmd+fEhVw342KWWIwxJoi8tCiF9s0bcNmg9oEOpcossRhjTJBYvesQK3dmcdNZcdQLD92v59CN3Bhjapl/L95B84b1mDi8Y8UbBzFLLMYYEwSS07NZuOUg14/sHJL3rniyxGKMMUHglSUpRIaHccOoLoEOpdossRhjTIClHz3Jf9fs5er4WKKb1A90ONXmVWIRkW4iUt95PlpE7hKRFv4NzRhj6obZS3dSUFTELWfFBToUn/C2xfIBUCgi3YFXga7AOxXtJCJjRWSbiCSLyLRS1k8REZeIJDqPmz3WFXqUz/co7yoiK0Rku4jMFZFIL+tgjDFBR1X5cG0a5/dpQ+dWjQMdjk94m1iKVLUAuBx4WlXvAdqVt4OIhAMzgYuBvsAkEelbyqZzVXWw83jFo/yER/k4j/K/AzNUtQdwCLjJyzoYY0zQ2XrgGAeP5nJhnzaBDsVnvE0s+SIyCbgB+NQpq1fBPiOAZFVNUdU8YA4wvmphuon7NtTzgHlO0RvAhOoc0xhjAmlxkguAs3uG5rhgpfE2sdwIjAQeV9VUEekK/KeCfToAezyW05yykq4UkfUiMk9EPDtvNxCRBBFZLiLFyaMVcNhpPZV3TETkVmf/BJfLVUGoxhgTGIu3uejdtiltmzcIdCg+41ViUdXNqnqXqr4rIi2Bpqr6RAW7lTbIjZZY/gTooqoDga9xt0CKdVLVeOBa4GkR6eblMYtjnqWq8aoaHxNTe34JGGNqj+zcAhJ2ZXFOiI5iXBZve4UtEpFmIhIFrANeE5F/VbBbGuDZAokF9nluoKqZqprrLL4MDPNYt8/5NwVYBAwBMoAWIlJ899AvjmmMMaHix+QM8guVc2rRaTDw/lRYc1U9ClwBvKaqw4ALKthnFdDD6cUVCUwE5ntuICKeHQDGAVuc8pYe3ZujgTOAzaqqwHfAVc4+NwAfe1kHY4wJKouTXDSODCe+c1SgQ/EpbxNLhJMEruHni/flcq6DTAUW4E4Y76nqJhGZLiLFvbzuEpFNIrIOuAuY4pT3ARKc8u+AJ1R1s7Puj8C9IpKM+5rLq17WwRhjgoaqsmibi1Hdo4mMqF33qns7IM103AliqaquEpE4YHtFO6nq58DnJcoe8Xj+IPBgKfv9CAwo45gpuHucGWNMyNrhymHv4RPccW63QIfic14lFlV9H3jfYzkFuNJfQRljTG33UzfjHrXr+gp4f/E+VkQ+FJF0ETkoIh+ISKy/gzPGmNpq0bZ0usU0pmNUo0CH4nPenth7DfeF9/a47xv5xCkzxhhTSSfyClmRmsXoXq0DHYpfeJtYYlT1NVUtcB6vA7Wv/WaMMTVgeWomeQVFta6bcTFvE0uGiPxGRMKdx2+ATH8GZowxtdXibS4a1AtjRNfa1c24mLeJ5be4uxofAPbjvo/kRn8FZYwxtZWqsjjJxci4VjSoFx7ocPzC2yFddqvqOFWNUdXWqjoB982SxhhjKmHmd8mkZuRwYd+2gQ7Fb6pzV869PovCGGPqgNeWpvKPr5K4YkgHJg7vWPEOIao6iaW0ASGNMcaU4r2EPTz6yWYu6teGJ68aSFhY7f0KrU5iKXVUYWOMMaf6bP1+pn2wnrN6RPPspCFEhNeuIVxKKvfOexE5RukJRICGfonIGGNqkZWpWdw9Zy3DOrfk39cNo35E7bxg76ncxKKqTWsqEGOMqW1O5hfyh3nraN+iIa9OGU6jSG+HZwxtdaOWxhgTAM98s52dmcd5++bTaNagotnca4/afaLPGGMCZNO+I8z6PoVr4mM5o3t0oMOpUZZYjDHGxwoKi/jjB+tp2SiS/72kT6DDqXF2KswYY3xs9tJUNu49ysxrh9KiUWSgw6lx1mIxxhgf2pWZw78WJnFBnzZcMqD23l1fHkssxhjjI6rKwx9tJCIsjMcm9EOk9t4EWR6/JhYRGSsi20QkWUSmlbJ+ioi4RCTRedzslA8WkWUisklE1ovIrz32eV1EUj32GezPOhhjjLc+27CfJdszuH9MT9o1r7u3+vntGouIhAMzgQuBNGCViMxX1c0lNp2rqlNLlB0HrlfV7SLSHlgtIgtU9bCz/gFVneev2I0xprKycwt47NPN9GvfjN+c3jnQ4QSUP1ssI4BkVU1R1TxgDjDemx1VNUlVtzvP9wHp2MRixpgg9vTCJNKP5fJ/E/rX+iFbKuLP2ncA9ngspzllJV3pnO6aJyK/GO5TREYAkcAOj+LHnX1miEh9n0ZtjDGVtGX/UV77cScTh3diSKeWgQ4n4PyZWEq7alVy3LFPgC6qOhD4GnjjlAOItAPeAm5U1SKn+EGgNzAciAL+WOqLi9wqIgkikuByuapeC2OMKUdRkfuCffOG9fjDRb0CHU5Q8GdiSQM8WyCxwD7PDVQ1U1VzncWXgWHF60SkGfAZ8LCqLvfYZ7+65QKv4T7l9guqOktV41U1PibGzqIZY/xj3po0Vu86xLSLe9Oycd27Z6U0/kwsq4AeItJVRCKBicB8zw2cFkmxccAWpzwS+BB4U1XfL20fcffjmwBs9FsNjDGmHEeO5/PEF1sZ1rklVw2NDXQ4QcNvvcJUtUBEpgILgHBgtqpuEpHpQIKqzgfuEpFxQAGQBUxxdr8GOBtoJSLFZVNUNRF4W0RicJ9qSwRu81cdjDGmPM98s51Dx/N4c9yIWj1xV2WJau2frys+Pl4TEhICHYYxphZJTs9m7NPfc3V8LH+7YmCgw/ELEVmtqvGV3a9u94kzxpgqevyzzTSsF859Y+yCfUmWWIwxppK+25bOd9tc3HV+D6Kb2B0PJVliMcaYSsgvLOL/Pt1M1+jG3DCqS6DDCUqWWIwxphLeWraLHa4cHr60D5ER9hVaGvurGGOMlw4fz+Ppr5M4q0c05/VuHehwgpYlFmOM8dL8dfs4erKAP47tXWeHxPeGJRZjjPHSR2v30rttU/p3aB7oUIKaJRZjjPHC7szjrNl9mHGD2wc6lKBnicUYY7wwf91eAMYNssRSEUssxhhTAVXlo8R9DO/SktiWjQIdTtCzxGKMMRXYsv8YyenZjB9c2pRSpiRLLMYYU4GPE/cSESZcMqBdxRsbSyzGGFOeoiJl/rp9nNMzhiibb8UrlliMMaYcK3dmsf/ISesNVgmWWIwxphwfJ+6jUWQ4F/ZtE+hQQoYlFmOMKUNeQRGfb9jPmL5taBTpt3kRax1LLMYYU4bvtqVz5EQ+44dYb7DKsMRijDGlyMrJY/onm4lt2ZAzu0cHOpyQYm07Y4wpobBIuXvOWlzHcpl3+0jqhdtv8Mrw619LRMaKyDYRSRaRaaWsnyIiLhFJdB43e6y7QUS2O48bPMqHicgG55jPig0xaozxsWe+TmLJ9gweHd+PgbEtAh1OyPFbYhGRcGAmcDHQF5gkIn1L2XSuqg52Hq84+0YBfwZOA0YAfxaRls72LwK3Aj2cx1h/1cEYU/d8s+Ugz36bzNXDYpk4vGOgwwlJ/myxjACSVTVFVfOAOcB4L/e9CFioqlmqeghYCIwVkXZAM1VdpqoKvAlM8Efwxpi6Z3fmce6Zm0i/9s14bEJ/m3OlivyZWDoAezyW05yykq4UkfUiMk9Ein8elLVvB+d5RcdERG4VkQQRSXC5XFWtgzGmjkhOP8b1s1cA8OLkYTSoFx7giEKXPxNLaaleSyx/AnRR1YHA18AbFezrzTHdhaqzVDVeVeNjYmK8DNkYUxd9tekAE2b+SHZuAbOnDKdTKxvBuDr8mVjSAM8TlLHAPs8NVDVTVXOdxZeBYRXsm+Y8L/OYxhjjraIiZcbCJG59azVxMY2ZP/VM4rtEBTqskOfPxLIK6CEiXUUkEpgIzPfcwLlmUmwcsMV5vgAYIyItnYv2Y4AFqrofOCYipzu9wa4HPvZjHYwxtVR+YRG3/Wc1z3yznSuHxvLe70bSvkXDQIdVK/jtPhZVLRCRqbiTRDgwW1U3ich0IEFV5wN3icg4oADIAqY4+2aJyGO4kxPAdFXNcp7fDrwONAS+cB7GGFMpMxYm8dXmgzx8aR9uOrOrXaj3IXF3rqrd4uPjNSEhIdBhGGOCxNLkDH7z6gp+Hd+RJ64cGOhwgpaIrFbV+MruZ7eTGmPqlMzsXO6Zm0hcdGMeuay0W+tMddmQLsaYOkNVeWDeeg4fz+f1G0fYiMV+Yi0WY0yd8fqPO/l2azoPXtKbvu2bBTqcWsvStTGmVlJVdmcdJ8WVww5XNjtcOXywOo3zerdmyqgugQ6vVrPEYkwIUlUKitRG3S3H459t4ZUfUn9ajmocyajurXjqqoHWA8zPLLEYE4JeXpLC898m88LkYZzZw+YKKc3q3Yfo264Z08f3Iy6mCVGNIwMdUp1hP3eMCUH/XbOXoycLmPLaSj5YnVbxDnVQakYOQzq1IL5LlCWVGmaJxZhKWJzkYt7qNDKycyve2E/SDh1n64Fj3HVed06Li+K+99fx3DfbqQv3pHkrKyePw8fz6RrdONCh1El2KswYL+3OPM4tbyaQV1CECAzu2IIL+rThon5t6d66SY3F8e3WdAAmDOnA1PN6MO2D9fxzYRJ7D5/g/yb0J8Kuu5CakQ1AXIwllkCwT6AxXpr+6WYiwoQ3fzuC35/fk8Ii5akF2xgzYzGPfrKJ7NyCGolj4eaDxEU3Ji6mCZERYfzzmkFMPbc7c1bt4U8fb7SWC5DiygEgLrrmEr75mbVYjPHCd1vT+XrLQf44tjdn94zh7J4x3H1BDw4ePcnz3ybz+o87+WLDAR4d34+L+rX1WxzZuQWsSMnihlGdfyoTEe6/qBcAz3+XTNtmDbn7gh5+iyEUpGTkEBEmxLa0QSUDwVosxjiO5xXwxYb9FBQWnVJ+Mr+Qv3yyibiYxtx0ZtdT1rVp1oDHJvTng9tH0aJRPX731mpufiOBLfuP+iXGJUku8gqLOL9Pm1+su29MT64cGsuMr5OYu2q3X14/VKS6cujUqpGdFgwQ+6uXY0VKJgs2HQh0GKYKCouUhJ1ZFBZ5f1roma+3c/vba7jm38vYk3X8p/JXlqSwK/M4j47rR2RE6f9lhnZqySd3nsmDF/fmxx0ZXPzMEq57dQVLtrt8emrq6y3pNG9Yj/jOLX+xTkR44soBnN0zhv/9cCPfOddiUlzZPP/tdi5+ZglXvLCU7QeP+SyeYJWakWOnwQLIToWVQVV59tvtrEzN4pUbhnNOT5uFMlB2ZuRw7cvLad2sAaO6tWJUt2iGdW5Jw8jSp44tKCzi/vfX8VHiPs7tFcMzk4bQrEG9cl/jeF4B767cTd92zdh+MJtLnl3CE1cMZHCnFjz/XTIX92/LWT3K/wzUCw/jd+d049fDO/L2it28/uNOrnt1Jb3bNuWcXjF0aNGQ9s0b0r5FQ+JiGld66tvCIuW7bemc2yumzF/i9cLDeGHyUCbOWsYdb6+ha3RjNjutp2GdW5KakcO455fy6Lh+XB0fWytvFCwsUlIzczinl/2fDRQbNr8cR07kM2nWclIysnnzt6cxoqvNLFfT8guLuOqlZaS6sunRpinr9hymoEiJDA/j0oHteOjSPkQ3qX/K9r+fk8hnG/Zz6YB2LNh0gC7RjXn1hng6tyq7h9DbK3bx0Icbef+2kbRp2oC75qwlcc9h2jVvwKHjeXxz32g6VHISqNyCQj5O3Meby3aSdCCbPI9TbK2b1mfm5KEMr8Rshat3ZXHli8t4btIQLhvUvtxt04+d5PpXV9KgXji/GtiOSwe2o13zhhw8epLfz0lkWUom4we35/HLB9Ckfu36fbkn6zhnPfkdf7tiAJNGdAp0OCGtqsPmW2KpQGZ2Ltf8exkHj+byzi2nMTC2hY+jM+X551fbeO7bZGZeO5RLB7YjJ7eAVTuzWLTNxTsrdtMwMpyHLu3D1cNiySssYuo7a1noTN5081lx/Lgjg9v/swYReHHyMEZ2a/WL11BVxsz4nsiIMD6980xEhPzCImYsTOLFxTt44KJe3DG6e7XqUVSkZOTksu/wSXZnHedfX21jz6ETTBvbm5vPOnWSqaSDx/h0/X7GD25Pt5ifT+f8/cutvPx9CmseubDCFlh5CouUF75LZsbXSXSKasTMyUPp1755teoXTL5PcnH97JXMvfV0Tov75fttvGeJpRzVnehr/5ETXP3SMnJyC5j7u5H0bNPUh9GZHa5s0o/mcnpc1ClfsCtTs5g4axlXDI3lH1cP+sV+yenHePC/G1i18xAj41pRLyKM75NcPDquHzd4DDK4MyOHm99MYGdGDk9PHMyvBp76a39pcgaTX1nBP64exFXDYk9Zl5mdS1TjSJ+fMjp6Mp8/vL+eLzcd4KJ+bfjr5QP4ITmDt5fvZuVO92SpzRpE8NJ1wxjVzT1ky4X/WkxM0/q8c8vpPolh1c4s7nxnLYeO5/HYhP5cE9/RJ8cNtNeXpvKXTzaz8qHzad20QaDDCWmWWMrhixkkd2XmcPVLy8gtKOKcnjGM6BrFiK5RdI9pQlhY7TtPXVOKipQxT39Pcno2Z/eM4ZFf9aF766YcOZHPJc8sISJc+Oyus8o8XVNUpLy7ajdPfL6V7LwCHp8wgGtP++Xpj6Mn87np9VWsTzvC+7eNPKXlefMbq1i7+zBLp51X6ese1aGqvPpDKk98sZVCVVShU1QjJp/WiVHdorn3vURSM3L42xUDOK1rK85+6jv+9Ku+v+iZVh0Z2bncPWctS5Mz+XV8Rx4d369G/wb+8OePN/LBmr1s+MuYWnkNqSZZYimHr6YmTk7P5umvk1iZmkX6MfeQHi0b1WPyaZ255ew4mjes+umJuurbrQf57esJXDaoPYu2pXM8r5DrR3Ym/WguX246wLzbRjKk0y97QJWUfuwk+w+fZFDHsk9VZmbnMu75pRQWKfPvPIPWTRuwKzOH0f9YxNRzu3PfmF6+rJrXVu/KYn7iPs7r04azukf/9EPlyIl87nh7NUuTMxkY25z1aUdY/MDocq8VVUVhkTJjYRLPf5dM33bNmDl5aEgPhXLdqys4fDyfT+48M9ChhLygTCwiMhZ4BggHXlHVJ8rY7irgfWC4qiaIyGTgAY9NBgJDVTVRRBYB7YATzroxqppeXhy+nvO+eJ6HlalZfLMlnS83HaBZgwhuG92NKaO62Kx0lTBp1nJ2Zeaw+A/ncvREPjynbzkAABZpSURBVP9cmMS7K3ejCvdd2JM7z/ftjX6b9h3hqheX0addU9699XT+/sU23ly2k6XTzqNNs+A7bZJfWMTDH25kbsIeurduwtf3nuO31/p260HumbuO/MIi/nJZ6PYaO+OJb4nv0pJnJg4JdCghL+gSi4iEA0nAhUAasAqYpKqbS2zXFPgMiASmqmpCifUDgI9VNc5ZXgTcX3K78vg6sZS0ad8R/vVVEt9sTSe6SX3uPr87E0d0qhNzZXy6fh8RYWGM7V/5u803pB3hsud/4KFL+nDL2XE/lW/ad4QVKVlcP7KzX25w+2z9fv7nnTVMGNyeb7akc27v1jw7KXi/hFSVeavT6NCiIaO6+3eI/H2HT3Dve4ksT8nikgFt+evlA2jRKHRGBj6ZX0ifR77k7vN78PsLegY6nJBX1cTiz2++EUCyqqaoah4wBxhfynaPAU8CJ8s4ziTgXf+E6Bv92jfn1SnD+eD2kXSLacyfPt7E2Ke/55stB2v1uE2qysMfbeS2/6zmX19tq3RdX16SQtP6EUwccepF437tm/PbM7v67a7pSwe2487zuvNR4j6O5RZw4xld/PI6viIiXB3f0e9JBaB9i4a8ffPp/HFsb77adJCLn1nC8pRMv7+ur+zKPI4qIX0qrzbwZ2LpAOzxWE5zyn4iIkOAjqr6aTnH+TW/TCyviUiiiPxJymiri8itIpIgIgkul6sK4VfesM5RzLn1dF6+Ph5VuOmNBCa/soJN+47UyOvXtN1Zxzl8PJ+4mMY8+20yv5+byMn8Qq/23Xv4BJ9t2M+k0zrRtBpdZ6vqngt6cvmQDlzQp7VX13DqkvAw4fbR3fjwjjNoUC+cya+s4OXvU3z2I6mgsIjVuyo3KoK3ikc19uymbWqePxNLaV/4P32SRCQMmAHcV+YBRE4DjqvqRo/iyao6ADjLeVxX2r6qOktV41U1Piam5u7AFREu7NuGBfeczaPj+rFl/1Eue+4H/vr5Fk7kefelGyrWp7kT5rMTh/DARb34OHEf1726gqycvAr3fe2HVAQCNvd4WJgw49eDefn6Srfy64wBsc2ZP/UMxvRtw+Ofb2HqO2t9MoLzS4t3cOWLy7hwxmI+WJ32i7HZqmOHM6pxF2uxBJQ/rzKnAZ7nOGKBfR7LTYH+wCKn0dEWmC8i4zyun0ykRGtFVfc6/x4TkXdwn3J70y81qIZ64WHcMKoLEwZ34IkvtzLr+xS+3HiAv14+ICBTyWZk5/LUl9u4/6JexDStX/EOXlifdpjIiDB6tW1K/w7N6RTViPveX8f5/1xEfJcoBndsweCOLRgY2/yUVsnRk/nMWbWHXw1sR/tK3s3ua6F4cbomNW1QjxcmD+Xf36fw5Jdb2XbwGI+O68e+wydYn3aE9WmH2Xv4BHeM7s6NZ3Sp8O+ZnVvAy0tSGRTbnNyCIu57fx3PfLOdO0Z348phsdW+LpmakUPrpvVr3WgCocaff/1VQA8R6QrsxZ0kri1eqapHgJ++YUtelHdaNFcDZ3tsEwG0UNUMEakH/Ar42o91qLbmjerxtysGMH5wex787wZ+8+oKrhoWy2Pj+5c51pU/vLlsF3MT9nD0ZD4v/maYT465Lu0Ifds1++nL4LJB7ekY1YjXl6ayLu0ICzcfBEAE+rVvxsi4Vozs1or1aUfIzi3g5rPiyju8CRIiwm3ndGNAh+bc+e5aJr+yAoAm9SPo36EZ3WKaMP3TzazamcXfrxpY7qgAby7byZET+Uwf358BHZrz9ZaDPPdtMtP+u4EP1+5l5uShpwzRU1mpGTk2uVcQ8FtiUdUCEZkKLMDd3Xi2qm4SkelAgqrOr+AQZwNpqpriUVYfWOAklXDcSeVlP4Tvc6fHteKLu8/iuW+38+KiHew7fIJXbxheI8mloLCI91btoUn9CL7YeIAvNuzn4gHtqnXMwiJl494jXF3iTvXBHVvwtNPN8/DxPNalHWHNrkMsT8nkjR938fKSVABGdWtF/w61ZxiRuuCM7tF8cfdZJOw8RK+2TYmLbkxYmKCqvLwkhb9/uY3Nz/3AzGuHlvre5uQW8MqSVEb3ivnpfqMx/dpyYd82fLBmLw99uIFxz/3Av6+LZ0Bs1T4bKa5sxvav3mfbVJ9f24uq+jnweYmyR8rYdnSJ5UXA6SXKcgDf/NwOgAb1wnngot50b92Ee99bxy1vJvDKDfF+v9N5cZKLA0dPMvPaoby4OJk/fbyJkd1aVasb6Q5XNsfzCssdO61Fo0jO6Rnz08jQJ/MLWbP7EGt2HapS92QTeG2aNeDSgad+cYsIt57djaGdWjL1nbVc8eKP/P3KAVw+5NQfHW+v2EVWTh53ntfjF/tfNSyW3m2b8ru3VnPlSz/yt8sHcGWJHy0VOZSTx6Hj+cTZ9ZWAq/03WgShy4fE8tRVg1i6I4Nb3kzwuidVVb27cjcxTeszpl8bnrxyEIeP5/F/n22p1jGLL9wP6uj9L8sG9cIZ1S2aqef1oHtrG2+ttonvEsVnd53J0E4tuGfuOv6zfNdP607kFTLr+xTO6uGe8qA0/Tu4OwsM7dSC+95fxz1zE0lOz/b69VMznemI7VRYwFliCZCrhsXy9ysGsmR7Br97a7Xfksv+Iyf4dms6VzsXRvu2b8Zt53Rj3uo0FidVvRv2+rTDNI4Mp6tNpmQ8tGpSn9dvHMF5vVvz8EcbefUH96nPd1buJiM7j7sqGEmhVZP6vHXTadw+uhufb9jPhTMW87u3Ekjcc7jC1y6e597uYQk8SywBdM3wjjxxxQAWJ7n49b+XsTMjx+ev8d6qNIoUJg7/eWDGO8/vTreYxvzvfzdUufvourQj9O/QnHAbgNOU0KBeOC/9ZhgX92/LY59uZsbCJF5avIORca28mn+mXngYfxzbm6XTzuN/Rndn2Y5MJsxcyjX/XsZHa/eW+SMsNSObiDChY1QjX1fJVJIllgCbOKITL04eSmpGDpc+u4R5q9N8diNaYZEyd9VuzuoRTadWP/9nqx8RzpNXDWLfkRNc9+oKktMrN1VtXkERW/YdLXfAR1O3RUaE8dykIYwf3J5nvtmO61huha2VkqKb1Of+i3rx44Pn87+X9Gbf4RP8fm4iIx7/mj99tJH1aYdPuckyNSOHTlGN6sRQSsHOOnsHgYsHtGNQxxbcMzeR+99fx6Jt6Tx++YBqj5b8/XYX+46c5OFf9f3FumGdW/LcpCE8/NFGLnnmB+6+oAe3nh3n1X/KpIPHyCssYmAVe+6YuiEiPIx/XTOYmCb1OXIin9PjqjYDa5P6Edx6djduPjOO5SmZzE3Yw9yEPby1fBf1I8LoFtOEHm2akLDzEAOsp2FQsMQSJNq3aMg7t5zOS4t3MGNhEuvTjjB7ynC6t676NYx3V+wmukkkF/RpU+r6Xw1sz2ldW/GX+Zt4asE2Plu/n39cPYi+7ZuVe9x1ae7z3QM7WIvFlC88TEr9YVMVYWHCqO7RjOoezfTj+SzccpBtB46SdDCbhJ2HSD+Wy9AyOgaYmmWJJYiEhwn/c253To9rxe/eSuCKF5by4m+GcUYVBh88ePQk32xN55az4oiMKLsVEuPMvX7ZxgP86eONTHp5OZ/eeWa556nX7zlCy0b16BgV2LvmTd3VvFG9X8z2eTK/kPrlfNZNzbF3IQgN69ySD+84g7bNG3DD7JXMWbm7UvvnFhTyt8+3UFikTBzu3XSzY/u3Zd5tIykqUqa+s4bcgrJ7qa1LO8yA2BY2HIoJKg3qhdtnMkhYYglSHaMaMe/2UYzs1opp/93A9E82s//IiQr325N1nKtfWsZHifu487zulRqMr3Orxjx19UDWpR3hr2Xc53Iir5Dt6dkMsusrxpgy2KmwINasQT1emzKcRz/ZzOylqcxemsrgji0Y278tY/q2oUurxj9NYwuwcPNB7nsvEQVmXTeMMf0qf3f72P7tuOnMrrz6QyrxXaK4bFD7U9Zv3n+EwiK1i6TGmDJZYglyEeFhPDahP1PO6MKXGw+wYNMBnvhiK098sZWIMCG6SX1inNFcl6Vk0r9DM164dtgp3Ysra9rFvVm7+xDTPlhPv/bNiPOY22LdnuI77u3CvTGmdH6d8z5Y+Htq4pqWdug4i5Nc7D10AtexXNKP5eI6lsvpca34w9hePhl7bN/hE1z67BJaNanP9PH9GBnXChHh93PWsiwlkxX/e4EPamKMCWZVnZrYWiwhKLZlIyaf1tmvr9G+RUOev3YoU99Zw7Uvr6B326ZMGdWFxD2Hyx140hhj7OK9KdMZ3aNZ9uD5PHnlQACm/XcDOzOPM9CurxhjymEtFlOuBvXCuWZ4R66Oj2Vlahafb9jPFZUcztwYU7dYYjFeERFOi2vFaXGtAh2KMSbI2akwY4wxPmWJxRhjjE9ZYjHGGONTfk0sIjJWRLaJSLKITCtnu6tEREUk3lnuIiInRCTRebzkse0wEdngHPNZscGBjDEmqPjt4r2IhAMzgQuBNGCViMxX1c0ltmsK3AWsKHGIHao6uJRDvwjcCiwHPgfGAl/4OHxjjDFV5M8WywggWVVTVDUPmAOML2W7x4AngZMVHVBE2gHNVHWZuocMeBOY4MOYjTHGVJM/E0sHYI/HcppT9hMRGQJ0VNVPS9m/q4isFZHFInKWxzHTyjumx7FvFZEEEUlwuVxVroQxxpjK8ed9LKVd+/hpYDIRCQNmAFNK2W4/0ElVM0VkGPCRiPSr6JinFKrOAmaBe6ywyoVujDGmqvyZWNIAz1mmYoF9HstNgf7AIuf6e1tgvoiMU9UEIBdAVVeLyA6gp3PM2HKOWarVq1dniMguoDlwBIgGMqpYLzyOU53tSltXsqy85eLnnmU1Ua/q1qlkWUXPq1unsmKozDa+eq88nwf6M+hteWXeKwiNz2CovVdlrauJ74uqDUqoqn554E5aKUBXIBJYB/QrZ/tFQLzzPAYId57HAXuBKGd5FXA67tbLF8AllYhplvNvQjXrNqu625W2rmRZecsedfEs83u9qlunsupRTv2qVaeaqpc375Uv61Xdz6C35ZV5r2qqXnXtvfJlvXz5XpX38FuLRVULRGQqsAAIB2ar6iYRme5UaH45u58NTBeRAqAQuE1Vs5x1twOvAw1xJ5bK9Aj7pJLVqO5xytuutHUly8pb/qSMbarDm2NVt04ly7x5Xl01US9v3itvY/FGdT+D3pbbe1V9tfX7okx1Yj6WkkQkQaswx0Cwq431qo11AqtXKKmNdQL/1quu3nk/K9AB+EltrFdtrBNYvUJJbawT+LFedbLFYowxxn/qaovFGGOMn1hiMcYY41Mhn1hEZLaIpIvIxirsW+aAliJypzOA5iYRedK3UXsVm8/rJSJ/EZG9HoN7XuL7yMuNyy/vlbP+fmcg02jfRex1bP54rx4TkfXO+/SViLT3feTlxuWPOj0lIluden0oIi18H3mFsfmjXlc73xNFxQPp1oTq1KWM490gItudxw0e5ZUf+Ndf/Zhr6oG7a/JQYGMV9l0JjOTne2IudsrPBb4G6jvLrWtJvf4C3F+b3itnXUfc3dp3AdG1oV64x8Qr3uYu4KVaUKcxQITz/O/A32vJe9UH6IXHvXjBXBcnzi4lyqJw33cYBbR0nrcsr97lPUK+xaKq3wNZnmUi0k1EvhSR1SKyRER6l9yvggEtbweeUNXiu//T/VuLX/JTvQLKj3WaAfyBMob38Td/1EtVj3ps2pgarpuf6vSVqhY4my7n1FE0aoSf6rVFVbfVRPyeqlqXMlwELFTVLFU9BCwExlb1+yTkE0sZZgF3quow4H7ghVK2KW9Ay57AWSKyQtyDYA73a7Teq269AKY6pyJmi0hL/4XqtWrVSUTGAXtVdZ2/A62kar9XIvK4iOwBJgOP+DFWb/ni81fstwTPdBe+rFegeVOX0pQ1aHCV6u3PscICQkSaAKOA9z1OBdYvbdNSyop/FUbgbg6eDgwH3hOROCdjB4SP6vUi7mkK1Pn3n7j/gwdEdeskIo2Ah3CfYgkaPnqvUNWHgIdE5EFgKvBnH4fqNV/VyTnWQ0AB8LYvY6wKX9Yr0Mqri4jcCNztlHUHPheRPCBVVS+n7PpVqd61LrHgboUd1hKThIl74rHVzuJ83F+yZQ1omQb810kkK0WkCPeAbYEcf7/a9VLVgx77vQyUNl1BTapunbrhHotunfMfKRZYIyIjVPWAn2Mvjy8+g57eAT4jgIkFH9XJuSj8K+D8QP5Q8+Dr9yqQSq0LgKq+BrwGICKLgCmqutNjkzRgtMdyLO5rMVUa+LdGL5z56wF0weMCFvAjcLXzXIBBZexX6oCWwG3AdOd5T9xNRKkF9Wrnsc09wJxQr1OJbXYSgIv3fnqvenhscycwrxbUaSywGYgJxHvk788gNXzxvqp1oeyL96m4z9S0dJ5XeeDfgL25PvzDvot7/pZ83Nn1Jty/Yr/EPaLyZuCRMvaNBzYCO4Dn+XkkgkjgP866NcB5taRebwEbgPW4f4W1q6n6+KtOJbbZSWB6hfnjvfrAKV+Pe+DADrWgTsm4f6QlOo8a7enmx3pd7hwrFzgILAjmulBKYnHKf+u8R8nAjRXVu7yHDelijDHGp2prrzBjjDEBYonFGGOMT1liMcYY41OWWIwxxviUJRZjjDE+ZYnF1Ekikl3Dr/eKiPT10bEKxT3q8UYR+aSiUYJFpIWI3OGL1zbGG9bd2NRJIpKtqk18eLwI/XmARb/yjF1E3gCSVPXxcrbvAnyqqv1rIj5jrMVijENEYkTkAxFZ5TzOcMpHiMiPIrLW+beXUz5FRN4XkU+Ar0RktIgsEpF54p535O3iuSuc8njnebYzwOQ6EVkuIm2c8m7O8ioRme5lq2oZPw/I2UREvhGRNeKeP2O8s80TQDenlfOUs+0DzuusF5FHffhnNMYSizEengFmqOpw4ErgFad8K3C2qg7BPcrwXz32GQncoKrnOctDgN8DfYE44IxSXqcxsFxVBwHfA7d4vP4zzutXOB6TM57V+bhHUQA4CVyuqkNxzyn0TyexTQN2qOpgVX1ARMYAPYARwGBgmIicXdHrGeOt2jgIpTFVdQHQ12Nk2GYi0hRoDrwhIj1wj+xaz2OfharqOSfGSlVNAxCRRNxjOf1Q4nXy+HkA0NXAhc7zkfw818U7wD/KiLOhx7FX4547A9xjOf3VSRJFuFsybUrZf4zzWOssN8GdaL4v4/WMqRRLLMb8LAwYqaonPAtF5DngO1W93LleschjdU6JY+R6PC+k9P9j+frzxc2ytinPCVUdLCLNcSeo/wGexT1vSwwwTFXzRWQn0KCU/QX4m6r+u5Kva4xX7FSYMT/7Cve8JwCISPHw482Bvc7zKX58/eW4T8EBTKxoY1U9gnva4vtFpB7uONOdpHIu0NnZ9BjQ1GPXBcBvnfk7EJEOItLaR3UwxhKLqbMaiUiax+Ne3F/S8c4F7c24p08AeBL4m4gsBcL9GNPvgXtFZCXQDjhS0Q6quhb3SLYTcU+cFS8iCbhbL1udbTKBpU735KdU9Svcp9qWicgGYB6nJh5jqsW6GxsTJJwZMU+oqorIRGCSqo6vaD9jgo1dYzEmeAwDnnd6ch0mgNNGG1Md1mIxxhjjU3aNxRhjjE9ZYjHGGONTlliMMcb4lCUWY4wxPmWJxRhjjE/9P3dpHZaY4fdyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_start=15, skip_end=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.510128</td>\n",
       "      <td>0.488822</td>\n",
       "      <td>0.829460</td>\n",
       "      <td>09:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.480796</td>\n",
       "      <td>0.456404</td>\n",
       "      <td>0.830959</td>\n",
       "      <td>09:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.437961</td>\n",
       "      <td>0.429785</td>\n",
       "      <td>0.844828</td>\n",
       "      <td>09:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.383612</td>\n",
       "      <td>0.423067</td>\n",
       "      <td>0.848201</td>\n",
       "      <td>10:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 1e-2\n",
    "lr_reduction_factor = 2.\n",
    "learn.fit_one_cycle(4, slice(lr/lr_reduction_factor,lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice here that we're starting to overfit - as with the language model, this is okay so long as our validation loss and accuracy metric both continue to improve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(clas_trial_prefix + 'third')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load(clas_trial_prefix + 'third');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.258947</td>\n",
       "      <td>0.439739</td>\n",
       "      <td>0.850075</td>\n",
       "      <td>01:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.249648</td>\n",
       "      <td>0.454287</td>\n",
       "      <td>0.847451</td>\n",
       "      <td>01:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.234214</td>\n",
       "      <td>0.457502</td>\n",
       "      <td>0.846702</td>\n",
       "      <td>01:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.220239</td>\n",
       "      <td>0.455409</td>\n",
       "      <td>0.849325</td>\n",
       "      <td>01:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 1e-3\n",
    "lr_reduction_factor = 2.\n",
    "learn.fit_one_cycle(4, slice(lr/lr_reduction_factor,lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we continue training, we see the training loss continue to drop, but the validation loss and accuracy are no longer improving, in fact they're starting to get a bit worse. \n",
    "\n",
    "This is a good time to export our our best model and begin our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load(clas_trial_prefix + 'third');\n",
    "learn.export('clas_model.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Analysis, Discussion, and Next Steps <a name=\"4\"></a>\n",
    "<a name=\"0\">[(... back to the top)](#top)</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's analyze our performance on the validation set. Note that the `TextClassificationInterpretation` class we're using currently only works for the AWD_LSTM model.\n",
    "\n",
    "Let's load our initial classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "clas_data_fn = 'data_clas.pkl'\n",
    "clas_model_fn = 'clas_model.pkl'\n",
    "bs = 512\n",
    "\n",
    "data = load_data(path, clas_data_fn, bs=bs)\n",
    "learn = load_learner(path, clas_model_fn)\n",
    "learn.data = data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's take a look at a few tweets that I made up to check that the predictions make sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dcato_000\\Anaconda3\\envs\\fastai-py3.7_1\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category positive, tensor(2), tensor([0.0014, 0.0013, 0.9973]))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"Thanks for everything!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dcato_000\\Anaconda3\\envs\\fastai-py3.7_1\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category neutral, tensor(1), tensor([0.1852, 0.6858, 0.1290]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"flying to #mco to visit my family\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dcato_000\\Anaconda3\\envs\\fastai-py3.7_1\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category negative, tensor(0), tensor([0.9395, 0.0530, 0.0075]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"I didn't like the flight at all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are all correct, which is promising.\n",
    "\n",
    "Next, let's get the model's predictions on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions, targets, and losses:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[torch.Size([2668, 3]), torch.Size([2668]), torch.Size([2668])]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_type = DatasetType.Valid\n",
    "\n",
    "preds = learn.get_preds(ds_type=ds_type, with_loss=True, ordered=True)\n",
    "# ^ note: when using `with_loss`, must set `ordered=True` or the losses will not correspond to the correct prediction\n",
    "print('predictions, targets, and losses:')\n",
    "[p.size() for p in preds]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `TextClassificationInterpretation` class has some basic convenience functions for interpreting the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "interp = TextClassificationInterpretation(learn, *preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from training that our final accuracy was 84.8% and average validation loss was 0.423.\n",
    "\n",
    "Our validation loss and accuracy metrics are useful as single numbers to improve on, but let's now explore how well we did at predicting each class.\n",
    "\n",
    "A more detailed way to characterize the model's performance is by using a confusion matrix. A confusion matrix compares predictions with their actual labels, each square corresponds to the number of examples in each bucket. Correct predictions are seen along the diagonal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARYAAAEmCAYAAACnN7/iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3gVZfrG8e+dUKQKCkhRwIIgsIgQijQFFRALKEoVxbKuupZdy+pPXRsW7L0XBEHEuiKgFCsgKFUFRUURRUC6IEWS8Pz+mEmMMYEAczIn5Plc17k4U87Mc0Jyn/edec+MzAznnItSStwFOOf2PB4szrnIebA45yLnweKci5wHi3Much4szrnIebC43SKpjKS3JP0q6ZXd2E5/SROirC0uktpL+jruOuIkH8dSPEjqB1wONAA2AHOB28xsym5udwBwCdDGzDJ2u9AkJ8mAema2MO5akpm3WIoBSZcDDwC3A/sBtYHHgO4RbL4O8E1xCJWCkFQi7hqSgpn5Yw9+AHsDvwGnb2ed0gTBszR8PACUDpcdDSwBrgBWAMuAs8NlNwNbgfRwH+cCNwHDc2y7LmBAiXB6IPA9QatpEdA/x/wpOV7XBpgB/Br+2ybHsg+AQcDUcDsTgCr5vLes+v+To/4eQDfgG2ANcG2O9VsC04B14bqPAKXCZR+F72Vj+H5759j+1cBy4IWseeFrDg730SycrgmsAo6O+3cjob93cRfgjwT/B0NXICPrDzufdW4BpgPVgKrAx8CgcNnR4etvAUqGf5CbgMrh8txBkm+wAOWA9UD9cFkNoFH4PDtYgH2AtcCA8HV9w+l9w+UfAN8BhwJlwunB+by3rPpvCOv/O7ASeBGoADQCtgAHhes3B1qH+60LfAX8K8f2DDgkj+3fSRDQZXIGS7jO38PtlAXGA/fE/XuR6Id3hfZ8+wKrbPtdlf7ALWa2wsxWErREBuRYnh4uTzezcQSf1vV3sZ5tQGNJZcxsmZnNz2OdE4BvzewFM8sws5HAAuCkHOsMMbNvzGwz8DLQdDv7TCc4npQOvARUAR40sw3h/ucDTQDMbJaZTQ/3+wPwJHBUAd7TjWb2e1jPn5jZ08C3wCcEYXrdDrZX5Hmw7PlWA1V20PevCSzOMb04nJe9jVzBtAkov7OFmNlGgu7DBcAySWMlNShAPVk11coxvXwn6lltZpnh86w//F9yLN+c9XpJh0oaI2m5pPUEx6WqbGfbACvNbMsO1nkaaAw8bGa/72DdIs+DZc83jaCp32M76ywlOAibpXY4b1dsJGjyZ6mec6GZjTez4wg+uRcQ/MHtqJ6smn7exZp2xuMEddUzs4rAtYB28JrtnlqVVJ7guNWzwE2S9omi0GTmwbKHM7NfCY4vPCqph6SykkpKOl7SXeFqI4HrJVWVVCVcf/gu7nIu0EFSbUl7A/+XtUDSfpJOllQO+J2gS5WZxzbGAYdK6iephKTeQENgzC7WtDMqEBwH+i1sTV2Ya/kvwEE7uc0HgVlmdh4wFnhit6tMch4sxYCZ3UcwhuV6ggOXPwEXA/8LV7kVmAl8DnwBzA7n7cq+JgKjwm3N4s9hkEJwdmkpwZmSo4CL8tjGauDEcN3VBGd0TjSzVbtS0066EuhHcLbpaYL3ktNNwFBJ6yT12tHGJHUnOIB+QTjrcqCZpP6RVZyEfICccy5y3mJxzkXOg8U5FzkPFudc5DxYnHOR8y9M7YBKlDGVqhB3GbE7vEHtuEtIGtrRqJZiZM7sWavMrGru+R4sO6BSFShdf4dnFfd47099MO4SkkbpEt7Qz1K2VEruEdKAd4WccwngweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Mucn7v5pg8cWN/ju/QmJVrNpB2+u0AXPePbpxzahtWrv0NgBsfGc34KV+S1qgOj/y3LxDckPy2J8Yx+v3PKV2qBJOe/RelSpWgRGoqb0yaw61PjIvtPUXtiUcfYuiQZ8GMM88+lwsvvoy1a9Zwzpl9+XHxYmrXqcOQF16iUuXKcZeacJmZmbRt3YKatWrx+v/e4uwzz2D2rJmULFmS5i1a8MhjT1KyZMm4y8xWZFsskipJuijHdE1Jr8ZZ08544a3pdP/no3+Z//Dw92ndZzCt+wxm/JQvAZj/3VLa9r+L1n0G0/2fj/Hw9X1JTU3h960ZdD3/IVr1HkyrPnfQuU1DWv6tbiG/k8T4cv48hg55lnc/msbkT2Yz/u2xfLfwW+6/9046HN2JWV8soMPRnbj/3jvjLrVQPPrwgzRocFj2dO++/Zg77ytmzPmcLZu3MOS5Z2Ks7q+KbLAAlYDsYDGzpWZ2Woz17JSps79jza+bCrTu5i3pZGZuA6B0qZKYWfayjZu3AlCyRColSqT+aVlR9s3XC2jRohVly5alRIkStG3XgTGj/8fbY96ib/8zAejb/0zGvTU65koTb8mSJbzz9jgGnnNu9ryux3dDEpJIa9GCn5csibHCv0pYsEiqK+krSU9Lmi9pgqQykg6W9I6kWZImS2oQrn+wpOmSZki6RdJv4fzykt6VNFvSF5K6h7sYDBwsaa6ku8P9zQtf84mkRjlq+UBSc0nlJD0X7mNOjm0ljQv6dODTUf/HEzf2p1KFMtnzWzSuw6xXr2PmK9dy6W0vZQdNSoqY/tI1/PjuYN6bvoAZ8xbHVXqkDmvYiI+nTmbN6tVs2rSJiePf5uclS1ix4heq16gBQPUaNVi5ckXMlSbef674N7fecScpKX/9c01PT+fFEcPp3KVrDJXlL9EtlnrAo2bWCFgH9ASeAi4xs+bAlcBj4boPAg+aWQtgaY5tbAFOMbNmQEfgXkkCrgG+M7OmZnZVrv2+BPQCkFQDqGlms4DrgPfCfXQE7pZULvJ3vYuefmUyDU+6iVZ9BrN81XoGX35q9rIZ8xbT/LTbaHfGXVx1TmdKlwoOj23bZrTuM5hDulxPWuM6NDy4RlzlR6p+g8O47PKrOOXErpzWvRuN/nY4JUqkxl1WoRs3dgxVq1WlWbPmeS6/7JKLaNe+PW3btS/kyrYv0cGyyMzmhs9nAXWBNsArkuYCTwJZfwlHAq+Ez1/MsQ0Bt0v6HJgE1AL228F+XwZOD5/3yrHdzsA14b4/APYCaud+saTzJc2UNNMyNhfgbUZjxZoNbNtmmBnPvT6VtMZ1/rLO14t+YePmrTQ6pOaf5v/622Y+mvktnds0LKxyE27AwHP4cNoMxk38gMqVK3PQwfWoVm0/li9bBsDyZcuoWrVazFUm1vSPpzJ2zFs0qHcgZ57Rlw/ff49zzhoAwG2DbmbVylXcefd9MVf5V4kOlt9zPM8E9gHWha2MrMdh+bw2S3+gKtDczJoCvxAEQr7M7GdgtaQmQG+CFgwEIdUzx75rm9lXebz+KTNLM7M0lSiTe3HCVK9SMft5906H8+V3wR9QnZr7kpoa/FfVrlGZQ+vux+Klq6lSuTx7lw/q26t0STq1qs/XP/xSaPUm2soVQTfnp59+ZMzo/3Farz50PeFERo4YBsDIEcM4/sST4iwx4W657Q4WLvqJBd8uYtjwkRzVsRPPDX2BIc89w6SJExg6/MU8u0hxK+zTzeuBRZJON7NXwi5NEzP7DJhO0FUaBfTJ8Zq9gRVmli6pI5D1Mb4BqLCdfb0E/AfY28y+COeNBy6RdImZmaQjzGxOdG+v4IbeMZD2zetRpVJ5Fr4ziEFPjKND83o0qb8/ZsbiZWu45NaRALQ54iCuPLsz6RmZbNtmXHb7KFav20jjejV5+pYBpKakkJIiXps4m7cnz4vj7STEmf1OZ+2aNZQoWZK773+ISpUr8+8rrubsAX0YPnQI+x9wAM8PHxV3mbG49J8XUrtOHY5u3waA7j1O4drrb4i5qj8oUWcRJNUFxphZ43D6SqA8MBR4nKALVBJ4ycxukVQPGE7QqhgLnG9mtSRVAd4K150LtAWON7MfJL0INAHeBh7Ntb/9gJ+BQWZ2czivDPAAQXdMwA9mduL23kdK2WpWun6vaH4oRdiyjx+Mu4SkUbpE8rUQ4lK2VMosM0vLPT9hwbKzJJUFNoctiT5AXzOL/ayNB0vAg+UPHix/yC9YkmnkbXPgkbB7tA44J+Z6nHO7KGmCxcwmA4fHXYdzbvd5m845FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5JLmFqvJqkmDA5j44f1xlxG7hct/i7uEpNGgZoW4S0h63mJxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHORy/di2pLeAiy/5WZ2ckIqcs4Vedu7Sv89hVaFc26Pkm+wmNmHhVmIc27PscP7CkmqB9wBNAT2yppvZgclsC7nXBFWkIO3Q4DHgQygIzAMeCGRRTnniraCBEsZM3sXkJktNrObgE6JLcs5V5QV5BarWySlAN9Kuhj4GaiW2LKcc0VZQVos/wLKApcCzYEBwFmJLMo5V7TtMFjMbIaZ/WZmS8zsbDM71cymF0ZxxcVlF/2dhgfVokOrptnzBg+6kaOObEbHtmmc3r0by5ctBcDMuPaqf9Py8MM46shmfD53TlxlR+7337dwVo9O9OvWll5dWvPk/bcDwXt+7J5B9OzUnNOPa8lLzz+R/ZpZ0yfT74R29OrSmvP7dIur9IRbt24dZ/Q9nWZNGtL88EZ8Mn0aa9as4eRunWnaqD4nd+vM2rVr4y4zm8zyHQMXrCC9Tx4D5cwsKY6zSKoLtDGzF3fhtb+ZWfntrdO0WXOb+GFic3Ta1MmUK1eei/9xNh99MheADevXU6FiRQCefvwRvv76K+554FEmjX+bZ558jJGvjWbWjE+5/urLeef9qQmtD2Dxqk0J34eZsXnTRsqWK09Gejrn9erKFTcMZtHCr5k1fTI33v04KSkprFm1kn2qVGXD+nWce1oXHhryKtVrHZA9P9Ea1KyQ8H3kdv65A2nTth0DzzmPrVu3smnTJu656w4qV96HK666mnvvvpN169Yy6LbBhVpXhb1SZ5lZWu75BekKXQlcFT7+C8wFZkZb3m6pC/TLa4GkghxDit2RbdtTqXLlP83LChWATZs2IgmAt8e9Ra++/ZFEWstW/PrrOn5ZvqxQ600USZQtF+R8RkY6GRnpSOK1Ec9x3iVXk5IS/Lpmhcc7b75Kxy4nUb3WAX+av6dZv349H0+ZzFlnnwtAqVKlqFSpEmPfGk3/M84EoP8ZZzJm9JtxlvknO/zDM7NZuWZNlbTbg+fClsbbwBSgDcFB4e5ATeBRoCqwCfi7mS2Q9DwwxsxeDV+f1doYDBwmaS4wFFgLnEAw5qacpJOBN4HKQEngejNLnv+B7bj9lv/y8sgRVKxYkdfHTgRg+dKl1Nz/gOx1atban2VLl7Jf9RpxlRmpzMxMBpx8FEsWL+L0M86jcdM0fv5xERPHvs4H48dQad8qXHnDndQ+8GB+XLSQjIwM/tH3BDZt/I0+Ay/ghFP7xv0WIvfDou+pUrUqF/z9HOZ98TlNj2jGXfc+wMoVv1C9RvD/Xr1GDVatXBFzpX/YYYtF0j45HlUkdQGqR7T/esCjZtYIWAf0BJ4CLjGz5gStpcd2sI1rgMlm1tTM7g/nHQmcFXbXtgCnmFkzgnE49yrr4z8fks6XNFPSzNWrVu3ym9td194wiLlffU/PXn159sngx5BX13UHb6dISU1N5cWxUxj78Xzmfz6LhV9/ydatWylVujTDRn9Aj95nMujqi4EghBbMm8sDz77Mw8+/zrMP383i7xfG/A6il5GRwdw5sznv/AuY+sksypUrx3133xl3WdtVkK7QLIKuzyxgGnAFcG5E+19kZnNz7KcuQevllbAF8iSwKx/FE81sTfhcwO2SPgcmAbWA/bb3YjN7yszSzCxt3ypVdmH30Tr19D6MHf0GADVq1WLpkp+yly39eUn2p9aepELFSjRv1Y5pH71Lteo16dQ1+M5rxy4n8e2C+QBUq16TIzscQ5my5ai0z74c0bIN3y6YF2fZCVGr1v7UqrU/LVq2AqD7KT2ZO3c2Vavtx/JlQTd4+bJlVKmaPKNAChIsh5nZQWZ2oJnVM7POwIyI9v97jueZwD7AurD1kfU4LFyekVVv2OIotZ3tbszxvD9Bt6q5mTUFfiHHVxOS1fcLv81+Pn7cGA45tD4AXY8/kZdHjsDMmPnpJ1SsuPce0w1au3oVG9avA2DLls18OvVD6h5Uj6OOO4GZH38EwOxPplD7wIMBOOq4bsyZMY2MjAy2bN7EvM9mUffgQ2OrP1H2q16dWvsfwDfffA3Ah++/R4PDGtLtxJMYMXwYACOGD+OEk5LnggMFObj5MdAs17xpecyLwnpgkaTTzeyVMECamNlnwA8E42heJjgWUzJ8zQZge4fp9wZWmFm6pI5AnQTUvVv+cfYZTJ3yEWtWr+LwBgfyn2tvYNKEt/nu229QSgoHHFCbux94FIBjuxzPpAnv0PLwwyhbtgwPPvZMzNVHZ9WK5dx01YVsy8xkmxnHdutB+2O60rRFa/77r/N58bnHKVuuHNcPfgiAAw+pT5ujjqVft7YoJYXuvQZwSP2GMb+LxLjn/gc5b+AAtm7dSt0DD+Txp55j27ZtnNW/Dy88/xz7H1CbYS+OirvMbPmebpZUnaDbMJzgrEtWR74i8ISZNditHQcHb8eYWeNw+kqgPMEB2McJukAlgZfM7BZJ+xEchE0B3iU4DlNeUkngHaAK8DzBwds0M7s43G4V4K1wW3OBtsDxZvZDspxuLgoK43RzURHH6eZkld/p5u0Fy1nAQCCNoOuTFSzrgaFm9npiSk0uHiwBD5Y/eLD8Ib9g2d71WIYCQyX1NLPXElqdc26PUpCDt80lVcqakFRZ0q0JrMk5V8QVJFiON7N1WRNmthbYc7+U4ZzbbQUJllRJpbMmJJUBSm9nfedcMVeQ083DgXclDQmnzyY4c+Occ3kqyHeF7gpHrR5LcGboHZJwLIhzLnkU9IZly4FtBN/lOQb4KmEVOeeKvO3dsOxQoA/QF1gNjCIY99KxkGpzzhVR2+sKLQAmAyeZ2UIASf8ulKqcc0Xa9rpCPQm6QO9LelrSMfwx+tY55/KVb7CY2Rtm1htoAHwA/BvYT9LjkjoXUn3OuSKoIBfT3mhmI8zsRGB/gi/yXZPwypxzRVZBzwoBYGZrzOzJZLmQtnMuOe1UsDjnXEF4sDjnIufB4pyLnAeLcy5yHizOuch5sDjnIufB4pyLnAeLcy5yHizOuch5sDjnIufB4pyLnAeLcy5yHizOuch5sDjnIleQ238UaykSZUv7j6nxAXvHXULSmPfTr3GXkPS8xeKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHi3Much4szrnIebA45yLnweKci5wHSxJqeOiBtGzWhCNbHEH7I1sA8Pprr5DWtDEV9kpl9qyZMVeYeFu2bKHdkS1p2exwmh3eiEE33wjAwAH9adKoPs2bNuYf551Denp6zJUmxu+/b+GsHp3o160tvbq05sn7bwfAzHjsnkH07NSc049ryUvPP/Gn183/bDatDtmHd8e9GUfZ2UrEuvddIOkCYJOZDZM0EJhgZkvDZc8A95nZl3HWGIVxE96jSpUq2dMNGzbmxVGvcenFF8RYVeEpXbo070x8j/Lly5Oenk6no9rRucvx9OnXnyHDhgNw1oB+DHn2Gc6/4MKYq41eqVKleXzEaMqWK09Gejrn9epKm6OPY9HCr/ll2RJemTSDlJQU1qxamf2azMxMHrnrRlq3PybGygNFLljMLGdEDwTmAUvDZefFUVNhaHDYYXGXUKgkUb58eQDS09PJSE9HEl2P75a9TlpaS37+eUlcJSaUJMqWC95/RkY6GRnB+39txHPc+sAzpKQEnY19qlTNfs2ooU/SscvJfPnF7FhqzqlQu0KS6kpaIGmopM8lvSqprKRjJM2R9IWk5ySVDtcfLOnLcN17wnk3SbpS0mlAGjBC0lxJZSR9IClN0oWS7sqx34GSHg6fnyHp0/A1T0pKLcyfQUEI0f2ELrRrncZzzzwVdzmxyczMpFXzptSuWY1Oxx5Hy1atspelp6czcsQLHNela4wVJlZmZib9TmhH5xb1aNW2I42bpvHzj4uYOPZ1zjz5aC49+zR+XPQdACuWL+WDCWPo2f+cmKsOxHGMpT7wlJk1AdYDlwPPA73N7G8EragLJe0DnAI0Cte9NedGzOxVYCbQ38yamtnmHItfBU7NMd0bGCXpsPB5WzNrCmQC/XMXKOl8STMlzVyVo6lZWCZ9MIWpn8zi9dHjeOqJx5gy+aNCryEZpKam8smsuSz8YQkzZ3zK/HnzspdddvFFtG3fgXbt2sdYYWKlpqby4tgpjP14PvM/n8XCr79k69atlCpdmmGjP6BH7zMZdPXFANw36P+45OqbSU1Njs/JOILlJzObGj4fDhwDLDKzb8J5Q4EOBKGzBXhG0qnApoLuwMxWAt9Lai1pX4IwmxruqzkwQ9LccPqgPF7/lJmlmVlalRxNzcJSo2ZNAKpVq8ZJ3Xswa8anhV5DMqlUqRIdjjqaCRPeAeC2QTezctVK7rrnvpgrKxwVKlaieat2TPvoXapVr0mnricD0LHLSXy7YD4AX30xh+suPYeT2/+N994ezZ03XsEHE8bEVnMcwWIFWsksA2gJvAb0AN7Zyf2MAnoBPYE3zMwAAUPDFk5TM6tvZjft5HYTauPGjWzYsCH7+XuTJtKwUeOYqyp8K1euZN26dQBs3ryZ996dRP36DRjy7DNMnDCeYcNHZh9n2BOtXb2KDeuD979ly2Y+nfohdQ+qx1HHncDMj4MW7OxPplD7wIMBePOjzxk9+QtGT/6CTsefzNU338vRnU+Mrf44Dt7WlnSkmU0D+gKTgH9IOsTMFgIDgA8llQfKmtk4SdOBhXlsawNQIZ/9vA5cBywGrg7nvQu8Kel+M1sRdrcqmNni6N7e7lnxyy/07RX04jIyMujVpy/HdenK6Dff4Mp/X8qqlSvp2eNEmjRpyptjdzZri47ly5bx93POIjMzk222jZ6n9aLbCSdSfq8S1K5Th6PbHQlA91NO5drrb4i52uitWrGcm666kG2ZmWwz49huPWh/TFeatmjNf/91Pi8+9zhly5Xj+sEPxV1qnhR8kBfSzqS6wDjgI6AN8C1BkBwJ3EMQdDOAC4F9gDeBvQhaGveY2VBJNwG/mdk9knoCtwObw228DVxpZjPD/Y0BGppZdndHUm/g/whaa+nAP81sen41N2ueZpOnzYjoJ1B0paYo7hKSxryffo27hKTR4qBKs8wsLff8OIJljJkVmba9B0vAg+UPHix/yC9Y9txOqnMuNoV6jMXMfgCKTGvFObdrvMXinIucB4tzLnIeLM65yHmwOOci58HinIucB4tzLnIeLM65yHmwOOci58HinIucB4tzLnIeLM65yHmwOOci58HinIucB4tzLnIeLM65yHmwOOci58HinIucB4tzLnIeLM65yHmwOOci58HinIucB4tzLnIeLM65yHmwOOci58HinItcod67uSiStBJYHHMZVYBVMdeQLPxnEUiWn0MdM6uae6YHSxEgaWZeN94ujvxnEUj2n4N3hZxzkfNgcc5FzoOlaHgq7gKSiP8sAkn9c/BjLM65yHmLxTkXOQ8W51zkPFicc5HzYHHORc6DJYlJKiOpftx1uOSgwBmSbgina0tqGXddefFgSVKSTgLmAu+E000ljY63qsIlaYOk9Xk8NkhaH3d9MXgMOBLoG05vAB6Nr5z8lYi7AJevm4CWwAcAZjZXUt34yil8ZlYh7hqSTCszayZpDoCZrZVUKu6i8uLBkrwyzOxXSXHXkTQkVQP2ypo2sx9jLCcO6ZJSAQOQVBXYFm9JefOuUPKaJ6kfkCqpnqSHgY/jLioOkk6W9C2wCPgQ+AF4O9ai4vEQ8AZQTdJtwBTg9nhLypuPvE1SksoC1wGdw1njgVvNbEt8VcVD0mdAJ2CSmR0hqSPQ18zOj7m0QiepAXAMIOBdM/sq5pLy5MGSpCQdYWZz4q4jGWRdIiAMmCPMbJukT80sKc+IJIqkB4FRZpb0LVc/xpK87pNUA3gFeMnM5sddUIzWSSoPfASMkLQCyIi5pjjMBq6XdChBl2iUmc2MuaY8eYsliUmqDvQCegMVCX6Rbo23qsInqRywmeCYYH9gb2CEma2OtbCYSNoH6An0AWqbWb2YS/oLD5YiQNLfgP8Avc0sKU8vJkp4FmS8mR0bdy3JIhwU1xvoAXxpZifFXNJf+FmhJCXpMEk3SZoHPEJwRmj/mMsqdGaWCWyStHfctcRN0p3h2bFbgPlA82QMFfBjLMlsCDAS6GxmS+MuJmZbgC8kTQQ2Zs00s0vjKykWi4AjzSwZLqK9Xd4VcklP0ll5zDYzG1boxcRAUgMzWyCpWV7LzWx2Yde0I95iSTKSXjazXpK+IBxhmbWI4I+pSUylxamSmT2Yc4aky+IqJgaXA+cD9+axzAjG+CQVb7EkGUk1zGyZpDp5LTezuO9xVOgkzTazZrnmzTGzI+KqKQ6S9so9QDKvecnAD94mGTNbFj69yMwW53wAF8VZW2GT1OQ4KW8AAAX8SURBVFfSW8CBkkbneLwPFMdTzXkNjEvKwXLeFUpexwFX55p3fB7z9mQfA8sI7vqXsxuwAfg8lopiEI5nqgWUkXQEQbcYgrFNZWMrbDs8WJKMpAsJWiYHScr5x1MBmBpPVfEIW2mLCa5BUpx1AQYSDDe4L8f8DcC1cRS0I36MJcmE4zUqA3cA1+RYtMHM1sRTVbwkbeCPA9mlgJLARjOrGF9VhU9STzN7Le46CsKDJcn5NUj+SlIPoKWZJeWnddQknWFmwyVdwZ/PFAJgZvfl8bJY+cHbJCXpJL8GSd7M7H8k4SnWBCoX/lueoEuc+5F0vMWSpPwaJH+QdGqOyRQgDTjKzIr7sZek5S2W5JUefns3RVKKmb0PNI27qJiclOPRheCgZfdYK4qBpLskVZRUUtK7klZJOiPuuvLiZ4WSl1+DJGRmZ8ddQ5LobGb/kXQKsAQ4HXgfGB5vWX/lLZbk1Z3gGiT/JrgFyHcEn9jFjqRDw0/oeeF0E0nXx11XDEqG/3YDRibzWUI/xuKSnqQPgauAJ7OG8UuaZ2aN462scEkaTHANls0Et4apBIwxs1axFpYHb7EkqXxu1vWTpDckHRR3fYWsrJl9mmtesesWmtk1BIMF08wsneASEkl5rMmPsSSv+4ClwIsEQ7j7ANWBr4HngKNjq6zwrZJ0MH/cT+c0gqH+xYqkksAAoEN4v6kPgSdiLSof3hVKUpI+yd3ElTTdzFpL+szMDo+rtsIWttCeAtoAawnG9vQvbt/0lvQMwXGWoeGsAUCmmZ0XX1V58xZL8tomqRfwajh9Wo5lxe3T4GeCK+q9D+wDrAfOIrhEY3HSItcHynvheKek48dYkld/gk+kFcAv4fMzJJUBLo6zsBi8SXBGLJ2ge/gbOS5RWYxkhl1CILsllxljPfnyrpBLesXxDFBeJB1D0HL7PpxVFzg7HDyZVLzFkqR87MaffBzeAqW4mwo8SXAj+G3h82mxVpQPb7EkKR+78QdJXwKHEBy0/Z1iev1fSS8THF8aEc7qC1Q2s9PjqypvfvA2eZU1s0/D04pZit3YjdDxcReQJOrnOnj7frIevPVgSV4+diNU3E4rb8ccSa3NbDqApFYk6VUFvSuUpHzshstN0ldAfSDrYl+1ga8IjrckVdfQgyVJSSpNMHalLn+M3TAzK25jN1wov1vCZEmmDx3vCiWvN4F1wGyCsRuumEum4NgRb7EkqeJ6BsjtGXwcS/LysRuuyPIWS5LysRuuKPNgSVJ+72ZXlHmwOOci58dYnHOR82BxzkXOg8VFTlKmpLmS5kl6RVLZ3djW0ZLGhM9PlnTNdtatJOmiXdjHTZKu3NUa3V95sLhE2GxmTcNxOFuBC3IuVGCnf/fMbLSZDd7OKpWAnQ4WFz0PFpdok4FDJNWV9JWkxwhGEx8gqbOkaZJmhy2b8gCSukpaIGkKkH17VUkDJT0SPt8vvGPBZ+GjDTAYODhsLd0drneVpBmSPpd0c45tXSfpa0mTCL5/4yLkweISRlIJgksefBHOqg8MC68vsxG4HjjWzJoBM4HLJe0FPE1wKcr2BHcmyMtDwIfhZQSaAfOBa4DvwtbSVZI6A/UI7sHTFGguqYOk5gR3PTiCILhaRPzWiz3/rpBLhDKS5obPJwPPAjWBxVlf+QdaAw2BqeE1Z0oRXA2tAbDIzL4FkDQcOD+PfXQCzgQws0zgV0mVc63TOXzMCafLEwRNBeANM9sU7mP0br1b9xceLC4RNpvZn25gH4ZHzgtgC5hoZn1zrdeU6O5CIOAOM3sy1z7+FeE+XB68K+TiMh1oK+kQAEllJR0KLAAOzHE1+r75vP5d4MLwtamSKgIbCFojWcYD5+Q4dlNLUjXgI+AUSWUkVaCY3hM7kTxYXCzMbCUwEBgp6XOCoGlgZlsIuj5jw4O3+X2F4TKgo6QvgFlAIzNbTdC1mifpbjObQHAnyWnheq8CFcxsNjAKmAu8RtBdcxHyIf3Ouch5i8U5FzkPFudc5DxYnHOR82BxzkXOg8U5FzkPFudc5DxYnHOR+3/vFiLOYr0cLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that the model has the toughest time distinguishing between negative and neutral texts. Additionally, we noted during data exporation that we might have to account for the fact that the negative class several times more examples than the other 2 classes. It appears that this is not an issue. In fact, as a percentage of the total, there are far more neutral examples misclassified than either of the other 2 classes. It is clear that the model has the most difficulty distinguishing between the negative and neutral classes.\n",
    "\n",
    "Next let's look at the top 5 validation losses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_8.0 @virginamerica has getaway deals through xxmaj may , from $ 59 one - way . xxmaj lots of cool cities http : / / t.co / xxunk # cheapflights # farecompare</td>\n",
       "      <td>positive</td>\n",
       "      <td>neutral</td>\n",
       "      <td>9.06</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_8.0 @virginamerica has getaway deals through xxmaj may , from $ 59 one - way . xxmaj lots of cool cities http : / / t.co / xxunk # cheapflights # farecompare</td>\n",
       "      <td>positive</td>\n",
       "      <td>neutral</td>\n",
       "      <td>8.72</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_75_130 xxretweetcount_0 xxtweet_createdlocalhour_9.0 @jetblue 's xxup ceo battles to appease passengers and xxmaj wall xxmaj street - http : / / t.co / xxunk http : / / t.co / xxunk</td>\n",
       "      <td>neutral</td>\n",
       "      <td>positive</td>\n",
       "      <td>8.09</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_14.0 xxmaj xxunk @kciairport “ @southwestair : xxmaj inclement weather may impact scheduled service in . reflight xxmaj booking xxmaj problems options available : http : / / t.co / xxunk ”</td>\n",
       "      <td>neutral</td>\n",
       "      <td>negative</td>\n",
       "      <td>6.68</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxtextlencount_&gt;130 xxretweetcount_0 xxtweet_createdlocalhour_7.0 @southwestair - xxmaj we left ipad in a seat pocket . xxmaj filed lost item report . xxmaj received it exactly 1 week xxmaj late xxmaj flightr . xxmaj is that a record ? # unbelievable</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>6.58</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading through these, it seems reasonable that these 5 top losses are some of the toughest examples to predict as several of these are misleading in that they initially seem to be one class, but on finishing reading the example, belong to a different class. And actually I think example 5 here is mislabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "\n",
    "Where to go from here depends on both our goal and our constraints. We started with the fairly open-ended goal of applying the ULMFiT modeling process to the Twitter US Airlines Sentiment data. \n",
    "\n",
    "By further clarifying the goal(s) and taking into account any operational constraints, we could make a clearer decision on which steps to take next. Here are a 2 possibilities:\n",
    "* Goal: maximize the accuracy (or other performance metric)\n",
    "  * To achieve academic performance, we could take steps mentioned above to find out where we should spend our time to improve the model.\n",
    "* Goal: deploy the model\n",
    "  * Some likely constraints: maximum inference time threshold + minimum accuracy threshold, variety of models depending on the power of devices (e.g. must run on mobile)\n",
    "\n",
    "If we decided that our model needed further improvements, then the next step would be to manually review a sample our validation losses and sort them according to what kinds of errors the model seems to be making. This should help us determine where the model has the most room for improvement. Then, by estimating how much effort it would take to improve each issue, we could make the best use of our time in achieving better results.\n",
    "\n",
    "One weakness of this model is the loss of natural ordering of numbers. Since all inputs are treated as tokens, we binned variables such as 'text_len' and 'retweet_count' to preserve some meaning without creating a lot of extra tokens. It would be better to concatenate these features to the model, treating them as numbers. Sticking with the fastai library, it would be straightforward to implement this as a callback.\n",
    "\n",
    "Since this is only the first model we've created for the sentiment classification task, I am not going to run the data to see the accuracy on the test set. This should only be done after we clarify our goal and build multiple candidate models. Only at this point would we measure the performance of our models over the test set. We would then use these results to choose the best model to operationalize."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}