4 AutoML Forecast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. AutoML Forecast with Azure ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/retkowsky/images/blob/master/AzureMLservicebanniere.png?raw=true'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) \\n[GCC 7.3.0]'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-09-22 12:18:55.070115\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version Azure ML : 1.13.0\n"
     ]
    }
   ],
   "source": [
    "#Version\n",
    "import azureml.core\n",
    "print(\"Version Azure ML :\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "import os\n",
    "\n",
    "# Squash warning messages for cleaner output in the notebook\n",
    "warnings.showwarning = lambda *args, **kwargs: None\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core import Experiment, Workspace, Dataset\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "subscription_id = os.environ.get(\"SUBSCRIPTION_ID\", \"70b8f39e-8863-49f7-b6ba-34a80799550c\") #ID soubscription Azure\n",
    "resource_group = os.environ.get(\"RESOURCE_GROUP\", \"workshopAML2020-rg\") #Resource group\n",
    "workspace_name = os.environ.get(\"WORKSPACE_NAME\", \"workshopAML2020\") #Nom workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Workspace</th>\n",
       "      <td>workshopAML2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resource Group</th>\n",
       "      <td>workshopAML2020-rg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Location</th>\n",
       "      <td>westeurope</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Run History Name</th>\n",
       "      <td>Exemple4-AutoMLtimeseries</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           \n",
       "Workspace         workshopAML2020          \n",
       "Resource Group    workshopAML2020-rg       \n",
       "Location          westeurope               \n",
       "Run History Name  Exemple4-AutoMLtimeseries"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# choose a name for the run history container in the workspace\n",
    "experiment_name = 'Exemple4-AutoMLtimeseries'\n",
    "\n",
    "# # project folder\n",
    "# project_folder = './sample_projects/automl-forecasting-energy-demand'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Run History Name'] = experiment_name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "outputDf = pd.DataFrame(data = output, index = [''])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data\n",
    "\n",
    "We will use energy consumption [data from New York City](http://mis.nyiso.com/public/P-58Blist.htm) for model training. The data is stored in a tabular format and includes energy demand and basic weather data at an hourly frequency. \n",
    "\n",
    "With Azure Machine Learning datasets you can keep a single copy of data in your storage, easily access data during model training, share data and collaborate with other users. Below, we will upload the datatset and create a [tabular dataset](https://docs.microsoft.com/bs-latn-ba/azure/machine-learning/service/how-to-create-register-datasets#dataset-types) to be used training and prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set up what we know about the dataset.\n",
    "\n",
    "<b>Target column</b> is what we want to forecast.<br></br>\n",
    "<b>Time column</b> is the time axis along which to predict.\n",
    "\n",
    "The other columns, \"temp\" and \"precip\", are implicitly designated as features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = 'demand'\n",
    "time_column_name = 'timeStamp'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>precip</th>\n",
       "      <th>temp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>4937.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>46.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>4752.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>45.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>4542.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>45.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>4357.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>45.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>4275.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timeStamp  demand  precip   temp\n",
       "0 2012-01-01 00:00:00  4937.5  0.0     46.13\n",
       "1 2012-01-01 01:00:00  4752.1  0.0     45.89\n",
       "2 2012-01-01 02:00:00  4542.6  0.0     45.04\n",
       "3 2012-01-01 03:00:00  4357.7  0.0     45.03\n",
       "4 2012-01-01 04:00:00  4275.5  0.0     42.61"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = Dataset.Tabular.from_delimited_files(path = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/nyc_energy.csv\").with_timestamp_columns(fine_grain_timestamp=time_column_name) \n",
    "dataset.take(5).to_pandas_dataframe().reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **NYC Energy dataset** is missing energy demand values for all datetimes later than August 10th, 2017 5AM. Below, we trim the rows containing these missing values from the end of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cut off the end of the dataset due to large number of nan values\n",
    "dataset = dataset.time_before(datetime(2017, 10, 10, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the data into train and test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first split we make is into train and test sets. Note that we are splitting on time. Data before and including August 8th, 2017 5AM will be used for training, and data after will be used for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>precip</th>\n",
       "      <th>temp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>49071</th>\n",
       "      <td>2017-08-08 01:00:00</td>\n",
       "      <td>5106.350</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49072</th>\n",
       "      <td>2017-08-08 02:00:00</td>\n",
       "      <td>4947.733</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49073</th>\n",
       "      <td>2017-08-08 03:00:00</td>\n",
       "      <td>4867.017</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49074</th>\n",
       "      <td>2017-08-08 04:00:00</td>\n",
       "      <td>4888.200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49075</th>\n",
       "      <td>2017-08-08 05:00:00</td>\n",
       "      <td>5120.308</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67.53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timeStamp    demand  precip   temp\n",
       "49071 2017-08-08 01:00:00  5106.350  0.0     68.30\n",
       "49072 2017-08-08 02:00:00  4947.733  0.0     68.44\n",
       "49073 2017-08-08 03:00:00  4867.017  0.0     68.78\n",
       "49074 2017-08-08 04:00:00  4888.200  0.0     68.70\n",
       "49075 2017-08-08 05:00:00  5120.308  0.0     67.53"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split into train based on time\n",
    "train = dataset.time_before(datetime(2017, 8, 8, 5), include_boundary=True)\n",
    "train.to_pandas_dataframe().reset_index(drop=True).sort_values(time_column_name).tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>precip</th>\n",
       "      <th>temp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-08-08 06:00:00</td>\n",
       "      <td>5590.992</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-08-08 07:00:00</td>\n",
       "      <td>6147.033</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-08-08 08:00:00</td>\n",
       "      <td>6592.425</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-08-08 09:00:00</td>\n",
       "      <td>6874.533</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-08-08 10:00:00</td>\n",
       "      <td>7010.542</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68.30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timeStamp    demand  precip   temp\n",
       "0 2017-08-08 06:00:00  5590.992  0.0     66.17\n",
       "1 2017-08-08 07:00:00  6147.033  0.0     66.29\n",
       "2 2017-08-08 08:00:00  6592.425  0.0     66.72\n",
       "3 2017-08-08 09:00:00  6874.533  0.0     67.37\n",
       "4 2017-08-08 10:00:00  7010.542  0.0     68.30"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split into test based on time\n",
    "test = dataset.time_between(datetime(2017, 8, 8, 6), datetime(2017, 8, 10, 5))\n",
    "test.to_pandas_dataframe().reset_index(drop=True).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting the maximum forecast horizon\n",
    "\n",
    "The forecast horizon is the number of periods into the future that the model should predict. It is generally recommend that users set forecast horizons to less than 100 time periods (i.e. less than 100 hours in the NYC energy example). Furthermore, **AutoML's memory use and computation time increase in proportion to the length of the horizon**, so consider carefully how this value is set. If a long horizon forecast really is necessary, consider aggregating the series to a coarser time scale. \n",
    "\n",
    "Learn more about forecast horizons in our [Auto-train a time-series forecast model](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-auto-train-forecast#configure-and-run-experiment) guide.\n",
    "\n",
    "In this example, we set the horizon to 48 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_horizon = 48"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "Instantiate an AutoMLConfig object. This config defines the settings and data used to run the experiment. We can provide extra configurations within 'automl_settings', for this forecasting task we add the name of the time column and the maximum forecast horizon.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**task**|forecasting|\n",
    "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
    "|**blocked_models**|Models in blocked_models won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
    "|**experiment_timeout_hours**|Maximum amount of time in hours that the experiment take before it terminates.|\n",
    "|**training_data**|The training data to be used within the experiment.|\n",
    "|**label_column_name**|The name of the label column.|\n",
    "|**compute_target**|The remote compute for training.|\n",
    "|**n_cross_validations**|Number of cross validation splits. Rolling Origin Validation is used to split time-series in a temporally consistent way.|\n",
    "|**enable_early_stopping**|Flag to enble early termination if the score is not improving in the short term.|\n",
    "|**time_column_name**|The name of your time column.|\n",
    "|**max_horizon**|The number of periods out you would like to predict past your training data. Periods are inferred from your data.|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    'time_column_name': time_column_name,\n",
    "    'max_horizon': max_horizon,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(task='forecasting',\n",
    "                             debug_log='automl.log',\n",
    "                             primary_metric='normalized_root_mean_squared_error',\n",
    "                             blocked_models = ['ExtremeRandomTrees', 'AutoArima', 'Prophet'],                             \n",
    "                             experiment_timeout_hours=0.25,\n",
    "                             iterations=10,\n",
    "                             enable_voting_ensemble=False, # Pas de Voting Ensemble\n",
    "                             enable_stack_ensemble=False, # Pas de Stack Ensemble\n",
    "                             training_data=train,\n",
    "                             label_column_name=target_column_name,\n",
    "                             enable_early_stopping=True,\n",
    "                             n_cross_validations=3,                             \n",
    "                             verbosity=logging.INFO,\n",
    "                            **automl_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while.\n",
    "One may specify `show_output = True` to print currently running iterations to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local machine\n",
      "Parent Run ID: AutoML_46a14286-b767-4a01-b6ce-aca9e8a1b433\n",
      "\n",
      "Current status: DatasetFeaturization. Beginning to featurize the dataset.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Converting non-string tag to string: (forecasting_target_lags: [0])\n",
      "WARNING - Converting non-string tag to string: (forecasting_target_rolling_window_size: 0)\n",
      "WARNING - Converting non-string tag to string: (forecasting_max_horizon: 48)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the dataset.\n",
      "Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Frequency detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  The time series was analyzed, all data points are aligned with detected frequency.\n",
      "              \n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       DONE\n",
      "DESCRIPTION:  If the missing values are expected, let the run complete. Otherwise cancel the current run and use a script to customize the handling of missing feature values that may be more appropriate based on the data type and business requirement.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "DETAILS:      \n",
      "+---------------------------------+---------------------------------+---------------------------------+\n",
      "|Column name                      |Missing value count              |Imputation type                  |\n",
      "+=================================+=================================+=================================+\n",
      "|precip                           |230                              |median                           |\n",
      "|temp                             |186                              |median                           |\n",
      "+---------------------------------+---------------------------------+---------------------------------+\n",
      "\n",
      "****************************************************************************************************\n",
      "Current status: ModelSelection. Beginning model selection.\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   MaxAbsScaler DecisionTree                      0:00:31       0.0538    0.0538\n",
      "         1   MinMaxScaler DecisionTree                      0:00:24       0.0577    0.0538\n",
      "         2   StandardScalerWrapper LassoLars                0:00:24       0.1367    0.0538\n",
      "         3   RobustScaler ElasticNet                        0:00:25       0.0895    0.0538\n",
      "         4   StandardScalerWrapper LassoLars                0:00:24       0.1367    0.0538\n",
      "         5   RobustScaler ElasticNet                        0:00:33       0.1347    0.0538\n",
      "         6   RobustScaler DecisionTree                      0:00:24       0.0634    0.0538\n",
      "         7   MinMaxScaler DecisionTree                      0:00:24       0.0473    0.0473\n",
      "         8   RobustScaler DecisionTree                      0:00:24       0.0896    0.0473\n",
      "         9   MinMaxScaler DecisionTree                      0:00:24       0.0653    0.0473\n",
      "****************************************************************************************************\n",
      "Current status: BestRunExplainModel. Best run model explanations started\n",
      "Current status: ModelExplanationDataSetSetup. Model explanations data setup completed\n",
      "Current status: PickSurrogateModel. Choosing LightGBM as the surrogate model for explanations\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features started\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features completed\n",
      "Current status: BestRunExplainModel. Best run model explanations completed\n",
      "****************************************************************************************************\n"
     ]
    }
   ],
   "source": [
    "local_run = experiment.submit(automl_config, show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>Exemple4-AutoMLtimeseries</td><td>AutoML_46a14286-b767-4a01-b6ce-aca9e8a1b433</td><td>automl</td><td>Completed</td><td><a href=\"https://ml.azure.com/experiments/Exemple4-AutoMLtimeseries/runs/AutoML_46a14286-b767-4a01-b6ce-aca9e8a1b433?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/workshopAML2020-rg/workspaces/workshopAML2020\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: Exemple4-AutoMLtimeseries,\n",
       "Id: AutoML_46a14286-b767-4a01-b6ce-aca9e8a1b433,\n",
       "Type: automl,\n",
       "Status: Completed)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "local_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve the Best Model\n",
    "Below we select the best model from all the training iterations using get_output method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "[('timeseriestransformer', TimeSeriesTransformer(featurization_config=None,\n",
       "           pipeline_type=<TimeSeriesPipelineType.FULL: 1>)), ('MinMaxScaler', MinMaxScaler(copy=True, feature_range=(0, 1))), ('DecisionTreeRegressor', DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=0.7,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=0.001953125,\n",
       "           min_samples_split=0.0012814223889440828,\n",
       "           min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "           splitter='best'))]"
      ],
      "text/plain": [
       "[('timeseriestransformer',\n",
       "  TimeSeriesTransformer(featurization_config=None,\n",
       "             pipeline_type=<TimeSeriesPipelineType.FULL: 1>)),\n",
       " ('MinMaxScaler', MinMaxScaler(copy=True, feature_range=(0, 1))),\n",
       " ('DecisionTreeRegressor',\n",
       "  DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=0.7,\n",
       "             max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "             min_impurity_split=None, min_samples_leaf=0.001953125,\n",
       "             min_samples_split=0.0012814223889440828,\n",
       "             min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "             splitter='best'))]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_run, fitted_model = local_run.get_output()\n",
    "fitted_model.steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Featurization\n",
    "You can access the engineered feature names generated in time-series featurization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "['precip', 'temp', 'precip_WASNULL', 'temp_WASNULL', 'year', 'half', 'quarter', 'month', 'day', 'hour', 'am_pm', 'hour12', 'wday', 'qday', 'week']"
      ],
      "text/plain": [
       "['precip',\n",
       " 'temp',\n",
       " 'precip_WASNULL',\n",
       " 'temp_WASNULL',\n",
       " 'year',\n",
       " 'half',\n",
       " 'quarter',\n",
       " 'month',\n",
       " 'day',\n",
       " 'hour',\n",
       " 'am_pm',\n",
       " 'hour12',\n",
       " 'wday',\n",
       " 'qday',\n",
       " 'week']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitted_model.named_steps['timeseriestransformer'].get_engineered_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View featurization summary\n",
    "You can also see what featurization steps were performed on different raw features in the user data. For each raw feature in the user data, the following information is displayed:\n",
    "\n",
    "+ Raw feature name\n",
    "+ Number of engineered features formed out of this raw feature\n",
    "+ Type detected\n",
    "+ If feature was dropped\n",
    "+ List of feature transformations for the raw feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RawFeatureName</th>\n",
       "      <th>TypeDetected</th>\n",
       "      <th>Dropped</th>\n",
       "      <th>EngineeredFeatureCount</th>\n",
       "      <th>Transformations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>precip</td>\n",
       "      <td>Numeric</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>[MedianImputer, ImputationMarker]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>temp</td>\n",
       "      <td>Numeric</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>[MedianImputer, ImputationMarker]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>timeStamp</td>\n",
       "      <td>DateTime</td>\n",
       "      <td>No</td>\n",
       "      <td>11</td>\n",
       "      <td>[DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  RawFeatureName TypeDetected Dropped  EngineeredFeatureCount  \\\n",
       "0  precip         Numeric      No      2                        \n",
       "1  temp           Numeric      No      2                        \n",
       "2  timeStamp      DateTime     No      11                       \n",
       "\n",
       "                                                                                                                                                                                                                           Transformations  \n",
       "0  [MedianImputer, ImputationMarker]                                                                                                                                                                                                        \n",
       "1  [MedianImputer, ImputationMarker]                                                                                                                                                                                                        \n",
       "2  [DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer]  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the featurization summary as a list of JSON\n",
    "featurization_summary = fitted_model.named_steps['timeseriestransformer'].get_featurization_summary()\n",
    "# View the featurization summary as a pandas dataframe\n",
    "pd.DataFrame.from_records(featurization_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forecasting\n",
    "\n",
    "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. First, we remove the target values from the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = test.to_pandas_dataframe().reset_index(drop=True)\n",
    "y_test = X_test.pop(target_column_name).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forecast Function\n",
    "For forecasting, we will use the forecast function instead of the predict function. Using the predict method would result in getting predictions for EVERY horizon the forecaster can predict at. This is useful when training and evaluating the performance of the forecaster at various horizons, but the level of detail is excessive for normal use. Forecast function also can handle more complicated scenarios, see the [forecast function notebook](../forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The featurized data, aligned to y, will also be returned.\n",
    "# This contains the assumptions that were made in the forecast\n",
    "# and helps align the forecast to the original data\n",
    "y_predictions, X_trans = fitted_model.forecast(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate\n",
    "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE).\n",
    "\n",
    "It is a good practice to always align the output explicitly to the input, as the count and order of the rows may have changed during transformations that span multiple rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from forecasting_helper import align_outputs\n",
    "\n",
    "df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Test data scores]\n",
      "\n",
      "median_absolute_error:   329.007\n",
      "r2_score:   0.706\n",
      "normalized_root_mean_squared_error:   0.160\n",
      "root_mean_squared_error:   505.682\n",
      "root_mean_squared_log_error:   0.068\n",
      "normalized_root_mean_squared_log_error:   0.140\n",
      "normalized_median_absolute_error:   0.104\n",
      "mean_absolute_error:   396.087\n",
      "explained_variance:   0.803\n",
      "normalized_mean_absolute_error:   0.126\n",
      "mean_absolute_percentage_error:   5.699\n",
      "spearman_correlation:   0.976\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3TcdZ3v8ec7P0o7IGmbgoelZIJ7OVxduiLtqV1goRC2ULhdwNVzWQcsoDdnW1bb9bhrPdGtoDnqdfcs6L2tJytobeeKivUKiLI1yp4rZxEKAlFwbYUkRJCWtER3A7RN3veP73fSSTIzmZnMZOY783qcM2cmn/l+v/P5Ttr395v39/N9f8zdERGR+tBQ6Q6IiMjcUdAXEakjCvoiInVEQV9EpI4o6IuI1JGmSncglyVLlnh7e3uluyEiEimPP/74K+5+Sqb3qjrot7e3s3fv3kp3Q0QkUsxsINt7Su+IiNQRBX0RkTpS1emdqY4ePcrQ0BCvv/56pbtSNebPn8/SpUtpbm6udFdEJAIiFfSHhoZ405veRHt7O2ZW6e5UnLszPDzM0NAQZ555ZqW7IyIREKn0zuuvv05ra6sCfsjMaG1t1V8+IpK3SAV9QAF/Cn0fItGRTEJ7OzQ0BM/J5PT2JUuCx9RlSiVyQT8qVq9eDcDmzZsZGxvLuMxdd9018fqDH/zgXHRLRCokmYTOThgYAPfgubMTNm6c3D48HDzSlyll4K/JoJ/taFoq4+PjeS97++2309jYmPG99KD/xS9+cdb9EpHq1dUFo6OT20ZHoadnevvUZbq6StePmgv62Y6mhQb+hx56iHXr1rF27VouvfRSnnjiCS655BLe/e5389WvfpX777+fiy66iPPPP58f/OAHAPT09LBq1Sq2bNkysZ3Vq1dz7Ngxfvvb37J27VpWr17Nxz72MXp6eujr62P16tX09fVx4YUXAvDUU09xwQUXsGrVKnbt2gXAjTfeyKZNm7jwwgu59dZbS/NFicicGhzM3J4lEZDXukVx96p9LF++3NM988wzPpN43D0I95Mf8fiMq07y4x//2C+77DJ3d7/77rv9M5/5jL/tbW/zY8eO+djYmF988cV+9OhRf+ONN/zSSy/1o0eP+sqVK/3o0aP+8MMP+8UXX+zuPrHchz70IX/wwQfd3X1sbMzd3S+44IKJz0u9XrdunT///PN+5MgRX7lypR85csTXr1/vu3fvdnf3lStXTutrPt+LiFRWttjU2Ji5fTbxC9jrWeJqXmf6ZrbJzH5uZr8ws81h22Iz22Nm+8LnRWG7mdkXzGy/mT1tZuelbWd9uPw+M1tfwmPXhGxHxGKOlO94xzsAOPfcc/nhD3/I29/+dhobG3nllVd49tlnueyyy1izZg0vvfQSBw8eJB6P09TUxPLly6dt61e/+hXnn38+AA0N2b/2w4cP097eTnNzM2eeeSYHDhwA4JxzzgFgwYIFhe+IiFRcdzfEYpPbYrEgEzG1feoy3d2l68eMQd/MzgH+B7ASeDvw38zsLGAL0OvuZwG94c8Aa4GzwkcnsD3czmJgK/DOcFtbUweKUmprK6w9l6eeemriuaOjYyJYL1myhGXLltHb28tDDz3EU089xSmnnMLAwABjY2P87Gc/m7ats88+m0ceeQQ4fk0g08ibhQsX0t/fz9GjR3nuuec49dRTsy4rItGRSAT5+3gczILnnh7Ytm1ye2tr8EhfJpEoXT/yOdN/K/CIu4+6+zHgX4FrgauBHeEyO4BrwtdXA18L/8p4BFhoZqcBlwN73P2Qux8G9gBXlG5XAtmOpsUcKZubm7niiivYtm0ba9asmWhvaGjgwx/+MB0dHVxyySVs3ryZpqYmbrrpJs4//3x27949bVtbtmzh85//PKtXr+bjH/84AGeccQZ/8Rd/wS9/+cuJ5W677Tbe+973cuGFF3LLLbfoTluRGpJIQH8/jI8Hz6lgnt7+yivBY+oyJZMt75N6EAT9XwGtQAz4N+CLwKtTljscPt8PXJjW3gusAD4CfDyt/RPARzJ8XiewF9jb1tY2KU+Vb+56164gB2YWPO/aVVg+zD3I6Xd1dRW+YgUopy8i6ciR05+xDIO7P2tmnyM4M/8P4CngWI5VMuUhPEf71M/rAXoAVqxYMe39fCQSZTg6iojUgLwu5Lr7ne5+nrtfBBwC9gEvh2kbwucD4eJDwBlpqy8FXszRXpVWr17Npz/96Up3Q0SkpPIdvXNq+NwGvAv4OnAvkBqBsx74bvj6XuB94SieVcCIu78EPAisMbNF4QXcNWGbiIjMkXyrbH7bzFqBo8At7n7YzD4LfNPM3g8MAu8Jl30AuBLYD4wCNwG4+yEz+xTwWLjcbe5+qET7ISIiecgr6Lv7n2ZoGwY6MrQ7cEuW7dwF3JXpPRERKb+aK8NQCv39/fzoRz/Ka9mvfvWrE+Pub7zxRvbv31/OronUjXLX0KpXNRn0k31J2m9vp+HWBtpvbyfZV9i/lkxBP1uRtfSgLyKlUaoaWjJdzQX9ZF+Szvs6GRgZwHEGRgbovK+zoMDf09PDzp07J26+ShVZSxVFg2B0z6OPPsqTTz5JR0cHO3fuBIJqmSqMJjI72SpSlrLaZL2quaDf1dvF6NHJ/1pGj47S1Zv/v5bOzk5uuOEG7rzzTg4cOMA3vvENbr755mnLrVy5knPPPZfe3l5uuOEGIDgY/OQnP+GBBx6Y3Y6I1LFS1tCSyWou6A+OZP5Xka19Jqkia+mCa9WZqTCayOyVsoaWTFZzQb+tJfO/imztmTQ3N0/MdpVeEdPdeeONN+jr68u4LKgwmkgplLKGlkxWc0G/u6ObWPPkfy2x5hjdHfn/aznnnHN4+OGH+ehHPzqp/cYbb+TCCy/kW9/61kTbVVddxTXXXMO3v/3t2XVcRCZkq0g5l+VVanX0kOVKVVTaihUrfO/evRM/P/vss7z1rW+dcb1kX5Ku3i4GRwZpa2mju6ObxLLaLcaT7/ciIvlJjR5Kv5gci839gadYZva4u6/I9F6+d+RGSmJZoqaDvIiUV67RQ1EI+rnUXHpHRGS2ann0UOSCfjWnoypB34dI6dXy6KFIBf358+czPDysQBdyd4aHh5k/f36luyJSU2p59FCkcvpLly5laGiIgwcPVrorVWP+/PksXbq00t0QqRrJZJB7HxwMzsy7u/PLw09db/16eOCB3Nsp9rMqKtuUWtXwWL58+SwnDRORerJrl3ss5h5U7AkesdjMU6YWs16xnzUXyDFdYqSGbIqI5NLeHhRnmyoeDyYZL+V6xX7WXMg1ZDNSOX0RkVyKHXVTzHpRHeGjoC8iNaPYUTfFrBfVET4K+iJSM4oddVPMelEd4aOgLyI1o9iaPcWsVw31gYqhC7kiIjVGF3JFRARQ0BcRqSsK+iIidURBX0Skjijoi4jUEQV9ESmLUkw3WKtTFlaSgr6IlFxqusGBgaAU2cBA8HM+QTsV6M3ghhsmb+P664MDgJkOAsVS0BeRkss13WAu6QcLCIL9VKm2Qg4kUZLsS9J+ezsNtzbQfns7yb7S7qBuzhKRkmtoyBywzWB8PPt62SpX5lINVS1nK9mXpKu3i4GRAQzDOf7lxZpj9KzrKWjeb92cJSJzqthiZMVUqKz2qpYzSfYl6byvk4GR4GiXHvABRo+O0tU7w59IBVDQF5GSK7YYWTEVKqu9quVMunq7GD06mnOZwZHSHdkU9EWk5IotRpbpYJFLFKpapsuUr88noLe1lO7Ippy+iFSVTPPOQtA2MACNjTA2FhxIIjEnbSiVxkk/q481x1jQtIDh14azrlfqnH5eQd/M/gb4AOBAH3ATcBpwN7AYeAK4wd2PmNkJwNeA5cAw8N/dvT/czseA9wNjwIfc/cFcn6ugLyK1ov329om8fbrWBa28duy1SQeD1MXceEuc7o7uggI+zPJCrpmdDnwIWOHu5wCNwHXA54B/cvezgMMEwZzw+bC7/xfgn8LlMLO3hev9EXAFsM3MGgvaExGRiMqWxjn02iF61vUQb4ljGPGWODvftRPf6vRv7i844M+kqYDlFpjZUSAGvARcCrw3fH8H8ElgO3B1+BrgHuB/mZmF7Xe7+xvA82a2H1gJ/Nvsd0NEpLq1tbRlPNNva2kjsSxR8uCezYxn+u7+G+AfgEGCYD8CPA686u7HwsWGgNPD16cDL4TrHguXb01vz7COiEjkFHIjVXdHN7HmyVepY80xujvm9kp0PumdRQRn6WcCfwCcCKzNsGjq4oBleS9b+9TP6zSzvWa29+DBgzN1T0SqUD3UzEkfX+84AyMDdN7XmTXwJ5YlpqVxCr1AWwr5pHcuA55394MAZrYbOB9YaGZN4dn8UuDFcPkh4AxgyMyagBbgUFp7Svo6E9y9B+iB4EJuMTslIpWTKqWQKsOQKpcA0Rlpk49M4+tTN1JlC+RzmcbJJp9x+oPAKjOLhbn5DuAZ4MfAu8Nl1gPfDV/fG/5M+P6PPBgidC9wnZmdYGZnAmcBj5ZmN0SkWhRbdydqsl2YLeWNVOWQT07/pwQXZJ8gGK7ZQHAm/lHgw+EF2VbgznCVO4HWsP3DwJZwO78AvklwwPgBcIu7j5V0b0Sk4rKVRYhauYSZ8vXZbpgq5Y1U5aCbs0SkpLIVTYtSYbRsN1Kl5+DzWaZSVHBNROZMsXV3Ki39zH79d9ZnzdenVMuF2ULlO05fRCQvqYu1U0spVPNF3Kln7WNZMs9T8/XVcGG2UAr6IlJyiUR1B/mp8ql0CdWfr8+H0jsiUvfyGXFTiRupykFBX0TqXrYz+EZrjFS+Ph8K+iJ1rlx3z0bprtxsJRJ2XLuD8a3jZSl8VikK+iJ1LH0icvfSTTZeru3m9dlFTCwe1ZE4xdA4fZE6Vq4x9ZUaq1/NY+fnksbpi0hG5bp7tlJ35eaqhyMBBX2ROpZtUvHFi8uz3XJPYh7VejhzSUFfpI51d0Nz8/T23/9+dvn3ct2VW6v1cOaSgr5IHUsk4OSTp7cfOTK7qpiJBPT0BDl8s+C5p6fwG7ZSQd5uNRpva+T63dfnrF9fLROVVDNdyBWpcw0NwQibqcxgfHzu+5Oy8Xsb+dLeL+HT51qaJN4Sp39z/8TPyb4kXb1dDI4M0tbSVtTE4lGX60KuyjCI1Lm2tswjbcqdf88m2Zdk0/c3MfzacF7L10I9nLmk9I5InStl/j3fG7KyLZcacplvwAfl6wulM32ROleqqpj5TpOYa7mug/kVPktRvr5wyumLSEnke0NWezsMnJyEji5oGYSRNujtJv67BIM3NcyYw09pXdDKHWvvUConA+X0RaTs8r0ha+DkJKzrhHnhGf3CAVjXycB9EG9pY2Akw5EjTbwlXpcXZ0tFOX0RKYl8b8hqvLzreMBPmTdK4+VdGYdcQnBWv+tdu/CtXlPFzypBQV9ESmLSBeFlSdhyEmw1Bm4Kxthv/N5GAMZOyvwnwdhJgxkLn+161y5e+btXFOhLROkdESmJRAIe/l2SLw1uwk8YBjv+3riPs33vdiB7CicejsLRkMvy0pm+SI2odP36ZF+SHYc78fmTA366nsd7dNdshSnoi9SActavz/dgks88s2M+Vle166uRhmyK1IB8h0smk4WNx586ph6CvH2mOjoNt8483LLRGjn298dm3B+ZHdXTF6lx+QyXLOavga6uyQEfgp8zFWPL587YzuWdMy4j5aWgL1ID8hkuWUgATxkcBNZuhE80wVYLntduzHiQyTbcEqDBGtiwYgPbrtqWe0ek7DR6R6QGdHdnTsOk188pZjar2Hs28p9v3X78wmzjGKzcTuxNAJMDeConX+8VLqudcvoiNWKmfH0x89Y23trEOGPT2htoZGyrcvPVSjl9kTqQSATBe3w8eJ56oTVXNc1sM1JlCvi52qX6Kb0jUieyVdN8eOFGvrT7+GQlqRmpABhvhIYMAX68cY56LaWmM32ROjL1rwH+OJlxdqrRo6N09XbBY51MG4XpBO0SSQr6IlWs1HfZTk3jbPr+pqxj6wdHBok/sw0e3QBjjUGwH2uERzcE7RJJSu+IVKl8JyXJe3vhrFSpu2ZnKmHc1tIWjgraxuj3jwf5WAy6ewr/fKkOOtMXqVLFjKvPub08yiSkGBYMt0wEd9/G48FE6fF45rtxJTpmDPpmdraZPZn2+J2ZbTazxWa2x8z2hc+LwuXNzL5gZvvN7GkzOy9tW+vD5feZ2fpy7phI1BUzrj4l02icqROIZ2MYf7XirybG1880KkiipaBx+mbWCPwGeCdwC3DI3T9rZluARe7+UTO7EvggcGW43B3u/k4zWwzsBVYQZAcfB5a7++Fsn6dx+lLPCh1Xn+xL0tXbxcDIAIZNytXHmmMsaFqQccLx1gWtnDTvJN1QVUNKOU6/A/i1uw8AVwM7wvYdwDXh66uBr3ngEWChmZ0GXA7scfdDYaDfA1xR4OeLVFypLq7OtJ1M4+rN4MorM2wrzNen8vSZRuMAGUsa37H2Dvo39zO+dVyzUtWBQoP+dcDXw9dvdveXAMLnU8P204EX0tYZCtuytU9iZp1mttfM9h48eLDA7omUV6lKGOeznUQC1q8PAn2KO+zYMf3z8snXD792SCWNJf/0jpnNA14E/sjdXzazV919Ydr7h919kZl9D/iMu/8kbO8F/g64FDjB3T8dtn8CGHX3f8z2mUrvSLUpppTBbLaTcbllSRov72L8pOPpmBt23zBzWeP/iHPs8wV0UiKrVOmdtcAT7v5y+PPLYdqG8PlA2D4EnJG23lKCg0W2dpHImM3F1WK2M225ZUlY18nYSQM4PnH37OIFi3N/4JEYYw9qZiopLOj/JcdTOwD3AqkROOuB76a1vy8cxbMKGAnTPw8Ca8xsUTjSZ03YJhIZ+ZQwLuV2Fl+chL9dEpQ13mpw7ftg3uQ0TrZ8PW7BkIlX43BfD/HfKY0jeQZ9M4sBfwbsTmv+LPBnZrYvfO+zYfsDwHPAfuCfgY0A7n4I+BTwWPi4LWwTiYxcRctKvZ1kX5KR1TfDieGcswY0jGfc3qG0fD0YNhKH3TvhVofb+4n9OlFwH6U2qbSySIEKnXKw2O20394+412zKfGWOP2b+0veR4mmXDl9BX2RKpXPnLMQpHU0CkfSqZ6+SBXKVsM+Jdecs43WqGGXUhQVXBOpgEzFz1I17FMBvLujm5u/ezNHxo5MWre5oZmvXPMVBXopis70RUKlLmOcS6abqSZq2IcSyxLcdfVdtC5onWhrXdDKB5Z8ha51iTnpp9QenemLUPoyxjPJVvxsantiWWLSGf1c91Nqj870RSh9GeOZZMvX58rjw9z3U2qPgr4IpbvTNl/dHd0Zi591d+QeTD/X/ZTao6AvQunutM1XYlmiqOJnc91PqT3K6YtAOC3g5NRJMXfaFmJqvj4flein1Bad6YtAQdMCbvzeRppua8JuNZpua2Lj9zZWZT9FMtEduSIF2Pi9jWzfu31a+4YVG9h21bYMa4jMPd2RK1IiPY/3FNQuUm0U9EUKMOZjBbWLVBsFfZHQTLVwIKh5k0m2dpFqo6AvwuSJxdNnpJoa+DuXd2ZcP1u7SLVR0Bchv1o4ANuu2saGFRsmzuwbrVEXcSVSNHpHhOy16w1jfGvm2apEqpVG74jMoNhaOCJRo6AvQvG1cESiRkFfhOJr4YhEjXL6IiI1Rjl9EREBFPRFROqKgr6ISB1R0JfIyKdMgojkpklUJBJSZRJSd82myiQAGmEjUgCd6UvV2rg9SdPftmOfbOD6e9bnVSZhLiST0N4ODQ3Bc1J/cEiEKOhLVZgaSC/7cJLtv+lk7KQBMIeGzKWLB0fmdkbwZDKYrnBgANyD585OBX6JDqV3pOKSSbjpn5IcvbYLWgYZGGljoOk/oHl0xnUXN81tmYSursnz00Lwc1eXpiyUaNCZvlTcpi8nOXp5JywMz+oXDkBseOYVj8Tgh3NbJmEwyx8W2dpFqo2CfpUpR7640jnomT5/+NwumDfl9NmybGysEdzg1Tjc18Ohf53b0+u2LH9YZGsXqTZK71SRVL44lT5I5Yuh+NRBObZZ8s9vyXKa7EwO/kdicF8P9B3veFu81D3Orbt78v4AxGJBu0gUqPZOFWlvD4LiVPE49PdXzzYL/vyTk9AR5OsZaYPebuK/S0x8/pLudoaPTe/kCeOtHBs9ibETB2n4fRv2o27Gnjwe8GMx6OmZ+1x6Mhnk8AcHgzP87m7l86W65Kq9o6BfRRoaghEhU5nBeJHzeJRjm4WwP07Cus7J6ZvwjN2fDiJlsi/Jzd/p5IgfX2aexbjr2slVLhVsRfIz64JrZrbQzO4xs1+a2bNm9idmttjM9pjZvvB5UbismdkXzGy/mT1tZuelbWd9uPw+M1tfmt2rHeXIF1c6B914eYZ8/bzRoD2UWJbgrmsnlzWeGvAhCPD9/cHBqr9fAV+kGPleyL0D+IG7/1fg7cCzwBag193PAnrDnwHWAmeFj05gO4CZLQa2Au8EVgJbUwcKCXR3BymLdLPNF5djm4UYOylzvn5qe2JZgv7N/YxvHad/c7/ushUpkxmDvpmdDFwE3Ang7kfc/VXgamBHuNgO4Jrw9dXA1zzwCLDQzE4DLgf2uPshdz8M7AGuKOneRFwiEeSo4/Eg/RKPzz5nXY5tFiKeZbrBbO0iUl75nOm/BTgIfMXMfmZmXzazE4E3u/tLAOHzqeHypwMvpK0/FLZla5/EzDrNbK+Z7T148GDBOxR15UhhVDItomkIRapLPkG/CTgP2O7u7wD+k+OpnEwyjbCeOvguvX1yg3uPu69w9xWnnHJKHt2TaqZpCEWqSz7j9IeAIXf/afjzPQRB/2UzO83dXwrTNwfSlj8jbf2lwIth++op7Q8V33WJisSyhIK8SJWY8Uzf3X8LvGBmZ4dNHcAzwL1AagTOeuC74et7gfeFo3hWASNh+udBYI2ZLQov4K4J20REZI7ke0fuB4Gkmc0DngNuIjhgfNPM3g8MAu8Jl30AuBLYD4yGy+Luh8zsU8Bj4XK3ufuhkuyFiIjkRTdniYjUmFnfnCUiIrVBQb/OaJ5ZkfqmKpt1RPPMiojO9Muk0jXsJ/UlPLu/fvf1VTPPrIhUhoJ+GZR7HtVCDiips/uBkQz1lUNzPc+siFSOgn4Z5JpHdbYKPaB09XZNO7ufqk11cETqhoJ+GZRzHtVsB5RNX858gXams3jVwRGpL7qQWwZtbZlnqypFDfuMB45lSYbP72R4ZPoF2raWtqypnXhLnO6Obl3EFakjOtMvg3LWsG9rA5YlYXM7bG0Inq/YNG2iktQF2mxVLne9a5fq1ovUIQX9MihnDfsrP5qEP++EhQNgHjzHhjMuOzgyqCqXIjKJyjBETPvt7TlH4qSLt8Tp39xf3g6JSNVRGYaIynT3bL7DK3WBVkQy0YXcKpPsS9LV28XAyACG4eE8M6mLs4sXLGb4tenpnNYFrZw07yQGRwZpa2nTBVoRyUhBv4pMLZPgUyYWGz06yoKmBcSaY5PG3seaY9yx9g4FeRGZkdI7VSSfG6kOvXZIF2ZFpGg606+QVBonPR2TT76+raVN0w+KSNEU9CsgW7XLbPn6FF2cFZHZUnqnAjKlcVI/T72RyjAApXFEpCQU9Msk12Ql2dI4mfL1O9+1E9/quntWREpC6Z0SS/Yl2fT9TZPSNFMnK8lWD0f5ehEpt5o/0y92MpNi1kv2Jbn5O50Z8/Lpk5V0d3QzzyancebZ9Hx9QXXz81y2miZ3EZG5V9Nn+qna86lSxKna85C7Dk6x6226t4sjnn3I5URa5+kEfi/wp13QMggjbfj/64Y/TMCywvuQ77LF7peI1I6arr3T3p65xHE8Dv39pV/PPtkQFEHLIlULJ5/tF9KHfJctdr9EJFrqtvZOsZOZFD0Jykj2gvnpwy3z2X4hfch32XJO7iIi0VDTQT/bpCUzTWZS7HqtT3bDkSmF9B3s9dZJwy3z2X4hfch32WL3S0RqR00G/dTFyoGBoJ59unwmM+nuhoZ1G+ETTbDV4BNNNKzbOON6d3wgQfODPfBqHNzg1TjN9+9i5zmvTBqRk88kK4VMxJLvsuWc3EVEIsLdq/axfPlyL9SuXe6xmHswbXjwMAue4/Hg/ZlsuH+D80mmPTbcvyGvz4/Hg8/M9Xn5LJfvtkr9uSISbcBezxJXa+5CbikuVjbd1sSYj01rb7RGjv39sYL6IyIy1+rqQm4pLlZmCvi52kVEoqLmgn4pLlY2WmNB7SIiUVFzQb8UFys7l3cW1C4iEhU1F/QTCejpCXL4ZsFzT09hd5xuu2obG1ZsmDizb7RGNqzYwLartpWp1yIic6PmLuROlWmyEhU0E5FalutCbk3W3plpcnFAgV9E6lJe6R0z6zezPjN70sz2hm2LzWyPme0LnxeF7WZmXzCz/Wb2tJmdl7ad9eHy+8xsfTl2KDUrVap0cabJxVPVLkVE6k0hOf1L3P3ctD8ZtgC97n4W0Bv+DLAWOCt8dALbIThIAFuBdwIrga2pA0Up5TO5eD5z0YqI1KLZXMi9GtgRvt4BXJPW/rXwxrBHgIVmdhpwObDH3Q+5+2FgD3DFLD4/o3wnFxcRqUf5Bn0H/sXMHjez1LjFN7v7SwDh86lh++nAC2nrDoVt2donMbNOM9trZnsPHjyY/56EZgromlxcROpZvkH/Anc/jyB1c4uZXZRjWcvQ5jnaJze497j7Cndfccopp+TZveO6O7o1ubiISBZ5jd5x9xfD5wNm9h2CnPzLZnaau78Upm8OhIsPAWekrb4UeDFsXz2l/aFZ9T6DVEDXME0RkelmHKdvZicCDe7++/D1HuA2oAMYdvfPmtkWYLG7/52ZXQX8NXAlwUXbL7j7yvBC7uNAajTPE8Bydz+U7bNLMU5fRKTezHac/puB71hQmL4J+D/u/gMzewz4ppm9HxgE3hMu/wBBwN8PjAI3Abj7ITP7FPBYuNxtuQK+iIiUXs3fkSsiUm/qqrSyiIhkp6AvIlJHFPRFROqIgr6ISB1R0BcRqSMK+iIidURBX0Skjijoi4jUEQV9EZE6oqAvIlJHFPRFROqIgr6ISB1R0BcRqSMK+lkkk9DeDlsYh7IAAAWKSURBVA0NwXMyWekeiYjMXl4zZ9WbZBI6O2F0NPh5YCD4GSChCbhEJMJ0pp9BV9fxgJ8yOhq0i4hEmYJ+BoODhbWLiESFgn4GbW2FtYuIRIWCfgbd3RCLTW6LxYJ2EZEoU9DPIJGAnh6Ix8EseO7p0UVcEYk+jd7JIpFQkBeR2qMzfRGROqKgLyJSRxT0RUTqiIK+iEgdUdAXEakj5u6V7kNWZnYQGChy9SXAKyXszlxT/ysr6v2H6O+D+l+8uLufkumNqg76s2Fme919RaX7USz1v7Ki3n+I/j6o/+Wh9I6ISB1R0BcRqSO1HPR7Kt2BWVL/Kyvq/Yfo74P6XwY1m9MXEZHpavlMX0REplDQFxGpI5EK+mbWb2Z9Zvakme0N2xab2R4z2xc+Lwrbzcy+YGb7zexpMzsvbTvrw+X3mdn6Cvf/k2b2m7DtSTO7Mm35j4X9/3czuzyt/Yqwbb+ZbZnD/i80s3vM7Jdm9qyZ/UmUvv8c+xCJ34GZnZ3WxyfN7Hdmtjkqv4Mc/Y/E9x9+7t+Y2S/M7Odm9nUzm29mZ5rZT8Pv8htmNi9c9oTw5/3h++0z7deccPfIPIB+YMmUtv8JbAlfbwE+F76+Evg+YMAq4Kdh+2LgufB5Ufh6UQX7/0ngIxmWfRvwFHACcCbwa6AxfPwaeAswL1zmbXPU/x3AB8LX84CFUfr+c+xDZH4HaX1rBH4LxKP2O8jQ/0h8/8DpwPPAgvDnbwI3hs/XhW1fAjaErzcCXwpfXwd8I9d+zdV3H6kz/SyuJviPTPh8TVr71zzwCLDQzE4DLgf2uPshdz8M7AGumOtO5+Fq4G53f8Pdnwf2AyvDx353f87djwB3h8uWlZmdDFwE3Ang7kfc/VUi9P3n2Idsqup3MEUH8Gt3HyBCv4M06f3Pphq//yZggZk1ATHgJeBS4J7w/anff+r3cg/QYWZG9v2aE1EL+g78i5k9bmadYdub3f0lgPD51LD9dOCFtHWHwrZs7XMhU/8B/jr88/uu1J/mOfpZqf6/BTgIfMXMfmZmXzazE4nW959tHyAav4N01wFfD19H6XeQkt5/iMD37+6/Af4BGCQI9iPA48Cr7n4sQ18m+hm+PwK0Vqr/KVEL+he4+3nAWuAWM7sox7KWoc1ztM+FTP3fDvwhcC7BP6R/DJettv43AecB2939HcB/EqQSsqm2/kP2fYjK7wCAMGf858C3Zlo0Q1s19j8S3394MLqaICXzB8CJBP+Xs/WlqvqfEqmg7+4vhs8HgO8Q/En0cvgnK+HzgXDxIeCMtNWXAi/maC+7TP1395fdfczdx4F/5vifedXW/yFgyN1/Gv58D0EAjcz3T5Z9iNDvIGUt8IS7vxz+HKXfAUzpf4S+/8uA5939oLsfBXYD5xOkzVJTz6b3ZaKf4fstwCEq/P1HJuib2Ylm9qbUa2AN8HPgXiA1+mA98N3w9b3A+8IRDKuAkfBP3weBNWa2KDxyrwnbKtL/1H/W0LXhPqX6f104AuBM4CzgUeAx4KxwxMA8gj+T7y13/939t8ALZnZ22NQBPENEvv9c+xCV30Gav2RyaiQyv4PQpP5H6PsfBFaZWSzMzaf+D/wYeHe4zNTvP/V7eTfwIw+u5Gbbr7kxV1eMZ/sgyMc+FT5+AXSF7a1AL7AvfF4cthvwvwmujPcBK9K2dTPBxZP9wE0V7v/OsH9PE/xjOC1tna6w//8OrE1rvxL4Vfhe1xz+Ds4F9oZ9/b8EIz8i8f3PsA9R+h3EgGGgJa0tMr+DLP2P0vd/K/BLggPTToIROG8hCNr7CVJWJ4TLzg9/3h++/5aZ9msuHirDICJSRyKT3hERkdlT0BcRqSMK+iIidURBX0Skjijoi4jUEQV9EZE6oqAvIlJH/j9ePm18RTwqWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.automl.core.shared import constants\n",
    "from azureml.automl.runtime.shared.score import scoring\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# use automl metrics module\n",
    "scores = scoring.score_regression(\n",
    "    y_test=df_all[target_column_name],\n",
    "    y_pred=df_all['predicted'],\n",
    "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
    "\n",
    "print(\"[Test data scores]\\n\")\n",
    "for key, value in scores.items():    \n",
    "    print('{}:   {:.3f}'.format(key, value))\n",
    "    \n",
    "# Plot outputs\n",
    "%matplotlib inline\n",
    "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
    "test_test = plt.scatter(df_all[target_column_name], df_all[target_column_name], color='g')\n",
    "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at `X_trans` is also useful to see what featurization happened to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>precip</th>\n",
       "      <th>temp</th>\n",
       "      <th>precip_WASNULL</th>\n",
       "      <th>temp_WASNULL</th>\n",
       "      <th>year</th>\n",
       "      <th>half</th>\n",
       "      <th>quarter</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>am_pm</th>\n",
       "      <th>hour12</th>\n",
       "      <th>wday</th>\n",
       "      <th>qday</th>\n",
       "      <th>week</th>\n",
       "      <th>_automl_target_col</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timeStamp</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-08-08 06:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>66.17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>5702.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 07:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>66.29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6163.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 08:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>66.72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6960.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 09:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>67.37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6960.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 10:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>68.30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6960.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 11:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>68.89</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7266.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 12:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>70.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7161.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 13:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>72.83</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7478.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 14:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>73.33</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7478.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 15:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>74.89</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>8167.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 16:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>77.90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>8167.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 17:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>79.24</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>8167.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 18:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>78.68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>8211.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 19:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>76.80</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7686.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 20:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>74.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7319.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 21:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>72.58</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>7084.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 22:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>70.67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6191.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-08 23:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>69.15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>32</td>\n",
       "      <td>6191.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 00:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>68.47</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 01:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>67.74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 02:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>67.31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 03:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>67.50</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 04:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>65.22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5210.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 05:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>64.11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>4478.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 06:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>63.79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>5330.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 07:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>64.28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>6163.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 08:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.03</td>\n",
       "      <td>66.56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>6960.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 09:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>70.87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>7585.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 10:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>74.92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>7919.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 11:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>78.04</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8778.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 12:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>80.18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8504.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 13:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>82.01</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8504.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 14:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>82.85</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8504.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 15:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>84.61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>9110.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 16:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>84.20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>9110.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 17:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>84.04</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>9110.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 18:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>82.86</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8659.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 19:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>79.66</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>8211.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 20:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>76.28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>7686.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 21:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.58</td>\n",
       "      <td>73.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>7290.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 22:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.04</td>\n",
       "      <td>72.26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>6731.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-09 23:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.48</td>\n",
       "      <td>71.10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>6731.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 00:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>71.51</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>6101.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 01:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.02</td>\n",
       "      <td>71.06</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>6101.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 02:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.01</td>\n",
       "      <td>71.21</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>6101.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 03:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.01</td>\n",
       "      <td>70.39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>5713.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 04:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>69.18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-08-10 05:00:00</th>\n",
       "      <th>_automl_dummy_grain_col</th>\n",
       "      <td>0.00</td>\n",
       "      <td>68.09</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>5326.32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             precip  temp  precip_WASNULL  \\\n",
       "timeStamp           _automl_dummy_grain_col                                 \n",
       "2017-08-08 06:00:00 _automl_dummy_grain_col 0.00    66.17  0                \n",
       "2017-08-08 07:00:00 _automl_dummy_grain_col 0.00    66.29  0                \n",
       "2017-08-08 08:00:00 _automl_dummy_grain_col 0.00    66.72  0                \n",
       "2017-08-08 09:00:00 _automl_dummy_grain_col 0.00    67.37  0                \n",
       "2017-08-08 10:00:00 _automl_dummy_grain_col 0.00    68.30  0                \n",
       "2017-08-08 11:00:00 _automl_dummy_grain_col 0.00    68.89  0                \n",
       "2017-08-08 12:00:00 _automl_dummy_grain_col 0.00    70.60  0                \n",
       "2017-08-08 13:00:00 _automl_dummy_grain_col 0.00    72.83  0                \n",
       "2017-08-08 14:00:00 _automl_dummy_grain_col 0.00    73.33  0                \n",
       "2017-08-08 15:00:00 _automl_dummy_grain_col 0.00    74.89  0                \n",
       "2017-08-08 16:00:00 _automl_dummy_grain_col 0.00    77.90  0                \n",
       "2017-08-08 17:00:00 _automl_dummy_grain_col 0.00    79.24  0                \n",
       "2017-08-08 18:00:00 _automl_dummy_grain_col 0.00    78.68  0                \n",
       "2017-08-08 19:00:00 _automl_dummy_grain_col 0.00    76.80  0                \n",
       "2017-08-08 20:00:00 _automl_dummy_grain_col 0.00    74.60  0                \n",
       "2017-08-08 21:00:00 _automl_dummy_grain_col 0.00    72.58  0                \n",
       "2017-08-08 22:00:00 _automl_dummy_grain_col 0.00    70.67  0                \n",
       "2017-08-08 23:00:00 _automl_dummy_grain_col 0.00    69.15  0                \n",
       "2017-08-09 00:00:00 _automl_dummy_grain_col 0.00    68.47  0                \n",
       "2017-08-09 01:00:00 _automl_dummy_grain_col 0.00    67.74  0                \n",
       "2017-08-09 02:00:00 _automl_dummy_grain_col 0.00    67.31  0                \n",
       "2017-08-09 03:00:00 _automl_dummy_grain_col 0.00    67.50  0                \n",
       "2017-08-09 04:00:00 _automl_dummy_grain_col 0.00    65.22  0                \n",
       "2017-08-09 05:00:00 _automl_dummy_grain_col 0.00    64.11  0                \n",
       "2017-08-09 06:00:00 _automl_dummy_grain_col 0.00    63.79  0                \n",
       "2017-08-09 07:00:00 _automl_dummy_grain_col 0.00    64.28  0                \n",
       "2017-08-09 08:00:00 _automl_dummy_grain_col 0.03    66.56  0                \n",
       "2017-08-09 09:00:00 _automl_dummy_grain_col 0.00    70.87  0                \n",
       "2017-08-09 10:00:00 _automl_dummy_grain_col 0.00    74.92  0                \n",
       "2017-08-09 11:00:00 _automl_dummy_grain_col 0.00    78.04  0                \n",
       "2017-08-09 12:00:00 _automl_dummy_grain_col 0.00    80.18  0                \n",
       "2017-08-09 13:00:00 _automl_dummy_grain_col 0.00    82.01  0                \n",
       "2017-08-09 14:00:00 _automl_dummy_grain_col 0.00    82.85  0                \n",
       "2017-08-09 15:00:00 _automl_dummy_grain_col 0.00    84.61  0                \n",
       "2017-08-09 16:00:00 _automl_dummy_grain_col 0.00    84.20  0                \n",
       "2017-08-09 17:00:00 _automl_dummy_grain_col 0.00    84.04  0                \n",
       "2017-08-09 18:00:00 _automl_dummy_grain_col 0.00    82.86  0                \n",
       "2017-08-09 19:00:00 _automl_dummy_grain_col 0.00    79.66  0                \n",
       "2017-08-09 20:00:00 _automl_dummy_grain_col 0.00    76.28  0                \n",
       "2017-08-09 21:00:00 _automl_dummy_grain_col 0.58    73.60  0                \n",
       "2017-08-09 22:00:00 _automl_dummy_grain_col 0.04    72.26  0                \n",
       "2017-08-09 23:00:00 _automl_dummy_grain_col 0.48    71.10  0                \n",
       "2017-08-10 00:00:00 _automl_dummy_grain_col 0.00    71.51  0                \n",
       "2017-08-10 01:00:00 _automl_dummy_grain_col 0.02    71.06  0                \n",
       "2017-08-10 02:00:00 _automl_dummy_grain_col 0.01    71.21  0                \n",
       "2017-08-10 03:00:00 _automl_dummy_grain_col 0.01    70.39  0                \n",
       "2017-08-10 04:00:00 _automl_dummy_grain_col 0.00    69.18  0                \n",
       "2017-08-10 05:00:00 _automl_dummy_grain_col 0.00    68.09  0                \n",
       "\n",
       "                                             temp_WASNULL  year  half  \\\n",
       "timeStamp           _automl_dummy_grain_col                             \n",
       "2017-08-08 06:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 07:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 08:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 09:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 10:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 11:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 12:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 13:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 14:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 15:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 16:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 17:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 18:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 19:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 20:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 21:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 22:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-08 23:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 00:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 01:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 02:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 03:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 04:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 05:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 06:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 07:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 08:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 09:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 10:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 11:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 12:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 13:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 14:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 15:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 16:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 17:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 18:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 19:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 20:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 21:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 22:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-09 23:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 00:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 01:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 02:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 03:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 04:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "2017-08-10 05:00:00 _automl_dummy_grain_col  0             2017  2      \n",
       "\n",
       "                                             quarter  month  day  hour  am_pm  \\\n",
       "timeStamp           _automl_dummy_grain_col                                     \n",
       "2017-08-08 06:00:00 _automl_dummy_grain_col  3        8      8    6     0       \n",
       "2017-08-08 07:00:00 _automl_dummy_grain_col  3        8      8    7     0       \n",
       "2017-08-08 08:00:00 _automl_dummy_grain_col  3        8      8    8     0       \n",
       "2017-08-08 09:00:00 _automl_dummy_grain_col  3        8      8    9     0       \n",
       "2017-08-08 10:00:00 _automl_dummy_grain_col  3        8      8    10    0       \n",
       "2017-08-08 11:00:00 _automl_dummy_grain_col  3        8      8    11    0       \n",
       "2017-08-08 12:00:00 _automl_dummy_grain_col  3        8      8    12    1       \n",
       "2017-08-08 13:00:00 _automl_dummy_grain_col  3        8      8    13    1       \n",
       "2017-08-08 14:00:00 _automl_dummy_grain_col  3        8      8    14    1       \n",
       "2017-08-08 15:00:00 _automl_dummy_grain_col  3        8      8    15    1       \n",
       "2017-08-08 16:00:00 _automl_dummy_grain_col  3        8      8    16    1       \n",
       "2017-08-08 17:00:00 _automl_dummy_grain_col  3        8      8    17    1       \n",
       "2017-08-08 18:00:00 _automl_dummy_grain_col  3        8      8    18    1       \n",
       "2017-08-08 19:00:00 _automl_dummy_grain_col  3        8      8    19    1       \n",
       "2017-08-08 20:00:00 _automl_dummy_grain_col  3        8      8    20    1       \n",
       "2017-08-08 21:00:00 _automl_dummy_grain_col  3        8      8    21    1       \n",
       "2017-08-08 22:00:00 _automl_dummy_grain_col  3        8      8    22    1       \n",
       "2017-08-08 23:00:00 _automl_dummy_grain_col  3        8      8    23    1       \n",
       "2017-08-09 00:00:00 _automl_dummy_grain_col  3        8      9    0     0       \n",
       "2017-08-09 01:00:00 _automl_dummy_grain_col  3        8      9    1     0       \n",
       "2017-08-09 02:00:00 _automl_dummy_grain_col  3        8      9    2     0       \n",
       "2017-08-09 03:00:00 _automl_dummy_grain_col  3        8      9    3     0       \n",
       "2017-08-09 04:00:00 _automl_dummy_grain_col  3        8      9    4     0       \n",
       "2017-08-09 05:00:00 _automl_dummy_grain_col  3        8      9    5     0       \n",
       "2017-08-09 06:00:00 _automl_dummy_grain_col  3        8      9    6     0       \n",
       "2017-08-09 07:00:00 _automl_dummy_grain_col  3        8      9    7     0       \n",
       "2017-08-09 08:00:00 _automl_dummy_grain_col  3        8      9    8     0       \n",
       "2017-08-09 09:00:00 _automl_dummy_grain_col  3        8      9    9     0       \n",
       "2017-08-09 10:00:00 _automl_dummy_grain_col  3        8      9    10    0       \n",
       "2017-08-09 11:00:00 _automl_dummy_grain_col  3        8      9    11    0       \n",
       "2017-08-09 12:00:00 _automl_dummy_grain_col  3        8      9    12    1       \n",
       "2017-08-09 13:00:00 _automl_dummy_grain_col  3        8      9    13    1       \n",
       "2017-08-09 14:00:00 _automl_dummy_grain_col  3        8      9    14    1       \n",
       "2017-08-09 15:00:00 _automl_dummy_grain_col  3        8      9    15    1       \n",
       "2017-08-09 16:00:00 _automl_dummy_grain_col  3        8      9    16    1       \n",
       "2017-08-09 17:00:00 _automl_dummy_grain_col  3        8      9    17    1       \n",
       "2017-08-09 18:00:00 _automl_dummy_grain_col  3        8      9    18    1       \n",
       "2017-08-09 19:00:00 _automl_dummy_grain_col  3        8      9    19    1       \n",
       "2017-08-09 20:00:00 _automl_dummy_grain_col  3        8      9    20    1       \n",
       "2017-08-09 21:00:00 _automl_dummy_grain_col  3        8      9    21    1       \n",
       "2017-08-09 22:00:00 _automl_dummy_grain_col  3        8      9    22    1       \n",
       "2017-08-09 23:00:00 _automl_dummy_grain_col  3        8      9    23    1       \n",
       "2017-08-10 00:00:00 _automl_dummy_grain_col  3        8      10   0     0       \n",
       "2017-08-10 01:00:00 _automl_dummy_grain_col  3        8      10   1     0       \n",
       "2017-08-10 02:00:00 _automl_dummy_grain_col  3        8      10   2     0       \n",
       "2017-08-10 03:00:00 _automl_dummy_grain_col  3        8      10   3     0       \n",
       "2017-08-10 04:00:00 _automl_dummy_grain_col  3        8      10   4     0       \n",
       "2017-08-10 05:00:00 _automl_dummy_grain_col  3        8      10   5     0       \n",
       "\n",
       "                                             hour12  wday  qday  week  \\\n",
       "timeStamp           _automl_dummy_grain_col                             \n",
       "2017-08-08 06:00:00 _automl_dummy_grain_col  6       1     39    32     \n",
       "2017-08-08 07:00:00 _automl_dummy_grain_col  7       1     39    32     \n",
       "2017-08-08 08:00:00 _automl_dummy_grain_col  8       1     39    32     \n",
       "2017-08-08 09:00:00 _automl_dummy_grain_col  9       1     39    32     \n",
       "2017-08-08 10:00:00 _automl_dummy_grain_col  10      1     39    32     \n",
       "2017-08-08 11:00:00 _automl_dummy_grain_col  11      1     39    32     \n",
       "2017-08-08 12:00:00 _automl_dummy_grain_col  12      1     39    32     \n",
       "2017-08-08 13:00:00 _automl_dummy_grain_col  1       1     39    32     \n",
       "2017-08-08 14:00:00 _automl_dummy_grain_col  2       1     39    32     \n",
       "2017-08-08 15:00:00 _automl_dummy_grain_col  3       1     39    32     \n",
       "2017-08-08 16:00:00 _automl_dummy_grain_col  4       1     39    32     \n",
       "2017-08-08 17:00:00 _automl_dummy_grain_col  5       1     39    32     \n",
       "2017-08-08 18:00:00 _automl_dummy_grain_col  6       1     39    32     \n",
       "2017-08-08 19:00:00 _automl_dummy_grain_col  7       1     39    32     \n",
       "2017-08-08 20:00:00 _automl_dummy_grain_col  8       1     39    32     \n",
       "2017-08-08 21:00:00 _automl_dummy_grain_col  9       1     39    32     \n",
       "2017-08-08 22:00:00 _automl_dummy_grain_col  10      1     39    32     \n",
       "2017-08-08 23:00:00 _automl_dummy_grain_col  11      1     39    32     \n",
       "2017-08-09 00:00:00 _automl_dummy_grain_col  0       2     40    32     \n",
       "2017-08-09 01:00:00 _automl_dummy_grain_col  1       2     40    32     \n",
       "2017-08-09 02:00:00 _automl_dummy_grain_col  2       2     40    32     \n",
       "2017-08-09 03:00:00 _automl_dummy_grain_col  3       2     40    32     \n",
       "2017-08-09 04:00:00 _automl_dummy_grain_col  4       2     40    32     \n",
       "2017-08-09 05:00:00 _automl_dummy_grain_col  5       2     40    32     \n",
       "2017-08-09 06:00:00 _automl_dummy_grain_col  6       2     40    32     \n",
       "2017-08-09 07:00:00 _automl_dummy_grain_col  7       2     40    32     \n",
       "2017-08-09 08:00:00 _automl_dummy_grain_col  8       2     40    32     \n",
       "2017-08-09 09:00:00 _automl_dummy_grain_col  9       2     40    32     \n",
       "2017-08-09 10:00:00 _automl_dummy_grain_col  10      2     40    32     \n",
       "2017-08-09 11:00:00 _automl_dummy_grain_col  11      2     40    32     \n",
       "2017-08-09 12:00:00 _automl_dummy_grain_col  12      2     40    32     \n",
       "2017-08-09 13:00:00 _automl_dummy_grain_col  1       2     40    32     \n",
       "2017-08-09 14:00:00 _automl_dummy_grain_col  2       2     40    32     \n",
       "2017-08-09 15:00:00 _automl_dummy_grain_col  3       2     40    32     \n",
       "2017-08-09 16:00:00 _automl_dummy_grain_col  4       2     40    32     \n",
       "2017-08-09 17:00:00 _automl_dummy_grain_col  5       2     40    32     \n",
       "2017-08-09 18:00:00 _automl_dummy_grain_col  6       2     40    32     \n",
       "2017-08-09 19:00:00 _automl_dummy_grain_col  7       2     40    32     \n",
       "2017-08-09 20:00:00 _automl_dummy_grain_col  8       2     40    32     \n",
       "2017-08-09 21:00:00 _automl_dummy_grain_col  9       2     40    32     \n",
       "2017-08-09 22:00:00 _automl_dummy_grain_col  10      2     40    32     \n",
       "2017-08-09 23:00:00 _automl_dummy_grain_col  11      2     40    32     \n",
       "2017-08-10 00:00:00 _automl_dummy_grain_col  0       3     41    32     \n",
       "2017-08-10 01:00:00 _automl_dummy_grain_col  1       3     41    32     \n",
       "2017-08-10 02:00:00 _automl_dummy_grain_col  2       3     41    32     \n",
       "2017-08-10 03:00:00 _automl_dummy_grain_col  3       3     41    32     \n",
       "2017-08-10 04:00:00 _automl_dummy_grain_col  4       3     41    32     \n",
       "2017-08-10 05:00:00 _automl_dummy_grain_col  5       3     41    32     \n",
       "\n",
       "                                             _automl_target_col  \n",
       "timeStamp           _automl_dummy_grain_col                      \n",
       "2017-08-08 06:00:00 _automl_dummy_grain_col 5702.61              \n",
       "2017-08-08 07:00:00 _automl_dummy_grain_col 6163.60              \n",
       "2017-08-08 08:00:00 _automl_dummy_grain_col 6960.29              \n",
       "2017-08-08 09:00:00 _automl_dummy_grain_col 6960.29              \n",
       "2017-08-08 10:00:00 _automl_dummy_grain_col 6960.29              \n",
       "2017-08-08 11:00:00 _automl_dummy_grain_col 7266.81              \n",
       "2017-08-08 12:00:00 _automl_dummy_grain_col 7161.13              \n",
       "2017-08-08 13:00:00 _automl_dummy_grain_col 7478.13              \n",
       "2017-08-08 14:00:00 _automl_dummy_grain_col 7478.13              \n",
       "2017-08-08 15:00:00 _automl_dummy_grain_col 8167.52              \n",
       "2017-08-08 16:00:00 _automl_dummy_grain_col 8167.52              \n",
       "2017-08-08 17:00:00 _automl_dummy_grain_col 8167.52              \n",
       "2017-08-08 18:00:00 _automl_dummy_grain_col 8211.74              \n",
       "2017-08-08 19:00:00 _automl_dummy_grain_col 7686.30              \n",
       "2017-08-08 20:00:00 _automl_dummy_grain_col 7319.17              \n",
       "2017-08-08 21:00:00 _automl_dummy_grain_col 7084.07              \n",
       "2017-08-08 22:00:00 _automl_dummy_grain_col 6191.92              \n",
       "2017-08-08 23:00:00 _automl_dummy_grain_col 6191.92              \n",
       "2017-08-09 00:00:00 _automl_dummy_grain_col 5326.32              \n",
       "2017-08-09 01:00:00 _automl_dummy_grain_col 5326.32              \n",
       "2017-08-09 02:00:00 _automl_dummy_grain_col 5326.32              \n",
       "2017-08-09 03:00:00 _automl_dummy_grain_col 5326.32              \n",
       "2017-08-09 04:00:00 _automl_dummy_grain_col 5210.66              \n",
       "2017-08-09 05:00:00 _automl_dummy_grain_col 4478.68              \n",
       "2017-08-09 06:00:00 _automl_dummy_grain_col 5330.49              \n",
       "2017-08-09 07:00:00 _automl_dummy_grain_col 6163.60              \n",
       "2017-08-09 08:00:00 _automl_dummy_grain_col 6960.29              \n",
       "2017-08-09 09:00:00 _automl_dummy_grain_col 7585.97              \n",
       "2017-08-09 10:00:00 _automl_dummy_grain_col 7919.98              \n",
       "2017-08-09 11:00:00 _automl_dummy_grain_col 8778.39              \n",
       "2017-08-09 12:00:00 _automl_dummy_grain_col 8504.33              \n",
       "2017-08-09 13:00:00 _automl_dummy_grain_col 8504.33              \n",
       "2017-08-09 14:00:00 _automl_dummy_grain_col 8504.33              \n",
       "2017-08-09 15:00:00 _automl_dummy_grain_col 9110.69              \n",
       "2017-08-09 16:00:00 _automl_dummy_grain_col 9110.69              \n",
       "2017-08-09 17:00:00 _automl_dummy_grain_col 9110.69              \n",
       "2017-08-09 18:00:00 _automl_dummy_grain_col 8659.11              \n",
       "2017-08-09 19:00:00 _automl_dummy_grain_col 8211.74              \n",
       "2017-08-09 20:00:00 _automl_dummy_grain_col 7686.30              \n",
       "2017-08-09 21:00:00 _automl_dummy_grain_col 7290.38              \n",
       "2017-08-09 22:00:00 _automl_dummy_grain_col 6731.42              \n",
       "2017-08-09 23:00:00 _automl_dummy_grain_col 6731.42              \n",
       "2017-08-10 00:00:00 _automl_dummy_grain_col 6101.30              \n",
       "2017-08-10 01:00:00 _automl_dummy_grain_col 6101.30              \n",
       "2017-08-10 02:00:00 _automl_dummy_grain_col 6101.30              \n",
       "2017-08-10 03:00:00 _automl_dummy_grain_col 5713.49              \n",
       "2017-08-10 04:00:00 _automl_dummy_grain_col 5326.32              \n",
       "2017-08-10 05:00:00 _automl_dummy_grain_col 5326.32              "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_trans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Training <a id=\"advanced_training\"></a>\n",
    "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, grain and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using lags and rolling window features\n",
    "Now we will configure the target lags, that is the previous values of the target variables, meaning the prediction is no longer horizon-less. We therefore must still specify the `max_horizon` that the model will learn to forecast. The `target_lags` keyword specifies how far back we will construct the lags of the target variable, and the `target_rolling_window_size` specifies the size of the rolling window over which we will generate the `max`, `min` and `sum` features.\n",
    "\n",
    "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset.  You can choose to remove models from the blocked_models list but you may need to increase the iteration_timeout_minutes parameter value to get results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_advanced_settings = {\n",
    "    'time_column_name': time_column_name,\n",
    "    'max_horizon': max_horizon,\n",
    "    'target_lags': 12,\n",
    "    'country_or_region' : 'US', #jours féries\n",
    "    'target_rolling_window_size': 4,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(task='forecasting',                             \n",
    "                             primary_metric='normalized_root_mean_squared_error',\n",
    "                             debug_log='automl.log',\n",
    "                             blocked_models = ['ElasticNet','ExtremeRandomTrees','GradientBoosting','XGBoostRegressor','ExtremeRandomTrees', 'AutoArima', 'Prophet'], #These models are blocked for tutorial purposes, remove this for real use cases.                            \n",
    "                             experiment_timeout_hours=0.25,\n",
    "                             iterations=5,\n",
    "                             training_data=train,\n",
    "                             label_column_name=target_column_name,\n",
    "                             enable_early_stopping = True,\n",
    "                             enable_voting_ensemble=False, # Pas de Voting Ensemble\n",
    "                             enable_stack_ensemble=False, # Pas de Stack Ensemble\n",
    "                             n_cross_validations=3,                             \n",
    "                             verbosity=logging.INFO,\n",
    "                            **automl_advanced_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now start a new remote run, this time with lag and rolling window featurization. AutoML applies featurizations in the setup stage, prior to iterating over ML models. The full training set is featurized first, followed by featurization of each of the CV splits. Lag and rolling window features introduce additional complexity, so the run will take longer than in the previous example that lacked these featurizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local machine\n",
      "Parent Run ID: AutoML_ddad88bb-da21-4b3c-93fe-2e8a1c57bd3c\n",
      "\n",
      "Current status: DatasetFeaturization. Beginning to featurize the dataset.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Converting non-string tag to string: (forecasting_target_lags: [12])\n",
      "WARNING - Converting non-string tag to string: (forecasting_target_rolling_window_size: 4)\n",
      "WARNING - Converting non-string tag to string: (forecasting_max_horizon: 48)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the dataset.\n",
      "Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "Current status: DatasetFeaturization. Beginning to featurize the CV split.\n",
      "Current status: DatasetFeaturizationCompleted. Completed featurizing the CV split.\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Frequency detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  The time series was analyzed, all data points are aligned with detected frequency.\n",
      "              \n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       DONE\n",
      "DESCRIPTION:  If the missing values are expected, let the run complete. Otherwise cancel the current run and use a script to customize the handling of missing feature values that may be more appropriate based on the data type and business requirement.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "DETAILS:      \n",
      "+---------------------------------+---------------------------------+---------------------------------+\n",
      "|Column name                      |Missing value count              |Imputation type                  |\n",
      "+=================================+=================================+=================================+\n",
      "|precip                           |230                              |median                           |\n",
      "|temp                             |186                              |median                           |\n",
      "+---------------------------------+---------------------------------+---------------------------------+\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Memory Issues Detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  The selected horizon, lag and rolling window values were analyzed, and no potential memory issues were detected.\n",
      "              Learn more about time-series forecasting configurations: https://aka.ms/AutomatedMLForecastingConfiguration\n",
      "\n",
      "****************************************************************************************************\n",
      "Current status: ModelSelection. Beginning model selection.\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   MaxAbsScaler DecisionTree                      0:02:35       0.0651    0.0651\n",
      "         1   MinMaxScaler DecisionTree                      0:02:12       0.0599    0.0599\n",
      "         2   StandardScalerWrapper LassoLars                0:01:31       0.0845    0.0599\n",
      "         3   StandardScalerWrapper LassoLars                0:01:35       0.0845    0.0599\n",
      "         4   RobustScaler DecisionTree                      0:02:08       0.0521    0.0521\n",
      "****************************************************************************************************\n",
      "Current status: BestRunExplainModel. Best run model explanations started\n",
      "Current status: ModelExplanationDataSetSetup. Model explanations data setup completed\n",
      "Current status: PickSurrogateModel. Choosing LightGBM as the surrogate model for explanations\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features started\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features completed\n",
      "Current status: BestRunExplainModel. Best run model explanations completed\n",
      "****************************************************************************************************\n"
     ]
    }
   ],
   "source": [
    "advanced_run = experiment.submit(automl_config, show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run_lags, fitted_model_lags = advanced_run.get_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "[('timeseriestransformer', TimeSeriesTransformer(featurization_config=None,\n",
       "           pipeline_type=<TimeSeriesPipelineType.FULL: 1>)), ('MinMaxScaler', MinMaxScaler(copy=True, feature_range=(0, 1))), ('DecisionTreeRegressor', DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=0.7,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=0.001953125,\n",
       "           min_samples_split=0.0012814223889440828,\n",
       "           min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "           splitter='best'))]"
      ],
      "text/plain": [
       "[('timeseriestransformer',\n",
       "  TimeSeriesTransformer(featurization_config=None,\n",
       "             pipeline_type=<TimeSeriesPipelineType.FULL: 1>)),\n",
       " ('MinMaxScaler', MinMaxScaler(copy=True, feature_range=(0, 1))),\n",
       " ('DecisionTreeRegressor',\n",
       "  DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=0.7,\n",
       "             max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "             min_impurity_split=None, min_samples_leaf=0.001953125,\n",
       "             min_samples_split=0.0012814223889440828,\n",
       "             min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "             splitter='best'))]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitted_model.steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "['precip', 'temp', 'precip_WASNULL', 'temp_WASNULL', 'year', 'half', 'quarter', 'month', 'day', 'hour', 'am_pm', 'hour12', 'wday', 'qday', 'week']"
      ],
      "text/plain": [
       "['precip',\n",
       " 'temp',\n",
       " 'precip_WASNULL',\n",
       " 'temp_WASNULL',\n",
       " 'year',\n",
       " 'half',\n",
       " 'quarter',\n",
       " 'month',\n",
       " 'day',\n",
       " 'hour',\n",
       " 'am_pm',\n",
       " 'hour12',\n",
       " 'wday',\n",
       " 'qday',\n",
       " 'week']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitted_model.named_steps['timeseriestransformer'].get_engineered_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RawFeatureName</th>\n",
       "      <th>TypeDetected</th>\n",
       "      <th>Dropped</th>\n",
       "      <th>EngineeredFeatureCount</th>\n",
       "      <th>Transformations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>precip</td>\n",
       "      <td>Numeric</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>[MedianImputer, ImputationMarker]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>temp</td>\n",
       "      <td>Numeric</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>[MedianImputer, ImputationMarker]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>timeStamp</td>\n",
       "      <td>DateTime</td>\n",
       "      <td>No</td>\n",
       "      <td>11</td>\n",
       "      <td>[DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  RawFeatureName TypeDetected Dropped  EngineeredFeatureCount  \\\n",
       "0  precip         Numeric      No      2                        \n",
       "1  temp           Numeric      No      2                        \n",
       "2  timeStamp      DateTime     No      11                       \n",
       "\n",
       "                                                                                                                                                                                                                           Transformations  \n",
       "0  [MedianImputer, ImputationMarker]                                                                                                                                                                                                        \n",
       "1  [MedianImputer, ImputationMarker]                                                                                                                                                                                                        \n",
       "2  [DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer, DateTimeTransformer]  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the featurization summary as a list of JSON\n",
    "featurization_summary = fitted_model.named_steps['timeseriestransformer'].get_featurization_summary()\n",
    "# View the featurization summary as a pandas dataframe\n",
    "pd.DataFrame.from_records(featurization_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Results<a id=\"advanced_results\"></a>\n",
    "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, grain and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The featurized data, aligned to y, will also be returned.\n",
    "# This contains the assumptions that were made in the forecast\n",
    "# and helps align the forecast to the original data\n",
    "y_predictions, X_trans = fitted_model_lags.forecast(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from forecasting_helper import align_outputs\n",
    "df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Test data scores]\n",
      "\n",
      "median_absolute_error:   259.010\n",
      "r2_score:   0.856\n",
      "normalized_root_mean_squared_error:   0.112\n",
      "root_mean_squared_error:   354.464\n",
      "root_mean_squared_log_error:   0.059\n",
      "normalized_root_mean_squared_log_error:   0.120\n",
      "normalized_median_absolute_error:   0.082\n",
      "mean_absolute_error:   292.110\n",
      "explained_variance:   0.903\n",
      "normalized_mean_absolute_error:   0.093\n",
      "mean_absolute_percentage_error:   4.496\n",
      "spearman_correlation:   0.950\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3Bc1X338fdXsgTehNhGNhli45WT+CEluDigBwh4GIOosclDTNpk4kQBY9JRxiapTZo2pmqHH4mmbX4MOJ2YVAk/DN4GCIXEJAQwpszzhBkCNjEo/AoKSELgYGOD88MBC+n7/HHP2itpd7UrrbS/Pq+Znd397rn3nrsLX12fc+455u6IiEh1qCl2BUREZPIo6YuIVBElfRGRKqKkLyJSRZT0RUSqyJRiVyCbmTNnemNjY7GrISJSVnbs2PG6u89K91lJJ/3Gxka2b99e7GqIiJQVM+vJ9Jmad0REqoiSvohIFSnp5p3h+vv76evr46233ip2VUrGkUceyZw5c6irqyt2VUSkDJRV0u/r6+Ooo46isbERMyt2dYrO3dm7dy99fX3Mmzev2NURkTJQVs07b731Fg0NDUr4gZnR0NCgf/mISM7KKukDSvjD6PsQqSyJzgSN1zVSc3UNjdc1kuhMFHT/ZZf0y8XixYsBWLduHQMDA2nL3HjjjYdef+lLX5qMaolICUt0Jmi9p5We/T04Ts/+HlrvaS1o4q/IpJ9IQGMj1NREz4nC/qFkcHAw57LXXXcdtbW1aT9LTfr/8R//Me56iUj5SnQmWHn3Sg70HxgSP9B/gLZtbQU7TsUl/UQCWluhpwfco+fW1vwT/8MPP8wFF1zAsmXLOOecc3jiiSc4++yz+eQnP8nNN9/MT3/6U8466yzOOOMM7rvvPgA6Ojo4/fTTWb9+/aH9LF68mHfeeYff/e53LFu2jMWLF3PFFVfQ0dFBZ2cnixcvprOzk0WLFgHw5JNPcuaZZ3L66aezefNmAC655BLWrl3LokWLuPrqqwvzRYlIyUhe4Q94+laB3v29BTtWWY3eyUVbGxwY+oeSAweieEtLfvt666232Lp1K7fffjsPPPAAu3fv5sEHH8TMOOecc3jooYcYHBxk2bJlnHvuudxwww088sgjPPbYYzz66KND9vWv//qvXH755SxZsoTBwUFqamq45ZZbePjhh4eU+5d/+RcSiQSzZ89m0aJFfPrTnwaiPx4bNmzgtNNO48orr8z3axGREta2rW3EFX6qudPmFuxYFXel35vhD2KmeDYf+chHAFi4cCEPPvggJ510ErW1tbz++us8++yznHvuuSxZsoRdu3axZ88e4vE4U6ZM4ZRTThmxr9/85jecccYZANTUZP7a33jjDRobG6mrq2PevHns3r0bgBNPPBGAqVOn5n8iIlISMnXSZruSj9XFaG9uL1gdKu5Kf+7cqEknXTxfTz755KHn5uZmnn76aQBmzpzJggULuP/++6mtraW/vx8zo6enh4GBAX71q1+N2Nfxxx/Po48+yrnnnnvoSj/dyJvp06fT3d3N7NmzefHFFznmmGMAjdIRKXfJJpzkFX2ykxaiK/me/SMTV63V0nFBBy0L8mymyKLirvTb2yEWGxqLxaJ4vurq6li6dCkbN25kyZIlh+I1NTV8+ctfprm5mbPPPpt169YxZcoUVq1axRlnnMFdd901Yl/r16/nm9/8JosXL+af//mfATjuuOP4m7/5G5577rlD5a655ho++9nPsmjRIi677DLdaStSIdI14SQ7adub24nVDU1csboYmz6xqaAJH8BKeWH0pqYmT51l89lnn+Uv/uIvRt0ukYja8Ht7oyv89vb82/MffvhhHnzwQb7+9a/nW+1Jl+v3IiLFU3N1Dc7IfGsYg1cOkuhM0Latjd79vcydNpf25vYxJ3wz2+HuTek+y6l5x8wuB/4WcKATWAUcC9wGHA08AVzk7gfN7AjgFuAUYC/waXfvDvu5Avg8MAD8nbvfP6YzGkVLS/5JXkRkImVqwkl20rYsaCn4VX06ozbvmNls4O+AJnc/EagFVgD/Dlzr7vOBN4iSOeH5DXf/IHBtKIeZnRC2+zCwFNhoZukHsJeAxYsXl8VVvoiUh0xNOIXspM1Frm36U4CpZjYFiAG7gHOAO8Pnm4ALw+vl4T3h82aLeiGXA7e5+9vu/hLQBZw6/lMQESl9LQta6Ligg/i0OIYRnxYveCdtLkZt3nH3V8zsW0Av8GfgAWAH8Ka7vxOK9QGzw+vZwMth23fMbD/QEOKpg9dTtznEzFqBVoC5YxlyIyIySfJth5+sJpxscmnemUF0lT4PeB/wLmBZmqLJHop0Yws9S3xowL3D3ZvcvWnWrLRLPIqIFN1kzJMzEXJp3jkXeMnd97h7P3AXcAYwPTT3AMwBXg2v+4DjAMLn04B9qfE025SU7u5uHnrooZzK3nzzzYfm4rnkkkvo6uqayKqJSInINgSzlOWS9HuB080sFtrmm4FngP8BPhnKrAR+El5vCe8Jnz/k0bjQLcAKMzvCzOYB84HHCnMaQ413atJ0ST/TJGupSV9Eqkemu2gLOU/ORBg16bv7L4k6ZJ8gGq5ZA3QAXwW+bGZdRG32N4RNbgAaQvzLwPqwn6eBO4j+YNwHXOaeYXahcSjEP7k6Ojq49dZbD918lZxkLTkpGkSjex577DF27txJc3Mzt956KxDNlqmJ0UQqX6b5cAo5T85EyGn0jrtf6e4fcvcT3f2iMALnRXc/1d0/6O6fcve3Q9m3wvsPhs9fTNlPu7t/wN2Pd/efT8QJFeKfXK2trVx00UXccMMN7N69m9tvv51LL710RLlTTz2VhQsXsm3bNi666CIg+mPwi1/8gnvvvXd8JyIiRTVai0GpDMHMV8VNw1Dof3IlJ1lLle0uZk2MJlJ8Y1lTI3WbmWcnuOTOoS0Gl949tMWgZUELK2d0UPvHOLhR+8c4K2dkH4KZrV7Jz8xgypToeSLWA6m8CddGuestF3V1dYdWu0qdEdPdefvtt3n++edHlJ0yJfoqNTGaSHEl19RITrGeXFMDMt+pn0jAqmsT9H+iDab1snewBmqGtj4f9AOs3dJ2KKknErDpKy0MHIjeDwCbYnDme9IfJ1u9YOhnycX2cql7viruSr8Q/+Q68cQTeeSRR/jqV786JH7JJZewaNEifvSjHx2KfexjH+PCCy/kv//7v8dXcREpiGxramSy9gcJ+s9rhek9YA616bsb9/YfbjHI9zjZyqf7LNe656syJ1wr4MRF5UATrokcVlMTrZo3nBlkGmhnlzdGCX80b8bxa7vHdJxs5SH9Z6PtM3P5cU64Vm5K4a43ESmOMa2pMS2HPr+DMRp2Hm4xyPc4o5VP99lo+xyLimveEZHqNpY1NRrqMmTVgVpwgzfj1N3fwYa/PXwxme9xspVP91mudc9X2SX9Um6OKgZ9HyJDtbRARwfE41GzSDwevc/WEbrh4+3U29CsO2UwRsP/24RdM0j87m5uurxlyD7yPU628qmfASQHDOZS93yVVZv+Sy+9xFFHHUVDQ4NGyRAl/L179/KHP/yBefPmFbs6ImWtkvoCs7Xpl1XS7+/vp6+vj7feequItSotRx55JHPmzNGyiiJySMV05NbV1emKVkRGVUlX7YVWVklfRGQ0yfm3ktOxJOffApT4KcOOXBGRbMp1yuPJoqQvIhWlXKc8nixK+iJSUcp1yuPJoqQvImWlUqc8nizqyBWRkpccjdOzvwfD8LC8drpO2uSzRu+kV1bj9EWkuiQ6E6z9+Vr2/nlv1nLxaXG613VPTqXKQMWM0xeR6pDoTPCFe77An/r/lFN5ddLmbtQ2fTM73sx2pjx+b2brzOwqM3slJX5+yjZXmFmXmT1vZuelxJeGWJeZrZ+okxKR8pXoTLDqx6tyTvigTtp8jHql7+7PAwsBzKwWeAW4G1gFXOvu30otb2YnACuADwPvAx40s/8VPv4u8FdAH/C4mW1x92cKdC4iUgHatrXRP9ifc3l10uYn39E7zcBv3T3bagPLgdvC4ukvAV3AqeHRFRZUPwjcFsqKiBySS1ONEU24GJ8Wp+OC7OvSylD5tumvAH6Y8v6LZnYxsB34e3d/A5gNPJpSpi/EAF4eFj9t+AHMrBVoBZhbyJUDRKQsZFrnOqlhagMblm1Qoh+jnK/0zawe+DiQXCD2euADRE0/u4BvJ4um2dyzxIcG3Dvcvcndm2bNmpVr9USkjGQba9/e3E5dTfpZY1c3reb1f3xdCX8c8rnSXwY84e6vASSfAczs+8BPw9s+4LiU7eYAr4bXmeIiUiVGmxAtmdBTh2rq6r5wch6nb2a3Afe7+03h/bHuviu8vhw4zd1XmNmHgf8iasN/H7ANmE90pf8bon6BV4DHgc+6+9OZjqlx+iKVp/G6xrTNNxprXzjjHqdvZjGiUTdfSAl/w8wWEjXRdCc/c/enzewO4BngHeAydx8I+/kicD9QC9yYLeGLSHkbfmNV8mpdE6IVl+7IFZGCW/OzNVy//foR8fraeo6qPyrtHba60i+cbFf6mnBNRAoqU8IHODhwEEATohWRkr6IFEyiM8H3tn8va5l9f95HxwUdxKfFMUxj7SeZ5t4RkYJp29Z2aAbMTOZOmztklI5MLl3pi0jBjNYZW19br2acIlPSF5GCyTbx2bvr382Ny2/UFX6RKemLSMGkW7XKMFY3reYPV/xBCb8EKOmLSFZrfraGKddMwa42plwzhTU/W5OxbMuClhGdtLf+9a1s/NjGSayxZKNx+iKSUabhl6ubViuRlzCN0xeRMenY0ZFXXEqfkr6IZDQQzaCSc1xKn5K+SJVLdCaY+Y2Z2NWGXW3M/MbMQ1Md11pt2m0yxaX0KemLVIhEAhoboaYmek4kRtsiSvgr71o1ZC6cvX/ey6U/uTSaAvmU1rTbZYqn1mXmTDA7/Jg583CdRqtruu1raw/vZ+bM/M5TDtMduSIVIJGA1lY4EE1RT09P9B6gJcsoybVb2hhg5Hq0BwcO0ratjfZZ3fznDhhc2AE1AzBYS83OVs78YOZO3EQCLr0UDh4cGt+7N4o/8ghs2pS5rokErFoF/cOqNTh4eD9JuZ6nHKbROyIVoLExSoDDxePQ3X34faIzQdu2Nnr390bLEr7Zk35NO6Lx9XNvGsxpv7nUJam2FgbSdAkk9zna9ulkq081Gvd8+iJS2nozzH6QGk+3YlWU8dNf+M2dNjen/ebzGaRP+Knbjbb9WI4ph6lNX6QCzM0w+0FqvG1b26GEf4h5+pw/EM2Rk8t+8/kMoiv9bNuNtv1YjimHKemLVID2dogNnf2AWCyKJ2WdDO1PDVHyd+BAA6uPjebIyWW/6epSX5/+s/r6qA0+2z7b26Eu/broaY1WHxlq1KRvZseb2c6Ux+/NbJ2ZHW1mW83shfA8I5Q3M/uOmXWZ2VNmdnLKvlaG8i+Y2cqJPDGRatLSAh0dUdu2GTQsTjD1nxq5qKuGxusaSXQmMk6G1lAXJ37H69g1TvxmZ/Nfvs7G1S1p9xuPR++zdZq2tMCNN0JDw7DjNETxjRuz77OlBW66aeT2NTWH99PQkHt9ZKi8OnLNrJZoUfPTgMuAfe7+b2a2Hpjh7l81s/OBLwHnh3Ib3P00Mzsa2A40EV1P7ABOcfc3Mh1PHbkiuUt20vbs78GwIfPax+pirDxpJZue3DSkiSdWF9MCJhWokNMwNAO/dfceYDmwKcQ3AReG18uBWzzyKDDdzI4FzgO2uvu+kOi3AkvzPL6IpJHspI06ZxmxkMmB/gPc+8K9WrFK8h69swL4YXj9XnffBeDuu8zsmBCfDbycsk1fiGWKD2FmrUArwFz1zojkJG0n7TC9+3u1YpXkfqVvZvXAx4EfjVY0TcyzxIcG3Dvcvcndm2bNmpVr9USq2mgrVkH2BU6keuTTvLMMeMLdXwvvXwvNNoTn3SHeBxyXst0c4NUscRHJQ6IzQeN1jdRcPXonbVKsLqZlCgXIL+l/hsNNOwBbgOQInJXAT1LiF4dRPKcD+0Mz0P3AEjObEUb6LAkxEclRatu94/Ts76H1nlbOn39+2hWrALXdyxA5tembWQz4K+ALKeF/A+4ws88DvcCnQvxeopE7XcABYBWAu+8zs68Bj4dy17j7vnGfgUgFS3QmWPvztYcmRKuxGgZ9cEiZ1E7a1CkW2pvblehlBM29I1KiEp0JLv3JpRwcODhqWcMYvHJw1HJSHbRylkgZatvWllPCB3XSSu6U9EVKVC4jckCdtJIfJX2REpXt6r3WanWDlYyJplYWKVHtze1p2/Trauq46cKblOhlTHSlL1Ik6cbbp2pZ0MKNy2+kYerhmccapjYo4cu4aPSOSBEMX9AENPmZFI5G70jJy2VR77Es/D0RClGPdHPlHOg/QNu2thHHSl0gPHVx8bHUMd/4WM63VH4nycDdS/ZxyimnuFS+zZvdYzF3OPyIxaJ4PmVKpa654CpzriLNw4Ycq75+6LHAva4u+/Ey1XH16vzimzfnf76l8jtVO2C7Z8irat6RostlUe9cF/6eaIWqx5R/aGTg3SN3VPvHOO98szvrsUY7XqbtMi1Inm2hcsjvfEvld6p22Zp3lPSl6GpqomvC4cxgcDD3MpOhUPWwv0zABa1Qn9LEczAG93TgT7VkPdZox8u2XT4szIubz/mWyu9U7dSmLyUtl8W3x7JA90QoVD3iv2+BezrgzTi4Rc/3dETxHPY5ls8yLUiebaHyfM+3VH4nySJTu08pPNSmXx2qsU0/13NWm76MBVna9Iue2LM9lPSrx+bN7vG4u1n0nC5J5FJmMhSqHrmec0PD4QTa0JDb8TLtO9/4WM63VH6napYt6atNXyRPyQXINYWxlKpsbfqahkEkD4nOBKt+vIr+wX4Aevb3sOrHqwCU+KUsqCNXJA9rf772UMJP6h/sZ+3P1xapRiL5UdIXyUNyBatc4yKlRklfRKSK5JT0zWy6md1pZs+Z2bNm9lEzu8rMXjGzneFxfkr5K8ysy8yeN7PzUuJLQ6zLzNZPxAmJTKTUGS9ziYuUmlyv9DcA97n7h4CTgGdD/Fp3Xxge9wKY2QnACuDDwFJgo5nVmlkt8F1gGXAC8JlQVqQkjDbVMcCGZRuor60fEquvrWfDsg2TVU2RcRl19I6ZvQc4C7gEwN0PAgcteY/2SMuB29z9beAlM+sCTg2fdbn7i2G/t4Wyz4znBEQKYfhUxz37e2i9pxUYOion+VpDNqVc5TJk8/3AHuAmMzsJ2AEkhyp80cwuBrYDf+/ubwCzgUdTtu8LMYCXh8VPG34wM2sFWgHm6t5tmSTZpjoentBbFrQoyUvZyqV5ZwpwMnC9u38E+BOwHrge+ACwENgFfDuUT/dPAM8SHxpw73D3JndvmjVrVg7VExm/TIuQ57o4uUi5yCXp9wF97v7L8P5O4GR3f83dB9x9EPg+h5tw+oDjUrafA7yaJS5SdJkWIc+2OLlIORo16bv774CXzez4EGoGnjGzY1OKfQL4dXi9BVhhZkeY2TxgPvAY8Dgw38zmmVk9UWfvlgKdh8i4tDe3E6uLDYnF6mK0N7cXqUYiEyPXaRi+BCRCsn4RWAV8x8wWEjXRdANfAHD3p83sDqIO2neAy9x9AMDMvgjcD9QCN7r70wU8F5ExUwetVAtNuCYiUmG0iIqIiABK+iIiVUVJX0pWIhEttF1TEz0nRt4gW5TjTXa9Su34UuYyra5SCg+tnFW9JnvZvVyPV+zlAIt9fCkPaOUsKTeNjdDTMzIej0N3d/GON9n1Gq7Yx5fyoI5cKTu96W6EXZCg5xPZJ0Qr6PHSxHMtN1GKfXwpf0r6UpJGTLu0IAEXtML0Hhw/NCFaoRJ/pmmehsdzLTdRin18KX9K+lKS2tshlnqDbHMb1KefEG1Cjkf0vr19bOUmSrGPL+VPSV9KUksLrPxWgtqvNMKVNTAtTUM2hZsQraUFOjqitnGz6LmjI4qPpdxEKfbxpfypI1dK0vD57TOJT4vTva57ciolUiaydeTmOveOyIRKdCaGzHvzx4N/HDXha0I0kfwp6UvRpVu1KhvDNCGayBgp6UvRpVu1KhM154iMjzpypehy7YxVc47I+CnpS9FlWp2qYWoD8WlxDCM+LU7HBR1qzhEZJyV9mXCJzgSN12W+kzbTqlUblm2ge103g1cO0r2uWwlfpACU9GVCJTtpe/ZnvpO2ZUELHRd06KpeZBJonL5MqMbrGtOOxlGHrMjEGfeEa2Y23czuNLPnzOxZM/uomR1tZlvN7IXwPCOUNTP7jpl1mdlTZnZyyn5WhvIvmNnKwpyelLJMnbSFupNWRPKTa/POBuA+d/8QcBLwLLAe2Obu84Ft4T3AMmB+eLQC1wOY2dHAlcBpwKnAlck/FFK5MnXSZoqLyMQaNemb2XuAs4AbANz9oLu/CSwHNoVim4ALw+vlwC1hLv9HgelmdixwHrDV3fe5+xvAVmBpQc9GSk6mTloNvRQpjlyu9N8P7AFuMrNfmdkPzOxdwHvdfRdAeD4mlJ8NvJyyfV+IZYoPYWatZrbdzLbv2bMn7xOS0qJOWpHSkssduVOAk4EvufsvzWwDh5ty0rE0Mc8SHxpw7wA6IOrIzaF+UuJaFrQoyYuUiFyu9PuAPnf/ZXh/J9EfgddCsw3heXdK+eNStp8DvJolLiIik2TUpO/uvwNeNrPjQ6gZeAbYAiRH4KwEfhJebwEuDqN4Tgf2h+af+4ElZjYjdOAuCTEREZkkuU649iUgYWb1wIvAKqI/GHeY2eeBXuBToey9wPlAF3AglMXd95nZ14DHQ7lr3H1fQc5CRERyopuzREQqzLhvzhIRkcqgpC8iUkWU9EVEqoiSvohIFVHSr0KjzW8vIpVLa+RWmXSLkLfe0wqgu2ZFqoCu9KtA6pX9yrtXjliE/ED/Adq2tRWpdiIymXSlX+GGX9kP+EDacprfXqQ66Eq/wrVtaxtxZZ+O5rcXqQ5K+hUulyt4zW8vUj2U9CtIulE5ma7ga61W89uLVCG16VeITKNyVp60kk1PbhrSxBOriynRi1QpXelXiHRt9wf6D3DvC/dq5SoROURX+mUo0ZmgbVsbvft7mTttLu3N7Rnb7nv392rlKhE5REm/zGRqxjl66tHs/fPeEeU1KkdEUql5p8xkasaBqK0+lUbliMhwSvolLN1onEzNOPv+vE9t9yIyKq2cVaKGN+NAdOU+dcrUtM048Wlxutd1T2INRaRUjXvlLDPrNrNOM9tpZttD7CozeyXEdprZ+SnlrzCzLjN73szOS4kvDbEuM1s/3hOrRMmr+8/d9Tk144hIweXTvHO2uy8c9tfj2hBb6O73ApjZCcAK4MPAUmCjmdWaWS3wXWAZcALwmVBWguTVfc/+noxl1IwjIuMxEaN3lgO3ufvbwEtm1gWcGj7rcvcXAczstlD2mQmoQ1nKZZ6cudPmagimiIxZrlf6DjxgZjvMrDUl/kUze8rMbjSzGSE2G3g5pUxfiGWKD2FmrWa23cy279mzJ+cTKTf5dNImqRlHRMYr16R/prufTNQ0c5mZnQVcD3wAWAjsAr4dylqa7T1LfGjAvcPdm9y9adasWTlWr7ykNuM4PmSsfSZqxhGRQsipecfdXw3Pu83sbuBUd/+/yc/N7PvAT8PbPuC4lM3nAK+G15niVSXTWPupU6YSq4tpnhwRmTCjXumb2bvM7Kjka2AJ8GszOzal2CeAX4fXW4AVZnaEmc0D5gOPAY8D881snpnVE3X2bincqZQPjbUXkWLJpXnnvcAvzOxJouT9M3e/D/hGGMb5FHA2cDmAuz8N3EHUQXsfcJm7D7j7O8AXgfuBZ4E7QtkJlUhAYyPU1ETPiTzWAB/LtqnbvPujCWr/vhG7qoYp/9DImuujHWSaGmHutLnwVAtc1w1XD/LHr3ez9uyWEcfPt17Dy69ZM/bvZCzG8xuISIG5e8k+TjnlFB+PzZvdYzF3OPyIxaL4RGw7ZJsFm51/ijlXcfjRFvPVGzf76o2bnbaRnzVfvnnEMYcff/Xq/OqV7jzS7TeX72QsxvMbiMjYANs9Q16t6DtyGxuhJ82Q93gcursLv21jI/S8JwHNbTCtJ23Xde0f48z5UXdKuV7YPxe2tVP7TAsD6ZewPbx9LWnLZKpXpvPIdfvxGs9vICJjk+2O3IpO+jU10bXlcGYwOFj4be0vE3BBK9RnGWvvhl0zmHbf45GpXpnOI9ftx2s8v4GIjM24p2EoV3MzzCqcKT7ebWvPa8ue8IHaP83NuI/a2tHrlanMeM41n3L5Gm+9RKSwKjrpt7dDbOg0NcRiUXwith149yiLkPfHaH1/e8Z9t7aOjOdSJlu90h0r3X5z+U7GYjy/gYhMgEyN/aXwGG9HrnvUYRiPu5tFz/l0IOa7bfza+NDO2eTjSrz2K3FfvfHwDjLtOzXe0BA9spXJpV7Dy69ePfbvZCzG8xuISP6o1o7cyZZpOmSNtReRyVS1bfqTrWVBi26uEpGSpiv9LNb8bA0dOzoY8AFqrZbWU1rZ+LGNRauPiEgusl3pa2H0DNb8bA3Xb7/+0PsBHzj0XolfRMqVmncy6NjRkVdcRKQcKOlnMODpb43NFBcRKQdK+hnUWvq7oDLFRUTKgZJ+Bq2ntOYVFxEpB+rIzSDZWavROyJSSTRkU0SkwlT1zVnpFiAXEalWFd28M3xahOQC5IDukhWRqpTTlb6ZdYelEXea2fYQO9rMtprZC+F5RoibmX3HzLrM7CkzOzllPytD+RfMbOXEnNLhq/vP3fW5tAuQt21rm6hDi4iUtHyad85294Up7UTrgW3uPh/YFt4DLCNaDH0+0ApcD9EfCeBK4DTgVODK5B+KQkpe3ffsz7xcVKaFyUVEKt142vSXA5vC603AhSnxW8IMn48C083sWOA8YKu773P3N4CtwNJxHD+ttm1tI67uh8u0MLmISKXLNek78ICZ7TCz5ED197r7LoDwfEyIzwZeTtm2L8QyxQtqtKv4WF2M9mat4CEi1SnXjtwz3f1VMzsG2Gpmz2Upm2Y5cDxLfOjG0R+VVoC5Y1hTb+60uRmbduLT4rQ3t6sTV6dQSz8AAAhnSURBVESqVk5X+u7+anjeDdxN1Cb/Wmi2ITzvDsX7gONSNp8DvJolPvxYHe7e5O5Ns2bNyu9sgPbmdmJ1Q9fni9XF2PzXm+le162ELyJVbdSkb2bvMrOjkq+BJcCvgS1AcgTOSuAn4fUW4OIwiud0YH9o/rkfWGJmM0IH7pIQKygtZCIiklkuzTvvBe42s2T5/3L3+8zsceAOM/s80At8KpS/Fzgf6AIOAKsA3H2fmX0NeDyUu8bd9xXsTFK0LGhRkhcRSUPTMIiIVJiqnoahmiQS0NgINTXRc0IzTojIMBU9DUM1SSSgtRUOhFsUenqi9wAtaukSkUBX+hWire1wwk86cCCKi4gkKelXiN4M96RliotIdVLSrxCZ7mMbw/1tIlLBlPQrRHs7xIbek0YsFsVFRJKU9CtESwt0dEA8DmbRc0eHOnFFZCiN3qkgLS1K8iKSna70RUSqiJK+iEgVUdIXEakiSvoiIlVESV9EpIoo6YuIVBElfRGRKqKkLyJSRZT0RUSqiJK+iEgVyTnpm1mtmf3KzH4a3t9sZi+Z2c7wWBjiZmbfMbMuM3vKzE5O2cdKM3shPFZmOpaIiEyMfObeWQs8C7wnJfYP7n7nsHLLgPnhcRpwPXCamR0NXAk0AQ7sMLMt7v7GWCsvIiL5yelK38zmAB8DfpBD8eXALR55FJhuZscC5wFb3X1fSPRbgaVjrHdWhVorVmvOikilybV55zrgH4HBYfH20IRzrZkdEWKzgZdTyvSFWKb4EGbWambbzWz7nj17cqzeYcm1Ynt6wP3wWrH5JuxC7UdEpJSMmvTN7P8Au919x7CPrgA+BPxv4Gjgq8lN0uzGs8SHBtw73L3J3ZtmzZo1WvVGKNRasVpzVkQqUS5X+mcCHzezbuA24Bwz2+zuu0ITztvATcCpoXwfcFzK9nOAV7PEC6pQa8VqzVkRqUSjJn13v8Ld57h7I7ACeMjdPxfa6TEzAy4Efh022QJcHEbxnA7sd/ddwP3AEjObYWYzgCUhVlCFWitWa86KSCUazzj9hJl1Ap3ATODrIX4v8CLQBXwfWAPg7vuArwGPh8c1IVZQhVorVmvOikglMvcRzeolo6mpybdv3573dolE1Pbe2xtdmbe3j20ZwULtR0RkMpnZDndvSvtZJSZ9EZFqli3paxoGEZEqoqQvIlJFlPRFRKqIkr6ISBVR0hcRqSIlPXrHzPYAPWPcfCbwegGrM9lU/+Ir93NQ/YurmPWPu3vaeWxKOumPh5ltzzRkqRyo/sVX7ueg+hdXqdZfzTsiIlVESV9EpIpUctLvKHYFxkn1L75yPwfVv7hKsv4V26YvIiIjVfKVvoiIDKOkLyJSRcoq6ZtZt5l1mtlOM9seYkeb2VYzeyE8zwhxM7PvmFlXWMf35JT9rAzlXzCzlUWu/1Vm9kqI7TSz81PKXxHq/7yZnZcSXxpiXWa2fhLrP93M7jSz58zsWTP7aDl9/1nOoSx+AzM7PqWOO83s92a2rlx+gyz1L4vvPxz3cjN72sx+bWY/NLMjzWyemf0yfJe3m1l9KHtEeN8VPm8c7bwmhbuXzQPoBmYOi30DWB9erwf+Pbw+H/g50dq8pwO/DPGjiRZ5ORqYEV7PKGL9rwK+kqbsCcCTwBHAPOC3QG14/BZ4P1AfypwwSfXfBPxteF0PTC+n7z/LOZTNb5BSt1rgd0C83H6DNPUvi+8fmA28BEwN7+8ALgnPK0Lse8Dq8HoN8L3wegVwe7bzmqzvvqyu9DNYTvQ/MuH5wpT4LR55FJhu0RKP5wFb3X2fu78BbAWWTnalc7AcuM3d33b3l4hWIjs1PLrc/UV3P0i0bvHyia6Mmb0HOAu4AcDdD7r7m5TR95/lHDIpqd9gmGbgt+7eQxn9BilS659JKX7/U4CpZjYFiAG7gHOAO8Pnw7//5O9yJ9BsZkbm85oU5Zb0HXjAzHaYWWuIvdejNXgJz8eE+Gzg5ZRt+0IsU3wypKs/wBfDP79vTP7TPEs9i1X/9wN7gJvM7Fdm9gMzexfl9f1nOgcoj98g1Qrgh+F1Of0GSan1hzL4/t39FeBbQC9Rst8P7ADedPd30tTlUD3D5/uBhmLVP6nckv6Z7n4ysAy4zMzOylLW0sQ8S3wypKv/9cAHgIVE/yF9O5QttfpPAU4Grnf3jwB/ImpKyKTU6g+Zz6FcfgMAQpvxx4EfjVY0TawU618W33/4Y7ScqEnmfcC7iP5fzlSXkqp/UlklfXd/NTzvBu4m+ifRa+GfrITn3aF4H3BcyuZzgFezxCdcuvq7+2vuPuDug0QLySf/mVdq9e8D+tz9l+H9nUQJtGy+fzKcQxn9BknLgCfc/bXwvpx+AxhW/zL6/s8FXnL3Pe7eD9wFnEHUbDYlTV0O1TN8Pg3YR5G//7JJ+mb2LjM7KvkaWAL8GtgCJEcfrAR+El5vAS4OIxhOB/aHf/reDywxsxnhL/eSECtK/ZP/swafCOeUrP+KMAJgHjAfeAx4HJgfRgzUE/0zectE19/dfwe8bGbHh1Az8Axl8v1nO4dy+Q1SfIahTSNl8xsEQ+pfRt9/L3C6mcVC23zy/4H/AT4Zygz//pO/yyeBhzzqyc10XpNjsnqMx/sgao99MjyeBtpCvAHYBrwQno8OcQO+S9Qz3gk0pezrUqLOky5gVZHrf2uo31NE/zEcm7JNW6j/88CylPj5wG/CZ22T+BssBLaHuv6YaORHWXz/o5xDOf0GMWAvMC0lVja/QYb6l9P3fzXwHNEfpluJRuC8nyhpdxE1WR0Ryh4Z3neFz98/2nlNxkPTMIiIVJGyad4REZHxU9IXEakiSvoiIlVESV9EpIoo6YuIVBElfRGRKqKkLyJSRf4/xVgz670lypMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.automl.core.shared import constants\n",
    "from azureml.automl.runtime.shared.score import scoring\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# use automl metrics module\n",
    "scores = scoring.score_regression(\n",
    "    y_test=df_all[target_column_name],\n",
    "    y_pred=df_all['predicted'],\n",
    "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
    "\n",
    "print(\"[Test data scores]\\n\")\n",
    "for key, value in scores.items():    \n",
    "    print('{}:   {:.3f}'.format(key, value))\n",
    "    \n",
    "# Plot outputs\n",
    "%matplotlib inline\n",
    "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
    "test_test = plt.scatter(df_all[target_column_name], df_all[target_column_name], color='g')\n",
    "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/retkowsky/images/blob/master/Powered-by-MS-Azure-logo-v2.png?raw=true\" height=\"300\" width=\"300\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "erwright"
   }
  ],
  "categories": [
   "how-to-use-azureml",
   "automated-machine-learning"
  ],
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}