Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example of versioning ML experiments using DVC\n",
    "\n",
    "This notebook aims to be a guideline for versioning your ML projects using DVC, from a Jupyter notebook.\n",
    "\n",
    "This notebook allows you to experiment as much as you like, and when you are in a state that you would like to preserve for future reference as a git commit, use the DVC cells to version all your relevant files. \n",
    "\n",
    "The cells marked with a green markdown box are responsible for creating a snapshot of your raw data, processed data, and trained models.\n",
    "\n",
    "This snapshot is implemented as md5 hashes of the respective files saved as text in the `.dvc` files. The hashes in the .dvc files will be part of the git commit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and global declarations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn import datasets\n",
    "from sklearn import preprocessing\n",
    "from sklearn import metrics\n",
    "from sklearn import model_selection\n",
    "from sklearn.linear_model import LinearRegression\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import joblib\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Download and version raw data</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[0m                                                                            "
     ]
    }
   ],
   "source": [
    "raw_data = datasets.fetch_california_housing(data_home=\"data/raw\")\n",
    "# Save the raw input data for reproducibility\n",
    "!dvc commit -f raw.dvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_dataframe(X, y):\n",
    "    return pd.concat([\n",
    "            pd.DataFrame(data=X, columns=raw_data.feature_names),\n",
    "            pd.DataFrame(data=y, columns=['Value'])\n",
    "        ],\n",
    "        axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.3252</td>\n",
       "      <td>41.0</td>\n",
       "      <td>6.984127</td>\n",
       "      <td>1.023810</td>\n",
       "      <td>322.0</td>\n",
       "      <td>2.555556</td>\n",
       "      <td>37.88</td>\n",
       "      <td>-122.23</td>\n",
       "      <td>4.526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.3014</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6.238137</td>\n",
       "      <td>0.971880</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>2.109842</td>\n",
       "      <td>37.86</td>\n",
       "      <td>-122.22</td>\n",
       "      <td>3.585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.2574</td>\n",
       "      <td>52.0</td>\n",
       "      <td>8.288136</td>\n",
       "      <td>1.073446</td>\n",
       "      <td>496.0</td>\n",
       "      <td>2.802260</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.24</td>\n",
       "      <td>3.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.6431</td>\n",
       "      <td>52.0</td>\n",
       "      <td>5.817352</td>\n",
       "      <td>1.073059</td>\n",
       "      <td>558.0</td>\n",
       "      <td>2.547945</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.8462</td>\n",
       "      <td>52.0</td>\n",
       "      <td>6.281853</td>\n",
       "      <td>1.081081</td>\n",
       "      <td>565.0</td>\n",
       "      <td>2.181467</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.422</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "0  8.3252      41.0  6.984127   1.023810       322.0  2.555556     37.88   \n",
       "1  8.3014      21.0  6.238137   0.971880      2401.0  2.109842     37.86   \n",
       "2  7.2574      52.0  8.288136   1.073446       496.0  2.802260     37.85   \n",
       "3  5.6431      52.0  5.817352   1.073059       558.0  2.547945     37.85   \n",
       "4  3.8462      52.0  6.281853   1.081081       565.0  2.181467     37.85   \n",
       "\n",
       "   Longitude  Value  \n",
       "0    -122.23  4.526  \n",
       "1    -122.22  3.585  \n",
       "2    -122.24  3.521  \n",
       "3    -122.25  3.413  \n",
       "4    -122.25  3.422  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_df = to_dataframe(raw_data.data, raw_data.target)\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.870671</td>\n",
       "      <td>28.639486</td>\n",
       "      <td>5.429000</td>\n",
       "      <td>1.096675</td>\n",
       "      <td>1425.476744</td>\n",
       "      <td>3.070655</td>\n",
       "      <td>35.631861</td>\n",
       "      <td>-119.569704</td>\n",
       "      <td>2.068558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.899822</td>\n",
       "      <td>12.585558</td>\n",
       "      <td>2.474173</td>\n",
       "      <td>0.473911</td>\n",
       "      <td>1132.462122</td>\n",
       "      <td>10.386050</td>\n",
       "      <td>2.135952</td>\n",
       "      <td>2.003532</td>\n",
       "      <td>1.153956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.499900</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.846154</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.692308</td>\n",
       "      <td>32.540000</td>\n",
       "      <td>-124.350000</td>\n",
       "      <td>0.149990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.563400</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>4.440716</td>\n",
       "      <td>1.006079</td>\n",
       "      <td>787.000000</td>\n",
       "      <td>2.429741</td>\n",
       "      <td>33.930000</td>\n",
       "      <td>-121.800000</td>\n",
       "      <td>1.196000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.534800</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>5.229129</td>\n",
       "      <td>1.048780</td>\n",
       "      <td>1166.000000</td>\n",
       "      <td>2.818116</td>\n",
       "      <td>34.260000</td>\n",
       "      <td>-118.490000</td>\n",
       "      <td>1.797000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.743250</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>6.052381</td>\n",
       "      <td>1.099526</td>\n",
       "      <td>1725.000000</td>\n",
       "      <td>3.282261</td>\n",
       "      <td>37.710000</td>\n",
       "      <td>-118.010000</td>\n",
       "      <td>2.647250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>15.000100</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>141.909091</td>\n",
       "      <td>34.066667</td>\n",
       "      <td>35682.000000</td>\n",
       "      <td>1243.333333</td>\n",
       "      <td>41.950000</td>\n",
       "      <td>-114.310000</td>\n",
       "      <td>5.000010</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             MedInc      HouseAge      AveRooms     AveBedrms    Population  \\\n",
       "count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   \n",
       "mean       3.870671     28.639486      5.429000      1.096675   1425.476744   \n",
       "std        1.899822     12.585558      2.474173      0.473911   1132.462122   \n",
       "min        0.499900      1.000000      0.846154      0.333333      3.000000   \n",
       "25%        2.563400     18.000000      4.440716      1.006079    787.000000   \n",
       "50%        3.534800     29.000000      5.229129      1.048780   1166.000000   \n",
       "75%        4.743250     37.000000      6.052381      1.099526   1725.000000   \n",
       "max       15.000100     52.000000    141.909091     34.066667  35682.000000   \n",
       "\n",
       "           AveOccup      Latitude     Longitude         Value  \n",
       "count  20640.000000  20640.000000  20640.000000  20640.000000  \n",
       "mean       3.070655     35.631861   -119.569704      2.068558  \n",
       "std       10.386050      2.135952      2.003532      1.153956  \n",
       "min        0.692308     32.540000   -124.350000      0.149990  \n",
       "25%        2.429741     33.930000   -121.800000      1.196000  \n",
       "50%        2.818116     34.260000   -118.490000      1.797000  \n",
       "75%        3.282261     37.710000   -118.010000      2.647250  \n",
       "max     1243.333333     41.950000   -114.310000      5.000010  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_X, test_X, train_y, test_y = model_selection.train_test_split(raw_df[raw_df.columns[:-1]], raw_df['Value'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6374</th>\n",
       "      <td>9.4667</td>\n",
       "      <td>32.0</td>\n",
       "      <td>7.325758</td>\n",
       "      <td>1.037879</td>\n",
       "      <td>1558.0</td>\n",
       "      <td>2.950758</td>\n",
       "      <td>34.17</td>\n",
       "      <td>-118.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13711</th>\n",
       "      <td>3.0587</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.284007</td>\n",
       "      <td>1.272978</td>\n",
       "      <td>2382.0</td>\n",
       "      <td>2.189338</td>\n",
       "      <td>34.08</td>\n",
       "      <td>-117.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5543</th>\n",
       "      <td>5.6110</td>\n",
       "      <td>36.0</td>\n",
       "      <td>5.727891</td>\n",
       "      <td>1.024943</td>\n",
       "      <td>996.0</td>\n",
       "      <td>2.258503</td>\n",
       "      <td>33.98</td>\n",
       "      <td>-118.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4263</th>\n",
       "      <td>1.3157</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.911826</td>\n",
       "      <td>1.151854</td>\n",
       "      <td>3049.0</td>\n",
       "      <td>2.133660</td>\n",
       "      <td>34.10</td>\n",
       "      <td>-118.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11504</th>\n",
       "      <td>4.7773</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3.535461</td>\n",
       "      <td>0.929078</td>\n",
       "      <td>531.0</td>\n",
       "      <td>1.882979</td>\n",
       "      <td>33.74</td>\n",
       "      <td>-118.10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "6374   9.4667      32.0  7.325758   1.037879      1558.0  2.950758     34.17   \n",
       "13711  3.0587       5.0  5.284007   1.272978      2382.0  2.189338     34.08   \n",
       "5543   5.6110      36.0  5.727891   1.024943       996.0  2.258503     33.98   \n",
       "4263   1.3157      43.0  1.911826   1.151854      3049.0  2.133660     34.10   \n",
       "11504  4.7773      37.0  3.535461   0.929078       531.0  1.882979     33.74   \n",
       "\n",
       "       Longitude  \n",
       "6374     -118.02  \n",
       "13711    -117.21  \n",
       "5543     -118.40  \n",
       "4263     -118.33  \n",
       "11504    -118.10  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize feature columns by training data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6374</th>\n",
       "      <td>2.947156</td>\n",
       "      <td>0.264726</td>\n",
       "      <td>0.741246</td>\n",
       "      <td>-0.122564</td>\n",
       "      <td>0.112211</td>\n",
       "      <td>-0.010226</td>\n",
       "      <td>-0.684439</td>\n",
       "      <td>0.775534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13711</th>\n",
       "      <td>-0.426527</td>\n",
       "      <td>-1.884132</td>\n",
       "      <td>-0.059881</td>\n",
       "      <td>0.355262</td>\n",
       "      <td>0.818678</td>\n",
       "      <td>-0.080542</td>\n",
       "      <td>-0.726595</td>\n",
       "      <td>1.180526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5543</th>\n",
       "      <td>0.917208</td>\n",
       "      <td>0.583075</td>\n",
       "      <td>0.114287</td>\n",
       "      <td>-0.148854</td>\n",
       "      <td>-0.369627</td>\n",
       "      <td>-0.074154</td>\n",
       "      <td>-0.773436</td>\n",
       "      <td>0.585537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4263</th>\n",
       "      <td>-1.344181</td>\n",
       "      <td>1.140186</td>\n",
       "      <td>-1.383034</td>\n",
       "      <td>0.109085</td>\n",
       "      <td>1.390540</td>\n",
       "      <td>-0.085683</td>\n",
       "      <td>-0.717227</td>\n",
       "      <td>0.620537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11504</th>\n",
       "      <td>0.478282</td>\n",
       "      <td>0.662662</td>\n",
       "      <td>-0.745964</td>\n",
       "      <td>-0.343695</td>\n",
       "      <td>-0.768301</td>\n",
       "      <td>-0.108833</td>\n",
       "      <td>-0.885854</td>\n",
       "      <td>0.735534</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  \\\n",
       "6374   2.947156  0.264726  0.741246  -0.122564    0.112211 -0.010226   \n",
       "13711 -0.426527 -1.884132 -0.059881   0.355262    0.818678 -0.080542   \n",
       "5543   0.917208  0.583075  0.114287  -0.148854   -0.369627 -0.074154   \n",
       "4263  -1.344181  1.140186 -1.383034   0.109085    1.390540 -0.085683   \n",
       "11504  0.478282  0.662662 -0.745964  -0.343695   -0.768301 -0.108833   \n",
       "\n",
       "       Latitude  Longitude  \n",
       "6374  -0.684439   0.775534  \n",
       "13711 -0.726595   1.180526  \n",
       "5543  -0.773436   0.585537  \n",
       "4263  -0.717227   0.620537  \n",
       "11504 -0.885854   0.735534  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = preprocessing.StandardScaler()\n",
    "train_X_scaled = pd.DataFrame(scaler.fit_transform(train_X), index=train_X.index, columns=train_X.columns)\n",
    "train_X_scaled.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12916</th>\n",
       "      <td>0.112904</td>\n",
       "      <td>-0.690322</td>\n",
       "      <td>0.230982</td>\n",
       "      <td>-0.199550</td>\n",
       "      <td>0.774096</td>\n",
       "      <td>-0.023481</td>\n",
       "      <td>1.409347</td>\n",
       "      <td>-0.864434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3938</th>\n",
       "      <td>0.929580</td>\n",
       "      <td>0.503487</td>\n",
       "      <td>0.013131</td>\n",
       "      <td>-0.316807</td>\n",
       "      <td>0.000754</td>\n",
       "      <td>-0.000943</td>\n",
       "      <td>-0.661018</td>\n",
       "      <td>0.495539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11816</th>\n",
       "      <td>0.438585</td>\n",
       "      <td>-1.167846</td>\n",
       "      <td>0.277412</td>\n",
       "      <td>-0.276662</td>\n",
       "      <td>-0.246167</td>\n",
       "      <td>0.002580</td>\n",
       "      <td>1.568606</td>\n",
       "      <td>-0.744436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14983</th>\n",
       "      <td>-0.020716</td>\n",
       "      <td>-1.247434</td>\n",
       "      <td>0.294054</td>\n",
       "      <td>-0.190097</td>\n",
       "      <td>-0.607117</td>\n",
       "      <td>0.026106</td>\n",
       "      <td>-1.363631</td>\n",
       "      <td>1.290523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9272</th>\n",
       "      <td>0.793222</td>\n",
       "      <td>-0.690322</td>\n",
       "      <td>0.650213</td>\n",
       "      <td>-0.136036</td>\n",
       "      <td>-0.708286</td>\n",
       "      <td>-0.034950</td>\n",
       "      <td>1.170459</td>\n",
       "      <td>-1.509421</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  \\\n",
       "12916  0.112904 -0.690322  0.230982  -0.199550    0.774096 -0.023481   \n",
       "3938   0.929580  0.503487  0.013131  -0.316807    0.000754 -0.000943   \n",
       "11816  0.438585 -1.167846  0.277412  -0.276662   -0.246167  0.002580   \n",
       "14983 -0.020716 -1.247434  0.294054  -0.190097   -0.607117  0.026106   \n",
       "9272   0.793222 -0.690322  0.650213  -0.136036   -0.708286 -0.034950   \n",
       "\n",
       "       Latitude  Longitude  \n",
       "12916  1.409347  -0.864434  \n",
       "3938  -0.661018   0.495539  \n",
       "11816  1.568606  -0.744436  \n",
       "14983 -1.363631   1.290523  \n",
       "9272   1.170459  -1.509421  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_X_scaled = pd.DataFrame(scaler.transform(test_X), index=test_X.index, columns=test_X.columns)\n",
    "test_X_scaled.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.concat([train_X_scaled, train_y], axis=1)\n",
    "test_df = pd.concat([test_X_scaled, test_y], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Optional: Version the processed data with DVC for efficiency and/or reproducibility</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0% Transferring|                                   |0/4 [00:00<?,     ?file/s]\n",
      "!\u001B[A\n",
      "  0%|          |california_households_scaler.pkl   0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "                                                                                \u001B[A\n",
      "!\u001B[A\n",
      "  0%|          |california_households_train.csv    0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "\n",
      "  0%|          |california_households_test.csv     0.00/? [00:00<?,        ?B/s]\u001B[A\u001B[A\n",
      "\n",
      "                                                                                \u001B[A\u001B[A\n",
      "                                                                                \u001B[A\n",
      "!\u001B[A\n",
      "  0%|          |memory://.UjZXBmfvLgdiWjagdxFmGC.tm0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "  0%|          |memory://.UjZXBmfvLgdiWjagdxFmGC.0.00/273 [00:00<?,        ?B/s]\u001B[A\n",
      "\n",
      "!\u001B[A\u001B[A\n",
      "\n",
      "  0%|          |.UjZXBmfvLgdiWjagdxFmGC.tmp        0.00/? [00:00<?,        ?B/s]\u001B[A\u001B[A\n",
      "                                                                                \u001B[A\n",
      "\n",
      "Updating lock file 'dvc.lock'                                                   \u001B[A\u001B[A\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "os.makedirs('data/processed/')\n",
    "train_df.to_csv('data/processed/california_households_train.csv', index_label='Index')\n",
    "test_df.to_csv('data/processed/california_households_test.csv', index_label='Index')\n",
    "joblib.dump(scaler, 'data/processed/california_households_scaler.pkl')\n",
    "!dvc commit -f process_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this cell to reload processed data, after switching branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('data/processed/california_households_train.csv', index_col=0)\n",
    "test_df = pd.read_csv('data/processed/california_households_train.csv', index_col=0)\n",
    "scaler = joblib.load('data/processed/california_households_scaler.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearRegression</label><div class=\"sk-toggleable__content\"><pre>LinearRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LinearRegression()\n",
    "X = train_df[train_df.columns[:-1]]\n",
    "y = train_df['Value']\n",
    "model.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Save the trained model for reproducibility</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0% Transferring|                                   |0/2 [00:00<?,     ?file/s]\n",
      "!\u001B[A\n",
      "  0%|          |california_households.pkl          0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "                                                                                \u001B[A\n",
      "!\u001B[A\n",
      "  0%|          |memory://.UrehUoTg9RCLnhuQcrzhfe.tm0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "  0%|          |memory://.UrehUoTg9RCLnhuQcrzhfe0.00/85.0 [00:00<?,        ?B/s]\u001B[A\n",
      "\n",
      "!\u001B[A\u001B[A\n",
      "\n",
      "  0%|          |.UrehUoTg9RCLnhuQcrzhfe.tmp        0.00/? [00:00<?,        ?B/s]\u001B[A\u001B[A\n",
      "                                                                                \u001B[A\n",
      "\n",
      "Updating lock file 'dvc.lock'                                                   \u001B[A\u001B[A\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "os.makedirs('models')\n",
    "joblib.dump(model, 'models/california_households.pkl')\n",
    "!dvc commit -f train_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this cell to reload the model, after switching branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = joblib.load('models/california_households.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>R2</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "      <th>median_absolute_error</th>\n",
       "      <th>loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.601755</td>\n",
       "      <td>0.537834</td>\n",
       "      <td>0.535798</td>\n",
       "      <td>0.42066</td>\n",
       "      <td>0.535798</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         R2       MAE       MSE  median_absolute_error      loss\n",
       "0  0.601755  0.537834  0.535798                0.42066  0.535798"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = model.predict(test_df[test_df.columns[:-1]])\n",
    "truth = test_df['Value']\n",
    "metrics_dict = {}\n",
    "metrics_dict['R2'] = metrics.r2_score(truth, predictions)\n",
    "metrics_dict['MAE'] = metrics.mean_absolute_error(truth, predictions)\n",
    "metrics_dict['MSE'] = metrics.mean_squared_error(truth, predictions)\n",
    "metrics_dict['median_absolute_error'] = metrics.median_absolute_error(truth, predictions)\n",
    "metrics_dict['loss'] = metrics_dict['MSE']\n",
    "pd.DataFrame(metrics_dict, index=[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Save the computed metrics for easy display in DVC and DAGsHub</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0% Transferring|                                   |0/1 [00:00<?,     ?file/s]\n",
      "!\u001B[A\n",
      "  0%|          |metrics.json                       0.00/? [00:00<?,        ?B/s]\u001B[A\n",
      "Updating lock file 'dvc.lock'                                                   \u001B[A\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "with open('metrics/metrics.json', 'w') as f:\n",
    "    json.dump(metrics_dict, f, indent=2)\n",
    "!dvc commit -f eval_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Versioning section - use the following cells to create a full commit of your current state</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make sure all data and models are committed to DVC\n",
    "The output of the following cell should be: `Pipeline is up to date. Nothing to reproduce.`\n",
    "\n",
    "If you get something else, then maybe you forgot to `dvc commit` earlier in the notebook.\n",
    "We recommend to make sure that the current contents in the data and models directories are to your liking,\n",
    "and if so, use the commit cell below to automatically commit all current files to DVC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data and pipelines are up to date.                                              \n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "!dvc status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[0m                                                                            "
     ]
    }
   ],
   "source": [
    "# Use this if dvc status is not up-to-date and you're sure the current state is OK.\n",
    "!dvc commit -f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}