spark-california-housing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "GEqQaHzsZ_bA",
   "metadata": {
    "id": "GEqQaHzsZ_bA"
   },
   "source": [
    "# Predicting median house value using PySpark and MLlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd50f402-afc1-43eb-92fd-1f79e8b0438a",
   "metadata": {},
   "source": [
    "**The first objective** is to train Linear Regression model on California 1990 housing data using PySpark and MLlib. Predictions are made on test data. Model's quality is estimated using RMSE, MAE and R2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1d851f-4177-44b8-8cb9-0a0dd827e91c",
   "metadata": {},
   "source": [
    "**The second objective** is to compare two models trained on different feature sets:\n",
    "* a model trained using all features\n",
    "* a model trained usig only numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c600882-7b82-4368-8212-a979a0eb95a0",
   "metadata": {},
   "source": [
    "**NOTE:**\n",
    "\n",
    "For this project a modified dataset was used, based on this data: https://www.kaggle.com/datasets/camnugent/california-housing-prices. The main purpose of this project is to get expierenced with distributed file system and evaluating technologies so score values and models used are not important here. Actual dataset is not available for sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc56f64e-95e3-4cea-bc4e-eb1ded180e7c",
   "metadata": {},
   "source": [
    "Target feature is `median_house_proximity`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fd450b3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.types import *\n",
    "import pyspark.sql.functions as F\n",
    "\n",
    "from pyspark.ml import Pipeline\n",
    "\n",
    "from pyspark.ml.feature import StringIndexer, VectorAssembler, StandardScaler, OneHotEncoder\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "    \n",
    "# random state\n",
    "RND = 1337"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "85102b39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.0.2'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking pyspark version\n",
    "pyspark.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5e26e3a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialising local pyspark session\n",
    "spark = SparkSession.builder \\\n",
    "                    .master(\"local\") \\\n",
    "                    .appName(\"ML California Housing\") \\\n",
    "                    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53871372",
   "metadata": {
    "id": "53871372"
   },
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f598a8b7",
   "metadata": {},
   "source": [
    "### Open the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "df92e3ec",
   "metadata": {
    "id": "df92e3ec"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# open the dataset\n",
    "df_housing = spark.read.csv('./datasets/housing.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6e940f79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- longitude: double (nullable = true)\n",
      " |-- latitude: double (nullable = true)\n",
      " |-- housing_median_age: double (nullable = true)\n",
      " |-- total_rooms: double (nullable = true)\n",
      " |-- total_bedrooms: double (nullable = true)\n",
      " |-- population: double (nullable = true)\n",
      " |-- households: double (nullable = true)\n",
      " |-- median_income: double (nullable = true)\n",
      " |-- median_house_value: double (nullable = true)\n",
      " |-- ocean_proximity: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# dataset scheme\n",
    "df_housing.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c24a60-4d63-4e15-8970-bac5318f26f6",
   "metadata": {},
   "source": [
    "All columns' types are numeric except for `ocean_proximity` which is a string type. Columns `longitude` and `latitude` mean coordinates; theese features are individual and don't affect target directly, so we will remove them from processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "413c7e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+-----------+--------------+----------+----------+-------------+------------------+---------------+\n",
      "|housing_median_age|total_rooms|total_bedrooms|population|households|median_income|median_house_value|ocean_proximity|\n",
      "+------------------+-----------+--------------+----------+----------+-------------+------------------+---------------+\n",
      "|              41.0|      880.0|         129.0|     322.0|     126.0|       8.3252|          452600.0|       NEAR BAY|\n",
      "|              21.0|     7099.0|        1106.0|    2401.0|    1138.0|       8.3014|          358500.0|       NEAR BAY|\n",
      "|              52.0|     1467.0|         190.0|     496.0|     177.0|       7.2574|          352100.0|       NEAR BAY|\n",
      "+------------------+-----------+--------------+----------+----------+-------------+------------------+---------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# removing longitude and latitude columns\n",
    "df_housing = df_housing.select([col for col in df_housing.columns if col not in ['longitude', 'latitude']])\n",
    "df_housing.show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79cfaaf6-e0fa-4b32-9bd8-763fdf8dc588",
   "metadata": {},
   "source": [
    "Check statistics and missing values using `describe()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d9ebaed7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>summary</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>count</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20433</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean</td>\n",
       "      <td>28.639486434108527</td>\n",
       "      <td>2635.7630813953488</td>\n",
       "      <td>537.8705525375618</td>\n",
       "      <td>1425.4767441860465</td>\n",
       "      <td>499.5396802325581</td>\n",
       "      <td>3.8706710029070246</td>\n",
       "      <td>206855.81690891474</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>stddev</td>\n",
       "      <td>12.58555761211163</td>\n",
       "      <td>2181.6152515827944</td>\n",
       "      <td>421.38507007403115</td>\n",
       "      <td>1132.46212176534</td>\n",
       "      <td>382.3297528316098</td>\n",
       "      <td>1.899821717945263</td>\n",
       "      <td>115395.61587441359</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>min</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4999</td>\n",
       "      <td>14999.0</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>max</td>\n",
       "      <td>52.0</td>\n",
       "      <td>39320.0</td>\n",
       "      <td>6445.0</td>\n",
       "      <td>35682.0</td>\n",
       "      <td>6082.0</td>\n",
       "      <td>15.0001</td>\n",
       "      <td>500001.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  summary  housing_median_age         total_rooms      total_bedrooms  \\\n",
       "0   count               20640               20640               20433   \n",
       "1    mean  28.639486434108527  2635.7630813953488   537.8705525375618   \n",
       "2  stddev   12.58555761211163  2181.6152515827944  421.38507007403115   \n",
       "3     min                 1.0                 2.0                 1.0   \n",
       "4     max                52.0             39320.0              6445.0   \n",
       "\n",
       "           population         households       median_income  \\\n",
       "0               20640              20640               20640   \n",
       "1  1425.4767441860465  499.5396802325581  3.8706710029070246   \n",
       "2    1132.46212176534  382.3297528316098   1.899821717945263   \n",
       "3                 3.0                1.0              0.4999   \n",
       "4             35682.0             6082.0             15.0001   \n",
       "\n",
       "   median_house_value ocean_proximity  \n",
       "0               20640           20640  \n",
       "1  206855.81690891474            None  \n",
       "2  115395.61587441359            None  \n",
       "3             14999.0       <1H OCEAN  \n",
       "4            500001.0      NEAR OCEAN  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_describe = df_housing.describe().toPandas()\n",
    "df_describe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abc1495d",
   "metadata": {},
   "source": [
    "### Fill missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf5f25eb-b8a3-464e-a948-a3c35fe96672",
   "metadata": {},
   "source": [
    "Let's count all the objects in the dataset and compare this data to the row `count` of the table above. Thus we will know how much missing values are there. Due to `summary` column being object type the Series object is not numeric so we fix that too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7bafc9bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Всего строк в датафрейме: 20640\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "housing_median_age      0\n",
       "total_rooms             0\n",
       "total_bedrooms       -207\n",
       "population              0\n",
       "households              0\n",
       "median_income           0\n",
       "median_house_value      0\n",
       "ocean_proximity         0\n",
       "Name: 0, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_count = df_housing.count()\n",
    "print(f'Всего строк в датафрейме: {total_count}')\n",
    "pd.to_numeric(df_describe.loc[0, 'housing_median_age':]) - total_count"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1558768f-019d-4de0-af0c-a6af0ca851cb",
   "metadata": {},
   "source": [
    "Column `total_bedrooms` has less values than there are objacts in the dataset, so that means there are missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c44e0ce6-0d55-48cd-b6fe-2620730ed472",
   "metadata": {},
   "source": [
    "We will check data distribution for this column in order to understand how to deal with the missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e2586bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAV6ElEQVR4nO3df7RdZX3n8ffHhB9CaAJiMzRQg5XaosxYmkEYW+ciFhBEXGtsFx1Go6WLdoY6OrWj0NYybWEGu7SInarNCCM61IjUUQpaS8E7Xcwso8TfgJQIAZLyQ+WHDVo1+J0/znPpMSa5Nzc399yT5/1a66y797Ofvff3IYfP2fc5+5ybqkKS1IenjLoASdL8MfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx97dWSvDfJRdP0mUiyaQ7PWUmeNVfHk+aSoa+RS7IxyYvnuq+kH2boS/MoyaJR16C+GfoaqSTvB34c+MskW5K8McnLktya5NEkk0l+ekd9W/uHkjyQ5LEkf5vkObOs5beTfL39NnH2UPt+Sd6a5N4kDyZ5d5KnDm3/z0nuT/L3SX5lm2O+N8m7knwsyePAiUl+uo3r0TbOlw31X5rkfUm+luSeJL+b5Clt26uT/N8kl7Z970ryr1r7fUkeSrJ66FinJbktyT8k2Zzkt2bz30V7F0NfI1VVrwTuBc6oqiXAR4APAK8Hng58jEHI77tt36r6o3aYjwNHAT8KfBa4ahal/DPgUGAFsBpYk+TZbdslwE8CzwOe1fr8HkCSU4HfAn6h1bC9qad/C1wMHASsA/4S+OtW72uBq4bO9SfAUuCZwL8GXgW8ZuhYzwe+CDwN+HNgLfAvW13/DvjvSZa0vpcDv1ZVBwHPBW7a5f8q2vtUlQ8fI30AG4EXt+U3A1cPbXsKsBmY2LbvDo61DChgaVt/L3DRNOefALYCBw61Xd1qCfA48BND204A7m7LVwCXDG37yXb+Zw2d/31D238eeAB4ylDbB4D/AiwCvgscPbTt14DJtvxq4M6hbce0cy0favsG8Ly2fG/b/0dG/W/sY+E8vNLXQvNjwD1TK1X1feA+BlfXPyTJoiSXJPlqkm8yeFGAwVX7rnikqh4fWr+n1fJ04ABgfZtSeRT4q9Y+Ve992+y3reHtPwbc18Y1vM+KVvM+2xxjatuUB4eWvw1QVdu2TV3p/xvgNOCeJP8nyQnbqU2dMfS1EAx/v/ffA8+YWkkS4AgGV/vb9oXB1MmZDKZVlgIrp3bdxRoOTnLg0PqPt1q+ziBIn1NVy9pjaQ2mogDub/UN77etbcd3xNQ8/dA+m9u5vsfQ+Ie27bKq+kxVnclgGukjDH57UecMfS0EDzKYw4ZBMJ2e5KQk+wBvAL4D/L/t9IXBPPl3GExrHAD8192o4/eT7Jvk54GXAh9qV+T/A7g0yY8CJFmR5JShel+d5OgkBwAXTnOOdcC3gDcm2SfJBHAGsLaqnmjHuzjJQUmeAfwm8L92dSBtHGcnWVpV3wO+CXx/uv209zP0tRD8N+B329TJGQzekPwTBle+ZzB44/a72/Ztd6O8j8EUyGbgNuBTs6zhAeARBlfiVwG/XlVfadveBGwAPtWmkP4GeDZAVX0ceDuDN0k3MM2bpW0cZwAvaeN7J/CqoXO9lsF7CHcBNzN4s/aKWY7plcDGVvOvA2dP018dSJV/OUuSeuGVviR1xNBXF9oHr7Zs5/HxUdcmzSendySpI4tHXcDOHHroobVy5cpZ7fv4449z4IEHTt9xgRrn+q19NMa5dhjv+hda7evXr/96VT19e9sWdOivXLmSW265ZVb7Tk5OMjExMbcFzaNxrt/aR2Oca4fxrn+h1Z5kex8SBJzTl6SuGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjizoT+SO0srzr5/1vhsvOX0OK5GkueOVviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHZhT6Sf5TkluTfDnJB5Lsn+TIJOuSbEjywST7tr77tfUNbfvKoeNc0NrvSHLKHhqTJGkHpg39JCuA/wisqqrnAouAs4C3AJdW1bOAR4Bz2i7nAI+09ktbP5Ic3fZ7DnAq8M4ki+Z2OJKknZnp9M5i4KlJFgMHAPcDLwKuaduvBF7els9s67TtJyVJa19bVd+pqruBDcBxuz0CSdKMTRv6VbUZeCtwL4OwfwxYDzxaVVtbt03Aira8Ariv7bu19X/acPt29pEkzYPF03VIcjCDq/QjgUeBDzGYntkjkpwLnAuwfPlyJicnZ3WcLVu2zHpfgDccs3X6TjuwO+edsrv1j5K1j8Y41w7jXf841T5t6AMvBu6uqq8BJPkw8AJgWZLF7Wr+cGBz678ZOALY1KaDlgLfGGqfMrzPk6pqDbAGYNWqVTUxMTGLYQ2Cd7b7Arz6/Otnve/Gs2d/3im7W/8oWftojHPtMN71j1PtM5nTvxc4PskBbW7+JOA24JPAK1qf1cBH2/K1bZ22/aaqqtZ+Vru750jgKODTczMMSdJMTHulX1XrklwDfBbYCnyOwZX49cDaJBe1tsvbLpcD70+yAXiYwR07VNWtSa5m8IKxFTivqp6Y4/FIknZiJtM7VNWFwIXbNN/Fdu6+qap/BH5xB8e5GLh4F2uUJM0RP5ErSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4sHnUBe9LK868fdQmStKDM6Eo/ybIk1yT5SpLbk5yQ5JAkNyS5s/08uPVNknck2ZDki0mOHTrO6tb/ziSr99SgJEnbN9PpncuAv6qqnwL+BXA7cD5wY1UdBdzY1gFeAhzVHucC7wJIcghwIfB84DjgwqkXCknS/Jg29JMsBV4IXA5QVd+tqkeBM4ErW7crgZe35TOB99XAp4BlSQ4DTgFuqKqHq+oR4Abg1DkciyRpGqmqnXdIngesAW5jcJW/HngdsLmqlrU+AR6pqmVJrgMuqaqb27YbgTcBE8D+VXVRa38z8O2qeus25zuXwW8ILF++/GfXrl07q4Ft2bKFux97Ylb77q5jVizd7WNs2bKFJUuWzEE188/aR2Oca4fxrn+h1X7iiSeur6pV29s2kzdyFwPHAq+tqnVJLuOfpnIAqKpKsvNXjxmqqjUMXmRYtWpVTUxMzOo4k5OTvO3mx+eipF228eyJ3T7G5OQksx37qFn7aIxz7TDe9Y9T7TOZ098EbKqqdW39GgYvAg+2aRvaz4fa9s3AEUP7H97adtQuSZon04Z+VT0A3Jfk2a3pJAZTPdcCU3fgrAY+2pavBV7V7uI5Hnisqu4HPgGcnOTg9gbuya1NkjRPZnqf/muBq5LsC9wFvIbBC8bVSc4B7gF+qfX9GHAasAH4VutLVT2c5A+Bz7R+f1BVD8/JKCRJMzKj0K+qzwPbe1PgpO30LeC8HRznCuCKXahPkjSH/BoGSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVkxqGfZFGSzyW5rq0fmWRdkg1JPphk39a+X1vf0LavHDrGBa39jiSnzPloJEk7tStX+q8Dbh9afwtwaVU9C3gEOKe1nwM80tovbf1IcjRwFvAc4FTgnUkW7V75kqRdMaPQT3I4cDrwnrYe4EXANa3LlcDL2/KZbZ22/aTW/0xgbVV9p6ruBjYAx83BGCRJM7R4hv3eDrwROKitPw14tKq2tvVNwIq2vAK4D6CqtiZ5rPVfAXxq6JjD+zwpybnAuQDLly9ncnJyhiX+oC1btvCGY56Y1b67a7Y1D9uyZcucHGcUrH00xrl2GO/6x6n2aUM/yUuBh6pqfZKJPV1QVa0B1gCsWrWqJiZmd8rJyUnedvPjc1jZzG08e2K3jzE5Oclsxz5q1j4a41w7jHf941T7TK70XwC8LMlpwP7AjwCXAcuSLG5X+4cDm1v/zcARwKYki4GlwDeG2qcM7yNJmgfTzulX1QVVdXhVrWTwRuxNVXU28EngFa3bauCjbfnatk7bflNVVWs/q93dcyRwFPDpORuJJGlaM53T3543AWuTXAR8Dri8tV8OvD/JBuBhBi8UVNWtSa4GbgO2AudV1Wgm3SWpU7sU+lU1CUy25bvYzt03VfWPwC/uYP+LgYt3tUhJ0tzwE7mS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1ZHf+iIp2YOX51896342XnD6HlUjSD/JKX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHZk29JMckeSTSW5LcmuS17X2Q5LckOTO9vPg1p4k70iyIckXkxw7dKzVrf+dSVbvuWFJkrZnJlf6W4E3VNXRwPHAeUmOBs4Hbqyqo4Ab2zrAS4Cj2uNc4F0weJEALgSeDxwHXDj1QiFJmh/Thn5V3V9Vn23L/wDcDqwAzgSubN2uBF7els8E3lcDnwKWJTkMOAW4oaoerqpHgBuAU+dyMJKknUtVzbxzshL4W+C5wL1Vtay1B3ikqpYluQ64pKpubttuBN4ETAD7V9VFrf3NwLer6q3bnONcBr8hsHz58p9du3btrAa2ZcsW7n7siVntO0rHrFgKDOpfsmTJiKuZHWsfjXGuHca7/oVW+4knnri+qlZtb9vimR4kyRLgL4DXV9U3Bzk/UFWVZOavHjtRVWuANQCrVq2qiYmJWR1ncnKSt938+FyUNK82nj0BDOqf7dhHzdpHY5xrh/Guf5xqn9HdO0n2YRD4V1XVh1vzg23ahvbzoda+GThiaPfDW9uO2iVJ82Qmd+8EuBy4var+eGjTtcDUHTirgY8Otb+q3cVzPPBYVd0PfAI4OcnB7Q3ck1ubJGmezGR65wXAK4EvJfl8a/tt4BLg6iTnAPcAv9S2fQw4DdgAfAt4DUBVPZzkD4HPtH5/UFUPz8UgJEkzM23otzdks4PNJ22nfwHn7eBYVwBX7EqBkqS54ydyJakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdmfEfRtf8WHn+9QC84ZitvLotz9TGS07fEyVJ2ot4pS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjviFa3uRlbv4BW3b8gvbpL2fV/qS1BFDX5I6YuhLUkcMfUnqiKEvSR3x7h09aXfu/vHOH2k8eKUvSR2Z9yv9JKcClwGLgPdU1SXzXYPm3vBvCbv6R939LUGaP/Ma+kkWAX8K/AKwCfhMkmur6rb5rEMLi9NK0vyZ7yv944ANVXUXQJK1wJmAoa9Z2d1PIW9rV39LmS1frDQqqar5O1nyCuDUqvrVtv5K4PlV9RtDfc4Fzm2rzwbumOXpDgW+vhvljto412/tozHOtcN417/Qan9GVT19exsW3N07VbUGWLO7x0lyS1WtmoOSRmKc67f20Rjn2mG86x+n2uf77p3NwBFD64e3NknSPJjv0P8McFSSI5PsC5wFXDvPNUhSt+Z1eqeqtib5DeATDG7ZvKKqbt1Dp9vtKaIRG+f6rX00xrl2GO/6x6b2eX0jV5I0Wn4iV5I6YuhLUkf2ytBPcmqSO5JsSHL+qOsBSHJFkoeSfHmo7ZAkNyS5s/08uLUnyTta/V9McuzQPqtb/zuTrJ6n2o9I8skktyW5NcnrxqX+JPsn+XSSL7Taf7+1H5lkXavxg+3GApLs19Y3tO0rh451QWu/I8kpe7r2ofMuSvK5JNeNYe0bk3wpyeeT3NLaFvzzpp1zWZJrknwlye1JThiX2neqqvaqB4M3iL8KPBPYF/gCcPQCqOuFwLHAl4fa/gg4vy2fD7ylLZ8GfBwIcDywrrUfAtzVfh7clg+eh9oPA45tywcBfwccPQ71txqWtOV9gHWtpquBs1r7u4F/35b/A/DutnwW8MG2fHR7Lu0HHNmeY4vm6bnzm8CfA9e19XGqfSNw6DZtC/550857JfCrbXlfYNm41L7TcY3y5HvoH+oE4BND6xcAF4y6rlbLSn4w9O8ADmvLhwF3tOU/A355237ALwN/NtT+A/3mcRwfZfD9SWNVP3AA8Fng+Qw+Pbl42+cMgzvLTmjLi1u/bPs8Gu63h2s+HLgReBFwXatlLGpv59rID4f+gn/eAEuBu2k3u4xT7dM99sbpnRXAfUPrm1rbQrS8qu5vyw8Ay9vyjsYw8rG1KYOfYXDFPBb1t+mRzwMPATcwuNJ9tKq2bqeOJ2ts2x8Dnjaq2oG3A28Evt/Wn8b41A5QwF8nWZ/BV6zAeDxvjgS+BvzPNrX2niQHMh6179TeGPpjqQaXAQv6/tkkS4C/AF5fVd8c3raQ66+qJ6rqeQyumo8Dfmq0Fc1MkpcCD1XV+lHXsht+rqqOBV4CnJfkhcMbF/DzZjGD6dh3VdXPAI8zmM550gKufaf2xtAfp696eDDJYQDt50OtfUdjGNnYkuzDIPCvqqoPt+axqR+gqh4FPslgSmRZkqkPJw7X8WSNbftS4BuMpvYXAC9LshFYy2CK57IxqR2Aqtrcfj4E/G8GL7rj8LzZBGyqqnVt/RoGLwLjUPtO7Y2hP05f9XAtMPVu/moGc+VT7a9qdwQcDzzWfqX8BHBykoPbXQMnt7Y9KkmAy4Hbq+qPx6n+JE9PsqwtP5XBexG3Mwj/V+yg9qkxvQK4qV3RXQuc1e6QORI4Cvj0nqy9qi6oqsOraiWD5/FNVXX2ONQOkOTAJAdNLTP49/4yY/C8qaoHgPuSPLs1ncTgK+AXfO3TGuUbCnvqweCd9L9jMHf7O6Oup9X0AeB+4HsMriLOYTDfeiNwJ/A3wCGtbxj8sZmvAl8CVg0d51eADe3xmnmq/ecY/Br7ReDz7XHaONQP/HPgc632LwO/19qfySD4NgAfAvZr7fu39Q1t+zOHjvU7bUx3AC+Z5+fPBP90985Y1N7q/EJ73Dr1/+I4PG/aOZ8H3NKeOx9hcPfNWNS+s4dfwyBJHdkbp3ckSTtg6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SO/H8k1HYLyqNafgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bedrooms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>537.870553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median</th>\n",
       "      <td>435.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        total_bedrooms\n",
       "mean        537.870553\n",
       "median      435.000000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_total_bedrooms = df_housing.select('total_bedrooms').toPandas()\n",
    "df_total_bedrooms.hist(bins=20)\n",
    "plt.show()\n",
    "df_total_bedrooms.agg(['mean', 'median'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce1b7330-881c-4031-9a60-9fd629597f6b",
   "metadata": {},
   "source": [
    "Distribution is shifted to the left so we will use median values to fill in missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c483e1f3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "435.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# saving median value to the variable\n",
    "total_bedrooms_median = df_total_bedrooms.median()[0]\n",
    "total_bedrooms_median"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d9bb37-63f3-4df6-975b-a996288b9508",
   "metadata": {},
   "source": [
    "Filling missing values in `total_bedrooms` using median value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "06ca429f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# only one column\n",
    "df_housing = df_housing.na.fill({'total_bedrooms': total_bedrooms_median})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c82e0308",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>summary</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>count</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean</td>\n",
       "      <td>28.639486434108527</td>\n",
       "      <td>2635.7630813953488</td>\n",
       "      <td>536.8388565891473</td>\n",
       "      <td>1425.4767441860465</td>\n",
       "      <td>499.5396802325581</td>\n",
       "      <td>3.8706710029070246</td>\n",
       "      <td>206855.81690891474</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>stddev</td>\n",
       "      <td>12.58555761211163</td>\n",
       "      <td>2181.6152515827944</td>\n",
       "      <td>419.39187792168866</td>\n",
       "      <td>1132.46212176534</td>\n",
       "      <td>382.3297528316098</td>\n",
       "      <td>1.899821717945263</td>\n",
       "      <td>115395.61587441359</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>min</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4999</td>\n",
       "      <td>14999.0</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>max</td>\n",
       "      <td>52.0</td>\n",
       "      <td>39320.0</td>\n",
       "      <td>6445.0</td>\n",
       "      <td>35682.0</td>\n",
       "      <td>6082.0</td>\n",
       "      <td>15.0001</td>\n",
       "      <td>500001.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  summary  housing_median_age         total_rooms      total_bedrooms  \\\n",
       "0   count               20640               20640               20640   \n",
       "1    mean  28.639486434108527  2635.7630813953488   536.8388565891473   \n",
       "2  stddev   12.58555761211163  2181.6152515827944  419.39187792168866   \n",
       "3     min                 1.0                 2.0                 1.0   \n",
       "4     max                52.0             39320.0              6445.0   \n",
       "\n",
       "           population         households       median_income  \\\n",
       "0               20640              20640               20640   \n",
       "1  1425.4767441860465  499.5396802325581  3.8706710029070246   \n",
       "2    1132.46212176534  382.3297528316098   1.899821717945263   \n",
       "3                 3.0                1.0              0.4999   \n",
       "4             35682.0             6082.0             15.0001   \n",
       "\n",
       "   median_house_value ocean_proximity  \n",
       "0               20640           20640  \n",
       "1  206855.81690891474            None  \n",
       "2  115395.61587441359            None  \n",
       "3             14999.0       <1H OCEAN  \n",
       "4            500001.0      NEAR OCEAN  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_housing.describe().toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e539f23e",
   "metadata": {},
   "source": [
    "### Split the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ee400701",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16565 4075\n"
     ]
    }
   ],
   "source": [
    "# split data on train and test samples\n",
    "df_train, df_test = df_housing.randomSplit([0.8, 0.2], seed=RND)\n",
    "print(df_train.count(), df_test.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d764c0d5",
   "metadata": {},
   "source": [
    "## Build a pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef19413",
   "metadata": {},
   "source": [
    "### OHE and standartisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "01687897",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['housing_median_age',\n",
       " 'total_rooms',\n",
       " 'total_bedrooms',\n",
       " 'population',\n",
       " 'households',\n",
       " 'median_income']"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['ocean_proximity']"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'median_house_value'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# list to append stages\n",
    "stages = []\n",
    "# numeric columns\n",
    "num_cols = [col for col in df_housing.columns if col not in ['median_house_value', 'ocean_proximity']]\n",
    "# categorical columns (only one)\n",
    "cat_cols = ['ocean_proximity']\n",
    "# target\n",
    "target = 'median_house_value'\n",
    "\n",
    "display(num_cols, cat_cols, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "460b3468-d742-4c84-8315-d0c3d3210d75",
   "metadata": {},
   "source": [
    "There is only one categorical feature - ocean proximity. We will process it using OHE and add to the pipepline stages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c1d163c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# string values to numeric\n",
    "encoder_idx = StringIndexer(inputCols=cat_cols, outputCols=[col + '_idx' for col in cat_cols])\n",
    "stages += [encoder_idx]\n",
    "\n",
    "# OHE\n",
    "encoder_ohe = OneHotEncoder(inputCols=[col + '_idx' for col in cat_cols], outputCols=[col + '_ohe' for col in cat_cols])\n",
    "stages += [encoder_ohe]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2080c7f-6c4e-44a9-bab4-92272230c224",
   "metadata": {},
   "source": [
    "We will add pipeline stages that assemble numeric features to a vector and standartise them then"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "54e3a214",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assembling a vector\n",
    "num_assembler = VectorAssembler(inputCols=num_cols, outputCol='num_features')\n",
    "stages += [num_assembler]\n",
    "\n",
    "# scaling features\n",
    "scaler_std = StandardScaler(inputCol='num_features', outputCol='num_features_scaled')\n",
    "stages += [scaler_std]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afe809fb-7251-4080-b6cc-d2948d5fdd20",
   "metadata": {},
   "source": [
    "Add a pipeline stage that assembles a vector of all features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0ee2a1f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# all features to vector\n",
    "all_assembler = VectorAssembler(inputCols=['ocean_proximity_ohe', 'num_features_scaled'], outputCol='all_features')\n",
    "stages += [all_assembler]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f8e1d48f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[StringIndexer_a4d438d1deb7,\n",
       " OneHotEncoder_5fa8122a4a6c,\n",
       " VectorAssembler_594a7cc4d4f7,\n",
       " StandardScaler_f145abc32c6d,\n",
       " VectorAssembler_67452192c54d]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checkin stages\n",
    "stages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cea03cb",
   "metadata": {},
   "source": [
    "### Assemble the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819e974a-a76b-4179-9c9b-be8a5f219d51",
   "metadata": {},
   "source": [
    "Due to **the second objective** we will assemble **two** pipelines: one for all features training and one for numeric only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "cfbc021c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[StringIndexer_a4d438d1deb7,\n",
       " OneHotEncoder_5fa8122a4a6c,\n",
       " VectorAssembler_594a7cc4d4f7,\n",
       " StandardScaler_f145abc32c6d,\n",
       " VectorAssembler_67452192c54d,\n",
       " LinearRegression_4658be078f26]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all features trainig\n",
    "lin_reg_all = LinearRegression(labelCol=target, featuresCol='all_features', regParam=0.01)\n",
    "\n",
    "# assembling pipeline for all features trainig\n",
    "pipeline_all = Pipeline(stages=stages+[lin_reg_all])\n",
    "pipeline_all.getStages()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d9d7b796",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[VectorAssembler_594a7cc4d4f7,\n",
       " StandardScaler_f145abc32c6d,\n",
       " LinearRegression_b3b325905ff7]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only numeric features traing\n",
    "lin_reg_num = LinearRegression(labelCol=target, featuresCol='num_features_scaled', regParam=0.01)\n",
    "\n",
    "# assenbling pipeline for numeric only trainig\n",
    "# removing categorical features encoding and vectorising as we don't need those\n",
    "pipeline_num = Pipeline(stages=stages[2:4]+[lin_reg_num])\n",
    "pipeline_num.getStages()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f32c44bc",
   "metadata": {},
   "source": [
    "## Train models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ded45b8",
   "metadata": {},
   "source": [
    "### Train the models and get predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "714da1de",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "23/02/24 20:44:51 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "23/02/24 20:44:51 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "23/02/24 20:44:51 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "23/02/24 20:44:51 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n",
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Обучение на всех признаках\n",
      "+------------------+------------------+\n",
      "|median_house_value|        prediction|\n",
      "+------------------+------------------+\n",
      "|          434700.0|159406.90908675164|\n",
      "|          237500.0|167878.62979386686|\n",
      "|          114200.0| 89552.41292299845|\n",
      "|          298500.0| 312018.2360306425|\n",
      "|          196700.0|201476.48984807797|\n",
      "+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n",
      "Обучение только на числовых признаках\n",
      "+------------------+------------------+\n",
      "|median_house_value|        prediction|\n",
      "+------------------+------------------+\n",
      "|          434700.0|111980.78992996781|\n",
      "|          237500.0|116771.63695244258|\n",
      "|          114200.0|106869.17784938044|\n",
      "|          298500.0|285275.20850078337|\n",
      "|          196700.0|232176.34731505078|\n",
      "+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# all features\n",
    "pred_all = pipeline_all.fit(df_train).transform(df_test)\n",
    "\n",
    "# numeric features only\n",
    "pred_num = pipeline_num.fit(df_train).transform(df_test)\n",
    "\n",
    "print('Обучение на всех признаках')\n",
    "pred_all.select([target, 'prediction']).show(5)\n",
    "print('Обучение только на числовых признаках')\n",
    "pred_num.select([target, 'prediction']).show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fecaf528",
   "metadata": {},
   "source": [
    "### Evaluate scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b034dc38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to evaluate RMSE, MAE and R2\n",
    "def rmse_mae_r2(data, y_true_col, y_pred_col):\n",
    "    '''\n",
    "    Return dictionary with RMSE, MAE and R2 scores\n",
    "    data - DataFrame object\n",
    "    y_true_col - column name with true labels (labelCol)\n",
    "    y_pred_col - column name with predicted labels (predictionCol)\n",
    "    '''\n",
    "    return {\n",
    "        'rmse': RegressionEvaluator(labelCol=y_true_col, predictionCol=y_pred_col, metricName='rmse').evaluate(data),\n",
    "        'mae': RegressionEvaluator(labelCol=y_true_col, predictionCol=y_pred_col, metricName='mae').evaluate(data),\n",
    "        'r2': RegressionEvaluator(labelCol=y_true_col, predictionCol=y_pred_col, metricName='r2').evaluate(data)\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3d3c49ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rmse</th>\n",
       "      <th>mae</th>\n",
       "      <th>r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>all</th>\n",
       "      <td>67612.469670</td>\n",
       "      <td>49831.377876</td>\n",
       "      <td>0.649156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num</th>\n",
       "      <td>73593.523899</td>\n",
       "      <td>55163.733340</td>\n",
       "      <td>0.584339</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             rmse           mae        r2\n",
       "all  67612.469670  49831.377876  0.649156\n",
       "num  73593.523899  55163.733340  0.584339"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = pd.DataFrame([rmse_mae_r2(pred_all, target, 'prediction'),\n",
    "                        rmse_mae_r2(pred_num, target, 'prediction')], index=['all', 'num'])\n",
    "metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58917f5c-5828-4a6a-9ce8-d3f221716bbd",
   "metadata": {},
   "source": [
    "### Stop PySpark session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "baece733",
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757fd103",
   "metadata": {
    "id": "757fd103"
   },
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "51853f1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Изменение метрики rmse: 0.088\n",
      "Изменение метрики mae: 0.107\n",
      "Изменение метрики r2: -0.100\n"
     ]
    }
   ],
   "source": [
    "for m in ['rmse', 'mae', 'r2']:\n",
    "    print(f\"Изменение метрики {m}: {((metrics.loc['num', m] - metrics.loc['all', m]) / metrics.loc['all', m]):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e98d808-f8e3-4de8-8c45-271b3c9f2c4d",
   "metadata": {},
   "source": [
    "* LinReg model trained on all features showed better results on test data than the one trained on only numeric features\n",
    "* * Scores are *~10%* worse\n",
    "* Coefficient of determination ($R^2$ score) obtained shows that prediction variance is almost 3 times (**2.849**) lower than variance of target itself"
   ]
  }
 ],
 "metadata": {
  "ExecuteTimeLog": [
   {
    "duration": 3,
    "start_time": "2023-02-21T20:24:15.032Z"
   },
   {
    "duration": 25153,
    "start_time": "2023-02-21T20:25:15.521Z"
   },
   {
    "duration": 11098,
    "start_time": "2023-02-21T20:26:49.240Z"
   },
   {
    "duration": 17,
    "start_time": "2023-02-21T20:27:49.940Z"
   },
   {
    "duration": 425,
    "start_time": "2023-02-21T20:31:26.012Z"
   },
   {
    "duration": 11,
    "start_time": "2023-02-21T20:34:57.420Z"
   },
   {
    "duration": 5159,
    "start_time": "2023-02-21T20:39:38.590Z"
   },
   {
    "duration": 1831,
    "start_time": "2023-02-21T20:46:39.224Z"
   },
   {
    "duration": 2249,
    "start_time": "2023-02-21T20:46:50.684Z"
   },
   {
    "duration": 68,
    "start_time": "2023-02-21T20:53:05.988Z"
   },
   {
    "duration": 42,
    "start_time": "2023-02-21T20:53:15.448Z"
   },
   {
    "duration": 31,
    "start_time": "2023-02-21T20:54:07.830Z"
   },
   {
    "duration": 23,
    "start_time": "2023-02-21T20:54:17.714Z"
   },
   {
    "duration": 103,
    "start_time": "2023-02-21T20:54:34.871Z"
   },
   {
    "duration": 21,
    "start_time": "2023-02-21T20:54:38.066Z"
   },
   {
    "duration": 16,
    "start_time": "2023-02-21T20:55:41.251Z"
   },
   {
    "duration": 499,
    "start_time": "2023-02-21T20:55:55.314Z"
   },
   {
    "duration": 536,
    "start_time": "2023-02-21T20:55:59.508Z"
   },
   {
    "duration": 421,
    "start_time": "2023-02-21T20:56:09.120Z"
   },
   {
    "duration": 464,
    "start_time": "2023-02-21T20:56:46.353Z"
   },
   {
    "duration": 48,
    "start_time": "2023-02-21T20:56:51.638Z"
   },
   {
    "duration": 273,
    "start_time": "2023-02-21T20:57:16.542Z"
   },
   {
    "duration": 293,
    "start_time": "2023-02-21T21:00:04.213Z"
   },
   {
    "duration": 685,
    "start_time": "2023-02-21T21:00:12.855Z"
   },
   {
    "duration": 1528,
    "start_time": "2023-02-21T21:00:36.157Z"
   },
   {
    "duration": 309,
    "start_time": "2023-02-21T21:01:27.601Z"
   },
   {
    "duration": 568,
    "start_time": "2023-02-21T21:02:45.178Z"
   },
   {
    "duration": 31,
    "start_time": "2023-02-21T21:03:12.960Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-21T21:03:22.852Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T21:03:38.411Z"
   },
   {
    "duration": 807,
    "start_time": "2023-02-21T21:03:50.519Z"
   },
   {
    "duration": 546,
    "start_time": "2023-02-21T21:06:07.518Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T21:09:24.811Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T21:10:00.559Z"
   },
   {
    "duration": 9,
    "start_time": "2023-02-21T21:10:43.978Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-21T21:10:49.986Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-21T21:10:54.176Z"
   },
   {
    "duration": 18,
    "start_time": "2023-02-21T21:11:24.486Z"
   },
   {
    "duration": 14,
    "start_time": "2023-02-21T21:11:32.278Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T21:11:35.565Z"
   },
   {
    "duration": 19,
    "start_time": "2023-02-21T21:11:43.201Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-21T21:11:46.074Z"
   },
   {
    "duration": 38,
    "start_time": "2023-02-21T21:18:52.863Z"
   },
   {
    "duration": 1929,
    "start_time": "2023-02-21T21:19:16.505Z"
   },
   {
    "duration": 351,
    "start_time": "2023-02-21T21:45:42.964Z"
   },
   {
    "duration": 937,
    "start_time": "2023-02-21T21:46:00.073Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T21:46:01.013Z"
   },
   {
    "duration": 16722,
    "start_time": "2023-02-21T21:46:01.022Z"
   },
   {
    "duration": 9901,
    "start_time": "2023-02-21T21:46:17.798Z"
   },
   {
    "duration": 23,
    "start_time": "2023-02-21T21:46:27.702Z"
   },
   {
    "duration": 992,
    "start_time": "2023-02-21T21:46:27.728Z"
   },
   {
    "duration": 5024,
    "start_time": "2023-02-21T21:46:28.723Z"
   },
   {
    "duration": 1117,
    "start_time": "2023-02-21T21:46:33.795Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-21T21:46:34.914Z"
   },
   {
    "duration": 71,
    "start_time": "2023-02-21T21:46:34.929Z"
   },
   {
    "duration": 2235,
    "start_time": "2023-02-21T21:46:35.002Z"
   },
   {
    "duration": 101,
    "start_time": "2023-02-21T22:02:22.566Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T22:02:44.844Z"
   },
   {
    "duration": 192,
    "start_time": "2023-02-21T22:25:47.115Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-21T22:26:38.065Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T22:27:48.383Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T22:27:52.666Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-21T22:29:09.020Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-21T22:29:20.274Z"
   },
   {
    "duration": 180,
    "start_time": "2023-02-21T22:30:14.824Z"
   },
   {
    "duration": 2526,
    "start_time": "2023-02-21T22:39:15.503Z"
   },
   {
    "duration": 977,
    "start_time": "2023-02-21T22:39:57.430Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T22:39:58.414Z"
   },
   {
    "duration": 18108,
    "start_time": "2023-02-21T22:39:58.423Z"
   },
   {
    "duration": 11482,
    "start_time": "2023-02-21T22:40:16.533Z"
   },
   {
    "duration": 91,
    "start_time": "2023-02-21T22:40:28.018Z"
   },
   {
    "duration": 930,
    "start_time": "2023-02-21T22:40:28.111Z"
   },
   {
    "duration": 5517,
    "start_time": "2023-02-21T22:40:29.094Z"
   },
   {
    "duration": 1140,
    "start_time": "2023-02-21T22:40:34.615Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T22:40:35.757Z"
   },
   {
    "duration": 135,
    "start_time": "2023-02-21T22:40:35.766Z"
   },
   {
    "duration": 2515,
    "start_time": "2023-02-21T22:40:35.906Z"
   },
   {
    "duration": 70,
    "start_time": "2023-02-21T22:40:38.423Z"
   },
   {
    "duration": 2777,
    "start_time": "2023-02-21T22:43:24.020Z"
   },
   {
    "duration": 861,
    "start_time": "2023-02-21T22:44:46.837Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-21T23:01:04.575Z"
   },
   {
    "duration": 125,
    "start_time": "2023-02-21T23:11:20.619Z"
   },
   {
    "duration": 110,
    "start_time": "2023-02-21T23:11:33.498Z"
   },
   {
    "duration": 1372,
    "start_time": "2023-02-21T23:22:13.920Z"
   },
   {
    "duration": 365,
    "start_time": "2023-02-21T23:22:33.259Z"
   },
   {
    "duration": 282,
    "start_time": "2023-02-21T23:22:57.126Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T23:23:05.554Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-21T23:23:22.755Z"
   },
   {
    "duration": 3723,
    "start_time": "2023-02-21T23:23:42.737Z"
   },
   {
    "duration": 195,
    "start_time": "2023-02-21T23:25:50.515Z"
   },
   {
    "duration": 407,
    "start_time": "2023-02-21T23:26:33.202Z"
   },
   {
    "duration": 890,
    "start_time": "2023-02-21T23:27:09.935Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-21T23:27:10.827Z"
   },
   {
    "duration": 14877,
    "start_time": "2023-02-21T23:27:10.835Z"
   },
   {
    "duration": 9089,
    "start_time": "2023-02-21T23:27:25.714Z"
   },
   {
    "duration": 14,
    "start_time": "2023-02-21T23:27:34.806Z"
   },
   {
    "duration": 894,
    "start_time": "2023-02-21T23:27:34.821Z"
   },
   {
    "duration": 4513,
    "start_time": "2023-02-21T23:27:35.717Z"
   },
   {
    "duration": 1324,
    "start_time": "2023-02-21T23:27:40.232Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T23:27:41.558Z"
   },
   {
    "duration": 62,
    "start_time": "2023-02-21T23:27:41.566Z"
   },
   {
    "duration": 2094,
    "start_time": "2023-02-21T23:27:41.630Z"
   },
   {
    "duration": 72,
    "start_time": "2023-02-21T23:27:43.726Z"
   },
   {
    "duration": 1894,
    "start_time": "2023-02-21T23:27:43.801Z"
   },
   {
    "duration": 615,
    "start_time": "2023-02-21T23:27:45.698Z"
   },
   {
    "duration": 194,
    "start_time": "2023-02-21T23:27:46.315Z"
   },
   {
    "duration": 1605,
    "start_time": "2023-02-21T23:27:46.512Z"
   },
   {
    "duration": 492,
    "start_time": "2023-02-21T23:27:48.120Z"
   },
   {
    "duration": 191,
    "start_time": "2023-02-21T23:36:53.395Z"
   },
   {
    "duration": 953,
    "start_time": "2023-02-21T23:37:46.183Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-21T23:37:47.138Z"
   },
   {
    "duration": 16358,
    "start_time": "2023-02-21T23:37:47.147Z"
   },
   {
    "duration": 10807,
    "start_time": "2023-02-21T23:38:03.508Z"
   },
   {
    "duration": 75,
    "start_time": "2023-02-21T23:38:14.318Z"
   },
   {
    "duration": 915,
    "start_time": "2023-02-21T23:38:14.394Z"
   },
   {
    "duration": 5190,
    "start_time": "2023-02-21T23:38:15.312Z"
   },
   {
    "duration": 1200,
    "start_time": "2023-02-21T23:38:20.504Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-21T23:38:21.706Z"
   },
   {
    "duration": 90,
    "start_time": "2023-02-21T23:38:21.716Z"
   },
   {
    "duration": 2123,
    "start_time": "2023-02-21T23:38:21.808Z"
   },
   {
    "duration": 9,
    "start_time": "2023-02-21T23:38:23.993Z"
   },
   {
    "duration": 2001,
    "start_time": "2023-02-21T23:38:24.004Z"
   },
   {
    "duration": 785,
    "start_time": "2023-02-21T23:38:26.007Z"
   },
   {
    "duration": 120,
    "start_time": "2023-02-21T23:38:26.795Z"
   },
   {
    "duration": 1715,
    "start_time": "2023-02-21T23:38:26.917Z"
   },
   {
    "duration": 87,
    "start_time": "2023-02-21T23:38:28.634Z"
   },
   {
    "duration": 402,
    "start_time": "2023-02-21T23:38:28.724Z"
   },
   {
    "duration": 789,
    "start_time": "2023-02-21T23:39:38.750Z"
   },
   {
    "duration": 1615,
    "start_time": "2023-02-21T23:40:33.266Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-21T23:42:31.261Z"
   },
   {
    "duration": 1161,
    "start_time": "2023-02-22T21:47:05.758Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T21:47:06.921Z"
   },
   {
    "duration": 22375,
    "start_time": "2023-02-22T21:47:06.928Z"
   },
   {
    "duration": 10438,
    "start_time": "2023-02-22T21:47:29.309Z"
   },
   {
    "duration": 15,
    "start_time": "2023-02-22T21:47:39.749Z"
   },
   {
    "duration": 984,
    "start_time": "2023-02-22T21:47:39.766Z"
   },
   {
    "duration": 5127,
    "start_time": "2023-02-22T21:47:40.752Z"
   },
   {
    "duration": 988,
    "start_time": "2023-02-22T21:47:45.884Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-22T21:47:46.874Z"
   },
   {
    "duration": 100,
    "start_time": "2023-02-22T21:47:46.883Z"
   },
   {
    "duration": 2263,
    "start_time": "2023-02-22T21:47:46.985Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-22T21:47:49.251Z"
   },
   {
    "duration": 2200,
    "start_time": "2023-02-22T21:47:49.263Z"
   },
   {
    "duration": 887,
    "start_time": "2023-02-22T21:47:51.466Z"
   },
   {
    "duration": 506,
    "start_time": "2023-02-22T21:47:52.355Z"
   },
   {
    "duration": 1508,
    "start_time": "2023-02-22T21:47:52.865Z"
   },
   {
    "duration": 99,
    "start_time": "2023-02-22T21:47:54.375Z"
   },
   {
    "duration": 1084,
    "start_time": "2023-02-22T21:47:54.476Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T21:47:55.561Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T21:59:34.301Z"
   },
   {
    "duration": 2803,
    "start_time": "2023-02-22T22:00:47.169Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T22:16:27.768Z"
   },
   {
    "duration": 413,
    "start_time": "2023-02-22T22:26:50.520Z"
   },
   {
    "duration": 3839,
    "start_time": "2023-02-22T22:27:54.724Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T22:28:10.424Z"
   },
   {
    "duration": 2335,
    "start_time": "2023-02-22T22:33:24.841Z"
   },
   {
    "duration": 2645,
    "start_time": "2023-02-22T22:41:04.232Z"
   },
   {
    "duration": 846,
    "start_time": "2023-02-22T22:44:38.459Z"
   },
   {
    "duration": 1701,
    "start_time": "2023-02-22T22:45:12.280Z"
   },
   {
    "duration": 2020,
    "start_time": "2023-02-22T22:46:05.540Z"
   },
   {
    "duration": 1604,
    "start_time": "2023-02-22T22:46:10.968Z"
   },
   {
    "duration": 2040,
    "start_time": "2023-02-22T22:46:37.133Z"
   },
   {
    "duration": 688,
    "start_time": "2023-02-22T22:47:47.023Z"
   },
   {
    "duration": 1829,
    "start_time": "2023-02-22T22:48:14.059Z"
   },
   {
    "duration": 389,
    "start_time": "2023-02-22T22:48:17.272Z"
   },
   {
    "duration": 567,
    "start_time": "2023-02-22T22:48:39.668Z"
   },
   {
    "duration": 1617,
    "start_time": "2023-02-22T22:49:02.474Z"
   },
   {
    "duration": 441,
    "start_time": "2023-02-22T22:49:08.428Z"
   },
   {
    "duration": 1121,
    "start_time": "2023-02-22T22:49:30.498Z"
   },
   {
    "duration": 399,
    "start_time": "2023-02-22T22:49:33.212Z"
   },
   {
    "duration": 1308,
    "start_time": "2023-02-22T22:49:38.805Z"
   },
   {
    "duration": 368,
    "start_time": "2023-02-22T22:49:41.505Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:03:18.795Z"
   },
   {
    "duration": 16,
    "start_time": "2023-02-22T23:04:02.611Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:04:33.725Z"
   },
   {
    "duration": 1315,
    "start_time": "2023-02-22T23:04:35.228Z"
   },
   {
    "duration": 1118,
    "start_time": "2023-02-22T23:05:37.634Z"
   },
   {
    "duration": 932,
    "start_time": "2023-02-22T23:06:37.435Z"
   },
   {
    "duration": 71,
    "start_time": "2023-02-22T23:08:43.114Z"
   },
   {
    "duration": 2242,
    "start_time": "2023-02-22T23:09:06.233Z"
   },
   {
    "duration": 1995,
    "start_time": "2023-02-22T23:10:28.455Z"
   },
   {
    "duration": 1120,
    "start_time": "2023-02-22T23:11:00.837Z"
   },
   {
    "duration": 323,
    "start_time": "2023-02-22T23:11:04.329Z"
   },
   {
    "duration": 1323,
    "start_time": "2023-02-22T23:11:06.419Z"
   },
   {
    "duration": 337,
    "start_time": "2023-02-22T23:11:09.956Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T23:11:13.624Z"
   },
   {
    "duration": 1634,
    "start_time": "2023-02-22T23:11:19.586Z"
   },
   {
    "duration": 837,
    "start_time": "2023-02-22T23:12:27.557Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T23:12:28.396Z"
   },
   {
    "duration": 13893,
    "start_time": "2023-02-22T23:12:28.402Z"
   },
   {
    "duration": 7473,
    "start_time": "2023-02-22T23:12:42.298Z"
   },
   {
    "duration": 20,
    "start_time": "2023-02-22T23:12:49.772Z"
   },
   {
    "duration": 634,
    "start_time": "2023-02-22T23:12:49.847Z"
   },
   {
    "duration": 3539,
    "start_time": "2023-02-22T23:12:50.548Z"
   },
   {
    "duration": 927,
    "start_time": "2023-02-22T23:12:54.148Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T23:12:55.076Z"
   },
   {
    "duration": 80,
    "start_time": "2023-02-22T23:12:55.083Z"
   },
   {
    "duration": 1613,
    "start_time": "2023-02-22T23:12:55.165Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-22T23:12:56.780Z"
   },
   {
    "duration": 1566,
    "start_time": "2023-02-22T23:12:56.787Z"
   },
   {
    "duration": 705,
    "start_time": "2023-02-22T23:12:58.355Z"
   },
   {
    "duration": 193,
    "start_time": "2023-02-22T23:12:59.062Z"
   },
   {
    "duration": 1095,
    "start_time": "2023-02-22T23:12:59.257Z"
   },
   {
    "duration": 25,
    "start_time": "2023-02-22T23:13:00.354Z"
   },
   {
    "duration": 1008,
    "start_time": "2023-02-22T23:13:00.380Z"
   },
   {
    "duration": 2658,
    "start_time": "2023-02-22T23:13:01.390Z"
   },
   {
    "duration": 3398,
    "start_time": "2023-02-22T23:13:04.050Z"
   },
   {
    "duration": 830,
    "start_time": "2023-02-22T23:13:07.450Z"
   },
   {
    "duration": 2278,
    "start_time": "2023-02-22T23:13:08.282Z"
   },
   {
    "duration": 846,
    "start_time": "2023-02-22T23:13:10.562Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T23:13:11.447Z"
   },
   {
    "duration": 4028,
    "start_time": "2023-02-22T23:13:11.454Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:15:20.424Z"
   },
   {
    "duration": 3105,
    "start_time": "2023-02-22T23:15:22.142Z"
   },
   {
    "duration": 792,
    "start_time": "2023-02-22T23:18:16.705Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:18:17.499Z"
   },
   {
    "duration": 14554,
    "start_time": "2023-02-22T23:18:17.504Z"
   },
   {
    "duration": 7408,
    "start_time": "2023-02-22T23:18:32.060Z"
   },
   {
    "duration": 81,
    "start_time": "2023-02-22T23:18:39.470Z"
   },
   {
    "duration": 704,
    "start_time": "2023-02-22T23:18:39.552Z"
   },
   {
    "duration": 3393,
    "start_time": "2023-02-22T23:18:40.257Z"
   },
   {
    "duration": 913,
    "start_time": "2023-02-22T23:18:43.652Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-22T23:18:44.566Z"
   },
   {
    "duration": 76,
    "start_time": "2023-02-22T23:18:44.574Z"
   },
   {
    "duration": 1523,
    "start_time": "2023-02-22T23:18:44.652Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-22T23:18:46.177Z"
   },
   {
    "duration": 1610,
    "start_time": "2023-02-22T23:18:46.248Z"
   },
   {
    "duration": 596,
    "start_time": "2023-02-22T23:18:47.860Z"
   },
   {
    "duration": 95,
    "start_time": "2023-02-22T23:18:48.458Z"
   },
   {
    "duration": 1123,
    "start_time": "2023-02-22T23:18:48.554Z"
   },
   {
    "duration": 93,
    "start_time": "2023-02-22T23:18:49.679Z"
   },
   {
    "duration": 779,
    "start_time": "2023-02-22T23:18:49.775Z"
   },
   {
    "duration": 2626,
    "start_time": "2023-02-22T23:18:50.556Z"
   },
   {
    "duration": 3293,
    "start_time": "2023-02-22T23:18:53.184Z"
   },
   {
    "duration": 770,
    "start_time": "2023-02-22T23:18:56.478Z"
   },
   {
    "duration": 2011,
    "start_time": "2023-02-22T23:18:57.250Z"
   },
   {
    "duration": 748,
    "start_time": "2023-02-22T23:18:59.263Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:19:00.013Z"
   },
   {
    "duration": 3451,
    "start_time": "2023-02-22T23:19:00.047Z"
   },
   {
    "duration": 1245,
    "start_time": "2023-02-22T23:49:52.370Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T23:50:13.625Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:50:47.413Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-22T23:51:24.312Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T23:56:10.018Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T23:56:27.434Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-22T23:56:35.178Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-22T23:56:39.404Z"
   },
   {
    "duration": 74,
    "start_time": "2023-02-22T23:59:32.867Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-23T00:00:35.962Z"
   },
   {
    "duration": 564,
    "start_time": "2023-02-23T00:23:00.510Z"
   },
   {
    "duration": 450,
    "start_time": "2023-02-23T00:23:08.980Z"
   },
   {
    "duration": 681,
    "start_time": "2023-02-23T00:23:12.795Z"
   },
   {
    "duration": 500,
    "start_time": "2023-02-23T00:23:22.482Z"
   },
   {
    "duration": 311,
    "start_time": "2023-02-23T00:23:30.219Z"
   },
   {
    "duration": 846,
    "start_time": "2023-02-23T00:36:50.745Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-23T00:36:51.593Z"
   },
   {
    "duration": 14490,
    "start_time": "2023-02-23T00:36:51.602Z"
   },
   {
    "duration": 8092,
    "start_time": "2023-02-23T00:37:06.094Z"
   },
   {
    "duration": 80,
    "start_time": "2023-02-23T00:37:14.188Z"
   },
   {
    "duration": 708,
    "start_time": "2023-02-23T00:37:14.270Z"
   },
   {
    "duration": 3495,
    "start_time": "2023-02-23T00:37:14.979Z"
   },
   {
    "duration": 882,
    "start_time": "2023-02-23T00:37:18.475Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-23T00:37:19.359Z"
   },
   {
    "duration": 59,
    "start_time": "2023-02-23T00:37:19.365Z"
   },
   {
    "duration": 1534,
    "start_time": "2023-02-23T00:37:19.425Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-23T00:37:20.960Z"
   },
   {
    "duration": 1781,
    "start_time": "2023-02-23T00:37:20.970Z"
   },
   {
    "duration": 697,
    "start_time": "2023-02-23T00:37:22.753Z"
   },
   {
    "duration": 42,
    "start_time": "2023-02-23T00:37:23.452Z"
   },
   {
    "duration": 1129,
    "start_time": "2023-02-23T00:37:23.548Z"
   },
   {
    "duration": 92,
    "start_time": "2023-02-23T00:37:24.678Z"
   },
   {
    "duration": 715,
    "start_time": "2023-02-23T00:37:24.771Z"
   },
   {
    "duration": 2683,
    "start_time": "2023-02-23T00:37:25.487Z"
   },
   {
    "duration": 2985,
    "start_time": "2023-02-23T00:37:28.171Z"
   },
   {
    "duration": 697,
    "start_time": "2023-02-23T00:37:31.158Z"
   },
   {
    "duration": 2231,
    "start_time": "2023-02-23T00:37:31.856Z"
   },
   {
    "duration": 859,
    "start_time": "2023-02-23T00:37:34.089Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-23T00:37:34.949Z"
   },
   {
    "duration": 4335,
    "start_time": "2023-02-23T00:37:34.954Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-23T00:37:39.348Z"
   },
   {
    "duration": 1262,
    "start_time": "2023-02-24T17:59:33.705Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:00:54.239Z"
   },
   {
    "duration": 20363,
    "start_time": "2023-02-24T18:00:57.821Z"
   },
   {
    "duration": 10403,
    "start_time": "2023-02-24T18:04:01.534Z"
   },
   {
    "duration": 31,
    "start_time": "2023-02-24T18:04:13.685Z"
   },
   {
    "duration": 753,
    "start_time": "2023-02-24T18:04:31.365Z"
   },
   {
    "duration": 4794,
    "start_time": "2023-02-24T18:04:39.013Z"
   },
   {
    "duration": 461,
    "start_time": "2023-02-24T18:20:15.832Z"
   },
   {
    "duration": 1718,
    "start_time": "2023-02-24T18:20:48.716Z"
   },
   {
    "duration": 1290,
    "start_time": "2023-02-24T18:23:18.053Z"
   },
   {
    "duration": 283,
    "start_time": "2023-02-24T18:23:37.135Z"
   },
   {
    "duration": 258,
    "start_time": "2023-02-24T18:23:47.361Z"
   },
   {
    "duration": 241,
    "start_time": "2023-02-24T18:23:59.775Z"
   },
   {
    "duration": 291,
    "start_time": "2023-02-24T18:24:13.204Z"
   },
   {
    "duration": 232,
    "start_time": "2023-02-24T18:24:34.887Z"
   },
   {
    "duration": 190,
    "start_time": "2023-02-24T18:24:53.929Z"
   },
   {
    "duration": 207,
    "start_time": "2023-02-24T18:25:02.529Z"
   },
   {
    "duration": 215,
    "start_time": "2023-02-24T18:25:10.053Z"
   },
   {
    "duration": 219,
    "start_time": "2023-02-24T18:25:43.714Z"
   },
   {
    "duration": 188,
    "start_time": "2023-02-24T18:25:53.634Z"
   },
   {
    "duration": 191,
    "start_time": "2023-02-24T18:25:58.519Z"
   },
   {
    "duration": 486,
    "start_time": "2023-02-24T18:26:27.568Z"
   },
   {
    "duration": 164,
    "start_time": "2023-02-24T18:26:45.033Z"
   },
   {
    "duration": 248,
    "start_time": "2023-02-24T18:28:56.284Z"
   },
   {
    "duration": 9,
    "start_time": "2023-02-24T18:29:54.501Z"
   },
   {
    "duration": 11,
    "start_time": "2023-02-24T18:30:03.105Z"
   },
   {
    "duration": 15,
    "start_time": "2023-02-24T18:30:13.016Z"
   },
   {
    "duration": 17,
    "start_time": "2023-02-24T18:30:34.228Z"
   },
   {
    "duration": 19,
    "start_time": "2023-02-24T18:30:55.732Z"
   },
   {
    "duration": 11,
    "start_time": "2023-02-24T18:31:00.852Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-24T18:31:20.856Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:31:35.451Z"
   },
   {
    "duration": 19,
    "start_time": "2023-02-24T18:31:48.646Z"
   },
   {
    "duration": 191,
    "start_time": "2023-02-24T18:33:37.187Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:33:40.140Z"
   },
   {
    "duration": 187,
    "start_time": "2023-02-24T18:33:58.017Z"
   },
   {
    "duration": 168,
    "start_time": "2023-02-24T18:34:02.701Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:34:35.267Z"
   },
   {
    "duration": 10,
    "start_time": "2023-02-24T18:34:53.496Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:35:16.159Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-24T18:35:21.772Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-24T18:35:28.812Z"
   },
   {
    "duration": 137,
    "start_time": "2023-02-24T18:36:45.123Z"
   },
   {
    "duration": 148,
    "start_time": "2023-02-24T18:39:44.666Z"
   },
   {
    "duration": 674,
    "start_time": "2023-02-24T18:40:43.837Z"
   },
   {
    "duration": 461,
    "start_time": "2023-02-24T18:41:29.529Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-24T18:42:44.617Z"
   },
   {
    "duration": 33,
    "start_time": "2023-02-24T18:49:00.049Z"
   },
   {
    "duration": 1206,
    "start_time": "2023-02-24T18:49:02.386Z"
   },
   {
    "duration": 1570,
    "start_time": "2023-02-24T18:53:17.940Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T18:59:50.348Z"
   },
   {
    "duration": 2,
    "start_time": "2023-02-24T19:00:35.722Z"
   },
   {
    "duration": 2,
    "start_time": "2023-02-24T19:03:23.293Z"
   },
   {
    "duration": 2,
    "start_time": "2023-02-24T19:03:36.437Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-24T19:03:39.678Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T19:03:46.807Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-24T19:04:01.009Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-24T19:04:07.300Z"
   },
   {
    "duration": 14,
    "start_time": "2023-02-24T19:21:13.168Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-24T19:21:28.830Z"
   },
   {
    "duration": 88,
    "start_time": "2023-02-24T19:21:33.514Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:21:38.918Z"
   },
   {
    "duration": 43,
    "start_time": "2023-02-24T19:27:21.714Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-24T19:34:20.515Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:35:50.688Z"
   },
   {
    "duration": 148,
    "start_time": "2023-02-24T19:41:20.011Z"
   },
   {
    "duration": 8,
    "start_time": "2023-02-24T19:41:35.033Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:41:46.637Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-24T19:41:49.998Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:42:49.510Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:42:53.414Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-24T19:43:01.509Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T19:43:04.769Z"
   },
   {
    "duration": 5,
    "start_time": "2023-02-24T19:43:20.897Z"
   },
   {
    "duration": 3,
    "start_time": "2023-02-24T19:43:24.545Z"
   },
   {
    "duration": 13,
    "start_time": "2023-02-24T19:43:45.681Z"
   },
   {
    "duration": 9,
    "start_time": "2023-02-24T19:45:52.295Z"
   },
   {
    "duration": 9549,
    "start_time": "2023-02-24T19:54:22.564Z"
   },
   {
    "duration": 5939,
    "start_time": "2023-02-24T19:55:12.954Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T20:02:20.855Z"
   },
   {
    "duration": 2455,
    "start_time": "2023-02-24T20:02:23.069Z"
   },
   {
    "duration": 154,
    "start_time": "2023-02-24T20:02:59.277Z"
   },
   {
    "duration": 12,
    "start_time": "2023-02-24T20:03:07.043Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T20:03:13.271Z"
   },
   {
    "duration": 16,
    "start_time": "2023-02-24T20:18:48.559Z"
   },
   {
    "duration": 902,
    "start_time": "2023-02-24T20:19:12.008Z"
   },
   {
    "duration": 7,
    "start_time": "2023-02-24T20:19:12.913Z"
   },
   {
    "duration": 16466,
    "start_time": "2023-02-24T20:19:12.923Z"
   },
   {
    "duration": 11227,
    "start_time": "2023-02-24T20:19:29.393Z"
   },
   {
    "duration": 80,
    "start_time": "2023-02-24T20:19:40.622Z"
   },
   {
    "duration": 988,
    "start_time": "2023-02-24T20:19:40.705Z"
   },
   {
    "duration": 4591,
    "start_time": "2023-02-24T20:19:41.695Z"
   },
   {
    "duration": 518,
    "start_time": "2023-02-24T20:19:46.288Z"
   },
   {
    "duration": 1217,
    "start_time": "2023-02-24T20:19:46.807Z"
   },
   {
    "duration": 62,
    "start_time": "2023-02-24T20:19:48.027Z"
   },
   {
    "duration": 99,
    "start_time": "2023-02-24T20:19:48.091Z"
   },
   {
    "duration": 2114,
    "start_time": "2023-02-24T20:19:48.192Z"
   },
   {
    "duration": 2107,
    "start_time": "2023-02-24T20:19:50.308Z"
   },
   {
    "duration": 76,
    "start_time": "2023-02-24T20:19:52.417Z"
   },
   {
    "duration": 104,
    "start_time": "2023-02-24T20:19:52.495Z"
   },
   {
    "duration": 19,
    "start_time": "2023-02-24T20:19:52.601Z"
   },
   {
    "duration": 13,
    "start_time": "2023-02-24T20:19:52.622Z"
   },
   {
    "duration": 50,
    "start_time": "2023-02-24T20:19:52.637Z"
   },
   {
    "duration": 40,
    "start_time": "2023-02-24T20:27:43.876Z"
   },
   {
    "duration": 11,
    "start_time": "2023-02-24T20:28:03.721Z"
   },
   {
    "duration": 8033,
    "start_time": "2023-02-24T20:28:15.792Z"
   },
   {
    "duration": 0,
    "start_time": "2023-02-24T20:28:23.828Z"
   },
   {
    "duration": 0,
    "start_time": "2023-02-24T20:28:23.830Z"
   },
   {
    "duration": 0,
    "start_time": "2023-02-24T20:28:23.832Z"
   },
   {
    "duration": 0,
    "start_time": "2023-02-24T20:28:23.833Z"
   },
   {
    "duration": 11,
    "start_time": "2023-02-24T20:42:41.881Z"
   },
   {
    "duration": 12,
    "start_time": "2023-02-24T20:43:25.170Z"
   },
   {
    "duration": 7057,
    "start_time": "2023-02-24T20:43:29.643Z"
   },
   {
    "duration": 918,
    "start_time": "2023-02-24T20:44:08.671Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-24T20:44:09.591Z"
   },
   {
    "duration": 15417,
    "start_time": "2023-02-24T20:44:09.598Z"
   },
   {
    "duration": 10071,
    "start_time": "2023-02-24T20:44:25.017Z"
   },
   {
    "duration": 19,
    "start_time": "2023-02-24T20:44:35.090Z"
   },
   {
    "duration": 785,
    "start_time": "2023-02-24T20:44:35.112Z"
   },
   {
    "duration": 4802,
    "start_time": "2023-02-24T20:44:35.898Z"
   },
   {
    "duration": 506,
    "start_time": "2023-02-24T20:44:40.702Z"
   },
   {
    "duration": 862,
    "start_time": "2023-02-24T20:44:41.210Z"
   },
   {
    "duration": 13,
    "start_time": "2023-02-24T20:44:42.073Z"
   },
   {
    "duration": 75,
    "start_time": "2023-02-24T20:44:42.088Z"
   },
   {
    "duration": 2003,
    "start_time": "2023-02-24T20:44:42.186Z"
   },
   {
    "duration": 2102,
    "start_time": "2023-02-24T20:44:44.191Z"
   },
   {
    "duration": 12,
    "start_time": "2023-02-24T20:44:46.297Z"
   },
   {
    "duration": 106,
    "start_time": "2023-02-24T20:44:46.311Z"
   },
   {
    "duration": 73,
    "start_time": "2023-02-24T20:44:46.419Z"
   },
   {
    "duration": 9,
    "start_time": "2023-02-24T20:44:46.494Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-24T20:44:46.504Z"
   },
   {
    "duration": 44,
    "start_time": "2023-02-24T20:44:46.511Z"
   },
   {
    "duration": 33,
    "start_time": "2023-02-24T20:44:46.556Z"
   },
   {
    "duration": 10827,
    "start_time": "2023-02-24T20:44:46.590Z"
   },
   {
    "duration": 4,
    "start_time": "2023-02-24T20:44:57.419Z"
   },
   {
    "duration": 2703,
    "start_time": "2023-02-24T20:44:57.424Z"
   },
   {
    "duration": 261,
    "start_time": "2023-02-24T20:45:00.129Z"
   },
   {
    "duration": 6,
    "start_time": "2023-02-24T20:45:00.392Z"
   }
  ],
  "colab": {
   "name": "PySparkSprint.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}