End-to-End Machine Learning Model using PySpark and MLlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build an End-to-End Machine Learning Model using PySpark and MLlib (1)\n",
    "#### Binary Classification with Imbalanced Classes problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-Memory computation and Parallel-Processing are some of the major reasons that Apache Spark has become very popular in the big data industry to deal with data products at large scale and perform faster analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we are going to use a real world dataset from Home Credit Default Risk competition on kaggle. The target variable is either 0 (applicants who were able to pay back their loans)or 1 (applicants who were NOT able to pay back their loans). it is a binary classification problem with a highly imbalanced target label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data: [Link](https://www.kaggle.com/c/home-credit-default-risk/data?select=application_train.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Let’s start by looking at how the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we use the findspark library to locate spark on our local machine\n",
    "import findspark\n",
    "findspark.init('C:/Users/bokhy/spark/spark-2.4.6-bin-hadoop2.7')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of Spark session start\n",
    "# spark = SparkSession.builder.getOrCreate()\n",
    "# aa = spark.sql('''select 'spark' as hello ''')\n",
    "# aa.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- SK_ID_CURR: integer (nullable = true)\n",
      " |-- TARGET: integer (nullable = true)\n",
      " |-- NAME_CONTRACT_TYPE: string (nullable = true)\n",
      " |-- CODE_GENDER: string (nullable = true)\n",
      " |-- FLAG_OWN_CAR: string (nullable = true)\n",
      " |-- FLAG_OWN_REALTY: string (nullable = true)\n",
      " |-- CNT_CHILDREN: integer (nullable = true)\n",
      " |-- AMT_INCOME_TOTAL: double (nullable = true)\n",
      " |-- AMT_CREDIT: double (nullable = true)\n",
      " |-- AMT_ANNUITY: double (nullable = true)\n",
      " |-- AMT_GOODS_PRICE: double (nullable = true)\n",
      " |-- NAME_TYPE_SUITE: string (nullable = true)\n",
      " |-- NAME_INCOME_TYPE: string (nullable = true)\n",
      " |-- NAME_EDUCATION_TYPE: string (nullable = true)\n",
      " |-- NAME_FAMILY_STATUS: string (nullable = true)\n",
      " |-- NAME_HOUSING_TYPE: string (nullable = true)\n",
      " |-- REGION_POPULATION_RELATIVE: double (nullable = true)\n",
      " |-- DAYS_BIRTH: integer (nullable = true)\n",
      " |-- DAYS_EMPLOYED: integer (nullable = true)\n",
      " |-- DAYS_REGISTRATION: double (nullable = true)\n",
      " |-- DAYS_ID_PUBLISH: integer (nullable = true)\n",
      " |-- OWN_CAR_AGE: double (nullable = true)\n",
      " |-- FLAG_MOBIL: integer (nullable = true)\n",
      " |-- FLAG_EMP_PHONE: integer (nullable = true)\n",
      " |-- FLAG_WORK_PHONE: integer (nullable = true)\n",
      " |-- FLAG_CONT_MOBILE: integer (nullable = true)\n",
      " |-- FLAG_PHONE: integer (nullable = true)\n",
      " |-- FLAG_EMAIL: integer (nullable = true)\n",
      " |-- OCCUPATION_TYPE: string (nullable = true)\n",
      " |-- CNT_FAM_MEMBERS: double (nullable = true)\n",
      " |-- REGION_RATING_CLIENT: integer (nullable = true)\n",
      " |-- REGION_RATING_CLIENT_W_CITY: integer (nullable = true)\n",
      " |-- WEEKDAY_APPR_PROCESS_START: string (nullable = true)\n",
      " |-- HOUR_APPR_PROCESS_START: integer (nullable = true)\n",
      " |-- REG_REGION_NOT_LIVE_REGION: integer (nullable = true)\n",
      " |-- REG_REGION_NOT_WORK_REGION: integer (nullable = true)\n",
      " |-- LIVE_REGION_NOT_WORK_REGION: integer (nullable = true)\n",
      " |-- REG_CITY_NOT_LIVE_CITY: integer (nullable = true)\n",
      " |-- REG_CITY_NOT_WORK_CITY: integer (nullable = true)\n",
      " |-- LIVE_CITY_NOT_WORK_CITY: integer (nullable = true)\n",
      " |-- ORGANIZATION_TYPE: string (nullable = true)\n",
      " |-- EXT_SOURCE_1: double (nullable = true)\n",
      " |-- EXT_SOURCE_2: double (nullable = true)\n",
      " |-- EXT_SOURCE_3: double (nullable = true)\n",
      " |-- APARTMENTS_AVG: double (nullable = true)\n",
      " |-- BASEMENTAREA_AVG: double (nullable = true)\n",
      " |-- YEARS_BEGINEXPLUATATION_AVG: double (nullable = true)\n",
      " |-- YEARS_BUILD_AVG: double (nullable = true)\n",
      " |-- COMMONAREA_AVG: double (nullable = true)\n",
      " |-- ELEVATORS_AVG: double (nullable = true)\n",
      " |-- ENTRANCES_AVG: double (nullable = true)\n",
      " |-- FLOORSMAX_AVG: double (nullable = true)\n",
      " |-- FLOORSMIN_AVG: double (nullable = true)\n",
      " |-- LANDAREA_AVG: double (nullable = true)\n",
      " |-- LIVINGAPARTMENTS_AVG: double (nullable = true)\n",
      " |-- LIVINGAREA_AVG: double (nullable = true)\n",
      " |-- NONLIVINGAPARTMENTS_AVG: double (nullable = true)\n",
      " |-- NONLIVINGAREA_AVG: double (nullable = true)\n",
      " |-- APARTMENTS_MODE: double (nullable = true)\n",
      " |-- BASEMENTAREA_MODE: double (nullable = true)\n",
      " |-- YEARS_BEGINEXPLUATATION_MODE: double (nullable = true)\n",
      " |-- YEARS_BUILD_MODE: double (nullable = true)\n",
      " |-- COMMONAREA_MODE: double (nullable = true)\n",
      " |-- ELEVATORS_MODE: double (nullable = true)\n",
      " |-- ENTRANCES_MODE: double (nullable = true)\n",
      " |-- FLOORSMAX_MODE: double (nullable = true)\n",
      " |-- FLOORSMIN_MODE: double (nullable = true)\n",
      " |-- LANDAREA_MODE: double (nullable = true)\n",
      " |-- LIVINGAPARTMENTS_MODE: double (nullable = true)\n",
      " |-- LIVINGAREA_MODE: double (nullable = true)\n",
      " |-- NONLIVINGAPARTMENTS_MODE: double (nullable = true)\n",
      " |-- NONLIVINGAREA_MODE: double (nullable = true)\n",
      " |-- APARTMENTS_MEDI: double (nullable = true)\n",
      " |-- BASEMENTAREA_MEDI: double (nullable = true)\n",
      " |-- YEARS_BEGINEXPLUATATION_MEDI: double (nullable = true)\n",
      " |-- YEARS_BUILD_MEDI: double (nullable = true)\n",
      " |-- COMMONAREA_MEDI: double (nullable = true)\n",
      " |-- ELEVATORS_MEDI: double (nullable = true)\n",
      " |-- ENTRANCES_MEDI: double (nullable = true)\n",
      " |-- FLOORSMAX_MEDI: double (nullable = true)\n",
      " |-- FLOORSMIN_MEDI: double (nullable = true)\n",
      " |-- LANDAREA_MEDI: double (nullable = true)\n",
      " |-- LIVINGAPARTMENTS_MEDI: double (nullable = true)\n",
      " |-- LIVINGAREA_MEDI: double (nullable = true)\n",
      " |-- NONLIVINGAPARTMENTS_MEDI: double (nullable = true)\n",
      " |-- NONLIVINGAREA_MEDI: double (nullable = true)\n",
      " |-- FONDKAPREMONT_MODE: string (nullable = true)\n",
      " |-- HOUSETYPE_MODE: string (nullable = true)\n",
      " |-- TOTALAREA_MODE: double (nullable = true)\n",
      " |-- WALLSMATERIAL_MODE: string (nullable = true)\n",
      " |-- EMERGENCYSTATE_MODE: string (nullable = true)\n",
      " |-- OBS_30_CNT_SOCIAL_CIRCLE: double (nullable = true)\n",
      " |-- DEF_30_CNT_SOCIAL_CIRCLE: double (nullable = true)\n",
      " |-- OBS_60_CNT_SOCIAL_CIRCLE: double (nullable = true)\n",
      " |-- DEF_60_CNT_SOCIAL_CIRCLE: double (nullable = true)\n",
      " |-- DAYS_LAST_PHONE_CHANGE: double (nullable = true)\n",
      " |-- FLAG_DOCUMENT_2: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_3: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_4: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_5: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_6: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_7: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_8: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_9: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_10: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_11: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_12: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_13: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_14: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_15: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_16: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_17: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_18: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_19: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_20: integer (nullable = true)\n",
      " |-- FLAG_DOCUMENT_21: integer (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_HOUR: double (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_DAY: double (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_WEEK: double (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_MON: double (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_QRT: double (nullable = true)\n",
      " |-- AMT_REQ_CREDIT_BUREAU_YEAR: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pyspark # only run after findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "# sc = pyspark.SparkContext(appName=\"imbalanced_binary_classification\")\n",
    "# initiate our session and read the main CSV file, then we print the #dataframe schema\n",
    "spark = SparkSession.builder.appName('imbalanced_binary_classification').getOrCreate()\n",
    "\n",
    "path = 'C:\\\\Users\\\\bokhy\\\\Desktop\\\\Python\\\\github\\\\Python-Projects'  \n",
    "\n",
    "new_df = spark.read.csv(os.path.join(path, 'application_train.csv'), header=True, inferSchema=True)\n",
    "new_df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we are going to drop the SK_ID_CURR column (we don't need it), \n",
    "and rename the “TARGET” column to “label” and see the distribution of our target variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>24825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>282686</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label   count\n",
       "0      1   24825\n",
       "1      0  282686"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "drop_col = ['SK_ID_CURR']\n",
    "new_df = new_df.select([column for column in new_df.columns if column not in drop_col])\n",
    "new_df = new_df.withColumnRenamed('TARGET', 'label')\n",
    "new_df.groupby('label').count().toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "307511\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1da2d7ad408>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuoAAAJQCAYAAADYNkG1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG5xJREFUeJzt3X/M7ndd3/HX2x7wxxAp9sCwrZbMM2NlDulZ6XRZVLJyIFkKpmwl055gkxpXNk3MYvWPlYAskoFGHKup4djWOSsBGd1S7LpKJGyIPVXWH1TSk8rosR09cCpgDLLie3/c3xOunt7n9D6Hc5/7fZ8+HsmV+7re3x/X5+aP5nkuvtf3ru4OAAAwy9dt9QIAAICnEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABtqx1QuY4pxzzukLLrhgq5cBAMAZ7u677/5sd+98uv2E+uKCCy7I/v37t3oZAACc4arq/2xkP5e+AADAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGGjHVi+Ar7ro39y81UsAtom7//2VW70EADaZT9QBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIE2LdSr6vyq+lBVPVBV91fVTy3zN1XVn1fVx5fHq1eO+bmqOlBVn6yqV67M9yyzA1V17cr8xVX1sap6sKp+p6qevcy/fnl9YNl+wWb9ngAAsBk28xP1J5L8THd/d5JLklxTVRcu2365u1+6PG5LkmXbFUm+J8meJP+xqs6qqrOSvCvJq5JcmOT1K+d523KuXUkeT3LVMr8qyePd/Z1JfnnZDwAAto1NC/XufrS7/3h5/sUkDyQ59ziHXJbklu7+6+7+syQHkly8PA5090Pd/eUktyS5rKoqyQ8nee9y/E1JXrNyrpuW5+9N8oplfwAA2BZOyzXqy6Un35fkY8vojVV1T1Xtq6qzl9m5SR5eOezgMjvW/FuT/EV3P3HU/EnnWrZ/ftkfAAC2hU0P9ap6TpL3Jfnp7v5CkuuT/J0kL03yaJJ3HNl1ncP7JObHO9fRa7u6qvZX1f5Dhw4d9/cAAIDTaVNDvaqelbVI/63u/t0k6e7PdPdXuvtvkvx61i5tSdY+ET9/5fDzkjxynPlnkzyvqnYcNX/SuZbt35Lk8NHr6+4bunt3d+/euXPn1/rrAgDAKbOZd32pJO9O8kB3/9LK/EUru702yX3L81uTXLHcseXFSXYl+aMkdyXZtdzh5dlZ+8Lprd3dST6U5PLl+L1JPrByrr3L88uT/P6yPwAAbAs7nn6Xk/YDSX4syb1V9fFl9vNZu2vLS7N2KcqnkvxEknT3/VX1niSfyNodY67p7q8kSVW9McntSc5Ksq+771/O97NJbqmqX0jyJ1n7h0GWn79ZVQey9kn6FZv4ewIAwCm3aaHe3R/J+teK33acY96a5K3rzG9b77jufihfvXRmdf6lJK87kfUCAMAk/jIpAAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADbVqoV9X5VfWhqnqgqu6vqp9a5s+vqjuq6sHl59nLvKrqnVV1oKruqaqXrZxr77L/g1W1d2V+UVXduxzzzqqq470HAABsF5v5ifoTSX6mu787ySVJrqmqC5Ncm+TO7t6V5M7ldZK8Ksmu5XF1kuuTtehOcl2Slye5OMl1K+F9/bLvkeP2LPNjvQcAAGwLmxbq3f1od//x8vyLSR5Icm6Sy5LctOx2U5LXLM8vS3Jzr/nDJM+rqhcleWWSO7r7cHc/nuSOJHuWbc/t7o92dye5+ahzrfceAACwLZyWa9Sr6oIk35fkY0le2N2PJmsxn+QFy27nJnl45bCDy+x484PrzHOc9wAAgG1h00O9qp6T5H1Jfrq7v3C8XdeZ9UnMT2RtV1fV/qraf+jQoRM5FAAANtWmhnpVPStrkf5b3f27y/gzy2UrWX4+tswPJjl/5fDzkjzyNPPz1pkf7z2epLtv6O7d3b17586dJ/dLAgDAJtjMu75UkncneaC7f2ll061Jjty5ZW+SD6zMr1zu/nJJks8vl63cnuTSqjp7+RLppUluX7Z9saouWd7ryqPOtd57AADAtrBjE8/9A0l+LMm9VfXxZfbzSX4xyXuq6qokn07yumXbbUleneRAkr9K8oYk6e7DVfWWJHct+725uw8vz38yyY1JvjHJB5dHjvMeAACwLWxaqHf3R7L+deRJ8op19u8k1xzjXPuS7Ftnvj/JS9aZf2699wAAgO3CXyYFAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAGwr1qrpzIzMAAODU2HG8jVX1DUm+Kck5VXV2klo2PTfJt23y2gAA4BnruKGe5CeS/HTWovzufDXUv5DkXZu4LgAAeEY7bqh3968k+ZWq+lfd/aunaU0AAPCM93SfqCdJuvtXq+r7k1ywekx337xJ6wIAgGe0jX6Z9DeTvD3JP0ryD5bH7qc5Zl9VPVZV963M3lRVf15VH18er17Z9nNVdaCqPllVr1yZ71lmB6rq2pX5i6vqY1X1YFX9TlU9e5l//fL6wLL9gg39LwEAAINs6BP1rEX5hd3dJ3DuG5P8hyRHf+r+y9399tVBVV2Y5Iok35O16+H/R1X93WXzu5L8kyQHk9xVVbd29yeSvG051y1V9WtJrkpy/fLz8e7+zqq6Ytnvn5/AugEAYMtt9D7q9yX52ydy4u7+cJLDG9z9siS3dPdfd/efJTmQ5OLlcaC7H+ruLye5JcllVVVJfjjJe5fjb0rympVz3bQ8f2+SVyz7AwDAtrHRUD8nySeq6vaquvXI4yTf841Vdc9yaczZy+zcJA+v7HNwmR1r/q1J/qK7nzhq/qRzLds/v+z/FFV1dVXtr6r9hw4dOslfBwAATr2NXvryplP0ftcneUuSXn6+I8mP56u3fVzVWf8fEn2c/fM025487L4hyQ1Jsnv37hO5rAcAADbVRu/68gen4s26+zNHnlfVryf5b8vLg0nOX9n1vCSPLM/Xm382yfOqasfyqfnq/kfOdbCqdiT5lmz8EhwAABhho3d9+WJVfWF5fKmqvlJVXzjRN6uqF628fG3Wrn1PkluTXLHcseXFSXYl+aMkdyXZtdzh5dlZ+8LprcuXWj+U5PLl+L1JPrByrr3L88uT/P4JfgkWAAC23EY/Uf/m1ddV9ZqsfdHzmKrqt5P8YJJzqupgkuuS/GBVvTRrl6J8Kmt/+TTdfX9VvSfJJ5I8keSa7v7Kcp43Jrk9yVlJ9nX3/ctb/GySW6rqF5L8SZJ3L/N3J/nNqjqQtU/Sr9jI7wgAAJNs9Br1J+nu/7J6T/Nj7PP6dcbvXmd2ZP+3JnnrOvPbkty2zvyhrPOPhe7+UpLXHW9tAAAw3YZCvap+ZOXl12XtvuouJwEAgE2y0U/U/+nK8yeydtnKZad8NQAAQJKNX6P+hs1eCAAA8FUbvevLeVX1/qp6rKo+U1Xvq6rzNntxAADwTLXRv0z6G1m77eG3Ze0vf/7XZQYAAGyCjYb6zu7+je5+YnncmGTnJq4LAACe0TYa6p+tqh+tqrOWx48m+dxmLgwAAJ7JNhrqP57knyX5v0kezdpf/PQFUwAA2CQbvT3jW5Ls7e7Hk6Sqnp/k7VkLeAAA4BTb6Cfq33sk0pOkuw8n+b7NWRIAALDRUP+6qjr7yIvlE/WNfhoPAACcoI3G9juS/K+qem+Sztr16m/dtFUBAMAz3Eb/MunNVbU/yQ8nqSQ/0t2f2NSVAQDAM9iGL19ZwlycAwDAabDRa9QBAIDTSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAw0KaFelXtq6rHquq+ldnzq+qOqnpw+Xn2Mq+qemdVHaiqe6rqZSvH7F32f7Cq9q7ML6qqe5dj3llVdbz3AACA7WQzP1G/Mcmeo2bXJrmzu3cluXN5nSSvSrJreVyd5PpkLbqTXJfk5UkuTnLdSnhfv+x75Lg9T/MeAACwbWxaqHf3h5McPmp8WZKbluc3JXnNyvzmXvOHSZ5XVS9K8sokd3T34e5+PMkdSfYs257b3R/t7k5y81HnWu89AABg2zjd16i/sLsfTZLl5wuW+blJHl7Z7+AyO9784Drz470HAABsG1O+TFrrzPok5if2plVXV9X+qtp/6NChEz0cAAA2zekO9c8sl61k+fnYMj+Y5PyV/c5L8sjTzM9bZ36893iK7r6hu3d39+6dO3ee9C8FAACn2ukO9VuTHLlzy94kH1iZX7nc/eWSJJ9fLlu5PcmlVXX28iXSS5Pcvmz7YlVdstzt5cqjzrXeewAAwLaxY7NOXFW/neQHk5xTVQezdveWX0zynqq6Ksmnk7xu2f22JK9OciDJXyV5Q5J09+GqekuSu5b93tzdR76g+pNZu7PMNyb54PLIcd4DAAC2jU0L9e5+/TE2vWKdfTvJNcc4z74k+9aZ70/yknXmn1vvPQAAYDuZ8mVSAABghVAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMtCWhXlWfqqp7q+rjVbV/mT2/qu6oqgeXn2cv86qqd1bVgaq6p6petnKevcv+D1bV3pX5Rcv5DyzH1un/LQEA4ORt5SfqP9TdL+3u3cvra5Pc2d27kty5vE6SVyXZtTyuTnJ9shb2Sa5L8vIkFye57kjcL/tcvXLcns3/dQAA4NSZdOnLZUluWp7flOQ1K/Obe80fJnleVb0oySuT3NHdh7v78SR3JNmzbHtud3+0uzvJzSvnAgCAbWGrQr2T/Pequruqrl5mL+zuR5Nk+fmCZX5ukodXjj24zI43P7jO/Cmq6uqq2l9V+w8dOvQ1/koAAHDq7Nii9/2B7n6kql6Q5I6q+tPj7Lve9eV9EvOnDrtvSHJDkuzevXvdfQAAYCtsySfq3f3I8vOxJO/P2jXmn1kuW8ny87Fl94NJzl85/LwkjzzN/Lx15gAAsG2c9lCvqr9VVd985HmSS5Pcl+TWJEfu3LI3yQeW57cmuXK5+8slST6/XBpze5JLq+rs5Uuklya5fdn2xaq6ZLnby5Ur5wIAgG1hKy59eWGS9y93TNyR5D939+9V1V1J3lNVVyX5dJLXLfvfluTVSQ4k+askb0iS7j5cVW9Jctey35u7+/Dy/CeT3JjkG5N8cHkAAMC2cdpDvbsfSvL315l/Lskr1pl3kmuOca59SfatM9+f5CVf82IBAGCLTLo9IwAAsBDqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQDu2egEA8LX49Jv/3lYvAdgmvv3f3rvVSzghPlEHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIGEOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBAQh0AAAYS6gAAMJBQBwCAgYQ6AAAMJNQBAGAgoQ4AAAMJdQAAGEioAwDAQEIdAAAGEuoAADCQUAcAgIHO2FCvqj1V9cmqOlBV1271egAA4ESckaFeVWcleVeSVyW5MMnrq+rCrV0VAABs3BkZ6kkuTnKgux/q7i8nuSXJZVu8JgAA2LAzNdTPTfLwyuuDywwAALaFHVu9gE1S68z6KTtVXZ3k6uXlX1bVJzd1VXByzkny2a1eBLPU2/du9RJgOv/t5KmuWy8Rt8R3bGSnMzXUDyY5f+X1eUkeOXqn7r4hyQ2na1FwMqpqf3fv3up1AGwn/tvJmeBMvfTlriS7qurFVfXsJFckuXWL1wQAABt2Rn6i3t1PVNUbk9ye5Kwk+7r7/i1eFgAAbNgZGepJ0t23Jbltq9cBp4DLswBOnP92su1V91O+YwkAAGyxM/UadQAA2NaEOgxWVXuq6pNVdaCqrt3q9QBMV1X7quqxqrpvq9cCXyuhDkNV1VlJ3pXkVUkuTPL6qrpwa1cFMN6NSfZs9SLgVBDqMNfFSQ5090Pd/eUktyS5bIvXBDBad384yeGtXgecCkId5jo3ycMrrw8uMwDgGUCow1zr/Z1jt2kCgGcIoQ5zHUxy/srr85I8skVrAQBOM6EOc92VZFdVvbiqnp3kiiS3bvGaAIDTRKjDUN39RJI3Jrk9yQNJ3tPd92/tqgBmq6rfTvLRJN9VVQer6qqtXhOcLH+ZFAAABvKJOgAADCTUAQBgIKEOAAADCXUAABhIqAMAwEBCHYAnqaq/fJrtF1TVfSd4zhur6vKvbWUAzyxCHQAABhLqAKyrqp5TVXdW1R9X1b1VddnK5h1VdVNV3VNV762qb1qOuaiq/qCq7q6q26vqRVu0fIBtT6gDcCxfSvLa7n5Zkh9K8o6qqmXbdyW5obu/N8kXkvzLqnpWkl9Ncnl3X5RkX5K3bsG6Ac4IO7Z6AQCMVUn+XVX94yR/k+TcJC9ctj3c3f9zef6fkvzrJL+X5CVJ7lh6/qwkj57WFQOcQYQ6AMfyL5LsTHJRd/+/qvpUkm9YtvVR+3bWwv7+7v6Hp2+JAGcul74AcCzfkuSxJdJ/KMl3rGz79qo6EuSvT/KRJJ9MsvPIvKqeVVXfc1pXDHAGEeoAHMtvJdldVfuz9un6n65seyDJ3qq6J8nzk1zf3V9OcnmSt1XV/07y8STff5rXDHDGqO6j/99LAABgq/lEHQAABhLqAAAwkFAHAICBhDoAAAwk1AEAYCChDgAAAwl1AAAYSKgDAMBA/x/h5RJyhIsKYQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's have a look at the distribution of our target variable:\n",
    "# to make it look better, we first convert our spark df to a Pandas\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "df_pd = new_df.toPandas()\n",
    "print(len(df_pd))\n",
    "plt.figure(figsize=(12,10))\n",
    "sns.countplot(x='label', data=df_pd, order=df_pd['label'].value_counts().index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>NAME_CONTRACT_TYPE</th>\n",
       "      <th>CODE_GENDER</th>\n",
       "      <th>FLAG_OWN_CAR</th>\n",
       "      <th>FLAG_OWN_REALTY</th>\n",
       "      <th>CNT_CHILDREN</th>\n",
       "      <th>AMT_INCOME_TOTAL</th>\n",
       "      <th>AMT_CREDIT</th>\n",
       "      <th>AMT_ANNUITY</th>\n",
       "      <th>AMT_GOODS_PRICE</th>\n",
       "      <th>...</th>\n",
       "      <th>FLAG_DOCUMENT_18</th>\n",
       "      <th>FLAG_DOCUMENT_19</th>\n",
       "      <th>FLAG_DOCUMENT_20</th>\n",
       "      <th>FLAG_DOCUMENT_21</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_HOUR</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_DAY</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_WEEK</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_MON</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_QRT</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_YEAR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>202500.0</td>\n",
       "      <td>406597.5</td>\n",
       "      <td>24700.5</td>\n",
       "      <td>351000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>270000.0</td>\n",
       "      <td>1293502.5</td>\n",
       "      <td>35698.5</td>\n",
       "      <td>1129500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>Revolving loans</td>\n",
       "      <td>M</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>67500.0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>6750.0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>312682.5</td>\n",
       "      <td>29686.5</td>\n",
       "      <td>297000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>121500.0</td>\n",
       "      <td>513000.0</td>\n",
       "      <td>21865.5</td>\n",
       "      <td>513000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>99000.0</td>\n",
       "      <td>490495.5</td>\n",
       "      <td>27517.5</td>\n",
       "      <td>454500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>1</td>\n",
       "      <td>171000.0</td>\n",
       "      <td>1560726.0</td>\n",
       "      <td>41301.0</td>\n",
       "      <td>1395000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>360000.0</td>\n",
       "      <td>1530000.0</td>\n",
       "      <td>42075.0</td>\n",
       "      <td>1530000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>112500.0</td>\n",
       "      <td>1019610.0</td>\n",
       "      <td>33826.5</td>\n",
       "      <td>913500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>Revolving loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>405000.0</td>\n",
       "      <td>20250.0</td>\n",
       "      <td>405000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label NAME_CONTRACT_TYPE CODE_GENDER FLAG_OWN_CAR FLAG_OWN_REALTY  \\\n",
       "0      1         Cash loans           M            N               Y   \n",
       "1      0         Cash loans           F            N               N   \n",
       "2      0    Revolving loans           M            Y               Y   \n",
       "3      0         Cash loans           F            N               Y   \n",
       "4      0         Cash loans           M            N               Y   \n",
       "5      0         Cash loans           M            N               Y   \n",
       "6      0         Cash loans           F            Y               Y   \n",
       "7      0         Cash loans           M            Y               Y   \n",
       "8      0         Cash loans           F            N               Y   \n",
       "9      0    Revolving loans           M            N               Y   \n",
       "\n",
       "   CNT_CHILDREN  AMT_INCOME_TOTAL  AMT_CREDIT  AMT_ANNUITY  AMT_GOODS_PRICE  \\\n",
       "0             0          202500.0    406597.5      24700.5         351000.0   \n",
       "1             0          270000.0   1293502.5      35698.5        1129500.0   \n",
       "2             0           67500.0    135000.0       6750.0         135000.0   \n",
       "3             0          135000.0    312682.5      29686.5         297000.0   \n",
       "4             0          121500.0    513000.0      21865.5         513000.0   \n",
       "5             0           99000.0    490495.5      27517.5         454500.0   \n",
       "6             1          171000.0   1560726.0      41301.0        1395000.0   \n",
       "7             0          360000.0   1530000.0      42075.0        1530000.0   \n",
       "8             0          112500.0   1019610.0      33826.5         913500.0   \n",
       "9             0          135000.0    405000.0      20250.0         405000.0   \n",
       "\n",
       "   ... FLAG_DOCUMENT_18 FLAG_DOCUMENT_19 FLAG_DOCUMENT_20 FLAG_DOCUMENT_21  \\\n",
       "0  ...                0                0                0                0   \n",
       "1  ...                0                0                0                0   \n",
       "2  ...                0                0                0                0   \n",
       "3  ...                0                0                0                0   \n",
       "4  ...                0                0                0                0   \n",
       "5  ...                0                0                0                0   \n",
       "6  ...                0                0                0                0   \n",
       "7  ...                0                0                0                0   \n",
       "8  ...                0                0                0                0   \n",
       "9  ...                0                0                0                0   \n",
       "\n",
       "  AMT_REQ_CREDIT_BUREAU_HOUR  AMT_REQ_CREDIT_BUREAU_DAY  \\\n",
       "0                        0.0                        0.0   \n",
       "1                        0.0                        0.0   \n",
       "2                        0.0                        0.0   \n",
       "3                        NaN                        NaN   \n",
       "4                        0.0                        0.0   \n",
       "5                        0.0                        0.0   \n",
       "6                        0.0                        0.0   \n",
       "7                        0.0                        0.0   \n",
       "8                        0.0                        0.0   \n",
       "9                        NaN                        NaN   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_WEEK  AMT_REQ_CREDIT_BUREAU_MON  \\\n",
       "0                         0.0                        0.0   \n",
       "1                         0.0                        0.0   \n",
       "2                         0.0                        0.0   \n",
       "3                         NaN                        NaN   \n",
       "4                         0.0                        0.0   \n",
       "5                         0.0                        0.0   \n",
       "6                         0.0                        1.0   \n",
       "7                         0.0                        0.0   \n",
       "8                         0.0                        0.0   \n",
       "9                         NaN                        NaN   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_QRT  AMT_REQ_CREDIT_BUREAU_YEAR  \n",
       "0                        0.0                         1.0  \n",
       "1                        0.0                         0.0  \n",
       "2                        0.0                         0.0  \n",
       "3                        NaN                         NaN  \n",
       "4                        0.0                         0.0  \n",
       "5                        1.0                         1.0  \n",
       "6                        1.0                         2.0  \n",
       "7                        0.0                         0.0  \n",
       "8                        0.0                         1.0  \n",
       "9                        NaN                         NaN  \n",
       "\n",
       "[10 rows x 121 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's see how everything look in Pandas\n",
    "import pandas as pd\n",
    "pd.DataFrame(new_df.take(10), columns= new_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16  categorical features\n",
      "104  numerical features\n"
     ]
    }
   ],
   "source": [
    "## [Categorical and Numerical column] ##\n",
    "\n",
    "# See how many categorical + numerical features:\n",
    "cat_cols = [item[0] for item in new_df.dtypes if item[1].startswith('string')] \n",
    "num_cols = [item[0] for item in new_df.dtypes if item[1].startswith('int') | item[1].startswith('double')][1:]\n",
    "\n",
    "print(str(len(cat_cols)) + '  categorical features')\n",
    "print(str(len(num_cols)) + '  numerical features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your selected dataframe has 121 columns.\n",
      "There are 67 columns that have missing values.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>% of Total Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>COMMONAREA_MEDI</th>\n",
       "      <td>214865</td>\n",
       "      <td>69.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COMMONAREA_AVG</th>\n",
       "      <td>214865</td>\n",
       "      <td>69.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COMMONAREA_MODE</th>\n",
       "      <td>214865</td>\n",
       "      <td>69.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NONLIVINGAPARTMENTS_MEDI</th>\n",
       "      <td>213514</td>\n",
       "      <td>69.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NONLIVINGAPARTMENTS_MODE</th>\n",
       "      <td>213514</td>\n",
       "      <td>69.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EXT_SOURCE_2</th>\n",
       "      <td>660</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AMT_GOODS_PRICE</th>\n",
       "      <td>278</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AMT_ANNUITY</th>\n",
       "      <td>12</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CNT_FAM_MEMBERS</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DAYS_LAST_PHONE_CHANGE</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>67 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Missing Values  % of Total Values\n",
       "COMMONAREA_MEDI                   214865               69.9\n",
       "COMMONAREA_AVG                    214865               69.9\n",
       "COMMONAREA_MODE                   214865               69.9\n",
       "NONLIVINGAPARTMENTS_MEDI          213514               69.4\n",
       "NONLIVINGAPARTMENTS_MODE          213514               69.4\n",
       "...                                  ...                ...\n",
       "EXT_SOURCE_2                         660                0.2\n",
       "AMT_GOODS_PRICE                      278                0.1\n",
       "AMT_ANNUITY                           12                0.0\n",
       "CNT_FAM_MEMBERS                        2                0.0\n",
       "DAYS_LAST_PHONE_CHANGE                 1                0.0\n",
       "\n",
       "[67 rows x 2 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## [Missing Values] ##\n",
    "\n",
    "# we use the below function to find more information about the #missing values\n",
    "def info_missing_table(df_pd):\n",
    "    \"\"\"Input pandas dataframe and Return columns with missing value and percentage\"\"\"\n",
    "    mis_val = df_pd.isnull().sum() #count total of null in each columns in dataframe\n",
    "#count percentage of null in each columns\n",
    "    mis_val_percent = 100 * df_pd.isnull().sum() / len(df_pd) \n",
    "    mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1) \n",
    " #join to left (as column) between mis_val and mis_val_percent\n",
    "    mis_val_table_ren_columns = mis_val_table.rename(\n",
    "    columns = {0 : 'Missing Values', 1 : '% of Total Values'}) \n",
    "#rename columns in table\n",
    "    mis_val_table_ren_columns = mis_val_table_ren_columns[\n",
    "    mis_val_table_ren_columns.iloc[:,1] != 0].sort_values('% of Total Values', ascending=False).round(1) \n",
    "        \n",
    "    print (\"Your selected dataframe has \" + str(df_pd.shape[1]) + \" columns.\\n\"    #.shape[1] : just view total columns in dataframe  \n",
    "    \"There are \" + str(mis_val_table_ren_columns.shape[0]) +              \n",
    "    \" columns that have missing values.\") #.shape[0] : just view total rows in dataframe\n",
    "    return mis_val_table_ren_columns\n",
    "missings = info_missing_table(df_pd)\n",
    "missings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- There are 67 columns out of 121 that has missing values in them. \n",
    "- Most of these 67 columns have more than 50 percent missing values (which is a lot) \n",
    "- So fill the numerical missing values with the average of each column and the categorical missing values with the most frequent category of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the number of missing values in each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_missings(spark_df):\n",
    "    null_counts = []        \n",
    "    for col in spark_df.dtypes:    \n",
    "        cname = col[0]     \n",
    "        ctype = col[1]      \n",
    "        nulls = spark_df.where( spark_df[cname].isNull()).count() #check count of null in column name\n",
    "        result = tuple([cname, nulls])  #new tuple, (column name, null count)\n",
    "        null_counts.append(result)      #put the new tuple in our result list\n",
    "    null_counts=[(x,y) for (x,y) in null_counts if y!=0]  #view just columns that have missing values\n",
    "    return null_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('AMT_ANNUITY', 12),\n",
       " ('AMT_GOODS_PRICE', 278),\n",
       " ('NAME_TYPE_SUITE', 1292),\n",
       " ('OWN_CAR_AGE', 202929),\n",
       " ('OCCUPATION_TYPE', 96391),\n",
       " ('CNT_FAM_MEMBERS', 2),\n",
       " ('EXT_SOURCE_1', 173378),\n",
       " ('EXT_SOURCE_2', 660),\n",
       " ('EXT_SOURCE_3', 60965),\n",
       " ('APARTMENTS_AVG', 156061),\n",
       " ('BASEMENTAREA_AVG', 179943),\n",
       " ('YEARS_BEGINEXPLUATATION_AVG', 150007),\n",
       " ('YEARS_BUILD_AVG', 204488),\n",
       " ('COMMONAREA_AVG', 214865),\n",
       " ('ELEVATORS_AVG', 163891),\n",
       " ('ENTRANCES_AVG', 154828),\n",
       " ('FLOORSMAX_AVG', 153020),\n",
       " ('FLOORSMIN_AVG', 208642),\n",
       " ('LANDAREA_AVG', 182590),\n",
       " ('LIVINGAPARTMENTS_AVG', 210199),\n",
       " ('LIVINGAREA_AVG', 154350),\n",
       " ('NONLIVINGAPARTMENTS_AVG', 213514),\n",
       " ('NONLIVINGAREA_AVG', 169682),\n",
       " ('APARTMENTS_MODE', 156061),\n",
       " ('BASEMENTAREA_MODE', 179943),\n",
       " ('YEARS_BEGINEXPLUATATION_MODE', 150007),\n",
       " ('YEARS_BUILD_MODE', 204488),\n",
       " ('COMMONAREA_MODE', 214865),\n",
       " ('ELEVATORS_MODE', 163891),\n",
       " ('ENTRANCES_MODE', 154828),\n",
       " ('FLOORSMAX_MODE', 153020),\n",
       " ('FLOORSMIN_MODE', 208642),\n",
       " ('LANDAREA_MODE', 182590),\n",
       " ('LIVINGAPARTMENTS_MODE', 210199),\n",
       " ('LIVINGAREA_MODE', 154350),\n",
       " ('NONLIVINGAPARTMENTS_MODE', 213514),\n",
       " ('NONLIVINGAREA_MODE', 169682),\n",
       " ('APARTMENTS_MEDI', 156061),\n",
       " ('BASEMENTAREA_MEDI', 179943),\n",
       " ('YEARS_BEGINEXPLUATATION_MEDI', 150007),\n",
       " ('YEARS_BUILD_MEDI', 204488),\n",
       " ('COMMONAREA_MEDI', 214865),\n",
       " ('ELEVATORS_MEDI', 163891),\n",
       " ('ENTRANCES_MEDI', 154828),\n",
       " ('FLOORSMAX_MEDI', 153020),\n",
       " ('FLOORSMIN_MEDI', 208642),\n",
       " ('LANDAREA_MEDI', 182590),\n",
       " ('LIVINGAPARTMENTS_MEDI', 210199),\n",
       " ('LIVINGAREA_MEDI', 154350),\n",
       " ('NONLIVINGAPARTMENTS_MEDI', 213514),\n",
       " ('NONLIVINGAREA_MEDI', 169682),\n",
       " ('FONDKAPREMONT_MODE', 210295),\n",
       " ('HOUSETYPE_MODE', 154297),\n",
       " ('TOTALAREA_MODE', 148431),\n",
       " ('WALLSMATERIAL_MODE', 156341),\n",
       " ('EMERGENCYSTATE_MODE', 145755),\n",
       " ('OBS_30_CNT_SOCIAL_CIRCLE', 1021),\n",
       " ('DEF_30_CNT_SOCIAL_CIRCLE', 1021),\n",
       " ('OBS_60_CNT_SOCIAL_CIRCLE', 1021),\n",
       " ('DEF_60_CNT_SOCIAL_CIRCLE', 1021),\n",
       " ('DAYS_LAST_PHONE_CHANGE', 1),\n",
       " ('AMT_REQ_CREDIT_BUREAU_HOUR', 41519),\n",
       " ('AMT_REQ_CREDIT_BUREAU_DAY', 41519),\n",
       " ('AMT_REQ_CREDIT_BUREAU_WEEK', 41519),\n",
       " ('AMT_REQ_CREDIT_BUREAU_MON', 41519),\n",
       " ('AMT_REQ_CREDIT_BUREAU_QRT', 41519),\n",
       " ('AMT_REQ_CREDIT_BUREAU_YEAR', 41519)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "miss_counts = count_missings(new_df)\n",
    "miss_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cateogrical columns_miss: ['NAME_TYPE_SUITE', 'OCCUPATION_TYPE', 'FONDKAPREMONT_MODE', 'HOUSETYPE_MODE', 'WALLSMATERIAL_MODE', 'EMERGENCYSTATE_MODE']\n",
      "numerical columns_miss: ['AMT_ANNUITY', 'AMT_GOODS_PRICE', 'OWN_CAR_AGE', 'CNT_FAM_MEMBERS', 'EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'APARTMENTS_AVG', 'BASEMENTAREA_AVG', 'YEARS_BEGINEXPLUATATION_AVG', 'YEARS_BUILD_AVG', 'COMMONAREA_AVG', 'ELEVATORS_AVG', 'ENTRANCES_AVG', 'FLOORSMAX_AVG', 'FLOORSMIN_AVG', 'LANDAREA_AVG', 'LIVINGAPARTMENTS_AVG', 'LIVINGAREA_AVG', 'NONLIVINGAPARTMENTS_AVG', 'NONLIVINGAREA_AVG', 'APARTMENTS_MODE', 'BASEMENTAREA_MODE', 'YEARS_BEGINEXPLUATATION_MODE', 'YEARS_BUILD_MODE', 'COMMONAREA_MODE', 'ELEVATORS_MODE', 'ENTRANCES_MODE', 'FLOORSMAX_MODE', 'FLOORSMIN_MODE', 'LANDAREA_MODE', 'LIVINGAPARTMENTS_MODE', 'LIVINGAREA_MODE', 'NONLIVINGAPARTMENTS_MODE', 'NONLIVINGAREA_MODE', 'APARTMENTS_MEDI', 'BASEMENTAREA_MEDI', 'YEARS_BEGINEXPLUATATION_MEDI', 'YEARS_BUILD_MEDI', 'COMMONAREA_MEDI', 'ELEVATORS_MEDI', 'ENTRANCES_MEDI', 'FLOORSMAX_MEDI', 'FLOORSMIN_MEDI', 'LANDAREA_MEDI', 'LIVINGAPARTMENTS_MEDI', 'LIVINGAREA_MEDI', 'NONLIVINGAPARTMENTS_MEDI', 'NONLIVINGAREA_MEDI', 'TOTALAREA_MODE', 'OBS_30_CNT_SOCIAL_CIRCLE', 'DEF_30_CNT_SOCIAL_CIRCLE', 'OBS_60_CNT_SOCIAL_CIRCLE', 'DEF_60_CNT_SOCIAL_CIRCLE', 'DAYS_LAST_PHONE_CHANGE', 'AMT_REQ_CREDIT_BUREAU_HOUR', 'AMT_REQ_CREDIT_BUREAU_DAY', 'AMT_REQ_CREDIT_BUREAU_WEEK', 'AMT_REQ_CREDIT_BUREAU_MON', 'AMT_REQ_CREDIT_BUREAU_QRT', 'AMT_REQ_CREDIT_BUREAU_YEAR']\n"
     ]
    }
   ],
   "source": [
    "# Separate categorical and numerical columns with missing values based on types\n",
    "\n",
    "list_cols_miss=[x[0] for x in miss_counts]\n",
    "df_miss= new_df.select(*list_cols_miss)\n",
    "\n",
    "# categorical columns\n",
    "catcolums_miss=[item[0] for item in df_miss.dtypes if item[1].startswith('string')]  #will select name of column with string data type\n",
    "print(\"cateogrical columns_miss:\", catcolums_miss)\n",
    "\n",
    "# numerical columns\n",
    "numcolumns_miss = [item[0] for item in df_miss.dtypes if item[1].startswith('int') | item[1].startswith('double')] #will select name of column with integer or double data type\n",
    "print(\"numerical columns_miss:\", numcolumns_miss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill the missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME_TYPE_SUITE Unaccompanied\n",
      "OCCUPATION_TYPE Laborers\n",
      "FONDKAPREMONT_MODE reg oper account\n",
      "HOUSETYPE_MODE block of flats\n",
      "WALLSMATERIAL_MODE Panel\n",
      "EMERGENCYSTATE_MODE No\n"
     ]
    }
   ],
   "source": [
    "# (1) categorical --> the most frequent category\n",
    "\n",
    "from pyspark.sql.functions import rank,sum,col\n",
    "df_Nomiss=new_df.na.drop()\n",
    "for x in catcolums_miss:                  \n",
    "    mode=df_Nomiss.groupBy(x).count().sort(col(\"count\").desc()).collect()[0][0] \n",
    "    print(x, mode) #print name of columns and it's most categories \n",
    "    new_df = new_df.na.fill({x:mode})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AMT_ANNUITY 27109.0\n",
      "AMT_GOODS_PRICE 538396.0\n",
      "OWN_CAR_AGE 12.0\n",
      "CNT_FAM_MEMBERS 2.0\n",
      "EXT_SOURCE_1 1.0\n",
      "EXT_SOURCE_2 1.0\n",
      "EXT_SOURCE_3 1.0\n",
      "APARTMENTS_AVG 0.0\n",
      "BASEMENTAREA_AVG 0.0\n",
      "YEARS_BEGINEXPLUATATION_AVG 1.0\n",
      "YEARS_BUILD_AVG 1.0\n",
      "COMMONAREA_AVG 0.0\n",
      "ELEVATORS_AVG 0.0\n",
      "ENTRANCES_AVG 0.0\n",
      "FLOORSMAX_AVG 0.0\n",
      "FLOORSMIN_AVG 0.0\n",
      "LANDAREA_AVG 0.0\n",
      "LIVINGAPARTMENTS_AVG 0.0\n",
      "LIVINGAREA_AVG 0.0\n",
      "NONLIVINGAPARTMENTS_AVG 0.0\n",
      "NONLIVINGAREA_AVG 0.0\n",
      "APARTMENTS_MODE 0.0\n",
      "BASEMENTAREA_MODE 0.0\n",
      "YEARS_BEGINEXPLUATATION_MODE 1.0\n",
      "YEARS_BUILD_MODE 1.0\n",
      "COMMONAREA_MODE 0.0\n",
      "ELEVATORS_MODE 0.0\n",
      "ENTRANCES_MODE 0.0\n",
      "FLOORSMAX_MODE 0.0\n",
      "FLOORSMIN_MODE 0.0\n",
      "LANDAREA_MODE 0.0\n",
      "LIVINGAPARTMENTS_MODE 0.0\n",
      "LIVINGAREA_MODE 0.0\n",
      "NONLIVINGAPARTMENTS_MODE 0.0\n",
      "NONLIVINGAREA_MODE 0.0\n",
      "APARTMENTS_MEDI 0.0\n",
      "BASEMENTAREA_MEDI 0.0\n",
      "YEARS_BEGINEXPLUATATION_MEDI 1.0\n",
      "YEARS_BUILD_MEDI 1.0\n",
      "COMMONAREA_MEDI 0.0\n",
      "ELEVATORS_MEDI 0.0\n",
      "ENTRANCES_MEDI 0.0\n",
      "FLOORSMAX_MEDI 0.0\n",
      "FLOORSMIN_MEDI 0.0\n",
      "LANDAREA_MEDI 0.0\n",
      "LIVINGAPARTMENTS_MEDI 0.0\n",
      "LIVINGAREA_MEDI 0.0\n",
      "NONLIVINGAPARTMENTS_MEDI 0.0\n",
      "NONLIVINGAREA_MEDI 0.0\n",
      "TOTALAREA_MODE 0.0\n",
      "OBS_30_CNT_SOCIAL_CIRCLE 1.0\n",
      "DEF_30_CNT_SOCIAL_CIRCLE 0.0\n",
      "OBS_60_CNT_SOCIAL_CIRCLE 1.0\n",
      "DEF_60_CNT_SOCIAL_CIRCLE 0.0\n",
      "DAYS_LAST_PHONE_CHANGE -963.0\n",
      "AMT_REQ_CREDIT_BUREAU_HOUR 0.0\n",
      "AMT_REQ_CREDIT_BUREAU_DAY 0.0\n",
      "AMT_REQ_CREDIT_BUREAU_WEEK 0.0\n",
      "AMT_REQ_CREDIT_BUREAU_MON 0.0\n",
      "AMT_REQ_CREDIT_BUREAU_QRT 0.0\n",
      "AMT_REQ_CREDIT_BUREAU_YEAR 2.0\n"
     ]
    }
   ],
   "source": [
    "# (2) numerical --> average value of each column\n",
    "\n",
    "from pyspark.sql.functions import mean, round\n",
    "for i in numcolumns_miss:\n",
    "    meanvalue = new_df.select(round(mean(i))).collect()[0][0] \n",
    "    print(i, meanvalue) \n",
    "    new_df=new_df.na.fill({i:meanvalue})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let’s work on how to deal with the imbalanced classes. there are different methods to mitigate this problem. one way is to under sample the majority class or over sample the minority class to make a more balanced results. another way is to assign weights for each class to penalize the majority class by assigning less weight and boost the minority class by assigning bigger weight. we are going to create a new column in the dataset named “weights” and assign the inverse ratio of each class as weights. here is how it’s done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adding the new column weights and fill it with ratios\n",
    "from pyspark.sql.functions import when\n",
    "ratio = 0.91\n",
    "def weight_balance(labels):\n",
    "    return when(labels == 1, ratio).otherwise(1*(1-ratio))\n",
    "new_df = new_df.withColumn('weights', weight_balance(col('label')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>NAME_CONTRACT_TYPE</th>\n",
       "      <th>CODE_GENDER</th>\n",
       "      <th>FLAG_OWN_CAR</th>\n",
       "      <th>FLAG_OWN_REALTY</th>\n",
       "      <th>CNT_CHILDREN</th>\n",
       "      <th>AMT_INCOME_TOTAL</th>\n",
       "      <th>AMT_CREDIT</th>\n",
       "      <th>AMT_ANNUITY</th>\n",
       "      <th>AMT_GOODS_PRICE</th>\n",
       "      <th>...</th>\n",
       "      <th>FLAG_DOCUMENT_19</th>\n",
       "      <th>FLAG_DOCUMENT_20</th>\n",
       "      <th>FLAG_DOCUMENT_21</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_HOUR</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_DAY</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_WEEK</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_MON</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_QRT</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_YEAR</th>\n",
       "      <th>weights</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>202500.0</td>\n",
       "      <td>406597.5</td>\n",
       "      <td>24700.5</td>\n",
       "      <td>351000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>270000.0</td>\n",
       "      <td>1293502.5</td>\n",
       "      <td>35698.5</td>\n",
       "      <td>1129500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>Revolving loans</td>\n",
       "      <td>M</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>67500.0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>6750.0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>312682.5</td>\n",
       "      <td>29686.5</td>\n",
       "      <td>297000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>121500.0</td>\n",
       "      <td>513000.0</td>\n",
       "      <td>21865.5</td>\n",
       "      <td>513000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>99000.0</td>\n",
       "      <td>490495.5</td>\n",
       "      <td>27517.5</td>\n",
       "      <td>454500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>1</td>\n",
       "      <td>171000.0</td>\n",
       "      <td>1560726.0</td>\n",
       "      <td>41301.0</td>\n",
       "      <td>1395000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>360000.0</td>\n",
       "      <td>1530000.0</td>\n",
       "      <td>42075.0</td>\n",
       "      <td>1530000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>112500.0</td>\n",
       "      <td>1019610.0</td>\n",
       "      <td>33826.5</td>\n",
       "      <td>913500.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>Revolving loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>405000.0</td>\n",
       "      <td>20250.0</td>\n",
       "      <td>405000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 122 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label NAME_CONTRACT_TYPE CODE_GENDER FLAG_OWN_CAR FLAG_OWN_REALTY  \\\n",
       "0      1         Cash loans           M            N               Y   \n",
       "1      0         Cash loans           F            N               N   \n",
       "2      0    Revolving loans           M            Y               Y   \n",
       "3      0         Cash loans           F            N               Y   \n",
       "4      0         Cash loans           M            N               Y   \n",
       "5      0         Cash loans           M            N               Y   \n",
       "6      0         Cash loans           F            Y               Y   \n",
       "7      0         Cash loans           M            Y               Y   \n",
       "8      0         Cash loans           F            N               Y   \n",
       "9      0    Revolving loans           M            N               Y   \n",
       "\n",
       "   CNT_CHILDREN  AMT_INCOME_TOTAL  AMT_CREDIT  AMT_ANNUITY  AMT_GOODS_PRICE  \\\n",
       "0             0          202500.0    406597.5      24700.5         351000.0   \n",
       "1             0          270000.0   1293502.5      35698.5        1129500.0   \n",
       "2             0           67500.0    135000.0       6750.0         135000.0   \n",
       "3             0          135000.0    312682.5      29686.5         297000.0   \n",
       "4             0          121500.0    513000.0      21865.5         513000.0   \n",
       "5             0           99000.0    490495.5      27517.5         454500.0   \n",
       "6             1          171000.0   1560726.0      41301.0        1395000.0   \n",
       "7             0          360000.0   1530000.0      42075.0        1530000.0   \n",
       "8             0          112500.0   1019610.0      33826.5         913500.0   \n",
       "9             0          135000.0    405000.0      20250.0         405000.0   \n",
       "\n",
       "   ... FLAG_DOCUMENT_19 FLAG_DOCUMENT_20 FLAG_DOCUMENT_21  \\\n",
       "0  ...                0                0                0   \n",
       "1  ...                0                0                0   \n",
       "2  ...                0                0                0   \n",
       "3  ...                0                0                0   \n",
       "4  ...                0                0                0   \n",
       "5  ...                0                0                0   \n",
       "6  ...                0                0                0   \n",
       "7  ...                0                0                0   \n",
       "8  ...                0                0                0   \n",
       "9  ...                0                0                0   \n",
       "\n",
       "  AMT_REQ_CREDIT_BUREAU_HOUR AMT_REQ_CREDIT_BUREAU_DAY  \\\n",
       "0                        0.0                       0.0   \n",
       "1                        0.0                       0.0   \n",
       "2                        0.0                       0.0   \n",
       "3                        0.0                       0.0   \n",
       "4                        0.0                       0.0   \n",
       "5                        0.0                       0.0   \n",
       "6                        0.0                       0.0   \n",
       "7                        0.0                       0.0   \n",
       "8                        0.0                       0.0   \n",
       "9                        0.0                       0.0   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_WEEK  AMT_REQ_CREDIT_BUREAU_MON  \\\n",
       "0                         0.0                        0.0   \n",
       "1                         0.0                        0.0   \n",
       "2                         0.0                        0.0   \n",
       "3                         0.0                        0.0   \n",
       "4                         0.0                        0.0   \n",
       "5                         0.0                        0.0   \n",
       "6                         0.0                        1.0   \n",
       "7                         0.0                        0.0   \n",
       "8                         0.0                        0.0   \n",
       "9                         0.0                        0.0   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_QRT  AMT_REQ_CREDIT_BUREAU_YEAR  weights  \n",
       "0                        0.0                         1.0     0.91  \n",
       "1                        0.0                         0.0     0.09  \n",
       "2                        0.0                         0.0     0.09  \n",
       "3                        0.0                         2.0     0.09  \n",
       "4                        0.0                         0.0     0.09  \n",
       "5                        1.0                         1.0     0.09  \n",
       "6                        1.0                         2.0     0.09  \n",
       "7                        0.0                         0.0     0.09  \n",
       "8                        0.0                         1.0     0.09  \n",
       "9                        0.0                         2.0     0.09  \n",
       "\n",
       "[10 rows x 122 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(new_df.take(10), columns= new_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PySpark has a great feature engineering handling, so that we do not need to do much for extracting features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Apply StringIndexer() to assign indices to each category in our categorical columns\n",
    "2. Apply OneHotEncoderEstimator() to convert categorical columns to onehot encoded vectors\n",
    "3. Apply VectorAssembler() to create a feature vector from all categorical and numerical features and we call the final vector as “features”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use the OneHotEncoderEstimator from MLlib in spark to convert #aech v=categorical feature into one-hot vectors\n",
    "# next, we use VectorAssembler to combine the resulted one-hot ector #and the rest of numerical features into a \n",
    "# single vector column. we append every step of the process in a #stages array\n",
    "from pyspark.ml.feature import OneHotEncoderEstimator, StringIndexer, VectorAssembler\n",
    "stages = []\n",
    "for categoricalCol in cat_cols:\n",
    "    stringIndexer = StringIndexer(inputCol = categoricalCol, outputCol = categoricalCol + 'Index')\n",
    "    encoder = OneHotEncoderEstimator(inputCols=[stringIndexer.getOutputCol()], outputCols=[categoricalCol + \"classVec\"])\n",
    "    stages += [stringIndexer, encoder]\n",
    "    \n",
    "assemblerInputs = [c + \"classVec\" for c in cat_cols] + num_cols\n",
    "assembler = VectorAssembler(inputCols=assemblerInputs, outputCol=\"features\")\n",
    "stages += [assembler]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s put everything into a pipeline. here we are performing a sequence of transformations so we do all of them at once using a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "cols = new_df.columns\n",
    "pipeline = Pipeline(stages = stages)\n",
    "pipelineModel = pipeline.fit(new_df)\n",
    "new_df = pipelineModel.transform(new_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>features</th>\n",
       "      <th>label</th>\n",
       "      <th>NAME_CONTRACT_TYPE</th>\n",
       "      <th>CODE_GENDER</th>\n",
       "      <th>FLAG_OWN_CAR</th>\n",
       "      <th>FLAG_OWN_REALTY</th>\n",
       "      <th>CNT_CHILDREN</th>\n",
       "      <th>AMT_INCOME_TOTAL</th>\n",
       "      <th>AMT_CREDIT</th>\n",
       "      <th>AMT_ANNUITY</th>\n",
       "      <th>...</th>\n",
       "      <th>FLAG_DOCUMENT_19</th>\n",
       "      <th>FLAG_DOCUMENT_20</th>\n",
       "      <th>FLAG_DOCUMENT_21</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_HOUR</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_DAY</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_WEEK</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_MON</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_QRT</th>\n",
       "      <th>AMT_REQ_CREDIT_BUREAU_YEAR</th>\n",
       "      <th>weights</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>202500.0</td>\n",
       "      <td>406597.5</td>\n",
       "      <td>24700.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>270000.0</td>\n",
       "      <td>1293502.5</td>\n",
       "      <td>35698.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Revolving loans</td>\n",
       "      <td>M</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>67500.0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>6750.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>F</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>135000.0</td>\n",
       "      <td>312682.5</td>\n",
       "      <td>29686.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Cash loans</td>\n",
       "      <td>M</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>121500.0</td>\n",
       "      <td>513000.0</td>\n",
       "      <td>21865.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 123 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            features  label  \\\n",
       "0  (1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...      1   \n",
       "1  (1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, ...      0   \n",
       "2  (0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...      0   \n",
       "3  (1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...      0   \n",
       "4  (1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, ...      0   \n",
       "\n",
       "  NAME_CONTRACT_TYPE CODE_GENDER FLAG_OWN_CAR FLAG_OWN_REALTY  CNT_CHILDREN  \\\n",
       "0         Cash loans           M            N               Y             0   \n",
       "1         Cash loans           F            N               N             0   \n",
       "2    Revolving loans           M            Y               Y             0   \n",
       "3         Cash loans           F            N               Y             0   \n",
       "4         Cash loans           M            N               Y             0   \n",
       "\n",
       "   AMT_INCOME_TOTAL  AMT_CREDIT  AMT_ANNUITY  ...  FLAG_DOCUMENT_19  \\\n",
       "0          202500.0    406597.5      24700.5  ...                 0   \n",
       "1          270000.0   1293502.5      35698.5  ...                 0   \n",
       "2           67500.0    135000.0       6750.0  ...                 0   \n",
       "3          135000.0    312682.5      29686.5  ...                 0   \n",
       "4          121500.0    513000.0      21865.5  ...                 0   \n",
       "\n",
       "  FLAG_DOCUMENT_20 FLAG_DOCUMENT_21 AMT_REQ_CREDIT_BUREAU_HOUR  \\\n",
       "0                0                0                        0.0   \n",
       "1                0                0                        0.0   \n",
       "2                0                0                        0.0   \n",
       "3                0                0                        0.0   \n",
       "4                0                0                        0.0   \n",
       "\n",
       "  AMT_REQ_CREDIT_BUREAU_DAY AMT_REQ_CREDIT_BUREAU_WEEK  \\\n",
       "0                       0.0                        0.0   \n",
       "1                       0.0                        0.0   \n",
       "2                       0.0                        0.0   \n",
       "3                       0.0                        0.0   \n",
       "4                       0.0                        0.0   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_MON  AMT_REQ_CREDIT_BUREAU_QRT  \\\n",
       "0                        0.0                        0.0   \n",
       "1                        0.0                        0.0   \n",
       "2                        0.0                        0.0   \n",
       "3                        0.0                        0.0   \n",
       "4                        0.0                        0.0   \n",
       "\n",
       "   AMT_REQ_CREDIT_BUREAU_YEAR  weights  \n",
       "0                         1.0     0.91  \n",
       "1                         0.0     0.09  \n",
       "2                         0.0     0.09  \n",
       "3                         2.0     0.09  \n",
       "4                         0.0     0.09  \n",
       "\n",
       "[5 rows x 123 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selectedCols = ['features']+cols\n",
    "new_df = new_df.select(selectedCols)\n",
    "\n",
    "# New dataset after feature engineering:\n",
    "pd.DataFrame(new_df.take(5), columns=new_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Start the Training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "246015\n",
      "61496\n"
     ]
    }
   ],
   "source": [
    "# split the data into trainign and testin sets\n",
    "train, test = new_df.randomSplit([0.80, 0.20], seed = 42)\n",
    "print(train.count())\n",
    "print(test.count())\n",
    "\n",
    "# first we check how LogisticRegression perform \n",
    "from pyspark.ml.classification import LogisticRegression\n",
    "LR = LogisticRegression(featuresCol = 'features', labelCol = 'label', maxIter=15)\n",
    "LR_model = LR.fit(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8lOW5//HPxRL2sIUdQlgFZG9kcd9QxCrWFXetlh6rbY/1tLWbPVXbX6u1Vj0udV/qXjeqVNxwVyCy7wQSIGxJCCQESMhy/f6YIcaYZQKZTGbm+369eDnPzD0z3yeJc81zP/dz3+buiIiIADSLdAAREWk6VBRERKSCioKIiFRQURARkQoqCiIiUkFFQUREKqgoiIhIBRUFiSlmlmlm+82s0My2m9mTZta+SpujzewDM9tjZvlm9m8zG1GlTaKZ/d3MNgVfKz24nVTD+5qZ/cTMlpvZXjPLMrOXzWxUOPdXpKGpKEgsOsvd2wNjgXHArw4+YGaTgXeAN4DewABgCfCZmQ0MtkkA3geOBKYCicDRwE5gQg3veQ/wU+AnQBdgKPA6cGZ9w5tZi/o+R6ShmK5ollhiZpnAte7+XnD7DuBIdz8zuP0JsMzdf1Tlef8Bctz9CjO7FvgjMMjdC0N4zyHAamCyu8+voc2HwD/d/dHg9lXBnMcGtx24AfhvoAUwByh09/+p9BpvAB+5+9/MrDdwH3A8UAjc7e73hvAjEqmVjhQkZplZX+AMID243ZbAN/6Xq2n+EjAlePtU4O1QCkLQKUBWTQWhHs4BJgIjgOeAi8zMAMysM3Aa8IKZNQP+TeAIp0/w/f/bzE4/zPcXUVGQmPS6me0BNgPZwO+D93ch8De/rZrnbAMOni/oWkObmtS3fU3+n7vnuft+4BPAgeOCj50PfOHuW4GjgG7ufqu7H3D3DcAjwIwGyCBxTkVBYtE57t4BOBEYxtcf9ruAcqBXNc/pBeQGb++soU1N6tu+JpsP3vBAv+4LwMXBuy4Bng3e7g/0NrPdB/8BvwZ6NEAGiXMqChKz3P0j4Engr8HtvcAXwAXVNL+QwMllgPeA082sXYhv9T7Q18xSa2mzF2hbabtndZGrbD8PnG9m/Ql0K70SvH8zkOHunSr96+Du00LMK1IjFQWJdX8HppjZ2OD2zcCVweGjHcyss5ndDkwG/hBs8wyBD95XzGyYmTUzs65m9msz+9YHr7uvAx4AnjezE80swcxam9kMM7s52GwxcK6ZtTWzwcA1dQV390VADvAoMMfddwcfmg8UmNkvzayNmTU3s5FmdtSh/IBEKlNRkJjm7jnA08DvgtufAqcD5xI4D7CRwLDVY4Mf7rh7MYGTzauBd4ECAh/EScC8Gt7qJ8D/AfcDu4H1wPcInBAGuBs4AOwAnuLrrqC6PB/M8lylfSoDziIw5DaDQLfXo0DHEF9TpEYakioiIhV0pCAiIhVUFEREpIKKgoiIVFBREBGRClE38VZSUpKnpKREOoaISFT56quvct29W13toq4opKSkkJaWFukYIiJRxcw2htJO3UciIlJBRUFERCqoKIiISAUVBRERqaCiICIiFcJWFMzscTPLNrPlNTxuZnZvcEH0pWY2PlxZREQkNOE8UniSwKLnNTkDGBL8NxN4MIxZREQkBGG7TsHdPzazlFqaTAeeDq4w9aWZdTKzXu7eEMsaiohEraKSMrbnF7G9oIgdBUXsLDxA/v4SThnendF9O4X1vSN58VofKi0/CGQF7/tWUTCzmQSOJkhOTm6UcCIi4VBe7mwvKCIzdy+bd+1jR0Fx4MM/v4it+UVsz9/Prn0l1T63W4dWMV0UrJr7ql3cwd0fBh4GSE1N1QIQItLk5e8rIT1nD+t2FJKxcy+ZuXvJzN1H5s69FJeWf6Ntl3YJdO/Qit6d2jA+uRO9OramZ8c29ExsTY/EViS1b0Vim5Y0b1bdx2bDimRRyAL6VdruC2yNUBYRkXpzd3buPcC6HYWkZ+9hXXYh6dmFrMsuJGdPcUW7hObN6NelDQOS2nHckCRSktoxIKkdyV3a0j2xFa1aNI/gXnxTJIvCLOAGM3uBwKLk+TqfICJNlbuzKW8fS7PyWbYln6VZu1m9fQ+7K3X1tG/VgsHd23PC0G4M7t6eId3bM7h7e/p2btso3/IbQtiKgpk9D5wIJJlZFvB7oCWAuz8EzAamAenAPuDqcGUREamPsnJnQ04hK7YWsHJbAcuy8lm+NZ89RaVA4Jv/8N6JnDGyF0O6t2dIj8CHf8/E1phFx4d/TcI5+ujiOh534Ppwvb+ISCh27zvAqm17WL29gNXB/67ZsYeikkC/f0KLZgzv2YGzxvRmZO+OjO7bkaE9OpDQIjav/Y26qbNFRA5VUUkZS7PySduYx8KNu1i+pYDtBUUVj3dpl8DwXh24ZEJ/juydyJF9EhnUrT0tm8dmAaiOioKIxKTi0jJWbC1g0abdrNiaz8qtBazLLqSsPDCAcWC3dkwa2IXhvRIZ3iuRYT070K1Dq6jv/jlcKgoiEvXcna35RSzetJuFm3axaFPgKOBAWaALqFuHVhzZO5FThndnXL/OjO/fmS7tEiKcumlSURCRqLPvQCnzNuTx5YadLA8eBRy84CuhRTNG9+nIlUf35zv9OzMuuTM9EltHOHH0UFEQkSZv194DLN68m0WbdjE/M4+vNu6ipMxJaN6MoT3bc9qInhzZJ5ExfTsxvFdizJ4EbgwqCiLSpLg7G3fuY35mHmmZeaRl7mJD7l4AmhkM75XI948ZwLFDkjgqpQutWzadC79igYqCiERUSVk5K7cWsHDTLhZk5rEgc1fF1cCd2rYktX8XLkjtx7jkTozq05F2rfSxFU766YpIoyopK2fJ5t18si6XL9bvZEnW7oq5gPp0asMxg7py1IAuTEjpwqBu7WkWJVcCxwoVBREJu235+/lwTQ4frsnm8/Sd7CkupZnBqD4duXRif8b378T45M707tQm0lHjnoqCiDS4/QfKmJexk0/W5fLpulzW7NgDQO+OrfnumF4cP6QbRw9KomPblhFOKlWpKIjIYSsqKWNeRmCI6PyMPJZm7Q6MDmrRjAkpXTh3fB9OGtadId3bx/3FYU2dioKIHJLNefv4cG0Oc1dn8/n6XIpKymnRzBjVtyPXHDuQyYO6MnGARgdFGxUFEQnJgdJy5mfk8cHqbD5cm82GnMAw0eQubZlxVDInHtGNCQO60DZBHyvRTL89EanR1t37+WRdDh+vzeXjtTnsKS4loUUzJg3symUT+3PCEd0YmNROXUIxREVBRCrsO1DKvIw8Pl6bw8drc1gfPBrokdiKaaN6ceqIHhw7OIk2CeoSilUqCiJxrqCohPdX7WD2su18tDaHA6XltGrRjIkDu3LxhGSOH9pNJ4jjiIqCSBwqKinjwzXZvLZoC3NX53CgrJyeia25ZEIyJw/rzgSdII5bKgoicaKopIxP1uUyZ8V23lmxnYKiUpLat+KySf05c3QvxvXrpKuHRUVBJJbtPxA4Inhr2TY+WJ3NvgNldGjdginDe3DOuD4cPagrLeJoVTGpm4qCSIxxd77ckMeLCzbxzsod7DtQRtd2CUwf24czRvZk0sCumlpaaqSiIBIjcguLeeWrLF5YsJmM3L10aN2C6WP7cNboXkwY0EVHBBISFQWRKFZW7nyWnssLCzbx7sodlJQ5R6V05oaTBjNtVC8NHZV6U1EQiTLuzsJNu/j3km28uXQbuYXFdG7bkisnpzBjQj8Gd+8Q6YgSxVQURKJEdkER/1qYxctpWWTk7qVVi2acMrw7Z43uzcnDu9OqhY4K5PCpKIg0YeXlzufrd/L0F5m8vzqbsnJnwoAu/OjEQUwd2ZMOrTX1tDQsFQWRJqigqITXFm7hqc8z2ZC7ly7tErj2uAFclNqPgd3aRzqexDAVBZEmwt1ZvHk3z83bxL+XbqWopJwx/Tpx90VjOGNkL11hLI1CRUEkwgqKSnhj0Raem7+ZVdsKaJvQnHPG9uGSicmM7tsp0vEkzqgoiETI6u0FPP3FRl5buIX9JWWM6JXI7eeMZPrY3jpXIBGjoiDSiErKynlnxQ6e/iKTeRl5tGrRjOlje3PZpP6M6tNRM5FKxKkoiDSCHQVFPD9/E8/P38SOgmL6dm7Dr84YxoWp/ejcLiHS8UQqhLUomNlU4B6gOfCou/+5yuPJwFNAp2Cbm919djgziTQWd+eL9Tv557yNzFmxg7Jy54Sh3fjT9/pz4hHdaa4ZSaUJCltRMLPmwP3AFCALWGBms9x9ZaVmvwVecvcHzWwEMBtICVcmkcbg7nywOpu731vL8i0FdGrbkmuOHcAlE5JJSWoX6XgitQrnkcIEIN3dNwCY2QvAdKByUXAgMXi7I7A1jHlEwsrdmbsmm3veW8eSrHySu7TljvNGc/bY3hpOKlEjnEWhD7C50nYWMLFKm/8F3jGzHwPtgFOreyEzmwnMBEhOTm7woCKHo6zcmb1sG/fPTWf19j307dyGv5w3inPH96WlZiaVKBPOolBdh6lX2b4YeNLd7zKzycAzZjbS3cu/8ST3h4GHAVJTU6u+hkhE7Ckq4eW0LJ78PJNNefsY2K0df71gDNPH9lYxkKgVzqKQBfSrtN2Xb3cPXQNMBXD3L8ysNZAEZIcxl8hhyS4o4pFPNvDC/M3sKS7lO/0786szhnHakT118liiXjiLwgJgiJkNALYAM4BLqrTZBJwCPGlmw4HWQE4YM4kcsi279/OPj9bzwoLNlJU7Z47qxfePHcDYfrrqWGJH2IqCu5ea2Q3AHALDTR939xVmdiuQ5u6zgJuAR8zsRgJdS1e5u7qHpEnZnl/E/81dx4sLAqfIzhvfl+tOHET/rhpJJLEnrNcpBK85mF3lvlsq3V4JHBPODCKHanPePh77NIPn5m+ivNy58Kh+XH/SYPp0ahPpaCJhoyuaRapIy8zj0U8yeGfldpqZce74Pvz45CH069I20tFEwk5FQSQoM3cvt7+1ivdW7aBjm5b88IRBXDG5P7066shA4oeKgsS9HQVFPDA3nefnb6Zlc+OXU4dx5dH9aZug/z0k/uivXuJWdkERD360nmfnBc4ZXJDalxunDKV7h9aRjiYSMSoKEneydu3jHx9t4MW0wNDS83TOQKSCioLEjfTsPTz00QZeX7QFs8DQ0v86YZAmqROpREVBYt7yLfnc+/463lm5g9Ytm3HZpP7MPH4gvTW0VORbVBQkZm3PL+KOOat5deEWElu34CcnD+bKo1Po2r5VpKOJNFl1FgUza0vgyuNkd/+BmQ0BjnD3N8OeTuQQ5O8v4bFPM3jk4w2UlTs/PGEg1580mESteyxSp1COFJ4AvgImB7ezgJcBFQVpUvYUlfDEZ5k8+skGCopKmTaqJzdPHU5yV51AFglVKEVhkLtfZGYXA7j7ftPq4tKElJSV89y8Tdzz/jry9h5gyoge/PepQziyd8dIRxOJOqEUhQNm1obgWghmNggoDmsqkRC4O++vyuZP/1nFhpy9TB7YlV9NG8bovpq1VORQhVIU/hd4G+hnZs8SmMDu6nCGEqlLenYht765ko/X5jCwWzsevSKVU4Z3RwexIoenzqLg7u+Y2VfAJAKrqf3U3XPDnkykGvsOlHLPe+t47NMM2iQ055bvjuDyyf210plIAwll9NH77n4K8FY194k0ms/Tc7n51WVsytvHhal9+cXUYSRpeKlIg6qxKASXxmwLJJlZZ75eczkR6N0I2UQAyC0s5s631/Bi2mZSurblhZmTmDSwa6RjicSk2o4Ufgj8N4EC8BVfF4UC4P4w5xKhqKSMxz7N4MEP17O/pIyZxw/kxlOH0iaheaSjicSsGouCu98D3GNmP3b3+xoxk8Q5d2fOiu3c+u+VbM0vYsqIHtx8xjAGdWsf6WgiMS+UE833mdlIYATQutL9T4czmMSnzXn7uOWN5cxdk8Ownh2468KxTB6kriKRxhLKiebfAycSKAqzgTOATwEVBWkwpWXlPPppBn9/by3NzPjtmcO56ugUWmhUkUijCuU6hfOBMcAid7/azHoAj4Y3lsST5Vvy+eUrS1mxtYApI3rwh7OP1AymIhESSlHY7+7lZlZqZolANjAwzLkkDhQUlXD3u2t5+ouNdGmXwIOXjmfqyJ66AE0kgkIpCmlm1gl4hMAopEJgflhTSUxzd2Yt2crtb60it7CYSyYk84vTh9GxrWYxFYm0UE40/yh48yEzextIdPel4Y0lsSq3sJhfv7qMd1buYEzfjjx2ZarmKhJpQuq1yI67Z5rZEWb2iLv/IFyhJDa9u3IHv3p1KQX7S/nNtOF8/9gBNG+mriKRpqS2K5pHA38lcPHa68B9wAPAROCuRkknMWHfgVJue3MVz8/fxPBeiTx77ViO6Nkh0rFEpBq1HSk8AjwIfAFMBRYCzwGXuntRI2STGLAsK5+fvrCIjJ17+eEJA7lpyhEktNAwU5Gmqrai0MrdnwzeXmNm/wPc7O5l4Y8l0a60rJyHPlrP399bR1L7Vjx77USOHpQU6VgiUofaikJrMxvH13MeFQKjD6665u4Lwx1OolNm7l5+9tJiFm7azXdH9+L2c0bSqW1CpGOJSAhqKwrbgL9V2t5eaduBk8MVSqLXG4u38KtXl9GimXHPjLFMH9sn0pFEpB5qmxDvpMN9cTObCtwDNAcedfc/V9PmQgKruzmwxN0vOdz3lcZXXFrG7W+u4pkvN3JUSmfuvXgcvTrqqmSRaFOvIan1YWbNCUyxPQXIAhaY2Sx3X1mpzRDgV8Ax7r7LzLqHK4+Ez9bd+7nu2YUs2bybHxw3gF9MHaaV0ESiVNiKAjABSHf3DQBm9gIwHVhZqc0PgPvdfReAu2eHMY+EwbwNO7n+uYUUlZTz0GXjmTqyV6QjichhCOfXuT7A5krbWcH7KhsKDDWzz8zsy2B307eY2UwzSzOztJycnDDFlfpwd576PJNLH51HYpuWvH790SoIIjEglKmzDbgUGOjut5pZMtDT3eua/6i6S1W9mvcfQmBq7r7AJ2Y20t13f+NJ7g8DDwOkpqZWfQ1pZAVFJfzq1WW8tXQbpw7vzt8uGktia81bJBILQuk+egAoJzDa6FZgD/AKcFQdz8sC+lXa7gtsrabNl+5eAmSY2RoCRWJBCLkkApZm7eaG5xaxZfd+fjH1CP7r+EE001QVIjEjlO6jie5+PVAEEOz/D2XQ+QJgiJkNMLMEYAYwq0qb14GTAMwsiUB30oYQs0sjKi93Hv54Pec9+DklZeW8OHMSPzpxsAqCSIwJ5UihJDiSyAHMrBuBI4dauXupmd0AzCEwJPVxd19hZrcCae4+K/jYaWa2EigDfu7uOw9xXyRMtucXcdPLi/ksfSenH9mDP587ms7tdDGaSCwy99q76M3sUuAiYDzwFIGV2H7r7i+HP963paamelpaWiTeOi7NXZ3NjS8tpriknN+fNYKLjuqnRXBEopCZfeXuqXW1C2U9hWfN7CvgFAInj89x91UNkFGasNKycu5+by33z13P8F6J3H/JOAZ2ax/pWCISZqGMProHeNHd72+EPNIEbM8v4sYXF/PFhp1clNqPP0w/ktYtm0c6log0glDOKSwEfmtmQ4HXCBQI9d/EqFlLtvLb15ZRUubcef5oLkjtV/eTRCRmhNJ99BTwlJl1Ac4D/mJmye4+JOzppNHs3neAW95YwawlWxmX3Im/XTiWAUntIh1LRBpZfaa5GAwMA1L45lQVEuUWbtrFDc8uJHtPMTdNGcp1Jw6iheYuEolLoZxT+AtwLrAeeAm4reoVxxKdDk5V8cfZq+jZsTWv/uhoRvftFOlYIhJBoRwpZACT3T033GGk8RSXlvGLfy3ljcVbOXV4d+66YCwd22qqCpF4V2NRMLNh7r4amA8kB+c8qqCV16LX7n0HmPn0V8zPzOPnpx/BdSdoqgoRCajtSOFnwEzgrmoe08prUWpz3j6ufGI+WXn7ufficZw9pnekI4lIE1LbymszgzfPcPeiyo+ZWeuwppKwWLE1nysfX0BJWTnPXDOBiQO7RjqSiDQxoQwx+TzE+6QJ+zw9l4v+8SUJzY1XrpusgiAi1artnEJPAovitDGzcXy9PkIi0LYRskkDeWvpNm58cTEpSW156vsTtHayiNSotnMKpwNXEVgH4W+V7t8D/DqMmaQB/fPLjfzujeV8J7kzj115lEYYiUitajuncPBK5vPc/ZVGzCQNwN2574N0/vbuWk4e1p37LxlPmwTNXyQitaut++gyd/8nkGJmP6v6uLv/rZqnSRNQXu7c+uZKnvw8k3PH9+Ev542mpa5QFpEQ1NZ9dHDiG82XHEVKysr5xb+W8tqiLVxz7AB+M224rkEQkZDV1n30j+B//9B4ceRwFJWUcf2zC3l/dTY/P/0IfnTiIC2IIyL1UmefgpndYWaJZtbSzN43s1wzu6wxwkno9haXcvUTC/hgTTa3nTOS608arIIgIvUWSkfzae5eAHwXyAKGAj8Payqplz1FJVz5+HzmZezk7gvHcvmk/pGOJCJRKpQJ8Q6OYZwGPO/uefoG2nTk7w8UhOVb8rnv4vGcObpXpCOJSBQLpSj828xWA/uBH5lZN6CojudIIygoKuGKx+axclsBD1w6ntOO7BnpSCIS5ersPnL3m4HJQKq7lwB7genhDia1O9hltHJbAQ9e+h0VBBFpEKEsstMSuBw4Ptht9BHwUJhzSS0Ki0u56okFLMvK5/5Lx3PqiB6RjiQiMSKU7qMHCZxXeCC4fXnwvmvDFUpqtu9AKd9/YgGLN+/m/y4ex+k6QhCRBhRKUTjK3cdU2v7AzJaEK5DUbP+BMq55Mo20jXncM2McZ4zSSWURaVihDEktM7NBBzfMbCBQFr5IUp2ikjJmPpPGlxk7uevCMZylxXFEJAxCOVL4OTDXzDYQmD67P3B1WFPJN5SWlfPj5xfxybpc7jx/NN8b1zfSkUQkRtVZFNz9fTMbAhxBoCisdvfisCcTIDC53S9fWca7K3fwh7OP5ILUfpGOJCIxrMbuIzMbYmZvmNly4Elgp7svUUFoPO7O7W+t4pWFWfxsylCuPDol0pFEJMbVdk7hceBN4DxgIXBfoySSCo98soHHP8vg6mNS+PHJgyMdR0TiQG3dRx3c/ZHg7TvNbGFjBJKAt5Zu40+zV3Pm6F787swRmtxORBpFbUcKrc1snJmNN7PxBNdqrrRdJzObamZrzCzdzG6upd35ZuZmllrfHYhFaZl53PjSYlL7d+auC8ZoPQQRaTS1HSls45trM2+vtO3AybW9sJk1B+4HphCYXXWBmc1y95VV2nUAfgLMq1/02JSRu5cfPJ1Gn05tePiKVFq31BKaItJ4altk56TDfO0JQLq7bwAwsxcIzJm0skq724A7gP85zPeLejsLi7nqifmYGU9efRRd2iVEOpKIxJlwLtzbB9hcaTsreF8FMxsH9HP3N2t7ITObaWZpZpaWk5PT8EmbgKKSMq59Oo3t+UU8emUq/bu2q/tJIiINLJxFobqOcK940KwZcDdwU10v5O4Pu3uqu6d269atASM2DeXlzo0vLmbx5t3cM2Ms45M7RzqSiMSpcBaFLKDylVZ9ga2VtjsAI4EPzSwTmATMireTze7ObW+t5D/Lt/ObacOZOlLzGYlI5ISyRrOZ2WVmdktwO9nMJoTw2guAIWY2wMwSgBnArIMPunu+uye5e4q7pwBfAme7e9oh7UmUeuzTDJ74LJPvHzOAa48bGOk4IhLnQjlSeIDAIjsXB7f3EBhVVCt3LwVuAOYAq4CX3H2Fmd1qZmcfYt6Y8ubSrdz+1iqmjerJb88cHuk4IiIhTYg30d3Hm9kiAHffFfzmXyd3nw3MrnLfLTW0PTGU14wVy7fkc9NLS0jt35m/XThW1yKISJMQypFCSfCaAwcIrtFcHtZUMS5v7wF++MxXdGmXwEOXf0fXIohIkxFKUbgXeA3obmZ/BD4F/hTWVDGstKycG55bSE5hMf+4/DsktW8V6UgiIhVCmTr7WTP7CjiFwDDTc9x9VdiTxag731nD5+t38tcLxjC6b6dIxxER+YZQRh8NAjLc/X5gOTDFzPRpdgg+XpvDPz7awCUTkzn/O1ooR0SanlC6j14hsCTnYOBRYADwXFhTxaDcwmJ+9tIShnRvz+/OHBHpOCIi1QqlKJQHh5eeC9zj7jcCusKqHtydn7+8hIKiEu67ZBxtEnRiWUSaplBHH10MXEFg0R2AluGLFHv+OW8Tc9fk8JtpwxnWMzHScUREahRKUbiawMVrf3T3DDMbAPwzvLFix7ode/jjWys5bkgSV0zuH+k4IiK1CmX00UoC6x0c3M4A/hzOULGiqKSMG55bRPtWLbjrwjFaPU1Emrwai4KZLaPSrKZVufvosCSKIbe9uZI1O/bw1Pcn0L1D60jHERGpU21HCt9ttBQx6O3l23l23iZ+ePxAThgae9N9i0hsqm3ltY2NGSSW7Cws5jevLWNkn0RuOu2ISMcREQlZKBevTTKzBWZWaGYHzKzMzAoaI1y0umXWCgqKSvjrBWNIaBHOJStERBpWKJ9Y/0dg2ux1QBvgWuC+cIaKZm8t3cZbS7fx01OGaPipiESdUKbOxt3Tzay5u5cBT5jZ52HOFZVyC4v53RvLGdWnI/91wqBIxxERqbdQisK+4PoJi83sDmAboFXlq3B3fvvacgqLSrnrwjG0aK5uIxGJPqF8cl0ebHcDsJfAusvnhTNUNJq1ZCtvr9jOjVOGMrRHh0jHERE5JLVdp5Ds7psqjUIqAv7QOLGiS3ZBEbe8sYJxyZ2YebzWWRaR6FXbkcLrB2+Y2SuNkCUquTu/fm0ZRSVl/PWCMTTXspoiEsVqKwqVP9309bcG767cwXursrnptKEM6tY+0nFERA5LbUXBa7gtQUUlZdz65kqG9mjP1ccMiHQcEZHDVtvoozHBi9QMaFPpgjUD3N3jfhD+Qx+tJ2vXfp7/wSRaarSRiMSA2qa50Eowtdict48HP1zPWWN6M3lQ10jHERFpEPp6e4j++NYqmjczfj1tWKSjiIg0GBWFQzBvw07eXrGd604YRK+ObSIdR0Skwaj0af8lAAANXUlEQVQo1FN5ufPH2avomdiaa4/ToCwRiS0qCvU0a8lWlmbl8/PTj6BNgk67iEhsUVGoh6KSMu54ezUj+yTyvXF9Ih1HRKTBqSjUw2OfZrA1v4hfTxtOM125LCIxSEUhRLmFxTz44XpOHd6dowclRTqOiEhYhLUomNlUM1tjZulmdnM1j//MzFaa2VIze9/M+oczz+H4+3tr2V9Sxs1nDI90FBGRsAlbUTCz5sD9wBnACOBiMxtRpdkiINXdRwP/Au4IV57DkZ69h+fnb+bSickM7q75jUQkdoXzSGECkO7uG9z9APACML1yA3ef6+77gptfAn3DmOeQ/b/Zq2nbsjk/PWVIpKOIiIRVOItCH2Bzpe2s4H01uQb4T3UPmNlMM0szs7ScnJwGjFi3+Rl5vL86m+tOGkTX9q0a9b1FRBpbOItCdcNzqp1t1cwuA1KBO6t73N0fdvdUd0/t1q1bA0asnbtz55zVdO/QiquP1iyoIhL7wlkUsggs3XlQX2Br1UZmdirwG+Bsdy8OY556+3BtDgsyd/HjU4boQjURiQvhLAoLgCFmNsDMEoAZwKzKDcxsHPAPAgUhO4xZ6q283PnrnDX069KGi1L71f0EEZEYELai4O6lwA3AHGAV8JK7rzCzW83s7GCzO4H2wMtmttjMZtXwco1u9vJtrNhawM+mDCWhhS7nEJH4UNsiO4fN3WcDs6vcd0ul26eG8/0Plbvz4IfrGdStHWeP0XQWIhI/9BW4GvMz8lixtYBrjxtIc01nISJxREWhGo9/lkHnti016Z2IxB0VhSo25+3j3ZU7uGRiMq1basSRiMQXFYUqnvo8k2ZmXD4pJdJRREQanYpCJYXFpby4YDPTRvWiZ8fWkY4jItLoVBQqeW3RFvYUl3LVMSmRjiIiEhEqCkHuzjNfZDKqT0fG9esU6TgiIhGhohA0LyOPtTsKuXxSf8w0DFVE4pOKQtAzX26kY5uWnDWmd6SjiIhEjIoCkF1QxJzl27kwta8mvhORuKaiADw/fzOl5c6lE5vsaqAiIo0i7ouCu/OvhZs5dnASKUntIh1HRCSi4r4oLNy0i815+zWlhYgIKgq8tmgLrVs24/SRPSMdRUQk4uK6KBwoLefNpds4bURP2rcK6yziIiJRIa6Lwkdrc9i9r0RdRyIiQXFdFF5blEXXdgkcOyQp0lFERJqEuC0Ke4pKeG9VNmeN6U3L5nH7YxAR+Ya4/TT8aG0OB0rLOXN0r0hHERFpMuK2KLyzYgdd2yUwPrlzpKOIiDQZcVkUDpSWM3d1NqcO76E1mEVEKonLovDFhp3sKS7ltCN7RDqKiEiTEpdF4Z0V22mb0JxjBmvUkYhIZXFXFMrLnXdX7uDEI7rRuqVmRBURqSzuisKSrN1k7ynmtBGa1kJEpKq4KwqfpecCcMLQbhFOIiLS9MRdUZifuYsjenSgc7uESEcREWly4qoolJU7Czfu4qgBujZBRKQ6cVUUVm0roLC4lKNSukQ6iohIkxRXRWFBZh6AioKISA3irij06dSG3p3aRDqKiEiTFNaiYGZTzWyNmaWb2c3VPN7KzF4MPj7PzFLClcXdmZ+xiwkDdJQgIlKTsBUFM2sO3A+cAYwALjazEVWaXQPscvfBwN3AX8KVJ3PnPnILi9V1JCJSi3AeKUwA0t19g7sfAF4ApldpMx14Knj7X8ApZhaWGeoWZATOJ0zQyCMRkRqFsyj0ATZX2s4K3ldtG3cvBfKBrlVfyMxmmlmamaXl5OQcUphObVsyZUQPBnVrf0jPFxGJB+Fcrb66b/x+CG1w94eBhwFSU1O/9XgoTjuyJ6cdqaktRERqE84jhSygX6XtvsDWmtqYWQugI5AXxkwiIlKLcBaFBcAQMxtgZgnADGBWlTazgCuDt88HPnD3QzoSEBGRwxe27iN3LzWzG4A5QHPgcXdfYWa3AmnuPgt4DHjGzNIJHCHMCFceERGpWzjPKeDus4HZVe67pdLtIuCCcGYQEZHQxdUVzSIiUjsVBRERqaCiICIiFVQURESkgkXbCFAzywE2HuLTk4DcBowTDbTP8UH7HB8OZ5/7u3ud6xBHXVE4HGaW5u6pkc7RmLTP8UH7HB8aY5/VfSQiIhVUFEREpEK8FYWHIx0gArTP8UH7HB/Cvs9xdU5BRERqF29HCiIiUgsVBRERqRCTRcHMpprZGjNLN7Obq3m8lZm9GHx8npmlNH7KhhXCPv/MzFaa2VIze9/M+kciZ0Oqa58rtTvfzNzMon74Yij7bGYXBn/XK8zsucbO2NBC+NtONrO5ZrYo+Pc9LRI5G4qZPW5m2Wa2vIbHzczuDf48lprZ+AYN4O4x9Y/ANN3rgYFAArAEGFGlzY+Ah4K3ZwAvRjp3I+zzSUDb4O3r4mGfg+06AB8DXwKpkc7dCL/nIcAioHNwu3ukczfCPj8MXBe8PQLIjHTuw9zn44HxwPIaHp8G/IfAypWTgHkN+f6xeKQwAUh39w3ufgB4AZhepc104Kng7X8Bp5hZdUuDRos699nd57r7vuDmlwRWwotmofyeAW4D7gCKGjNcmISyzz8A7nf3XQDunt3IGRtaKPvsQGLwdke+vcJjVHH3j6l9BcrpwNMe8CXQycx6NdT7x2JR6ANsrrSdFbyv2jbuXgrkA10bJV14hLLPlV1D4JtGNKtzn81sHNDP3d9szGBhFMrveSgw1Mw+M7MvzWxqo6ULj1D2+X+By8wsi8D6LT9unGgRU9//3+slrIvsREh13/irjrsNpU00CXl/zOwyIBU4IayJwq/WfTazZsDdwFWNFagRhPJ7bkGgC+lEAkeDn5jZSHffHeZs4RLKPl8MPOnud5nZZAKrOY509/Lwx4uIsH5+xeKRQhbQr9J2X759OFnRxsxaEDjkrO1wrakLZZ8xs1OB3wBnu3txI2ULl7r2uQMwEvjQzDIJ9L3OivKTzaH+bb/h7iXungGsIVAkolUo+3wN8BKAu38BtCYwcVysCun/90MVi0VhATDEzAaYWQKBE8mzqrSZBVwZvH0+8IEHz+BEqTr3OdiV8g8CBSHa+5mhjn1293x3T3L3FHdPIXAe5Wx3T4tM3AYRyt/26wQGFWBmSQS6kzY0asqGFco+bwJOATCz4QSKQk6jpmxcs4ArgqOQJgH57r6toV485rqP3L3UzG4A5hAYufC4u68ws1uBNHefBTxG4BAzncARwozIJT58Ie7znUB74OXgOfVN7n52xEIfphD3OaaEuM9zgNPMbCVQBvzc3XdGLvXhCXGfbwIeMbMbCXSjXBXNX/LM7HkC3X9JwfMkvwdaArj7QwTOm0wD0oF9wNUN+v5R/LMTEZEGFovdRyIicohUFEREpIKKgoiIVFBREBGRCioKIiJSQUVBmiQz62pmi4P/tpvZlkrbCQ34PqeaWX7wdVeZ2W8O4TWam9knwdsDzWxGpccmmtndDZxztZn9OYTnjI+BaS6kkakoSJPk7jvdfay7jwUeAu4+uB2cGO3gFMIN8Tc8N/g+RwHXmNmYemYtc/fjgpsDqXTdi7vPc/cbGyBj5ZzjgfPMbGId7ccDKgpSLyoKElXMbLCZLTezh4CFQD8z213p8Rlm9mjwdg8ze9XM0sxsfvDqzxq5e2HwNQeZWRsze8rMlpnZQjM7Pviao8xsQfAb+9LgkUGLShn+DJwUfPwnwW/4rwePJjaaWWLwdczMNphZ0iHk3EdgCuk+wdeaZGZfWGA9gc/MbIiZtQFuAS4NZjnfzNqb2ZPB91hkZmfV/zcgsU5FQaLRCOAxdx8HbKml3b3AHe6eClwIPFrbi5pZNwJTNa8AfgIccPdRwOUEroBPILAWx18rHVlUnXPmZoLf6N393oN3unsZ8CZfT/t8NLDW3XMPIWcXAkcknwbvWgUcG/x53Abc7u77gVuBZ4NZ/kWgSLzt7hOAk4G7zKx1be8l8SfmprmQuLDe3ReE0O5U4Aj7eqmMzmbWJviBWdlJZrYIKAduc/c1ZnYsgalBCE6rsBUYDHwO/NYCK9e96u7pFphUMRQvAr8AniG4uNMh5FwKDAvmPDiHVSfgaTMbVMf7nwacYV+vXtYaSAbWhphf4oCKgkSjvZVul/PNqYQrf/M1YMLBcxC1mOvu51S5r9pFl9z9GTP7AjgTeNfMriRQKELxCfCkmXUFzgZ+dyg5zWwYgSmxX3f3ZcAfgTnu/oCZDQberuH5Bpzj7utDzCtxSN1HEtWCc+bvCvajNwO+V+nh94DrD26Y2dh6vPTHwKXB5w0HegHpZjbQ3dPd/R7gLWB0leftITBtd3VZHXgD+DuwpNIaB/XK6e6rCawm94vgXR35uhvtqlqyzCHQLXbwfcbV9j4Sn1QUJBb8ksC34/cJzDV/0PXAMcETwisJLFUZqvuANma2DHgWuCL4Tf4SM1thZosJ9Ov/s8rzFgHNzWyJmf2Eb3sRuIyvu44ONecDBJaRTQb+AtxpZp9VafMBMCZ4Uvl84A9A2+DJ8xUEViwT+QbNkioiIhV0pCAiIhVUFEREpIKKgoiIVFBREBGRCioKIiJSQUVBREQqqCiIiEiF/w+D3Q/2+5TA/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set ROC: 0.7202497542477039\n"
     ]
    }
   ],
   "source": [
    "#plotting the ROC Curve\n",
    "trainingSummary = LR_model.summary\n",
    "roc = trainingSummary.roc.toPandas()\n",
    "plt.plot(roc['FPR'],roc['TPR'])\n",
    "plt.ylabel('False Positive Rate')\n",
    "plt.xlabel('True Positive Rate')\n",
    "plt.title('ROC Curve')\n",
    "plt.show()\n",
    "print('Training set ROC: ' + str(trainingSummary.areaUnderROC))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking on the testing set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (1) Logistic Regression "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test SET ROC: 0.7193116947055981\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "predictions_LR = LR_model.transform(test)\n",
    "evaluator = BinaryClassificationEvaluator()\n",
    "print(\"Test SET ROC: \" + str(evaluator.evaluate(predictions_LR, {evaluator.metricName: \"areaUnderROC\"})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (2) Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test_SET (Area Under ROC): 0.7323118190985889\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.classification import GBTClassifier\n",
    "gbt = GBTClassifier(maxIter=15)\n",
    "GBT_Model = gbt.fit(train)\n",
    "gbt_predictions = GBT_Model.transform(test)\n",
    "evaluator = BinaryClassificationEvaluator()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test SET ROC: 0.7323118190985882\n"
     ]
    }
   ],
   "source": [
    "print(\"Test SET ROC: \" + str(evaluator.evaluate(gbt_predictions, {evaluator.metricName: \"areaUnderROC\"})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the Gradient Boosting result then apply hyper-parameter tuning using grid search and after that we run cross validation to better improve the performance of GBT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.tuning import ParamGridBuilder, CrossValidator\n",
    "paramGrid = (ParamGridBuilder()\n",
    "             .addGrid(gbt.maxDepth, [2, 4, 6])\n",
    "             .addGrid(gbt.maxBins, [20, 30])\n",
    "             .addGrid(gbt.maxIter, [10, 15])\n",
    "             .build())\n",
    "cv = CrossValidator(estimator=gbt, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=5)\n",
    "# Run cross validations.\n",
    "cvModel = cv.fit(train)\n",
    "gbt_cv_predictions = cvModel.transform(test)\n",
    "evaluator.evaluate(gbt_cv_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}