17_logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Classifier TensorFlow Binary, \n",
    "\n",
    "# What is Linear Classifier?\n",
    "\n",
    "The two most common supervised learning tasks are linear regression and\n",
    "linear classifier. Linear regression predicts a value while the linear\n",
    "classifier predicts a class. This tutorial is focused on Linear\n",
    "Classifier.\n",
    "\n",
    "Classification problems represent roughly 80 percent of the machine\n",
    "learning task. Classification aims at predicting the probability of each\n",
    "class given a set of inputs. The label (i.e., the dependent variable) is\n",
    "a discrete value, called a class.\n",
    "\n",
    "1.  If the label has only two classes, the learning algorithm is a\n",
    "    binary classifier.\n",
    "\n",
    "2.  Multiclass classifier tackles labels with more than two classes.\n",
    "\n",
    "For instance, a typical binary classification problem is to predict the\n",
    "likelihood a customer makes a second purchase. Predict the type of\n",
    "animal displayed on a picture is multiclass classification problem since\n",
    "there are more than two varieties of animal existing.\n",
    "\n",
    "The theoretical part of this tutorial puts primary focus on the binary\n",
    "class. You will learn more about the multiclass output function in a\n",
    "future tutorial.\n",
    "\n",
    "In this tutorial, you will learn\n",
    "\n",
    "## How Binary classifier works?\n",
    "\n",
    "You learned in the previous tutorial that a function is composed of two\n",
    "kind of variables, a dependent variable and a set of features\n",
    "(independent variables). In the linear regression, a dependent variable\n",
    "is a real number without range. The primary objective is to predict its\n",
    "value by minimizing the mean squared error.\n",
    "\n",
    "For a binary task, the label can have had two possible integer values.\n",
    "In most case, it is either `[0,1]` or `[1,2]`. For instance, the\n",
    "objective is to predict whether a customer will buy a product or not.\n",
    "The label is defined as follow:\n",
    "\n",
    "-   `Y = 1` (customer purchased the product)\n",
    "\n",
    "-   `Y = 0` (customer does not purchase the product)\n",
    "\n",
    "The model uses the features `X` to classify each customer in the most\n",
    "likely class he belongs to, namely, potential buyer or not.\n",
    "\n",
    "The probability of success is computed with **logistic regression**. The\n",
    "algorithm will compute a probability based on the feature `X` and\n",
    "predicts a success when this probability is above 50 percent. More\n",
    "formally, the probability is calculated as follow:\n",
    "\n",
    "$$P(Y = 1|x) = \\frac{1}{1 + exp( - (\\theta^{T}x + b))}$$\n",
    "\n",
    "where $\\theta$ is the set of weights, $x$ the features and b the bias.\n",
    "\n",
    "The function can be decomposed into two parts:\n",
    "\n",
    "-   The linear model\n",
    "\n",
    "-   The logistic function\n",
    "\n",
    "**Linear model**\n",
    "\n",
    "You are already familiar with the way the weights are computed. Weights\n",
    "are computed using a dot product: $\\theta^{T}x + b$ that is $\\sum_{i=0}^{n} x_i w_i +b$. `Y` is a linear function of all the\n",
    "features $x_i$. If the model does not have features, the prediction is\n",
    "equal to the bias, $b$.\n",
    "\n",
    "The weights indicate the direction of the correlation between the\n",
    "features $x_i$ and the label $Y$. A positive correlation increases the\n",
    "probability of the positive class while a negative correlation leads the\n",
    "probability closer to 0, (i.e., negative class).\n",
    "\n",
    "The linear model returns only real number, which is inconsistent with\n",
    "the probability measure of range `[0,1]`. The logistic function is\n",
    "required to convert the linear model output to a probability,\n",
    "\n",
    "**Logistic function**\n",
    "\n",
    "The logistic function, or sigmoid function, has an S-shape and the\n",
    "output of this function is always between 0 and 1.\n",
    "\n",
    "$$\\frac{1}{1 + exp( - t)}$$\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/17_logit_v4_files/image011.png)\n",
    "\n",
    "It is easy to substitute the output of the linear regression into the\n",
    "sigmoid function. It results in a new number with a probability between\n",
    "0 and 1.\n",
    "\n",
    "The classifier can transform the probability into a class\n",
    "\n",
    "-   Values between 0 to 0.49 become class 0\n",
    "\n",
    "-   Values between 0.5 to 1 become class 1\n",
    "\n",
    "## How to Measure the performance of Linear Classifier?\n",
    "\n",
    "### Accuracy\n",
    "\n",
    "The overall performance of a classifier is measured with the accuracy\n",
    "metric. Accuracy collects all the correct values divided by the total\n",
    "number of observations. For instance, an accuracy value of 80 percent\n",
    "means the model is correct in 80 percent of the cases.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/17_logit_v4_files/image013.png)\n",
    "\n",
    "You can note a shortcoming with this metric, especially for imbalance\n",
    "class. An imbalance dataset occurs when the number of observations per\n",
    "group is not equal. Let’s say; you try to classify a rare event with a\n",
    "logistic function. Imagine the classifier tries to estimate the death of\n",
    "a patient following a disease. In the data, 5 percent of the patients\n",
    "pass away. You can train a classifier to predict the number of death and\n",
    "use the accuracy metric to evaluate the performances. If the classifier\n",
    "predicts 0 death for the entire dataset, it will be correct in 95\n",
    "percent of the case.\n",
    "\n",
    "### Confusion matrix\n",
    "\n",
    "A better way to assess the performance of a classifier is to look at the\n",
    "confusion matrix.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/17_logit_v4_files/image015.png)\n",
    "\n",
    "The confusion matrix visualizes the accuracy of a classifier by\n",
    "comparing the actual and predicted classes. The binary confusion matrix\n",
    "is composed of squares:\n",
    "\n",
    "-   TP: True Positive: Predicted values correctly predicted as actual\n",
    "    positive\n",
    "\n",
    "-   FP: Predicted values incorrectly predicted an actual positive. i.e.,\n",
    "    Negative values predicted as positive\n",
    "\n",
    "-   FN: False Negative: Positive values predicted as negative\n",
    "\n",
    "-   TN: True Negative: Predicted values correctly predicted as actual\n",
    "    negative\n",
    "\n",
    "From the confusion matrix, it is easy to compare the actual class and\n",
    "predicted class.\n",
    "\n",
    "### Precision and Sensitivity\n",
    "\n",
    "The confusion matrix provides a good insight into the true positive and\n",
    "false positive. In some case, it is preferable to have a more concise\n",
    "metric.\n",
    "\n",
    "**Precision**\n",
    "\n",
    "The precision metric shows the accuracy of the positive class. It\n",
    "measures how likely the prediction of the positive class is correct.\n",
    "\n",
    "$$Precision = \\frac{\\text{TP}}{TP + FP}$$\n",
    "\n",
    "The maximum score is 1 when the classifier perfectly classifies all the\n",
    "positive values. Precision alone is not very helpful because it ignores\n",
    "the negative class. The metric is usually paired with Recall metric.\n",
    "Recall is also called sensitivity or true positive rate.\n",
    "\n",
    "**Sensitivity**\n",
    "\n",
    "Sensitivity computes the ratio of positive classes correctly detected.\n",
    "This metric gives how good the model is to recognize a positive class.\n",
    "\n",
    "$$Recall = \\frac{\\text{TP}}{TP + FN}$$\n",
    "\n",
    "## Linear Classifier with TensorFlow\n",
    "\n",
    "For this tutorial, we will use the census dataset. The purpose is to use\n",
    "the variables in the census dataset to predict the income level. Note\n",
    "that the income is a binary variable\n",
    "\n",
    "-   with a value of 1 if the income &gt; 50k\n",
    "\n",
    "-   0 if income &lt; 50k.\n",
    "\n",
    "This variable is your label\n",
    "\n",
    "This dataset includes eight categorical variables:\n",
    "\n",
    "-   workplace\n",
    "\n",
    "-   education\n",
    "\n",
    "-   marital\n",
    "\n",
    "-   occupation\n",
    "\n",
    "-   relationship\n",
    "\n",
    "-   race\n",
    "\n",
    "-   sex\n",
    "\n",
    "-   native\\_country\n",
    "\n",
    "moreover, six continuous variables:\n",
    "\n",
    "-   age\n",
    "\n",
    "-   fnlwgt\n",
    "\n",
    "-   education\\_num\n",
    "\n",
    "-   capital\\_gain\n",
    "\n",
    "-   capital\\_loss\n",
    "\n",
    "-   hours\\_week\n",
    "\n",
    "Through this example, you will understand how to train a linear\n",
    "classifier with TensorFlow estimator and how to improve the accuracy\n",
    "metric.\n",
    "\n",
    "We will proceed as follow:\n",
    "\n",
    "Step 1: Import the data\n",
    "\n",
    "Step 2: Data Conversion\n",
    "\n",
    "Step 3: Train the classifier\n",
    "\n",
    "Step 4: Improve the model\n",
    "\n",
    "Step 5: Hyperparameter:Lasso & Ridge\n",
    "\n",
    "**Step 1**: Import the data\n",
    "\n",
    "You first import the libraries used during the tutorial.\n",
    "\n",
    "import tensorflow as tf\\\n",
    "import pandas as pd\n",
    "\n",
    "Next, you import the data from the archive of UCI and defines the\n",
    "columns names. You will use the `COLUMNS` to name the columns in a\n",
    "pandas data frame.\n",
    "\n",
    "Note that you will train the classifier using a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "## Define path data\n",
    "COLUMNS = ['age','workclass', 'fnlwgt', 'education', 'education_num', 'marital',\n",
    "           'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss',\n",
    "           'hours_week', 'native_country', 'label']\n",
    "PATH = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\"\n",
    "PATH_test = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data stored online are already divided between a train set and test\n",
    "set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = pd.read_csv(PATH, skipinitialspace=True, names = COLUMNS, index_col=False)\n",
    "df_test = pd.read_csv(PATH_test,skiprows = 1, skipinitialspace=True, names = COLUMNS, index_col=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train set contains 32,561 observations and the test set 16,281"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32561, 15) (16281, 15)\n",
      "age                int64\n",
      "workclass         object\n",
      "fnlwgt             int64\n",
      "education         object\n",
      "education_num      int64\n",
      "marital           object\n",
      "occupation        object\n",
      "relationship      object\n",
      "race              object\n",
      "sex               object\n",
      "capital_gain       int64\n",
      "capital_loss       int64\n",
      "hours_week         int64\n",
      "native_country    object\n",
      "label             object\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(df_train.shape, df_test.shape)\n",
    "print(df_train.dtypes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensorflow requires a Boolean value to train the classifier. You need to\n",
    "cast the values from string to integer. The label is store as an object,\n",
    "however, you need to convert it into a numeric value. The code below\n",
    "creates a dictionary with the values to convert and loop over the column\n",
    "item. Note that you perform this operation twice, one for the train\n",
    "test, one for the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "label = {'<=50K': 0,'>50K': 1}\n",
    "df_train.label = [label[item] for item in df_train.label]\n",
    "label_t = {'<=50K.': 0,'>50K.': 1}\n",
    "df_test.label = [label_t[item] for item in df_test.label]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the train data, there are 24,720 incomes lower than 50k and 7841 above. The ratio is almost the same for the test set. Please refer this tutorial on Facets for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    24720\n",
      "1     7841\n",
      "Name: label, dtype: int64\n",
      "0    12435\n",
      "1     3846\n",
      "Name: label, dtype: int64\n",
      "age                int64\n",
      "workclass         object\n",
      "fnlwgt             int64\n",
      "education         object\n",
      "education_num      int64\n",
      "marital           object\n",
      "occupation        object\n",
      "relationship      object\n",
      "race              object\n",
      "sex               object\n",
      "capital_gain       int64\n",
      "capital_loss       int64\n",
      "hours_week         int64\n",
      "native_country    object\n",
      "label              int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(df_train[\"label\"].value_counts())\n",
    "### The model will be correct in atleast 70% of the case\n",
    "print(df_test[\"label\"].value_counts())\n",
    "## Unbalanced label\n",
    "print(df_train.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2**: Data Conversion \n",
    "\n",
    "A few steps are required before you train a linear classifier with\n",
    "Tensorflow. You need to prepare the features to include in the model. In\n",
    "the benchmark regression, you will use the original data without\n",
    "applying any transformation.\n",
    "\n",
    "The estimator needs to have a list of\n",
    "features to train the model. Hence, the column's data requires to be\n",
    "converted into a tensor.\n",
    "\n",
    "A good practice is to define two lists of features based on their type\n",
    "and then pass them in the `feature_columns` of the estimator.\n",
    "\n",
    "You will begin by converting continuous features, then define a bucket\n",
    "with the categorical data.\n",
    "\n",
    "The features of the dataset have two formats:\n",
    "\n",
    "-   Integer\n",
    "\n",
    "-   Object\n",
    "\n",
    "Each feature is listed in the next two variables as per their types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Add features to the bucket: \n",
    "### Define continuous list\n",
    "CONTI_FEATURES  = ['age', 'fnlwgt','capital_gain', 'education_num', 'capital_loss', 'hours_week']\n",
    "### Define the categorical list\n",
    "CATE_FEATURES = ['workclass', 'education', 'marital', 'occupation', 'relationship', 'race', 'sex', 'native_country']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `feature_column` is equipped with an object `numeric_column` to help\n",
    "in the transformation of the continuous variables into tensor. In the\n",
    "code below, you convert all the variables from CONTI\\_FEATURES into a\n",
    "tensor with a numeric value. This is compulsory to construct the model.\n",
    "All the independent variables need to be converted into the proper type\n",
    "of tensor.\n",
    "\n",
    "Below we write a code to let you see what is happening behind\n",
    "`feature_column.numeric_column`. We will print the converted value for\n",
    "age It is for explanatory purpose, hence there is no need to understand\n",
    "the python code. You can refer to the official documentation to\n",
    "understand the codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_transformation(feature = \"age\", continuous = True, size = 2): \n",
    "    #X = fc.numeric_column(feature)\n",
    "    ## Create feature name\n",
    "    feature_names = [\n",
    "    feature]\n",
    "\n",
    "    ## Create dict with the data\n",
    "    d = dict(zip(feature_names, [df_train[feature]]))\n",
    "\n",
    "    ## Convert age\n",
    "    if continuous == True:\n",
    "        c = tf.feature_column.numeric_column(feature)\n",
    "        feature_columns = [c]\n",
    "    else: \n",
    "        c = tf.feature_column.categorical_column_with_hash_bucket(feature, hash_bucket_size=size) \n",
    "        c_indicator = tf.feature_column.indicator_column(c)\n",
    "        feature_columns = [c_indicator]\n",
    "    \n",
    "## Use input_layer to print the value\n",
    "    input_layer = tf.feature_column.input_layer(\n",
    "        features=d,\n",
    "        feature_columns=feature_columns\n",
    "        )\n",
    "    ## Create lookup table\n",
    "    zero = tf.constant(0, dtype=tf.float32)\n",
    "    where = tf.not_equal(input_layer, zero)\n",
    "    ## Return lookup tble\n",
    "    indices = tf.where(where)\n",
    "    values = tf.gather_nd(input_layer, indices)\n",
    "    ## Initiate graph\n",
    "    sess = tf.Session()\n",
    "    ## Print value\n",
    "    print(sess.run(input_layer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[39.]\n",
      " [50.]\n",
      " [38.]\n",
      " ...\n",
      " [58.]\n",
      " [22.]\n",
      " [52.]]\n"
     ]
    }
   ],
   "source": [
    "print_transformation(feature = \"age\", continuous = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values are exactly the same as in df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "continuous_features = [tf.feature_column.numeric_column(k) for k in CONTI_FEATURES]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to TensorFlow documentation, there are different ways to\n",
    "convert categorical data. If the vocabulary list of a feature is known\n",
    "and does not have plenty of values, it is possible to create the\n",
    "categorical column with `categorical_column_with_vocabulary_list`. It\n",
    "will assign to all unique vocabulary list an ID.\n",
    "\n",
    "For instance, if a variable `statu`s has three distinct values:\n",
    "\n",
    "-   Husband\n",
    "\n",
    "-   Wife\n",
    "\n",
    "-   Single\n",
    "\n",
    "Then three ID will be attributed. For instance, Husband will have the ID\n",
    "1, Wife the ID 2 and so on.\n",
    "\n",
    "For illustration purpose, you can use this code to convert an object\n",
    "variable to a categorical column in TensorFlow.\n",
    "\n",
    "The feature sex can only have two value: male or female. When we will\n",
    "convert the feature sex, Tensorflow will create 2 new columns, one for\n",
    "male and one for female. If the sex is equal to male, then the new\n",
    "column male will be equal to 1 and female to 0. This example is\n",
    "displayed in the table below:\n",
    "\n",
    "| rows | sex    | after transformation | male | female |\n",
    "|------|--------|----------------------|------|--------|\n",
    "| 1    | male   | =>                   | 1    | 0      |\n",
    "| 2    | male   | =>                   | 1    | 0      |\n",
    "| 3    | female | =>                   | 0    | 1      |\n",
    "\n",
    "In tensorflow:\n",
    "\n",
    "print\\_transformation(feature = \"sex\", continuous = False, size = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "relationship = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'relationship', [\n",
    "        'Husband', 'Not-in-family', 'Wife', 'Own-child', 'Unmarried',\n",
    "        'Other-relative'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we added Python code to print the encoding. Again, you don’t need to understand the code, the purpose is to see the transformation\n",
    "\n",
    "However, a faster way to transform the data is to use the method\n",
    "`categorical_column_with_hash_bucket`. Altering string variables in a sparse matrix will be useful. A sparse matrix is a matrix with mostly\n",
    "zero. The method takes care of everything. You only need to specify the\n",
    "number of buckets and the key column. The number of buckets is the\n",
    "maximum amount of groups that Tensorflow can create. The key column is\n",
    "simply the name of the column to convert.\n",
    "\n",
    "In the code below, you create a loop over all the categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = [tf.feature_column.categorical_column_with_hash_bucket(k, hash_bucket_size=1000) for k in CATE_FEATURES]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3**: Train the Classifier\n",
    "\n",
    "TensorFlow currently provides an estimator for the linear regression and\n",
    "linear classification.\n",
    "\n",
    "-   Linear regression: `LinearRegressor`\n",
    "\n",
    "-   Linear classification: `LinearClassifier`\n",
    "\n",
    "The syntax of the linear classifier is the same as in the tutorial on\n",
    "linear regression except for one argument, n\\_class. You need to define\n",
    "the feature column, the model directory and, compare with the linear\n",
    "regressor; you have the define the number of class. For a logit\n",
    "regression, it the number of class is equal to 2.\n",
    "\n",
    "The model will compute the weights of the columns contained in\n",
    "`continuous_features` and `categorical_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'ongoing/train', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xb1b1488d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "model = tf.estimator.LinearClassifier(\n",
    "    n_classes = 2,\n",
    "    model_dir=\"ongoing/train\", \n",
    "    feature_columns=categorical_features+ continuous_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the classifier is defined, you can create the input function.\n",
    "The method is the same as in the linear regressor tutorial. Here, you\n",
    "use a batch size of 128 and you shuffle the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURES = ['age','workclass', 'fnlwgt', 'education', 'education_num', 'marital', 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_week', 'native_country']\n",
    "LABEL= 'label'\n",
    "def get_input_fn(data_set, num_epochs=None, n_batch = 128, shuffle=True):\n",
    "    return tf.estimator.inputs.pandas_input_fn(\n",
    "       x=pd.DataFrame({k: data_set[k].values for k in FEATURES}),\n",
    "       y = pd.Series(data_set[LABEL].values),\n",
    "       batch_size=n_batch,   \n",
    "       num_epochs=num_epochs,\n",
    "       shuffle=shuffle)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You create a function with the arguments required by the linear estimator, i.e., number of epochs, number of batches and shuffle the dataset or note. Since you use the Pandas method to pass the data into the model, you need to define the X variables as a pandas data frame. Note that you loop over all the data stored in FEATURES. \n",
    "\n",
    "Let’s train the model with the object model.train. You use the function\n",
    "previously defined to feed the model with the appropriate values. Note\n",
    "that you set the batch size to 128 and the number of epochs to None. The\n",
    "model will be trained over a thousand steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ongoing/train/model.ckpt.\n",
      "INFO:tensorflow:loss = 88.722855, step = 1\n",
      "INFO:tensorflow:global_step/sec: 67.5773\n",
      "INFO:tensorflow:loss = 52583.64, step = 101 (1.485 sec)\n",
      "INFO:tensorflow:global_step/sec: 128.235\n",
      "INFO:tensorflow:loss = 25203.816, step = 201 (0.776 sec)\n",
      "INFO:tensorflow:global_step/sec: 131.015\n",
      "INFO:tensorflow:loss = 54924.312, step = 301 (0.764 sec)\n",
      "INFO:tensorflow:global_step/sec: 127.105\n",
      "INFO:tensorflow:loss = 68509.31, step = 401 (0.788 sec)\n",
      "INFO:tensorflow:global_step/sec: 142.797\n",
      "INFO:tensorflow:loss = 9151.754, step = 501 (0.702 sec)\n",
      "INFO:tensorflow:global_step/sec: 138.293\n",
      "INFO:tensorflow:loss = 34576.06, step = 601 (0.719 sec)\n",
      "INFO:tensorflow:global_step/sec: 129.156\n",
      "INFO:tensorflow:loss = 36047.117, step = 701 (0.775 sec)\n",
      "INFO:tensorflow:global_step/sec: 123.81\n",
      "INFO:tensorflow:loss = 22608.148, step = 801 (0.807 sec)\n",
      "INFO:tensorflow:global_step/sec: 118.505\n",
      "INFO:tensorflow:loss = 22201.918, step = 901 (0.844 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into ongoing/train/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 5444.363.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0xb1b115c50>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.train(input_fn=get_input_fn(df_train, \n",
    "                                      num_epochs=None,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the loss decreased subsequently during the last 100 steps, i.e., from 901 to 1000.\n",
    "\n",
    "The final loss after one thousand iterations is 5444. You can estimate\n",
    "your model on the test set and see the performance. To evaluate the\n",
    "performance of your model, you need to use the object evaluate. You feed\n",
    "the model with the test set and set the number of epochs to 1, i.e., the\n",
    "data will go to the model only one time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-11:25:31\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ongoing/train/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [100/1000]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-11:25:33\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7615626, accuracy_baseline = 0.76377374, auc = 0.63300294, auc_precision_recall = 0.50891197, average_loss = 47.12155, global_step = 1000, label/mean = 0.23622628, loss = 5993.6406, precision = 0.49401596, prediction/mean = 0.18454961, recall = 0.38637546\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: ongoing/train/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.7615626,\n",
       " 'accuracy_baseline': 0.76377374,\n",
       " 'auc': 0.63300294,\n",
       " 'auc_precision_recall': 0.50891197,\n",
       " 'average_loss': 47.12155,\n",
       " 'label/mean': 0.23622628,\n",
       " 'loss': 5993.6406,\n",
       " 'precision': 0.49401596,\n",
       " 'prediction/mean': 0.18454961,\n",
       " 'recall': 0.38637546,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(input_fn=get_input_fn(df_test, \n",
    "                                      num_epochs=1,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow returns all the metrics you learnt in the theoretical part.\n",
    "Without surprise, the accuracy is large due to the unbalanced label.\n",
    "Actually, the model performs slightly better than a random guess.\n",
    "Imagine the model predict all household with income lower than 50K, then\n",
    "the model has an accuracy of 70 percent. On a closer analysis, you can\n",
    "see the prediction and recall are quite low.\n",
    "\n",
    "**Step 4**: Improve the model\n",
    "\n",
    "Now that you have a benchmark model, you can try to improve it, that is,\n",
    "increase the accuracy. In the previous tutorial, you learned how to\n",
    "improve the prediction power with an interaction term. In this tutorial,\n",
    "you will revisit this idea by adding a polynomial term to the\n",
    "regression.\n",
    "\n",
    "Polynomial regression is instrumental when there is non-linearity in the\n",
    "data. There are two ways to capture non-linearity in the data.\n",
    "\n",
    "-   Add polynomial term\n",
    "\n",
    "-   Bucketize the continuous variable into a categorical variable\n",
    "\n",
    "**Polynomial term**\n",
    "\n",
    "From the picture below, you can see what a polynomial regression is. It\n",
    "is an equation with X variables with different power. A second-degree\n",
    "polynomial regression has two variables, X and X squared. Third degree\n",
    "has three variables, X, $X^{2},\\ and\\ X^{3}$\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/17_logit_v4_files/image020.png)\n",
    "\n",
    "Below, we constructed a graph with two variables, `X` and `Y`. It is\n",
    "obvious the relationship is not linear. If we add a linear regression,\n",
    "we can see the model is unable to capture the pattern (left picture).\n",
    "\n",
    "Now, look at the left picture from the picture below, we added five-term\n",
    "to the regression (that is $y = x + x^{2} + x^{3} + x^{4} + x^{5}$. The\n",
    "model now captures way better the pattern. This is the power of\n",
    "polynomial regression.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/17_logit_v4_files/image021.png)\n",
    "\n",
    "Let's go back to our example. Age is *not* in a linear relationship with\n",
    "income. Early age might have a flat income close to zero because\n",
    "children or young people do not work. Then it increases in working age\n",
    "and decreases during retirement. It is typically an Inversed-U shape.\n",
    "One way to capture this pattern is by adding a power two to the\n",
    "regression.\n",
    "\n",
    "Let’s see if it increases the accuracy.\n",
    "\n",
    "You need to add this new feature to the dataset and in the list of\n",
    "continuous feature.\n",
    "\n",
    "You add the new variable in the train and test dataset, so it is more\n",
    "convenient to write a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def square_var(df_t, df_te, var_name = 'age'):\n",
    "    df_t['new'] = df_t[var_name].pow(2) \n",
    "    df_te['new'] = df_te[var_name].pow(2) \n",
    "    return df_t, df_te"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function has 3 arguments:\n",
    "\n",
    "df\\_t: define the training set\n",
    "\n",
    "df\\_te: define the test set\n",
    "\n",
    "var\\_name = 'age': Define the variable to transform\n",
    "\n",
    "You can use the object pow(2) to square the variable age. Note that the\n",
    "new variable is named 'new'\n",
    "\n",
    "Now that the function `square_var` is written, you can create the new\n",
    "datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train_new, df_test_new = square_var(df_train, df_test, var_name = 'age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the new dataset has one more feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32561, 16) (16281, 16)\n"
     ]
    }
   ],
   "source": [
    "print(df_train_new.shape, df_test_new.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The square variable is called `new` in the dataset. You need to add it\n",
    "to the list of continuous features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTI_FEATURES_NEW  = ['age', 'fnlwgt','capital_gain', 'education_num', 'capital_loss', 'hours_week', 'new']\n",
    "continuous_features_new = [tf.feature_column.numeric_column(k) for k in CONTI_FEATURES_NEW]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** that you changed the directory of the Graph. You can’t train\n",
    "different models in the same directory. It means, you need to change the\n",
    "path of the argument model\\_dir. If you don’t TensorFlow will throw an\n",
    "error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'ongoing/train1', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xb1b148c88>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "model_1 = tf.estimator.LinearClassifier(\n",
    "    model_dir=\"ongoing/train1\", \n",
    "    feature_columns=categorical_features+ continuous_features_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURES_NEW = ['age','workclass', 'fnlwgt', 'education', 'education_num', 'marital', 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_week', 'native_country', 'new']\n",
    "def get_input_fn(data_set, num_epochs=None, n_batch = 128, shuffle=True):\n",
    "    return tf.estimator.inputs.pandas_input_fn(\n",
    "       x=pd.DataFrame({k: data_set[k].values for k in FEATURES_NEW}),\n",
    "       y = pd.Series(data_set[LABEL].values),\n",
    "       batch_size=n_batch,   \n",
    "       num_epochs=num_epochs,\n",
    "       shuffle=shuffle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the classifier is designed with the new dataset, you can train\n",
    "and evaluate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ongoing/train1/model.ckpt.\n",
      "INFO:tensorflow:loss = 88.722855, step = 1\n",
      "INFO:tensorflow:global_step/sec: 71.305\n",
      "INFO:tensorflow:loss = 70077.66, step = 101 (1.404 sec)\n",
      "INFO:tensorflow:global_step/sec: 120.086\n",
      "INFO:tensorflow:loss = 49522.082, step = 201 (0.833 sec)\n",
      "INFO:tensorflow:global_step/sec: 148.581\n",
      "INFO:tensorflow:loss = 107120.57, step = 301 (0.672 sec)\n",
      "INFO:tensorflow:global_step/sec: 154.018\n",
      "INFO:tensorflow:loss = 12814.152, step = 401 (0.649 sec)\n",
      "INFO:tensorflow:global_step/sec: 135.308\n",
      "INFO:tensorflow:loss = 19573.898, step = 501 (0.743 sec)\n",
      "INFO:tensorflow:global_step/sec: 143.553\n",
      "INFO:tensorflow:loss = 26381.986, step = 601 (0.692 sec)\n",
      "INFO:tensorflow:global_step/sec: 56.3572\n",
      "INFO:tensorflow:loss = 23417.719, step = 701 (1.775 sec)\n",
      "INFO:tensorflow:global_step/sec: 136.765\n",
      "INFO:tensorflow:loss = 23946.049, step = 801 (0.731 sec)\n",
      "INFO:tensorflow:global_step/sec: 141.925\n",
      "INFO:tensorflow:loss = 3309.5786, step = 901 (0.707 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into ongoing/train1/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 28861.898.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0xb1b148cc0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_1.train(input_fn=get_input_fn(df_train, \n",
    "                                      num_epochs=None,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-11:25:52\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ongoing/train1/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [100/1000]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-11:25:55\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7944229, accuracy_baseline = 0.76377374, auc = 0.6093755, auc_precision_recall = 0.54885805, average_loss = 111.0046, global_step = 1000, label/mean = 0.23622628, loss = 14119.265, precision = 0.6682401, prediction/mean = 0.09116262, recall = 0.2576703\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: ongoing/train1/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.7944229,\n",
       " 'accuracy_baseline': 0.76377374,\n",
       " 'auc': 0.6093755,\n",
       " 'auc_precision_recall': 0.54885805,\n",
       " 'average_loss': 111.0046,\n",
       " 'label/mean': 0.23622628,\n",
       " 'loss': 14119.265,\n",
       " 'precision': 0.6682401,\n",
       " 'prediction/mean': 0.09116262,\n",
       " 'recall': 0.2576703,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_1.evaluate(input_fn=get_input_fn(df_test_new, \n",
    "                                      num_epochs=1,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The squared variable improved the accuracy from 0.76 to 0.79. Let’s see\n",
    "if you can do better by combining bucketization and interaction term\n",
    "together.\n",
    "\n",
    "## Bucketization and interaction\n",
    "\n",
    "As you saw before, a linear classifier is unable to capture the\n",
    "age-income pattern correctly. That is because it learns a single weight\n",
    "for each feature. To make it easier for the classifier, one thing you\n",
    "can do is bucket the feature. Bucketing transforms a numeric feature\n",
    "into several certain ones based on the range it falls into, and each of\n",
    "these new features indicates whether a person's age falls within that\n",
    "range.\n",
    "\n",
    "With these new features, the linear model can capture the relationship\n",
    "by learning different weights for each bucket.\n",
    "\n",
    "In TensorFlow, it is done with `bucketized_column`. You need to add the\n",
    "range of values in the boundaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "age = tf.feature_column.numeric_column('age')\n",
    "age_buckets = tf.feature_column.bucketized_column(\n",
    "    age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You already know age is non-linear with income. Another way to improve\n",
    "the model is through interaction. In the word of TensorFlow, it is\n",
    "feature crossing. Feature crossing is a way to create new features that\n",
    "are combinations of existing ones, which can be helpful for a linear\n",
    "classifier that can’t model interactions between features.\n",
    "\n",
    "You can break down `age` with another feature like `education`. That is\n",
    "is, some groups are likely to have a high income and others low (Think\n",
    "about the Ph.D. student).\n",
    "\n",
    "."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "education_x_occupation = [tf.feature_column.crossed_column(\n",
    "    ['education', 'occupation'], hash_bucket_size=1000)]\n",
    "age_buckets_x_education_x_occupation = [tf.feature_column.crossed_column(\n",
    "    [age_buckets, 'education', 'occupation'], hash_bucket_size=1000)]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a cross feature column, you use crossed_column with the variables to cross in a bracket. The hash_bucket_size indicates the maximum crossing possibilities. To create interaction between variables (at least one variable needs to be categorical), you can use tf.feature_column.crossed_column. To use this object, you need to add in square bracket the variable to interact and a second argument, the bucket size. The bucket size is the maximum number of group possible within a variable. Here you set it at 1000 as you do not know the exact number of groups\n",
    "\n",
    "`age_buckets` needs to be squared before to add it to the feature\n",
    "columns. You also add the new features to the features columns and\n",
    "prepare the estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'ongoing/train3', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xb19b05eb8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "base_columns = [\n",
    "    age_buckets,\n",
    "]\n",
    "\n",
    "model_imp = tf.estimator.LinearClassifier(\n",
    "    model_dir=\"ongoing/train3\", \n",
    "    feature_columns=categorical_features+base_columns+education_x_occupation+age_buckets_x_education_x_occupation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURES_imp = ['age','workclass', 'education', 'education_num', 'marital',\n",
    "                'occupation', 'relationship', 'race', 'sex', 'native_country', 'new']\n",
    "\n",
    "def get_input_fn(data_set, num_epochs=None, n_batch = 128, shuffle=True):\n",
    "    return tf.estimator.inputs.pandas_input_fn(\n",
    "       x=pd.DataFrame({k: data_set[k].values for k in FEATURES_imp}),\n",
    "       y = pd.Series(data_set[LABEL].values),\n",
    "       batch_size=n_batch,   \n",
    "       num_epochs=num_epochs,\n",
    "       shuffle=shuffle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are ready to estimate the new model and see if it improves the\n",
    "accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ongoing/train3/model.ckpt.\n",
      "INFO:tensorflow:loss = 88.722855, step = 1\n",
      "INFO:tensorflow:global_step/sec: 66.9377\n",
      "INFO:tensorflow:loss = 50.334488, step = 101 (1.497 sec)\n",
      "INFO:tensorflow:global_step/sec: 150.613\n",
      "INFO:tensorflow:loss = 56.153225, step = 201 (0.662 sec)\n",
      "INFO:tensorflow:global_step/sec: 179.291\n",
      "INFO:tensorflow:loss = 45.792007, step = 301 (0.563 sec)\n",
      "INFO:tensorflow:global_step/sec: 174.064\n",
      "INFO:tensorflow:loss = 37.485672, step = 401 (0.572 sec)\n",
      "INFO:tensorflow:global_step/sec: 151.93\n",
      "INFO:tensorflow:loss = 56.48449, step = 501 (0.657 sec)\n",
      "INFO:tensorflow:global_step/sec: 160.733\n",
      "INFO:tensorflow:loss = 32.528934, step = 601 (0.624 sec)\n",
      "INFO:tensorflow:global_step/sec: 154.025\n",
      "INFO:tensorflow:loss = 37.438057, step = 701 (0.651 sec)\n",
      "INFO:tensorflow:global_step/sec: 138.016\n",
      "INFO:tensorflow:loss = 61.1075, step = 801 (0.721 sec)\n",
      "INFO:tensorflow:global_step/sec: 131.868\n",
      "INFO:tensorflow:loss = 44.69645, step = 901 (0.762 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into ongoing/train3/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 44.18133.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0xb19b05f98>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_imp.train(input_fn=get_input_fn(df_train_new, \n",
    "                                      num_epochs=None,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-11:26:13\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ongoing/train3/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [100/1000]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-11:26:15\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8358209, accuracy_baseline = 0.76377374, auc = 0.88401634, auc_precision_recall = 0.69599575, average_loss = 0.35122654, global_step = 1000, label/mean = 0.23622628, loss = 44.67437, precision = 0.68986726, prediction/mean = 0.23320661, recall = 0.55408216\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: ongoing/train3/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.8358209,\n",
       " 'accuracy_baseline': 0.76377374,\n",
       " 'auc': 0.88401634,\n",
       " 'auc_precision_recall': 0.69599575,\n",
       " 'average_loss': 0.35122654,\n",
       " 'label/mean': 0.23622628,\n",
       " 'loss': 44.67437,\n",
       " 'precision': 0.68986726,\n",
       " 'prediction/mean': 0.23320661,\n",
       " 'recall': 0.55408216,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_imp.evaluate(input_fn=get_input_fn(df_test_new, \n",
    "                                      num_epochs=1,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new accuracy level is 83.58 percent. It is four percent higher than\n",
    "the previous model.\n",
    "\n",
    "Finally, you can add a regularization term to prevent overfitting.\n",
    "\n",
    "**Step 5**: Hyperparameter:Lasso & Ridge\n",
    "\n",
    "Your model can suffer from **overfitting** or **underfitting**.\n",
    "\n",
    "-   Overfitting: The model is unable to generalize the prediction to new\n",
    "    data\n",
    "\n",
    "-   Underfitting: The model is unable to capture the pattern of the\n",
    "    data. i.e., linear regression when the data is non-linear\n",
    "\n",
    "When a model has lots of parameters and a relatively low amount of data,\n",
    "it leads to poor predictions. Imagine, one group only have three\n",
    "observations; the model will compute a weight for this group. The weight\n",
    "is used to make a prediction; if the observations of the test set for\n",
    "this particular group is entirely different from the training set, then\n",
    "the model will make a wrong prediction. During the evaluation with the\n",
    "training set, the accuracy is good, but not good with the test set\n",
    "because the weights computed is not the true one to generalize the\n",
    "pattern. In this case, it does not make a reasonable prediction on\n",
    "unseen data.\n",
    "\n",
    "To prevent overfitting, regularization gives you the possibilities to\n",
    "control for such complexity and make it more generalizable. There are\n",
    "two regularization techniques:\n",
    "\n",
    "-   L1: Lasso\n",
    "\n",
    "-   L2: Ridge\n",
    "\n",
    "In TensorFlow, you can add these two hyperparameters in the `optimizer`.\n",
    "For instance, the higher the hyperparameter `L2`, the weight tends to be\n",
    "very low and close to zero. The fitted line will be very flat, while an\n",
    "`L2` close to zero implies the weights are close to the regular linear\n",
    "regression.\n",
    "\n",
    "You can try by yourself the different value of the hyperparameters and\n",
    "see if you can increase the accuracy level.\n",
    "\n",
    "**Note** that if you change the hyperparameter, you need to delete the\n",
    "folder `ongoing/train4` otherwise the model will start with the\n",
    "previously trained model.\n",
    "\n",
    "Let's see how is the accuracy with the hype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'ongoing/train4', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xb1995c668>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ongoing/train4/model.ckpt.\n",
      "INFO:tensorflow:loss = 88.722855, step = 1\n",
      "INFO:tensorflow:global_step/sec: 60.1856\n",
      "INFO:tensorflow:loss = 50.38778, step = 101 (1.664 sec)\n",
      "INFO:tensorflow:global_step/sec: 133.773\n",
      "INFO:tensorflow:loss = 55.38014, step = 201 (0.749 sec)\n",
      "INFO:tensorflow:global_step/sec: 144.301\n",
      "INFO:tensorflow:loss = 46.806694, step = 301 (0.696 sec)\n",
      "INFO:tensorflow:global_step/sec: 130.997\n",
      "INFO:tensorflow:loss = 38.68271, step = 401 (0.762 sec)\n",
      "INFO:tensorflow:global_step/sec: 170.429\n",
      "INFO:tensorflow:loss = 56.99398, step = 501 (0.584 sec)\n",
      "INFO:tensorflow:global_step/sec: 124.577\n",
      "INFO:tensorflow:loss = 33.263622, step = 601 (0.804 sec)\n",
      "INFO:tensorflow:global_step/sec: 142.742\n",
      "INFO:tensorflow:loss = 37.7902, step = 701 (0.703 sec)\n",
      "INFO:tensorflow:global_step/sec: 163.573\n",
      "INFO:tensorflow:loss = 61.732605, step = 801 (0.608 sec)\n",
      "INFO:tensorflow:global_step/sec: 174.756\n",
      "INFO:tensorflow:loss = 46.938225, step = 901 (0.574 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into ongoing/train4/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 43.4942.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0xb1995c5f8>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_regu = tf.estimator.LinearClassifier(\n",
    "    model_dir=\"ongoing/train4\", feature_columns=categorical_features+base_columns+education_x_occupation+age_buckets_x_education_x_occupation,\n",
    "    optimizer=tf.train.FtrlOptimizer(\n",
    "        learning_rate=0.1,\n",
    "        l1_regularization_strength=0.9,\n",
    "        l2_regularization_strength=5))\n",
    "\n",
    "model_regu.train(input_fn=get_input_fn(df_train_new, \n",
    "                                      num_epochs=None,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-11:26:35\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ongoing/train4/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [100/1000]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-11:26:37\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.83833915, accuracy_baseline = 0.76377374, auc = 0.8869794, auc_precision_recall = 0.7014905, average_loss = 0.34691378, global_step = 1000, label/mean = 0.23622628, loss = 44.12581, precision = 0.69720596, prediction/mean = 0.23662092, recall = 0.5579823\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: ongoing/train4/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.83833915,\n",
       " 'accuracy_baseline': 0.76377374,\n",
       " 'auc': 0.8869794,\n",
       " 'auc_precision_recall': 0.7014905,\n",
       " 'average_loss': 0.34691378,\n",
       " 'label/mean': 0.23622628,\n",
       " 'loss': 44.12581,\n",
       " 'precision': 0.69720596,\n",
       " 'prediction/mean': 0.23662092,\n",
       " 'recall': 0.5579823,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_regu.evaluate(input_fn=get_input_fn(df_test_new, \n",
    "                                      num_epochs=1,\n",
    "                                      n_batch = 128,\n",
    "                                      shuffle=False),\n",
    "                                      steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this hyperparameter, you slightly increase the accuracy metrics. In\n",
    "the next tutorial, you will learn how to improve a linear classifier\n",
    "using a kernel method.\n",
    "\n",
    "# Summary\n",
    "\n",
    "To train a model, you need to:\n",
    "\n",
    "-   Define the features: Independent variables: X\n",
    "\n",
    "-   Define the label: Dependent variable: y\n",
    "\n",
    "-   Construct a train/test set\n",
    "\n",
    "-   Define the initial weight\n",
    "\n",
    "-   Define the loss function: MSE\n",
    "\n",
    "-   Optimize the model: Gradient descent\n",
    "\n",
    "-   Define:\n",
    "\n",
    "-   Learning rate\n",
    "\n",
    "-   Number of epoch\n",
    "\n",
    "-   Batch size\n",
    "\n",
    "-   Number of class\n",
    "\n",
    "In this tutorial, you learned how to use the high-level API for a linear\n",
    "regression classifier. You need to define:\n",
    "\n",
    "1.  Feature columns. If continuous:\n",
    "    `tf.feature_column.numeric_column()`. You can populate a list with\n",
    "    python list comprehension\n",
    "\n",
    "2.  The estimator: `tf.estimator.LinearClassifier(feature_columns, model_dir, n_classes = 2)`\n",
    "\n",
    "3.  A function to import the data, the batch size and epoch: `input_fn()`\n",
    "\n",
    "After that, you are ready to train, evaluate and make a prediction with\n",
    "`train()`, `evaluate()` and `predict()`\n",
    "\n",
    "To improve the performance of the model, you can:\n",
    "\n",
    "-   Use polynomial regression\n",
    "\n",
    "-   Interaction term: `tf.feature_column.crossed_column`\n",
    "\n",
    "-   Add regularization parameter"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}