L1.Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/ML-Challenge/week3-supervised-learning/blob/master/L1.Classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download lesson datasets\n",
    "# Required if you're using Google Colab\n",
    "!wget \"https://github.com/ML-Challenge/week3-supervised-learning/raw/master/datasets.zip\"\n",
    "!unzip -o datasets.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import utils, we'll be using this module throughout the lesson\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:03:55.892434Z",
     "start_time": "2020-02-05T11:03:55.734855Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import dependencies\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "# and setting the size of all plots.\n",
    "plt.rcParams['figure.figsize'] = [11, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson, we will introduce classification problems and show how to solve them using supervised learning techniques. We will apply the concepts to a political dataset, where we classify the party affiliation of United States members of Congress based on their voting records."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yogi Berra said, \"You can observe a lot by watching\". The same is true for data. If we can appropriately display our data, we can already start to draw conclusions from it. Therefore exploring the data is a crucial step in analysis. By exploring, we mean organizing and plotting the data, and computing a few numerical summaries about it. This idea is known as exploratory data analysis, or EDA, and was developed by one of the greatest statisticians of all time, [John Tukey](https://en.wikipedia.org/wiki/John_Tukey). He wrote a book entitled `Exploratory Data Analysis` in 1977, where he laid out the principles. In that book, he said, \"Exploratory data analysis can never be the whole story, but nothing else can serve as the foundation stone.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numerical EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we'll be working with a dataset obtained from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records) consisting of votes made by the US House of Representatives Congressmen. Our goal will be to predict their party affiliation ('Democrat' or 'Republican') based on how they voted on certain key issues. Here, it's worth noting that we have preprocessed this dataset to deal with missing values. This is so that the focus can be directed towards understanding how to train and evaluate supervised learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:03:56.522846Z",
     "start_time": "2020-02-05T11:03:56.496924Z"
    }
   },
   "outputs": [],
   "source": [
    "utils.house_votes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:03:56.677602Z",
     "start_time": "2020-02-05T11:03:56.654642Z"
    }
   },
   "outputs": [],
   "source": [
    "utils.house_votes.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:03:56.862678Z",
     "start_time": "2020-02-05T11:03:56.815819Z"
    }
   },
   "outputs": [],
   "source": [
    "utils.house_votes.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visual EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Numerical EDA we did in the previous example gave us some very important information, such as the names and data types of the columns, and the dimensions of the DataFrame. Following this with some visual EDA will give us an even better understanding of the data. Because all the features in this dataset are binary; that is, they are either 0 or 1, we can use [Seaborn's](http://seaborn.pydata.org/generated/seaborn.countplot.html) [countplot](http://seaborn.pydata.org/generated/seaborn.countplot.html) to visualize the data: \n",
    "\n",
    "```\n",
    "plt.figure()\n",
    "sns.countplot(x='education', hue='party', data=df, palette='RdBu')\n",
    "plt.xticks([0,1], ['No', 'Yes'])\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `sns.countplot()`, we specify the x-axis data to be `'education'`, and hue to be `'party'`. Recall that `'party'` is also our target variable. So the resulting plot shows the difference in voting behavior between the two parties for the `'education'` bill, with each party colored differently. We manually specified the color to be `'RdBu'`, as the Republican party has been traditionally associated with red, and the Democratic party with blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:03:57.918175Z",
     "start_time": "2020-02-05T11:03:57.442421Z"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "plt.figure()\n",
    "sns.countplot(x='education', hue='party', data=utils.house_votes, palette='RdBu')\n",
    "plt.xticks([0,1], ['No', 'Yes'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like Democrats voted resoundingly *against* this bill, compared to Republicans. This is the kind of information that our machine learning model will seek to learn when we try to predict party affiliation solely based on voting behavior. An expert in U.S politics may be able to predict this without machine learning, but probably not instantaneously - and certainly not if we are dealing with hundreds of samples!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The classification challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a set of labeled data, and we want to build a classifier that takes unlabeled data as input and outputs a label. So how do we construct this classifier? We first need to choose a type of classifier, and it needs to learn from the already labeled data. For this reason, we call the already labeled data the training data. So let's build our first classifier!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-Nearest Neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will choose a simple algorithm called K-nearest neighbors. The basic idea of K-nearest neighbors, or KNN, is to predict the label of any data point by looking at the K, for example, 3, closest labeled data points, and getting them to vote on what label the unlabeled point should have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following image, there's and example of KNN in two dimensions: how do we classify the data point in the middle?\n",
    "\n",
    "![Two dimensions KNN](assets/KNN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, if `k = 3` then the class is `B` but if `k = 6` the class is `A`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-NN: Intuition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a bit of intuition for KNN, let's checkout out a scatter plot of two dimensions of the [Iris dataset](https://archive.ics.uci.edu/ml/datasets/Iris), petal length and petal width. The following holds for higher dimensions, however, we'll show the 2D case for illustrative purposes.\n",
    "\n",
    "![Iris Sample](assets/iris.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the KNN algorithm essentially does is to create a set of decision boundaries, and we visualize the 2D case here. Any new data point will be identified according to the zone it is in.\n",
    "\n",
    "![KNN boundaries](assets/iris_knn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scikit-learn fit and predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All machine learning models in `scikit-learn` are implemented as python classes. These classes serve two purposes: they implement the algorithms for learning a model, and predicting, while also storing all the information that is learned from the data.\n",
    "\n",
    "Training a model on the data is also called fitting the model to the data. In `scikit-learn`, we use the `.fit()` method to do this.\n",
    "\n",
    "Similarly, the `.predict()` method is what we use to predict the label of an unlabeled data point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-Nearest Neighbors: Fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having explored the Congressional voting records dataset, it is time now to build our first classifier. In this example, we will fit a k-Nearest Neighbors classifier to the voting dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must ensure that the data adheres to the format required by the scikit-learn API. The features need to be in an array where each column is a feature, and each row a different observation or data point - in this case, a Congressman's voting record. The target needs to be a single column with the same number of observations as the feature data. Notice we named the feature array `X` and response variable `y`: This is in accordance with the common scikit-learn practice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create an instance of a k-NN classifier with 6 neighbors (by specifying the n_neighbors parameter) and then fit it to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:00.653685Z",
     "start_time": "2020-02-05T11:04:00.120730Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import KNeighborsClassifier from sklearn.neighbors\n",
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:00.731510Z",
     "start_time": "2020-02-05T11:04:00.717516Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create arrays for the features and the response variable\n",
    "y = utils.house_votes['party'].values\n",
    "X = utils.house_votes.drop('party', axis=1).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:00.808947Z",
     "start_time": "2020-02-05T11:04:00.796436Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a k-NN classifier with 6 neighbors\n",
    "knn = KNeighborsClassifier(n_neighbors=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:00.887556Z",
     "start_time": "2020-02-05T11:04:00.873559Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fit the classifier to the data\n",
    "knn.fit(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-Nearest Neighbors: Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having fit a k-NN classifier, we can now use it to predict the label of a new data point. However, there is no unlabeled data available since all of it was used to fit the model! We can still use the `.predict()` method on the `X` that was used to fit the model, but it is not a good indicator of the model's ability to generalize to new, unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the classifier to predict the label for this new data point, as well as on the training data X that the model has already seen. Using `.predict()` on X_new will generate 1 prediction, while using it on X will generate 435 predictions: 1 for each sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:01.217647Z",
     "start_time": "2020-02-05T11:04:01.196702Z"
    }
   },
   "outputs": [],
   "source": [
    "# Predict the labels for the training data X\n",
    "y_pred = knn.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:01.403524Z",
     "start_time": "2020-02-05T11:04:01.381584Z"
    }
   },
   "outputs": [],
   "source": [
    "X_new = pd.DataFrame([[0.69646919, 0.28613933, 0.22685145, 0.55131477, 0.71946897, 0.42310646, 0.9807642 , 0.68482974, 0.4809319 , 0.39211752, 0.34317802, 0.72904971, 0.43857224, 0.0596779 , 0.39804426, 0.73799541]], columns=utils.house_votes_columns[1:])\n",
    "X_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:01.559760Z",
     "start_time": "2020-02-05T11:04:01.548790Z"
    }
   },
   "outputs": [],
   "source": [
    "# Predict and print the label for the new data point X_new\n",
    "new_prediction = knn.predict(X_new)\n",
    "print(\"Prediction: {}\".format(new_prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measuring model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to fit a classifier and use it to predict the labels of previously unseen data, we need to figure out how to measure its performance. That is, we need a metric. In classification problems, accuracy is a commonly-used metric.\n",
    "\n",
    "The accuracy of a classifier is defined as the number of correct predictions divided by the total number of data points. This begs the questions, though: which data do we use to compute accuracy?\n",
    "What we are interested in is how well our model will perform on new data, that is, samples that the algorithm has never seen before.\n",
    "\n",
    "Well, we could compute the accuracy of the data that we used to fit the classifier. However, as this data was used to train it, the classifier's performance will not be indicative of how well it can generalize to unseen data. For this reason, it is common practice to split the data into two sets, a training set, and a test set.\n",
    "\n",
    "We train or fit the classifier on the training set. Then we make predictions on the labeled test set and compare these predictions with the known labels. We then compute the accuracy of our predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The digits recognition dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up until now, we have been performing binary classification, since the target variable had two possible outcomes. However, we can also perform a multi-class classification, where the target variable could take on many possible outcomes. In the following examples, we'll be working with the [MNIST](http://yann.lecun.com/exdb/mnist/) digits recognition dataset, which has 10 classes, the digits 0 through 9! A reduced version of the MNIST dataset is one of scikit-learn's included datasets, and that is the one we will use in this example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sample in this scikit-learn dataset is an 8x8 image representing a handwritten digit. Each pixel is represented by an integer in the range 0 to 16, indicating varying levels of black.`scikit-learn's` built-in datasets are of type `Bunch`, which are dictionary-like objects. Helpfully for the MNIST dataset, scikit-learn provides an `'images'` key in addition to the `'data'` and `'target'` keys. Because it is a 2D array of the images corresponding to each sample, this `'images'` key is useful for visualizing the images, as we'll see in this example. On the other hand, the 'data' key contains the feature array - that is, the images as a flattened array of 64 pixels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we can access the keys of these Bunch objects in two different ways: By using the `.` notation, as in `digits.images`, or the `[]` notation, as in `digits['images']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:02.710321Z",
     "start_time": "2020-02-05T11:04:02.682429Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:02.943844Z",
     "start_time": "2020-02-05T11:04:02.862636Z"
    }
   },
   "outputs": [],
   "source": [
    "# Load the digits dataset: digits\n",
    "digits = datasets.load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:03.039512Z",
     "start_time": "2020-02-05T11:04:03.024584Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Print the keys and DESCR of the dataset\n",
    "print(digits.keys())\n",
    "print(digits.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:03.193101Z",
     "start_time": "2020-02-05T11:04:03.183161Z"
    }
   },
   "outputs": [],
   "source": [
    "# Print the shape of the images and data keys\n",
    "print(digits.images.shape)\n",
    "print(digits.data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:03.488318Z",
     "start_time": "2020-02-05T11:04:03.358570Z"
    }
   },
   "outputs": [],
   "source": [
    "# Display digit 1010\n",
    "plt.imshow(digits.images[1010], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Test/ Split + Fit/Predict/Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have learned about the importance of splitting our data into training and test sets, it's time to practice doing this on the digits dataset! After creating arrays for the features and target variable, we will split them into training and test sets, fit a k-NN classifier to the training data, and then compute its accuracy using the `.score()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:03.892547Z",
     "start_time": "2020-02-05T11:04:03.874564Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:04.047863Z",
     "start_time": "2020-02-05T11:04:04.033914Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create feature and target arrays\n",
    "X = digits.data\n",
    "y = digits.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first argument is the feature data, the second the targets or labels. The `test_size` keyword argument specifies what proportion of the original data is used for the test set. Lastly, the `random_state` kwarg sets a seed for the random number generator that splits the data into a train and test. Setting the seed with the same argument later will allow us to reproduce the exact split and the downstream results.\n",
    "\n",
    "`train_test_split` returns four arrays: the training data, the test data, the training labels, and the test labels. We unpack these into four variables: `X_train`, `X_test`, `y_train`, `y_test`, respectively. By default, `train_test_split` splits the data into 75% training data and 25% test data, which is a good rule of thumb. In the next example, we specify the size of the test set to 20%.\n",
    "\n",
    "It is also best practice to perform the split so that the test split reflects the labels of the data. That is, we want the labels to be distributed in train and test sets as they are in the original dataset. To achieve this, we use the keyword argument `stratify=y`, where `y` is the list or array containing the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:04.388718Z",
     "start_time": "2020-02-05T11:04:04.380756Z"
    }
   },
   "outputs": [],
   "source": [
    "# Split into training and test set and stratify the split according to the labels so that they are distributed in the training and test sets as they are in the original dataset.\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:04.544680Z",
     "start_time": "2020-02-05T11:04:04.527726Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a k-NN classifier with 7 neighbors: knn\n",
    "knn = KNeighborsClassifier(n_neighbors=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:04.714401Z",
     "start_time": "2020-02-05T11:04:04.705424Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fit the classifier to the training data\n",
    "knn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check out the accuracy of our model, we use the `.score()` method of the model and pass it `X_test` and `y_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:05.097601Z",
     "start_time": "2020-02-05T11:04:05.025869Z"
    }
   },
   "outputs": [],
   "source": [
    "# Print the accuracy\n",
    "print(knn.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the accuracy of our K-nearest neighbors model is approximately `98%`, which is pretty good for an out-of-the-box model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model complexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We discussed the concept of a decision boundary. Here, we visualize a decision boundary for several, increasing values of `K` in a KNN model. Note that, as `K` increases, the decision boundary gets smoother and less curvy. Therefore, we consider it to be a less complex model that those with a lower `K`.\n",
    "\n",
    "![Decision Boundaries](assets/decision_boundaries.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, complex models run the risk of being sensitive to noise in the specific data that we have, rather than reflecting general trends in the data. This is known as `overfitting`.\n",
    "\n",
    "If we increase the `K` even more and make the model even simpler, then the model will perform less well on both test and training sets, as indicated in this schematic figure, knows as a model complexity curve.\n",
    "\n",
    "![Model Complexity Curve](assets/underfitting_and_overfitting.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there is a sweet spot in the middle that gives us the best performance on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overfitting and underfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will compute and plot the training and testing accuracy scores for a variety of different neighbor values. By observing how the accuracy scores differ for the training and testing sets with different values of k, we will develop some intuition for `overfitting` and `underfitting`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:07.553685Z",
     "start_time": "2020-02-05T11:04:07.542747Z"
    }
   },
   "outputs": [],
   "source": [
    "# Setup arrays to store train and test accuracies\n",
    "neighbors = np.arange(1, 9)\n",
    "train_accuracy = np.empty(len(neighbors))\n",
    "test_accuracy = np.empty(len(neighbors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:09.657523Z",
     "start_time": "2020-02-05T11:04:07.791512Z"
    }
   },
   "outputs": [],
   "source": [
    "# Loop over different values of k\n",
    "for i, k in enumerate(neighbors):\n",
    "    # Setup a k-NN Classifier with k neighbors: knn\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "\n",
    "    # Fit the classifier to the training data\n",
    "    knn.fit(X_train, y_train)\n",
    "    \n",
    "    # Compute accuracy on the training set\n",
    "    train_accuracy[i] = knn.score(X_train, y_train)\n",
    "\n",
    "    # Compute accuracy on the testing set\n",
    "    test_accuracy[i] = knn.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-05T11:04:09.920578Z",
     "start_time": "2020-02-05T11:04:09.766813Z"
    }
   },
   "outputs": [],
   "source": [
    "# Generate plot\n",
    "plt.title('k-NN: Varying Number of Neighbors')\n",
    "plt.plot(neighbors, test_accuracy, label = 'Testing Accuracy')\n",
    "plt.plot(neighbors, train_accuracy, label = 'Training Accuracy')\n",
    "plt.legend()\n",
    "plt.xlabel('Number of Neighbors')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Week 3 - Supervised Learning](https://radu-enuca.gitbook.io/ml-challenge/supervised-learning)**\n",
    "\n",
    "*Have questions or comments? Visit the ML Challenge Mattermost Channel.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}