machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A short introduction to machine learning\n",
    "\n",
    "Machine learning is really just fancy statistics. \n",
    "Still, machine learning is fun, and here we’ll take a look at what we can do.\n",
    "There are three major types of machine learning:\n",
    "- Supervised learning \n",
    "- Unsupervised learning \n",
    "- Reinforcement learning \n",
    "\n",
    "Here, we will only discuss supervised and unsupervised learning.\n",
    "\n",
    "But first, we will need to import some libraries for machine learning.\n",
    "- scikit-learn (`sklearn`) is a toolkit for machine learning\n",
    "- matplotlib is a library for plotting and visualizing results\n",
    "- numpy is a package for scientific computing with Python "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import make_moons, make_circles, make_classification, make_blobs\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.linear_model import Perceptron\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "import numpy as np\n",
    "\n",
    "# define colors\n",
    "colors = plt.cm.RdBu\n",
    "colors_bright = ListedColormap(['#FF0000', '#0000FF'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised learning: classification\n",
    "\n",
    "By supervision we mean human supervision.\n",
    "Supervised learning works with labeled (annotated) data. The labels must be made manually by people, and this can require a lot of work for large data sets.\n",
    "\n",
    "A common machine learning task is *classification* of items.\n",
    "That is, determining which group or class an item belongs to.\n",
    "*Neural networks* are powerful classifiers that can be used with complex data sets. But there are also many simpler classifiers.\n",
    "We will try a couple of different non-neural classifiers, and then try neural networks in an exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start coding the machine learning part, we will need a way to visualize the results. \n",
    "Below, we define a function to plot the decision boundary produced by classifier.\n",
    "The details of this function are not important, so you don't need to spend much time understanding it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_boundary(classifier, data, datasets):\n",
    "    X_train, X_test, y_train, y_test = datasets\n",
    "    score = classifier.score(X_test, y_test)\n",
    "    figure = plt.figure(figsize=(7, 7))\n",
    "    h = .02  # step size in the mesh\n",
    "    x_min, x_max = data[:, 0].min() - .5, data[:, 0].max() + .5\n",
    "    y_min, y_max = data[:, 1].min() - .5, data[:, 1].max() + .5\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                         np.arange(y_min, y_max, h))\n",
    "\n",
    "    # Plot the decision boundary. For that, we will assign a color to each\n",
    "    # point in the mesh [x_min, x_max]x[y_min, y_max].\n",
    "    if hasattr(classifier, \"decision_function\"):\n",
    "        Z = classifier.decision_function(np.c_[xx.ravel(), yy.ravel()])\n",
    "    else:\n",
    "        Z = classifier.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]\n",
    "\n",
    "    # Put the result into a color plot\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    plt.contourf(xx, yy, Z, cmap=colors, alpha=.8)\n",
    "\n",
    "    # Plot the training points\n",
    "    plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=colors_bright,\n",
    "               edgecolors='k')\n",
    "    # Plot the testing points\n",
    "    plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=colors_bright,\n",
    "               edgecolors='k', alpha=0.6)\n",
    "    \n",
    "    #add labels\n",
    "    plt.xlabel('lazy - active')\n",
    "    plt.ylabel('size')\n",
    "    plt.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score),\n",
    "             size=15, horizontalalignment='right')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need some data to work with.\n",
    "As an example, we will use a simulated, random data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, labels = make_classification(n_features=2, n_redundant=0, n_informative=2,\n",
    "                                   random_state=2, n_clusters_per_class=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can examine the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can visualize them with matplotlib.\n",
    "The plot shows that this data set is nearly linearly separable.\n",
    "The blue class has a couple of extreme values or *outliers*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Plot the data\n",
    "plt.scatter(data[:, 0], data[:, 1], c=labels, cmap=colors_bright)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define a function that tests given classifier with our data set.\n",
    "\n",
    "When training a supervised machine learning algorithm, we want to be able to verify that the resulting model works on new data. We need to test that the model generalizes to unseen data.\n",
    "Therefore, we always split the data set into training and test sets.\n",
    "\n",
    "Then we train the classifier on the training set. This is called *fitting* the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_classifier(classifier, X, y):\n",
    "    datasets = train_test_split(X, y, test_size=.4, random_state=42)\n",
    "    X_train, X_test, y_train, y_test = datasets\n",
    "    classifier.fit(X_train, y_train)\n",
    "    plot_boundary(classifier, X, datasets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support-vector machine (SVM) classifier\n",
    "\n",
    "Let's try a linear SVM classifier.\n",
    "The accuracy on the test set is listed in the lower right corner of the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "classifier = SVC(kernel=\"linear\", C=0.025)\n",
    "run_classifier(classifier, data, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this classifier on new data. This is also called *predicting* their classes or labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(classifier.predict([[1.1, 2.0],\n",
    "                          [-1, -2]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-Nearest Neighbors (kNN)\n",
    "\n",
    "k-Nearest Neighbors (kNN) is an efficient classifier.\n",
    "kNN assigns a data point the same class as the majority of its neighbors.\n",
    "The value k specifies how many neighbors should be considered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "classifier = KNeighborsClassifier(3)\n",
    "run_classifier(classifier, data, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linearly separable data\n",
    "\n",
    "The data we have worked with so far, are (nearly) linearly separable.\n",
    "Let's try the SVM and kNN classifiers with a data set that is not linearly separable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2, labels2 = make_circles(noise=0.2, factor=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The linear SVM classifier works poorly with this data set, and yields a low accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = SVC(kernel=\"linear\", C=0.025)\n",
    "run_classifier(classifier, data2, labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " kNN, on the other hand, works well with data that is not linearly separable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "classifier = KNeighborsClassifier(3)\n",
    "run_classifier(classifier, data2, labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised clustering\n",
    "\n",
    "In unsupervised learning there are no labels.\n",
    "The task is to find patterns in the data, for example by clustering them.\n",
    "We avoid the human effort to label the data but get less information as a result.\n",
    "\n",
    "First, let's plot the unlabeled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(7, 7))\n",
    "plt.scatter(data[:, 0], data[:, 1])#, s=5, linewidth=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-means clustering\n",
    "\n",
    "k-means clustering is a common clustering algorithm.\n",
    "It takes the parameter *k*, which specifies the number of clusters we want to find.\n",
    "In other words, the user needs to know beforehand, *a priori*, how many clusters they wish to look for.\n",
    "We can of course try different values for k, to see which works better.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Kmean = KMeans(n_clusters=2)\n",
    "Kmean.fit(data)\n",
    "clusters = Kmean.predict(data)\n",
    "figure = plt.figure(figsize=(7, 7))\n",
    "plt.scatter(data[:, 0], data[:, 1])#, s=5)#, linewidth=0, c=clusters)\n",
    "for cluster_x, cluster_y in Kmean.cluster_centers_:\n",
    "    plt.scatter(cluster_x, cluster_y, s=100, c='r', marker='x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Decision tree classifier\n",
    "\n",
    "Below, we define a decision tree classifier.\n",
    "Use this classifier to classify both our data sets, data and data2.\n",
    "Discuss and explain the results, especially the decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "decision_tree_classifier = DecisionTreeClassifier(max_depth=5)\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Neural network classifier\n",
    "\n",
    "Below, we define two neural network classifiers.\n",
    "Use these classifiers to classify both our data sets, data and data2.\n",
    "Compare and discuss the results. What does the decision boundary\n",
    "tell us about these classifiers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "single_layer_network = Perceptron()\n",
    "multi_layer_network = MLPClassifier(max_iter=1000)\n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: k-Nearest Neighbors (kNN) classifier\n",
    "\n",
    "This is the same kNN classifier as we saw above. Try out different values for k and the number of data points, n_samples.\n",
    "Can you find a good value for k for 300 samples?\n",
    "\n",
    "The accuracy score is in the lower right corner of the plot.\n",
    "Which accuracy do you get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data3, labels3 = make_circles(n_samples = 100, noise=0.2, factor=0.5, random_state=5)\n",
    "\n",
    "kNN_classifier = KNeighborsClassifier(10)\n",
    "run_classifier(kNN_classifier, data3, labels3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}