linear-regression-exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HTW-Berlin - Informatik und Wirtschaft - Aktuelle Trends - Machine Learning: Linear Regression Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "* [Introduction](#Introduction)\n",
    "  * [Required Knowledge](#Required-Knowledge)\n",
    "  * [Required Python Modules](#Required-Python-Modules)\n",
    "\n",
    "\n",
    "* [Linear Regression](#Linear-Regression)\n",
    "  * [Hypothesis](#Hypothesis)\n",
    "  * [Exercise: Cost](#Exercise:-Cost)\n",
    "  * [Exercise: Gradient](#Exercise:-Gradient)\n",
    "  * [Exercise: Stochastic Gradient Descent](#Exercise:-Stochastic-Gradient-Descent)\n",
    "  * [Exercise: Plot Cost per Epoch](#Exercise:-Plot-Cost-per-Epoch)\n",
    "\n",
    "\n",
    "* [Summary and Outlook](#Summary-and-Outlook)\n",
    "* [Licenses](#Licenses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Goal of this exercise is to implement Linear Regression in Python. You can use the Python standard library and math functions from numpy. This notebook guides you through the implementation process.\n",
    "\n",
    "This notebooks implements tests using `assert` or `np.testing.assert_almost_equal`. If you run the corresponding notebook cell and no output appears, the test has passed. Otherwise an exception is raised.\n",
    "\n",
    "**General Hint:**\n",
    "\n",
    "If you have problems with the implementation (e.g. you don't know how to call a certain function or you don't know how to loop through the dataset), make use of the interactive nature of the notebook. You can at all times add new cells to the notebook to inspect defined variables or to try small code snippets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required Knowledge\n",
    "\n",
    "This exercise is part of the course \"Aktuelle Trends der Informations- und Kommunikationstechnik\". The fundamentals of Linear Regression are taught in class.\n",
    "\n",
    "* The PDF slides used in class are [available](../../../../../slides/courses/htw-berlin/informatik-und-wirtschaft/aktuelle-trends/linear-regression.pdf) in the educational-materials repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required Python Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from deep_teaching_commons.data.fundamentals.iris import Iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_data_dir = os.path.expanduser('~/deep.TEACHING/data')\n",
    "dm = Iris(base_data_dir=base_data_dir)  # data manager\n",
    "iris = dm.dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = iris['petal_width'].values\n",
    "Y = iris['petal_length'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape, Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale `X` using the Scikit-Learn `StandardScaler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X.reshape(-1, 1)).flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_scaled[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X_scaled, Y, c='grey')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hypothesis\n",
    "\n",
    "The linear hypothesis is defined as follows:\n",
    "\n",
    "$$\n",
    "h(x) = x \\cdot w + b\n",
    "$$\n",
    "\n",
    "The hypothesis is a function of `x`, where the parameters `w` and `b` are treated as constants.\n",
    "\n",
    "As the following code shows, in Python we can use a *closure* to create functions with built-in constant values. Work through the given code below, it is used as an example for the following exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_linear_hypothesis(w, b):\n",
    "    # this is a closure\n",
    "    def linear_hypothesis(x):\n",
    "        return x * w + b\n",
    "    \n",
    "    return linear_hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = 1.9\n",
    "b = 3.8\n",
    "# create a function h with constant values for w and b\n",
    "h = make_linear_hypothesis(w, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# as you can see h is function\n",
    "type(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hypothesis $h$ is used to predict values for $y$ given $x$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -1.2 is a useful value for x because we have already scaled x\n",
    "# take a look at your plot above for comparison\n",
    "x = -1.2\n",
    "h(x)  # apply function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run a test\n",
    "np.testing.assert_almost_equal(h(-1.2), 1.52)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Cost\n",
    "\n",
    "The cost function is defined as follows:\n",
    "\n",
    "$$\n",
    "J(w, b) = \\frac{1}{2m} \\sum\\limits_{i=1}^{m} (h_{w, b}(x_i) - y_i)^2\n",
    "$$\n",
    "\n",
    "Implement this function using a closure, similar to the `make_linear_hypothesis` function above, where `mse_cost` ($J$) is defined by `make_mse_cost`. The mean squared error (MSE) cost function loops through the complete dataset of X and Y, to sum over the the squared distances between $h(x_i)$ (prediction) and $y_i$. As a last step the summation is divided by $2m$, where $m$ is the number of samples in $X$.\n",
    "\n",
    "$J$ is a function of $w$ and $b$, because we want to find the best parameters $w$ and $b$ providing the lowest possible cost. Therefore the data $X$ and $Y$ is treated as a constant. You can use the already implemented function `make_linear_hypothesis` inside of `mse_cost` to create a new hypothesis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_mse_cost(X, Y):\n",
    "    def mse_cost(w, b):\n",
    "        raise NotImplementedError('implement this function')\n",
    "    \n",
    "    return mse_cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "J = make_mse_cost(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = 1.9\n",
    "b = 3.8\n",
    "J(w, b)  # apply function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run a test\n",
    "np.testing.assert_almost_equal(J(1.9, 3.8), 2.832163333333332)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Gradient\n",
    "\n",
    "The partial derivatives (gradient) are used by the Stochastic Gradient Descent optimizer and are defined as follows:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w} J(w, b) &= \\frac{1}{m}\\sum_{i=1}^{m}(h_{w, b}(x_i) - y_i) \\cdot x_i\\\\\n",
    "\\frac{\\partial}{\\partial b} J(w, b) &= \\frac{1}{m}\\sum_{i=1}^{m}(h_{w, b}(x_i) - y_i)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Implement a function `gradient`, which calculates at a point $w, b$ the partial derivatives $pd\\_w$ and $pd\\_b$. Return both values from the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_gradient(X, Y):\n",
    "    def gradient(w, b):\n",
    "        raise NotImplementedError('implement this function')\n",
    "        \n",
    "        pd_w = None\n",
    "        pd_b = None\n",
    "        \n",
    "        return pd_w, pd_b\n",
    "    \n",
    "    return gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gradient = make_gradient(X_scaled, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = 1.9\n",
    "b = 3.8\n",
    "gradient(w, b)  # apply function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run a test\n",
    "np.testing.assert_almost_equal(gradient(1.9, 3.8), (0.20696354830509436, 0.04133333333333284))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Stochastic Gradient Descent\n",
    "\n",
    "The following pseude code shows the iterative parameter updates of Stochastic Gradient Descent:\n",
    "\n",
    "---\n",
    "\n",
    "Randomly initialize w and b.\n",
    "\n",
    "For a number of epochs repeat:\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "pd\\_w &:= \\frac{\\partial}{\\partial w} J(w, b)\\\\\n",
    "pd\\_b &:= \\frac{\\partial}{\\partial b} J(w, b)\\\\\\\\\n",
    "w &:= w - \\alpha * pd\\_w\\\\\n",
    "b &:= b - \\alpha * pd\\_b\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "The function to be implemented is `stochastic_gradient_descent(X, Y, w, b, alpha, epochs)`, where `X, Y` is the data, `w, b` are the randomly initialized parameters, `alpha` is the learning rate and `epochs` is the number of training iterations. You should return the values of $w$ and $b$, as well as a list of the MSE cost values. The cost has to be calculated once before the training starts and after each training epoch (e.g. 101 cost values if number of epochs is 100)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sgd(X, Y, w, b, alpha, epochs):\n",
    "    raise NotImplementedError('implement this function')\n",
    "    \n",
    "    cost_per_epoch = []\n",
    "    \n",
    "    return w, b, cost_per_epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.1\n",
    "epochs = 100\n",
    "np.random.seed(0)\n",
    "w, b = np.random.randn(2)\n",
    "w, b, cost_per_epoch = sgd(X_scaled, Y, w, b, alpha, epochs)  # Implementierung in der Übung\n",
    "w, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(cost_per_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run tests\n",
    "test_w, test_b, test_cost_per_epoch = sgd(X_scaled, Y, 1.292771109464669, -1.2955399908557792, 0.1, 100)\n",
    "np.testing.assert_almost_equal(len(test_cost_per_epoch), 101)\n",
    "np.testing.assert_almost_equal((test_w, test_b), (1.6930258200874897, 3.7585324198675756))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Plot Cost per Epoch\n",
    "\n",
    "Plot `cost_per_epoch` result of `sgd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_over_time(cost_per_epoch):\n",
    "    raise NotImplementedError('implement this function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.1\n",
    "epochs = 100\n",
    "np.random.seed(0)\n",
    "w, b = np.random.randn(2)\n",
    "w, b, cost_per_epoch = sgd(X_scaled, Y, w, b, alpha, epochs)\n",
    "plot_over_time(cost_per_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try out different `alpha` values and see how the training performance changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary and Outlook\n",
    "\n",
    "You have learned how to implement Linear Regression with one input and one output variable to solve simple regression problems. The algorithms were implemented in Python, without the help of higher level libraries like Tensorflow or Keras.\n",
    "\n",
    "The next part of the course covers Logistic Regression for classification tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Licenses\n",
    "\n",
    "### Notebook License (CC-BY-SA 4.0)\n",
    "\n",
    "*The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g. images).*\n",
    "\n",
    "HTW-Berlin - Informatik und Wirtschaft - Aktuelle Trends - Machine Learning: Linear Regression Exercise<br/>\n",
    "by [Christoph Jansen (deep.TEACHING - HTW Berlin)](https://www.htw-berlin.de/hochschule/personen/person/?eid=9225)<br/>\n",
    "is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/).<br/>\n",
    "Based on a work at https://gitlab.com/deep.TEACHING.\n",
    "\n",
    "\n",
    "### Code License (MIT)\n",
    "\n",
    "*The following license only applies to code cells of the notebook.*\n",
    "\n",
    "Copyright 2018 Christoph Jansen\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "deep_teaching_kernel",
   "language": "python",
   "name": "deep_teaching_kernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}