03-your-first-cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: left; margin-bottom: 1em\" src=\"images/PRACE.png\" width=\"200\">\n",
    "<img style=\"float: right; margin-bottom: 1em\" src=\"images/surfsara.png\" width=\"150\">\n",
    "<hr style=\"clear: both\"/>\n",
    "\n",
    "# Your first convolutional neural network\n",
    "In this notebook you will learn how to build and train a convolutional neural network (CNN) yourself, and to figure out what the network is learning.\n",
    "\n",
    "We will continue with the leaf classification problem from before, and make it a bit harder. Instead of the ten classes from before, we now have 15, with and build a first CNN to evaluate its performance against a typical dense network.\n",
    "\n",
    "Let's load the data first. It consists of a training and validation set. Let's print the class names as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lib\n",
    "import keras\n",
    "\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "(X_train, Y_train), (X_val, Y_val), labels = lib.dataset_plant_village_small()\n",
    "\n",
    "for index, name in enumerate(labels):\n",
    "    print('{}: {}'.format(str(index).rjust(2), name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the examples in the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "lib.plot_examples(X_train[:32], Y_train[:32], labels);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "It's always a good idea to establish a baseline performance to compare later models against. Build a small dense network to solve this 15-class classification problem by filling out the skeleton below. Create a hidden layer with softmax activation, and two dense hidden layers before that with 512 and 256 units, respectively. Train the network for 10 epochs with learning rate 0.001."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Conv2D, MaxPool2D, Flatten, Dense\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=data_train.image_shape))\n",
    "# <FILL IN>\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=<FILL IN>), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(\n",
    "    X_train, Y_train,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    epochs=<FILL IN>\n",
    ")\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The images used in the previous exercise are quite small: 64 by 64 pixels. Already, with a relatively small network and a small training set, the training time is quite high. This problem becomes worse with bigger images. When an image becomes twice as big, the number of parameters in the first dense layer will increase by a factor four. In addition, although the model improves, it does so very slowly.\n",
    "\n",
    "Clearly, we need a more scalable solution. CNNs provide a less complex model to reduce training time, and also allow us to do **local feature detection**: convolutional layers detect patterns in the image independent of their position.\n",
    "\n",
    "## Exercise 2\n",
    "Train a simple convolutional neural network to solve this problem by filling out the skeleton below.\n",
    "\n",
    "In the first convolutional layer, use 32 filters and a kernel size of 3, with 'relu' activation. The last layer should be a single dense layer of 15 units with softmax activation, similar to the last layer in the dense network you created in exercise 1.\n",
    "\n",
    "Train for 10 epochs with a learning rate of 0.001."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Conv2D, MaxPool2D\n",
    "\n",
    "model = Sequential()\n",
    "# First conv. -> max. pool block\n",
    "\n",
    "model.add(Conv2D(filters=<FILL IN>, kernel_size=<FILL IN>, activation='<FILL IN>', input_shape=X_train.shape[1:]))\n",
    "model.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2)))\n",
    "\n",
    "# Dense layers for final classification\n",
    "\n",
    "model.add(Flatten())\n",
    "# <FILL IN>\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=<FILL IN>), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(\n",
    "    X_train, Y_train,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    epochs=<FILL IN>\n",
    ")\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "We see that the CNN performs better than the dense network because the accuracy at epoch 10 is much higher for the CNN. Why do you think it performs much better than the dense network? Write down your answer in the cell below.\n",
    "\n",
    "**Hint**: convolutional layers activate **independent** of the location of a particular pattern. How would that help for this particular classification problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4\n",
    "In exercise 2 you created a network with a single convolution 'block', consisting of a convolution layer and max pooling layer. The power of convolutional neural networks comes from building a **feature hierarchy** by stacking several of these blocks after each other.\n",
    "\n",
    "In the skeleton below, stack a number of these blocks on top of each other by copy-pasting the first block. Each block should model more complexity, so multiply the number of kernels (the first parameter, 32 in the first block) for each block. This means that the first convolutional layer should have 32 kernels, the second 64, the third 128, etc. Try to have at least three of these blocks after each other.\n",
    "\n",
    "Train the network with 10 epochs and a learning rate of your choice, and try to get the training accuracy above 0.95."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape=X_train.shape[1:]))\n",
    "model.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2)))\n",
    "\n",
    "# <FILL IN>\n",
    "\n",
    "# Dense network for end classification\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(15, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=0.001), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(\n",
    "    X_train, Y_train,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    epochs=10\n",
    ")\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network performs extremely well. It will build a feature hierarchy and in this way compress the original input image to a very small image with many feature channels (the last dimension in the output shape in the summary). These feature channels are passed along to a dense layer for the final classification.\n",
    "\n",
    "## Class activation maps\n",
    "We can have a look at the **class activation maps** for the healthy and metastasised classes. These will tell us where the filters of the convolutional layers activate in order to predict a certain class. In other words, the class activation maps will tell us which areas of the image are important **for that layer** to predict the target class.\n",
    "\n",
    "We will show an overlay for an input image for each convolutional layer with respect to the first four classes. Generating the activation images is an optimization problem and will take a lot of time if we generate them for each combination of convolution layer and class.\n",
    "\n",
    "**Hint**: you can try changing the `example_index` variable to look at the class activation maps of different examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "example_index = 0\n",
    "lib.plot_cam(model, X_train[example_index], np.argmax(Y_train[example_index]), labels);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the image above will show is where a particular convolutional layer will activate in order to maximise the prediction of a certain class. It may give you some idea about where a particular filter is 'looking' in order to come to a certain decision. The row label in bold is the correct label for that image.\n",
    "\n",
    "## Exercise 5\n",
    "The plots above show the activation maps from the first convolutional layer to the last, from left to right. We can see the activations become focused on larger areas, and become less granular we go deeper into the network. Why do you think that is?\n",
    "\n",
    "**Hint**: think about the last operation in each convolutional block. How will it affect filter responses?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activations\n",
    "Another way to find out what the network is doing is by looking at the convolutional filters at each layer. The best way to investigate these is to generate an image that would maximally activate a given filter. That is: if we provide a certain filter with an image, it should activate to its maximum extent. In this way, it will show us what visual pattern it has learnt to detect.\n",
    "\n",
    "## Exercise 6\n",
    "In the cell below, fill in the name of the **second** convolutional layer you trained in exercise 4, and run the cell. It will show you the patterns for the first 32 filters of the second convolutional layer. Do the patterns make sense to you? What patterns do you expect to see? Fill in your answer in the cell below.\n",
    "\n",
    "**Notes**:\n",
    "* This can take a minute or three, since each image is generated by optimisation;\n",
    "* Not all activations will show you a pattern. This is because not all filters will be optimised evenly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_name = '<FILL IN>'\n",
    "lib.plot_activations(model, layer_name, 32);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7\n",
    "Rerun exercise 6, this time with the **last** convolutional layer. Do you see a difference in the complexity of the patterns? Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_name = '<FILL IN>'\n",
    "lib.plot_activations(model, layer_name, 32);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise (hard)\n",
    "Create a fully convolutional network. This network does not have any dense layers. The last layer will consists of a convolutional layer that outputs two probabilities using sigmoid activation. How does its performance compare to the network you designed in exercise 4?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# SOLUTION\n",
    "\n",
    "model = Sequential()\n",
    "# FILL IN\n",
    "model.add(Flatten())\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=0.0001), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(\n",
    "    X_train, Y_train,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    epochs=10\n",
    ")\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise (hard)\n",
    "Look at the number of parameters for the convolutional layers in exercise 4. How do you arrive at the numbers listed in the summary? Write down your answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}