05a - Deep Neural Networks (TensorFlow).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning with TensorFlow\n",
    "\n",
    "Classical machine learning relies on using statistics to determine relationships between features and labels, and can be very effective for creating predictive models. However, a massive growth in the availability of data coupled with advances in the computing technology required to process it has led to the emergence of new machine learning techniques that mimic the way the brain processes information in a structure called an artificial neural network.\n",
    "\n",
    "TensorFlow is a framework for creating machine learning models, including deep neural networks (DNNs). In this example, we'll use Tensorflow to create a simple neural network that classifies penguins into species based on the length and depth of their culmen (bill), their flipper length, and their body mass.\n",
    "\n",
    "> **Citation**: The penguins dataset used in the this exercise is a subset of data collected and made available by [Dr. Kristen\n",
    "Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php)\n",
    "and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a\n",
    "member of the [Long Term Ecological Research\n",
    "Network](https://lternet.edu/).\n",
    "\n",
    "## Explore the dataset\n",
    "\n",
    "Before we start using TensorFlow to create a model, let's load the data we need from the Palmer Islands penguins dataset, which contains observations of three different species of penguin.\n",
    "\n",
    "> **Note**: In reality, you can solve the penguin classification problem easily using classical machine learning techniques without the need for a deep learning model; but it's a useful, easy to understand dataset with which to demonstrate the principles of neural networks in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# load the training dataset (excluding rows with null values)\n",
    "penguins = pd.read_csv('data/penguins.csv').dropna()\n",
    "\n",
    "# Deep Learning models work best when features are on similar scales\n",
    "# In a real solution, we'd implement some custom normalization for each feature, but to keep things simple\n",
    "# we'll just rescale the FlipperLength and BodyMass so they're on a similar scale to the bill measurements\n",
    "penguins['FlipperLength'] = penguins['FlipperLength']/10\n",
    "penguins['BodyMass'] = penguins['BodyMass']/100\n",
    "\n",
    "# The dataset is too small to be useful for deep learning\n",
    "# So we'll oversample it to triple its size\n",
    "for i in range(1,3):\n",
    "    penguins = penguins.append(penguins)\n",
    "\n",
    "# Display a random sample of 10 observations\n",
    "sample = penguins.sample(10)\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Species** column is the label our model will predict. Each label value represents a class of penguin species, encoded as 0, 1, or 2. The following code shows the actual species to which these class labels corrrespond."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguin_classes = ['Amelie', 'Gentoo', 'Chinstrap']\n",
    "print(sample.columns[0:5].values, 'SpeciesName')\n",
    "for index, row in penguins.sample(10).iterrows():\n",
    "    print('[',row[0], row[1], row[2],row[3], int(row[4]), ']',penguin_classes[int(row[-1])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As is common in a supervised learning problem, we'll split the dataset into a set of records with which to train the model, and a smaller set with which to validate the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "features = ['CulmenLength','CulmenDepth','FlipperLength','BodyMass']\n",
    "label = 'Species'\n",
    "   \n",
    "# Split data 70%-30% into training set and test set\n",
    "x_train, x_test, y_train, y_test = train_test_split(penguins[features].values,\n",
    "                                                    penguins[label].values,\n",
    "                                                    test_size=0.30,\n",
    "                                                    random_state=0)\n",
    "\n",
    "print ('Training Set: %d, Test Set: %d \\n' % (len(x_train), len(x_test)))\n",
    "print(\"Sample of features and labels:\")\n",
    "\n",
    "# Take a look at the first 25 training features and corresponding labels\n",
    "for n in range(0,24):\n",
    "    print(x_train[n], y_train[n], '(' + penguin_classes[y_train[n]] + ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *features* are the cupmen measurements for each penguin observation, and the *label* is a numeric value that indicates the species of penguin that the observation represents (Amelie, Gentoo, or Chinstrap).\n",
    "\n",
    "## Install and import TensorFlow libraries\n",
    "\n",
    "Since we plan to use TensorFlow to create our penguin classifier, we'll need to run the following two cells to install and import the libraries we intend to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras import utils\n",
    "from tensorflow.keras import optimizers\n",
    "\n",
    "# Set random seed for reproducability\n",
    "tensorflow.random.set_seed(0)\n",
    "\n",
    "print(\"Libraries imported.\")\n",
    "print('Keras version:',keras.__version__)\n",
    "print('TensorFlow version:',tensorflow.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data for TensorFlow\n",
    "\n",
    "We've already loaded our data and split it into training and validation datasets. However, we need to do some further data preparation so that our data will work correctly with TensorFlow. Specifically, we need to set the data type of our features to 32-bit floating point numbers, and specify that the labels represent categorical classes rather than numeric values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set data types for float features\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "\n",
    "# Set data types for categorical labels\n",
    "y_train = utils.to_categorical(y_train)\n",
    "y_test = utils.to_categorical(y_test)\n",
    "print('Ready...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a neural network\n",
    "\n",
    "Now we're ready to define our neural network. In this case, we'll create a network that consists of 3 fully-connected layers:\n",
    "* An input layer that receives an input value for each feature (in this case, the four penguin measurements) and applies a *ReLU* activation function.\n",
    "* A hidden layer that receives ten inputs and applies a *ReLU* activation function.\n",
    "* An output layer that uses a *SoftMax* activation function to generate an output for each penguin species (which represent the classification probabilities for each of the three possible penguin species)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a classifier network\n",
    "hl = 10 # Number of hidden layer nodes\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(hl, input_dim=len(features), activation='relu'))\n",
    "model.add(Dense(hl, input_dim=hl, activation='relu'))\n",
    "model.add(Dense(len(penguin_classes), input_dim=hl, activation='softmax'))\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "To train the model, we need to repeatedly feed the training values forward through the network, use a loss function to calculate the loss, use an optimizer to backpropagate the weight and bias value adjustments, and validate the model using the test data we withheld.\n",
    "\n",
    "To do this, we'll apply an Adam optimizer to a categorical cross-entropy loss function iteratively over 50 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#hyper-parameters for optimizer\n",
    "learning_rate = 0.001\n",
    "opt = optimizers.Adam(lr=learning_rate)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=opt,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Train the model over 50 epochs using 10-observation batches and using the test holdout dataset for validation\n",
    "num_epochs = 50\n",
    "history = model.fit(x_train, y_train, epochs=num_epochs, batch_size=10, validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review training and validation loss\n",
    "\n",
    "After training is complete, we can examine the loss metrics we recorded while training and validating the model. We're really looking for two things:\n",
    "* The loss should reduce with each epoch, showing that the model is learning the right weights and biases to predict the correct labels.\n",
    "* The training loss and validations loss should follow a similar trend, showing that the model is not overfitting to the training data.\n",
    "\n",
    "Let's plot the loss metrics and see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "epoch_nums = range(1,num_epochs+1)\n",
    "training_loss = history.history[\"loss\"]\n",
    "validation_loss = history.history[\"val_loss\"]\n",
    "plt.plot(epoch_nums, training_loss)\n",
    "plt.plot(epoch_nums, validation_loss)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the learned weights and biases\n",
    "\n",
    "The trained model consists of the final weights and biases that were determined by the optimizer during training. Based on our network model we should expect the following values for each layer:\n",
    "* Layer 1: There are four input values going to ten output nodes, so there should be 4 x 10 weights and 10 bias values.\n",
    "* Layer 2: There are ten input values going to ten output nodes, so there should be 10 x 10 weights and 10 bias values.\n",
    "* Layer 3: There are ten input values going to three output nodes, so there should be 10 x 3 weights and 3 bias values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for layer in model.layers:\n",
    "    weights = layer.get_weights()[0]\n",
    "    biases = layer.get_weights()[1]\n",
    "    print('------------\\nWeights:\\n',weights,'\\nBiases:\\n', biases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance\n",
    "\n",
    "So, is the model any good? The raw accuracy reported from the validation data would seem to indicate that it predicts pretty well; but it's typically useful to dig a little deeper and compare the predictions for each possible class. A common way to visualize the performace of a classification model is to create a *confusion matrix* that shows a crosstab of correct and incorrect predictions for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensorflow doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "class_probabilities = model.predict(x_test)\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "true_labels = np.argmax(y_test, axis=1)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm = confusion_matrix(true_labels, predictions)\n",
    "plt.imshow(cm, interpolation=\"nearest\", cmap=plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "tick_marks = np.arange(len(penguin_classes))\n",
    "plt.xticks(tick_marks, penguin_classes, rotation=85)\n",
    "plt.yticks(tick_marks, penguin_classes)\n",
    "plt.xlabel(\"Actual Class\")\n",
    "plt.ylabel(\"Predicted Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix should show a strong diagonal line indicating that there are more correct than incorrect predictions for each class.\n",
    "\n",
    "## Use the trained model\n",
    "\n",
    "Now that we have a model we believe is reasonably accurate, we can use it to predict the species of new penguin observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save the trained model\n",
    "modelFileName = 'models/penguin-classifier.h5'\n",
    "model.save(modelFileName)\n",
    "\n",
    "# Load the saved model when required\n",
    "del model  # deletes the existing model variable\n",
    "model = models.load_model(modelFileName) # loads the saved model\n",
    "\n",
    "x_new = np.array([[50.4,15.3,20,50]])\n",
    "print ('New sample: {}'.format(x_new))\n",
    "\n",
    "class_probabilities = model.predict(x_new)\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "\n",
    "print(penguin_classes[predictions[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "This notebook was designed to help you understand the basic concepts and principles involved in deep neural networks, using a simple Tensorflow example. To learn more about Tensorflow, take a look at the <a href=\"https://www.tensorflow.org/\" target=\"_blank\">Tensorflow web site</a>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}