HW9.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "HW9.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ArtsiomStrok/google-colaboratory/blob/master/HW9.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "DvRcQIxISJ30",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **Homework 9: Variational Autoencoders**"
      ]
    },
    {
      "metadata": {
        "id": "tJQ_deIUSca8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **About**\n",
        "\n",
        "### **Due**\n",
        "\n",
        "Monday 4/24/19, 11:59 PM CST\n",
        "\n",
        "### **Goal**\n",
        "\n",
        "This homework focuses on creating variational autoencoders applied to the MNIST dataset."
      ]
    },
    {
      "metadata": {
        "id": "d_TOu0F9Sze-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Dev Environment\n",
        "### Working on Google Colab\n",
        "You may choose to work locally or on Google Colaboratory. You have access to free compute through this service. \n",
        "1. Visit https://colab.research.google.com/drive \n",
        "2. Navigate to the **`Upload`** tab, and upload your `HW9.ipynb`\n",
        "3. Now on the top right corner, under the `Comment` and `Share` options, you should see a `Connect` option. Once you are connected, you will have access to a VM with 12GB RAM, 50 GB disk space and a single GPU. The dropdown menu will allow you to connect to a local runtime as well.\n",
        "\n",
        "**Notes:** \n",
        "* **If you do not have a working setup for Python 3, this is your best bet. It will also save you from heavy installations like `tensorflow` if you don't want to deal with those.**\n",
        "* ***There is a downside*. You can only use this instance for a single 12-hour stretch, after which your data will be deleted, and you would have redownload all your datasets, any libraries not already on the VM, and regenerate your logs**.\n",
        "\n",
        "\n",
        "### Installing PyTorch and Dependencies\n",
        "\n",
        "The instructions for installing and setting up PyTorch can be found at https://pytorch.org/get-started/locally/. Make sure you follow the instructions for your machine. For any of the remaining libraries used in this assignment:\n",
        "* We have provided a `hw8_requirements.txt` file on the homework web page. \n",
        "* Download this file, and in the same directory you can run `pip3 install -r hw8_requirements.txt`\n",
        "\n",
        "Check that PyTorch installed correctly by running the following:"
      ]
    },
    {
      "metadata": {
        "id": "YB1d0Rm6SWiB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "torch.rand(5, 3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4dLgW4qRTEqj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The output should look something like\n",
        "\n",
        "```python\n",
        "tensor([[0.3380, 0.3845, 0.3217],\n",
        "        [0.8337, 0.9050, 0.2650],\n",
        "        [0.2979, 0.7141, 0.9069],\n",
        "        [0.1449, 0.1132, 0.1375],\n",
        "        [0.4675, 0.3947, 0.1426]])\n",
        "```\n",
        "\n",
        "### Let's get started with the assignment."
      ]
    },
    {
      "metadata": {
        "id": "ysPO_A4iTJvm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Instructions**\n",
        "##**(NOTE: THESE ARE JUST GUIDELINES. YOU ARE NOT REQUIRED TO EXACTLY FOLLOW THIS FORMAT)**\n",
        "\n",
        "### **Part 1 - Datasets and Dataloaders**\n",
        "\n",
        "This part of the assignment is similar to HW 8. \n",
        "\n",
        "**Create a directory named hw9_data with the following command.**"
      ]
    },
    {
      "metadata": {
        "id": "VFBEkk5aT0Xy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!mkdir hw9_data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3UK59qgDT7pU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "**Now use `torch.datasets.MNIST` to load the Train and Test data into `hw9_data`.** \n",
        "* ** Use the directory you created above as the `root` directory for your datasets**\n",
        "* ** Populate the `transformations` variable with any transformations you would like to perform on your data.** (Hint: You will need to do at least one)\n",
        "* **Pass your `transformations` variable to `torch.datasets.MNIST`. This allows you to perform arbitrary transformations to your data at loading time.**"
      ]
    },
    {
      "metadata": {
        "id": "acNAlKImT7Ta",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torchvision import datasets, transforms\n",
        "\n",
        "## YOUR CODE HERE ##\n",
        "transformations = None\n",
        "mnist_train = None \n",
        "mnist_test = None"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jH35P-FcULyr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Any file in our dataset will now be read at runtime, and the specified transformations we need on it will be applied when we need it.**. \n",
        "\n",
        "We could iterate through these directly using a loop, but this is not idiomatic. PyTorch provides us with this abstraction in the form of `DataLoaders`. The module of interest is `torch.utils.data.DataLoader`. \n",
        "\n",
        "`DataLoader` allows us to do lots of useful things\n",
        "* Group our data into batches\n",
        "* Shuffle our data\n",
        "* Load the data in parallel using `multiprocessing` workers\n",
        "\n",
        "**Use `DataLoader` to create a loader for the training set and one for the testing set**\n",
        "* **Use a `batch_size` of 32 to start, you may change it if you wish.**\n",
        "* **Set the `shuffle` parameter to `True`.** \n",
        "\n",
        "**Check that the data was loaded successfully before proceeding to the next sections. **"
      ]
    },
    {
      "metadata": {
        "id": "q3DO5eriUhZF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "## YOUR CODE HERE ##\n",
        "train_loader = None\n",
        "test_loader = None"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EccGF9s3UgOm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Part 2 - Encoder and Decoders (0 points)**\n",
        "\n",
        "In this section we will be creating the encoder and decoder for our variational autoencoder (VAE). \n",
        "\n",
        "You can take a look at the following to understand how VAE's work. \n",
        "\n",
        "*   https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf\n",
        "*  http://kvfrans.com/variational-autoencoders-explained/\n",
        "*  https://jmetzen.github.io/2015-11-27/vae.html\n",
        "\n",
        "VAEs work around a latent space who's dimension can be chosen by us. We will leave this as a parameter for the Encoder and Decoder classes that you will have to populate. \n",
        "\n",
        "Feel free to use any network architecture that you wish. Try simpler network structures like a few linear layers before trying anything more complicated. \n",
        "\n",
        "### For the Encoder:\n",
        "\n",
        "*   **Finish the __init__() function.**\n",
        "*  **Finish the forward() function.** \n",
        "*  **Assume that input to forward, x, is of shape (batch_size, 28,28)**\n",
        "*  **forward() should return two tensors of size latent_dim like a standard encoder of a VAE**\n",
        "* **One of the tensors should correspond to the mean of the encoding and the other tensor should correspond to the variance. In practice, it is easier to model the output as the log of the variance (logvar) and we will too**\n",
        "\n",
        "### For the Decoder:\n",
        "\n",
        "*   **Finish the __init__() function.**\n",
        "*  **Finish the forward() function.** \n",
        "*  **Assume that input to forward, x, is of shape (batch_size, latent_dim)**\n",
        "*  **forward() should return a tensor of shape (batch_size, 28,28)**\n",
        "* **Make sure that the output lies in the same range as the input to the encoder (Hint: Sigmoid?)**\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "NL4wDY2TYVGP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch import nn\n",
        "class Encoder(nn.Module):\n",
        "  def __init__(self, latent_dim):\n",
        "    super(Encoder, self).__init__()\n",
        "    ## YOUR CODE HERE ##\n",
        "  \n",
        "  def forward(self, x):\n",
        "    ## YOUR CODE HERE ##\n",
        "    return \n",
        "\n",
        "class Decoder(nn.Module):\n",
        "  def __init__(self, latent_dim):\n",
        "    super(Decoder, self).__init__()\n",
        "    ## YOUR CODE HERE ##\n",
        "    \n",
        "  def forward(self,x):\n",
        "    ## YOUR CODE HERE ##\n",
        "    return "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "I9Su2FOMZG93",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Part 3: Training and loss functions** (0 points)\n",
        "\n",
        "Recall that the encoder outputs the mean (mu) and the log of the variance (logvar). This implies that the latent vector of the input image follows a gaussian distribution with mean (mu) and standard deviation (e^[0.5\\*logvar]). To decode this information, the decoder needs a sample from this distribution. \n",
        "\n",
        "**Complete the sample function to generate these samples **"
      ]
    },
    {
      "metadata": {
        "id": "0-w9oxDddxSc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def sample(mu, logvar):\n",
        "  ## YOUR CODE HERE ##\n",
        "  return"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4wUHpN7hd-6h",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We also need to create the loss function. Assume that x are your input images and x_hat are your reconstructions of these input images, complete the following loss for a VAE. (Hint: You will need to use mu and logvar as well)"
      ]
    },
    {
      "metadata": {
        "id": "7esviVxoecxX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def vae_loss(x, x_hat, mu, logvar):\n",
        "  ## YOUR CODE HERE ##\n",
        "  # MSE LOSS + KL DIVERGENCE \n",
        "  return"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m5T1tOI0exar",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the following we will instantiate an Encoder and Decoder with a latent dimension of 32.\n",
        "\n",
        "We also define a single optimizer that optimizes the parameters of both the Encoder and the Decoder together. Feel free to use any optimizer of your choice. "
      ]
    },
    {
      "metadata": {
        "id": "0JGv_rs2fJkA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch import optim\n",
        "\n",
        "## YOUR CODE HERE ##\n",
        "encoder = None\n",
        "decoder = None\n",
        "params = list(encoder.parameters())+list(decoder.parameters())\n",
        "optimizer = optim.Adam(params, lr=1e-3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RwJkFq5cfn0w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Complete the train function that takes input encoder, decoder, train_loader, optimizer, and number of epochs you wish to train your model for.\n",
        "\n",
        "Training will involve:\n",
        "\n",
        "1.   **One epoch is defined as a full pass of your dataset through your model. We choose the number of epochs we wish to train our model for.**\n",
        "2.   **For each batch, use the encoder to generate the mu and logvar.**\n",
        "3. **Sample a latent vector for each image in the batch and feed this to the decoder to generate the decoded images.**\n",
        "4. **Calculate the loss function for this batch.**\n",
        "5. **Now calculate the gradients for each parameter you are optimizing over. (Hint: Your loss function object can do this for you)**\n",
        "6. **Update your model parameters (Hint: The optimizer comes in here)**\n",
        "7. ** Set the gradients in your model to zero for the next batch.**\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "F17NVLO8hkP8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train(encoder, decoder, train_loader, optimizer, num_epochs = 10):\n",
        "  pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dLIjuvFkhuxH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally call train with the relevant parameters.\n",
        "\n",
        "Note : This function may take a while to complete if you're training for many epochs on a cpu. This is where it comes in handy to be running on Google Colab, or just have a GPU on hand."
      ]
    },
    {
      "metadata": {
        "id": "puFTbp4Nh2Hb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train(encoder, decoder, train_loader, optimizer, num_epochs = 10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-ml3F2uSh7-p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Part 4: Visualizing the VAE output** (90 points)\n",
        "\n",
        "We will look at how well the codes produced by the VAE can be interpolated. **For this section we will only use the MNIST test set. **\n",
        "\n",
        "To create an interpolation between two images A and B, we encode both these images and generate a sample code for each of them. We now consider 7 equally spaced points in between these two sample codes giving us a total of 9 points including the samples. We then decode these images to get interpolated images in between A and B.\n",
        "\n",
        "Complete the interpolation function below that takes a pair of images A and B and returns 9 images. (You are free to use any data structure you want to return these images)"
      ]
    },
    {
      "metadata": {
        "id": "JRE_LDjNjnAX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from torchvision import utils\n",
        "%matplotlib inline\n",
        "import numpy as np\n",
        "\n",
        "def create_interpolates(A, B, encoder, decoder):\n",
        "  ## YOUR CODE HERE ##\n",
        "  return "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uCBT8zvEk_zn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**For 10 pairs of MNIST test images of the same digit (1 pair for \"0\", 1 pair for \"1\", etc.), selected at random, compute the code for each image of the pair. Now compute 7 evenly spaced linear interpolates between these codes, and decode the result into images. Prepare a figure showing this interpolate. Lay out the figure so each interpolate is a row. On the left of the row is the first test image; then the interpolate closest to it; etc; to the last test image. You should have a 10 rows (1 row per digit) and 9 columns (7 interpolates + 2 selected test images) of images. (45 points)**"
      ]
    },
    {
      "metadata": {
        "id": "zoaMBIu0jABd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "similar_pairs = {}\n",
        "for _, (x, y) in enumerate(test_loader):\n",
        "  for i in range(len(y)):\n",
        "    if y[i].item() not in similar_pairs:\n",
        "      similar_pairs[y[i].item()] = []\n",
        "    if len(similar_pairs[y[i].item()])<2:\n",
        "      similar_pairs[y[i].item()].append(x[i])\n",
        "  \n",
        "  done = True\n",
        "  for i in range(10):\n",
        "    if i not in similar_pairs or len(similar_pairs[i])<2:\n",
        "      done = False\n",
        "  \n",
        "  if done:\n",
        "    break\n",
        "\n",
        "# similar_pairs[i] contains two images indexed at 0 and 1 that have images of the digit i\n",
        "\n",
        "## YOUR CODE HERE ##"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "k4q7-6jTlQj9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**For 10 pairs of MNIST test images of different digits selected at random, compute the code for each image of the pair. Now compute 7 evenly spaced linear interpolates between these codes, and decode the result into images. Prepare a figure showing this interpolate. Lay out the figure so each interpolate is a row. On the left of the row is the first test image; then the interpolate closest to it; etc; to the last test image. You should have a 10 rows and 9 columns of images. (45 points)**"
      ]
    },
    {
      "metadata": {
        "id": "iKgDR1L-lU11",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "## YOUR CODE HERE ##"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}