C4AIScholarsChallenge-2022.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/varun-ml/cohere-scholars-application/blob/main/C4AIScholarsChallenge-2022.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fRUrAHlZKdWd"
      },
      "source": [
        "# C4AI Scholars Program Takehome Challenge\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PY4wvFv6lTbY"
      },
      "source": [
        "### Background\n",
        "\n",
        "Welcome to the C4AI Scholars Program Take-Home Challenge! This exercise is designed to allow you to showcase your engineering and problem solving skills. The Challenge consists of three parts:\n",
        "\n",
        "* Part One of the challenge requires identifying bugs, and getting the code working. This is designed to test your ability to grapple with real world engineering challenges.\n",
        "* Part Two of the challenge tests your ability to generate code for a specified problem. \n",
        "* Part Three of the challenge is an opportunity for you to attempt an optional challenge question that extends the original problem set. \n",
        "\n",
        "\n",
        "These tasks were chosen as a setting to see how you think about problems, even if they are not in your own research field of interest. The tasks and dataset are not meant to be indicative of the research goals of the Scholar Program. We purposefully have selected a simple toy problem so the focus is on how you think, and does not require significant machine learning resources. \n",
        "\n",
        "Good luck! If you have questions about the framing of the questions, please contact info@for.ai  \n",
        "\n",
        "\n",
        "### How to Use and Submit this Document\n",
        "\n",
        "* **Make a copy of this document** and **rename** it **Firstname_Lastname_C4AIScholarsChallenge**\n",
        "* Once you’ve completed all tasks, **save and pin your revisions**\n",
        "* **Submit a link** to your final document via the [Cohere For AI Scholars Program application](https://jobs.lever.co/cohere/?department=Cohere%20For%20AI). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LmbPmOHhLx4U"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qc-enDOhazoe"
      },
      "source": [
        "## Overview of Singular Value Decomposition\n",
        "\n",
        "In this takehome, you will be working on a problem involving singular value decomposition. Singular Value Decomposition (SVD) exists for every rectangular matrix. The nice thing about SVD is that the original matrix can be expressed as the sum of outer products of left and singular vectors scaled by the corresponding singular values. Formally:\n",
        "\n",
        "> Let 𝛢 be a rectangular matrix of dimensions 𝑚𝘹𝑛, then the SVD of the matrix A is given by $ A = U𝛴V^T$ where $U$ is an orthogonal matrix of shape mxm containing the left singular vectors, $V$ is an orthogonal matrix of shape nxn containing the right singular vectors and $𝛴$ is a diagonal matrix containing the singular values of $A$. This formulation of SVD can be re-expressed as \\begin{align} A = \\sum_{i=1}^{r} s_i. u_i v_i^T \\end{align} where $r = \\text{min}(m,n)$ represents the rank of the matrix, $s_i$ is the $i$th singular value and $u_i v_i^T$ is the outer product of the $i$th left and right singular vectors. \n",
        "\n",
        "<!-- \\begin{align}\n",
        "A = \\sum_{i=1}^{\\text{min}(m,n)} s_i. u_i v_i^T\n",
        "\\end{align}\n",
        "\\begin{align} -->\n",
        "\n",
        "> The singular values $𝛴$ are decreasing in order. So, each outer product is scaled by a smaller value as we compute each term in the sum above. This gives us an opportunity to approximate $A$ using only the sum of the first $k$ outer products where $k < \\text{min}(m,n)$ $-$ this effectively means that we are zero-ing out some of the singular values by assuming that the contribution to the sum is negligible. This is called low-rank approximation.\n",
        "\n",
        "If you aren't familiar with singular value decomposition, or the above feels rusty, don't worry. Take a moment to brush up your knowledge using any of the following resources:\n",
        "* [stanford lecture notes on low rank approximations](https://web.stanford.edu/class/cs168/l/l9.pdf)\n",
        "* [youtube series of short and beginner friendly lectures](https://www.youtube.com/watch?v=gXbThCXjZFM&list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AcL7u9wJjPF3"
      },
      "source": [
        "## Check for understanding (3 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-v-iHTH3nasL"
      },
      "source": [
        "#### Q1: What are some real world applications of low rank approximations?\n",
        "\n",
        "\n",
        "#### Answer:\n",
        "\n",
        "\n",
        "1. Deep Neural Network compression: Compression of deep neural networks helps in reducing the size of the neural network and reducing computations to improve inference time for the network. Cost benefits and lower environmental impact from lower compute needs.\n",
        "2. Image Lossy Compression: Low-rank decompositions help in generating lossy compression of images, which are good approximations for the original image\n",
        "3. Collaborative filtering in recommender systems: We make predictions about a user's interests by collating information about the claims of other users.   \n",
        "4. Image denoising: Low-rank approximations are used to remove noise to generate a better-quality image\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UHGqpn3tkFRL"
      },
      "source": [
        "#### Q2: What are the benefits of compressing a deep neural network? How would you measure the benefits of compression?\n",
        "\n",
        "\n",
        "#### Answer:\n",
        "\n",
        "\n",
        "Benefits of compressing a deep neural network\n",
        "*   Reduction in size of the network due to fewer parameters. This results in a smaller memory footprint and allows models to run on resource-constrained machines. This makes these models accessible to a larger audience\n",
        "*   Improvement in inference time due to fewer computations.\n",
        "*   Reduced energy requirements to run a model, resulting in lower costs and less environmental impact\n",
        "* Reduction in training time, models with half-precision take considerably less time to train\n",
        " \n",
        "Measure benefits due to compression\n",
        "*   Model size can be measured. It should be considerably less compared to the original model. This is similar to measuring the number of parameters in the networks\n",
        "*   Counting the number of flops to make an inference, or measuring average inference time to run the model\n",
        "*   Measuring compute costs for a model in production\n",
        " \n",
        " \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9r4v5vng9ma"
      },
      "source": [
        "#### Q3: In this takehome, we will consider how singular value decomposition can be used to compress a deep neural network. Compared to other compression methods used for deep neural networks such as pruning, quantization, or efficient architectures, what are the relative merits/demerits of low rank approximations? Choose one or two alternative compression methods and compare with singular value decomposition.\n",
        "\n",
        "#### Answer:\n",
        " \n",
        "Merits:\n",
        " \n",
        " \n",
        "*   Cheap way to compress an over-parameterized model: decomposition is typically done layer by layer. In addition to computing the decomposition matrices, we simply need to redefine the model architecture with the new weights\n",
        "*   SVD low-rank approximation doesn't need retraining, fine-tuning may improve performance\n",
        "*   Generates models that have good performance with a smaller number of parameters and a lower number of floating point operations although they will have more layers. Especially, in the case of SVD when the number of singular values chosen is << then the rank of the matrix\n",
        "*   Smaller models can be deployed to resource-constrained machines without any need for specialized support\n",
        " \n",
        "Demerits:\n",
        "*   Approach cannot be used to improve training time, we need a pre-trained over-parameterized model, to begin with\n",
        "*   Only linear Deep Neural Network layers like convolution (tucker and cp decomposition) and fully connected layers benefit, algorithms ignore the nonlinearity that comes after them \n",
        "*   SVD decomposition works layer by layer ignoring interactions between layers. In that sense, they are greedy and maybe sub-optimal\n",
        "---\n",
        " \n",
        "Comparing SVD with Pruning and efficient architectures\n",
        " \n",
        " \n",
        "* Efficient architectures are the go-to technique when we need to train a model from scratch. Unlike, SVD and Pruning we don't require a pre-trained model\n",
        " \n",
        "* Pruning typically requires multiple forward and backward passes on the model using the data to effectively perform model compression\n",
        " \n",
        "* Low-rank approximations seem like the fastest way to achieve a compressed model, we just need to redefine the model arch in a specific way and compute the decompositions.\n",
        " \n",
        "* Pruning and efficient architecture for a problem achieve model compression by learning from the data in a global sense, as opposed to SVD which greedily picks up a layer and does the decomposition\n",
        " \n",
        "* Most Pruning algorithms need specialized support to improve inference time, currently libraries' support for sparse tensor operations seems limited. Libraries like PyTorch use masking at runtime to implement pruning but this will not help the runtime performance of the model\n",
        " \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O81HH6D3Lugd"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xoWy4-iQIYJ0"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wBFa6wXqAKgK",
        "outputId": "a856d0cc-d6b7-44cc-ae1a-38c7ad8456e1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting dm-haiku\n",
            "  Downloading dm_haiku-0.0.8-py3-none-any.whl (350 kB)\n",
            "\u001b[K     |████████████████████████████████| 350 kB 27.3 MB/s \n",
            "\u001b[?25hCollecting optax\n",
            "  Downloading optax-0.1.3-py3-none-any.whl (145 kB)\n",
            "\u001b[K     |████████████████████████████████| 145 kB 63.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from dm-haiku) (4.1.1)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from dm-haiku) (0.8.10)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from dm-haiku) (1.21.6)\n",
            "Collecting jmp>=0.0.2\n",
            "  Downloading jmp-0.0.2-py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from dm-haiku) (1.3.0)\n",
            "Requirement already satisfied: jaxlib>=0.1.37 in /usr/local/lib/python3.7/dist-packages (from optax) (0.3.22+cuda11.cudnn805)\n",
            "Collecting chex>=0.0.4\n",
            "  Downloading chex-0.1.5-py3-none-any.whl (85 kB)\n",
            "\u001b[K     |████████████████████████████████| 85 kB 5.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: jax>=0.1.55 in /usr/local/lib/python3.7/dist-packages (from optax) (0.3.23)\n",
            "Requirement already satisfied: dm-tree>=0.1.5 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.4->optax) (0.1.7)\n",
            "Requirement already satisfied: toolz>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.4->optax) (0.12.0)\n",
            "Requirement already satisfied: opt-einsum in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->optax) (3.3.0)\n",
            "Requirement already satisfied: etils[epath] in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->optax) (0.8.0)\n",
            "Requirement already satisfied: scipy>=1.5 in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->optax) (1.7.3)\n",
            "Requirement already satisfied: zipp in /usr/local/lib/python3.7/dist-packages (from etils[epath]->jax>=0.1.55->optax) (3.10.0)\n",
            "Requirement already satisfied: importlib_resources in /usr/local/lib/python3.7/dist-packages (from etils[epath]->jax>=0.1.55->optax) (5.10.0)\n",
            "Installing collected packages: jmp, chex, optax, dm-haiku\n",
            "Successfully installed chex-0.1.5 dm-haiku-0.0.8 jmp-0.0.2 optax-0.1.3\n"
          ]
        }
      ],
      "source": [
        "!pip install dm-haiku optax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wzdlf_milUbR"
      },
      "outputs": [],
      "source": [
        "from typing import Iterator, Mapping, Tuple\n",
        "from copy import deepcopy\n",
        "import time\n",
        "from absl import app\n",
        "import haiku as hk\n",
        "import matplotlib.pyplot as plt\n",
        "import jax\n",
        "import jax.numpy as jnp\n",
        "import numpy as np\n",
        "import optax\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "from functools import partial\n",
        "import math\n",
        "\n",
        "Batch = Tuple[np.ndarray, np.ndarray]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsYJmUqz-uFT"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EjTGxEfqnSmd"
      },
      "source": [
        "## Coding Challenge Part I : Debugging Challenge (10 Points)\n",
        "\n",
        "\n",
        "We are now going to explore using SVD to compute low rank approximations of the parameters of a small deep neural network. You are using a very simple toy model as a first baseline. Section 3 will give you the chance to improve baseline accuracy beyond this very simple model -- this is just a toy setting to first explore low rank approximations.\n",
        "\n",
        "The first part of this challenge is primarily a debugging challenge. It will require removing bugs in order to train a very simple network. We have introduced several bugs -- some are subtle and will not break your code but will degrade final performance. These subtle bugs are introduced to understand your grasp of fundamental machine learning principles. There are also more obvious bugs designed to break your code. \n",
        "\n",
        "* [**4 points**] Your goal is to get the code working. There are 4 bugs in the code, all 4 of these are subtle bugs which are designed to impair test accuracy but not break the code. You will get partial points for each of the 4 bugs you find. After finding all bugs, your test performance should be around 66-67% test accuracy. \n",
        "\n",
        "* [**2 points**] We will give extra points for also adding improved documentation to each of the functions we introduce in this section, and for describing the fixes to the bugs. \n",
        "\n",
        "* [**4 points**] There are also two functions you will need to code up in this section -- we indicate where these code changes need to happen with TODO comments. \n",
        "\n",
        "* Do not alter the model architecture or the learning rate.\n",
        "\n",
        "\n",
        "Useful tips:\n",
        "* To iterate faster and avoid training for 10000 steps each time you want to test whether you have found all the bugs, a good sign you have caught the bugs is wheter after 1000/10000 steps your accuracy >40%.\n",
        "* The colab difftool is useful to track what code you have changed during the debugging challenge (incase you need to revert code). You can access this via tools > diff notebooks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IqpgWQJs-evw"
      },
      "outputs": [],
      "source": [
        "CIFAR10_MEAN = (0.4914, 0.4822, 0.4465)\n",
        "CIFAR10_STD = (0.2023, 0.1994, 0.2010)\n",
        "\n",
        "\n",
        "def net_fn(batch: Batch) -> jnp.ndarray:\n",
        "  \"\"\"Defines the network\"\"\"\n",
        "  x = normalize(batch[0])\n",
        "  \n",
        "  # Do NOT alter the architecture definition below.\n",
        "  net = hk.Sequential([\n",
        "      hk.Conv2D(output_channels=6*3, kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      jax.nn.relu,\n",
        "      hk.Conv2D(output_channels=16*3, kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      hk.Flatten(),\n",
        "      hk.Linear(3000), jax.nn.relu,\n",
        "      hk.Linear(2000), jax.nn.relu,\n",
        "      hk.Linear(2000), jax.nn.relu,\n",
        "      hk.Linear(1000), jax.nn.relu,\n",
        "      hk.Linear(10),\n",
        "  ])\n",
        "  return net(x)\n",
        "\n",
        "def load_dataset(\n",
        "    split: str,\n",
        "    *,\n",
        "    is_training: bool,\n",
        "    batch_size: int,\n",
        ") -> Iterator[tuple]:\n",
        "  \"\"\"Loads the dataset as a generator of batches.\"\"\"\n",
        "  # cache the dataset to memory and repeat it indefinitely, epochs have reshuffled ordered data since *reshuffle_each_iteration* defaults to TRUE\n",
        "  ds = tfds.load('cifar10', split=split, as_supervised=True).cache().repeat()\n",
        "  if is_training:\n",
        "    # shuffle the batches to improve generalization of the trained network\n",
        "    ds = ds.shuffle(10 * batch_size, seed=0)\n",
        "  ds = ds.batch(batch_size)\n",
        "  return iter(tfds.as_numpy(ds))\n",
        "\n",
        "# Recommended: this can be jit'd and it should result in faster training time \n",
        "# @jax.jit\n",
        "def compute_loss(params: hk.Params, batch: Batch) -> jnp.ndarray:\n",
        "  \"\"\"Compute the loss of the network, including L2.\"\"\"\n",
        "  x, y = batch\n",
        "  logits = net.apply(params, batch)\n",
        "  labels = jax.nn.one_hot(y, 10)\n",
        "\n",
        "  # DONE: TODO: add code below to compute the l2_loss variable\n",
        "  param_values = jax.tree_util.tree_leaves(params)\n",
        "  l2_loss = sum(jnp.sum(jnp.square(value)) for value in param_values)\n",
        "\n",
        "  weighted_l2_loss = 0.5 * l2_loss\n",
        "\n",
        "  # NOT A BUG: usually we take mean of the loss, but since the loss is usually adjusted by a scaling factor it's effectively equivalent\n",
        "  # I am not taking mean of the loss here, since there are some scale issues between l2 loss and cross entropy loss that need to be adjusted \n",
        "  softmax_xent = -jnp.sum(jnp.multiply(labels, jax.nn.log_softmax(logits)))\n",
        "\n",
        "  # BUG 1: l2 loss should be added to the xent loss instead of being subtracted, \n",
        "  # BUG 2: we should be using the weighted l2 loss here\n",
        "  # subtracting will encourage network params with large l2 norms and would impair training progress\n",
        "  return softmax_xent + (1e-4 * weighted_l2_loss) \n",
        "\n",
        "\n",
        "@jax.jit\n",
        "def compute_accuracy(params: hk.Params, batch: Batch) -> jnp.ndarray:\n",
        "  \"\"\"Compute prediction accuracy for a batch of data\"\"\"\n",
        "  # using the global. the correct net gets binded to the params. \n",
        "  # Every time the shape of the params change, it triggers a JIT compile and JIT binds the net function\n",
        "  # https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#pure-functions\n",
        "  predictions = net.apply(params, batch)\n",
        "\n",
        "  # DONE:: TODO: add code below to compute the accuracy over the batch.\n",
        "  _, y = batch\n",
        "  prediction_label = jnp.argmax(predictions, axis=1)\n",
        "  # using the jax.lax equals method here to avoid recompilations \n",
        "  # https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#structured-control-flow-primitives\n",
        "  accuracy = jnp.mean(jnp.equal(prediction_label,y))\n",
        "  \n",
        "  return accuracy\n",
        "\n",
        "@jax.jit\n",
        "def update(\n",
        "    params: hk.Params,\n",
        "    opt_state: optax.OptState,\n",
        "    batch: Batch,\n",
        ") -> Tuple[hk.Params, optax.OptState]:\n",
        "  \"\"\"Compute gradients and update the weights\"\"\"\n",
        "  grads = jax.grad(compute_loss)(params, batch)\n",
        "  updates, opt_state = opt.update(grads, opt_state)\n",
        "  new_params = optax.apply_updates(params, updates)\n",
        "  return new_params, opt_state\n",
        "\n",
        "@jax.jit\n",
        "def ema_update(params, avg_params):\n",
        "  \"\"\"Incrementally update parameters via polyak averaging.\"\"\"\n",
        "  # Polyak averaging tracks an (exponential moving) average of the past parameters of a model, for use at test/evaluation time.\n",
        "  return optax.incremental_update(params, avg_params, step_size=0.001)\n",
        "\n",
        "\n",
        "def normalize(images):\n",
        "  # numpy array creation could be moved out of the method\n",
        "  mean = np.asarray(CIFAR10_MEAN)\n",
        "  std = np.asarray(CIFAR10_STD)\n",
        "\n",
        "  # BUG 3: int8 range is -128 to 127 causing issues with numbers > 127, uint8 / 255. is float64, casting as float32 since that's the type jax prefers\n",
        "  x = images.astype(jnp.float32) / 255.\n",
        "\n",
        "  # BUG 4: x /- mean just divides x by -1. * mean without reassignment to x\n",
        "  x -= mean  # operations uses numpy broadcasting to work correctly\n",
        "  x /= std \n",
        "\n",
        "  return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d5iI930cIjzM"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 339,
          "referenced_widgets": [
            "125cb4682d91435fa63ccc69f4272ce5",
            "0dd7df3cdac14cb49cb90fa53b5aa1a0",
            "155e0e12d8b74a0caaecb45cfe42c909",
            "41f10b9116eb470b92300c54b8599bb9",
            "5c5ec9ae3120432f81163593d0fca729",
            "a3165c2388df4f6c9be17799387613b3",
            "a14169df4a08430fb539d23bcafd3bf3",
            "482e2a1f2e5a4a45bb9710a74109801e",
            "7acf4ea181764f26a53c9517665b40a5",
            "792e93d718f44308a107428d02d087e6",
            "c16f5fd1ee9c46ffbd4cf70e5a76a3d9",
            "5dddc88f52974a6c9f89ccc293378a0b",
            "78b3d6c661da4f3e829a47a9a3982f96",
            "672b454d76444221a214aa2ecc56ce53",
            "b31d0cc11b414f98ac562fbf2a5ab32b",
            "024994efa9a94649a6e8c40c8775149d",
            "7011fcb71fb54224bbf2ca3017e7cfe5",
            "fc22a3537aa04be58497faad28cadf2a",
            "47552858457e4ae788ffcc7aceeebee3",
            "adf8d3c7221c49d59179994b64386281",
            "0163b13eaaae4617a19509ceff3bd339",
            "f265fc5b189443b5bc26788cc170dd95",
            "489dab16ca0f4b47a26098dabe6656e3",
            "63ee59a41b4d4f4387e88433f62f40e1",
            "73cd458fd3684e80a73271a67ce4c449",
            "7215c69ac4fa40e0923efce59fa963d4",
            "a367896ef0e74e3095acc55e2252e2b4",
            "d5d77ab171df40d5b7793f407367861b",
            "136fefb48f1b46f594a436a76e64c6f6",
            "328d5b4cdda0436bb479f86ac82680ce",
            "0800eb1c3b264aa78d1b9143a51adcb6",
            "12a35d35359f43d89abba53fed35181d",
            "4e07817836074afd95ffa0ce13d887c0",
            "1499954e23304688985f8624b34f9fd7",
            "a3d410be51dd4db3ac73d23488e9af47",
            "6784e2dad4b54f78ae53e52262376c83",
            "69fabfeab44e4569b3f8bcc48c6bf004",
            "7dddbd5b30844d6fbda588ac199b6edb",
            "cae30c2ce3ea4dac80ae6b974ce8f033",
            "ee43e3f18dc64dfea3fc251ad3a0a34e",
            "b54c15bf12f64c099555958a0ded00b9",
            "d24658c56af34cfbb6d9817ed2381444",
            "d374df00d7f54d6d8b6374fa1a139828",
            "08995c1f4e31419d8a8f0535bef8ef1a",
            "d053418d04674749a98afa17e91aa6b1",
            "f166794fe7ef47b49f32b8ce294c06db",
            "92b96ac0af404aa7ac29febd6601a26c",
            "c31d6f41c7d34cd78a879d56abd7610d",
            "9770e4ce645d46b296de1cfaf7e7fb0a",
            "084440fb123d4b168377878cfa3d8589",
            "c33964ccf95543038947138a2459a80d",
            "d24b46dc02df4d22b143ddbb90845067",
            "c44bc25746cf4c02aa7ed95b04139782",
            "d5dd43bb0bc7425385df5685af62d440",
            "2e70dbfca6d7459d8f0ed687a5d76e6c",
            "486b13e0d6e6403a982e60aad25b0210",
            "3a1643d595f4419a9727fb131ef4d548",
            "1625117f98854c4b87eafe9ea4ea7893",
            "7fc0f0022c3a49c7a60736ddadb26015",
            "6165ea42c4f84caf96298900496e7e78",
            "67d15cc85fa942b6859d301c70bbbcf4",
            "0e3740e900704125b30d31fd2e1abb47",
            "c55bacbdc65d49269c628495573b46cd",
            "9209390e1b0a482db1b62101ddb1c306",
            "4cd9c329f7b847d086b222e4febae367",
            "fa72b99270734b3c88bbb850724e9acf",
            "a9f21872578242b4973c308a3674264a",
            "4ae45fedfe884787a2aa673e2428b77e",
            "c0a1dd36d5d046f7af9726ea39151f46",
            "eca2a4226b23426b8f6984795a35abfa",
            "a1264146ed274496aaa68d35cf9adde6",
            "1bde4bf945f14497a869854243124f60",
            "28fa5c3c59234abf99872abd1e8c4678",
            "92ddfb58a7084ed3be1ebf9294712bde",
            "d11803d4b0134aee947e6557dc70d5db",
            "cd532039d76349fab7142f50a5794317",
            "58b954b4db2c43ef9c7a986803cc1f19",
            "cc30e5efb79b4c9cb35aaf5126eafd66",
            "bcb105a141df4adf9156720173f7ead9",
            "f7253ab1829546139269cc957db0d7b1",
            "b87a8542a970482584a1460ceb2b4595",
            "03fcf0536efd456a906202273a24debc",
            "307965ca273f4b0e89c55ccfa590fce0",
            "04231c29fa4c48d78cecb7dd23ef4725",
            "5dd8d88a1c434cc0b1fb17d41da471a7",
            "b50999734a6845b5bc7b67b26c2e3336",
            "66c12a744e12466db36dcc83766278c7",
            "d39ed98855aa41a3b220556a353d2a9b"
          ]
        },
        "id": "Q4-HuMSH_Cbw",
        "outputId": "3aaf6db4-a606-41c2-b2ba-c8104cf041bc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1mDownloading and preparing dataset 162.17 MiB (download: 162.17 MiB, generated: 132.40 MiB, total: 294.58 MiB) to ~/tensorflow_datasets/cifar10/3.0.2...\u001b[0m\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "125cb4682d91435fa63ccc69f4272ce5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Dl Completed...: 0 url [00:00, ? url/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5dddc88f52974a6c9f89ccc293378a0b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Dl Size...: 0 MiB [00:00, ? MiB/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "489dab16ca0f4b47a26098dabe6656e3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extraction completed...: 0 file [00:00, ? file/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1499954e23304688985f8624b34f9fd7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating splits...:   0%|          | 0/2 [00:00<?, ? splits/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d053418d04674749a98afa17e91aa6b1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train examples...:   0%|          | 0/50000 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "486b13e0d6e6403a982e60aad25b0210",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Shuffling ~/tensorflow_datasets/cifar10/3.0.2.incompleteH429IC/cifar10-train.tfrecord*...:   0%|          | 0/…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a9f21872578242b4973c308a3674264a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating test examples...:   0%|          | 0/10000 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "cc30e5efb79b4c9cb35aaf5126eafd66",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Shuffling ~/tensorflow_datasets/cifar10/3.0.2.incompleteH429IC/cifar10-test.tfrecord*...:   0%|          | 0/1…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1mDataset cifar10 downloaded and prepared to ~/tensorflow_datasets/cifar10/3.0.2. Subsequent calls will reuse this data.\u001b[0m\n",
            "[Step 0] Validation / Test accuracy: 0.101 / 0.103.\n",
            "[Step 1000] Validation / Test accuracy: 0.468 / 0.469.\n",
            "[Step 2000] Validation / Test accuracy: 0.605 / 0.612.\n",
            "[Step 3000] Validation / Test accuracy: 0.619 / 0.625.\n",
            "[Step 4000] Validation / Test accuracy: 0.624 / 0.630.\n",
            "[Step 5000] Validation / Test accuracy: 0.621 / 0.630.\n",
            "[Step 6000] Validation / Test accuracy: 0.623 / 0.630.\n",
            "[Step 7000] Validation / Test accuracy: 0.625 / 0.628.\n",
            "[Step 8000] Validation / Test accuracy: 0.625 / 0.627.\n",
            "[Step 9000] Validation / Test accuracy: 0.623 / 0.627.\n",
            "[Step 10000] Validation / Test accuracy: 0.623 / 0.626.\n"
          ]
        }
      ],
      "source": [
        "net = hk.without_apply_rng(hk.transform(net_fn))\n",
        "\n",
        "# Do not change learning rate\n",
        "opt = optax.adam(1e-3)\n",
        "\n",
        "# using 20% of the dataset for training, 80% for validation, although it's weird to use maximum available data for validation\n",
        "# BUG: using 80% train data, since performance improvement beyond 50% wasn't seen with 20% train data \n",
        "train = load_dataset(\"train[:80%]\", is_training=True, batch_size=1000)\n",
        "validation = load_dataset(\"train[80%:]\", is_training=False, batch_size=10000)\n",
        "\n",
        "test = load_dataset(\"test\", is_training=False, batch_size=10000)\n",
        "\n",
        "# Alternative: supplying a dummy data instead of actual data for Haiku to init the network\n",
        "# dummy_data = jnp.ones([100, 32, 32, 3]), 1\n",
        "\n",
        "# Recommended: param objects should be deepcopied, although in this case it doesn't cause issues\n",
        "# we draw an input to get shapes.\n",
        "params = net.init(jax.random.PRNGKey(42), next(train))\n",
        "avg_params = deepcopy(params)\n",
        "opt_state = opt.init(params)\n",
        "\n",
        "# Do not alter the number of steps\n",
        "for step in range(10001):\n",
        "  if step % 1000 == 0:\n",
        "    val_accuracy = compute_accuracy(avg_params, next(validation))\n",
        "    test_accuracy = compute_accuracy(avg_params, next(test))\n",
        "    val_accuracy, test_accuracy = jax.device_get(\n",
        "        (val_accuracy, test_accuracy))    \n",
        "    print(f\"[Step {step}] Validation / Test accuracy: \"\n",
        "          f\"{val_accuracy:.3f} / {test_accuracy:.3f}.\")\n",
        "    \n",
        "  params, opt_state = update(params, opt_state, next(train))\n",
        "  avg_params = ema_update(params, avg_params)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QBWQhbV-MPQA"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQmOspeD0jCh"
      },
      "source": [
        "## Coding Challenge Part 2: Compression through Low Rank Approximation (8 points)\n",
        "\n",
        "In this section, you will add code to compute the low rank approximation and to compute evaluation metrics. We will evaluate whether the low rank approximation allows for speed up in inference time. We define inference time as the average time to compute the prediction for all examples in the test set.\n",
        "\n",
        "* [**4 points**] You will need to add code to define both the compute_eval_metrics and rank_approximated weight function. \n",
        "* [**4 points**] Q4 and Q5 are worth 2 points each."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sdZqO7W_KSX8"
      },
      "outputs": [],
      "source": [
        "import timeit\n",
        "\n",
        "@jax.jit\n",
        "def apply_(params: hk.Params, batch: Batch) -> jnp.ndarray:\n",
        "  \"\"\"Get predictions from the network\"\"\"\n",
        "  # this function will get compiled only once for Q2 but will compile 10 times for Q3 and Q4. Each new net function triggers recompilation\n",
        "  # https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#pure-functions\n",
        "\n",
        "  # net is a global\n",
        "  predictions = net.apply(params, batch)  \n",
        "  return predictions\n",
        "\n",
        "@jax.jit\n",
        "def accuracy_(predictions: jnp.ndarray, y: jnp.ndarray) -> jnp.ndarray:\n",
        "  \"\"\"Compute accuracy from the predictions\"\"\"\n",
        "  prediction_label = jnp.argmax(predictions, axis=1)\n",
        "  accuracy = jnp.mean(jnp.equal(prediction_label, y))\n",
        "\n",
        "  return accuracy\n",
        "\n",
        "def compute_accuracy(params: hk.Params, batch: Batch) -> Tuple[jnp.ndarray, jnp.ndarray]:\n",
        "  # adding timer to this method\n",
        "  start = timeit.default_timer()\n",
        "  # adding block until ready to measure correct time\n",
        "  predictions = apply_(params, batch).block_until_ready()\n",
        "  duration = timeit.default_timer() - start\n",
        "\n",
        "  # DONE:: TODO: add code below to compute the accuracy over the batch.\n",
        "  _, y = batch\n",
        "  accuracy = accuracy_(predictions, y)\n",
        "  \n",
        "  return accuracy, duration\n",
        "\n",
        "def compute_eval_metrics(params, batch, n_samples):\n",
        "# DONE: TODO: add code to compute the time for inference.\n",
        "  duration_list = []\n",
        "  accuracy_list = []\n",
        "  for _ in range(n_samples):\n",
        "    # DONE: Added above, TODO: add code to correctly compute the accuracy on a given batch.\n",
        "    acc, duration = compute_accuracy(params, batch)\n",
        "    duration_list.append(duration)\n",
        "    accuracy_list.append(acc)\n",
        "\n",
        "  return accuracy_list, duration_list"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X8U8Nlp9IS4q"
      },
      "outputs": [],
      "source": [
        "def rank_approximated_weight(weight: jnp.ndarray, rank_fraction: float):\n",
        "  U, a, V = jnp.linalg.svd(weight)\n",
        "  # DONE: TODO: replace the code below with code to compute the SVD of the matrix to return the rank approximated weights u and v for a given matrix.\n",
        "  # u = jax.random.normal(jax.random.PRNGKey(42), shape=weight.shape)\n",
        "  # size = weight.shape[1]\n",
        "  # v = jax.random.normal(jax.random.PRNGKey(42), shape=(size,size))\n",
        "  m, n = weight.shape\n",
        "\n",
        "  if n > m:\n",
        "    rank = m\n",
        "  else:\n",
        "    # default case where m >= n\n",
        "    rank = n\n",
        "\n",
        "  k = (rank * rank_fraction).astype(int)\n",
        "  diag_matrix = jnp.diag(a[:k])\n",
        "  V = diag_matrix@V[:k, :]\n",
        "\n",
        "  return U[:, :k], V"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvm9InR4JG4F"
      },
      "source": [
        "### Evaluations at different ranks\n",
        "\n",
        "The code below first replaces the weights with the low rank factorizations at different rank fractions. For each modified net, we compute the new eval accuracy. Firstly, add code for the rank_approximated_weight and add code to correctly compute the time for inference (the duration)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PEjLxaDPEGEY",
        "outputId": "6f1eefe0-c212-4f7a-ae95-c3417455acd9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Evaluating the model at 1.0\n",
            "Rank Fraction / Test accuracy: 1.00 / 0.626.\n",
            "Rank Fraction / Duration (milli sec): 1.00 / 214.8488.\n",
            "Evaluating the model at 0.9\n",
            "Rank Fraction / Test accuracy: 0.90 / 0.626.\n",
            "Rank Fraction / Duration (milli sec): 0.90 / 209.3016.\n",
            "Evaluating the model at 0.8\n",
            "Rank Fraction / Test accuracy: 0.80 / 0.625.\n",
            "Rank Fraction / Duration (milli sec): 0.80 / 208.5936.\n",
            "Evaluating the model at 0.7000000000000001\n",
            "Rank Fraction / Test accuracy: 0.70 / 0.614.\n",
            "Rank Fraction / Duration (milli sec): 0.70 / 209.3766.\n",
            "Evaluating the model at 0.6000000000000001\n",
            "Rank Fraction / Test accuracy: 0.60 / 0.617.\n",
            "Rank Fraction / Duration (milli sec): 0.60 / 209.3382.\n",
            "Evaluating the model at 0.5000000000000001\n",
            "Rank Fraction / Test accuracy: 0.50 / 0.616.\n",
            "Rank Fraction / Duration (milli sec): 0.50 / 209.0742.\n",
            "Evaluating the model at 0.40000000000000013\n",
            "Rank Fraction / Test accuracy: 0.40 / 0.581.\n",
            "Rank Fraction / Duration (milli sec): 0.40 / 208.7761.\n",
            "Evaluating the model at 0.30000000000000016\n",
            "Rank Fraction / Test accuracy: 0.30 / 0.513.\n",
            "Rank Fraction / Duration (milli sec): 0.30 / 209.1387.\n",
            "Evaluating the model at 0.20000000000000018\n",
            "Rank Fraction / Test accuracy: 0.20 / 0.343.\n",
            "Rank Fraction / Duration (milli sec): 0.20 / 208.8565.\n",
            "Evaluating the model at 0.1000000000000002\n",
            "Rank Fraction / Test accuracy: 0.10 / 0.186.\n",
            "Rank Fraction / Duration (milli sec): 0.10 / 208.7439.\n"
          ]
        }
      ],
      "source": [
        "# BUG: we should be copying avg_params and not params\n",
        "params = deepcopy(avg_params)\n",
        "rank_truncated_params = deepcopy(params)\n",
        "ranks_and_accuracies = []\n",
        "ranks_and_times = []\n",
        "for rank_fraction in np.arange(1.0, 0.0, -0.1):\n",
        "\n",
        "  print(f\"Evaluating the model at {rank_fraction}\")\n",
        "  for layer in params.keys():\n",
        "    if 'conv' in layer:\n",
        "      continue\n",
        "    weight = params[layer]['w']\n",
        "    # DONE: TODO: complete coding the rank_approximated_weight function to compute the SVD of the matrix to return the rank approximated weights u and v for a given matrix.\n",
        "    u, v = rank_approximated_weight(weight, rank_fraction)\n",
        "    rank_truncated_params[layer]['w'] = u@v\n",
        "\n",
        "  test_batch = next(test)\n",
        "  # we compute metrics over 50 samples to reduce noise in the measurement.\n",
        "  n_samples = 50\n",
        "  # DONE: TODO: complete coding the compute_eval_metrics function to compute the time taken for inference.\n",
        "  test_accuracy, latency = compute_eval_metrics(rank_truncated_params, next(test), n_samples)\n",
        "  print(f\"Rank Fraction / Test accuracy: \"\n",
        "          f\"{rank_fraction:.2f} / {np.mean(test_accuracy):.3f}.\")\n",
        "  ranks_and_accuracies.append((rank_fraction, np.mean(test_accuracy)))\n",
        "  print(f\"Rank Fraction / Duration (milli sec): \"\n",
        "          f\"{rank_fraction:.2f} / {np.mean(latency)*1000:.4f}.\")\n",
        "  ranks_and_times.append((rank_fraction, np.mean(latency)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzyvqIr38eWw"
      },
      "source": [
        "### Q4: What do you observe as the relationship between rank fraction and test accuracy?\n",
        "\n",
        "Plot this relationship showing accuracy (y-axis) vs rank percentage of the matrix (x-axis). You should use the ranks_and_accuracies list computed above.\n",
        "\n",
        "Answer: The test accuracy is comparable to accuracy on the full network and starts significantly dropping as the rank fraction dips below 0.5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "TCdYJ6lSEKM9",
        "outputId": "bf0a7d32-709e-4876-a83e-71fd3ba1733d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Accuracy')"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVTklEQVR4nO3dfbRldX3f8fdnZmTBDUSazmgVmDs0awihJvHhlpj8EQ1CilohrU0KvdaQWicmoqlPS9Yal7W4WK24Yrva0Oo1i2rjNQgmtWOCslIKJSVCGeRBHoKOhIEhrjoaosFRefr2j71v5sz13plzGfY5Z85+v9Y665z9O7979nftubM/d+/fPr+dqkKS1F/rxl2AJGm8DAJJ6jmDQJJ6ziCQpJ4zCCSp5zaMu4C12rhxY23ZsmXcZUjSEeXWW2/9RlVtWum9Iy4ItmzZws6dO8ddhiQdUZLsXu09Tw1JUs8ZBJLUcwaBJPWcQSBJPWcQSFLPGQSStIrFRdiyBdata54XF6ezDoNA0sSZhB3w4iJs2wa7d0NV87xt2+hrGUUdOdKmoZ6bmyu/RyBNr6Ud3759+9tmZmBhAebnR1fHli3NTne52Vl44IEjr44kt1bV3IrvGQSSJsmk7IDXrWv+Al8ugaeeOvLqOFgQeGpImgCTcCpkUjz44Nrau7J589raj+Q6DAL13rh3wpNyLnpSTMoO+JJLmlNSg2Zmmvapq6OqjqjHS17ykpKeKZ/4RNXMTFWzC24eMzNN+6jMzh64/qXH7OzoaljyiU80602a51Fuh8Eaxv1vMljLuLfHM1UHsLNW2a86RqBem4Tz0ZNyLnpSBmmXatm+vTkdtHlz89fvqGuYNg4WS6uYhJ3wJITRJNWhbjhYLK1iEs5HT8q56EkZpNXoGQTqtUnYCc/PN6dfZmebI5HZ2fGcjpmEUNR4GATqtUnZCc/PN6dfnnqqeR7H+fBJCEWNR6dBkOTsJPcl2ZXkolX6/HKSe5LcneSTXdYjrWQSdsKTYFJCUaPX2WBxkvXAl4GzgD3ALcD5VXXPQJ+twJXAGVX1SJLnVNXXD/a5DhZL0tqNa7D4dGBXVd1fVY8BVwDnLuvzRuCyqnoE4FAhIEl65nUZBCcADw0s72nbBp0CnJLkxiQ3JTl7pQ9Ksi3JziQ79+7d21G5ktRP4x4s3gBsBV4OnA98NMnxyztV1UJVzVXV3KZNm0ZcoiRNty6D4GHgpIHlE9u2QXuAHVX1eFX9Oc2YwtYOa5IkLdNlENwCbE1ycpKjgPOAHcv6fIbmaIAkG2lOFd3fYU2SpGU6C4KqegK4ELgGuBe4sqruTnJxknPabtcA30xyD3Ad8K6q+mZXNUmSfpBzDUlSDzjXkCbSuO8DIKmxYdwFqJ+WT3m8dDMW8Jus0qh5RKCx2L79wHnvoVnevn089Uh9ZhBoLJzyWJocBoHGwimPpclhEGgsnPJYmhwGgcbCKY+lyeFVQxqb+Xl3/NIk8IhAknrOIJCknjMIJKnnDAJJ6jmDQJJ6ziCQpJ4zCCSp5wwCSeo5g0CSes4gkKSeMwgkqecMAknqOYNAknrOIJCknjMIJKnnDAJJ6jmDQJJ6ziCQpJ4zCCSp5wwCSeo5g0CSes4gkKSeMwh6aHERtmyBdeua58XFcVckaZw2jLsAjdbiImzbBvv2Ncu7dzfLAPPz46tL0vh4RNAz27fvD4El+/Y17ZL6ySDomQcfXFu7pOlnEPTM5s1ra5c0/QyCnrnkEpiZObBtZqZpl9RPnQZBkrOT3JdkV5KLVnj/giR7k9zePv5ll/WoGRBeWIDZWUia54UFB4qlPuvsqqEk64HLgLOAPcAtSXZU1T3Lun6qqi7sqg79oPl5d/yS9uvyiOB0YFdV3V9VjwFXAOd2uD5J0tPQZRCcADw0sLynbVvutUnuTPLpJCet9EFJtiXZmWTn3r17u6hVknpr3IPFnwW2VNVPAn8MfHylTlW1UFVzVTW3adOmkRYoSdOuyyB4GBj8C//Etu1vVNU3q+r77eLvAC/psB5J0gq6DIJbgK1JTk5yFHAesGOwQ5LnDSyeA9zbYT2SpBV0dtVQVT2R5ELgGmA9cHlV3Z3kYmBnVe0A3prkHOAJ4C+BC7qqR5K0slTVuGtYk7m5udq5c+e4y5CkI0qSW6tqbqX3xj1YLEkaM4NAknrOIJCknjMIJKnnDAJJ6jmDQJJ6ziCQpJ4zCCSp5wwCSeo5g0CSes4gkKSeMwgkqecMAknqOYNAknrOIJCknjMIJKnnDhkESV6TxMCQpCk1zA7+nwJfSXJpklO7LkiSNFqHDIKqeh3wIuCrwMeSfCHJtiTHdV6dJKlzQ53yqapvA58GrgCeB/wj4ItJ3tJhbZKkERhmjOCcJP8duB54FnB6Vb0S+CngHd2WJ0nq2oYh+rwW+PdVdcNgY1XtS/KGbsqSJI3KMEHwPuBrSwtJjgGeW1UPVNW1XRUmSRqNYcYIrgKeGlh+sm2TJE2BYYJgQ1U9trTQvj6qu5IkSaM0TBDsTXLO0kKSc4FvdFeSJGmUhhkjeBOwmOS3gQAPAa/vtCpJ0sgcMgiq6qvAS5Mc2y4/2nlVkqSRGeaIgCSvBv4ecHQSAKrq4g7rkiSNyDBfKPswzXxDb6E5NfRLwGzHdUmSRmSYweKfrarXA49U1b8BfgY4pduyJEmjMkwQfK993pfk+cDjNPMNSZKmwDBjBJ9NcjzwQeCLQAEf7bQqSdLIHDQI2hvSXFtVfwX8fpI/BI6uqm+NpDpJUucOemqoqp4CLhtY/r4hIEnTZZgxgmuTvDZL141KkqbKMEHwazSTzH0/ybeT/HWSbw/z4UnOTnJfkl1JLjpIv9cmqSRzQ9YtSXqGDPPN4qd1S8ok62lOK50F7AFuSbKjqu5Z1u844DeBm5/OeiRJh+eQQZDk51ZqX36jmhWcDuyqqvvbz7kCOBe4Z1m/9wMfAN51yGolSc+4YS4fHdxBH02zg78VOOMQP3cCzQR1S/YAPz3YIcmLgZOq6o+SrBoESbYB2wA2b948RMmSpGENc2roNYPLSU4C/sPhrri9NPVDwAVD1LAALADMzc3V4a5bkrTfMIPFy+0BfnyIfg8DJw0sn9i2LTkOeAFwfZIHgJcCOxwwlqTRGmaM4D/RfJsYmuB4Ic03jA/lFmBrkpNpAuA84J8tvdl+H2HjwHquB95ZVTuHLV6SdPiGGSMY3DE/AfxeVd14qB+qqieSXAhcA6wHLq+qu5NcDOysqh1Pq2JJ0jNqmCD4NPC9qnoSmstCk8xU1b5D/WBVXQ1cvaztvav0ffkQtUiSnmFDfbMYOGZg+Rjgf3ZTjiRp1IYJgqMHb0/Zvp7priRJ0igNEwTfaa/3ByDJS4DvdleSJGmUhhkj+FfAVUn+guZWlX+H5taVkqQpMMwXym5JcirwY23TfVX1eLdlSZJGZZib178Z+KGququq7gKOTfIb3ZcmSRqFYcYI3tjeoQyAqnoEeGN3JUmSRmmYIFg/eFOadnrpo7orSZI0SsMMFn8e+FSSj7TLvwZ8rruSJEmjNEwQvJtmCug3tct30lw5JEmaAoc8NdTewP5m4AGaexGcAdzbbVmSpFFZ9YggySnA+e3jG8CnAKrq50dTmiRpFA52aujPgD8B/mFV7QJI8raRVCVJGpmDnRr6x8DXgOuSfDTJK2i+WSxJmiKrBkFVfaaqzgNOBa6jmWriOUn+S5JfGFWBkqRuDTNY/J2q+mR77+ITgdtoriSSJE2BNd2zuKoeqaqFqnpFVwVNs8VF2LIF1q1rnhcXx12RJA33PQI9AxYXYds22Nfe12337mYZYH5+fHVJ0pqOCPT0bd++PwSW7NvXtEvSOBkEI/Lgg2trl6RRMQhGZPPmtbVL0qgYBCNyySUws+xOzzMzTbskjZNBMCLz87CwALOzkDTPCwsOFEsaP68aGqH5eXf8kiaPRwSS1HMGgST1nEEgST1nEEhSzxkEktRzBoEk9ZxBIEk9ZxBIUs8ZBJLUcwaBJPWcQSBJPWcQSFLPdRoESc5Ocl+SXUkuWuH9NyX5UpLbk/yfJKd1WY8k6Qd1FgRJ1gOXAa8ETgPOX2FH/8mq+omqeiFwKfChruqRJK2syyOC04FdVXV/VT0GXAGcO9ihqr49sPhDQHVYjyRpBV3ej+AE4KGB5T3ATy/vlOTNwNuBo4AzVvqgJNuAbQCbvbejJD2jxj5YXFWXVdWPAu8G3rNKn4WqmququU2bNo22QEmacl0GwcPASQPLJ7Ztq7kC+MUO65EkraDLILgF2Jrk5CRHAecBOwY7JNk6sPhq4Csd1iNJWkFnYwRV9USSC4FrgPXA5VV1d5KLgZ1VtQO4MMmZwOPAI8CvdFWPJGllnd68vqquBq5e1vbegde/2eX6JUmHNvbBYknSeBkEktRzBoEk9ZxBIEk9ZxBIUs8ZBJLUcwaBJPWcQSBJPWcQSFLPGQSS1HMGgST1nEEgST1nEEhSzxkEktRzBoEk9ZxBIEk9ZxBIUs8ZBJLUcwaBJPWcQSBJPWcQSFLPGQSS1HMGgST1nEEgST1nEEhSzxkEktRzBoEk9ZxBIEk9ZxBIUs8ZBJLUcwaBJPWcQSBJPWcQSFLPGQSS1HMGgST1XKdBkOTsJPcl2ZXkohXef3uSe5LcmeTaJLNd1iNJ+kGdBUGS9cBlwCuB04Dzk5y2rNttwFxV/STwaeDSruqRJK2syyOC04FdVXV/VT0GXAGcO9ihqq6rqn3t4k3AiR3WI0laQZdBcALw0MDynrZtNW8APtdhPZKkFWwYdwEASV4HzAEvW+X9bcA2gM2bN4+wMkmafl0eETwMnDSwfGLbdoAkZwLbgXOq6vsrfVBVLVTVXFXNbdq0qZNiJamvugyCW4CtSU5OchRwHrBjsEOSFwEfoQmBr3dYiyRpFZ0FQVU9AVwIXAPcC1xZVXcnuTjJOW23DwLHAlcluT3JjlU+7rAsLsKWLbBuXfO8uNjFWiTpyNTpGEFVXQ1cvaztvQOvz+xy/dDs9Ldtg33ttUm7dzfLAPPzXa9dkibf1H+zePv2/SGwZN++pl2S1IMgePDBtbVLUt9MfRCsdrWpV6FKUmPqg+CSS2Bm5sC2mZmmXZLUgyCYn4eFBZidhaR5XlhwoFiSlkzEN4u7Nj/vjl+SVjP1RwSSpIMzCCSp5wwCSeo5g0CSes4gkKSeS1WNu4Y1SbIX2D3uOg7TRuAb4y5igrg99nNbHMjtcaDD2R6zVbXiPP5HXBBMgyQ7q2pu3HVMCrfHfm6LA7k9DtTV9vDUkCT1nEEgST1nEIzHwrgLmDBuj/3cFgdyexyok+3hGIEk9ZxHBJLUcwaBJPWcQdChJGcnuS/JriQXrfD+25Pck+TOJNcmmR1HnaNwqG0x0O+1SSrJVF8yOMz2SPLL7e/H3Uk+OeoaR2mI/yubk1yX5Lb2/8urxlHnKCS5PMnXk9y1yvtJ8h/bbXVnkhcf9kqrykcHD2A98FXg7wJHAXcApy3r8/PATPv614FPjbvucW2Ltt9xwA3ATcDcuOse8+/GVuA24G+1y88Zd91j3h4LwK+3r08DHhh33R1uj58DXgzctcr7rwI+BwR4KXDz4a7TI4LunA7sqqr7q+ox4Arg3MEOVXVdVe1rF28CThxxjaNyyG3Rej/wAeB7oyxuDIbZHm8ELquqRwCq6usjrnGUhtkeBfxw+/rZwF+MsL6RqqobgL88SJdzgf9WjZuA45M873DWaRB05wTgoYHlPW3bat5Ak/LT6JDboj28Pamq/miUhY3JML8bpwCnJLkxyU1Jzh5ZdaM3zPZ4H/C6JHuAq4G3jKa0ibTWfcsh9eIOZZMuyeuAOeBl465lHJKsAz4EXDDmUibJBprTQy+nOVK8IclPVNVfjbWq8Tkf+FhV/VaSnwF+N8kLquqpcRc2DTwi6M7DwEkDyye2bQdIciawHTinqr4/otpG7VDb4jjgBcD1SR6gOe+5Y4oHjIf53dgD7Kiqx6vqz4Ev0wTDNBpme7wBuBKgqr4AHE0zAVsfDbVvWQuDoDu3AFuTnJzkKOA8YMdghyQvAj5CEwLTfA74oNuiqr5VVRuraktVbaEZLzmnqnaOp9zOHfJ3A/gMzdEASTbSnCq6f5RFjtAw2+NB4BUASX6cJgj2jrTKybEDeH179dBLgW9V1dcO5wM9NdSRqnoiyYXANTRXRVxeVXcnuRjYWVU7gA8CxwJXJQF4sKrOGVvRHRlyW/TGkNvjGuAXktwDPAm8q6q+Ob6quzPk9ngH8NEkb6MZOL6g2ktopk2S36P5I2BjOybyr4FnAVTVh2nGSF4F7AL2Ab962Ouc0m0pSRqSp4YkqecMAknqOYNAknrOIJCknjMIJKnnDAId8ZI8meT2JHcl+WyS4w/jsx4dos9bk9ybZPHprmfgsy5I8vyB5d9Jctrhfq60Fl4+qiNekker6tj29ceBL1fVJYf7WQfp82fAmVW1Z1n7hqp6Yo3rux545xR/eU5HAI8ING2+QDsBV5LTk3yhncP+T5P8WNt+QZI/SPL5JF9JcunyD0mysf3ZVy9r/zDNdMmfS/K2JO9L8rtJbqSZ/2ZLkj9J8sX28bMDP/vuJF9KckeSf5fkn9DMMbXYHtEck+T6pak1kpzf9r8ryQcGPufRJJe0n3NTkue27b/U9r0jyQ3P9IbVFBv33Ns+fBzuA3i0fV4PXAWc3S7/MLChfX0m8Pvt6wtopmt4Ns1UBbtpZj4FeBR4LnAzcNYq63sA2Ni+fh9wK3BMuzwDHN2+3krzzViAVwJ/yv77T/xI+3w9A/deWFoGnk8zrcImmhkA/hfwi22fAl7Tvr4UeE/7+kvACe3r48f97+LjyHk4xYSmwTFJbqc5ErgX+OO2/dnAx5Nspdl5PmvgZ66tqm8BtNM4zNJM7fss4FrgzVX1v4dc/46q+m77+lnAbyd5Ic3UEKe07WcC/7Xa+09U1cHmmwf4+8D1VbW3rXGR5oYlnwEeA/6w7XcrcFb7+kbgY0muBP5gyNolTw1pKny3ql5IszMP8Oa2/f3AdVX1AuA1NH/9Lxmc6fVJ9s+79QTNzvUfrGH93xl4/Tbg/wE/RfOX/VFr+JxhPV5VS4N7f1N7Vb0JeA/NzJS3JvnbHaxbU8gg0NRo/9p+K/COJBtojgiWpue9YNiPAf4FcGqSdz+NMp4NfK2aefL/Oc3pKmiOUn41yQxAkh9p2/+aZhru5f4v8LJ2rGI9zXz8Bz1CSfKjVXVzVb2XZmbOkw7WX1piEGiqVNVtwJ00O85LgX+b5DbWMNNuVT3Z/vwZSX5jjSX8Z+BXktwBnEp7tFBVn6eZPnhnexrrnW3/jwEfXhosHqjha8BFwHU09/C9tar+xyHW/cGlwWWa8Yg71li7esrLRyWp5zwikKSeMwgkqecMAknqOYNAknrOIJCknjMIJKnnDAJJ6rn/D7r6x7jiqExmAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "## DONE: TODO: add your code below to plot the relationship between time and test set accuracy\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "x = [x for x, _ in ranks_and_accuracies]\n",
        "y = [y for _, y in ranks_and_accuracies]\n",
        "\n",
        "plt.plot(x, y, 'bo')\n",
        "plt.xlabel(\"Rank fractions\")\n",
        "plt.ylabel(\"Accuracy\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rRkI8rAO5UYe"
      },
      "source": [
        "### Q5: Does replacing the weight matrix with the low factor matrix result in latency speed ups?\n",
        "\n",
        "Plot the relationship of time (y-axis) vs rank percentage (x-axis). To do so add code to compute the ranks_and_times list.\n",
        "\n",
        "Answer: No, Latency is similar to latency on the full matrix, this is because the number of multiplications required to do inference remains the same in this implementation. The latency at rank fraction 1 is marginally higher becuase of jit comilation, which happens when the first inference is done. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "f7jlMYxhi7E-",
        "outputId": "78361f40-7482-4c41-ad1c-4ae0a7002840"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Time taken for infence in milli-seconds')"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAdvklEQVR4nO3dfZQdVZ3u8e8DBhQTiEhkeEnSoIBG9Iq3B0G8ekUFBwElite5AQW5ZEZRQdAFii/oDEvFC3fpODPQYxTUOAoDaBABIwQZEJjphCAJEUflRTRqFCSBKBDy3D+qutNpu8+pfjmnTnc/n7XOOlV1dlX9utI5v961d+0t20RERABsU3cAERHROZIUIiKiX5JCRET0S1KIiIh+SQoREdHvaXUHMBa77LKLu7q66g4jImJCWb58+e9szxrqswmdFLq6uujt7a07jIiICUXS/cN91rLbR5JmS1om6W5JqyWdWm4/tlzfLKl7QPkuSX+UtLJ8Xdiq2CIiYmitrClsAs6wvULSDGC5pKXAKmA+cNEQ+/zM9ktaGFNERDTQsqRgey2wtlzeIGkNsIftpQCSWnXqiIgYpbb0PpLUBRwA3N6k6F6S7pD0A0n/Y5hjLZTUK6l33bp14xxpRMTU1vKkIGk6cDlwmu31DYquBebYPgA4Hfi6pB0HF7LdY7vbdvesWUM2nkdExCi1NClImkaREBbbvqJRWduP2/59ubwc+Bmwbyvji4iYaBYvhq4u2Gab4n3x4vE9fsvaFFQ0GiwC1ti+oEL5WcBDtp+StDewD/DzVsUXETHRLF4MCxfCxo3F+v33F+sACxaMzzlaWVM4BDgeOHRAN9MjJB0j6UHgYOBqSdeV5V8J/EjSSuDfgL+1/VAL44uImFDOPntLQuizcWOxfby0svfRzcBwXYyuHKL85RS3miIiYggPPDCy7aORsY8iIiaIOXNGtn00khQiIiaIc8+FHXbYetsOOxTbx0uSQkTEBLFgAfT0wNy5IBXvPT3j18gME3xAvIiIqWbBgvFNAoM1rSlIOk/SjpKmSbpe0jpJx7UupIiIqEuV20eHlU8iHwncBzwP+GArg4qIiHpUSQp9t5jeAFxm+5EWxhMRETWq0qbwHUk/Bv4IvKt88vhPrQ0rIiLq0LSmYPss4OVAt+0ngY3AG1sdWEREtN+wNQVJ84fYNnC14QB3EREx8TS6fXRU+f4ciprCDeX6q4EfkqQQETHpDJsUbJ8IIOl7wLxyJjUk7QZc3JboIiKirar0PprdlxBKvwHGcaSNiIjoFFV6H11fDm/9r+X6/wK+37qQIiKiLk2Tgu33lI3OfXMm99j+s6GvIyJi4qs09lE5lWYaliMiJrkqYx/Nl/Rfkh6RtF7SBknr2xFcRES0V5WawnnAUbbXtDqYiIioV5XeR79JQoiImBqq1BR6JX0T+BbweN/Gsp0hIiImkSpJYUeK8Y4OG7DNpOE5ImLSqdIl9cR2BBIREfWr0vtoT0lXSvpt+bpc0p7tCC4iItqrSkPzl4ElwO7l66pyW0OSZktaJuluSaslnVpuP7Zc3yype4j95kh6VNIHRvajRETEWFVJCrNsf9n2pvJ1MTCrwn6bgDNszwMOAk6RNA9YBcwHbhpmvwuAayocPyIixlmVpPB7ScdJ2rZ8HQf8vtlOttfaXlEubwDWAHvYXmP7nqH2kfQm4F5gdfUfISIixkuVpPBO4K3Ar4G1wFuAETU+S+oCDgBub1BmOnAm8Ikmx1ooqVdS77p160YSRkRENFGl99H9wNGjPUH5ZX85cJrtRsNjnAP8P9uPDprhbXA8PUAPQHd3t0cbV0RE/LkqvY8ukTRzwPqzJH2pysElTaNICIsrPOz2MuA8SfcBpwEflvSeKueJiIjxUeXhtRfb/kPfiu2HJR3QbCcVf+4vAtbYvqBZedt9Q3Mj6RzgUdtfqBBfRESMkyptCttIelbfiqSdqZZMDgGOBw6VtLJ8HSHpGEkPAgcDV5cT+ERERAeo8uV+PnCrpMvK9WOBc5vtZPtmYLjGgYaT9Ng+p0JcERExzqo0NH9FUi9waLlpvu27WxtWRETUocrtI4CdgcfKe/zrJO3VwpgiIqImVXoffZzi+YEPlZumAV9rZVAREVGPKjWFYyieU3gMwPavgBmtDCoiIupRJSk8YdsUcygg6ZmtDSkiIupSJSlcKukiYKakk4HvA//S2rAiIqIOVXof/V9JrwPWA/sBH7O9tOWRRURE2zVNCuXtohtsL5W0H7CfpGm2n2x9eBER0U5Vbh/dBGwvaQ/gWoqnlC9uZVAREVGPKklBtjdSTIzzz7aPBV7Y2rAiIqIOlZKCpIOBBcDV5bZtWxdSRETUpUpSOJXiwbUrba+WtDewrLVhRUREHar0PrqJcj5lSX9h++fA+1odWEREtF/VsY/6fLclUUREREcYaVIYfp7MiIiY8EaaFPIkc0TEJDZsm4KkHW2vL2da6/ONvnXbD7U8uoiIaKtGDc1fB44EllMMhqdB73u3PLqIiGirYZOC7SPL90yoExExRTS6ffTSRjvaXjH+4URERJ0a3T46v8FnZsuczRERMUk0un306nYGEhER9Wt0+2h+ox1tXzH+4URERJ0a3T46qsFnBhomBUmzga8Au5ble2x/TtKxwDnAC4ADbfeW5Q8Eevp2B86xfWWVHyIiIsZHo9tHJ47x2JuAM2yvkDQDWC5pKbCKYhjuiwaVXwV0294kaTfgTklX2d40xjgiIqKiRrePjrP9NUmnD/W57QsaHdj2WmBtubxB0hpgj76pPCUNLr9xwOrTKWoXERHRRo1uHz2zfJ8x1pNI6gIOAG5vUu5lwJeAucDxQ9USJC0EFgLMmTNnrKFFRMQAjW4fXVS+f2IsJ5A0HbgcOM32+kZlbd8OvFDSC4BLJF1j+0+DyvRQtj10d3enNhERMY6azqcgaS/gvUDXwPK2j66w7zSKhLB4JL2VbK+R9CiwP9Bbdb+IiBibpkkB+BawCLgK2Fz1wCoaDRYBa5q1P5Tl9wJ+UTY0zwWeD9xX9XwRETF2VZLCn2x/fhTHPgQ4HrhL0spy24eB7YF/AGYBV0taaftw4BXAWZKepEg+77b9u1GcNyIiRkl249vykv43sA/wPeDxvu2dMPZRd3e3e3tzdykiYiQkLbfdPdRnVWoKL6L4i/9Qttw+ythHERGTUJWkcCywt+0nWh1MRETUq8p0nKuAma0OJCIi6lelpjAT+LGk/2TrNoWmXVIjImJiqZIUPt7yKCIioiM0TQq2f9COQCIion5V2hQiImKKSFKIiIh+SQoREdGvyoB4h1DMlDa3LC/AtvdubWgREdFuVXofLQLeDywHnmptOBERUacqSeER29e0PJKIiKhdlaSwTNJngSvosAHxIiJifFVJCi8r3weOqJcB8SIiJqEqD6+9uh2BRERE/YZNCpKOs/01SacP9XmV2dQiImJiaVRTeGb5PqMdgURERP2GTQq2LyrfP9G+cCIiok55ojkiIvolKURERL8khYiI6Nc0KUjaVdIiSdeU6/MkndT60CIiot2q1BQuBq4Ddi/XfwKc1mwnSbMlLZN0t6TVkk4ttx9brm+W1D2g/OskLZd0V/meh+MiItqsSlLYxfalwGYA25uoNjDeJuAM2/OAg4BTJM0DVgHzgZsGlf8dcJTtFwHvAL5a7UeIiIjxUmWYi8ckPZtiaAskHQQ80mwn22uBteXyBklrgD1sLy2PM7j8HQNWVwPPkLS97ceJiIi2qJIUTgeWAM+VdAswC3jLSE4iqQs4ALi94i5vBlYMlRAkLQQWAsyZM2ckYURERBNVxj5aIelVwH4UE+zcY/vJqieQNB24HDjN9voK5V8IfAY4bJh4eoAegO7ubleNIyIimqvS++gUYLrt1bZXAdMlvbvKwSVNo0gIi21fUaH8nsCVwNtt/6zKOSIiYvxUaWg+2fYf+lZsPwyc3GwnFY0Gi4A1VQbPkzQTuBo4y/YtFeKKiIhxViUpbKsBrcKStgW2q7DfIcDxwKGSVpavIyQdI+lB4GDgaknXleXfAzwP+NiA8s8Z2Y8TERFjUaWh+Vrgm5IuKtf/ptzWkO2bKdoghnLlEOX/Hvj7CvFERESLVEkKZ1IkgneV60uBL7YsooiIqE2V3kebgX8uXxERMYk1TQqSDgHOAeaW5QXY9t6tDS0iItqtyu2jRcD7geVUG94iIiImqCpJ4RHb17Q8koiIqF2VpLBM0meBK4D+YSdsr2hZVBERUYsqSeFl5Xv3gG0GMrR1RMQkU6X30avbEUhERNQvM69FRES/ls28FhERE08rZ16LiIgJpkpSGNXMaxERMfG0Zea1iIiYGIZNCpKOtX0Z8DAw6pnXIiJi4mh0++hD5fvltjf1zbyWhBARMXk1un30e0nfA/aStGTwh7aPbl1YERFRh0ZJ4Q3AS4GvAue3J5yIiKjTsEnB9hPAbZJebntdG2OKiIiaVOl99CxJ5wJdA8vbzthHERGTTJWkcBlwIcUUnHloLSJiEquSFDbZzlScERFTQJUnmq+S9G5Ju0naue/V8sgiIqLtqtQU3lG+f3DANgOZozkiYpJpWlOwvdcQr6YJQdJsScsk3S1ptaRTy+3HluubJXUPKP/ssvyjkr4wth8rIiJGo9EwF4favkHS/KE+t31Fk2NvAs6wvULSDGC5pKXAKmA+cNGg8n8CPgrsX74iIqLNGt0+ehVwA3DUEJ+ZYs7mYdleC6wtlzdIWgPsYXspgKTB5R8Dbpb0vMrRR0TEuGr08NrHy/cTx3oSSV3AAcDt43CshcBCgDlz5oz1cBERMUCV3kdjImk6cDlwmu31Yz2e7R7b3ba7Z82aNfYAIyKiX0uTgqRpFAlhcYU2iIiIqFnDpCBpG0kvH82BVTQaLALW2L5gNMeIiIj2avicgu3Nkv6Roj1gpA4BjgfukrSy3PZhYHvgHyhmcLta0krbhwNIug/YEdhO0puAw2zfPYpzR0TEKFR5eO16SW8GrrDtqge2fTPFTG1DuXKYfbqqHj8iIsZflTaFv6EYFO8JSeslbZA05gbjiIjoPE1rCrZntCOQiIioX5XbR0g6GnhluXqj7e+0LqSIiKhL09tHkj4NnArcXb5OlfSpVgcWERHtV6WmcATwEtubASRdAtwBfKiVgUVERPtVfXht5oDlnVoRSERE1K9KTeFTwB2SllF0MX0lcFZLo4qIiFo0Gjr7ENu3UIyGeiPwl+VHZ9r+dRtii4iINmtUU/g88N+BW22/FFjSnpAiIqIujZLCk5J6gD0lfX7wh7bf17qwIiKiDo2SwpHAa4HDgeXtCSciIurUaJKd3wHfkLTG9p1tjCkiImrStEtqEkJExNTR8pnXIiJi4khSiIiIfk0fXpO0PfBmoGtgedufbF1YERFRhypPNH8beISiB9LjrQ0nIiLqVCUp7Gn79S2PJCIialelTeGHkl7U8kgiIqJ2VWoKrwBOkHQvxe0jAbb94pZGFhERbVclKfxVy6OIiIiOUOXhtfuB2cCh5fLGKvtFRMTEU2U6zo8DZ7JlprVpwNcq7Ddb0jJJd0taLenUcvux5fpmSd2D9vmQpJ9KukfS4SP/cSIiYiyq3D46BjgAWAFg+1eSZlTYbxNwhu0VZfnlkpYCq4D5wEUDC0uaB7wNeCGwO/B9SfvafqryTxMREWNS5TbQE7YNGEDSM6sc2PZa232JZAOwBtjD9hrb9wyxyxuBb9h+3Pa9wE+BA6ucKyIixkeVpHCppIuAmZJOBr4PfHEkJ5HURVHbuL1BsT2AXwxYf7DcNvhYCyX1Supdt27dSMKIiIgmqtw+Op9iXoX1wH7Ax4Cbqp5A0nTgcuA02+tHE+RAtnuAHoDu7m6P9XgREbFFlaSwyPY7gaXQ/yX/XeA1zXaUNI0iISy2fUWT4r+k6OXUZ89yW0REtEmV20e/lPRPAJKeBXyPar2PBCwC1ti+oMJ5lgBvk7S9pL2AfYD/qLBfRESMkyrPKXwUeFTShRQJ4XzbX65w7EOA44FDJa0sX0dIOkbSg8DBwNWSrivPsxq4FLgbuBY4JT2PIiLaS0XHoiE+kOYPXAU+SvGX+7UAFW4HtVx3d7d7e3vrDiMiYkKRtNx291CfNWpTOGrQ+h0UD64dRdE9tfakEBER42vYpGD7xHYGEhER9asy89rTgZMonjR+et/2skdSRERMIlV6H30V+AvgcOAHFF1FN7QyqIiIqEeVpPC8sgfSY7YvAd4AvKy1YUVERB2qJIUny/c/SNof2Al4TutCioiIulR5ormnfGjtIxQPmE2n6J4aERGTTJWkcL3thynGO9oboHziOCIiJpkqt48uH2Lbv413IBERUb9hawqSnk/RDXWnQU8378iArqkRETF5NLp9tB9wJDCTrZ9u3gCc3MqgIiKiHsPePrL97fKp5iNtnzjg9T7bP2xjjBFRg8WLoasLttmmeF+8uO6Ioh2qjJJ6azsCiYgt6v5CXrwYFi6E++8Hu3hfuDCJYSqo0tAcEW3UCV/IZ58NGzduvW3jxmJ7TG5JChEdphO+kB94YGTbp4K6a2/t0jQpSNpV0iJJ15Tr8ySd1PrQIqamTvhCnjNnZNsnu06ovbVLlZrCxcB1wO7l+k+A01oVUMRU1wlfyOeeCzvssPW2HXYotk9FnVB7a5cqSWEX25cCmwFsbwIyTWZEi3TCF/KCBdDTA3PnglS89/QU26eiTqi9tUuVpPCYpGdTzLaGpIOAR1oaVcQU1ilfyAsWwH33webNxftUTQjQGbW3dqmSFE6nGAjvuZJuAb4CvLelUUXbTJXGs4kmX8idpRNqb+3SdEA82yskvYriCWcB99h+ssluMQH0NZ713SvtazyDfAlFDNT3/+Hss4tbRnPmFAlhMv4/ke3GBaRtKSbW6WJAErF9QUsjq6C7u9u9vb11hzFhdXUViWCwuXOLv06nosWLp8Z//JjaJC233T3UZ1WGzr4K+BNwF2Vjc0wOU6nxrIrUnCKqJYU9bb94pAeWNJui/WFXikbqHtufk7Qz8E2Kmsd9wFttP1xO5PMl4LkUSeidtleN9LxR3Zw5Q9cUJmPjWRWNuh0mKcRUUaWh+RpJh43i2JuAM2zPAw4CTpE0DziLYuKefYDry3WADwMrywT0duBzozhnjMBUajyrIjWniGpJ4TbgSkl/lLRe0gZJ65vtZHut7RXl8gZgDbAH8EbgkrLYJcCbyuV5wA1l+R8DXZJ2HdFPEyPSKV0fO8VU6nYYMZwqSeEC4GBgB9s72p5he8eRnERSF3AAcDuwq+215Ue/pri9BHAnML8sfyAwF9hziGMtlNQrqXfdunUjCSOGkK6PW6TmFFEtKfwCWOVm3ZSGIWk6xZSep9neqoZRHrPvuJ8GZkpaSfEcxB0M8eS07R7b3ba7Z82aNZqQIoaUmlNEtYbmnwM3lgPiPd63sUqXVEnTKBLCYttXlJt/I2k322sl7Qb8tjzeeuDEcj8B95bnnpTS9bEzLViQf4eY2qrUFO6laBDeDpgx4NVQ+cW+CFgzKIEsAd5RLr8D+HZZfqak7crt/we4aXDNYrKYSiMuRsTE0vThtVEfWHoF8O9s/XzDhynaFS4F5gD3U3RJfUjSwRQNzwZWAyfZfrjROSbqw2t5aOzPpeYU0T6jenhN0hdsv0fSVWy579/P9tGNTmr7ZophMYbymiHK3wrs2+iY46XuL6B0fdxaHhqL6BzD1hQkrbe9Yznu0Z+x/YOWRlbBaGoKg7+AoOhh0s4GxdQUtpbrEdFejWoKjZLCHbYPaGlkYzSapNAJX0CdkJg6yTbbFG0rg0lFV9mIGF+jHftolqTTh/uwEwbEG41OuHUzlUZcrCLDbUR0jka9j7YFprN1j6PKvY86Vac8tZqHxrbIQ2MRnaNRTWGt7U+2LZI2OffcoW/d5AuoPqk5RSN1dwyZaholheF6Dk1o+QLqTHloLIaSnmnt16iheWfbD7U5nhGZqM8pREQ1ndAxZDJq1NA8bJtCpyeEiJj8OqFjyFRTZZiLiIhadErHkKkkSSEiOlZ6prVfkkJEdKwMZ95+VYbOjoioTXqmtVdqChER0S9JISIi+iUpREREvySFiIjol6QQERH9WjYdZztIWkcxpedEtgvwu7qD6CC5HlvL9dgi12JrY7kec23PGuqDCZ0UJgNJvcONQTIV5XpsLddji1yLrbXqeuT2UURE9EtSiIiIfkkK9eupO4AOk+uxtVyPLXItttaS65E2hYiI6JeaQkRE9EtSiIiIfkkKbSLp9ZLukfRTSWcN8fnpku6W9CNJ10uaW0ec7dLsegwo92ZJljRpuyJWuRaS3lr+fqyW9PV2x9hOFf6vzJG0TNId5f+XI+qIsx0kfUnSbyWtGuZzSfp8ea1+JOmlYz6p7bxa/AK2BX4G7A1sB9wJzBtU5tXADuXyu4Bv1h13ndejLDcDuAm4DeiuO+4afzf2Ae4AnlWuP6fuuGu+Hj3Au8rlecB9dcfdwuvxSuClwKphPj8CuAYQcBBw+1jPmZpCexwI/NT2z20/AXwDeOPAAraX2d5Yrt4G7NnmGNup6fUo/R3wGeBP7Qyuzapci5OBf7T9MIDt37Y5xnaqcj0M7Fgu7wT8qo3xtZXtm4CHGhR5I/AVF24DZkrabSznTFJojz2AXwxYf7DcNpyTKLL/ZNX0epTV4Nm2r25nYDWo8ruxL7CvpFsk3Sbp9W2Lrv2qXI9zgOMkPQh8F3hve0LrSCP9bmkqM691GEnHAd3Aq+qOpS6StgEuAE6oOZRO8TSKW0j/k6IGeZOkF9n+Q61R1eevgYttny/pYOCrkva3vbnuwCaD1BTa45fA7AHre5bbtiLptcDZwNG2H29TbHVodj1mAPsDN0q6j+Je6ZJJ2thc5XfjQWCJ7Sdt3wv8hCJJTEZVrsdJwKUAtm8Fnk4xONxUVOm7ZSSSFNrjP4F9JO0laTvgbcCSgQUkHQBcRJEQJvM9Y2hyPWw/YnsX2122uyjaWI623VtPuC3V9HcD+BZFLQFJu1DcTvp5O4NsoyrX4wHgNQCSXkCRFNa1NcrOsQR4e9kL6SDgEdtrx3LA3D5qA9ubJL0HuI6id8WXbK+W9Emg1/YS4LPAdOAySQAP2D66tqBbqOL1mBIqXovrgMMk3Q08BXzQ9u/ri7p1Kl6PM4B/kfR+ikbnE1x2xZlsJP0rxR8Eu5RtKB8HpgHYvpCiTeUI4KfARuDEMZ9zkl7LiIgYhdw+ioiIfkkKERHRL0khIiL6JSlERES/JIWIiOiXpBCTiqSnJK2UtErSVZJmjuFYj1Yo8z5JayQtHu15BhzrBEm7D1j/oqR5Yz1uxEikS2pMKpIetT29XL4E+Intc8d6rAZlfgy81vaDg7Y/zfamEZ7vRuADk/QhvZggUlOIyexWysHBJB0o6dZyDP4fStqv3H6CpCskXSvpvySdN/ggknYp933DoO0XUgzxfI2k90s6R9JXJd1CMR5Pl6R/l7SifL18wL5nSrpL0p2SPi3pLRRjXi0uazrPkHRj39Aekv66LL9K0mcGHOdRSeeWx7lN0q7l9mPLsndKumm8L2xMYnWPF55XXuP5Ah4t37cFLgNeX67vCDytXH4tcHm5fALFkBE7UQyXcD/F6KwAjwK7ArcDrxvmfPcBu5TL5wDLgWeU6zsATy+X96F4Ihfgr4AfsmX+jJ3L9xsZMG9E3zqwO8XQDrMoRiG4AXhTWcbAUeXyecBHyuW7gD3K5Zl1/7vkNXFeGeYiJptnSFpJUUNYAywtt+8EXCJpH4ov0mkD9rne9iMA5VAScymGI54GXA+cYvsHFc+/xPYfy+VpwBckvYRieIp9y+2vBb7scv4M243Gywf4S+BG2+vKGBdTTL7yLeAJ4DtlueXA68rlW4CLJV0KXFEx9ojcPopJ54+2X0LxxS7glHL73wHLbO8PHEVRK+gzcETap9gyJtgmii/aw0dw/scGLL8f+A3w3yj+4t9uBMep6knbfQ2D/bHb/lvgIxQjaC6X9OwWnDsmoSSFmJTKv8LfB5wh6WkUNYW+IYVPqHoY4J3A8yWdOYowdgLWuhjn/3iKW1pQ1F5OlLQDgKSdy+0bKIYNH+w/gFeVbRvbUswn0LDmIum5tm+3/TGKEURnNyof0SdJISYt23cAP6L4Ej0P+JSkOxjB6MC2nyr3P1TSu0cYwj8B75B0J/B8ylqE7WsphjzuLW91faAsfzFwYV9D84AY1gJnAcso5ixebvvbTc792b6GaYr2iztHGHtMUemSGhER/VJTiIiIfkkKERHRL0khIiL6JSlERES/JIWIiOiXpBAREf2SFCIiot//B/APOCj3hFCpAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "## DONE: TODO: add your code below to plot the relationship between time and rank percentage\n",
        "\n",
        "x = [x for x, _ in ranks_and_times]\n",
        "y = [y*1000 for _, y in ranks_and_times]\n",
        "\n",
        "\n",
        "plt.plot(x, y, 'bo')\n",
        "plt.xlabel(\"Rank fractions\")\n",
        "plt.ylabel(\"Time taken for infence in milli-seconds\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KR3JTDWb2QII"
      },
      "source": [
        "## Coding Challenge Part 3: Perform evaluations on the dataset in factorized space. (4 points)\n",
        "\n",
        "In this section, you will perform evaluations on the dataset in factorized space.\n",
        "\n",
        "* [**4 points**] 2 pts for question 6 and question 7."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OctdUxerSZut"
      },
      "outputs": [],
      "source": [
        "def low_rank_net_fn(batch: Batch, rank: float) -> jnp.ndarray:\n",
        "  \n",
        "  x = normalize(batch[0])\n",
        "  \n",
        "  total_input_dim = np.prod(x.shape[1:])\n",
        "\n",
        "  # Do not alter the architecture code.\n",
        "  net = hk.Sequential([\n",
        "      hk.Conv2D(output_channels=6*3, kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      jax.nn.relu,\n",
        "      hk.Conv2D(output_channels=16*3, kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      hk.Flatten(),\n",
        "      hk.Linear(int(rank * min(total_input_dim, 3000)), with_bias=False),\n",
        "      hk.Linear(3000), jax.nn.relu,\n",
        "      hk.Linear(int(rank * 2000), with_bias=False), \n",
        "      hk.Linear(2000), jax.nn.relu,\n",
        "      hk.Linear(int(rank * 2000), with_bias=False), \n",
        "      hk.Linear(2000), jax.nn.relu,      \n",
        "      hk.Linear(int(rank * 1000), with_bias=False), \n",
        "      hk.Linear(1000), jax.nn.relu,\n",
        "      hk.Linear(int(rank * 10), with_bias=False),\n",
        "      hk.Linear(10),\n",
        "  ])\n",
        "  return net(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zi2TN_WV-jgZ",
        "outputId": "2d81e60e-e526-408f-da40-f96a33325625"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Evaluating the model at 1.00\n",
            "Rank Fraction / Test accuracy: 1.00 / 0.626.\n",
            "Rank Fraction / Duration (milli sec): 1.00 / 448.5849.\n",
            "Evaluating the model at 0.90\n",
            "Rank Fraction / Test accuracy: 0.90 / 0.626.\n",
            "Rank Fraction / Duration (milli sec): 0.90 / 450.0596.\n",
            "Evaluating the model at 0.80\n",
            "Rank Fraction / Test accuracy: 0.80 / 0.625.\n",
            "Rank Fraction / Duration (milli sec): 0.80 / 405.1707.\n",
            "Evaluating the model at 0.70\n",
            "Rank Fraction / Test accuracy: 0.70 / 0.614.\n",
            "Rank Fraction / Duration (milli sec): 0.70 / 373.9225.\n",
            "Evaluating the model at 0.60\n",
            "Rank Fraction / Test accuracy: 0.60 / 0.617.\n",
            "Rank Fraction / Duration (milli sec): 0.60 / 335.1806.\n",
            "Evaluating the model at 0.50\n",
            "Rank Fraction / Test accuracy: 0.50 / 0.616.\n",
            "Rank Fraction / Duration (milli sec): 0.50 / 298.5186.\n",
            "Evaluating the model at 0.40\n",
            "Rank Fraction / Test accuracy: 0.40 / 0.581.\n",
            "Rank Fraction / Duration (milli sec): 0.40 / 264.1593.\n",
            "Evaluating the model at 0.30\n",
            "Rank Fraction / Test accuracy: 0.30 / 0.513.\n",
            "Rank Fraction / Duration (milli sec): 0.30 / 238.5024.\n",
            "Evaluating the model at 0.20\n",
            "Rank Fraction / Test accuracy: 0.20 / 0.343.\n",
            "Rank Fraction / Duration (milli sec): 0.20 / 198.8206.\n",
            "Evaluating the model at 0.10\n",
            "Rank Fraction / Test accuracy: 0.10 / 0.186.\n",
            "Rank Fraction / Duration (milli sec): 0.10 / 170.3905.\n"
          ]
        }
      ],
      "source": [
        "vanilla_to_low_rank_map = {\n",
        "    'conv2_d': 'conv2_d',\n",
        "    'conv2_d_1': 'conv2_d_1',\n",
        "    'linear': ['linear', 'linear_1'],\n",
        "    'linear_1': ['linear_2', 'linear_3'],\n",
        "    'linear_2': ['linear_4', 'linear_5'],\n",
        "    'linear_3': ['linear_6', 'linear_7'],\n",
        "    'linear_4': ['linear_8', 'linear_9']\n",
        "}\n",
        "\n",
        "\n",
        "ranks_and_accuracies = []\n",
        "ranks_and_times = []\n",
        "for rank_fraction in np.arange(1.0, 0.0, -0.1):\n",
        "  low_rank_net_fn_partial = partial(low_rank_net_fn, rank=rank_fraction)\n",
        "  net = hk.without_apply_rng(hk.transform(low_rank_net_fn_partial)) \n",
        "  low_rank_params = net.init(jax.random.PRNGKey(42), next(train))\n",
        "\n",
        "  print(f\"Evaluating the model at \" f\"{rank_fraction:.2f}\")\n",
        "\n",
        "  for layer in vanilla_to_low_rank_map.keys():\n",
        "    if 'conv' in layer:\n",
        "      low_rank_params[layer] = params[layer]\n",
        "      continue\n",
        "    weight = params[layer]['w']\n",
        "    # TODO: complete coding the rank_approximated_weight function to compute the SVD of the matrix to return the rank approximated weights u and v for a given matrix.\n",
        "    u, v = rank_approximated_weight(weight, rank_fraction)\n",
        "    low_rank_params[vanilla_to_low_rank_map[layer][0]]['w'] = u\n",
        "    low_rank_params[vanilla_to_low_rank_map[layer][1]]['w'] = v\n",
        "    low_rank_params[vanilla_to_low_rank_map[layer][1]]['b'] = params[layer]['b']\n",
        "  \n",
        "  # TODO: modify the compute_eval_metrics function below to compute the time taken for inference.\n",
        "  test_accuracy, duration = compute_eval_metrics(low_rank_params, next(test), 50)\n",
        "  ranks_and_times.append((rank_fraction, np.mean(duration)))\n",
        "  ranks_and_accuracies.append((rank_fraction, np.mean(test_accuracy)))\n",
        "  print(f\"Rank Fraction / Test accuracy: \"\n",
        "          f\"{rank_fraction:.2f} / {np.mean(test_accuracy):.3f}.\")\n",
        "  print(f\"Rank Fraction / Duration (milli sec): \"\n",
        "          f\"{rank_fraction:.2f} / {np.mean(duration)*1000:.4f}.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObBn-Pf_996r"
      },
      "source": [
        "### Q6: Plot a curve showing time vs rank percentage of the matrix "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "c0bJJFL4LM7q",
        "outputId": "3e803e6c-5dd4-4e66-8be1-0523acbfa2d3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Time taken for infence in milli-seconds')"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeaklEQVR4nO3de5gdVZnv8e8vIYAtlwCJDOTWoAgHLwNMiyDnDAMoIpeAeBl9GhTkoUdFRXAcZOIMXqaPV2D0cQZpjYTLnkFElMCAGLnIjApjB0IA8RIxCcQMRIVwaQ0E3vNHrd3s7tO9u3bSVdXZ/fs8Tz1Vter2dtH0m6q1ai1FBGZmZgBTqg7AzMwmDicFMzMb5KRgZmaDnBTMzGyQk4KZmQ3aquoANseMGTOis7Oz6jDMzLYoS5cu/V1EzBxp2xadFDo7O+nv7686DDOzLYqkVaNt8+sjMzMb5KRgZmaDnBTMzGyQk4KZmQ1yUjAzs0GFJwVJUyXdLen6tL5I0m8kLUvTfqlckr4saYWk5ZIOKDo2M7O8ajXo7IQpU7J5rVZ1RMUo40nhTOCBYWUfjYj90rQslb0J2CtNPcBFJcRmZjamWg16emDVKojI5j091SSGopNToUlB0mzgGODrOXY/HrgsMncA0yXtVmR8ZmZ5LFgAAwNDywYGsvIylZGcin5S+Gfg74Dnh5X3pldEF0raJpXNAh5q2OfhVDaEpB5J/ZL6161bV0jQZmaNVq9urbwoZSSnwpKCpGOBRyNi6bBN5wL7AK8BdgbOaeW8EdEXEV0R0TVz5ohfaZuZjau5c1srL0oZyanIJ4VDgPmSVgJXAodLuiIi1qZXRBuAS4AD0/5rgDkNx89OZWZmlerthY6OoWUdHVl5mcpIToUlhYg4NyJmR0Qn8A7glog4qV5PIEnACcB96ZDFwLtSK6SDgPURsbao+MzM8uruhr4+mDcPpGze15eVl6mM5FRFh3g1STMBAcuA96byG4CjgRXAAHBqBbGZmY2ou7v8JDBSDJDVIaxenT0h9PaOb1yKiOY7SJ8H/gn4I/A94NXAWRFxxfiFsWm6urrCvaSambVG0tKI6BppW57XR0dGxBPAscBK4GXAR8cvPDMzmyjyJIX6K6ZjgG9FxPoC4zEzswrlqVO4XtLPyV4fvS/VB/yp2LDMzKwKYz4pRMTHgNcBXRHxLFkl8PFFB2ZmZuUb9UlB0okjlDWuXlNEQGZmVp1mr4+OS/OXkD0p3JLWDwN+jJOCmVnbGTUpRMSpAJK+D+xb/5AsfXy2qJTozMysVHlaH80Z9mXxI0DJPX6YmVkZ8rQ+ulnSTcC/p/W/Bn5QXEhmZlaVMZNCRHwgVTr/n1TUFxHfKTYsMzOrQq6+jyLiGlyxbGbW9sasU5B0oqRfSVov6QlJT0p6oozgzMysXHmeFD4PHBcRw8dZNjOzNpOn9dEjTghmZpNDnieFfknfBL4LbKgXpnoGMzNrI3mSwg5k/R0d2VAWuOLZzKzt5GmS6hHQzMwmiTytj2ZL+o6kR9P0bUmzywjOzMzKlaei+RJgMbB7mq5LZWZm1mbyJIWZEXFJRGxM0yJgZsFxmZlZBfIkhd9LOknS1DSdBPy+6MDMzKx8eZLCe4C3A/8DrAXeCrjy2cysDeVpfbQKmF9CLGZmVrE8rY8ulTS9YX0nSd8oNiwzs0ytBp2dMGVKNq/Vqo6oveX5eO3VEfF4fSUiHpO0f4ExmZkBWQLo6YGBgWx91apsHaC7u7q42lmeOoUpknaqr0jamZxdbpuZbY4FC15ICHUDA1m5FSNPUjgf+ImkT0v6NPBjsp5Tc0ktlu6WdH1a30PSnZJWSPqmpK1T+TZpfUXa3tn6j2Nm7WT16tbKbfONmRQi4jLgRLKxmR8BToyIy1u4xplAYy+rnwMujIiXAY8Bp6Xy04DHUvmFaT8zm8TmjjIa/GjltvnyPCkA7Aw8HRFfAdZJ2iPPQak7jGOAr6d1AYcDV6ddLgVOSMvHp3XS9iPS/mY2SfX2QkfH0LKOjqzcipGn9dF5wDnAualoGnBFzvP/M/B3wPNpfRfg8YjYmNYfBmal5VnAQwBp+/q0v5lNUt3d0NcH8+aBlM37+lzJXKQ8FcZvBvYH7gKIiN9K2n6sgyQdCzwaEUsl/dVmRTn0vD1AD8BcP0Oatb3ubieBMuV5ffRMRATZGApIenHOcx8CzJe0EriS7LXRl4DpkurJaDawJi2vAeaka2wF7MgI3WlERF9EdEVE18yZ7oLJzGw85UkKV0m6mOyP+enAD4CvjXVQRJwbEbMjohN4B3BLRHQDt5J1lQHwbuDatLw4rZO235KSkZmZlSRPNxdflPQG4Algb+AfI2LJZlzzHOBKSf8E3A0sTOULgcslrQD+QJZIzMysRGMmhfS66JaIWCJpb2BvSdMi4tm8F4mI24Db0vKDwIEj7PMn4G15z2lmZuMvz+uj24FtJM0CvgecDCwqMigzM6tGnqSgiBgg+4Dtooh4G/CKYsMyM7Mq5EoKkg4GuoH/SGVTiwvJzMyqkicpnEn24dp3IuJ+SXuStSAyM7M2k6f10e1k9QpI+rNUUfyhogMzM7Py5e37qO6GQqIwM7MJodWk4A7qzMzaWKtJYcwvmc3MbMs1ap2CpB0i4ok00lrdlfX1iPhD4dGZmVmpmlU0/xtwLLCUrDM8DZvvWXh0ZmZWqlFfH0XEsWm+R0TsOXxeXohmVoVaDTo7YcqUbF6rVR2RlaHZ66MDmh0YEXeNfzhmNhHUatDTAwMD2fqqVdk6eGyDdqfReqeW1OwDtYiIw4sJKb+urq7o7++vOgyzttPZmSWC4ebNg5Ury47GxpukpRHRNdK2UZ8UIuKw4kIys4ls9erWyq19NHt9dGKzAyPimvEPx8wmgrlzR35S8Ai47a9Z66PjmmwLwEnBrE319g6tUwDo6MjKrb01e310apmBmNnEUa9MXrAge2U0d26WEFzJ3P6avT46KSKukHT2SNsj4oLiwjKzqnV3OwlMRs1eH704zbcvIxAzM6tes9dHF6f5J8sLx8zMqjTmeAqS9gA+CHQ27h8R84sLy8zMqjBmUgC+CywErgOeLzYcMzOrUp6k8KeI+HLhkZiZWeXyJIUvSToP+D6woV7ovo/MzNpPnqTwKuBk4HBeeH0Uad3MzNpInqTwNmDPiHim6GDMzKxaeYbjvA+YXnQgZmZWvTxPCtOBn0v6KUPrFJo2SZW0LXA7sE26ztURcZ6kRcChwPq06ykRsUySgC8BRwMDqdz1FmZmJcqTFM7bxHNvAA6PiKckTQP+S9KNadtHI+LqYfu/CdgrTa8FLkpzMzMryZhJISJ+uCknjmz0nqfS6rQ0jTyiT+Z44LJ03B2SpkvaLSLWbsr1zcysdXnqFDaZpKmSlgGPAksi4s60qVfSckkXStomlc0CHmo4/OFUNvycPZL6JfWvW7euyPDNzCadQpNCRDwXEfsBs4EDJb0SOBfYB3gNsDNwTovn7IuIrojomjlz5rjHbGY2mRWaFOoi4nHgVuCoiFgbmQ3AJcCBabc1wJyGw2anMjMzK8mYSUHSIZKWSPqlpAcl/UbSgzmOmylpelp+EfAGslZMu6UyASeQNXkFWAy8S5mDgPWuTzAzK1ee1kcLgbOApcBzLZx7N+BSSVPJks9VEXG9pFskzQQELAPem/a/gaw56gqyJqke+c3MrGR5ksL6iLhx7N2GiojlwP4jlI/YPUZqdXRGq9cxM7Pxkycp3CrpC8A1uEM8M7O2lqei+bVAF/B/gfPT9MUigzKb7Go16OyEKVOyea1WdUQ2WeT5eO2wMgIxs0ytBj09MDCQra9ala0DdHdXF5dNDspe5Y+wQTopIq6QdPZI2yPigkIjy6Grqyv6+/urDsNsXHV2ZolguHnzYOXKsqOxdiRpaUR0jbSt2ZPCi9N8+/EPycxGs3p1a+Vm42nUpBARF6f5J8sLx8zmzh35SWHu3PJjscmnlC+azSy/3l7o6Bha1tGRlZsVzUnBbILp7oa+vqwOQcrmfX2uZLZy5PlOwcxK1t3tJGDVyNP30a6SFtYHyJG0r6TTig/NzMzKluf10SLgJmD3tP5L4MNFBWRmZtXJkxRmRMRVwPMAEbGR1jrGMzOzLUSepPC0pF1IQ2nWu7UuNCozM6tEnorms8nGOnippB8BM4G3FhqVmZlVIk/fR3dJOhTYm2wMhF9ExLOFR2ZmZqXL0/roDGC7iLg/Iu4DtpP0/uJDMzOzsuWpUzg9jbEMQEQ8BpxeXEhmZlaVPElhahpPGYA0vObWxYVkZmZVyVPR/D3gm5IuTut/k8rMzKzN5EkK55Algvel9SXA1wuLyMzMKpOn9dHzwEVpMjOzNjZmUpB0CPAJYF7aX0BExJ7FhmZmZmXL8/poIXAWsBR3b2Fm1tbyJIX1EXFj4ZGYmVnl8iSFWyV9AbgG2FAvjIi7CovKzMwqkScpvDbNuxrKAjh8/MMxM7Mq5Wl9dNimnFjStsDtwDbpOldHxHmS9gCuBHYhq6c4OSKekbQNcBnwF8Dvgb+OiJWbcm0zM9s0RY68tgE4PCL+HNgPOCp1u/054MKIeBnwGFA/12nAY6n8wrSfWalqNejshClTsnmtVnVEZuUqbOS1yDyVVqelqf7a6epUfilwQlo+Pq2Tth/R2L2GWdFqNejpgVWrICKb9/Q4MdjkUujIa5KmSloGPEr2JfSvgcfTOQAeBmal5VnAQw3XWE/2imn4OXsk9UvqX7duXZ4wzHJZsAAGBoaWDQxk5WaTRaEjr0XEcxGxHzAbOBDYZ1MDbThnX0R0RUTXzJkzN/d0ZoNWr26t3Kwd5UkKw0deuwz4YCsXSV1v3wocDEyXVK/gng2sSctrgDkAafuOZBXOZqWYO7e1crN2NGpSkPS2tPgYcCjwOrKO8V4REcvHOrGkmZKmp+UXAW8AHiBLDvXhPN8NXJuWF6d10vZbIiJa+mnMNkNvL3R0DC3r6MjKzSaLZk8K56b5tyNiY33ktRaG4tyN7MO35cBPgSURcT1Zr6tnS1pBVmewMO2/ENgllZ8NfKzVH8Zsc3R3Q18fzJsHUjbv68vKzSYLjfaPcUlLyOoRXgP85/DtETG/2NDG1tXVFf39/VWHYWa2RZG0NCK6RtrW7OO1Y4ADgMuB84sIzMzMJpZRk0JEPAPcIel1EeG2n2Zmk0Cevo92ktQLdDbuHxHu+8jMrM3kSQrfAr5KNgSnx1MwM2tjeZLCxojwUJxmZpNAno/XrpP0fkm7Sdq5PhUemZmZlS7Pk0L9g7KPNpQF4DGazczaTJ7xFPYoIxAzM6veqElB0uERcYukE0faHhHXFBeWmZlVodmTwqHALcBxI2wLsjGbzcysjTT7eO28ND+1vHDMzKxKeVofmZnZJOGkYGZmg5omBUlTJL2urGDMzKxaTZNCRDwP/EtJsZiZWcXyvD66WdJbJKnwaMzMrFJ5ksLfkHWK94ykJyQ9KemJguMyM7MKjJkUImL7iJgSEdMiYoe0vkMZwdnkUatBZydMmZLNa7WqIzKbnPL0fYSk+cBfptXb0ljLZuOiVoOeHhgYyNZXrcrWweMjm5VtzCcFSZ8FzgR+lqYzJX2m6MBs8liw4IWEUDcwkJWbWbnyPCkcDeyXWiIh6VLgbuDcIgOzyWP16tbKzaw4eT9em96wvGMRgdjkNXdua+VmVpw8SeEzwN2SFqWnhKVAb7Fh2WTS2wsdHUPLOjqycjMr16hJQdIhafEa4KA0/zZwcER8s4TYbJLo7oa+Ppg3D6Rs3tfnSmazKigiRt4gLY2Iv5B0V0QcUHJcuXR1dUV/f3/VYZiZbVHS3/eukbY1q2h+VlIfMFvSl4dvjIgPjVeAZmY2MTSrUziWbJCdP5LVIwyfmpI0R9Ktkn4m6X5JZ6byT0haI2lZmo5uOOZcSSsk/ULSGzfnBzMzs9Y1G2Tnd8CVkh6IiHs24dwbgY9ExF2StgeWSlqStl0YEV9s3FnSvsA7gFcAuwM/kPTyiHhuE65tZmabIE83F5uSEIiItRFxV1p+EngAmNXkkOOBKyNiQ0T8BlgBHLgp1zYzs01TyiA7kjqB/YE7U9EHJC2X9A1JO6WyWcBDDYc9TPMkYmZm46zwpCBpO7KmrB+OiCeAi4CXAvsBa4HzWzxfj6R+Sf3r1q0b93jNzCazMbu5kLQN8Bags3H/iPhUjmOnkSWEWkRck457pGH714B653prgDkNh89OZUNERB/QB1mT1LFiMDOz/PI8KVxL9r5/I/B0w9RUGpRnIfBARFzQUL5bw25vBu5Ly4uBd0jaRtIewF7Af+f5IczMbHzk6RBvdkQctQnnPgQ4GbhX0rJU9vfAOyXtBwSwkmwQHyLifklXkfXEuhE4wy2PzMzKlScp/FjSqyLi3lZOHBH/BYw0hOcNTY7pxf0qmZlVJs/ro/9N9o3BL1KLoXslLS86MCuHRzwzs0Z5nhTeVHgUVgmPeGZmw+X5eG0VWaugw9PyQJ7jbOLziGdmNlye4TjPA87hhZHWpgFXFBmUlcMjnpnZcHn+xf9mYD6pGWpE/BbYvsigrBwe8czMhsuTFJ6JbNCFAJD04mJDsrJ4xDMzGy5PUrhK0sXAdEmnAz8Avl5sWFYGj3hmZsONOvLa4A7Zl8mvB44k++7gJuD2iNhQfHjNeeQ1M7PWberIa3ULI+I9wJJ0su3IPkA7YvxCNDOziSDP66M1kv4VIHVz/X3c+sjMrC3l+U7hH4CnJH2VLCGcHxGXFB6ZmZmVbtTXR5JObFi9E/gHsl5LQ9KJ9a6wzcysfTSrUzhu2PrdZB+uHUfWPNVJwcyszYyaFCLi1DIDMTOz6uUZeW1b4DTgFcC29fLUIsnMzNpIntZHlwN/BrwR+CHZMJlPFhmUmZlVI09SeFlqgfR0RFwKHAO8ttiwzMysCnmSwrNp/rikVwI7Ai8pLiQzM6tKni+a+9JHax8HFgPbkTVPNTOzNpMnKdwcEY8BtwN7Akjao9CozMysEnleH317hLKrxzsQMzOrXrMvmvcha4a647Cvm3egoWmqmZm1j2avj/YGjgWmM/Tr5ieB04sMyszMqtHsi+ZrgWslHRwRPykxJjMzq0ieXlKdEMzMJok8Fc1mZjZJOCmYmdmgMZOCpF0lLZR0Y1rfV9JpOY6bI+lWST+TdL+kM1P5zpKWSPpVmu+UyiXpy5JWSFou6YDN/eHMzKw1eZ4UFgE3Abun9V8CH85x3EbgIxGxL3AQcIakfYGPkX0Qtxdwc1oHeBOwV5p6gIty/gxbpFoNOjthypRsXqtVHZGZWb6kMCMirgKeB4iIjcBzYx0UEWsj4q60/CTwADALOB64NO12KXBCWj4euCwydwDTJe3Wyg+zpajVoKcHVq2CiGze0+PEYGbVy5MUnpa0C9loa0g6CFjfykUkdQL7kw3ruWtErE2b/gfYNS3PAh5qOOzhVDb8XD2S+iX1r1u3rpUwJowFC2BgYGjZwEBWbmZWpTx9H51N1hHeSyX9CJgJvDXvBSRtR9ZVxocj4glJg9siIiRFKwFHRB/QB9DV1dXSsRPF6tWtlZuZlWXMpBARd0k6lOwLZwG/iIhnxzgMAEnTyBJCLSLqYzo/Imm3iFibXg89msrXAHMaDp+dytrO3LnZK6ORys3MqpSn9dFU4GjgCOBI4IOSzs5xnICFwAMRcUHDpsXAu9Pyu4FrG8rflVohHQSsb3jN1FZ6e6GjY2hZR0dWbmZWpTyvj64D/gTcS6pszukQ4GTgXknLUtnfA58FrkrNWlcBb0/bbiBLPiuAAeDUFq61RenuzuYLFmSvjObOzRJCvdzMrCqKaP5aXtLyiHh1SfG0pKurK/r7+6sOw8xsiyJpaUR0jbQtT+ujGyUdOc4xmZnZBJTn9dEdwHckTSEbr1lkDYd2KDQyMzMrXZ6kcAFwMHBvjPWuyczMtmh5Xh89BNznhGBm1v7yPCk8CNyWOsTbUC8c1szUzMzaQJ6k8Js0bZ0mMzNrU3m+aP5kGYGYmVn1Rk0Kkr4SER+QdB2pM7xGETG/0MjMzKx0zZ4U3gV8APhiSbGYmVnFmiWFXwNExA9LisXMzCrWLCnMbNbxnVsfmZm1n2ZJYSqwHdkXzGZmNgk0SwprI+JTpUVSklrNvZOamY2mWVJouyeE+tjI9aEw62MjgxODmRk07+biiNKiKInHRjYza27UpBARfygzkDJ4bGQzs+bydIjXNkYbA9ljI5uZZSZVUvDYyGZmzU2qpNDdDX19MG8eSNm8r8+VzGZmdXl6SW0r3d1OAmZmo5lUTwpmZtack4KZmQ1yUjAzs0FOCmZmNshJwczMBini/xtUbYshaR2wquo4NtMM4HdVBzGB+H4M5fvxAt+LoTbnfsyLiJkjbdiik0I7kNQfEV1VxzFR+H4M5fvxAt+LoYq6H359ZGZmg5wUzMxskJNC9fqqDmCC8f0YyvfjBb4XQxVyP1ynYGZmg/ykYGZmg5wUzMxskJNCSSQdJekXklZI+tgI28+W9DNJyyXdLGleFXGWZaz70bDfWySFpLZtipjnXkh6e/r9uF/Sv5UdY5ly/L8yV9Ktku5O/78cXUWcZZD0DUmPSrpvlO2S9OV0r5ZLOmCzLxoRngqegKnAr4E9ga2Be4B9h+1zGNCRlt8HfLPquKu8H2m/7YHbgTuArqrjrvB3Yy/gbmCntP6SquOu+H70Ae9Ly/sCK6uOu8D78ZfAAcB9o2w/GrgREHAQcOfmXtNPCuU4EFgREQ9GxDPAlcDxjTtExK0RMZBW7wBmlxxjmca8H8mngc8BfyozuJLluRenA/8SEY8BRMSjJcdYpjz3I4Ad0vKOwG9LjK9UEXE78IcmuxwPXBaZO4DpknbbnGs6KZRjFvBQw/rDqWw0p5Fl/3Y15v1Ij8FzIuI/ygysAnl+N14OvFzSjyTdIemo0qIrX5778QngJEkPAzcAHywntAmp1b8tY5p0I69NdJJOArqAQ6uOpSqSpgAXAKdUHMpEsRXZK6S/InuCvF3SqyLi8Uqjqs47gUURcb6kg4HLJb0yIp6vOrB24CeFcqwB5jSsz05lQ0h6PbAAmB8RG0qKrQpj3Y/tgVcCt0laSfaudHGbVjbn+d14GFgcEc9GxG+AX5IliXaU536cBlwFEBE/AbYl6xxuMsr1t6UVTgrl+Cmwl6Q9JG0NvANY3LiDpP2Bi8kSQju/M4Yx7kdErI+IGRHRGRGdZHUs8yOiv5pwCzXm7wbwXbKnBCTNIHud9GCZQZYoz/1YDRwBIOl/kSWFdaVGOXEsBt6VWiEdBKyPiLWbc0K/PipBRGyU9AHgJrLWFd+IiPslfQroj4jFwBeA7YBvSQJYHRHzKwu6QDnvx6SQ817cBBwp6WfAc8BHI+L31UVdnJz34yPA1ySdRVbpfEqkpjjtRtK/k/2DYEaqQzkPmAYQEV8lq1M5GlgBDACnbvY12/RempnZJvDrIzMzG+SkYGZmg5wUzMxskJOCmZkNclIwM7NBTgrWViQ9J2mZpPskXSdp+mac66kc+3xI0gOSapt6nYZznSJp94b1r0vad3PPa9YKN0m1tiLpqYjYLi1fCvwyIno391xN9vk58PqIeHhY+VYRsbHF690G/G2bfqRnWwg/KVg7+wmpczBJB0r6SeqD/8eS9k7lp0i6RtL3JP1K0ueHn0TSjHTsMcPKv0rWxfONks6S9AlJl0v6EVl/PJ2S/lPSXWl6XcOx50i6V9I9kj4r6a1kfV7V0pPOiyTdVu/aQ9I70/73Sfpcw3mektSbznOHpF1T+dvSvvdIun28b6y1sar7C/fkaTwn4Kk0nwp8Czgqre8AbJWWXw98Oy2fQtZlxI5k3SWsIuudFeApYFfgTuANo1xvJTAjLX8CWAq8KK13ANum5b3IvsgFeBPwY14YP2PnNL+NhnEj6uvA7mRdO8wk64XgFuCEtE8Ax6XlzwMfT8v3ArPS8vSq/7t42nImd3Nh7eZFkpaRPSE8ACxJ5TsCl0rai+wP6bSGY26OiPUAqSuJeWTdEU8DbgbOiIgf5rz+4oj4Y1qeBnxF0n5k3VO8PJW/Hrgk0vgZEdGsv3yA1wC3RcS6FGONbPCV7wLPANen/ZYCb0jLPwIWSboKuCZn7GZ+fWRt548RsR/ZH3YBZ6TyTwO3RsQrgePIngrqGnukfY4X+gTbSPaH9o0tXP/phuWzgEeAPyf7F//WLZwnr2cjol4xOBh7RLwX+DhZD5pLJe1SwLWtDTkpWFtK/wr/EPARSVuRPSnUuxQ+Je9pgPcA+0g6ZxPC2BFYG1k//yeTvdKC7OnlVEkdAJJ2TuVPknUbPtx/A4emuo2pZOMJNH1ykfTSiLgzIv6RrAfROc32N6tzUrC2FRF3A8vJ/oh+HviMpLtpoXfgiHguHX+4pPe3GMK/Au+WdA+wD+kpIiK+R9blcX961fW3af9FwFfrFc0NMawFPgbcSjZm8dKIuHaMa3+hXjFNVn9xT4ux2yTlJqlmZjbITwpmZjbIScHMzAY5KZiZ2SAnBTMzG+SkYGZmg5wUzMxskJOCmZkN+n9O07rgXurCWQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# DONE: TODO: add code to plot the relationship between time vs percentage rank of the matrix.\n",
        "\n",
        "x = [x for x, _ in ranks_and_times]\n",
        "y = [y*1000 for _, y in ranks_and_times]\n",
        "\n",
        "plt.plot(x, y, 'bo')\n",
        "plt.xlabel(\"Rank fractions\")\n",
        "plt.ylabel(\"Time taken for infence in milli-seconds\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "313ALwDu93k1"
      },
      "source": [
        "### Q7: What do you observe between time and the percentage rank of the matrix.\n",
        "\n",
        "### Put your answer here:\n",
        "\n",
        "\n",
        "*   We see a linear drop-off in latency over each iteration. This is in relation to what is expected -- \n",
        "\n",
        "\n",
        "Maths Working Below:\n",
        "```\n",
        "(naive algorithm for multiplication) \n",
        "x*U where u has dimension (n,o) and x has dimensions (m, n) will require \n",
        "(2n-1)*m*o operations\n",
        "\n",
        "decomposing U into u and v with dimensions (n, a), (a, o) \n",
        "x*u*v will require (2n -1)*a*m + (2a-1)*m*o \n",
        "\n",
        "which is approximately equal to 2n*a*m + 2a*m*o = 2a*m(o+n)  (since 2a>>1 and 2n>>1)\n",
        "number of multiplications decrease linearly compared with size of a\n",
        "```\n",
        "\n",
        "\n",
        "*   When rank_fraction is greater than 0.3 we see inference time of the model is comparable to the network where no decomposition was done. ie > 200ms\n",
        "\n",
        "```\n",
        "Using our equations from earlier we see\n",
        "\n",
        "Without SVD we take approx 2n*m*o operations\n",
        "After SVD we take approx 2a*m*(o+n) \n",
        "\n",
        "a == frac*n; After SVD we have 2*frac*n*o*m + 2*frac*n*m*n\n",
        "```\n",
        "\n",
        "**consider the following:**\n",
        "\n",
        "n is larger then o, then number of operations after SVD is still high, \n",
        "\n",
        "Maths Working Below:\n",
        "```\n",
        "in our network for the linear layers n is 2x to x times o\n",
        "(Linear layers are (6000+,3000), (3000, 2000), (2000, 2000), (2000, 1000)\n",
        "\n",
        "setting n as 2*o and frac as 0.5 we have\n",
        "After SVD -- n*o*m + 2*n*m*o = 3*m*n*o\n",
        "Note: Before SVD we needed 2*m*n*o operations, so in this case we have a 50% increase in latency \n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QHcBKkogM6uV"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jEAu9Vu-0rWX"
      },
      "source": [
        "## Coding Challenge Part 4: Take this Further (10 bonus points)\n",
        "\n",
        "This part of the challenge is designed to be open ended. If you wanted to show some more skills, here is your chance to shine. We include two options below -- **only do one of the options**:\n",
        "\n",
        "**Option 1:** Implement a change that isn't SVD but minimizes inference latency while preserving accuracy. Can you outperform SVD? \n",
        "\n",
        "\n",
        "\n",
        "**Option 2:** Improve the quality of code for this takehome. Pretend you are reviewing a peer and add comments to cells with suggestions of how to improve the code quality. Try and make your comments action orientated and precise. \n",
        "\n",
        "\n",
        "**For Option 1, DO NOT alter the previous code sections, instead add any new code below. You should not need to add new code for Option 2, instead just add comments to cells.**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOWDlWX9IO8o"
      },
      "source": [
        "## Structured Iterative Pruning on Linear Layers\n",
        "\n",
        "### Algorithm\n",
        "\n",
        "1. While validation_accuracy dips below threshold:\n",
        "> 1. Reduce neurons according to a schedule -- only linear layers since we have large linear layers\n",
        "> 2. Fine-tune train pruned network\n",
        "> 3. Measure validation accuracy\n",
        "\n",
        "2. Measure test accuracy\n",
        "3. Measure inference time \n",
        "\n",
        "### Depth Separable Convolutions (Other attempt) \n",
        "<s> Depth Separable Convolutions, will reduce inference time for convolutions </s>\n",
        "\n",
        "Dropped the idea as it's not in the spirit of model compression, \n",
        "\n",
        "Also, it's difficult to identify the right arch and train the model to desired accuracy and latency\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "02AB0gJ5wsfD"
      },
      "outputs": [],
      "source": [
        "# This works as expected because, the correct net gets binded to the params do to jit compiling behavior\n",
        "orig_accuracy, _ = compute_accuracy(avg_params, next(validation))\n",
        "threshold_on_accuracy = 0.1\n",
        "\n",
        "desired_accuracy = (1 - threshold_on_accuracy) * orig_accuracy\n",
        "params_fin = deepcopy(avg_params)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HSooTcXKyaFm"
      },
      "outputs": [],
      "source": [
        "def l2_structured_pruning(b: jnp.ndarray, weights_in: jnp.ndarray, weights_out: jnp.ndarray, sparse_fraction: float, algo=\"in\"):\n",
        "  \"\"\"Identify neurons that have small weights and manipulate weight matrices accordingly\"\"\"\n",
        "\n",
        "  if algo == \"in\":\n",
        "    # using incoming weights for the neuron\n",
        "    norms = jnp.linalg.norm(weights_in, ord=2, axis=0)\n",
        "    indices = jnp.argsort(norms)\n",
        "    k = int((1-sparse_fraction) * jnp.size(norms))\n",
        "    print(jnp.size(norms), \" neurons reduced to \", k)\n",
        "    keep_indices = jnp.greater_equal(np.arange(norms.size), norms.size - k)\n",
        "    weights_in_fin = weights_in[:, keep_indices]\n",
        "    weights_out_fin = weights_out[keep_indices, :]\n",
        "    b_fin = b[keep_indices]\n",
        "  \n",
        "  if algo == \"in_out\":\n",
        "    # using incoming and outgoing weights for the neuron\n",
        "    norms1 = jnp.linalg.norm(weights_in, ord=2, axis=0)\n",
        "    norms2 = jnp.linalg.norm(weights_out, ord=2, axis=1)\n",
        "    norms = norms1*norms2\n",
        "    indices = jnp.argsort(norms)\n",
        "    k = int((1-sparse_fraction) * jnp.size(norms))\n",
        "    print(jnp.size(norms), \" neurons reduced to \", k)\n",
        "    keep_indices = jnp.greater_equal(np.arange(norms.size), norms.size - k)\n",
        "    weights_in_fin = weights_in[:, keep_indices]\n",
        "    weights_out_fin = weights_out[keep_indices, :]\n",
        "    b_fin = b[keep_indices]\n",
        "\n",
        "  return b_fin, weights_in_fin, weights_out_fin"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eIqZD7TzSq3F"
      },
      "outputs": [],
      "source": [
        "def sparse_net_fn(batch: Batch, sparsity_fractions: jnp.ndarray, iteration: jnp.int8) -> jnp.ndarray:\n",
        "  \n",
        "  x = normalize(batch[0])\n",
        "  \n",
        "  total_input_dim = np.prod(x.shape[1:])\n",
        "  l1 = 3000\n",
        "  l2 = 2000\n",
        "  l3 = 2000\n",
        "  l4 = 1000\n",
        "  # calculate sizes for each layer\n",
        "  for i in np.arange(iteration):\n",
        "    l1 = int((1-sparsity_fractions[2])*l1)\n",
        "    l2 = int((1-sparsity_fractions[3])*l2)\n",
        "    l3 = int((1-sparsity_fractions[4])*l3)\n",
        "    l4 = int((1-sparsity_fractions[5])*l4)\n",
        "\n",
        "  # Do not alter the architecture code.\n",
        "  net = hk.Sequential([\n",
        "      hk.Conv2D(output_channels=int((1-sparsity_fractions[0]) * 6*3), kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      jax.nn.relu,\n",
        "      hk.Conv2D(output_channels=int((1-sparsity_fractions[1]) * 16*3), kernel_shape=(5,5)),\n",
        "      jax.nn.relu,\n",
        "      hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "      hk.Flatten(),\n",
        "      hk.Linear(l1), jax.nn.relu,\n",
        "      hk.Linear(l2), jax.nn.relu,\n",
        "      hk.Linear(l3), jax.nn.relu,\n",
        "      hk.Linear(l4), jax.nn.relu,\n",
        "      hk.Linear(10),\n",
        "  ])\n",
        "  return net(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qb2oKTWaSq3G",
        "outputId": "270f2d17-0aed-4a96-e017-50e5e333a430"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Evaluating the model at iteration 1\n",
            "3000  neurons reduced to  2760\n",
            "2000  neurons reduced to  1840\n",
            "2000  neurons reduced to  1760\n",
            "1000  neurons reduced to  880\n",
            "[Step 0] Validation / Test accuracy: 0.617 / 0.615.\n",
            "[Step 500] Validation / Test accuracy: 0.620 / 0.620.\n",
            "Evaluating the model at iteration 2\n",
            "2760  neurons reduced to  2539\n",
            "1840  neurons reduced to  1692\n",
            "1760  neurons reduced to  1548\n",
            "880  neurons reduced to  774\n",
            "[Step 0] Validation / Test accuracy: 0.611 / 0.612.\n",
            "[Step 500] Validation / Test accuracy: 0.613 / 0.619.\n",
            "Evaluating the model at iteration 3\n",
            "2539  neurons reduced to  2335\n",
            "1692  neurons reduced to  1556\n",
            "1548  neurons reduced to  1362\n",
            "774  neurons reduced to  681\n",
            "[Step 0] Validation / Test accuracy: 0.604 / 0.606.\n",
            "[Step 500] Validation / Test accuracy: 0.610 / 0.614.\n",
            "Evaluating the model at iteration 4\n",
            "2335  neurons reduced to  2148\n",
            "1556  neurons reduced to  1431\n",
            "1362  neurons reduced to  1198\n",
            "681  neurons reduced to  599\n",
            "[Step 0] Validation / Test accuracy: 0.597 / 0.607.\n",
            "[Step 500] Validation / Test accuracy: 0.606 / 0.614.\n",
            "Evaluating the model at iteration 5\n",
            "2148  neurons reduced to  1976\n",
            "1431  neurons reduced to  1316\n",
            "1198  neurons reduced to  1054\n",
            "599  neurons reduced to  527\n",
            "[Step 0] Validation / Test accuracy: 0.603 / 0.607.\n",
            "[Step 500] Validation / Test accuracy: 0.606 / 0.615.\n",
            "Evaluating the model at iteration 6\n",
            "1976  neurons reduced to  1817\n",
            "1316  neurons reduced to  1210\n",
            "1054  neurons reduced to  927\n",
            "527  neurons reduced to  463\n",
            "[Step 0] Validation / Test accuracy: 0.599 / 0.606.\n",
            "[Step 500] Validation / Test accuracy: 0.606 / 0.610.\n",
            "Evaluating the model at iteration 7\n",
            "1817  neurons reduced to  1671\n",
            "1210  neurons reduced to  1113\n",
            "927  neurons reduced to  815\n",
            "463  neurons reduced to  407\n",
            "[Step 0] Validation / Test accuracy: 0.588 / 0.597.\n",
            "[Step 500] Validation / Test accuracy: 0.601 / 0.605.\n",
            "Evaluating the model at iteration 8\n",
            "1671  neurons reduced to  1537\n",
            "1113  neurons reduced to  1023\n",
            "815  neurons reduced to  717\n",
            "407  neurons reduced to  358\n",
            "[Step 0] Validation / Test accuracy: 0.579 / 0.589.\n",
            "[Step 500] Validation / Test accuracy: 0.595 / 0.602.\n",
            "Evaluating the model at iteration 9\n",
            "1537  neurons reduced to  1414\n",
            "1023  neurons reduced to  941\n",
            "717  neurons reduced to  630\n",
            "358  neurons reduced to  315\n",
            "[Step 0] Validation / Test accuracy: 0.572 / 0.578.\n",
            "[Step 500] Validation / Test accuracy: 0.587 / 0.598.\n",
            "Evaluating the model at iteration 10\n",
            "1414  neurons reduced to  1300\n",
            "941  neurons reduced to  865\n",
            "630  neurons reduced to  554\n",
            "315  neurons reduced to  277\n",
            "[Step 0] Validation / Test accuracy: 0.572 / 0.579.\n",
            "[Step 500] Validation / Test accuracy: 0.590 / 0.596.\n",
            "Evaluating the model at iteration 11\n",
            "1300  neurons reduced to  1196\n",
            "865  neurons reduced to  795\n",
            "554  neurons reduced to  487\n",
            "277  neurons reduced to  243\n",
            "[Step 0] Validation / Test accuracy: 0.555 / 0.566.\n",
            "[Step 500] Validation / Test accuracy: 0.580 / 0.595.\n",
            "Evaluating the model at iteration 12\n",
            "1196  neurons reduced to  1100\n",
            "795  neurons reduced to  731\n",
            "487  neurons reduced to  428\n",
            "243  neurons reduced to  213\n",
            "[Step 0] Validation / Test accuracy: 0.564 / 0.575.\n",
            "[Step 500] Validation / Test accuracy: 0.581 / 0.596.\n",
            "Evaluating the model at iteration 13\n",
            "1100  neurons reduced to  1012\n",
            "731  neurons reduced to  672\n",
            "428  neurons reduced to  376\n",
            "213  neurons reduced to  187\n",
            "[Step 0] Validation / Test accuracy: 0.557 / 0.564.\n",
            "[Step 500] Validation / Test accuracy: 0.579 / 0.589.\n",
            "Evaluating the model at iteration 14\n",
            "1012  neurons reduced to  931\n",
            "672  neurons reduced to  618\n",
            "376  neurons reduced to  330\n",
            "187  neurons reduced to  164\n",
            "[Step 0] Validation / Test accuracy: 0.550 / 0.556.\n",
            "[Step 500] Validation / Test accuracy: 0.573 / 0.582.\n",
            "Evaluating the model at iteration 15\n",
            "931  neurons reduced to  856\n",
            "618  neurons reduced to  568\n",
            "330  neurons reduced to  290\n",
            "164  neurons reduced to  144\n",
            "[Step 0] Validation / Test accuracy: 0.550 / 0.560.\n",
            "[Step 500] Validation / Test accuracy: 0.575 / 0.586.\n",
            "Evaluating the model at iteration 16\n",
            "856  neurons reduced to  787\n",
            "568  neurons reduced to  522\n",
            "290  neurons reduced to  255\n",
            "144  neurons reduced to  126\n",
            "[Step 0] Validation / Test accuracy: 0.555 / 0.565.\n",
            "[Step 500] Validation / Test accuracy: 0.575 / 0.586.\n",
            "Evaluating the model at iteration 17\n",
            "787  neurons reduced to  724\n",
            "522  neurons reduced to  480\n",
            "255  neurons reduced to  224\n",
            "126  neurons reduced to  110\n",
            "[Step 0] Validation / Test accuracy: 0.546 / 0.554.\n",
            "[Step 500] Validation / Test accuracy: 0.568 / 0.576.\n",
            "Evaluating the model at iteration 18\n",
            "724  neurons reduced to  666\n",
            "480  neurons reduced to  441\n",
            "224  neurons reduced to  197\n",
            "110  neurons reduced to  96\n",
            "[Step 0] Validation / Test accuracy: 0.538 / 0.544.\n",
            "[Step 500] Validation / Test accuracy: 0.568 / 0.578.\n",
            "Evaluating the model at iteration 19\n",
            "666  neurons reduced to  612\n",
            "441  neurons reduced to  405\n",
            "197  neurons reduced to  173\n",
            "96  neurons reduced to  84\n",
            "[Step 0] Validation / Test accuracy: 0.522 / 0.525.\n",
            "[Step 500] Validation / Test accuracy: 0.558 / 0.563.\n",
            "Pruning Iteration / Test accuracy: 0 / 0.626.\n",
            "Pruning Iteration / Duration (milli sec): 0 / 211.4572.\n",
            "Pruning Iteration / Test accuracy: 1 / 0.620.\n",
            "Pruning Iteration / Duration (milli sec): 1 / 210.4836.\n",
            "Pruning Iteration / Test accuracy: 2 / 0.619.\n",
            "Pruning Iteration / Duration (milli sec): 2 / 194.8404.\n",
            "Pruning Iteration / Test accuracy: 3 / 0.614.\n",
            "Pruning Iteration / Duration (milli sec): 3 / 178.2445.\n",
            "Pruning Iteration / Test accuracy: 4 / 0.614.\n",
            "Pruning Iteration / Duration (milli sec): 4 / 163.2344.\n",
            "Pruning Iteration / Test accuracy: 5 / 0.615.\n",
            "Pruning Iteration / Duration (milli sec): 5 / 152.8298.\n",
            "Pruning Iteration / Test accuracy: 6 / 0.610.\n",
            "Pruning Iteration / Duration (milli sec): 6 / 141.3319.\n",
            "Pruning Iteration / Test accuracy: 7 / 0.605.\n",
            "Pruning Iteration / Duration (milli sec): 7 / 134.9351.\n",
            "Pruning Iteration / Test accuracy: 8 / 0.603.\n",
            "Pruning Iteration / Duration (milli sec): 8 / 124.9840.\n",
            "Pruning Iteration / Test accuracy: 9 / 0.598.\n",
            "Pruning Iteration / Duration (milli sec): 9 / 120.5721.\n",
            "Pruning Iteration / Test accuracy: 10 / 0.596.\n",
            "Pruning Iteration / Duration (milli sec): 10 / 113.1131.\n",
            "Pruning Iteration / Test accuracy: 11 / 0.595.\n",
            "Pruning Iteration / Duration (milli sec): 11 / 110.7695.\n",
            "Pruning Iteration / Test accuracy: 12 / 0.596.\n",
            "Pruning Iteration / Duration (milli sec): 12 / 101.2771.\n",
            "Pruning Iteration / Test accuracy: 13 / 0.589.\n",
            "Pruning Iteration / Duration (milli sec): 13 / 98.4989.\n",
            "Pruning Iteration / Test accuracy: 14 / 0.582.\n",
            "Pruning Iteration / Duration (milli sec): 14 / 100.5615.\n",
            "Pruning Iteration / Test accuracy: 15 / 0.586.\n",
            "Pruning Iteration / Duration (milli sec): 15 / 93.5944.\n",
            "Pruning Iteration / Test accuracy: 16 / 0.586.\n",
            "Pruning Iteration / Duration (milli sec): 16 / 92.5724.\n",
            "Pruning Iteration / Test accuracy: 17 / 0.576.\n",
            "Pruning Iteration / Duration (milli sec): 17 / 91.2499.\n",
            "Pruning Iteration / Test accuracy: 18 / 0.578.\n",
            "Pruning Iteration / Duration (milli sec): 18 / 87.8025.\n",
            "Pruning Iteration / Test accuracy: 19 / 0.563.\n",
            "Pruning Iteration / Duration (milli sec): 19 / 84.7204.\n"
          ]
        }
      ],
      "source": [
        "# % removal of neurons at each layer in each pruning iteration\n",
        "# only pruning linear layers, research has shown that the deeper layers have more tendency to be sparse, so pruning deeper layers aggressively\n",
        "layer_to_sparseness_map = [\n",
        "    ('conv2_d', 0.),\n",
        "    ('conv2_d_1', 0.),\n",
        "    ('linear', .08),\n",
        "    ('linear_1', .08),\n",
        "    ('linear_2', .12),\n",
        "    ('linear_3', .12),\n",
        "    ('linear_4', 0.),\n",
        "]\n",
        "params = deepcopy(avg_params)\n",
        "layers = [x for x, _ in layer_to_sparseness_map] \n",
        "sparsity_fractions = [y for _, y in layer_to_sparseness_map]\n",
        "\n",
        "iteration = 0\n",
        "params_list = []\n",
        "# without any sparsity\n",
        "params_list.append(avg_params)\n",
        "val_accuracy = 1.0 #initialization\n",
        "while val_accuracy > desired_accuracy:\n",
        "  iteration+=1\n",
        "  \n",
        "  sparse_net_fn_partial = partial(sparse_net_fn, sparsity_fractions=sparsity_fractions, iteration=iteration)\n",
        "  net = hk.without_apply_rng(hk.transform(sparse_net_fn_partial)) \n",
        "  sparse_net_fn_params = net.init(jax.random.PRNGKey(42), next(train))\n",
        "\n",
        "  print(f\"Evaluating the model at iteration \" f\"{iteration:.0f}\")\n",
        "\n",
        "  for i in np.arange(len(layers)-1):\n",
        "    layer = layers[i]\n",
        "    if 'conv' in layer: #last layer or conv layers\n",
        "      sparse_net_fn_params[layer] = params[layer]\n",
        "      continue\n",
        "\n",
        "    layer_next = layers[i+1]\n",
        "\n",
        "    weight_in_ = params[layer]['w']\n",
        "    weight_out_ = params[layer_next]['w']\n",
        "    b_ = params[layer]['b']\n",
        "\n",
        "    b, weights_in, weights_out = l2_structured_pruning(b_, weight_in_, weight_out_, sparsity_fractions[i], algo=\"in\")\n",
        "\n",
        "    # manipulating the weights to effect pruning\n",
        "    # updating new weights in params for next iteration\n",
        "    params[layer]['w'] = sparse_net_fn_params[layer]['w'] = weights_in\n",
        "    sparse_net_fn_params[layer]['b'] = b\n",
        "    params[layer_next]['w'] = sparse_net_fn_params[layer_next]['w'] = weights_out\n",
        "    sparse_net_fn_params[layer_next]['b'] = params[layer_next]['b']\n",
        "\n",
        "  # train model\n",
        "  sparse_net_fn_avg_params = deepcopy(sparse_net_fn_params)\n",
        "  opt_state = opt.init(sparse_net_fn_params)\n",
        "\n",
        "  # fine tuning the pruned network\n",
        "  for step in range(501):\n",
        "    if step % 500 == 0:\n",
        "      val_accuracy, _ = compute_accuracy(sparse_net_fn_avg_params, next(validation))\n",
        "      test_accuracy, _ = compute_accuracy(sparse_net_fn_avg_params, next(test))\n",
        "      val_accuracy, test_accuracy = jax.device_get(\n",
        "          (val_accuracy, test_accuracy))    \n",
        "      print(f\"[Step {step}] Validation / Test accuracy: \"\n",
        "            f\"{val_accuracy:.3f} / {test_accuracy:.3f}.\")\n",
        "      \n",
        "    sparse_net_fn_params, opt_state = update(sparse_net_fn_params, opt_state, next(train))\n",
        "    sparse_net_fn_avg_params = ema_update(sparse_net_fn_params, sparse_net_fn_avg_params)\n",
        "\n",
        "  # updating params for next iteration of pruning\n",
        "  params = deepcopy(sparse_net_fn_avg_params) \n",
        "\n",
        "  # params with sparsity at pruning iteration i\n",
        "  params_list.append(sparse_net_fn_avg_params)\n",
        "\n",
        "ranks_and_accuracies = []\n",
        "ranks_and_times = []\n",
        "# measuring accuracy and latency\n",
        "for i in range(len(params_list)):\n",
        "  params_ = params_list[i]\n",
        "  # This works as expected because, the correct net gets binded to the params do to jit compiling behavior\n",
        "  test_accuracy, duration = compute_eval_metrics(params_, next(test), 50)\n",
        "  print(f\"Pruning Iteration / Test accuracy: \"\n",
        "          f\"{i:.0f} / {np.mean(test_accuracy):.3f}.\")\n",
        "  ranks_and_accuracies.append((i, np.mean(test_accuracy)))\n",
        "  print(f\"Pruning Iteration / Duration (milli sec): \"\n",
        "          f\"{i:.0f} / {np.mean(duration)*1000:.4f}.\")\n",
        "  ranks_and_times.append((i, np.mean(duration)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "6xCyyfHOotd0",
        "outputId": "5c79edbe-9890-4299-e4c9-9a9f4e765ac5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Time taken for infence in milli-seconds')"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5RcVZn38e8v3DTcQkxASOg0IDILb4gtctERBBEQiOJlcDUKyEtGBQUdR2H61aBrZY03cAadAVqJAe2Xm4AEB0bCTdQxYgcRAnLJQAKJgQTBgEQCMc/7x9ndqTRd1aer6lRVV/8+a9U6dfY5VeepSnee3peztyICMzMzgAnNDsDMzFqHk4KZmQ1yUjAzs0FOCmZmNshJwczMBm3e7ABqMWXKlOjs7Gx2GGZmY8qiRYueioipwx0b00mhs7OT/v7+ZodhZjamSFpW7pibj8zMbJCTgpmZDXJSMDOzQU4KZmY2yEnBzMwGjbuk0NcHnZ0wYUK27etrdkRmZq1jTA9JHa2+Ppg1C9auzfaXLcv2Abq7mxeXmVmrGFc1hZ6ejQlhwNq1WXlermmYWTsbVzWFxx4bXflQrmmYWbsbVzWFjo7RlQ9Vj5qGmVkrG1dJYc4cmDhx07KJE7PyPGqtaZiZtbpxlRS6u6G3F2bMACnb9vbmb/qptaZhZtbqxlVSgCwBLF0KGzZk29H0BdRa0zAza3UjJgVJ35C0naQtJN0iabWkExoRXKuptaZhZtbq8tQUDo+IZ4GjgaXAa4B/LjKoVlZLTcPMrNXlSQoDw1bfC1wVEWsKjMfMzJooz30KP5X0APBX4JOSpgIvFBuWmZk1w4g1hYg4CzgQ6IqIl4C1wMyiAzMzs8YrW1OQdNwwZaW71xQRkJmZNU+l5qNj0nZHsprCrWn/EOB/cFIwM2s7ZZNCRJwMIOkmYO+IWJn2dwbmNSQ6MzNrqDyjj3YdSAjJk4Dv4TUza0N5Rh/dIulnwGVp/x+Am4sLyczMmiXP6KPTgYuAN6VHb0R8eqTXSdpV0m2S7pd0n6QzUvlkSQskPZy2O6RySTpf0hJJ90jat7aPZmZmo5VrPYWIuIbRdyyvB/4pIu6StC2wSNIC4CTgloj4mqSzgLOALwJHAnumx9uAC9LWzMwaJM/cR8elv+rXSHpW0nOSnh3pdRGxMiLuSs+fA/4ATCO7x+GSdNolwPvS85nApZFZCExKndpmZtYgeTqavwEcGxHbR8R2EbFtRGw3motI6gTeDPwG2Kmk4/oJYKf0fBrweMnLlqeyoe81S1K/pP7Vq1ePJgwzMxtBnqTwZET8odoLSNoGuBo4M02sNygiAojRvF9E9EZEV0R0TZ06tdqwzMxsGHn6FPolXQH8BFg3UJj6GSqStAVZQugrOf9JSTtHxMrUPLQqla8Adi15+fRUZmZmDZKnprAd2XxHh5Pd5XwM2TTaFSmbE+Ni4A8RcV7JofnAien5icB1JeUfS6OQ9gfWDLk/wszMCjZiTWHgzuYqHAR8FLhX0t2p7F+ArwFXSjoFWAZ8OB27ATgKWEKWhKq9rpmZVSnP6KPpkq6VtCo9rpY0faTXRcQvI0IR8caI2Cc9boiIP0XEoRGxZ0QcFhFPp/MjIk6LiD0i4g0R0V+PD9hq+vqgsxMmTMi2fX3NjsjMbKM8zUc/IGva2SU9rk9lNkp9fTBrFixbBhHZdtYsJwYzax15ksLUiPhBRKxPj3mAh/1UoacH1q7dtGzt2qzczKwV5EkKf5J0gqTN0uME4E9FB9aOHntsdOVmZo2WJyl8nKwz+AlgJfBB3AlclY4yc8uWKzcza7Q8E+Iti4hjI2JqROwYEe+LCP9tW4U5c2DixE3LJk7Mys3MWkGe0UeXSJpUsr+DpLnFhtWeuruhtxdmzAAp2/b2ZuVmZq0gzx3Nb4yIPw/sRMQzkt5cYExtrbvbScDMWleePoUJA2seQLYeAjmn3DYzs7Elz3/u5wK/lnRV2v8Q4FZwM7M2lGeai0sl9QPvSkXHRcT9xYZlZmbNkKf5CGAy8HxEfBdYLWm3AmMyM7MmyTP6aDbZcplnp6ItgB8VGZSZmTVHnprC+4FjgecBIuKPwLZFBmVmZs2RJym8WLpCmqStiw3JzMyaJU9SuFLSRcAkSacCNwPfKzYsMzNrhjyjj74l6d3As8BewJcjYkHhkZmZWcONmBRSc9GtEbFA0l7AXpK2iIiXig/PzMwaKU/z0R3AVpKmAf9NtsTmvCKDMjOz5siTFBQRa4HjgAsi4kPA64oNy8zMmiFXUpB0ANAN/Fcq26y4kMzMrFnyJIUzyG5cuzYi7pO0O3BbsWGZmVkz5Bl9dAdZvwKSXh0RjwCfKTowMzNrvLxzHw24Ie+JkuZKWiVpcUnZPpIWSrpbUr+k/VK5JJ0vaYmkeyTtO8q4zMysDkabFDSKc+cBRwwp+wbwlYjYB/hy2gc4EtgzPWYBF4wyLjMzq4PRJoXcdzKnZqenhxYD26Xn2wN/TM9nApdGZiHZ3dM7jzI2MzOrUdmkIGm7tJ088AAuL3lejTOBb0p6HPgWG2denQY8XnLe8lQ2XFyzUtNT/+rVq6sMY+zq64POTpgwIdv29TU7IjNrJ5U6mv8fcDSwiOwvfA3Z7l7F9T4JfDYirpb0YeBi4LDRvEFE9AK9AF1dXVFFDGNWXx/MmgVr12b7y5Zl++B1n82sPsrWFCLi6LTdLSJ2H7qt8nonAtek51cB+6XnK4BdS86bnsqsRE/PxoQwYO3arNzMrB7K1hRGGgEUEXdVcb0/Au8Ebidb3vPhVD4fOF3S5cDbgDURsbKK929rjz02unIzs9Gq1Hx0boVjwcY1m4cl6TLgYGCKpOXAbOBU4N8lbQ68QDbSCLKhrkcBS4C1wMl5gh9vOjqyJqPhys3M6qFsUoiIQ2p544j4SJlDbxnm3ABOq+V648GcOZv2KQBMnJiVm5nVQ6Xmo+MqvTAirql03OpvoDO5pydrMuroyBKCO5nNrF4qNR8dU+FYsLHD2Bqou9tJwMyKU6n5yO36ZmbjTKXmoxMi4keSPjfc8Yg4r7iwzMysGSo1H22dtts2IhAzM2u+Ss1HF6XtVxoXjpmZNdOI6ylI2g34NNBZen5EHFtcWGZm1gwjJgXgJ2RzFF0PbCg2HDMza6Y8SeGFiDi/8EjMzKzp8iSFf5c0G7gJWDdQWOXcR2Zm1sLyJIU3AB8lm+tooPloxLmPzMxs7MmTFD4E7B4RLxYdjJmZNVee5TgXA5OKDsTMzJovT01hEvCApN+yaZ+Ch6SambWZPElhduFRmJlZSxgxKUTEzxsRiJmZNV+ePgUzMxsnnBTGmb4+6OyECROybV9fsyMys1aSp0/B2kRf36bLeS5blu2DF+4xs8yINQVJB0laIOkhSY9IelTSI40Izuqrp2fT9Z0h2+/paU48ZtZ68tQULgY+CywC/lZsOFakxx4bXbmZjT95ksKaiLix8EiscB0dWZPRcOVmZpCvo/k2Sd+UdICkfQceI71I0lxJqyQtHlL+aUkPSLpP0jdKys+WtETSg5LeU8VnsRHMmQMTJ25aNnFiVm5mBvlqCm9L266SsjwT4s0DvgtcOlAg6RBgJvCmiFgnacdUvjdwPPA6YBfgZkmvjQg3V9XRQGdyT0/WZNTRkSUEdzKb2YA8N68dUs0bR8QdkjqHFH8S+FpErEvnrErlM4HLU/mjkpYA+wG/rubaVl53t5OAmZVXNilIOiEifiTpc8Mdj4jzqrjea4F3SJoDvAB8PiJ+C0wDFpactzyVmZlZA1WqKWydttvW+XqTgf2BtwJXStp9NG8gaRYwC6DDPaRmZnVVNilExEVp+5U6Xm85cE1EBHCnpA3AFGAFsGvJedNT2XBx9QK9AF1dXVHH2MzMxr1GT3PxE+AQAEmvBbYEngLmA8dL2krSbsCewJ0Njs3MbNwrbJoLSZcBBwNTJC0nm4J7LjA3DVN9ETgx1Rruk3QlcD+wHjjNI4/MzBpP2f/JY1NXV1f09/c3OwwzszFF0qKI6BruWJ65j3aSdLGkG9P+3pJOqXeQZmbWfHn6FOYBPyO7qQzgIeDMogIyM7PmyZMUpkTElcAGgIhYjyfGMzNrS3mSwvOSXkU2tQWS9gfWFBqVtTQv1GPWvvKMPvoc2ZDRPST9CpgKfLDQqKxleaEes/aWa/SRpM2BvQABD0bES0UHlodHHzVeZ+fw02/PmAFLlzY6GjOrRq2jj04DtomI+yJiMbCNpE/VO0gbG7xQj1l7y9OncGpE/HlgJyKeAU4tLiRrZeWmm/I0VGbtIU9S2EySBnYkbUY2PYWNQ16ox6y95UkK/w1cIelQSYcCl6UyG4e6u6G3N+tDkLJtb687mc3axYgdzZImAP8IHJqKFgDfb4W5idzRbGY2epU6mvOsvLYBuCA9zMysjY2YFCQdBJwDzEjnC4iIGNXiOGZm1vry3Lx2MfBZYBGe3sLMrK3lSQprIuLGwiMxM7Omy5MUbpP0TeAaYN1AYUTcVVhUZmbWFHmSwtvStrSnOoB31T8cMzNrpjyjjw5pRCBmZtZ8XnnNzMwGeeU1MzMb5JXXzMxskFdes4bzym1mrcsrr1lDeeU2s9ZWtqYg6UPp6TPAO4EDySbGe11E3DPSG0uaK2mVpMXDHPsnSSFpStqXpPMlLZF0j6R9q/o01vJ6ejYmhAFr12blZtZ8lZqPzk7bqyNi/cDKa6NYinMecMTQQkm7AocDpWt1HQnsmR6z8OR7bcsrt5m1tkrNR3+SdBOwm6T5Qw9GxLGV3jgi7pDUOcyhbwNfAK4rKZsJXBrZPN4LJU2StHNErBzpA9jY0tEx/BrPXrnNrDVUSgrvBfYFfgicW4+LSZoJrIiI35cs5gYwDXi8ZH95KntZUpA0i6w2QYf/Jxlz5szZtE8BvHKbWSspmxQi4kWyv9oPjIjVtV5I0kTgX8iajqoWEb1AL2SL7NQalzXWQGdyT0/WZNTRkSUEdzKbtYY8o492kDQH6Cw9PyJGO/fRHsBuwEAtYTpwl6T9gBXAriXnTk9l1oa6u50EzFpVnqRwFXAh8H1quGktIu4FdhzYl7QU6IqIp1KfxemSLiebgG+N+xPMzBovT1JYHxGjHg0k6TLgYGCKpOXA7Ii4uMzpNwBHAUuAtcDJo72emZnVLs8dzddL+pSknSVNHniM9KKI+EhE7BwRW0TE9KEJISI6I+Kp9Dwi4rSI2CMi3hAR/VV+HhsHfEe0WXHy1BROTNt/LikLwGs0W8P5jmizYim7NWBs6urqiv5+VyrGk87O4e9zmDEDli5tdDRmY5OkRRHRNdyxsjUFSe+KiFslHTfc8Yi4pl4BmuXlO6LNilWp+eidwK3AMcMcC7I1m80ayndEmxWr0s1rs9PWI4GsZfiOaLNi5Rl9ZNYyuruhtzfrQ5CybW+vO5nN6iXP6COzluI7os2KU7GmIGmCpAMbFYyZmTVXxaQQERuA/2hQLGZm1mR5+hRukfQBDZnr2szM2k+epPCPZJPivSjpWUnPSXq24LjMzKwJRuxojohtGxGImZk1X67RR5KOBf4+7d4eET8tLiQzM2uWEZuPJH0NOAO4Pz3OkPSvRQdmZmaNl6dP4Sjg3RExNyLmAkeQrd9sNiZ56m2z8vLevDYJeDo9376gWMwK56m3zSrLU1P4V+B3kuZJugRYBHimGRuTeno2nTcJsv2enubEY9ZqKk2dfVBE/IpsNtTbgbemQ1+MiCcaEJtZ3XnqbbPKKtUUzk/bX0fEyoiYnx5OCDZmlZti21Nvm2Uq9Sm8JKkXmC7p/KEHI+IzxYVlVgxPvW1WWaWkcDRwGPAesn4EszFvoDO5pydrMuroyBKCO5nNMiOu0SzpTRHx+wbFMypeo9maoa/PScXGtkprNI84+qjahCBprqRVkhaXlH1T0gOS7pF0raRJJcfOlrRE0oOS3lPNNc2KNjCkddkyiNg4pNX3Oli7KHLltXlkN7qVWgC8PiLeCDwEnA0gaW/geOB16TX/KWmzAmMzq4qHtFq7KywpRMQdbLzhbaDspohYn3YXAtPT85nA5RGxLiIeBZYA+xUVm1m1PKTV2t2IdzRL2gr4ANBZen5EfLXGa38cuCI9n0aWJAYsT2XDxTMLmAXQ4XGE1mAdHVmT0XDlZu0gT03hOrK/5NcDz5c8qiapJ73fqFtiI6I3Iroiomvq1Km1hGE2anPmZENYS3lIq7WTPHMfTY+IoX0DVZN0Etlw10Nj49CnFcCupddMZWYtxUNard3lqSn8j6Q31ONiko4AvgAcGxGl3XXzgeMlbSVpN2BP4M56XNOs3rq7YelS2LAh2zohWDvJU1N4O3CSpEeBdYCASCOIypJ0GXAwMEXScmA22WijrYAFacnnhRHxiYi4T9KVZOs1rAdOi4i/VfmZzMysSnluXpsxXHlEDNPd1li+ec3MbPRqvXltGVl7/7vS87V5XmdmZmNPnuU4ZwNfJN1oBmwB/KjIoMzMrDny/MX/fuBY0jDUiPgjsG2RQZm1My8Haq0sT0fzixERkgJA0tYFx2TWtrwcqLW6PDWFKyVdBEySdCpwM/D9YsMya0+eO8laXZ6kcC7wY+BqYC/gy8BFRQZl1q7qMXeSm5+sSHmajy6OiI+TzXCKpG2AG4BDiwzMrB3VOneSm5+saHlqCisk/SeApB2Am/DoI7Oq1Dp3kpufrGh57lP4EvAXSReSJYRzI+IHhUdm1oa6u6G3F2bMACnb9vbm/yvfU3db0co2H0k6rmT3N8CXyOYjCknHRcQ1RQdn1o66u6tv6vHU3Va0Sn0KxwzZ/x3ZjWvHAAE4KZg12Jw5m/YpgKfutvoqmxQi4uRGBmJmI/PU3Va0PCuvvQI4hWz95FcMlKcRSWbWYLU0P5mNJM/oox8CrwbeA/ycbAGc54oMyszMmiNPUnhNGoH0fERcArwXeFuxYZmZWTPkSQovpe2fJb0e2B7YsbiQzMysWfLc0dybblr7v2TLZm5DNjzVzMzaTJ6kcEtEPAPcAewOkNZRNjOzNpOn+ejqYcp+XO9AzMys+Srd0fx3ZMNQtx9yd/N2lAxNNTOz9lGpprAXcDQwiewu5oHHvsCpxYdmZkXw1NtWSaU7mq8DrpN0QET8uoExmVlBWmHq7b4+35HdyvLMklpVQpA0V9IqSYtLyiZLWiDp4bTdIZVL0vmSlki6R9K+1VzTzCpr9tTbA0lp2TKI2JiUXFtpHXk6mqs1DzhiSNlZZKOZ9gRuSfsARwJ7pscs4IIC4zIbt5o99Xazk5KNrLCkEBF3AE8PKZ4JXJKeXwK8r6T80sgsJFsPeueiYjMbr8pNsT2aqbdr6ZNodlKykY2YFCTtJOliSTem/b0lnVLl9XaKiJXp+RPATun5NODxkvOWp7Lh4pklqV9S/+rVq6sMw2x8qnXlt1qbf+qRlKxYeWoK84CfAbuk/YeAM2u9cEQE2boMo31db0R0RUTX1KlTaw3DbFypdeW3Wpt/ak1KVrw8SWFKRFwJbACIiPXA36q83pMDzUJpuyqVrwB2LTlveiozszrr7oalS2HDhmw7mpE/tTb/1JqUrHh5ksLzkl5F+qte0v7AmiqvNx84MT0/EbiupPxjaRTS/sCakmYmM2sR9Wj+qSUpWfHyJIXPkf2nvYekXwGXAp8e6UWSLgN+DewlaXnqh/ga8G5JDwOHpX2AG4BHgCXA94BPjfaDmFnx3PzT/kacEC8i7pL0TrI7nAU8GBEvjfAyIuIjZQ4dOsy5AZw20nuaWXN5OdD2p+z/4wonSJuRLazTSUkSiYjzCo0sh66urujv7292GGZmY4qkRRHRNdyxPM1H1wMnAa8Cti15mJk1nOduKlae9RSmR8QbC4/EzGwErTB3U7vLU1O4UdLhhUdiZjaCekyT4ZpGZXlqCguBayVNIFuvWWR9w9sVGpmZ2RC13ifhmsbI8tQUzgMOACZGxHYRsa0Tgpk1Q633SbimMbI8SeFxYHGMNEzJzKxgtd4nUa+aRjtP/Z0nKTwC3C7pbEmfG3gUHZiZ2VC1TpPRCjWNVpcnKTxKtvbBlnhIqpk1WS3TZDS7pjEW5Lmj+SuNCMTMrGi13pHd0ZE1GQ1X3i7K1hQkfTdtr5c0f+ijcSGamdVPM2sa0Pod1ZVqCh8DTge+1aBYzMxaWq01jbEwJLbs3EeSfhcRb25wPKPiuY/MbCzp7By++WnGjKzW0iiV5j6qVFOYWmmUUStMiGdmNpaMhY7qSklhM2AbsjuYzcysRmOho7pSUlgZEV9tWCRmZm1uzpxN+xSguo7qItezqHSfgmsIZmZ1VOvNd424o7pSR/PkiHi6fpeqP3c0m9l4Uq+O6qoW2Wn1hGBmNt40oqM6zzQXZmbWAmqduykPJwUzszGiHndUj8RJwcxsjKi1ozqPPCuv1Z2kzwL/BwjgXuBkYGfgcuBVwCLgoxHxYjPiMzNrVd3dxU6J0fCagqRpwGeAroh4PdlNcscDXwe+HRGvAZ4BTml0bGZm412zmo82B14paXNgIrASeBfw43T8EuB9TYrNzGzcanhSiIgVZDOvPkaWDNaQNRf9OSLWp9OWA9OGe72kWZL6JfWvXr26ESGbmY0bzWg+2gGYCewG7AJsDRyR9/UR0RsRXRHRNXXq1IKiNDMbn5rRfHQY8GhErI6Il4BrgIOASak5CWA6sKIJsZmZjWvNGH30GLC/pInAX4FDgX7gNuCDZCOQTgSuG+mNFi1a9JSkYW76zmUK8FSVr22EVo8PWj9Gx1cbx1ebVo5vRrkDZec+KpKkrwD/AKwHfkc2PHUaWUKYnMpOiIh1BcbQX27uj1bQ6vFB68fo+Grj+GrT6vGV05T7FCJiNjB7SPEjwH5NCMfMzBLf0WxmZoPGc1LobXYAI2j1+KD1Y3R8tXF8tWn1+IbVlD4FMzNrTeO5pmBmZkM4KZiZ2aC2TwqSjpD0oKQlks4a5vhWkq5Ix38jqbOBse0q6TZJ90u6T9IZw5xzsKQ1ku5Ojy83Kr50/aWS7k3Xftnap8qcn76/eyTt28DY9ir5Xu6W9KykM4ec0/DvT9JcSaskLS4pmyxpgaSH03aHMq89MZ3zsKQTGxjfNyU9kP4Nr5U0qcxrK/48FBjfOZJWlPw7HlXmtRV/3wuM74qS2JZKurvMawv//moWEW37IJuB9X+B3YEtgd8Dew8551PAhen58cAVDYxvZ2Df9Hxb4KFh4jsY+GkTv8OlwJQKx48CbgQE7A/8pon/1k8AM5r9/QF/D+wLLC4p+wZwVnp+FvD1YV43mWxo9mRgh/R8hwbFdziweXr+9eHiy/PzUGB85wCfz/EzUPH3vaj4hhw/F/hys76/Wh/tXlPYD1gSEY9EtjbD5WTzLpWaSTYrK2SztB4qSY0ILiJWRsRd6flzwB8oMxFgC5sJXBqZhWTTlezchDgOBf43Iqq9w71uIuIOYOga56U/Z+VmAX4PsCAino6IZ4AFjGJesFrii4ibYuOElAvJppppijLfXx55ft9rVim+9H/Hh4HL6n3dRmn3pDANeLxkf7jZVwfPSb8Ua8gW+mmo1Gz1ZuA3wxw+QNLvJd0o6XUNDSxbCOkmSYskzRrmeJ7vuBGOp/wvYjO/vwE7RcTK9PwJYKdhzmmV7/LjZLW/4Yz081Ck01Pz1twyzW+t8P29A3gyIh4uc7yZ318u7Z4UxgRJ2wBXA2dGxLNDDt9F1iTyJuA7wE8aHN7bI2Jf4EjgNEl/3+Drj0jSlsCxwFXDHG729/cykbUjtORYcEk9ZNPP9JU5pVk/DxcAewD7kE25f26DrjtaH6FyLaHlf5/aPSmsAHYt2R9u9tXBc5TN0ro98KeGRJddcwuyhNAXEdcMPR4Rz0bEX9LzG4AtJE1pVHyRrX9BRKwCruXlU5Hk+Y6LdiRwV0Q8OfRAs7+/Ek8ONKul7aphzmnqdynpJOBooDslrpfJ8fNQiIh4MiL+FhEbgO+VuW6zv7/NgeOAK8qd06zvbzTaPSn8FthT0m7pr8njgflDzplPNisrZLO03lruF6LeUvvjxcAfIuK8Mue8eqCPQ9J+ZP9mDUlakraWtO3Ac7LOyMVDTpsPfCyNQtofWFPSTNIoZf86a+b3N0Tpz1m5WYB/BhwuaYfUPHJ4KiucpCOALwDHRsTaMufk+XkoKr7Sfqr3l7lunt/3Ih0GPBARy4c72Mzvb1Sa3dNd9INsdMxDZKMSelLZV8l++AFeQdbssAS4E9i9gbG9nawZ4R7g7vQ4CvgE8Il0zunAfWQjKRYCBzYwvt3TdX+fYhj4/krjE/Af6fu9l2zt7Ub++25N9p/89iVlTf3+yBLUSuAlsnbtU8j6qW4BHgZuBianc7uA75e89uPpZ3EJcHID41tC1h4/8HM4MCJvF+CGSj8PDYrvh+nn6x6y/+h3Hhpf2n/Z73sj4kvl8wZ+7krObfj3V+vD01yYmdmgdm8+MjOzUXBSMDOzQU4KZmY2yEnBzMwGOSmYmdkgJwUbkyT9Lc00uVjSVZIm1ul9uySdX6f3OkfS59PzkyTtUo/3Te93sKQDS/Y/Ielj9Xp/G7+cFGys+mtE7BMRrwdeJLs3YVC6u3TUIqI/Ij5TjwCHOIlszHpuI3yGg4HBpBARF0bEpVVFZlbCScHawS+A16S/nn8haT5wv6TOIXPef17SOen57ZK+LulOSQ9JekcqP1jST9Pzc9Lka7dLekTSZ0re60tp3v5fSrpsoEYwHEkfJLtJrS/Vbl4p6S2Sfp4mRvtZyRQYt0v6tzTX/hmSjlG2zsfvJN0saac0eeIngM+m93vHkFrJPpIWauPaCDtU+sxmpZwUbExLf00fSXa3K2Tz3J8REa/N8fLNI2I/4Exgdplz/o5sSuv9gNmStpD0VuADwJvStbsqXSQifgz0k80ptA/ZhHPfAT4YEW8B5gJzSl6yZUR0RcS5wC+B/SPizWRTQX8hIpYCFwLfTrWlXwy55KXAFyPijWTfS+lny/OZbRyrqopt1gJeqY2rW/2CbA6pA4E7I+LRnO8xMAHhIqCzzDn/FRHrgHWSVpFNeX0QcF1EvAC8IOn6Uca+F/B6YEGalmkzsmkTBpROqDYduCLVJLYEKkWOQW0AAAFNSURBVH42SdsDkyLi56noEjadPTbPZ7ZxzEnBxqq/pr+6B6X/YJ8vKVrPprXhVwx5j3Vp+zfK/y6sK3le6bzREHBfRBxQ5njpZ/gOcF5EzJd0MNkKZLXI85ltHHPzkbWzJ4EdJb1K0lZk00LXw6+AYyS9QtlaGHne9zmyJVcBHgSmSjoAsunTVX7xn+3ZOP1z6ZrNpe83KCLWAM+U9Bd8FPj50PPMyvFfCta2IuIlSV8lm/12BfBAnd73t6kz+x6yxHMv2Yp9lcwDLpT0V+AAsmnaz0/NPZsD/0Y2c+ZQ5wBXSXoGuBXYLZVfD/xY0kzg00Nec2K61kSydZ5PHtUHtHHNs6SaVUHSNhHxl/Qf7x3ArEjrbZuNZa4pmFWnV9LeZP0UlzghWLtwTcHMzAa5o9nMzAY5KZiZ2SAnBTMzG+SkYGZmg5wUzMxs0P8HVeoDPsxbqvkAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "x = [x for x, _ in ranks_and_times]\n",
        "y = [y*1000 for _, y in ranks_and_times]\n",
        "\n",
        "plt.plot(x, y, 'bo')\n",
        "plt.xlabel(\"Pruning Iteration\")\n",
        "plt.ylabel(\"Time taken for infence in milli-seconds\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "ckvMf16PouYA",
        "outputId": "b246481e-f03d-4d8f-a844-83510ff13c0b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Accuracy')"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAbmUlEQVR4nO3df7QdZX3v8feHhAgRFWiiRZKcg20Q0WIKp1miYqkUDFqTWrwYeqrEVnJZbUTuqrbpOqvKyl25a6mrrRfKKutAuWJXgACtNlhpiAiCXMGcYIgkCISYQHIpHAPyw8iPkO/9Y2Ynw87sc+acvWf//LzW2mvveWZmz3fP2Xu+Z+Z55nkUEZiZmVU7pNUBmJlZe3KCMDOzXE4QZmaWywnCzMxyOUGYmVmuqa0OoFFmzJgR/f39rQ7DzKyjbNiw4ecRMTNvXtckiP7+fkZGRlodhplZR5G0o9Y8X2IyM7NcThBmZpbLCcLMzHI5QZiZWS4nCDMzy9XzCWLVKujvh0MOSZ5XrWp1RGZm7aFrmrlOxqpVsHQp7NmTTO/YkUwDDA62Li4zs3bQ02cQQ0MHkkPFnj1JuZlZr+vpBPHYYxMrNzPrJT2dIObMmVi5mVkv6ekEsXIlTJ/+2rLp05NyM7Ne19MJYnAQhoehrw+k5Hl42BXUZmbQ462YIEkGTghmZgfr6TMIMzOrzQnCzMxyOUGYmVkuJwgzM8vlBGFmZrmcIMzMLJcThJmZ5XKCMDOzXE4QZmaWywnCzMxyOUGYmVkuJ4g6echSM+tWpSYISQskPSRpq6TlNZY5V9IWSZslXZuWzZP0w7Rsk6RPlBnnZFWGLN2xAyIODFnqJGFm3UARUc4bS1OAh4EzgZ3AeuC8iNiSWWYucAPwwYh4RtKbI+IpSccDERGPSHorsAF4R0T8otb2BgYGYmRkpJTPUkt/f5IUqvX1wfbtTQ3FzGxSJG2IiIG8eWWeQcwHtkbEtoh4GbgeWFS1zAXA5RHxDEBEPJU+PxwRj6Sv/x/wFDCzxFgnxUOWmlk3KzNBHAs8npnemZZlHQ8cL+luSfdIWlD9JpLmA9OAR3PmLZU0ImlkdHS0gaEX4yFLzaybtbqSeiowFzgdOA+4UtKRlZmSjgH+Bfh0ROyrXjkihiNiICIGZs5s/glGI4YsdSW3mbWrMhPELmB2ZnpWWpa1E1gTEa9ExM9I6izmAkh6I/AfwFBE3FNinJNW75ClruQ2s3ZWZiX1VJID/hkkiWE98McRsTmzzAKSiuvzJc0AfgzMA54HbgFujoivFdleKyqp6+VKbjNrtZZUUkfEXmAZsBZ4ELghIjZLWiFpYbrYWmC3pC3A7cAXImI3cC7wAWCJpI3pY15ZsbaKK7nNrJ2VdgbRbD6DMDObuFY1c7VxNKKS28ysLE4QLVRvJbeZWZmmtjqAXjc46IRgZu3JZxBmZpbLCcLMzHI5QZiZWS4nCDMzy+UEYWZmuZwgrC7ubNCsezlB9Lh6DvDubNCsuzlBdLhWHuCHhmDPnteW7dmTlJtZ53NfTB2scoDPHqSnTy9+N3a9fUEdckiSWKpJsO+g0TvMrB25L6YuVe9/8PX2JusR9cy6mxNEB2v1Ad6dDZp1NyeIDtbqA7w7GzTrbk4QHawdDvCDg0l9xb59ybOTg1n3cG+uHaxyMB4aSi4rzZmTJIeJHuB9UDezPE4QHc4HeDMriy8xmZlZLicIMzPL5QRhZma5nCCspdzZn1n7KjVBSFog6SFJWyUtr7HMuZK2SNos6dpM+X9K+oWkb5cZo7WOO/sza2+lJQhJU4DLgbOBE4HzJJ1Ytcxc4G+A90XEO4GLM7O/CnyyrPis9dzZn1l7K/MMYj6wNSK2RcTLwPXAoqplLgAuj4hnACLiqcqMiLgNeL7E+KzF6u0qxMzKVWaCOBZ4PDO9My3LOh44XtLdku6RtGAiG5C0VNKIpJHR0dE6w7Vmc2d/Zu2t1ZXUU4G5wOnAecCVko4sunJEDEfEQEQMzJw5s6QQrSzu7M+svZWZIHYBszPTs9KyrJ3Amoh4JSJ+BjxMkjCsB7izP7P2VmaCWA/MlXScpGnAYmBN1TLfIjl7QNIMkktO20qMydqMO/sza1+lJYiI2AssA9YCDwI3RMRmSSskLUwXWwvslrQFuB34QkTsBpB0F3AjcIaknZI+VFasZmZ2MA85ambWwzzkqJmZTZgThJmZ5XKCMDOzXE4QZmaWywnCzMxyOUGYmVkuJwjraB5Pwqw8U1sdgNlkVcaTqHQZXhlPAnxHtlkj+AzCOpbHkzArlxOEdSyPJ2FWLicI61geT8KsXE4Q1rE8noRZuZwgrGN5PAmzcrkVk3W0wUEnBLOy+AzCeprvozCrzWcQ1rN8H4XZ2HwGYT3L91GYjc0JwnqW76MwG5sThPUs30dhNjYnCOtZvo/CbGxOENazfB+F2djcisl6mu+jMKtt3DMISR+V5DMNM7MeU+TA/wngEUlfkXTCRN5c0gJJD0naKml5jWXOlbRF0mZJ12bKz5f0SPo4fyLbNTOz+o17iSki/kTSG4HzgK9LCuD/ANdFxPO11pM0BbgcOBPYCayXtCYitmSWmQv8DfC+iHhG0pvT8qOBLwEDQAAb0nWfmewHNTOziSl06SgingNuAq4HjgE+Btwn6bNjrDYf2BoR2yLi5XTdRVXLXABcXjnwR8RTafmHgHUR8XQ6bx2woOBnMjOzBihSB7FQ0jeBO4BDgfkRcTbwbuAvx1j1WODxzPTOtCzreOB4SXdLukfSggmsi6SlkkYkjYyOjo73UczMbAKKtGI6B/iHiLgzWxgReyT9WQO2Pxc4HZgF3Cnpt4quHBHDwDDAwMBA1BmLmZllFLnEdAnwo8qEpMMl9QNExG1jrLcLmJ2ZnpWWZe0E1kTEKxHxM+BhkoRRZF0zMytRkQRxI7AvM/1qWjae9cBcScdJmgYsBtZULfMtkrMHJM0gueS0DVgLnCXpKElHAWelZWZtxd2FWzcrcolpalrJDEBEvJwe8McUEXslLSM5sE8Bro6IzZJWACMRsYYDiWALSeL5QkTsBpD0P0mSDMCKiHh6Qp/MrGTuLty6nSLGvnQvaR1wWXpAR9Ii4KKIOKMJ8RU2MDAQIyMjrQ7Dekh/f5IUqvX1wfbtzY7GbHIkbYiIgbx5Rc4gLgRWSfpHQCStiz7VwPjMOpK7C7duV+RGuUeB90g6Ip1+ofSozDrAnDn5ZxDuLty6RaHO+iR9BHgncJgkACJiRYlxmbW9lStfWwcB7i7cukuRG+WuIOmP6bMkl5j+G9BXclxmba8R3YW3uhVUq7dv7a1IJfWmiDgp83wEcEtEnNacEItxJbV1mupWUJCcgTRrTIpWb9/aw1iV1EXug3gxfd4j6a3AKyT9MZlZHYaGXntwhmR6aKg3tm/tr0gdxM2SjgS+CtxH0rvqlaVGZdYDWt0KqtXbt/Y35hlEOlDQbRHxi4j4V5K6hxMi4otNic6si9Vq7TSRVlD11CE0YvvW3cZMEBGxj2RMh8r0SxHxbOlRmfWAlSuTa/5ZE2kFValD2LEDIg7cyV00SdS7fet+ReogbpN0jirtW82sIeptBVVvHUIjWmFZdyvSiul54PXAXpIKawEREW8sP7zi3IrJes0hhyRnDtUk2Lfv4HKzPHV1tRERb2h8SGZWL9/JbWUbN0FI+kBeefUAQmbWXL6T28pWpJnrFzKvDyMZa3oD8MFSIjKzQip1BUNDSdPUOXOS5OA6BGuUIpeYPpqdljQb+FppEZlZYYODTghWniKtmKrtBN7R6EDMzKy9FKmDuIzk7mlIEso8kjuqzcysixWpg8i2Hd0LXBcRd5cUj5mZtYkiCeIm4MWIeBVA0hRJ0yNizzjrmZlZByt0JzVweGb6cOC75YRjZmbtokiCOCw7zGj6evoYy5uZWRcokiB+KenkyoSkU4BflReSmZm1gyIJ4mLgRkl3SfoBsBpYVuTNJS2Q9JCkrZKW58xfImlU0sb08ZnMvC9LeiB9fKLoBzIzs8YocqPcekknAG9Pix6KiFfGW0/SFJKuws8kuXdivaQ1EbGlatHVEbGsat2PACeTNKl9HXCHpFsi4rlxP5GZmTXEuGcQkv4CeH1EPBARDwBHSPrzAu89H9gaEdsi4mXgemBRwbhOBO6MiL0R8UtgE7Cg4LpmZtYARS4xXRARv6hMRMQzwAUF1jsWeDwzvTMtq3aOpE2Sbkq78QC4H1ggabqkGcDvAbOrV5S0VNKIpJHR0dECIZmZWVFFEsSU7GBB6aWjaQ3a/s1Af0ScBKwDrgGIiFuB7wD/F7gO+CHwavXKETEcEQMRMTBz5swGhWRmZlAsQfwnsFrSGZLOIDlg31JgvV289r/+WWnZfhGxOyJeSievAk7JzFsZEfMi4kySQYoeLrBNM2uiesbEtvZX5E7qvwaWAhem05uAXy+w3npgrqTjSBLDYuCPswtIOiYinkgnFwIPpuVTgCMjYrekk4CTgFsLbNPMmqQyJnZlPIrKmNjgHma7xbhnEBGxD7gX2E5S8fxB0gP5OOvtJWkOuzZd/oaI2CxphaSF6WIXSdos6X7gImBJWn4ocJekLcAw8Cfp+5lZm6h3TGxrfzXHpJZ0PHBe+vg5yf0Pn4+IvuaFV5zHpDZrLo+J3R3GGpN6rDOIn5KcLfxBRLw/Ii4jp6LYzHpTrbGvPSZ29xgrQfwR8ARwu6Qr0wpqjbG8mfWQlSuTMbCzPCZ2d6mZICLiWxGxGDgBuJ2ky403S/onSWc1K0Aza0+DgzA8DH19yWWlvr5kupkV1G5FVa4ildS/jIhr07GpZwE/JmnZZGY9bnAQtm9P6hy2b29+cli6NGk9FXGgFdVEkoQTzNhqVlJ3GldSm/WW/v4kKVTr60uS1Xiqm+lCcoms2WdBrTbZSmozs7b12GMTK6/mZrrjc4Iws45UbyuqehNML3CCMLOOVG8rKjfTHZ8ThJl1pHpbUbmZ7viK9MVkZtaWBgcnX6FcWW9oKLmsNGdOkhx6qYJ6PE4QZtaz6kkwvcCXmMzMLJcThJmZ5XKCMDOzXE4QZmaWywnCzMxyOUGYmVkuJwgzM8vlBGFmLePuttubb5Qzs5ao7m67Mp4D+Oa1duEzCDNrCXe33f6cIMysJdzddvtzgjCzlnB32+2v1AQhaYGkhyRtlbQ8Z/4SSaOSNqaPz2TmfUXSZkkPSrpUksqM1cyay91tt7/SEoSkKcDlwNnAicB5kk7MWXR1RMxLH1el674XeB9wEvAu4HeA3y0rVjNrvnrHc7DyldmKaT6wNSK2AUi6HlgEbCmwbgCHAdMAAYcCT5YUp5m1iLvbbm9lXmI6Fng8M70zLat2jqRNkm6SNBsgIn4I3A48kT7WRsSD1StKWippRNLI6Oho4z+BmVkPa3Ul9c1Af0ScBKwDrgGQ9JvAO4BZJEnlg5JOq145IoYjYiAiBmbOnNnEsM3Mul+ZCWIXMDszPSst2y8idkfES+nkVcAp6euPAfdExAsR8QJwC3BqibGamVmVMhPEemCupOMkTQMWA2uyC0g6JjO5EKhcRnoM+F1JUyUdSlJBfdAlJjMzK09pCSIi9gLLgLUkB/cbImKzpBWSFqaLXZQ2Zb0fuAhYkpbfBDwK/AS4H7g/Im4uK1Yzs1Zo976oFBGtjqEhBgYGYmRkpNVhmJkVUt0XFST3gTS7qa+kDRExkDev1ZXUZmY9qRP6onKCMDNrgU7oi8oJwsysBTqhLyonCDOzSaqnkrkT+qJygjAzm4RKJfOOHRBxYMCjokmiE/qicismM7NJ6O9PkkK1vj7Yvr3Z0UyeWzGZmTVYJ1Qy18sJwsxsEjqhkrleThBmZpPQCZXM9XKCMDObhE6oZK5XmQMGmZl1tW4f8MhnEGZmlssJwszMcjlBmJlZLicIMzPL5QRhZma5nCDMzCyXE4SZmeVygjAzs1xOEGZmlssJwszMcjlBmJlZrlIThKQFkh6StFXS8pz5SySNStqYPj6Tlv9epmyjpBcl/WGZsZqZ2WuV1lmfpCnA5cCZwE5gvaQ1EbGlatHVEbEsWxARtwPz0vc5GtgK3FpWrGZmdrAyzyDmA1sjYltEvAxcDyyaxPt8HLglIvY0NDozMxtTmQniWODxzPTOtKzaOZI2SbpJ0uyc+YuB6/I2IGmppBFJI6Ojo/VHbGZm+7W6kvpmoD8iTgLWAddkZ0o6BvgtYG3eyhExHBEDETEwc+bM0oM1M+slZSaIXUD2jGBWWrZfROyOiJfSyauAU6re41zgmxHxSmlRmplZrjITxHpgrqTjJE0juVS0JrtAeoZQsRB4sOo9zqPG5SUzMytXaa2YImKvpGUkl4emAFdHxGZJK4CRiFgDXCRpIbAXeBpYUllfUj/JGcj3y4rRzMxqU0S0OoaGGBgYiJGRkVaHYWbWUSRtiIiBvHmtrqQ2M7M25QRhZma5nCDMzCyXE4SZmeVygjAzs1xOEGZmlssJwszMcjlBmJlZLicIMzPL5QRhZma5nCDMzCyXE4SZmeVygjAz61CrVkF/PxxySPK8alVj37+07r7NzKw8q1bB0qWwZ08yvWNHMg0wONiYbfgMwsysAw0NHUgOFXv2JOWN4gRhZtaBHntsYuWT4QRhZtaB5syZWPlkOEGYmXWglSth+vTXlk2fnpQ3ihOEmVkHGhyE4WHo6wMpeR4eblwFNbgVk5lZxxocbGxCqOYzCDMzy+UEYWZmuZwgzMwslxOEmZnlcoIwM7NciohWx9AQkkaBHXW8xQzg5w0KpwyOrz6Orz6Orz7tHF9fRMzMm9E1CaJekkYiYqDVcdTi+Orj+Orj+OrT7vHV4ktMZmaWywnCzMxyOUEcMNzqAMbh+Orj+Orj+OrT7vHlch2EmZnl8hmEmZnlcoIwM7NcPZUgJC2Q9JCkrZKW58x/naTV6fx7JfU3MbbZkm6XtEXSZkmfy1nmdEnPStqYPr7YrPgyMWyX9JN0+yM58yXp0nQfbpJ0chNje3tm32yU9Jyki6uWaeo+lHS1pKckPZApO1rSOkmPpM9H1Vj3/HSZRySd38T4virpp+nf75uSjqyx7pjfhRLju0TSrszf8MM11h3z915ifKszsW2XtLHGuqXvv7pFRE88gCnAo8DbgGnA/cCJVcv8OXBF+noxsLqJ8R0DnJy+fgPwcE58pwPfbvF+3A7MGGP+h4FbAAHvAe5t4d/7v0huAmrZPgQ+AJwMPJAp+wqwPH29HPhyznpHA9vS56PS10c1Kb6zgKnp6y/nxVfku1BifJcAny/w9x/z915WfFXz/w74Yqv2X72PXjqDmA9sjYhtEfEycD2wqGqZRcA16eubgDMkqRnBRcQTEXFf+vp54EHg2GZsu8EWAd+IxD3AkZKOaUEcZwCPRkQ9d9fXLSLuBJ6uKs5+z64B/jBn1Q8B6yLi6Yh4BlgHLGhGfBFxa0TsTSfvAWY1ertF1dh/RRT5vddtrPjSY8e5wHWN3m6z9FKCOBZ4PDO9k4MPwPuXSX8gzwK/1pToMtJLW78N3Jsz+1RJ90u6RdI7mxpYIoBbJW2QtDRnfpH93AyLqf3DbPU+fEtEPJG+/i/gLTnLtMt+/FOSM8I8430XyrQsvQR2dY1LdO2w/04DnoyIR2rMb+X+K6SXEkRHkHQE8K/AxRHxXNXs+0gumbwbuAz4VrPjA94fEScDZwN/IekDLYhhTJKmAQuBG3Nmt8M+3C+Saw1t2dZc0hCwF1hVY5FWfRf+CfgNYB7wBMllnHZ0HmOfPbT9b6mXEsQuYHZmelZalruMpKnAm4DdTYku2eahJMlhVUT8W/X8iHguIl5IX38HOFTSjGbFl253V/r8FPBNklP5rCL7uWxnA/dFxJPVM9phHwJPVi67pc9P5SzT0v0oaQnwB8BgmsQOUuC7UIqIeDIiXo2IfcCVNbbb6v03FfgjYHWtZVq1/yailxLEemCupOPS/zAXA2uqllkDVFqLfBz4Xq0fR6Ol1yv/GXgwIv6+xjK/XqkTkTSf5O/XzAT2eklvqLwmqcx8oGqxNcCn0tZM7wGezVxOaZaa/7m1eh+mst+z84F/z1lmLXCWpKPSSyhnpWWlk7QA+CtgYUTsqbFMke9CWfFl67Q+VmO7RX7vZfp94KcRsTNvZiv334S0upa8mQ+SFjYPk7RuGErLVpD8EAAOI7kssRX4EfC2Jsb2fpJLDZuAjenjw8CFwIXpMsuAzSQtMu4B3tvk/fe2dNv3p3FU9mE2RgGXp/v4J8BAk2N8PckB/02ZspbtQ5JE9QTwCsl18D8jqde6DXgE+C5wdLrsAHBVZt0/Tb+LW4FPNzG+rSTX7yvfw0rLvrcC3xnru9Ck+P4l/W5tIjnoH1MdXzp90O+9GfGl5V+vfOcyyzZ9/9X7cFcbZmaWq5cuMZmZ2QQ4QZiZWS4nCDMzy+UEYWZmuZwgzMwslxOEdSRJr6a9YD4g6UZJ0xv0vgOSLm3Qe10i6fPp6yWS3tqI903f73RJ781MXyjpU416fzNwgrDO9auImBcR7wJeJrnXYb/0TtYJi4iRiLioEQFWWULSDr6wcT7D6cD+BBERV0TENyYVmVkNThDWDe4CfjP9r/ouSWuALZL6q/rp/7ykS9LXd0j6sqQfSXpY0mlp+emSvp2+viTtDO4OSdskXZR5r79Nxxr4gaTrKmcKeSR9nOQmuFXpWc/hkk6R9P20o7a1ma437pD0tXR8gM9J+qiSsUl+LOm7kt6SduZ4IfA/0vc7repsZZ6ke3RgPIejxvnM70zLNqbrzG3UH8Y6mxOEdbT0v+yzSe6shaRv/s9FxPEFVp8aEfOBi4Ev1VjmBJKut+cDX5J0qKTfAc4B3p1ue2CsjUTETcAISb9G80g6wLsM+HhEnAJcDazMrDItIgYi4u+AHwDviYjfJumy+q8iYjtwBfAP6VnUXVWb/Abw1xFxEsl+yX62vM98IfC/09gGSO4INmNSp+FmbeBwHRip6y6SfqzeC/woIn5W8D0qHSJuAPprLPMfEfES8JKkp0i65n4f8O8R8SLwoqSbJxj724F3AevSbqGmkHTXUJHt4G0WsDo9w5gGjPnZJL0JODIivp8WXcNre7XN+8w/BIYkzQL+LWp3T209xgnCOtWv0v9490sPtr/MFO3ltWfJh1W9x0vp86vU/i28lHk91nITIWBzRJxaY372M1wG/H1ErJF0OsloavU46DNHxLWS7gU+AnxH0n+PiO/VuR3rAr7EZN3sSeDNkn5N0utIuq9uhLuBj0o6TMn4HUXe93mSoWQBHgJmSjoVkm7eVXvgojdxoJvq7LjU2ffbLyKeBZ6p1C8AnwS+X71clqS3Adsi4lKSnmVPGv/jWC/wGYR1rYh4RdIKkp55dwE/bdD7rk8rwjeRJKGfkIw+OJavA1dI+hVwKkl38peml4SmAl8j6dWz2iXAjZKeAb4HHJeW3wzcJGkR8Nmqdc5PtzWdZCzrT48T27nAJyW9QjLC3f8aZ3nrEe7N1WwSJB0RES+kB+E7gaWRjilu1i18BmE2OcOSTiSp17jGycG6kc8gzMwslyupzcwslxOEmZnlcoIwM7NcThBmZpbLCcLMzHL9fzeI7ewjWk+wAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "x = [x for x, _ in ranks_and_accuracies]\n",
        "y = [y for _, y in ranks_and_accuracies]\n",
        "\n",
        "plt.plot(x, y, 'bo')\n",
        "plt.xlabel(\"Pruning Iterations\")\n",
        "plt.ylabel(\"Accuracy\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KoN-IhY9hNQE"
      },
      "outputs": [],
      "source": [
        "# TODO: add code for option 1 here\n",
        "\n",
        "# def depth_separable_net_fn(batch: Batch, width: float) -> jnp.ndarray:\n",
        "  \n",
        "#   x = normalize(batch[0])\n",
        "\n",
        "#   # Do not alter the architecture code.\n",
        "#   net = hk.Sequential([\n",
        "#       # hk.Conv2D(output_channels=3, kernel_shape=(3,3)),\n",
        "#       # jax.nn.relu,\n",
        "#       # hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "#       hk.DepthwiseConv2D(1, kernel_shape=5), \n",
        "#       jax.nn.relu,\n",
        "#       hk.Conv2D((width * 6*3).astype('int'), kernel_shape=(1, 1)), \n",
        "#       jax.nn.relu,\n",
        "#       hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "#       hk.DepthwiseConv2D(1, kernel_shape=5), \n",
        "#       jax.nn.relu,\n",
        "#       hk.Conv2D((width * 16*3).astype('int'), kernel_shape=(1, 1)), \n",
        "#       jax.nn.relu,\n",
        "#       hk.AvgPool(window_shape=(2,2), strides=(2,2), padding='VALID'),\n",
        "#       hk.Flatten(),\n",
        "#       hk.Linear((3000).astype('int')), jax.nn.relu,\n",
        "#       hk.Linear((2000).astype('int')), jax.nn.relu,\n",
        "#       hk.Linear((2000).astype('int')), jax.nn.relu,\n",
        "#       hk.Linear((1000).astype('int')), jax.nn.relu,\n",
        "#       hk.Linear(10),\n",
        "#   ])\n",
        "#   return net(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I_sfE7P0IW1K"
      },
      "outputs": [],
      "source": [
        "# import jmp\n",
        "\n",
        "# policy = jmp.get_policy('params=float16,compute=float16,output=float16')\n",
        "\n",
        "# ranks_and_accuracies = []\n",
        "# ranks_and_times = []\n",
        "# hk.mixed_precision.set_policy(depth_separable_net_fn, policy)\n",
        "# for width_fraction in np.arange(1.0, 0.0, -1.5):\n",
        "#   depth_separable_net_fn = partial(depth_separable_net_fn, width=width_fraction)\n",
        "#   # Quantization, using float 16 during compute and output to speed up training and inference\n",
        "\n",
        "#   # sampling multiple random numbers together to optimize init and apply of model\n",
        "#   # depth_separable_net_fn = hk.experimental.optimize_rng_use(depth_separable_net_fn)\n",
        "\n",
        "#   net = hk.without_apply_rng(hk.transform(depth_separable_net_fn)) \n",
        "#   depth_separable_params = net.init(jax.random.PRNGKey(42), next(train))\n",
        "#   depth_separable_avg_params = deepcopy(depth_separable_params)\n",
        "\n",
        "#   opt_state = opt.init(depth_separable_params)\n",
        "\n",
        "#   print(f\"Evaluating the model at \" f\"{width_fraction:.2f}\")\n",
        "\n",
        "#   # train network\n",
        "#   opt_state = opt.init(depth_separable_params)\n",
        "\n",
        "#   # Do not alter the number of steps\n",
        "#   for step in range(10001):\n",
        "#     if step % 1000 == 0:\n",
        "#       val_accuracy, d = compute_accuracy(depth_separable_avg_params, next(validation))\n",
        "#       test_accuracy, d = compute_accuracy(depth_separable_avg_params, next(test))\n",
        "#       val_accuracy, test_accuracy = jax.device_get(\n",
        "#           (val_accuracy, test_accuracy))    \n",
        "#       print(f\"[Step {step}] Validation / Test accuracy: \"\n",
        "#             f\"{val_accuracy:.3f} / {test_accuracy:.3f}.\")\n",
        "      \n",
        "#     depth_separable_params, opt_state = update(depth_separable_params, opt_state, next(train))\n",
        "#     depth_separable_avg_params = ema_update(depth_separable_params, depth_separable_avg_params)\n",
        "  \n",
        "#   # TODO: modify the compute_eval_metrics function below to compute the time taken for inference.\n",
        "#   test_accuracy, duration = compute_eval_metrics(depth_separable_avg_params, next(test), 50)\n",
        "#   ranks_and_times.append((width_fraction, np.mean(duration)))\n",
        "#   ranks_and_accuracies.append((width_fraction, np.mean(test_accuracy)))\n",
        "#   print(f\"Rank Fraction / Test accuracy: \"\n",
        "#           f\"{width_fraction:.2f} / {np.mean(test_accuracy):.3f}.\")\n",
        "#   print(f\"Rank Fraction / Duration (milli sec): \"\n",
        "#           f\"{width_fraction:.2f} / {np.mean(duration)*1000:.4f}.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fq2zcXoRNkYx"
      },
      "source": [
        "## You have made it to the end of the challenge!\n",
        "\n",
        "Before you submit your completed challenge document, please make sure to **save and pin your revisions** before submitting a link to your submission via the [Cohere For AI Scholars Program Application.](https://jobs.lever.co/cohere/) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i6eafeYtlhVL"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "-v-iHTH3nasL",
        "UHGqpn3tkFRL"
      ],
      "provenance": [],
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.6"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0163b13eaaae4617a19509ceff3bd339": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "024994efa9a94649a6e8c40c8775149d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "03fcf0536efd456a906202273a24debc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "04231c29fa4c48d78cecb7dd23ef4725": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0800eb1c3b264aa78d1b9143a51adcb6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "084440fb123d4b168377878cfa3d8589": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "08995c1f4e31419d8a8f0535bef8ef1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0dd7df3cdac14cb49cb90fa53b5aa1a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a3165c2388df4f6c9be17799387613b3",
            "placeholder": "​",
            "style": "IPY_MODEL_a14169df4a08430fb539d23bcafd3bf3",
            "value": "Dl Completed...: 100%"
          }
        },
        "0e3740e900704125b30d31fd2e1abb47": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "125cb4682d91435fa63ccc69f4272ce5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0dd7df3cdac14cb49cb90fa53b5aa1a0",
              "IPY_MODEL_155e0e12d8b74a0caaecb45cfe42c909",
              "IPY_MODEL_41f10b9116eb470b92300c54b8599bb9"
            ],
            "layout": "IPY_MODEL_5c5ec9ae3120432f81163593d0fca729"
          }
        },
        "12a35d35359f43d89abba53fed35181d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "136fefb48f1b46f594a436a76e64c6f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1499954e23304688985f8624b34f9fd7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a3d410be51dd4db3ac73d23488e9af47",
              "IPY_MODEL_6784e2dad4b54f78ae53e52262376c83",
              "IPY_MODEL_69fabfeab44e4569b3f8bcc48c6bf004"
            ],
            "layout": "IPY_MODEL_7dddbd5b30844d6fbda588ac199b6edb"
          }
        },
        "155e0e12d8b74a0caaecb45cfe42c909": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_482e2a1f2e5a4a45bb9710a74109801e",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7acf4ea181764f26a53c9517665b40a5",
            "value": 1
          }
        },
        "1625117f98854c4b87eafe9ea4ea7893": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c55bacbdc65d49269c628495573b46cd",
            "max": 50000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9209390e1b0a482db1b62101ddb1c306",
            "value": 50000
          }
        },
        "1bde4bf945f14497a869854243124f60": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28fa5c3c59234abf99872abd1e8c4678": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2e70dbfca6d7459d8f0ed687a5d76e6c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "307965ca273f4b0e89c55ccfa590fce0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "328d5b4cdda0436bb479f86ac82680ce": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "3a1643d595f4419a9727fb131ef4d548": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_67d15cc85fa942b6859d301c70bbbcf4",
            "placeholder": "​",
            "style": "IPY_MODEL_0e3740e900704125b30d31fd2e1abb47",
            "value": "Shuffling ~/tensorflow_datasets/cifar10/3.0.2.incompleteH429IC/cifar10-train.tfrecord*...:  78%"
          }
        },
        "41f10b9116eb470b92300c54b8599bb9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_792e93d718f44308a107428d02d087e6",
            "placeholder": "​",
            "style": "IPY_MODEL_c16f5fd1ee9c46ffbd4cf70e5a76a3d9",
            "value": " 1/1 [00:18&lt;00:00, 15.06s/ url]"
          }
        },
        "47552858457e4ae788ffcc7aceeebee3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "482e2a1f2e5a4a45bb9710a74109801e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "486b13e0d6e6403a982e60aad25b0210": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3a1643d595f4419a9727fb131ef4d548",
              "IPY_MODEL_1625117f98854c4b87eafe9ea4ea7893",
              "IPY_MODEL_7fc0f0022c3a49c7a60736ddadb26015"
            ],
            "layout": "IPY_MODEL_6165ea42c4f84caf96298900496e7e78"
          }
        },
        "489dab16ca0f4b47a26098dabe6656e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_63ee59a41b4d4f4387e88433f62f40e1",
              "IPY_MODEL_73cd458fd3684e80a73271a67ce4c449",
              "IPY_MODEL_7215c69ac4fa40e0923efce59fa963d4"
            ],
            "layout": "IPY_MODEL_a367896ef0e74e3095acc55e2252e2b4"
          }
        },
        "4ae45fedfe884787a2aa673e2428b77e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1bde4bf945f14497a869854243124f60",
            "placeholder": "​",
            "style": "IPY_MODEL_28fa5c3c59234abf99872abd1e8c4678",
            "value": "Generating test examples...: 100%"
          }
        },
        "4cd9c329f7b847d086b222e4febae367": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4e07817836074afd95ffa0ce13d887c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "58b954b4db2c43ef9c7a986803cc1f19": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5c5ec9ae3120432f81163593d0fca729": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5dd8d88a1c434cc0b1fb17d41da471a7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5dddc88f52974a6c9f89ccc293378a0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_78b3d6c661da4f3e829a47a9a3982f96",
              "IPY_MODEL_672b454d76444221a214aa2ecc56ce53",
              "IPY_MODEL_b31d0cc11b414f98ac562fbf2a5ab32b"
            ],
            "layout": "IPY_MODEL_024994efa9a94649a6e8c40c8775149d"
          }
        },
        "6165ea42c4f84caf96298900496e7e78": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "63ee59a41b4d4f4387e88433f62f40e1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d5d77ab171df40d5b7793f407367861b",
            "placeholder": "​",
            "style": "IPY_MODEL_136fefb48f1b46f594a436a76e64c6f6",
            "value": "Extraction completed...: 100%"
          }
        },
        "66c12a744e12466db36dcc83766278c7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "672b454d76444221a214aa2ecc56ce53": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_47552858457e4ae788ffcc7aceeebee3",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_adf8d3c7221c49d59179994b64386281",
            "value": 1
          }
        },
        "6784e2dad4b54f78ae53e52262376c83": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b54c15bf12f64c099555958a0ded00b9",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d24658c56af34cfbb6d9817ed2381444",
            "value": 2
          }
        },
        "67d15cc85fa942b6859d301c70bbbcf4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "69fabfeab44e4569b3f8bcc48c6bf004": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d374df00d7f54d6d8b6374fa1a139828",
            "placeholder": "​",
            "style": "IPY_MODEL_08995c1f4e31419d8a8f0535bef8ef1a",
            "value": " 2/2 [00:54&lt;00:00, 24.16s/ splits]"
          }
        },
        "7011fcb71fb54224bbf2ca3017e7cfe5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7215c69ac4fa40e0923efce59fa963d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_12a35d35359f43d89abba53fed35181d",
            "placeholder": "​",
            "style": "IPY_MODEL_4e07817836074afd95ffa0ce13d887c0",
            "value": " 1/1 [00:18&lt;00:00, 18.12s/ file]"
          }
        },
        "73cd458fd3684e80a73271a67ce4c449": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_328d5b4cdda0436bb479f86ac82680ce",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0800eb1c3b264aa78d1b9143a51adcb6",
            "value": 1
          }
        },
        "78b3d6c661da4f3e829a47a9a3982f96": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7011fcb71fb54224bbf2ca3017e7cfe5",
            "placeholder": "​",
            "style": "IPY_MODEL_fc22a3537aa04be58497faad28cadf2a",
            "value": "Dl Size...: 100%"
          }
        },
        "792e93d718f44308a107428d02d087e6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7acf4ea181764f26a53c9517665b40a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7dddbd5b30844d6fbda588ac199b6edb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "7fc0f0022c3a49c7a60736ddadb26015": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4cd9c329f7b847d086b222e4febae367",
            "placeholder": "​",
            "style": "IPY_MODEL_fa72b99270734b3c88bbb850724e9acf",
            "value": " 39029/50000 [00:00&lt;00:00, 143003.29 examples/s]"
          }
        },
        "9209390e1b0a482db1b62101ddb1c306": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "92b96ac0af404aa7ac29febd6601a26c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d24b46dc02df4d22b143ddbb90845067",
            "max": 50000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c44bc25746cf4c02aa7ed95b04139782",
            "value": 50000
          }
        },
        "92ddfb58a7084ed3be1ebf9294712bde": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9770e4ce645d46b296de1cfaf7e7fb0a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "a1264146ed274496aaa68d35cf9adde6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "a14169df4a08430fb539d23bcafd3bf3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a3165c2388df4f6c9be17799387613b3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a367896ef0e74e3095acc55e2252e2b4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a3d410be51dd4db3ac73d23488e9af47": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cae30c2ce3ea4dac80ae6b974ce8f033",
            "placeholder": "​",
            "style": "IPY_MODEL_ee43e3f18dc64dfea3fc251ad3a0a34e",
            "value": "Generating splits...: 100%"
          }
        },
        "a9f21872578242b4973c308a3674264a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4ae45fedfe884787a2aa673e2428b77e",
              "IPY_MODEL_c0a1dd36d5d046f7af9726ea39151f46",
              "IPY_MODEL_eca2a4226b23426b8f6984795a35abfa"
            ],
            "layout": "IPY_MODEL_a1264146ed274496aaa68d35cf9adde6"
          }
        },
        "adf8d3c7221c49d59179994b64386281": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b31d0cc11b414f98ac562fbf2a5ab32b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0163b13eaaae4617a19509ceff3bd339",
            "placeholder": "​",
            "style": "IPY_MODEL_f265fc5b189443b5bc26788cc170dd95",
            "value": " 162/162 [00:18&lt;00:00, 12.86 MiB/s]"
          }
        },
        "b50999734a6845b5bc7b67b26c2e3336": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b54c15bf12f64c099555958a0ded00b9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b87a8542a970482584a1460ceb2b4595": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_66c12a744e12466db36dcc83766278c7",
            "placeholder": "​",
            "style": "IPY_MODEL_d39ed98855aa41a3b220556a353d2a9b",
            "value": " 8337/10000 [00:00&lt;00:00, 83359.15 examples/s]"
          }
        },
        "bcb105a141df4adf9156720173f7ead9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_307965ca273f4b0e89c55ccfa590fce0",
            "placeholder": "​",
            "style": "IPY_MODEL_04231c29fa4c48d78cecb7dd23ef4725",
            "value": "Shuffling ~/tensorflow_datasets/cifar10/3.0.2.incompleteH429IC/cifar10-test.tfrecord*...:  83%"
          }
        },
        "c0a1dd36d5d046f7af9726ea39151f46": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_92ddfb58a7084ed3be1ebf9294712bde",
            "max": 10000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d11803d4b0134aee947e6557dc70d5db",
            "value": 10000
          }
        },
        "c16f5fd1ee9c46ffbd4cf70e5a76a3d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c31d6f41c7d34cd78a879d56abd7610d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d5dd43bb0bc7425385df5685af62d440",
            "placeholder": "​",
            "style": "IPY_MODEL_2e70dbfca6d7459d8f0ed687a5d76e6c",
            "value": " 49924/50000 [00:45&lt;00:00, 1162.33 examples/s]"
          }
        },
        "c33964ccf95543038947138a2459a80d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c44bc25746cf4c02aa7ed95b04139782": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c55bacbdc65d49269c628495573b46cd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cae30c2ce3ea4dac80ae6b974ce8f033": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc30e5efb79b4c9cb35aaf5126eafd66": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bcb105a141df4adf9156720173f7ead9",
              "IPY_MODEL_f7253ab1829546139269cc957db0d7b1",
              "IPY_MODEL_b87a8542a970482584a1460ceb2b4595"
            ],
            "layout": "IPY_MODEL_03fcf0536efd456a906202273a24debc"
          }
        },
        "cd532039d76349fab7142f50a5794317": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d053418d04674749a98afa17e91aa6b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f166794fe7ef47b49f32b8ce294c06db",
              "IPY_MODEL_92b96ac0af404aa7ac29febd6601a26c",
              "IPY_MODEL_c31d6f41c7d34cd78a879d56abd7610d"
            ],
            "layout": "IPY_MODEL_9770e4ce645d46b296de1cfaf7e7fb0a"
          }
        },
        "d11803d4b0134aee947e6557dc70d5db": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d24658c56af34cfbb6d9817ed2381444": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d24b46dc02df4d22b143ddbb90845067": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d374df00d7f54d6d8b6374fa1a139828": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d39ed98855aa41a3b220556a353d2a9b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d5d77ab171df40d5b7793f407367861b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d5dd43bb0bc7425385df5685af62d440": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eca2a4226b23426b8f6984795a35abfa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cd532039d76349fab7142f50a5794317",
            "placeholder": "​",
            "style": "IPY_MODEL_58b954b4db2c43ef9c7a986803cc1f19",
            "value": " 9985/10000 [00:08&lt;00:00, 1130.70 examples/s]"
          }
        },
        "ee43e3f18dc64dfea3fc251ad3a0a34e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f166794fe7ef47b49f32b8ce294c06db": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_084440fb123d4b168377878cfa3d8589",
            "placeholder": "​",
            "style": "IPY_MODEL_c33964ccf95543038947138a2459a80d",
            "value": "Generating train examples...: 100%"
          }
        },
        "f265fc5b189443b5bc26788cc170dd95": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f7253ab1829546139269cc957db0d7b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5dd8d88a1c434cc0b1fb17d41da471a7",
            "max": 10000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b50999734a6845b5bc7b67b26c2e3336",
            "value": 10000
          }
        },
        "fa72b99270734b3c88bbb850724e9acf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fc22a3537aa04be58497faad28cadf2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}