COLAB_Primer_MotionCapture_Fig3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Primer-MotionCapture_fig3.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DeepLabCut/Primer-MotionCapture/blob/master/COLAB_Primer_MotionCapture_Fig3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK255E7YoEIt",
        "colab_type": "text"
      },
      "source": [
        "# Motion Capture with Deep Learning: \n",
        "A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives\n",
        "\n",
        "[Alexander Mathis](https://github.com/AlexEMG) | [Steffen Schneider](https://github.com/stes) | [Jessy Lauer](https://github.com/jeylau) | [Mackenzie Mathis](https://github.com/MMathisLab)\n",
        "\n",
        "\n",
        "- https://github.com/DeepLabCut/Primer-MotionCapture\n",
        "\n",
        "- How you can use data augmentation to train neural networks:\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1593721822684-DJJIOEEC4YHRO2ZRW6Z3/ke17ZwdGBToddI8pDm48kMqyqP2xgoLMxs8NG4McAT9Zw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZamWLI2zvYWH8K3-s_4yszcp2ryTI0HqTOaaUohrI8PIT5G0e3JcWjRHcSN0vw_Zk3UAC_JV7w-s1xTX3wfPLu0/augs.png?format=1000w)\n",
        "\n",
        "This script uses Imgaug to display various augmentation methods \n",
        "for a few labeled images of birds (Data recorded with Alex' iphone \n",
        "at Aiguille du Midi near Mont Blanc)\n",
        "\n",
        "Uses Imgaug:\n",
        "Code: https://github.com/aleju/imgaug\n",
        "Docs: https://imgaug.readthedocs.io/en/latest/index.html\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txoddlM8hLKm",
        "colab_type": "text"
      },
      "source": [
        "Please click on the \"play' button to run the code in each cell.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q23BzhA6CXxu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#(this will take a few minutes to install all the dependences!)\n",
        "!pip install deeplabcut\n",
        "%reload_ext numpy\n",
        "%reload_ext scipy\n",
        "%reload_ext matplotlib\n",
        "%reload_ext mpl_toolkits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5WGHc0IiYkoL",
        "colab_type": "text"
      },
      "source": [
        "Import the dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y36K4Eux3h-X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "3088b6c0-ebf3-46e2-fa39-513a8470b3f2"
      },
      "source": [
        "# Use TensorFlow 1.x:\n",
        "%tensorflow_version 1.x\n",
        "import os\n",
        "os.environ[\"DLClight\"]=\"True\"\n",
        "import deeplabcut\n",
        "import tensorflow as tf\n",
        "vers = (tf.__version__).split(\".\")\n",
        "if int(vers[0]) == 1 and int(vers[1]) > 12:\n",
        "    TF = tf.compat.v1  # behaves differently before 1.13\n",
        "else:\n",
        "    TF = tf\n",
        "\n",
        "TF.logging.set_verbosity(TF.logging.ERROR)\n",
        "DEBUG = True and \"DEBUG\" in os.environ and os.environ[\"DEBUG\"]\n",
        "from deeplabcut import DEBUG\n",
        "\n",
        "import pandas as pd\n",
        "import os\n",
        "import numpy as np\n",
        "\n",
        "import imgaug as ia\n",
        "import imgaug.augmenters as iaa\n",
        "from imgaug.augmentables import Keypoint, KeypointsOnImage\n",
        "from deeplabcut.utils.auxfun_videos import imread, imresize\n",
        "import imageio "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGRsUQL2aPeI",
        "colab_type": "text"
      },
      "source": [
        "Let's grab some demo data!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KS4Q4UkR9rgG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Clone the Primer repo so we can use the demo data:\n",
        "!git clone -l -s git://github.com/DeepLabCut/Primer-MotionCapture.git cloned-DLC-Primer\n",
        "\n",
        "%cd cloned-DLC-Primer\n",
        "!ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8LNx3z_ybv7P",
        "colab_type": "text"
      },
      "source": [
        "Setting up some variables:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bfSPyAx9bwEy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "scale=.2\n",
        "ia.seed(1)\n",
        "#parameters for plotting:\n",
        "color=(400,0,0)\n",
        "size=7\n",
        "alpha=.01"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frnj1RVDyEqs",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## Loading data:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vhENAlQnFENJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 317
        },
        "outputId": "cfd18a5f-7809-4b20-8290-60af7dddbaff"
      },
      "source": [
        "imfolder='montblanc_images'\n",
        "Dataframe = pd.read_hdf(os.path.join(imfolder,\"CollectedData_Daniel.h5\"))\n",
        "\n",
        "scorer=Dataframe.columns.get_level_values(0)[0]\n",
        "individuals=Dataframe.columns.get_level_values(1)\n",
        "bodyparts=Dataframe.columns.get_level_values(2)\n",
        "Dataframe.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th>scorer</th>\n",
              "      <th colspan=\"64\" halign=\"left\">Daniel</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>individuals</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird1</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird2</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird3</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird4</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird5</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird6</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird7</th>\n",
              "      <th colspan=\"8\" halign=\"left\">bird8</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bodyparts</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">head</th>\n",
              "      <th colspan=\"2\" halign=\"left\">tail</th>\n",
              "      <th colspan=\"2\" halign=\"left\">leftwing</th>\n",
              "      <th colspan=\"2\" halign=\"left\">rightwing</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>coords</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "      <th>x</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>img0155.png</th>\n",
              "      <td>76.844318</td>\n",
              "      <td>1068.722389</td>\n",
              "      <td>91.885294</td>\n",
              "      <td>1073.296136</td>\n",
              "      <td>83.178698</td>\n",
              "      <td>1068.890492</td>\n",
              "      <td>85.340405</td>\n",
              "      <td>1066.281536</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>633.723407</td>\n",
              "      <td>265.969784</td>\n",
              "      <td>664.802644</td>\n",
              "      <td>278.419669</td>\n",
              "      <td>650.006421</td>\n",
              "      <td>262.713181</td>\n",
              "      <td>646.421219</td>\n",
              "      <td>282.682029</td>\n",
              "      <td>687.180030</td>\n",
              "      <td>254.177500</td>\n",
              "      <td>701.044000</td>\n",
              "      <td>257.237000</td>\n",
              "      <td>692.507979</td>\n",
              "      <td>265.632591</td>\n",
              "      <td>692.507979</td>\n",
              "      <td>252.579116</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>270.767000</td>\n",
              "      <td>441.059000</td>\n",
              "      <td>291.042197</td>\n",
              "      <td>445.404245</td>\n",
              "      <td>279.605967</td>\n",
              "      <td>445.585773</td>\n",
              "      <td>265.960000</td>\n",
              "      <td>439.351000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>img0719.png</th>\n",
              "      <td>527.883814</td>\n",
              "      <td>798.424352</td>\n",
              "      <td>527.883814</td>\n",
              "      <td>819.592270</td>\n",
              "      <td>555.564938</td>\n",
              "      <td>813.079065</td>\n",
              "      <td>493.689484</td>\n",
              "      <td>798.424352</td>\n",
              "      <td>866.570511</td>\n",
              "      <td>166.643397</td>\n",
              "      <td>848.659196</td>\n",
              "      <td>235.032057</td>\n",
              "      <td>921.932760</td>\n",
              "      <td>207.350933</td>\n",
              "      <td>785.155440</td>\n",
              "      <td>173.156603</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>img0382.png</th>\n",
              "      <td>1002.648118</td>\n",
              "      <td>448.081409</td>\n",
              "      <td>1016.503454</td>\n",
              "      <td>441.475958</td>\n",
              "      <td>974.937444</td>\n",
              "      <td>439.059329</td>\n",
              "      <td>1035.192048</td>\n",
              "      <td>444.537021</td>\n",
              "      <td>1345.764721</td>\n",
              "      <td>430.355034</td>\n",
              "      <td>1373.472296</td>\n",
              "      <td>425.958629</td>\n",
              "      <td>1323.271486</td>\n",
              "      <td>432.604357</td>\n",
              "      <td>1373.370054</td>\n",
              "      <td>420.335320</td>\n",
              "      <td>1208.005774</td>\n",
              "      <td>423.875068</td>\n",
              "      <td>1223.472197</td>\n",
              "      <td>412.919685</td>\n",
              "      <td>1183.839489</td>\n",
              "      <td>422.425091</td>\n",
              "      <td>1241.516356</td>\n",
              "      <td>416.302965</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>465.149323</td>\n",
              "      <td>910.426436</td>\n",
              "      <td>472.586078</td>\n",
              "      <td>922.821027</td>\n",
              "      <td>462.895761</td>\n",
              "      <td>920.680143</td>\n",
              "      <td>473.149468</td>\n",
              "      <td>915.609629</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>img1164.png</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>967.102545</td>\n",
              "      <td>772.498131</td>\n",
              "      <td>1052.904833</td>\n",
              "      <td>829.588369</td>\n",
              "      <td>1030.202282</td>\n",
              "      <td>825.915898</td>\n",
              "      <td>963.430073</td>\n",
              "      <td>818.570955</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1481.748136</td>\n",
              "      <td>780.182861</td>\n",
              "      <td>1551.512575</td>\n",
              "      <td>873.405211</td>\n",
              "      <td>1476.569117</td>\n",
              "      <td>831.973055</td>\n",
              "      <td>1520.438458</td>\n",
              "      <td>812.780218</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1281.219881</td>\n",
              "      <td>586.152006</td>\n",
              "      <td>1281.703207</td>\n",
              "      <td>597.912931</td>\n",
              "      <td>1277.353275</td>\n",
              "      <td>595.012977</td>\n",
              "      <td>1283.314293</td>\n",
              "      <td>593.563000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>img0071.png</th>\n",
              "      <td>322.057817</td>\n",
              "      <td>350.433043</td>\n",
              "      <td>338.709137</td>\n",
              "      <td>352.916134</td>\n",
              "      <td>337.832751</td>\n",
              "      <td>346.343245</td>\n",
              "      <td>318.990469</td>\n",
              "      <td>357.298061</td>\n",
              "      <td>704.509962</td>\n",
              "      <td>368.336837</td>\n",
              "      <td>713.463880</td>\n",
              "      <td>373.814528</td>\n",
              "      <td>717.361468</td>\n",
              "      <td>368.336837</td>\n",
              "      <td>695.872064</td>\n",
              "      <td>375.605312</td>\n",
              "      <td>792.096812</td>\n",
              "      <td>335.109416</td>\n",
              "      <td>801.484484</td>\n",
              "      <td>339.477937</td>\n",
              "      <td>808.734370</td>\n",
              "      <td>332.692788</td>\n",
              "      <td>781.500825</td>\n",
              "      <td>342.545197</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>476.196105</td>\n",
              "      <td>851.187857</td>\n",
              "      <td>497.623544</td>\n",
              "      <td>859.565502</td>\n",
              "      <td>480.626590</td>\n",
              "      <td>860.934925</td>\n",
              "      <td>485.057076</td>\n",
              "      <td>855.376679</td>\n",
              "      <td>1636.168946</td>\n",
              "      <td>717.076136</td>\n",
              "      <td>1656.772253</td>\n",
              "      <td>724.611060</td>\n",
              "      <td>1643.821603</td>\n",
              "      <td>722.962795</td>\n",
              "      <td>1645.587601</td>\n",
              "      <td>716.605203</td>\n",
              "      <td>338.311787</td>\n",
              "      <td>886.759027</td>\n",
              "      <td>350.230626</td>\n",
              "      <td>894.471217</td>\n",
              "      <td>343.419861</td>\n",
              "      <td>891.766942</td>\n",
              "      <td>336.709254</td>\n",
              "      <td>893.469634</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "scorer            Daniel                            ...                       \n",
              "individuals        bird1                            ...    bird8              \n",
              "bodyparts           head                      tail  ... leftwing rightwing    \n",
              "coords                 x            y            x  ...        y         x   y\n",
              "img0155.png    76.844318  1068.722389    91.885294  ...      NaN       NaN NaN\n",
              "img0719.png   527.883814   798.424352   527.883814  ...      NaN       NaN NaN\n",
              "img0382.png  1002.648118   448.081409  1016.503454  ...      NaN       NaN NaN\n",
              "img1164.png          NaN          NaN          NaN  ...      NaN       NaN NaN\n",
              "img0071.png   322.057817   350.433043   338.709137  ...      NaN       NaN NaN\n",
              "\n",
              "[5 rows x 64 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNi9s1dboEJN",
        "colab_type": "text"
      },
      "source": [
        "# Let's look at some Augmentations! \n",
        "\n",
        "*this takes a few minutes: please check the \"augmentation examples\" folder for the completed files!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "eMeUwgxPoEJP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#setting up augmentations\n",
        "Augmentations=[]\n",
        "\n",
        "augtype='rotateandscale'\n",
        "#rotate & scale \n",
        "seq = iaa.Sequential([\n",
        "    iaa.Multiply((1.2, 1.5)), # change brightness, doesn't affect keypoints\n",
        "    iaa.Affine(\n",
        "        rotate=23,\n",
        "        scale=(0.9, 1.1)\n",
        "    ) # rotate by exactly 23 deg and scale to 90-10%, affects keypoints\n",
        "])\n",
        "Augmentations.append([augtype, seq])\n",
        "\n",
        "augtype='fog'\n",
        "seq = iaa.Sequential([iaa.Fog()])\n",
        "Augmentations.append([augtype,seq])\n",
        "\n",
        "augtype='snow'\n",
        "seq = iaa.Sequential([iaa.Snowflakes(flake_size=(.2,.5),density=(0.005, 0.07), speed=(0.01, 0.05))])\n",
        "Augmentations.append([augtype,seq])        \n",
        "\n",
        "augtype='GuassianBlur'\n",
        "seq = iaa.Sequential([iaa.GaussianBlur(sigma=(0, 3.0))])\n",
        "Augmentations.append([augtype,seq]) \n",
        "\n",
        "for ind, imname in enumerate(Dataframe.index):\n",
        "        image=imresize(imread(os.path.join('montblanc_images',imname)),size=scale)\n",
        "        ny,nx,nc=np.shape(image)\n",
        "\n",
        "        kpts=[]\n",
        "        for i in individuals:\n",
        "            for b in bodyparts:\n",
        "                x, y=Dataframe.iloc[ind][scorer][i][b]['x'], Dataframe.iloc[ind][scorer][i][b]['y']\n",
        "                if np.isfinite(x) and np.isfinite(y):\n",
        "                    kpts.append(Keypoint(x=x*scale,y=y*scale))\n",
        "\n",
        "        kps=KeypointsOnImage(kpts, shape=image.shape)\n",
        "        \n",
        "        cells=[]\n",
        "\n",
        "        # image with keypoints before augmentation \n",
        "        image_before = kps.draw_on_image(image, color=color,size=size,alpha=alpha)\n",
        "        cells.append(image_before)\n",
        "        \n",
        "        for name, seq in Augmentations:\n",
        "            image_aug, kps_aug = seq(image=image, keypoints=kps)\n",
        "            image_after = kps_aug.draw_on_image(image_aug, color=color,size=size,alpha=alpha)\n",
        "            cells.append(image_after)\n",
        "    \n",
        "        grid_image = np.hstack(cells)  # Horizontally stack the images\n",
        "        imageio.imwrite('augmentationexamples/'+str(imfolder)+'_'+imname.split('.png')[0]+'_joint.png', grid_image)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-B_e9qdwMj5",
        "colab_type": "text"
      },
      "source": [
        "After (or while) the code above runs, you can go look into the \"augmentationexamples folder to see the images. In Colab, you can double click and they will open on the right side of this screen!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C4B5fN2Twhl2",
        "colab_type": "text"
      },
      "source": [
        "# Play around! What happens if you set the `augtype` to something else/or add more?\n",
        "\n",
        "Check out the options in imgaug: https://imgaug.readthedocs.io/en/latest/index.html\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y0077RSlxPoL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#For example, try adding this in the cell above, at line 23:\n",
        "augtype='GuassianBlur'\n",
        "seq = iaa.Sequential([iaa.GaussianBlur(sigma=(0, 3.0))])\n",
        "Augmentations.append([augtype,seq]) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_jwicl3ywI2",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1593748127529-O7LN1VK99YAKL6EU6MHY/ke17ZwdGBToddI8pDm48kDXDnTgl1slZNt80fPstB8hZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZamWLI2zvYWH8K3-s_4yszcp2ryTI0HqTOaaUohrI8PI7YJEkB-NQMg7QT3XqyKhWHKvgVbDU7y_mPhOqssYa08/montblanc_images_img0155_joint.png?format=1000w)\n",
        "\n",
        "\n",
        "- Can you find out all the augmentations used in the main Figure?\n",
        "\n",
        "- Want to use this in DLC, check out these suggestions: https://github.com/DeepLabCut/DeepLabCut/wiki/Data-Augmentation\n"
      ]
    }
  ]
}