05_Classification_de_textes.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicolashernandez/teaching_nlp/blob/main/05_Classification_de_textes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JxdW5mtMGqmy"
      },
      "source": [
        "# Classification de textes\n",
        "\n",
        "La **classification** consiste à attribuer une classe à chaque texte (objet, instance, point) à classer. On parle de *classification binaire* (_binary classification_) quand il y a deux classes exclusives. On parle de *classification en classes multiples* (_multiclass classification_) pour désigner la répartition d'un lot de textes entre plus de deux ensembles (ou classes), une classe par instance. On parle de *classification multi-étiquettes* (_multi-label classification_) pour désigner les problèmes de classification où plusieurs étiquettes (classes) peuvent être assignées à une même instance.\n",
        "\n",
        "Reconnaître si un email est un spam, si une photo contient une voiture... sont des problèmes de classification (binaire).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D7PRm8jl2cE0"
      },
      "source": [
        "# Sentiment analysis as a classification task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ee8F8DyZ9v7x"
      },
      "source": [
        "\n",
        "La tâche classique d'analyse de sentiment consiste à annoter un texte donné selon une polarité positive ou négative exprimée dans le texte.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbuBLXmt3NyE"
      },
      "source": [
        "## QUESTION\n",
        "\n",
        "* Selon vous la tâche d'analyse de sentiment tel que définie juste au dessus peut se définir comme un problème de 1) classification binaire, 2) classification en classes multiples ou 3) en classification multi-étiquettes ?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6sjOoxLi3m7E"
      },
      "source": [
        "## VOTRE REPONSE\n",
        "\n",
        "**TODO**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_h06bBaA34vf"
      },
      "source": [
        "# Configuration de l'environnement Google Colab en mode GPU\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WveM1n9KFNx6"
      },
      "source": [
        "Quel mode d'exécution utilisez-vous (via GPU ou via CPU) ? Que vous dis le code ci-dessous ?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "bwHyWGuWiCUu",
        "outputId": "1d5c29e4-2810-4ea3-ce23-3776eed5bdbc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting gputil\n",
            "  Downloading GPUtil-1.4.0.tar.gz (5.5 kB)\n",
            "Building wheels for collected packages: gputil\n",
            "  Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gputil: filename=GPUtil-1.4.0-py3-none-any.whl size=7410 sha256=f9aadf58dd38d4e617fdfbfd8d5276b3e3107aecfe0d04a1a84af345ccc656ef\n",
            "  Stored in directory: /root/.cache/pip/wheels/6e/f8/83/534c52482d6da64622ddbf72cd93c35d2ef2881b78fd08ff0c\n",
            "Successfully built gputil\n",
            "Installing collected packages: gputil\n",
            "Successfully installed gputil-1.4.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (5.4.8)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: humanize in /usr/local/lib/python3.7/dist-packages (0.5.1)\n",
            "Gen RAM Free: 12.6 GB  | Proc size: 96.1 MB\n",
            "GPU RAM Free: 15109MB | Used: 0MB | Util   0% | Total 15109MB\n"
          ]
        }
      ],
      "source": [
        "# memory footprint support libraries/code\n",
        "# https://medium.com/@oribarel/getting-the-most-out-of-your-google-colab-2b0585f82403\n",
        "!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi\n",
        "!pip install gputil\n",
        "!pip install psutil\n",
        "!pip install humanize\n",
        "import psutil\n",
        "import humanize\n",
        "import os\n",
        "import GPUtil as GPU\n",
        "GPUs = GPU.getGPUs()\n",
        "# XXX: only one GPU on Colab and isn’t guaranteed\n",
        "\n",
        "def printm():\n",
        " process = psutil.Process(os.getpid())\n",
        " print(\"Gen RAM Free: \" + humanize.naturalsize( psutil.virtual_memory().available ), \" | Proc size: \" + humanize.naturalsize( process.memory_info().rss))\n",
        " print(\"GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB\".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))\n",
        "#\n",
        "\n",
        "if len(GPUs) >0: \n",
        "  gpu = GPUs[0]\n",
        "  printm()\n",
        "else:\n",
        "  print ('no GPU. Are you sure the hardware accelerator is configured to GPU? To do this go to Runtime→Change runtime type and change the Hardware accelerator to GPU.') "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Quel carte GPU utilisez-vous ?"
      ],
      "metadata": {
        "id": "WzCqDlhgJtuM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%system nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZXxRxNLQJJTR",
        "outputId": "64968ce8-8e39-4655-a08b-755181eb9db5"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Mon Nov 14 12:37:02 2022       ',\n",
              " '+-----------------------------------------------------------------------------+',\n",
              " '| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |',\n",
              " '|-------------------------------+----------------------+----------------------+',\n",
              " '| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |',\n",
              " '| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |',\n",
              " '|                               |                      |               MIG M. |',\n",
              " '|===============================+======================+======================|',\n",
              " '|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |',\n",
              " '| N/A   40C    P8    11W /  70W |      3MiB / 15109MiB |      0%      Default |',\n",
              " '|                               |                      |                  N/A |',\n",
              " '+-------------------------------+----------------------+----------------------+',\n",
              " '                                                                               ',\n",
              " '+-----------------------------------------------------------------------------+',\n",
              " '| Processes:                                                                  |',\n",
              " '|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |',\n",
              " '|        ID   ID                                                   Usage      |',\n",
              " '|=============================================================================|',\n",
              " '|  No running processes found                                                 |',\n",
              " '+-----------------------------------------------------------------------------+']"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxVCJnn3OBq2"
      },
      "source": [
        "Configurez votre mode d'exécution du Google Colab en mode GPU.\n",
        "\n",
        "> To use the google colab in a GPU mode you have to make sure the hardware accelerator is configured to GPU. To do this go to Runtime→Change runtime type and change the Hardware accelerator to GPU. Sometimes, all GPUs are in use and there is no GPU available.\n",
        "\n",
        "Une fois configuré le GPU, vérifiez l'état de la mémoire sur la carte en réexécutant le code \"memory footprint\" ci-dessus. Le message a-t-il changé favorablement ?\n",
        "\n",
        "Le mode d'exécution GPU est nécessaire pour le fine-tuning de BERT ci-dessous. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VE0PnqNnM1tB"
      },
      "source": [
        "# Allociné dataset\n",
        "\n",
        "The [Allociné dataset](https://huggingface.co/datasets/allocine) is a French-language dataset for sentiment analysis. The texts are movie reviews written between 2006 and 2020 by members of the Allociné.fr community for various films. It contains 100k positive and 100k negative reviews divided into train (160k), validation (20k), and test (20k). \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxo9pzqyVwIG"
      },
      "source": [
        "### From huggingface datahub\n",
        "\n",
        "After execution of this cell, you must restart the runtime in order to use  newly installed versions.\n",
        "\n",
        "Just click on the button on the suggester \"restart runtime\" button (or go the runtime menu)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PXgmkHWnxIS8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0566d8d7-0948-4836-abd7-545d8d2079f5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting datasets\n",
            "  Downloading datasets-2.6.1-py3-none-any.whl (441 kB)\n",
            "\u001b[K     |████████████████████████████████| 441 kB 28.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from datasets) (3.8.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.3.5)\n",
            "Collecting dill<0.3.6\n",
            "  Downloading dill-0.3.5.1-py2.py3-none-any.whl (95 kB)\n",
            "\u001b[K     |████████████████████████████████| 95 kB 4.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (2022.10.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (4.64.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from datasets) (21.3)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (6.0.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from datasets) (1.21.6)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (2.23.0)\n",
            "Collecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (6.0)\n",
            "Collecting xxhash\n",
            "  Downloading xxhash-3.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[K     |████████████████████████████████| 212 kB 71.2 MB/s \n",
            "\u001b[?25hCollecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py37-none-any.whl (115 kB)\n",
            "\u001b[K     |████████████████████████████████| 115 kB 70.5 MB/s \n",
            "\u001b[?25hCollecting huggingface-hub<1.0.0,>=0.2.0\n",
            "  Downloading huggingface_hub-0.10.1-py3-none-any.whl (163 kB)\n",
            "\u001b[K     |████████████████████████████████| 163 kB 61.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from datasets) (4.13.0)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (2.1.1)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.8.1)\n",
            "Requirement already satisfied: asynctest==0.13.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (0.13.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (22.1.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (4.0.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (4.1.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (6.0.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->datasets) (3.0.9)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2022.9.24)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (3.0.4)\n",
            "Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1\n",
            "  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)\n",
            "\u001b[K     |████████████████████████████████| 127 kB 70.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->datasets) (3.10.0)\n",
            "Collecting multiprocess\n",
            "  Downloading multiprocess-0.70.13-py37-none-any.whl (115 kB)\n",
            "\u001b[K     |████████████████████████████████| 115 kB 73.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2022.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.15.0)\n",
            "Installing collected packages: urllib3, dill, xxhash, responses, multiprocess, huggingface-hub, datasets\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "  Attempting uninstall: dill\n",
            "    Found existing installation: dill 0.3.6\n",
            "    Uninstalling dill-0.3.6:\n",
            "      Successfully uninstalled dill-0.3.6\n",
            "Successfully installed datasets-2.6.1 dill-0.3.5.1 huggingface-hub-0.10.1 multiprocess-0.70.13 responses-0.18.0 urllib3-1.25.11 xxhash-3.1.0\n"
          ]
        }
      ],
      "source": [
        "!pip install datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Retrieve the [allociné dataset](https://huggingface.co/datasets/allocine) from the huggingface hub."
      ],
      "metadata": {
        "id": "1eCtRF0tbm5S"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qvYzPpHJwSEu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 562,
          "referenced_widgets": [
            "b6ca4143486641a3b73d59637328e2b4",
            "d2a818e98f3a48468dff2e729e626153",
            "123fbb1c677e459a9c396eba6c10209a",
            "e4ce991697f04cdd9307aa1f1dcac9de",
            "3aa79446cd4344569c887c5dfc920845",
            "8a3f5f76ed184afab5373acdf83092d0",
            "44212d9979634303a1f9fcf8ca48e4f7",
            "c2ef8b4c907d49f5a84186314e956df4",
            "844304d7c3d6458296188963a20c497c",
            "0540ef45da394a24bf9b8a56fb770aa4",
            "290665a7ce8244a6addfe5ac36dac895",
            "5bd35b298e854936a8b7675286a5ed6b",
            "6a1814b3a6ca4c02832e656cddf6ab19",
            "436eaf6fd1bc4fbeaba7b9d69deff439",
            "6f7122f4375c4886bd4c9f14cf0e7893",
            "70107a988d424ad5a38896465eedf7a9",
            "6101cd2353ce4c04b4b3496950b3ae08",
            "015aeb00d34a4cd5b7b9eda66e0d69c5",
            "0b25ef3cf844416e8d0e12a2c63fb421",
            "cf26db770b1d47fc9e40175b52599d2d",
            "7936a14f5d7846aaa481f036878a919e",
            "a93ae8a8f86548548e7cd3f5825eafc6",
            "79e9f189c0594597a596b9cea6575f94",
            "f5968cc4c37043a7bbb2827d7a43c345",
            "75c85e7bc6d1465482348c73f4f362c8",
            "88060cd5453149b1b09c583d1de6657d",
            "bb3b688410204dff82a72be1e34fd856",
            "292c1458931c44e897a665def8b91c87",
            "d53737423a9c4f399deb22a1ed9d4c31",
            "39e59fa8639c4a5b9a152ffa5e977662",
            "0f710f3aa1f9433a8dedbc9cf8a4beff",
            "a52dea77aa9e49f7979a85779d701775",
            "effd4418679f4fc99daccb1b90a0bdd3",
            "c210be85697848139222efc91b0a2622",
            "f39d47def2bb4348bf908b1300a3acf9",
            "a1f7c70f7a474c1495f3c64d6019597a",
            "0851f5d8339441cf8ebfe2d288585bd1",
            "3a0b31df2d8f4bb8b826a6a11bd18539",
            "bb01ccab0b064e1bb15af4569244f15b",
            "17f828ffbc6c41ce9be941cdbf81e8f5",
            "9ae6808d5b874956bb9daf98c881035e",
            "71ca06111d4947b2bd2a228d8e7c3ff5",
            "fcb57b52ff6746a2bcb6003038b339d1",
            "cbb5ed28efaa482a8f8c35210ea4ebb0",
            "7c32212ee16645ddb59cf8cf450af3a8",
            "4b3717e325d8417589c85225b14b1006",
            "497dcfc6e70348e58993c98e8d4db8ce",
            "bd7fbf8b063249b9b5952c9a597af85d",
            "d411b3a216784c21befb0e65481fcc60",
            "96166f4e8a95406b8541f1c1cb8578ba",
            "2429af983c9e4d338ed07c76e75cd853",
            "bcb9a4e292874a5d910a61195cce7716",
            "0e1048dbc4af46299504f39cc2cb10f3",
            "a54030d56e944d0787ab2d2798026ee1",
            "0d1a7c10f4574ad89386805051ec500d",
            "8d8821eb9d0845018531734443223ee5",
            "3b2d70a789e04e639e37ab4c8eb0db0b",
            "d06bcab78058463c82e248d51fdccd8f",
            "d8752ffde7f54643aab15b0bf075340a",
            "21c7af9682ab463ebbdeeb916d3a6794",
            "c518a3ba017a4340a1a3cf0919a37b04",
            "3795e5db3dc6443da11eb17b108e964d",
            "0c5331686b124aa79ca06f1f0e8da06e",
            "863154ee114b442f94c36ab147aad9b6",
            "a9be5759ab6e48f6b438ef3b17e142c1",
            "07d7ced7d8d74fb28b1487ccc9dc361a",
            "b052f51fc34649d69749824db4eb36e3",
            "51db043b08f9434b9eb30785d82ab43f",
            "ce25b9d87d2f4afe9b57351daf33b336",
            "7c56762e7e8644aeb0942f208aa44df3",
            "46eb11ce593642178655e9b7f441ae82",
            "c2d99cbc83424208b13583cf42b24d72",
            "0e99590640e146c4a6886507ea6317a0",
            "f94f480c488a4eb894395c840d095b73",
            "6245a9408a9642c2b6875a6603218eb7",
            "f7ede946fea7438fbf6e4342422aeaed",
            "86c84acfab1b4b63ba3bb6ef5a9ec8d9",
            "3104db207d224ee2b54972cf5898e774",
            "3473304e0c5f43b186b77a612a34764c",
            "4843f6b720a04f43b7c55671586c9592",
            "c9949e2640de4b2a9b105a7e401a95cb",
            "4ade5b9af208473eb9d2d7a5a20d8eee",
            "7208dbb9a99743c798ec59ba806a8956",
            "a81d69a20b314735a87865c2ccdca696",
            "975ef7a4f9084b2b962bcd240198f7ea",
            "612418dd948d418a8d00726269207c09",
            "150d263f96f740e19063ab6eb9b80c9b",
            "a094005d268346de9550a32b87dda977"
          ]
        },
        "outputId": "41279e87-eea1-4524-e49d-fa8a28e508ee"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/3.68k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b6ca4143486641a3b73d59637328e2b4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/1.96k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5bd35b298e854936a8b7675286a5ed6b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/9.11k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "79e9f189c0594597a596b9cea6575f94"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " Allocine Dataset: A Large-Scale French Movie Reviews Dataset.\n",
            " This is a dataset for binary sentiment classification, made of user reviews scraped from Allocine.fr.\n",
            " It contains 100k positive and 100k negative reviews divided into 3 balanced splits: train (160k reviews), val (20k) and test (20k).\n",
            "\n",
            "{'review': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}\n",
            "Downloading and preparing dataset allocine/allocine to /root/.cache/huggingface/datasets/allocine/allocine/1.0.0/ea86b1dc05eae3a45a07b6281f2d4033b5fe7927b1008d06aa457ca1eae660d0...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/66.6M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c210be85697848139222efc91b0a2622"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/160000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7c32212ee16645ddb59cf8cf450af3a8"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating validation split:   0%|          | 0/20000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8d8821eb9d0845018531734443223ee5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating test split:   0%|          | 0/20000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b052f51fc34649d69749824db4eb36e3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset allocine downloaded and prepared to /root/.cache/huggingface/datasets/allocine/allocine/1.0.0/ea86b1dc05eae3a45a07b6281f2d4033b5fe7927b1008d06aa457ca1eae660d0. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/3 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3104db207d224ee2b54972cf5898e774"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['review', 'label'],\n",
              "        num_rows: 160000\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['review', 'label'],\n",
              "        num_rows: 20000\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['review', 'label'],\n",
              "        num_rows: 20000\n",
              "    })\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "# https://huggingface.co/datasets/allocine\n",
        "\n",
        "from datasets import load_dataset_builder\n",
        "ds_builder = load_dataset_builder(\"allocine\")\n",
        "\n",
        "# Inspect dataset description\n",
        "print(ds_builder.info.description)\n",
        "\n",
        "# Inspect dataset features\n",
        "print(ds_builder.info.features)\n",
        "\n",
        "# get_dataset_split_names\n",
        "from datasets import get_dataset_split_names\n",
        "get_dataset_split_names(\"allocine\")\n",
        "\n",
        "# load_dataset\n",
        "from datasets import load_dataset\n",
        "allocine_dataset = load_dataset(\"allocine\")\n",
        "allocine_dataset\n",
        "#train_dataset = load_dataset(\"allocine\", split=\"train\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvdy-2-DV9zX"
      },
      "source": [
        "#### Select data ratio to process and create Pandas DataFrame\n",
        "\n",
        "Definition of a method to select a ratio of a dataset split. By default return 100 % of the split."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def get_dataset_split_ratio(dataset_split, seed=42, ratio=100):\n",
        "  if ratio > 100 or ratio < 1: ratio = 100\n",
        "  reviews = dataset_split['review']\n",
        "  labels = dataset_split['label']\n",
        "  \n",
        "  data = [pair for pair in zip(reviews, labels)]\n",
        "  \n",
        "  np.random.seed(seed)\n",
        "  np.random.shuffle(data)\n",
        "  \n",
        "  ratio_len = int(len(data)*ratio/100)\n",
        "  \n",
        "  data = list(zip(*data))\n",
        "\n",
        "  return {'review':data[0][:ratio_len], 'label':data[1][:ratio_len]}\n"
      ],
      "metadata": {
        "id": "AIdQWqGa7WmB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Definition of the method to create a dataframe from a huggingface dataset split (`{'review':[], 'label':[]}`) :"
      ],
      "metadata": {
        "id": "2v5boWFkDIhy"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BEkNPZg705jM"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# From huggingface dataset split to DataFrame\n",
        "def hf_dataset_split_to_df (huggingface_dataset_split):\n",
        "  df = pd.DataFrame(huggingface_dataset_split)\n",
        "  # https://github.com/amaiya/ktrain/blob/master/examples/text/ArabicHotelReviews-nbsvm.ipynb\n",
        "  df['label'] = df['label'].apply(lambda x: 'negative' if x == 0 else 'positive')\n",
        "  df = pd.concat([df, df.label.astype('str').str.get_dummies()], axis=1, sort=False)\n",
        "  df = df[['review', 'negative', 'positive']]\n",
        "  return df"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sélectionner la taille des données sur lesquelles vous voulez travailler et construire la dataframe correspondante.\n",
        "\n",
        "Le code ci-dessous extrait 5 % du corpus. C'est suffisant pour étudier comment fonctionne les modèles présentés ci-après sans attendre trop de temps d'entraînement."
      ],
      "metadata": {
        "id": "uqrzl25ODe2u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ratio = 5 # 100 # percent of the data\n",
        "\n",
        "train_df = hf_dataset_split_to_df(get_dataset_split_ratio(allocine_dataset['train'], 42, ratio))\n",
        "val_df = hf_dataset_split_to_df(get_dataset_split_ratio(allocine_dataset['validation'], 42, ratio))\n",
        "test_df = hf_dataset_split_to_df(get_dataset_split_ratio(allocine_dataset['test'], 42, ratio))\n",
        "\n",
        "print(train_df.head())\n",
        "print(train_df.describe())\n",
        "print('len(train_df) :', len(train_df))\n",
        "print('len(val_df) :', len(val_df))\n",
        "print('len(test_df) :', len(test_df))\n"
      ],
      "metadata": {
        "id": "jmwbpxGDCl66",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20dbdd64-ccb3-40cf-baba-a90967ac772f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                              review  negative  positive\n",
            "0  Un excellent thriller d'action où les scènes d...         0         1\n",
            "1  Si le scénariste, qui aurait pu faire un minim...         1         0\n",
            "2  Référence dans la filmographie de Bogart, \"Le ...         1         0\n",
            "3  Un bon scénario, un bon film, une histoire lou...         0         1\n",
            "4  Un scenario vide et une mise en scene trés sop...         1         0\n",
            "          negative     positive\n",
            "count  8000.000000  8000.000000\n",
            "mean      0.491125     0.508875\n",
            "std       0.499952     0.499952\n",
            "min       0.000000     0.000000\n",
            "25%       0.000000     0.000000\n",
            "50%       0.000000     1.000000\n",
            "75%       1.000000     1.000000\n",
            "max       1.000000     1.000000\n",
            "len(train_df) : 8000\n",
            "len(val_df) : 1000\n",
            "len(test_df) : 1000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLfw1G8wMxkz"
      },
      "source": [
        "# ktrain\n",
        "\n",
        "[ktrain](https://github.com/amaiya/ktrain) a lightweight wrapper for the deep learning library _TensorFlow Keras_ (and other libraries) to help build, train, and deploy neural networks and other machine learning models. Inspired by ML framework extensions like fastai and ludwig, ktrain is designed to make deep learning and AI more accessible and easier to apply for both newcomers and experienced practitioners.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Configuration et installation de ktrain\n",
        "\n",
        "Configuration de l'environnement"
      ],
      "metadata": {
        "id": "OMQniIpmoI6r"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jZ-F6wCZRYpw"
      },
      "outputs": [],
      "source": [
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline\n",
        "import os\n",
        "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
        "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"; "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qUvbU_z33-Lw"
      },
      "source": [
        "Installation de la bibliothèque ktrain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wxWRMgrKR7Q1",
        "outputId": "981eb115-40be-4d25-a9e4-cc37fd164f9f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting ktrain\n",
            "  Downloading ktrain-0.31.10.tar.gz (25.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 25.3 MB 1.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from ktrain) (1.0.2)\n",
            "Requirement already satisfied: matplotlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from ktrain) (3.2.2)\n",
            "Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from ktrain) (1.3.5)\n",
            "Requirement already satisfied: fastprogress>=0.1.21 in /usr/local/lib/python3.7/dist-packages (from ktrain) (1.0.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from ktrain) (2.23.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from ktrain) (1.2.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from ktrain) (21.3)\n",
            "Collecting langdetect\n",
            "  Downloading langdetect-1.0.9.tar.gz (981 kB)\n",
            "\u001b[K     |████████████████████████████████| 981 kB 63.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: jieba in /usr/local/lib/python3.7/dist-packages (from ktrain) (0.42.1)\n",
            "Collecting cchardet\n",
            "  Downloading cchardet-2.1.7-cp37-cp37m-manylinux2010_x86_64.whl (263 kB)\n",
            "\u001b[K     |████████████████████████████████| 263 kB 59.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: chardet in /usr/local/lib/python3.7/dist-packages (from ktrain) (3.0.4)\n",
            "Collecting syntok>1.3.3\n",
            "  Downloading syntok-1.4.4-py3-none-any.whl (24 kB)\n",
            "Collecting transformers==4.17.0\n",
            "  Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 46.8 MB/s \n",
            "\u001b[?25hCollecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3 MB 51.5 MB/s \n",
            "\u001b[?25hCollecting keras_bert>=0.86.0\n",
            "  Downloading keras-bert-0.89.0.tar.gz (25 kB)\n",
            "Collecting whoosh\n",
            "  Downloading Whoosh-2.7.4-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[K     |████████████████████████████████| 468 kB 65.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (4.13.0)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
            "\u001b[K     |████████████████████████████████| 880 kB 57.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (6.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (0.10.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (4.64.1)\n",
            "Collecting tokenizers!=0.11.3,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.6 MB 60.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (1.21.6)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (3.8.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->ktrain) (2022.6.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers==4.17.0->ktrain) (4.1.1)\n",
            "Collecting keras-transformer==0.40.0\n",
            "  Downloading keras-transformer-0.40.0.tar.gz (9.7 kB)\n",
            "Collecting keras-pos-embd==0.13.0\n",
            "  Downloading keras-pos-embd-0.13.0.tar.gz (5.6 kB)\n",
            "Collecting keras-multi-head==0.29.0\n",
            "  Downloading keras-multi-head-0.29.0.tar.gz (13 kB)\n",
            "Collecting keras-layer-normalization==0.16.0\n",
            "  Downloading keras-layer-normalization-0.16.0.tar.gz (3.9 kB)\n",
            "Collecting keras-position-wise-feed-forward==0.8.0\n",
            "  Downloading keras-position-wise-feed-forward-0.8.0.tar.gz (4.1 kB)\n",
            "Collecting keras-embed-sim==0.10.0\n",
            "  Downloading keras-embed-sim-0.10.0.tar.gz (3.6 kB)\n",
            "Collecting keras-self-attention==0.51.0\n",
            "  Downloading keras-self-attention-0.51.0.tar.gz (11 kB)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->ktrain) (0.11.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->ktrain) (3.0.9)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->ktrain) (1.4.4)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->ktrain) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->ktrain) (2022.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib>=3.0.0->ktrain) (1.15.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers==4.17.0->ktrain) (3.10.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->ktrain) (2022.9.24)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->ktrain) (1.25.11)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->ktrain) (2.10)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.17.0->ktrain) (7.1.2)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->ktrain) (1.7.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->ktrain) (3.1.0)\n",
            "Building wheels for collected packages: ktrain, keras-bert, keras-transformer, keras-embed-sim, keras-layer-normalization, keras-multi-head, keras-pos-embd, keras-position-wise-feed-forward, keras-self-attention, langdetect, sacremoses\n",
            "  Building wheel for ktrain (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ktrain: filename=ktrain-0.31.10-py3-none-any.whl size=25312981 sha256=a69826ab949b3d95ce27003ab23d034230089bb785720856460557fb4930e348\n",
            "  Stored in directory: /root/.cache/pip/wheels/9c/1c/1b/6df2db85720b8f5c6ea5e3ae37313cfc656f248abf910b7cfd\n",
            "  Building wheel for keras-bert (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-bert: filename=keras_bert-0.89.0-py3-none-any.whl size=33515 sha256=f67d5534f302a603c59225efe798af45a5099663c647c54b152b872c8fc77a06\n",
            "  Stored in directory: /root/.cache/pip/wheels/a4/e8/45/842b3a39831261aef9154b907eacbc4ac99499a99ae829b06f\n",
            "  Building wheel for keras-transformer (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-transformer: filename=keras_transformer-0.40.0-py3-none-any.whl size=12303 sha256=e3ea0af75f6595ee438aa2c95d1730594e48f7f3a516b6221742412751acf4e9\n",
            "  Stored in directory: /root/.cache/pip/wheels/46/68/26/692ed21edd832833c3b0a0e21615bcacd99ca458b3f9ed571f\n",
            "  Building wheel for keras-embed-sim (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-embed-sim: filename=keras_embed_sim-0.10.0-py3-none-any.whl size=3959 sha256=948231a97e4a8fa24aa4c3010b68e1f3220795a4e719d0ca64bd7c8356096972\n",
            "  Stored in directory: /root/.cache/pip/wheels/81/67/b5/d847588d075895281e1cf5590f819bd4cf076a554872268bd5\n",
            "  Building wheel for keras-layer-normalization (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-layer-normalization: filename=keras_layer_normalization-0.16.0-py3-none-any.whl size=4667 sha256=8f6c8fc7f06ba558275cc78c148bca21fb3066e168966239b9f935ea095f7d59\n",
            "  Stored in directory: /root/.cache/pip/wheels/85/5d/1c/2e619f594f69fbcf8bc20943b27d414871c409be053994813e\n",
            "  Building wheel for keras-multi-head (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-multi-head: filename=keras_multi_head-0.29.0-py3-none-any.whl size=14993 sha256=0764d79c630a8a13cd4cc1b056a750ee1ff85a39be6587862e2be7897a39f7b9\n",
            "  Stored in directory: /root/.cache/pip/wheels/86/aa/3c/9d15d24005179dae08ff291ce99c754b296347817d076fd9fb\n",
            "  Building wheel for keras-pos-embd (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-pos-embd: filename=keras_pos_embd-0.13.0-py3-none-any.whl size=6962 sha256=0602e5a4402dd65a6afe468c1ba8fe271387de7271ffea756b78ee0b3b92141a\n",
            "  Stored in directory: /root/.cache/pip/wheels/8d/c1/a0/dc44fcf68c857b7ff6be9a97e675e5adf51022eff1169b042f\n",
            "  Building wheel for keras-position-wise-feed-forward (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-position-wise-feed-forward: filename=keras_position_wise_feed_forward-0.8.0-py3-none-any.whl size=4983 sha256=955035fc94a6c6acfd713e91c118785185ed690116fc26d05e56146f0cbb71df\n",
            "  Stored in directory: /root/.cache/pip/wheels/c2/75/6f/d42f6e051506f442daeba53ff1e2d21a5f20ef8c411610f2bb\n",
            "  Building wheel for keras-self-attention (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-self-attention: filename=keras_self_attention-0.51.0-py3-none-any.whl size=18912 sha256=af9be4baea8dddf445932081231f855c9ab136e169073b246d12d9f75c3a65dc\n",
            "  Stored in directory: /root/.cache/pip/wheels/95/b1/a8/5ee00cc137940b2f6fa198212e8f45d813d0e0d9c3a04035a3\n",
            "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993242 sha256=e0b6d90f741ea2cb84f6403499c289dff44038788bf0b6d4572918a5b7e20d3e\n",
            "  Stored in directory: /root/.cache/pip/wheels/c5/96/8a/f90c59ed25d75e50a8c10a1b1c2d4c402e4dacfa87f3aff36a\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=bececfa930cfa8f80c184c2d983593d38715bccc9058a1010354c6002ae2c33a\n",
            "  Stored in directory: /root/.cache/pip/wheels/87/39/dd/a83eeef36d0bf98e7a4d1933a4ad2d660295a40613079bafc9\n",
            "Successfully built ktrain keras-bert keras-transformer keras-embed-sim keras-layer-normalization keras-multi-head keras-pos-embd keras-position-wise-feed-forward keras-self-attention langdetect sacremoses\n",
            "Installing collected packages: keras-self-attention, keras-position-wise-feed-forward, keras-pos-embd, keras-multi-head, keras-layer-normalization, keras-embed-sim, tokenizers, sacremoses, keras-transformer, whoosh, transformers, syntok, sentencepiece, langdetect, keras-bert, cchardet, ktrain\n",
            "Successfully installed cchardet-2.1.7 keras-bert-0.89.0 keras-embed-sim-0.10.0 keras-layer-normalization-0.16.0 keras-multi-head-0.29.0 keras-pos-embd-0.13.0 keras-position-wise-feed-forward-0.8.0 keras-self-attention-0.51.0 keras-transformer-0.40.0 ktrain-0.31.10 langdetect-1.0.9 sacremoses-0.0.53 sentencepiece-0.1.97 syntok-1.4.4 tokenizers-0.13.2 transformers-4.17.0 whoosh-2.7.4\n"
          ]
        }
      ],
      "source": [
        "!pip install ktrain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o7LRCuPB4p-y"
      },
      "source": [
        "Import de la bibliothèque ktrain dédié aux traitements de la modalité textuelle"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2avp8suURzMT"
      },
      "outputs": [],
      "source": [
        "# Execution time 30 s\n",
        "import ktrain\n",
        "from ktrain import text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VYGyWUwn9yPS"
      },
      "source": [
        "## Fonctionnement de ktrain\n",
        "\n",
        "\n",
        "Pour réaliser cette tâche de classification nous allons utiliser la bibliothèque ktrain pour faire un apprentissage par transfert.\n",
        "\n",
        "Les étapes sont les suivantes\n",
        "1. chargement des données avec application d'un prétraitement défini à la volée\n",
        "2. construction d'un modèle de classification sur la base d'un modèle pré-entraîné spécifié\n",
        "3. récupération d'une instance du modèle pour la personnalisation de celui-ci\n",
        "4. recherche d'un bon taux d'apprentissage\n",
        "5. entraînement du classifieur i.e. personnalisation du modèle de base à l'aide d'un taux d'apprentissage défini\n",
        "6. utilisation du nouveau modèle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UAvPw6g49WGa"
      },
      "source": [
        "*ETAPE 1 :* \n",
        "\n",
        "Le type de pré-traitement est fonction du modèle pré-entraîné spécifié."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OTBW_SEO9mXi"
      },
      "source": [
        "*ETAPE 2 :*\n",
        "\n",
        "ktrain vient avec quelques modèles pré-entraînés packagés. Pour les connaître, exécutez : "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P7FvrdBC9n2W",
        "outputId": "f026f082-cb4f-4dba-8030-b1a136f71a04"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fasttext: a fastText-like model [http://arxiv.org/pdf/1607.01759.pdf]\n",
            "logreg: logistic regression using a trainable Embedding layer\n",
            "nbsvm: NBSVM model [http://www.aclweb.org/anthology/P12-2018]\n",
            "bigru: Bidirectional GRU with pretrained fasttext word vectors [https://fasttext.cc/docs/en/crawl-vectors.html]\n",
            "standard_gru: simple 2-layer GRU with randomly initialized embeddings\n",
            "bert: Bidirectional Encoder Representations from Transformers (BERT) from keras_bert [https://arxiv.org/abs/1810.04805]\n",
            "distilbert: distilled, smaller, and faster BERT from Hugging Face transformers [https://arxiv.org/abs/1910.01108]\n"
          ]
        }
      ],
      "source": [
        "text.print_text_classifiers()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ae1TJ5nk_K7G"
      },
      "source": [
        "Vous connaissez **[fasttext](https://github.com/facebookresearch/fastText)** de (Facebook/Meta) car vous l'avez déjà utilisé dans un [précédent TP](https://github.com/nicolashernandez/teaching_nlp/blob/main/04_repr%C3%A9sentation_vectorielle_continue.ipynb)... fasttext permet de produire une représentation continue des mots en suivant une approche à la word2vec. Sa spécificité est qu'il propose de traiter la variabilité morphologique des mots en construisant des vecteurs non pas pour des mots mais pour des sous-mots (séquence de caractères). Le vecteur d'un mot est la somme de tous les vecteurs des sous-mots le composant. Cette approche est indépendante de la langue, et montre de meilleurs résultats que word2vec sur des tâches syntaxiques, surtout quand le corpus d'entraînement est petit. Word2vec est légèrement meilleur pour des tâches sémantiques. Un des avantage de FastText est de pouvoir fournir des vecteurs mêmes pour les mots hors vocabulaires.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6XtFCEfYWchi"
      },
      "source": [
        "[**NBSVM**](https://medium.com/@asmaiya/a-neural-implementation-of-nbsvm-in-keras-d4ef8c96cb7c) is an approach to text classification proposed by [Wang and Manning](https://www.aclweb.org/anthology/P12-2018) that takes a linear model such as SVM (or logistic regression) and infuses it with Bayesian probabilities by replacing word count features with Naive Bayes log-count ratios. Despite its simplicity, NBSVM models have been shown to be both fast and powerful across a wide range of different text classification datasets. \n",
        "Keras offers a NBSVM model implemented as a neural network using two embedding layers. The first stores the Naive Bayes log-count ratios. The second stores learned weights (or coefficients) for each feature (i.e., word) in this linear model. The prediction, then, is simply the dot product of these two vectors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0H3RJMw_xhB"
      },
      "source": [
        "**BERT** (_Bidirectional Encoder Representations from Transformers_), proposé par [Google AI Language](https://arxiv.org/pdf/1810.04805.pdf) est un encodeur bidirectionnel qui applique un modèle d'attention Transformers à la modélisation du language (_language modeling_). Il représente l'état de l'art.\n",
        "Les Transformers sont un mécanisme d'attention qui apprennent les relations contextuelles entre les mots dans un texte.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rB3bN2mE_xyi"
      },
      "source": [
        "Il est possible d'[**encapsuler les modèles Transformers du site _hugging face_**](https://github.com/amaiya/ktrain/blob/master/tutorials/tutorial-A3-hugging_face_transformers.ipynb). _hugging face_ diffuse les [modèles Transformers pré-entraînés \"officiels\"](https://huggingface.co/transformers/pretrained_models.html) ainsi que les [modèles Transformers construits par la communauté](https://huggingface.co/models).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6wINsCEpD1-s"
      },
      "source": [
        "*ETAPE 4 :*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4syPI6CI6Vt6"
      },
      "source": [
        "Le **taux d'apprentissage (_learning rate_)** est un hyperparamètre qui contrôle combien le modèle doit changer en réponse à l'erreur estimée à chaque fois que les poids du modèles sont mis à jour. Choisir un 'lr' trop petit conduit à une longue phase d'entraînement qui peut resté bloquée. Choisir un 'lr' trop grand conduit à un apprentissage sous-optimal des poids et à une instabilité du processus d'entraînement. \n",
        "\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CMJHB4zgRCTw"
      },
      "source": [
        "## FastText\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPc3RUc_TYDc"
      },
      "source": [
        "*ETAPE 1 :* \n",
        "\n",
        "Les méthodes [`texts_from_csv`](https://amaiya.github.io/ktrain/text/index.html#ktrain.text.texts_from_csv) ou [`texts_from_df`](https://amaiya.github.io/ktrain/text/index.html#ktrain.text.texts_from_df) charge le corpus, normalise les documents (définit un préprocesseur réutilisable), et découpe la collection en données d'entraînement et données de validation (à moins que des données de validation soient passées en argument).\n",
        "\n",
        "Cette étape est commune aux modèles FastText et NBSVM (`preprocess_mode='standard'`). BERT utilise son propre tokenizer. Il faudra faire un pré-traitement dédié (`preprocess_mode='bert'`).\n",
        "\n",
        "```\n",
        "* train_filepath(str): file path to training CSV\n",
        "* text_column(str): name of column containing the text\n",
        "* label_column(list): list of columns that are to be treated as labels\n",
        "* val_filepath(string): file path to test CSV.  If not supplied, 10% of documents in training CSV will be used for testing/validation.\n",
        "* max_features(int): max num of words to consider in vocabulary ; Note: This is only used for preprocess_mode='standard'.\n",
        "* maxlen(int): each document can be of most <maxlen> words. 0 is used as padding ID.\n",
        "* ngram_range(int): size of multi-word phrases to consider e.g., 2 will consider both 1-word phrases and 2-word phrases limited by max_features\n",
        "* preprocess_mode (str):  Either 'standard' (normal tokenization) or one of {'bert', 'distilbert'} tokenization and preprocessing for use with                BERT/DistilBert text classification model.\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "        \n",
        "  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uuKxqzFd3QBU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a81cbac4-851f-44f9-b6fb-e63cec11f4b5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['negative', 'positive']\n",
            "   negative  positive\n",
            "0         0         1\n",
            "1         1         0\n",
            "2         1         0\n",
            "3         0         1\n",
            "4         1         0\n",
            "['negative', 'positive']\n",
            "   negative  positive\n",
            "0         0         1\n",
            "1         0         1\n",
            "2         1         0\n",
            "3         1         0\n",
            "4         1         0\n",
            "language: fr\n",
            "Word Counts: 44776\n",
            "Nrows: 8000\n",
            "8000 train sequences\n",
            "train sequence lengths:\n",
            "\tmean : 87\n",
            "\t95percentile : 239\n",
            "\t99percentile : 305\n",
            "x_train shape: (8000,400)\n",
            "y_train shape: (8000, 2)\n",
            "Is Multi-Label? False\n",
            "1000 test sequences\n",
            "test sequence lengths:\n",
            "\tmean : 89\n",
            "\t95percentile : 247\n",
            "\t99percentile : 303\n",
            "x_test shape: (1000,400)\n",
            "y_test shape: (1000, 2)\n"
          ]
        }
      ],
      "source": [
        "# fasttext\n",
        "#NUM_WORDS = 50000\n",
        "#MAXLEN = 150\n",
        "#NGRAMS_SIZE = 1# 1 # 8 minutes avec 2 pour 10000 examples\n",
        "\n",
        "# nbsvm \n",
        "#NUM_WORDS = 80000\n",
        "#MAXLEN = 2000\n",
        "#NGRAMS_SIZE = 3\n",
        "\n",
        "(x_train_preproc, y_train_preproc), (x_val_preproc, y_val_preproc), preproc = text.texts_from_df (train_df, \n",
        "                                                                   'review', # name of column containing review text\n",
        "                                                                   label_columns=['negative', 'positive'],\n",
        "                                                                   val_df=val_df, # if None, 10% of data will be used for validation\n",
        "                                          #max_features=NUM_WORDS, \n",
        "                                          #maxlen=MAXLEN,\n",
        "                                          #ngram_range=NGRAMS_SIZE,\n",
        "                                          preprocess_mode='standard' # default\n",
        "                                          )\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LktiVGh8NVSW"
      },
      "source": [
        "Observons les 5 premières des données prétraitées. `x_` représente la donnée et `y_` la classe. On note que les données ont été transformées. Chaque mot est remplacé par un identifiant. On note que la classe est décrite par 2 colonnes avec deux codes \"1 0\" et \"0 1\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PzzWhW3ZHAq8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc458896-0a53-48b3-d48a-49b4ee347cc6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "x_train_preproc [[    0     0     0 ...   838     1  2855]\n",
            " [    0     0     0 ...     8     1   766]\n",
            " [    0     0     0 ...  2425     1     4]\n",
            " [    0     0     0 ...   453   368   707]\n",
            " [    0     0     0 ... 10369    46    85]]\n",
            "y_train_preproc [[0. 1.]\n",
            " [1. 0.]\n",
            " [1. 0.]\n",
            " [0. 1.]\n",
            " [1. 0.]]\n"
          ]
        }
      ],
      "source": [
        "print ('x_train_preproc', x_train_preproc[:5])\n",
        "print ('y_train_preproc', y_train_preproc[:5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHhCXpcQOPje"
      },
      "source": [
        "*ETAPE 2 et 3:* \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w3jEacvRRuih"
      },
      "outputs": [],
      "source": [
        "# Build and return a text classification model https://amaiya.github.io/ktrain/text/index.html#ktrain.text.text_classifier\n",
        "fasttext_model = text.text_classifier('fasttext', (x_train_preproc, y_train_preproc), preproc=preproc)\n",
        "\n",
        "# Returns a Learner instance that can be used to tune and train Keras models https://amaiya.github.io/ktrain/index.html#ktrain.get_learner\n",
        "fasttext_learner = ktrain.get_learner(fasttext_model, train_data=(x_train_preproc, y_train_preproc), val_data=(x_val_preproc, y_val_preproc))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ufKylXAeOVCU"
      },
      "source": [
        "*ETAPE 4 :* \n",
        "\n",
        "La méthode [`lr_find`](https://amaiya.github.io/ktrain/core.html#ktrain.core.Learner.lr_find) implémente la méthode [*Cyclical Learning Rates*](https://arxiv.org/abs/1506.01186) qui permet d'estimer un lr sans une exploration systématique d'un lr qui décroit. L'approche peut prendre seulement le temps de quelques itérations (i.e. du traitement de quelques batchs) sans même tourner sur plusieurs époques. Pour rappel, une époque correspond au traitement de l'ensemble des données d'entrainements par lot/batch d'exemples. *For example,\n",
        "CIFAR-10 has 50, 000 training images and the batchsize is\n",
        "100 so an epoch = 50, 000/100 = 500 iterations*. \n",
        "\n",
        "Then plots loss as learning rate is increased.\n",
        "\n",
        "*Highest learning rate corresponding to a still falling loss should be chosen*. \"[The point of optimality [...] is the point on the training rate plot where the slope is steepest in the downwards direction. That's because this plot shows the loss after one epoch. Points before the steepest slope are training too slowly. Points after the steepest slope are at risk of training too quickly: usually, but not always (it didn't happen in this demo case), they will fall off the mountain in terms of loss because they jump past the point of optimality](https://www.kaggle.com/code/residentmario/finding-an-optimal-learning-rate-with-lr-finder)\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_9_32BN5Ur7e"
      },
      "outputs": [],
      "source": [
        "# recherche d'un bon taux d'apprentissage \n",
        "# you can set max_epochs (e.g., max_epochs=5) to estimate LR\n",
        "fasttext_learner.lr_find() # \n",
        "fasttext_learner.lr_plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0eWERAHZ_Go"
      },
      "source": [
        "*ETAPE 5 :* \n",
        "\n",
        "[autofit](https://amaiya.github.io/ktrain/core.html#ktrain.core.Learner.autofit)\n",
        "Automatically train model using a default learning rate schedule shown to work well in practice.  By default, this method currently employs a triangular learning rate policy (https://arxiv.org/abs/1506.01186).  \n",
        "During each epoch, this learning rate policy varies the learning rate from lr/10 to lr and then back to a low learning rate that is near-zero. \n",
        "If epochs is None, then early_stopping and reduce_on_plateau are atomatically\n",
        "set to 6 and 3, respectively.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOMvkq1ZPXV2"
      },
      "source": [
        "#### QUESTION : training\n",
        "\n",
        "* Sur le graph ci-dessus, repérez approximativement la puissance `n` de `1/10^n` où la chute de la loss devient importante. Testez avec cette valeur comme learning rate, observez votre performance (accuracy) sur le train et le val.\n",
        "\n",
        "Dans l'extrait de log ci-dessous, loss et accuracy concerne le corpus de train tandis que  val_loss et val_accuracy le corpus de validation (sur un corpus sample).\n",
        "```\n",
        "Epoch 12/1024\n",
        "244/250 [============================>.] - ETA: 0s - loss: 0.1109 - accuracy: 0.9589\n",
        "Restoring model weights from the end of the best epoch: 7.\n",
        "250/250 [==============================] - 1s 6ms/step - loss: 0.1108 - accuracy: 0.9589 - val_loss: 0.3864 - val_accuracy: 0.8820\n",
        "Epoch 12: early stopping\n",
        "```\n",
        "* Est-ce normal d'observer un écart d'accuracy entre le train et le val ?\n",
        "* Stoquez le score obtenu en dernière étape, et relancez le finetuning sans changer le paramétrage. Obtenez-vous les mêmes résultats ? Pourquoi ? La réponse est à chercher dans l'initiatilisation des poids du réseau neuronal.\n",
        "* Volontairement (et selon le temps qu'a pris votre entraînement) testez des lr avec d'autres puissances de 10 (0.1, 0.01, 0.001, 0.0001). Attention les lr les plus petites prendront le plus de temps !!! Est-ce que cela marche mieux ? Attention, suivant votre choix de lr l'entraînement peut prendre quelques minutes à au moins 1h....\n",
        "\n",
        "Mon meilleur score est `loss: 0.1108 - accuracy: 0.9589 - val_loss: 0.3864 - val_accuracy: 0.8820` sur le corpus sample et vous ?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5vY7oJU7U4G0"
      },
      "outputs": [],
      "source": [
        "LEARNING_RATE = 0.01\n",
        "\n",
        "fasttext_learner.autofit(LEARNING_RATE)\n",
        "# Epoch 16/1024\n",
        "# 4491/4500 [============================>.] - ETA: 0s - loss: 0.1540 - accuracy: 0.9418Restoring model weights from the end of the best epoch: 11.\n",
        "# 4500/4500 [==============================] - 25s 6ms/step - loss: 0.1539 - accuracy: 0.9418 - val_loss: 0.2173 - val_accuracy: 0.9206\n",
        "# Epoch 16: early stopping\n",
        "\n",
        "# wi all allocine training dataset (at least one hour run) \n",
        "#learner.autofit(0.00001)\n",
        "# Epoch 129/1024\n",
        "# 4497/4500 [============================>.] - ETA: 0s - loss: 0.2197 - accuracy: 0.9135Restoring model weights from the end of the best epoch: 124.\n",
        "# 4500/4500 [==============================] - 29s 6ms/step - loss: 0.2197 - accuracy: 0.9135 - val_loss: 0.2058 - val_accuracy: 0.9204\n",
        "# Epoch 129: early stopping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqItionrS_z0"
      },
      "source": [
        "#### VOTRE REPONSE\n",
        "\n",
        "**TODO**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "save/reload a model\n",
        "\n",
        "https://github.com/amaiya/ktrain/blob/master/FAQ.md#how-do-i-resume-training-from-a-saved-checkpoint"
      ],
      "metadata": {
        "id": "ad4FYJW1GBy_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# save Predictor (i.e., model and Preprocessor instance) after partially training\n",
        "ktrain.get_predictor(fasttext_learner.model, preproc).save('fasttext_allocine.model+preproc')\n",
        "\n",
        "# reload Predictor and extract model\n",
        "#model = ktrain.load_predictor('/tmp/my_predictor').model"
      ],
      "metadata": {
        "id": "2v3AG0hQGBzC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ySqltk4KTHJ7"
      },
      "source": [
        "*ETAPE 6 :* Le code ci-dessous permet d'utiliser le modèle. Notez que le paramétrage `return_proba=True` permet d'obtenir les probas... "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qOiht1hPani6"
      },
      "outputs": [],
      "source": [
        "fasttext_predictor = ktrain.get_predictor(fasttext_learner.model, preproc)\n",
        "\n",
        "data = [ \"Ce film était horrible ! L'intrigue était ennuyeuse. Le jeu d'acteur était correct, cependant.\",\n",
        "         \"Le film est vraiment nul. Je veux qu'on me rende mon argent.\",\n",
        "        \"Quelle belle comédie romantique. 10/10 à revoir !\"]\n",
        "\n",
        "# Makes predictions for a list of strings where each string is a document or text snippet.\n",
        "print (fasttext_predictor.predict(data))\n",
        "\n",
        "# If return_proba is True, returns probabilities of each class.\n",
        "print (fasttext_predictor.predict(data,  return_proba=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k5l8TyRWThZF"
      },
      "source": [
        "#### QUESTION : évaluation qualitative légère\n",
        "\n",
        "* Tester le modèle. Arrivez-vous à piéger le modèle ? Avec quelle phrase (donnez le code)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JkowscJhTiPh"
      },
      "source": [
        "#### VOTRE REPONSE\n",
        "\n",
        "**TODO**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bA5hf8I1dXul"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNZjuU8Bc4qI"
      },
      "source": [
        "### QUESTION : Evaluation quantitative\n",
        "\n",
        "Génération d'une hypothèse pour les *review* du corpus de test (`list(test_df['review'])`)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# reviews \n",
        "x_test = list(test_df['review'])\n",
        "\n",
        "# labels (gold) \n",
        "y_test = list(test_df['positive'])"
      ],
      "metadata": {
        "id": "V6BGVxPPvzvd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "32x1JdOCc5Dw"
      },
      "outputs": [],
      "source": [
        "# hypothèse\n",
        "y_hyp = [0 if h == 'negative' else 1 for h in fasttext_predictor.predict(x_test) ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNPvtEt0PzK4"
      },
      "source": [
        "\n",
        "\n",
        "[SemEval 2022 isarcasmeval task](https://sites.google.com/view/semeval2022-isarcasmeval) used the F1-score for the sarcastic class. sklearn permet de spécifier cette mesure avec le paramètre `average = 'binary'` éventuellement accompagné du paramètre qui indique la classe à considérer `pos_label = 1`. This metric should not be confused with the regular macro-F1. sklearn permet de calculer un score macro à l'aide de la valeur `average = 'macro'`.\n",
        "Sans paramétrage c'est le F1-score d'une classe qui est retourné. Les performances rapportées par [Theophile Blart utilise aussi ce paramétrage par défaut](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YI0p4xFFPzch",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 236
        },
        "outputId": "73ff7458-7b43-43d7-ab5a-a5b3c21696d7"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-db23f447e36c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmetrics\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mf1_score\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprecision_score\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecall_score\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mf1_positive\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf1_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_hyp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maverage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"binary\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpos_label\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mf1_positive\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m#OR\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'y_hyp' is not defined"
          ]
        }
      ],
      "source": [
        "from sklearn.metrics import f1_score, precision_score, recall_score\n",
        "\n",
        "f1_positive = f1_score(y_test, y_hyp, average = \"binary\", pos_label = 1)\n",
        "f1_positive\n",
        "#OR\n",
        "#p_score_positive = precision_score(y_test, y_hyp, average = \"binary\", pos_label = 1)\n",
        "#r_score_positive = recall_score(y_test, y_hyp, average = \"binary\", pos_label = 1)\n",
        "#f1_positive = (2*p_score_positive*r_score_positive)/(p_score_positive +r_score_positive)\n",
        "\n",
        "# 5 %   0.8922764227642277\n",
        "# 100 % 0.9181408981471789"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQ1neD23ao8T"
      },
      "source": [
        "[Theophile Blard rapporte des performances avec différents modèles (CamemBERT, RNN, TF-IDF + LogReg, CNN, fastText (unigrams)) sur le dataset Allociné](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert). Comment vous positionnez-vous par rapport à ses résultats ?   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ApxsjcDHE4Im"
      },
      "source": [
        "## NBSVM\n",
        "\n",
        "Le code ci-dessous utilise le même pré-traitement que précédemment et applique un modèle neuronal plus \"simple\" que fasttext.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uGp2mrBsg17I"
      },
      "outputs": [],
      "source": [
        "# load an NBSVM model\n",
        "nbsvm_model = text.text_classifier('nbsvm', (x_train_preproc, y_train_preproc), preproc=preproc)\n",
        "nbsvm_learner = ktrain.get_learner(nbsvm_model, train_data=(x_train_preproc, y_train_preproc), val_data=(x_val_preproc, y_val_preproc))\n",
        "\n",
        "# fine tune\n",
        "LEARNING_RATE = 0.01\n",
        "nbsvm_learner.autofit(LEARNING_RATE)\n",
        "# Epoch 6/1024\n",
        "#4498/4500 [============================>.] - ETA: 0s - loss: 0.1264 - accuracy: 0.9544Restoring model weights from the end of the best epoch: 1.\n",
        "#4500/4500 [==============================] - 24s 5ms/step - loss: 0.1264 - accuracy: 0.9544 - val_loss: 0.2677 - val_accuracy: 0.9161\n",
        "#Epoch 6: early stopping\n",
        "\n",
        "# Finally, we will fit our model using and [SGDR learning rate](https://github.com/amaiya/ktrain/blob/master/example-02-tuning-learning-rates.ipynb) schedule by invoking the fit method with the cycle_len parameter (along with the cycle_mult parameter).\n",
        "# learner.fit(0.001, 3, cycle_len=1, cycle_mult=2)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "évaluation"
      ],
      "metadata": {
        "id": "JsEYbrTGxtv0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# données\n",
        "x_test = list(test_df['review']) # reviews \n",
        "y_test = list(test_df['positive']) # labels (gold) \n",
        "\n",
        "# prédiction\n",
        "nbsvm_predictor = ktrain.get_predictor(nbsvm_learner.model, preproc)\n",
        "y_hyp = [0 if h == 'negative' else 1 for h in nbsvm_predictor.predict(x_test) ]\n",
        "\n",
        "# évaluation\n",
        "print(f1_score(y_test, y_hyp))\n",
        "\n",
        "# 5   % 0.903981180311341\n",
        "# 100 % 0.9203585411085926\n"
      ],
      "metadata": {
        "id": "uQhc_d1F_St9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "save/reload a model\n",
        "\n",
        "https://github.com/amaiya/ktrain/blob/master/FAQ.md#how-do-i-resume-training-from-a-saved-checkpoint"
      ],
      "metadata": {
        "id": "j85WVlKO54h1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# save Predictor (i.e., model and Preprocessor instance) after partially training\n",
        "ktrain.get_predictor(nbsvm_learner.model, preproc).save('nbsvm_allocine.model+preproc')\n",
        "\n",
        "# reload Predictor and extract model\n",
        "#model = ktrain.load_predictor('/tmp/my_predictor').model"
      ],
      "metadata": {
        "id": "vZ8IjKQWFhFS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xveDFwSnUzj1"
      },
      "source": [
        "#### QUESTION\n",
        "\n",
        "* Le modèle nbsvm est-il plus performant que le précédent sur les données de validations ? Vous pouvez tester aussi différents learning rate."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ilof7YlQNtfz"
      },
      "source": [
        "## BERT \n",
        "\n",
        "L'usage du modèle bert requiert que l'on change le prétraitement des données en entrée. Le code suivant réalise le prétraitement, charge un modèle bert et lance la personnalisation (fine tuning) sur 1 cycle avec taux d'apprentissage fixé.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 521
        },
        "id": "iAZQI0ucQWSa",
        "outputId": "9d8fcd20-9fdb-4bb8-b7a8-cfe913292e41"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['negative', 'positive']\n",
            "   negative  positive\n",
            "0         0         1\n",
            "1         1         0\n",
            "2         1         0\n",
            "3         0         1\n",
            "4         1         0\n",
            "['negative', 'positive']\n",
            "   negative  positive\n",
            "0         0         1\n",
            "1         0         1\n",
            "2         1         0\n",
            "3         1         0\n",
            "4         1         0\n",
            "downloading pretrained BERT model (multi_cased_L-12_H-768_A-12.zip)...\n",
            "[██████████████████████████████████████████████████]\n",
            "extracting pretrained BERT model...\n",
            "done.\n",
            "\n",
            "cleanup downloaded zip...\n",
            "done.\n",
            "\n",
            "preprocessing train...\n",
            "language: fr\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
              "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "done."
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Is Multi-Label? False\n",
            "preprocessing test...\n",
            "language: fr\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
              "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "done."
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "# ETAPE 1 \n",
        "(x_train_preproc, y_train_preproc), (x_val_preproc, y_val_preproc), preproc = text.texts_from_df (train_df, \n",
        "                      'review',\n",
        "                      label_columns = [\"negative\", \"positive\"],\n",
        "                      val_df= val_df, # if None, 10% of data will be used for validation\n",
        "                      ##max_features=NUM_WORDS, \n",
        "                      #maxlen=MAXLEN,\n",
        "                      preprocess_mode='bert' \n",
        "                      )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kpI2LG_Wvqe"
      },
      "source": [
        "#### QUESTION\n",
        "* Exécutez le code en observant l'occupation de la RAM et attendez jusqu'à voir le temps prévisionnel s'afficher. Etes-vous choqué par le temps affiché ? BERT est très gros. Il requiert un peu de temps... Sur l'extrait le temps devrait être de 15 minutes à quelques heures sur le corpus complet.\n",
        "* Si vous êtes en phase d'exploration et que vous ne travaillez pas sur un extrait des données, stoppez l'exécution dans la cellule, et tentez de changer la taille du batch_size (quantité de données traitées en même temps). Vous pouvez tester 12 et éventuellement 128... comme valeurs. Ne lachez pas la barre de la RAM des yeux. Ça passe ? Quel problème rencontrez-vous ? \n",
        "* Reprenez... et passez à la suite..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G0ujB3xihhnG",
        "outputId": "7cf8b989-59c1-42b3-9708-6861a89e1433"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Is Multi-Label? False\n",
            "maxlen is 400\n",
            "done.\n",
            "\n",
            "\n",
            "begin training using onecycle policy with max lr of 2e-05...\n",
            "26667/26667 [==============================] - 18569s 695ms/step - loss: 0.1856 - accuracy: 0.9256 - val_loss: 0.1116 - val_accuracy: 0.9588\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f72f1272b10>"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "# ETAPE 2 et 3\n",
        "bert_model = text.text_classifier('bert', (x_train_preproc, y_train_preproc) , preproc=preproc)\n",
        "bert_learner = ktrain.get_learner(bert_model, \n",
        "                             train_data=(x_train_preproc, y_train_preproc), \n",
        "                             val_data=(x_val_preproc, y_val_preproc), \n",
        "                             batch_size=6)\n",
        "# ETAPE 5\n",
        "bert_learner.fit_onecycle(2e-5, 1)\n",
        "\n",
        "# Is Multi-Label? False\n",
        "# maxlen is 400\n",
        "# begin training using onecycle policy with max lr of 2e-05...\n",
        "# 26667/26667 [==============================] - 18569s 695ms/step - loss: 0.1856 - accuracy: 0.9256 - val_loss: 0.1116 - val_accuracy: 0.9588\n",
        "# <keras.callbacks.History at 0x7f72f1272b10>\n",
        "# Exécution ~ 5h"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "évaluation"
      ],
      "metadata": {
        "id": "N3x1kk3U0P2Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# données\n",
        "x_test = list(test_df['review'])   # reviews \n",
        "y_test = list(test_df['positive']) # labels (gold) \n",
        "\n",
        "# prédiction\n",
        "bert_predictor = ktrain.get_predictor(bert_learner.model, preproc)\n",
        "y_hyp = [0 if h == 'negative' else 1 for h in bert_predictor.predict(x_test) ]\n",
        "\n",
        "# évaluation\n",
        "from sklearn.metrics import f1_score, precision_score, recall_score\n",
        "print(f1_score(y_test, y_hyp))\n",
        "# 100 % 0.9581906836808797"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4bbb8db3-71dc-4816-a2b7-73cb7be5f6a4",
        "id": "NH7qjU610P2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.9581906836808797\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "save/reload a model\n",
        "\n",
        "https://github.com/amaiya/ktrain/blob/master/FAQ.md#how-do-i-resume-training-from-a-saved-checkpoint"
      ],
      "metadata": {
        "id": "0xEUlatHGQzE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# save Predictor (i.e., model and Preprocessor instance) after partially training\n",
        "ktrain.get_predictor(bert_learner.model, preproc).save('bert_allocine.model+preproc')\n",
        "\n",
        "# reload Predictor and extract model\n",
        "#model = ktrain.load_predictor('/tmp/my_predictor').model"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kSdge2OQzCRf",
        "outputId": "d931a005-1e6a-42cf-93bb-4f4419fa929a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Function `_wrapped_model` contains input name(s) Input-Token, Input-Segment with unsupported characters which will be renamed to input_token, input_segment in the SavedModel.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1aed390> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f1ae1690> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f180c810> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f18ffd10> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f1a5e690> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1909a50> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1742090> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f18f2bd0> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f0f1fe90> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1a5e750> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f19bcd50> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f19c2590> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f159fe90> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f15a26d0> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f0f00d10> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f12f69d0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f1304350> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1309bd0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f10273d0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f0fb93d0> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f0fc5290> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f193df50> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f1706d50> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f17a39d0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1636ed0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f1645ed0> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1652cd0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f14cea50> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f156ee50> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f14bc290> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f13dca90> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f13eb950> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f14eedd0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f12c4fd0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_multi_head.multi_head_attention.MultiHeadAttention object at 0x7f72f13c91d0> has the same name 'MultiHeadAttention' as a built-in Keras object. Consider renaming <class 'keras_multi_head.multi_head_attention.MultiHeadAttention'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f1264a90> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n",
            "WARNING:absl:<keras_layer_normalization.layer_normalization.LayerNormalization object at 0x7f72f11d71d0> has the same name 'LayerNormalization' as a built-in Keras object. Consider renaming <class 'keras_layer_normalization.layer_normalization.LayerNormalization'> to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O0r6CivJbBUF"
      },
      "source": [
        "## Modèle \"à la bert\" issu de hugging face\n",
        "\n",
        "De nombreux [modèles sont disponibles sur HuggingFace](https://huggingface.co/models). \n",
        "\n",
        "On va tester un modèle plus léger que BERT à savoir 'distilbert-base-uncased'. \n",
        "\n",
        "[DistilBERT](https://arxiv.org/abs/1910.01108) is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT’s performances as measured on the GLUE language understanding benchmark. \n",
        "\n",
        "*Knowledge distillation* (sometimes also referred to as *teacher-student learning*) is a [compression technique in which a small model is trained to reproduce the behavior of a larger model](https://medium.com/huggingface/distilbert-8cf3380435b5) (or an ensemble of models).\n",
        "\n",
        "In the teacher-student training, we train a student network to mimic the full output distribution of the teacher network (its knowledge). Rather than training with a cross-entropy over the hard targets (one-hot encoding of the gold class), we transfer the knowledge from the teacher to the student with a cross-entropy over the soft targets (probabilities of the teacher)...\n",
        "\n",
        "Other techniques intend to do the same. These include quantization (approximating the weights of a network with a smaller precision) and weights pruning (removing some connections in the network). See the [blog post of Rasa on Compressing BERT for faster prediction](https://medium.com/huggingface/distilbert-8cf3380435b5).\n",
        "\n",
        "<!-- comment -->"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x_train = list(train_df['review'])\n",
        "y_train = list(train_df['positive'])\n",
        "\n",
        "x_val = list(val_df['review'])\n",
        "y_val = list(val_df['positive'])\n",
        "\n",
        "x_test = list(test_df['review'])\n",
        "y_test = list(test_df['positive'])\n",
        "\n",
        "## Do not consider the following\n",
        "\n",
        "## Convert a list of string into lists of int then into np.array.\n",
        "## This step should be skiped when running BERT\n",
        "\n",
        "#import numpy as np\n",
        "#y_train = np.array(list(map(int, y_train)))\n",
        "#y_val = np.array(list(map(int, y_val)))\n",
        "#y_test = np.array(list(map(int, y_test)))\n",
        "\n",
        "#print ('x_train', len(x_train), x_train[:10])\n",
        "#print ('y_train',  len(y_train), y_train[:10])\n",
        "#print ('x_val', len(x_val), x_val[:10])\n",
        "#print ('y_val',  len(y_val), y_val[:10])\n",
        "#print ('x_test', len(x_test))\n",
        "#print ('y_test',  len(y_test))"
      ],
      "metadata": {
        "id": "kDBwr4fvzOCU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "52DqPKJE9d4s"
      },
      "outputs": [],
      "source": [
        "import ktrain\n",
        "from ktrain import text\n",
        "MODEL_NAME = 'distilbert-base-uncased'\n",
        "#MODEL_NAME = 'albert-base-v2'\n",
        "#MODEL_NAME = 'camembert-base'\n",
        "\n",
        "CLASS_NAMES = [\"negative\", \"positive\"]\n",
        "\n",
        "distilbert_preproc = text.Transformer(MODEL_NAME, maxlen=500, class_names=CLASS_NAMES)\n",
        "train_preproc = distilbert_preproc.preprocess_train(x_train, y_train)\n",
        "val_preproc = distilbert_preproc.preprocess_test(x_val, y_val)\n",
        "#print (type(trn))\n",
        "#x_train, y_train = trn\n",
        "#x_test, y_test = val  \n",
        "distilbert_model = distilbert_preproc.get_classifier()\n",
        "# batch_size (int):              Batch size to use in training. default:32  \n",
        "#learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=6) # 128 dépend de la ram dispo 13 Go par défaut\n",
        "distilbert_learner = ktrain.get_learner(distilbert_model, train_data=train_preproc, val_data=val_preproc, batch_size=12)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#del distilbert_preproc\n",
        "#del train_preproc\n",
        "#del val_preproc\n",
        "#del distilbert_model\n",
        "#del distilbert_learner"
      ],
      "metadata": {
        "id": "VrmoU03yM6-G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EqM1-j84-4eD"
      },
      "outputs": [],
      "source": [
        "distilbert_learner.fit_onecycle(0.01, 1)\n",
        "#learner.fit_onecycle(8e-5, 4)\n",
        "#8e-5 = 8 + 10^(-5)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# données\n",
        "x_test = list(test_df['review'])   # reviews \n",
        "y_test = list(test_df['positive']) # labels (gold) \n",
        "\n",
        "# prédiction\n",
        "distilbert_predictor = ktrain.get_predictor(distilbert_learner.model, distilbert_preproc)\n",
        "y_hyp = [0 if h == 'negative' else 1 for h in distilbert_predictor.predict(x_test) ]\n",
        "\n",
        "# évaluation\n",
        "print(f1_score(y_test, y_hyp))\n",
        "# distilbert-base-uncased train sur 5 % 0.6577181208053691\n",
        "# distilbert-base-uncased train sur 100 % 0.6482833198161666\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LH253Lbk_yXi",
        "outputId": "65b35bcb-b002-476a-e5fe-60a312ec9538"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.6482833198161666\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "backup model"
      ],
      "metadata": {
        "id": "qw8UvziY5vmc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# save Predictor (i.e., model and Preprocessor instance) after partially training\n",
        "ktrain.get_predictor(distilbert_learner.model, preproc).save('distilbert_allocine.ckpt')\n",
        "\n",
        "#import pickle\n",
        "#with open('distilbert_allocine.model', 'wb') as model_file:\n",
        "#  pickle.dump(distilbert_learner, model_file)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H3mr7y5I5eLf",
        "outputId": "55ca1691-af49-45e9-a231-a33ca0e65a03"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Found untraced functions such as embeddings_layer_call_fn, embeddings_layer_call_and_return_conditional_losses, transformer_layer_call_fn, transformer_layer_call_and_return_conditional_losses, LayerNorm_layer_call_fn while saving (showing 5 of 164). These functions will not be directly callable after loading.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xeM-jNIyb_7M"
      },
      "source": [
        "#### QUESTION\n",
        "* Quelle performance obtenez-vous avec BERT et distilBERT pour quel temps d'entraînement ? Quelle performance comparativement aux 2 modèles précédents nbsvm et fasttext ?\n",
        "* Quelles risques prenez-vous à ne pas réaliser les mêmes prétraitements ou normalisations (voire utiliser des outils différents pour réaliser les mêmes prétraitements ou normalisations supposées) que ceux réalisés sur les corpus ayant servis à construire les modèles ? \n",
        "* En résumé, en mettant dans la balance les questions de performance, les questions de taille de modèles, de temps de \"fine-tuning\"... quelles conclusions faites-vous de l'usage des modèles simples vs les modèles plus complexes à la BERT ?\n",
        "\n",
        "Si le code tourne toujours passez à la question suivante pour gagner du temps..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qni8ELuBcBB1"
      },
      "source": [
        "#### VOTRE REPONSE\n",
        "\n",
        "**TODO**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VzOyw5WCCJX4"
      },
      "source": [
        "# Devoir à rendre \n",
        "[Theophile Blard rapporte des performances avec différents modèles (CamemBERT, RNN, TF-IDF + LogReg, CNN, fastText (unigrams)) sur le dataset Allociné](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert). \n",
        "\n",
        "\n",
        "| Model                                        | Validation Accuracy | Validation F1-Score | Test Accuracy | Test F1-Score |\n",
        "| :--------------------------------------------|--------------------:| -------------------:| -------------:|--------------:|\n",
        "| **[CamemBERT][bert.ipynb]**                  |           **97.39** |           **97.36** |     **97.44** |     **97.34** |\n",
        "| [RNN]                    |               94.39 |               94.34 |         94.58 |         94.39 |\n",
        "| [TF-IDF + LogReg]              |               94.35 |               94.29 |         94.38 |         94.19 |\n",
        "| [CNN]                    |               93.69 |               93.72 |         94.10 |         93.98 |\n",
        "| [fastText (unigrams)]    |               92.88 |               92.75 |         92.90 |         92.57 |\n",
        "\n",
        "\n",
        "Arriverez-vous à faire mieux ?\n",
        "\n",
        "Bien entendu vous avez le droit de choisir les modèles et leur configuration.\n",
        "\n",
        "Explorez les thèmes suivants\n",
        "- data augmentation\n",
        "- model ensembling \n",
        "\n",
        "Ecrivez un rapport d'expériences.\n",
        "\n",
        "## Topics\n",
        "\n",
        "### Data augmentation in NLP\n",
        "TextAttack is a Python framework for adversarial attacks, adversarial training, and [data augmentation in NLP](https://textattack.readthedocs.io/en/latest/2notebook/3_Augmentations.html).\n",
        "\n",
        "Here some hints for [multi-language attacks](https://textattack.readthedocs.io/en/latest/2notebook/Example_4_CamemBERT.html)\n",
        "\n",
        "### Model ensembling \n",
        "\n",
        "- Train/fine-tune multiple models from various architectures and/or dataset folds\n",
        "- Then ensemble them by (hard/soft) voting mechanism or something else...\n",
        "\n",
        "\n",
        "\n",
        "## VOTRE REPONSE\n",
        "\n",
        "**TODO**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dU6olWGuGzN6"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_tzxoRA_RGl4"
      },
      "source": [
        "# Références\n",
        "* Text Classification Example: Sentiment Analysis with IMDb Movie Reviews¶ https://nbviewer.org/github/amaiya/ktrain/blob/master/tutorials/tutorial-04-text-classification.ipynb\n",
        "* https://nbviewer.org/github/amaiya/ktrain/blob/master/examples/text/IMDb-BERT.ipynb\n",
        "* ktrain examples of Binary text Classification (Sentiment Analysis with IMDb Movie Reviews) with a nbsvm model (also a bit of bert)  and \n",
        "Multi-Label Text Classification (toxic comments) with fasttext https://nbviewer.org/github/amaiya/ktrain/blob/master/tutorials/tutorial-04-text-classification.ipynb\n",
        "* Text Classification with Hugging Face Transformers in ktrain https://github.com/amaiya/ktrain/blob/master/tutorials/tutorial-A3-hugging_face_transformers.ipynb\n",
        "* ktrain api documentation https://amaiya.github.io/ktrain/\n",
        "* Evaluation of various models (CamemBERT, RNN, TF-IDF + LogReg, CNN, fastText (unigrams)) on allocine dataset https://github.com/TheophileBlard/french-sentiment-analysis-with-bert\n",
        "* The Allociné dataset is a French-language dataset for sentiment analysis. The texts are movie reviews written between 2006 and 2020 by members of the Allociné.fr community for various films. It contains 100k positive and 100k negative reviews divided into train (160k), validation (20k), and test (20k). \n",
        "https://huggingface.co/datasets/allocine\n",
        "* Amazon reviews for three product categories: books, DVD, and music. Each sample contains a review text and the associated rating from 1 to 5 stars. Reviews rated above 3 is labeled as positive, and those rated less than 3 is labeled as negative. https://github.com/getalp/Flaubert/tree/master/flue\n",
        "* Twitter API between May and September 2018. The sentiment was generated thanks to AWS Comprehend API. For Spanish and French, tweets were first translated to English using Google Translate, and then analyzed with AWS Comprehend. https://github.com/charlesmalafosse/open-dataset-for-sentiment-analysis/\n",
        "* French dataset for sentiment analysis (Data translated from English to French).  A collection of over 1.5 Million tweets data translated to French, with their sentiment. The data has two columns, polarity and status https://github.com/gamebusterz/French-Sentiment-Analysis-Dataset\n",
        "* English Classification datasets https://ludwig.ai/latest/examples/text_classification/"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Notes personnelles pour améliorer ce sujet\n",
        "\n",
        "- introduire le sujet en présentant le travail de Théophile\n",
        "- reproduire la démarche d'analyse d'erreur de Théophile\n",
        "- permettre de mesurer les temps d'entraînement et de prédiction (cas 1 exemple vs plusieurs)\n",
        "- en raison du temps de fine-tuning, une approche possible serait plusieurs petits modèles sur des parties du corpus "
      ],
      "metadata": {
        "id": "vrRImAlHZCHp"
      }
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNFNd0KJMhzPgAM4c65E0T6",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "b6ca4143486641a3b73d59637328e2b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d2a818e98f3a48468dff2e729e626153",
              "IPY_MODEL_123fbb1c677e459a9c396eba6c10209a",
              "IPY_MODEL_e4ce991697f04cdd9307aa1f1dcac9de"
            ],
            "layout": "IPY_MODEL_3aa79446cd4344569c887c5dfc920845"
          }
        },
        "d2a818e98f3a48468dff2e729e626153": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8a3f5f76ed184afab5373acdf83092d0",
            "placeholder": "​",
            "style": "IPY_MODEL_44212d9979634303a1f9fcf8ca48e4f7",
            "value": "Downloading builder script: 100%"
          }
        },
        "123fbb1c677e459a9c396eba6c10209a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c2ef8b4c907d49f5a84186314e956df4",
            "max": 3681,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_844304d7c3d6458296188963a20c497c",
            "value": 3681
          }
        },
        "e4ce991697f04cdd9307aa1f1dcac9de": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0540ef45da394a24bf9b8a56fb770aa4",
            "placeholder": "​",
            "style": "IPY_MODEL_290665a7ce8244a6addfe5ac36dac895",
            "value": " 3.68k/3.68k [00:00&lt;00:00, 76.9kB/s]"
          }
        },
        "3aa79446cd4344569c887c5dfc920845": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8a3f5f76ed184afab5373acdf83092d0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "44212d9979634303a1f9fcf8ca48e4f7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c2ef8b4c907d49f5a84186314e956df4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "844304d7c3d6458296188963a20c497c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0540ef45da394a24bf9b8a56fb770aa4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "290665a7ce8244a6addfe5ac36dac895": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5bd35b298e854936a8b7675286a5ed6b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6a1814b3a6ca4c02832e656cddf6ab19",
              "IPY_MODEL_436eaf6fd1bc4fbeaba7b9d69deff439",
              "IPY_MODEL_6f7122f4375c4886bd4c9f14cf0e7893"
            ],
            "layout": "IPY_MODEL_70107a988d424ad5a38896465eedf7a9"
          }
        },
        "6a1814b3a6ca4c02832e656cddf6ab19": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6101cd2353ce4c04b4b3496950b3ae08",
            "placeholder": "​",
            "style": "IPY_MODEL_015aeb00d34a4cd5b7b9eda66e0d69c5",
            "value": "Downloading metadata: 100%"
          }
        },
        "436eaf6fd1bc4fbeaba7b9d69deff439": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0b25ef3cf844416e8d0e12a2c63fb421",
            "max": 1960,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cf26db770b1d47fc9e40175b52599d2d",
            "value": 1960
          }
        },
        "6f7122f4375c4886bd4c9f14cf0e7893": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7936a14f5d7846aaa481f036878a919e",
            "placeholder": "​",
            "style": "IPY_MODEL_a93ae8a8f86548548e7cd3f5825eafc6",
            "value": " 1.96k/1.96k [00:00&lt;00:00, 66.7kB/s]"
          }
        },
        "70107a988d424ad5a38896465eedf7a9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6101cd2353ce4c04b4b3496950b3ae08": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "015aeb00d34a4cd5b7b9eda66e0d69c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0b25ef3cf844416e8d0e12a2c63fb421": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cf26db770b1d47fc9e40175b52599d2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7936a14f5d7846aaa481f036878a919e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a93ae8a8f86548548e7cd3f5825eafc6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "79e9f189c0594597a596b9cea6575f94": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f5968cc4c37043a7bbb2827d7a43c345",
              "IPY_MODEL_75c85e7bc6d1465482348c73f4f362c8",
              "IPY_MODEL_88060cd5453149b1b09c583d1de6657d"
            ],
            "layout": "IPY_MODEL_bb3b688410204dff82a72be1e34fd856"
          }
        },
        "f5968cc4c37043a7bbb2827d7a43c345": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_292c1458931c44e897a665def8b91c87",
            "placeholder": "​",
            "style": "IPY_MODEL_d53737423a9c4f399deb22a1ed9d4c31",
            "value": "Downloading readme: 100%"
          }
        },
        "75c85e7bc6d1465482348c73f4f362c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_39e59fa8639c4a5b9a152ffa5e977662",
            "max": 9107,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0f710f3aa1f9433a8dedbc9cf8a4beff",
            "value": 9107
          }
        },
        "88060cd5453149b1b09c583d1de6657d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a52dea77aa9e49f7979a85779d701775",
            "placeholder": "​",
            "style": "IPY_MODEL_effd4418679f4fc99daccb1b90a0bdd3",
            "value": " 9.11k/9.11k [00:00&lt;00:00, 90.9kB/s]"
          }
        },
        "bb3b688410204dff82a72be1e34fd856": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "292c1458931c44e897a665def8b91c87": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d53737423a9c4f399deb22a1ed9d4c31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "39e59fa8639c4a5b9a152ffa5e977662": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0f710f3aa1f9433a8dedbc9cf8a4beff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a52dea77aa9e49f7979a85779d701775": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "effd4418679f4fc99daccb1b90a0bdd3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c210be85697848139222efc91b0a2622": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f39d47def2bb4348bf908b1300a3acf9",
              "IPY_MODEL_a1f7c70f7a474c1495f3c64d6019597a",
              "IPY_MODEL_0851f5d8339441cf8ebfe2d288585bd1"
            ],
            "layout": "IPY_MODEL_3a0b31df2d8f4bb8b826a6a11bd18539"
          }
        },
        "f39d47def2bb4348bf908b1300a3acf9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bb01ccab0b064e1bb15af4569244f15b",
            "placeholder": "​",
            "style": "IPY_MODEL_17f828ffbc6c41ce9be941cdbf81e8f5",
            "value": "Downloading data: 100%"
          }
        },
        "a1f7c70f7a474c1495f3c64d6019597a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9ae6808d5b874956bb9daf98c881035e",
            "max": 66625305,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_71ca06111d4947b2bd2a228d8e7c3ff5",
            "value": 66625305
          }
        },
        "0851f5d8339441cf8ebfe2d288585bd1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fcb57b52ff6746a2bcb6003038b339d1",
            "placeholder": "​",
            "style": "IPY_MODEL_cbb5ed28efaa482a8f8c35210ea4ebb0",
            "value": " 66.6M/66.6M [00:01&lt;00:00, 59.9MB/s]"
          }
        },
        "3a0b31df2d8f4bb8b826a6a11bd18539": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bb01ccab0b064e1bb15af4569244f15b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "17f828ffbc6c41ce9be941cdbf81e8f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9ae6808d5b874956bb9daf98c881035e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "71ca06111d4947b2bd2a228d8e7c3ff5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fcb57b52ff6746a2bcb6003038b339d1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cbb5ed28efaa482a8f8c35210ea4ebb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7c32212ee16645ddb59cf8cf450af3a8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4b3717e325d8417589c85225b14b1006",
              "IPY_MODEL_497dcfc6e70348e58993c98e8d4db8ce",
              "IPY_MODEL_bd7fbf8b063249b9b5952c9a597af85d"
            ],
            "layout": "IPY_MODEL_d411b3a216784c21befb0e65481fcc60"
          }
        },
        "4b3717e325d8417589c85225b14b1006": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_96166f4e8a95406b8541f1c1cb8578ba",
            "placeholder": "​",
            "style": "IPY_MODEL_2429af983c9e4d338ed07c76e75cd853",
            "value": "Generating train split: 100%"
          }
        },
        "497dcfc6e70348e58993c98e8d4db8ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bcb9a4e292874a5d910a61195cce7716",
            "max": 160000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0e1048dbc4af46299504f39cc2cb10f3",
            "value": 160000
          }
        },
        "bd7fbf8b063249b9b5952c9a597af85d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a54030d56e944d0787ab2d2798026ee1",
            "placeholder": "​",
            "style": "IPY_MODEL_0d1a7c10f4574ad89386805051ec500d",
            "value": " 159391/160000 [00:19&lt;00:00, 12584.76 examples/s]"
          }
        },
        "d411b3a216784c21befb0e65481fcc60": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "96166f4e8a95406b8541f1c1cb8578ba": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2429af983c9e4d338ed07c76e75cd853": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bcb9a4e292874a5d910a61195cce7716": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0e1048dbc4af46299504f39cc2cb10f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a54030d56e944d0787ab2d2798026ee1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0d1a7c10f4574ad89386805051ec500d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8d8821eb9d0845018531734443223ee5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3b2d70a789e04e639e37ab4c8eb0db0b",
              "IPY_MODEL_d06bcab78058463c82e248d51fdccd8f",
              "IPY_MODEL_d8752ffde7f54643aab15b0bf075340a"
            ],
            "layout": "IPY_MODEL_21c7af9682ab463ebbdeeb916d3a6794"
          }
        },
        "3b2d70a789e04e639e37ab4c8eb0db0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c518a3ba017a4340a1a3cf0919a37b04",
            "placeholder": "​",
            "style": "IPY_MODEL_3795e5db3dc6443da11eb17b108e964d",
            "value": "Generating validation split: 100%"
          }
        },
        "d06bcab78058463c82e248d51fdccd8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0c5331686b124aa79ca06f1f0e8da06e",
            "max": 20000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_863154ee114b442f94c36ab147aad9b6",
            "value": 20000
          }
        },
        "d8752ffde7f54643aab15b0bf075340a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a9be5759ab6e48f6b438ef3b17e142c1",
            "placeholder": "​",
            "style": "IPY_MODEL_07d7ced7d8d74fb28b1487ccc9dc361a",
            "value": " 20000/20000 [00:14&lt;00:00, 10112.10 examples/s]"
          }
        },
        "21c7af9682ab463ebbdeeb916d3a6794": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "c518a3ba017a4340a1a3cf0919a37b04": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3795e5db3dc6443da11eb17b108e964d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0c5331686b124aa79ca06f1f0e8da06e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "863154ee114b442f94c36ab147aad9b6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a9be5759ab6e48f6b438ef3b17e142c1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "07d7ced7d8d74fb28b1487ccc9dc361a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b052f51fc34649d69749824db4eb36e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_51db043b08f9434b9eb30785d82ab43f",
              "IPY_MODEL_ce25b9d87d2f4afe9b57351daf33b336",
              "IPY_MODEL_7c56762e7e8644aeb0942f208aa44df3"
            ],
            "layout": "IPY_MODEL_46eb11ce593642178655e9b7f441ae82"
          }
        },
        "51db043b08f9434b9eb30785d82ab43f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c2d99cbc83424208b13583cf42b24d72",
            "placeholder": "​",
            "style": "IPY_MODEL_0e99590640e146c4a6886507ea6317a0",
            "value": "Generating test split:  99%"
          }
        },
        "ce25b9d87d2f4afe9b57351daf33b336": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f94f480c488a4eb894395c840d095b73",
            "max": 20000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6245a9408a9642c2b6875a6603218eb7",
            "value": 20000
          }
        },
        "7c56762e7e8644aeb0942f208aa44df3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f7ede946fea7438fbf6e4342422aeaed",
            "placeholder": "​",
            "style": "IPY_MODEL_86c84acfab1b4b63ba3bb6ef5a9ec8d9",
            "value": " 19747/20000 [00:08&lt;00:00, 11371.86 examples/s]"
          }
        },
        "46eb11ce593642178655e9b7f441ae82": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "c2d99cbc83424208b13583cf42b24d72": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0e99590640e146c4a6886507ea6317a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f94f480c488a4eb894395c840d095b73": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6245a9408a9642c2b6875a6603218eb7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f7ede946fea7438fbf6e4342422aeaed": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "86c84acfab1b4b63ba3bb6ef5a9ec8d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3104db207d224ee2b54972cf5898e774": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3473304e0c5f43b186b77a612a34764c",
              "IPY_MODEL_4843f6b720a04f43b7c55671586c9592",
              "IPY_MODEL_c9949e2640de4b2a9b105a7e401a95cb"
            ],
            "layout": "IPY_MODEL_4ade5b9af208473eb9d2d7a5a20d8eee"
          }
        },
        "3473304e0c5f43b186b77a612a34764c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7208dbb9a99743c798ec59ba806a8956",
            "placeholder": "​",
            "style": "IPY_MODEL_a81d69a20b314735a87865c2ccdca696",
            "value": "100%"
          }
        },
        "4843f6b720a04f43b7c55671586c9592": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_975ef7a4f9084b2b962bcd240198f7ea",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_612418dd948d418a8d00726269207c09",
            "value": 3
          }
        },
        "c9949e2640de4b2a9b105a7e401a95cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_150d263f96f740e19063ab6eb9b80c9b",
            "placeholder": "​",
            "style": "IPY_MODEL_a094005d268346de9550a32b87dda977",
            "value": " 3/3 [00:00&lt;00:00, 56.76it/s]"
          }
        },
        "4ade5b9af208473eb9d2d7a5a20d8eee": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7208dbb9a99743c798ec59ba806a8956": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a81d69a20b314735a87865c2ccdca696": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "975ef7a4f9084b2b962bcd240198f7ea": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "612418dd948d418a8d00726269207c09": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "150d263f96f740e19063ab6eb9b80c9b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a094005d268346de9550a32b87dda977": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}