google-ai-assistant-for-data-science-teaching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e2e356ae",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "papermill": {
     "duration": 0.009935,
     "end_time": "2024-04-07T11:30:42.742370",
     "exception": false,
     "start_time": "2024-04-07T11:30:42.732435",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center><h1>Google AI Assistant for Data Science teaching</h1></center>\n",
    "\n",
    "<center><img src=\"https://res.infoq.com/news/2024/02/google-gemma-open-model/en/headerimage/generatedHeaderImage-1708977571481.jpg\" width=\"400\"></center>\n",
    "\n",
    "# Introduction\n",
    "\n",
    "This Notebook will build an AI Assistant for Data Science teaching.\n",
    "\n",
    "## The method\n",
    "\n",
    "We will fine-tune a LLM with Data Science questions and answers. Then we will create a custom class that will query this model.\n",
    "\n",
    "## The model\n",
    "\n",
    "As model, we will use Gemma model, fine-tuned with our Data Science Q&A data, using LoRA.\n",
    "\n",
    "## The data\n",
    "\n",
    "As data for fine-tuning Gemma, we will use [Data Science Q&A Treasury](https://www.kaggle.com/datasets/memocan/data-science-interview-q-and-a-treasury) dataset. This dataset contains over 150 questions and answering about Data Science.\n",
    "\n",
    "## Previous work\n",
    "\n",
    "This work is largely based on previous work. Here I list the sources:\n",
    "\n",
    "1. Gemma Model Card, Kaggle Models, https://www.kaggle.com/models/google/gemma\n",
    "2. Kaggle QA with Gemma - KerasNLP Starter, Kaggle Code, https://www.kaggle.com/code/awsaf49/kaggle-qa-with-gemma-kerasnlp-starter (Version 11)\n",
    "3. Fine-tune Gemma models in Keras using LoRA, Kaggle Code, https://www.kaggle.com/code/nilaychauhan/fine-tune-gemma-models-in-keras-using-lora (Version 1)\n",
    "4. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, LoRA: Low-Rank Adaptation of Large Language Models, ArXiv, https://arxiv.org/pdf/2106.09685.pdf\n",
    "5. Abheesht Sharma, Matthew Watson, Parameter-efficient fine-tuning of GPT-2 with LoRA, https://keras.io/examples/nlp/parameter_efficient_finetuning_of_gpt2_with_lora/\n",
    "6. Keras 3 API documentation / KerasNLP / Models / Gemma, https://keras.io/api/keras_nlp/models/gemma/   \n",
    "7. Understanding LoRA with a minimal example, https://blogs.rstudio.com/ai/posts/2023-06-22-understanding-lora/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b94e1936",
   "metadata": {
    "papermill": {
     "duration": 0.009417,
     "end_time": "2024-04-07T11:30:42.761519",
     "exception": false,
     "start_time": "2024-04-07T11:30:42.752102",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction about Gemma\n",
    "\n",
    "\n",
    "Gemma is a collection of lightweight source generative AI models designed to be used mostly by developers and researchers. Created by Google DeepMind research lab that also developed Gemini, Gemma is available in several versions, with 2B and 7B parameters, as following:\n",
    "\n",
    "| Model                  | Parameters      | Tuned versions    | Description                                    | Recomemnded target platforms       |\n",
    "|------------------------|-----------------|-------------------|------------------------------------------------|------------------------------------|\n",
    "| `gemma_2b_en`          | 2.51B           | Pretrained        | 18-layer Gemma model (Gemma with 2B parameters)|Mobile devices and laptops          |\n",
    "| `gemma_instruct_2b_en` | 2.51B           | Instruction tuned | 18-layer Gemma model (Gemma with 2B parameters)| Mobile devices and laptops         | \n",
    "| `gemma_7b_en`          | 8.54B           | Pretrained        | 28-layer Gemma model (Gemma with 7B parameters)| Desktop computers and small servers|\n",
    "| `gemma_instruct_7b_en` | 8.54B           | Instruction tuned | 28-layer Gemma model (Gemma with 7B parameters)| Desktop computers and small servers|\n",
    "\n",
    "\n",
    "For this notebook, we will fine-tune `gemma_2b_en` model, one of the `2B` parameters Gemma models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dca646f",
   "metadata": {
    "papermill": {
     "duration": 0.009192,
     "end_time": "2024-04-07T11:30:42.780121",
     "exception": false,
     "start_time": "2024-04-07T11:30:42.770929",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# LoRA introduction\n",
    "\n",
    "LoRA stands for Low-Rank Adaptation. It is a method used to fine-tune large language models (LLMs) by freezing the weights of the LLM and injecting trainable rank-decomposition matrices.   \n",
    "The number of trainable parameters during fine-tunning will decrease therefore considerably.   \n",
    "According to LoRA paper, this number decreases 10,000 times, and the computational resources size decreases 3 times. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be81f676",
   "metadata": {
    "papermill": {
     "duration": 0.00913,
     "end_time": "2024-04-07T11:30:42.798640",
     "exception": false,
     "start_time": "2024-04-07T11:30:42.789510",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Installations and configurations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c6b7f517",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:30:42.818742Z",
     "iopub.status.busy": "2024-04-07T11:30:42.818430Z",
     "iopub.status.idle": "2024-04-07T11:31:14.901789Z",
     "shell.execute_reply": "2024-04-07T11:31:14.900758Z"
    },
    "papermill": {
     "duration": 32.096182,
     "end_time": "2024-04-07T11:31:14.904257",
     "exception": false,
     "start_time": "2024-04-07T11:30:42.808075",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n",
      "tensorflow-decision-forests 1.8.1 requires wurlitzer, which is not installed.\u001b[0m\u001b[31m\r\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n",
      "tensorflow-decision-forests 1.8.1 requires wurlitzer, which is not installed.\r\n",
      "tensorflow 2.15.0 requires keras<2.16,>=2.15.0, but you have keras 3.1.1 which is incompatible.\u001b[0m\u001b[31m\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n",
    "!pip install -q -U keras-nlp\n",
    "!pip install -q -U keras>=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "36e98d84",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:14.927175Z",
     "iopub.status.busy": "2024-04-07T11:31:14.926833Z",
     "iopub.status.idle": "2024-04-07T11:31:29.260641Z",
     "shell.execute_reply": "2024-04-07T11:31:29.259864Z"
    },
    "papermill": {
     "duration": 14.347993,
     "end_time": "2024-04-07T11:31:29.263091",
     "exception": false,
     "start_time": "2024-04-07T11:31:14.915098",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-04-07 11:31:19.502511: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-04-07 11:31:19.502615: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-04-07 11:31:19.647882: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KERAS_BACKEND\"] = \"jax\" # you can also use tensorflow or torch\n",
    "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\" # avoid memory fragmentation on JAX backend.\n",
    "\n",
    "import keras\n",
    "import keras_nlp\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm\n",
    "from time import time\n",
    "tqdm.pandas() # progress bar for pandas\n",
    "\n",
    "from IPython.display import display, Markdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8d4795cf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.285176Z",
     "iopub.status.busy": "2024-04-07T11:31:29.284160Z",
     "iopub.status.idle": "2024-04-07T11:31:29.289265Z",
     "shell.execute_reply": "2024-04-07T11:31:29.288370Z"
    },
    "papermill": {
     "duration": 0.017934,
     "end_time": "2024-04-07T11:31:29.291209",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.273275",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Config:\n",
    "    seed = 42\n",
    "    dataset_path = \"/kaggle/input/data-science-interview-q-and-a-treasury/dataset.csv\"\n",
    "    preset = \"gemma_2b_en\" # name of pretrained Gemma\n",
    "    sequence_length = 512 # max size of input sequence for training\n",
    "    batch_size = 1 # size of the input batch in training\n",
    "    epochs = 10 # number of epochs to train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b5aa6daa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.312703Z",
     "iopub.status.busy": "2024-04-07T11:31:29.312423Z",
     "iopub.status.idle": "2024-04-07T11:31:29.317037Z",
     "shell.execute_reply": "2024-04-07T11:31:29.316152Z"
    },
    "papermill": {
     "duration": 0.01761,
     "end_time": "2024-04-07T11:31:29.319014",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.301404",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "keras.utils.set_random_seed(Config.seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee71f18",
   "metadata": {
    "papermill": {
     "duration": 0.00983,
     "end_time": "2024-04-07T11:31:29.338741",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.328911",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's define as well a small utility function for the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "16298d9e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.359930Z",
     "iopub.status.busy": "2024-04-07T11:31:29.359272Z",
     "iopub.status.idle": "2024-04-07T11:31:29.364328Z",
     "shell.execute_reply": "2024-04-07T11:31:29.363515Z"
    },
    "papermill": {
     "duration": 0.017703,
     "end_time": "2024-04-07T11:31:29.366185",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.348482",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def colorize_text(text):\n",
    "    for word, color in zip([\"Question\", \"Answer\", \"Total time\"], [\"blue\", \"red\", \"green\"]):\n",
    "        text = text.replace(f\"\\n\\n{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22267036",
   "metadata": {
    "papermill": {
     "duration": 0.009454,
     "end_time": "2024-04-07T11:31:29.385503",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.376049",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load the data  \n",
    "\n",
    "We load the data we will use for fine-tunning with LoRA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "65163b72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.407106Z",
     "iopub.status.busy": "2024-04-07T11:31:29.406306Z",
     "iopub.status.idle": "2024-04-07T11:31:29.439022Z",
     "shell.execute_reply": "2024-04-07T11:31:29.438094Z"
    },
    "papermill": {
     "duration": 0.046067,
     "end_time": "2024-04-07T11:31:29.441346",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.395279",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is supervised machine learning? 👶</td>\n",
       "      <td>Supervised learning is a type of machine learn...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is regression? Which models can you use t...</td>\n",
       "      <td>Regression is a part of supervised ML. Regress...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is linear regression? When do we use it? 👶</td>\n",
       "      <td>Linear regression is a model that assumes a li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What are the main assumptions of linear regres...</td>\n",
       "      <td>There are several assumptions of linear regres...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What’s the normal distribution? Why do we care...</td>\n",
       "      <td>The normal distribution is a continuous probab...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0             What is supervised machine learning? 👶   \n",
       "1  What is regression? Which models can you use t...   \n",
       "2    What is linear regression? When do we use it? 👶   \n",
       "3  What are the main assumptions of linear regres...   \n",
       "4  What’s the normal distribution? Why do we care...   \n",
       "\n",
       "                                              answer  \n",
       "0  Supervised learning is a type of machine learn...  \n",
       "1  Regression is a part of supervised ML. Regress...  \n",
       "2  Linear regression is a model that assumes a li...  \n",
       "3  There are several assumptions of linear regres...  \n",
       "4  The normal distribution is a continuous probab...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(f\"{Config.dataset_path}\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c47c56f6",
   "metadata": {
    "papermill": {
     "duration": 0.009854,
     "end_time": "2024-04-07T11:31:29.462651",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.452797",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Fine-tune Gemma\n",
    "\n",
    "We prepare a template and generate a prompt using this template from each row in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "703f9665",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.484972Z",
     "iopub.status.busy": "2024-04-07T11:31:29.484185Z",
     "iopub.status.idle": "2024-04-07T11:31:29.515877Z",
     "shell.execute_reply": "2024-04-07T11:31:29.515015Z"
    },
    "papermill": {
     "duration": 0.047487,
     "end_time": "2024-04-07T11:31:29.520452",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.472965",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b356d0b8397447bb6178833e331ca31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/166 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "template = \"\\n\\nQuestion:\\n{question}\\n\\nAnswer:\\n{answer}\"\n",
    "df[\"prompt\"] = df.progress_apply(lambda row: template.format(question=row.question,\n",
    "                                                             answer=row.answer), axis=1)\n",
    "data = df.prompt.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09c734aa",
   "metadata": {
    "papermill": {
     "duration": 0.010473,
     "end_time": "2024-04-07T11:31:29.541336",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.530863",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Initialize the code for Gemma Causal LM\n",
    "\n",
    "We initialize `GemmaCausalML` with `gemma_2b_en` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1cfb08f1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:31:29.563578Z",
     "iopub.status.busy": "2024-04-07T11:31:29.563257Z",
     "iopub.status.idle": "2024-04-07T11:32:27.484780Z",
     "shell.execute_reply": "2024-04-07T11:32:27.483826Z"
    },
    "papermill": {
     "duration": 57.934869,
     "end_time": "2024-04-07T11:32:27.486623",
     "exception": false,
     "start_time": "2024-04-07T11:31:29.551754",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Attaching 'config.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'config.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'model.weights.h5' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'tokenizer.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'assets/tokenizer/vocabulary.spm' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">524,288,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,506,172,416\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m524,288,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_2b_en\")\n",
    "gemma_lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccb83210",
   "metadata": {
    "papermill": {
     "duration": 0.011714,
     "end_time": "2024-04-07T11:32:27.510270",
     "exception": false,
     "start_time": "2024-04-07T11:32:27.498556",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Gemma preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d63dd07b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:32:27.535596Z",
     "iopub.status.busy": "2024-04-07T11:32:27.535194Z",
     "iopub.status.idle": "2024-04-07T11:32:27.869977Z",
     "shell.execute_reply": "2024-04-07T11:32:27.868893Z"
    },
    "papermill": {
     "duration": 0.350276,
     "end_time": "2024-04-07T11:32:27.872312",
     "exception": false,
     "start_time": "2024-04-07T11:32:27.522036",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "x, y, sample_weight = gemma_lm.preprocessor(data[0:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38459d87",
   "metadata": {
    "papermill": {
     "duration": 0.011534,
     "end_time": "2024-04-07T11:32:27.896062",
     "exception": false,
     "start_time": "2024-04-07T11:32:27.884528",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Enable LoRA for the model\n",
    "\n",
    "We set LoRA rank to 5. The higher the LoRA rank, the higher the number of trainable parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8293dd09",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:32:27.921246Z",
     "iopub.status.busy": "2024-04-07T11:32:27.920562Z",
     "iopub.status.idle": "2024-04-07T11:32:28.390460Z",
     "shell.execute_reply": "2024-04-07T11:32:28.389579Z"
    },
    "papermill": {
     "duration": 0.484695,
     "end_time": "2024-04-07T11:32:28.392380",
     "exception": false,
     "start_time": "2024-04-07T11:32:27.907685",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,507,877,376</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">524,288,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,507,877,376\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m524,288,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,507,877,376</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,507,877,376\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,704,960</span> (6.50 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m1,704,960\u001b[0m (6.50 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Enable LoRA for the model and set the LoRA rank to 5.\n",
    "gemma_lm.backbone.enable_lora(rank=5)\n",
    "gemma_lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0278d36",
   "metadata": {
    "papermill": {
     "duration": 0.013053,
     "end_time": "2024-04-07T11:32:28.419159",
     "exception": false,
     "start_time": "2024-04-07T11:32:28.406106",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The total trainable parameters is 1.7 M (or 6.5 MB).  \n",
    "This is less than 0.06% of the 2,5G (9.35 GB) total trainable parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8234d7a3",
   "metadata": {
    "papermill": {
     "duration": 0.012852,
     "end_time": "2024-04-07T11:32:28.445024",
     "exception": false,
     "start_time": "2024-04-07T11:32:28.432172",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Run the fine-tuning sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5b071c24",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:32:28.472935Z",
     "iopub.status.busy": "2024-04-07T11:32:28.472581Z",
     "iopub.status.idle": "2024-04-07T11:53:03.202955Z",
     "shell.execute_reply": "2024-04-07T11:53:03.201993Z"
    },
    "papermill": {
     "duration": 1234.746989,
     "end_time": "2024-04-07T11:53:03.205038",
     "exception": false,
     "start_time": "2024-04-07T11:32:28.458049",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m143s\u001b[0m 731ms/step - loss: 0.5137 - sparse_categorical_accuracy: 0.5402\n",
      "Epoch 2/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 729ms/step - loss: 0.4342 - sparse_categorical_accuracy: 0.5783\n",
      "Epoch 3/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.4123 - sparse_categorical_accuracy: 0.5927\n",
      "Epoch 4/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 729ms/step - loss: 0.3994 - sparse_categorical_accuracy: 0.6016\n",
      "Epoch 5/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.3898 - sparse_categorical_accuracy: 0.6081\n",
      "Epoch 6/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.3777 - sparse_categorical_accuracy: 0.6162\n",
      "Epoch 7/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.3622 - sparse_categorical_accuracy: 0.6280\n",
      "Epoch 8/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 729ms/step - loss: 0.3435 - sparse_categorical_accuracy: 0.6435\n",
      "Epoch 9/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.3208 - sparse_categorical_accuracy: 0.6627\n",
      "Epoch 10/10\n",
      "\u001b[1m166/166\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m121s\u001b[0m 728ms/step - loss: 0.2951 - sparse_categorical_accuracy: 0.6857\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7b719c77d900>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Limit the input sequence length to 512 (to control memory usage).\n",
    "gemma_lm.preprocessor.sequence_length = Config.sequence_length \n",
    "\n",
    "# Compile the model with loss, optimizer, and metric\n",
    "gemma_lm.compile(\n",
    "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=8e-5),\n",
    "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
    ")\n",
    "\n",
    "# Train model\n",
    "gemma_lm.fit(data, epochs=Config.epochs, batch_size=Config.batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd6d6d1",
   "metadata": {
    "papermill": {
     "duration": 0.153473,
     "end_time": "2024-04-07T11:53:03.511041",
     "exception": false,
     "start_time": "2024-04-07T11:53:03.357568",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Test the fine-tuned model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe53464",
   "metadata": {
    "papermill": {
     "duration": 0.149099,
     "end_time": "2024-04-07T11:53:03.855408",
     "exception": false,
     "start_time": "2024-04-07T11:53:03.706309",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Define the specialized class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "823e058e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:04.154555Z",
     "iopub.status.busy": "2024-04-07T11:53:04.154215Z",
     "iopub.status.idle": "2024-04-07T11:53:04.160898Z",
     "shell.execute_reply": "2024-04-07T11:53:04.160015Z"
    },
    "papermill": {
     "duration": 0.158239,
     "end_time": "2024-04-07T11:53:04.162710",
     "exception": false,
     "start_time": "2024-04-07T11:53:04.004471",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class GemmaQA:\n",
    "    def __init__(self, max_length=512):\n",
    "        self.max_length = max_length\n",
    "        self.prompt = \"\\n\\nQuestion:\\n{question}\\n\\nAnswer:\\n{answer}\"\n",
    "        self.gemma_lm = gemma_lm\n",
    "        \n",
    "    def query(self, question):\n",
    "        t_start = time()\n",
    "        response = self.gemma_lm.generate(\n",
    "            self.prompt.format(\n",
    "                question=question,\n",
    "                answer=\"\"), \n",
    "            max_length=self.max_length)\n",
    "        t_end = time()\n",
    "        display(Markdown(colorize_text(f\"{response}\\n\\nTotal time: {round(t_end - t_start,2)} sec.\")))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4b78bace",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:04.467128Z",
     "iopub.status.busy": "2024-04-07T11:53:04.466261Z",
     "iopub.status.idle": "2024-04-07T11:53:04.470704Z",
     "shell.execute_reply": "2024-04-07T11:53:04.469862Z"
    },
    "papermill": {
     "duration": 0.160231,
     "end_time": "2024-04-07T11:53:04.472595",
     "exception": false,
     "start_time": "2024-04-07T11:53:04.312364",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "gemma_qa = GemmaQA()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de045232",
   "metadata": {
    "papermill": {
     "duration": 0.147879,
     "end_time": "2024-04-07T11:53:04.768950",
     "exception": false,
     "start_time": "2024-04-07T11:53:04.621071",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Test 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1ddf7e34",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:05.069150Z",
     "iopub.status.busy": "2024-04-07T11:53:05.068272Z",
     "iopub.status.idle": "2024-04-07T11:53:18.544034Z",
     "shell.execute_reply": "2024-04-07T11:53:18.543067Z"
    },
    "papermill": {
     "duration": 13.629699,
     "end_time": "2024-04-07T11:53:18.546197",
     "exception": false,
     "start_time": "2024-04-07T11:53:04.916498",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "What is linear regression? When do we use it? 👶\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Linear regression is a statistical method for predicting a single value for a new input, by solving a linear equation with known (or parameterized) constants, or feature values, or regression coefficients.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 13.47 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "row = df.iloc[2]\n",
    "gemma_qa.query(row.question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf32aaa",
   "metadata": {
    "papermill": {
     "duration": 0.153681,
     "end_time": "2024-04-07T11:53:18.852624",
     "exception": false,
     "start_time": "2024-04-07T11:53:18.698943",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Test 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "800ce376",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:19.159156Z",
     "iopub.status.busy": "2024-04-07T11:53:19.158387Z",
     "iopub.status.idle": "2024-04-07T11:53:22.120575Z",
     "shell.execute_reply": "2024-04-07T11:53:22.119582Z"
    },
    "papermill": {
     "duration": 3.116335,
     "end_time": "2024-04-07T11:53:22.122635",
     "exception": false,
     "start_time": "2024-04-07T11:53:19.006300",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "What is SGD  —  stochastic gradient descent? What’s the difference with the usual gradient descent? ‍⭐️\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Stochastic gradient descent is a variant of gradient descent where the gradient is calculated using only a random sample from the training set. This means that the update step is only based on a random sample from the training set. The usual gradient descent uses the whole training set to calculate the gradient. This means that the update step is based on all the training examples.\n",
       "\n",
       "The difference between SGD and gradient descent is that SGD is faster to compute, since it only requires one calculation of the gradient per iteration.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 2.96 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "row = df.iloc[10]\n",
    "gemma_qa.query(row.question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7169b53c",
   "metadata": {
    "papermill": {
     "duration": 0.149654,
     "end_time": "2024-04-07T11:53:22.423327",
     "exception": false,
     "start_time": "2024-04-07T11:53:22.273673",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Test 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fd191ab5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:22.726047Z",
     "iopub.status.busy": "2024-04-07T11:53:22.725591Z",
     "iopub.status.idle": "2024-04-07T11:53:23.085512Z",
     "shell.execute_reply": "2024-04-07T11:53:23.084589Z"
    },
    "papermill": {
     "duration": 0.512754,
     "end_time": "2024-04-07T11:53:23.087496",
     "exception": false,
     "start_time": "2024-04-07T11:53:22.574742",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "How to validate your models? 👶\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Answer here\n",
       "\n",
       "**<font color='green'>Total time:</font>** 0.35 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "row = df.iloc[15]\n",
    "gemma_qa.query(row.question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8c7c1c",
   "metadata": {
    "papermill": {
     "duration": 0.151509,
     "end_time": "2024-04-07T11:53:23.388726",
     "exception": false,
     "start_time": "2024-04-07T11:53:23.237217",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fresh question 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8b034f19",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:23.700418Z",
     "iopub.status.busy": "2024-04-07T11:53:23.699649Z",
     "iopub.status.idle": "2024-04-07T11:53:24.711228Z",
     "shell.execute_reply": "2024-04-07T11:53:24.710151Z"
    },
    "papermill": {
     "duration": 1.164677,
     "end_time": "2024-04-07T11:53:24.713249",
     "exception": false,
     "start_time": "2024-04-07T11:53:23.548572",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "What is regularization?\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Regularization is a technique for preventing your neural networks from overfitting. It does this by adding some penalty to your cost function.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 1.01 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question = \"What is regularization?\"\n",
    "gemma_qa.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84df5cab",
   "metadata": {
    "papermill": {
     "duration": 0.148685,
     "end_time": "2024-04-07T11:53:25.018087",
     "exception": false,
     "start_time": "2024-04-07T11:53:24.869402",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fresh question 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d623adce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:25.317680Z",
     "iopub.status.busy": "2024-04-07T11:53:25.316946Z",
     "iopub.status.idle": "2024-04-07T11:53:26.706727Z",
     "shell.execute_reply": "2024-04-07T11:53:26.705746Z"
    },
    "papermill": {
     "duration": 1.541778,
     "end_time": "2024-04-07T11:53:26.708803",
     "exception": false,
     "start_time": "2024-04-07T11:53:25.167025",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "What is SVM?\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Support Vector Machine (SVM) is a supervised machine learning algorithm that finds a optimal boundary between the data points, called support vectors, such that the data points closest to the boundary are called support vectors.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 1.38 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question = \"What is SVM?\"\n",
    "gemma_qa.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6902ce5e",
   "metadata": {
    "papermill": {
     "duration": 0.15006,
     "end_time": "2024-04-07T11:53:27.029850",
     "exception": false,
     "start_time": "2024-04-07T11:53:26.879790",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fresh question 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c8f6b173",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:27.329742Z",
     "iopub.status.busy": "2024-04-07T11:53:27.328760Z",
     "iopub.status.idle": "2024-04-07T11:53:28.284152Z",
     "shell.execute_reply": "2024-04-07T11:53:28.283188Z"
    },
    "papermill": {
     "duration": 1.108129,
     "end_time": "2024-04-07T11:53:28.286213",
     "exception": false,
     "start_time": "2024-04-07T11:53:27.178084",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "What is Dropout?\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Dropout is a technique that randomly turns off connections between layers in a neural network, which prevents the network from overfitting.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 0.95 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question = \"What is Dropout?\"\n",
    "gemma_qa.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1473c05",
   "metadata": {
    "papermill": {
     "duration": 0.151072,
     "end_time": "2024-04-07T11:53:28.588166",
     "exception": false,
     "start_time": "2024-04-07T11:53:28.437094",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fresh question 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3b6bee85",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:28.937369Z",
     "iopub.status.busy": "2024-04-07T11:53:28.936523Z",
     "iopub.status.idle": "2024-04-07T11:53:29.296858Z",
     "shell.execute_reply": "2024-04-07T11:53:29.295824Z"
    },
    "papermill": {
     "duration": 0.516367,
     "end_time": "2024-04-07T11:53:29.298870",
     "exception": false,
     "start_time": "2024-04-07T11:53:28.782503",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "Please describe the principle of decision trees.\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Answer here\n",
       "\n",
       "**<font color='green'>Total time:</font>** 0.35 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question = \"Please describe the principle of decision trees.\"\n",
    "gemma_qa.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b30124fa",
   "metadata": {
    "papermill": {
     "duration": 0.151242,
     "end_time": "2024-04-07T11:53:29.603351",
     "exception": false,
     "start_time": "2024-04-07T11:53:29.452109",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fresh question 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8a5a04c3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-07T11:53:29.907097Z",
     "iopub.status.busy": "2024-04-07T11:53:29.906695Z",
     "iopub.status.idle": "2024-04-07T11:53:31.948162Z",
     "shell.execute_reply": "2024-04-07T11:53:31.947280Z"
    },
    "papermill": {
     "duration": 2.196055,
     "end_time": "2024-04-07T11:53:31.950344",
     "exception": false,
     "start_time": "2024-04-07T11:53:29.754289",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Question:</font>**\n",
       "Please describe logistic regression principle.\n",
       "\n",
       "**<font color='red'>Answer:</font>**\n",
       "Logistic regression is a machine learning algorithm that is used for classification problems. It is also known as the logistic function or sigmoid function. The main idea behind logistic regression is to find a set of weights that minimizes the probability of a prediction being wrong. It is used when we have a binary dependent variable and many independent variables.\n",
       "\n",
       "**<font color='green'>Total time:</font>** 2.04 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question = \"Please describe logistic regression principle.\"\n",
    "gemma_qa.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c6fceb8",
   "metadata": {
    "papermill": {
     "duration": 0.153778,
     "end_time": "2024-04-07T11:53:32.260645",
     "exception": false,
     "start_time": "2024-04-07T11:53:32.106867",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusions\n",
    "\n",
    "\n",
    "We fine-tuned Gemma with a set of Data Science interview question and answers.   \n",
    "Then we tested the model with questions from the dataset used for fine-tuning.  \n",
    "At the end, we also tested with some free formulated questions, not from the dataset."
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "gpu",
   "dataSources": [
    {
     "databundleVersionId": 7669720,
     "sourceId": 64148,
     "sourceType": "competition"
    },
    {
     "datasetId": 4498747,
     "sourceId": 7705679,
     "sourceType": "datasetVersion"
    },
    {
     "modelInstanceId": 5171,
     "sourceId": 11371,
     "sourceType": "modelInstanceVersion"
    }
   ],
   "dockerImageVersionId": 30674,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1375.985971,
   "end_time": "2024-04-07T11:53:35.866357",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-04-07T11:30:39.880386",
   "version": "2.5.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "303507cc34674fafa843262c0df181df": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "49881056178d44ad9950119085772884": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "4b356d0b8397447bb6178833e331ca31": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_6972576a52db4874b8af879c90881ec8",
        "IPY_MODEL_8160f2ac3df7420cb7eb44b27288d900",
        "IPY_MODEL_71d40c14d4ea4f71abc2f6fd5726a001"
       ],
       "layout": "IPY_MODEL_9c091eb6b86f4485a718d8e2b7718835"
      }
     },
     "6972576a52db4874b8af879c90881ec8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_49881056178d44ad9950119085772884",
       "placeholder": "​",
       "style": "IPY_MODEL_ff017f8662f648788a9b4779f63ab51f",
       "value": "100%"
      }
     },
     "71d40c14d4ea4f71abc2f6fd5726a001": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_303507cc34674fafa843262c0df181df",
       "placeholder": "​",
       "style": "IPY_MODEL_ab8a4927a9f54e23aaf4c1c716602cd1",
       "value": " 166/166 [00:00&lt;00:00, 8267.68it/s]"
      }
     },
     "8160f2ac3df7420cb7eb44b27288d900": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_cf94252097bd4fb582e4ec82c4bc8abf",
       "max": 166.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_aee04bcc80984058a2a5d159ae1d0f6f",
       "value": 166.0
      }
     },
     "9c091eb6b86f4485a718d8e2b7718835": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ab8a4927a9f54e23aaf4c1c716602cd1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "aee04bcc80984058a2a5d159ae1d0f6f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "cf94252097bd4fb582e4ec82c4bc8abf": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ff017f8662f648788a9b4779f63ab51f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}