fine-tuning-gemma-2-model-using-lora-and-keras.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/gpreda/fine-tuning-gemma-2-model-using-lora-and-keras?scriptVersionId=205427313\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","id":"3ee885e0","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.012006,"end_time":"2024-11-05T17:23:21.339303","exception":false,"start_time":"2024-11-05T17:23:21.327297","status":"completed"},"tags":[]},"source":["<center><h1>Fine-tuning Gemma 2 model using LoRA and Keras</h1></center>\n","\n","<center><img src=\"https://res.infoq.com/news/2024/02/google-gemma-open-model/en/headerimage/generatedHeaderImage-1708977571481.jpg\" width=\"400\"></center>\n","\n","\n","# Introduction\n","\n","This notebook will demonstrate three things:\n","\n","1. How to fine-tune Gemma model using LoRA\n","2. Creation of a specialised class to query about Kaggle features\n","3. Some results of querying about Kaggle Docs\n","\n","This work is largely based on previous work. Here I list the sources:\n","\n","1. Gemma 2 Model Card, Kaggle Models,https://www.kaggle.com/models/google/gemma-2/\n","2. Kaggle QA with Gemma - KerasNLP Starter, Kaggle Code, https://www.kaggle.com/code/awsaf49/kaggle-qa-with-gemma-kerasnlp-starter (Version 11)  \n","3. Fine-tune Gemma models in Keras using LoRA, Kaggle Code, https://www.kaggle.com/code/nilaychauhan/fine-tune-gemma-models-in-keras-using-lora (Version 1) \n","4. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, LoRA: Low-Rank Adaptation of Large Language Models, ArXiv, https://arxiv.org/pdf/2106.09685.pdf\n","5. Abheesht Sharma, Matthew Watson, Parameter-efficient fine-tuning of GPT-2 with LoRA, https://keras.io/examples/nlp/parameter_efficient_finetuning_of_gpt2_with_lora/\n","6. Keras 3 API documentation / KerasNLP / Models / Gemma, https://keras.io/api/keras_nlp/models/gemma/\n","7. Unlock the Power of Gemma 2: Prompt it like a Pro, https://www.kaggle.com/code/gpreda/unlock-the-power-of-gemma-2-prompt-it-like-a-pro  \n","8. Fine-tune Gemma using LoRA and Keras, https://www.kaggle.com/code/gpreda/fine-tune-gemma-using-lora-and-keras\n","9. Fine-tunning Gemma model with Kaggle Docs data, https://www.kaggle.com/code/gpreda/fine-tunning-gemma-model-with-kaggle-docs-data\n","10. Kaggle Docs, Kaggle Dataset, https://www.kaggle.com/datasets/awsaf49/kaggle-docs  \n","\n","\n","**Let's go**!\n"]},{"cell_type":"markdown","id":"11fc1632","metadata":{"papermill":{"duration":0.010966,"end_time":"2024-11-05T17:23:21.361715","exception":false,"start_time":"2024-11-05T17:23:21.350749","status":"completed"},"tags":[]},"source":["# What is Gemma 2?\n","\n","Gemma is a collection of lightweight, advanced open models developed by Google, leveraging the same research and technology behind the Gemini models. These models are text-to-text, decoder-only large language models available in English, with open weights provided for both pre-trained and instruction-tuned versions. Gemma models excel in a range of text generation tasks, such as question answering, summarization, and reasoning. Their compact size allows for deployment in resource-constrained environments like laptops, desktops, or personal cloud infrastructure, making state-of-the-art AI models more accessible and encouraging innovation for all. \n","\n","Gemma 2 represent the 2nd generation of Gemma models. These models were trained on a dataset of text data that includes a wide variety of sources. The **27B** model was trained with **13 trillion** tokens, the **9B** model was trained with **8 trillion tokens**, and **2B** model was trained with **2 trillion** tokens. Here is a summary of their key components: \n","* **Web Documents**: A diverse collection of web text ensures the model is exposed to a broad range of linguistic styles, topics, and vocabulary. Primarily English-language content.\n","* **Code**: Exposing the model to code helps it to learn the syntax and patterns of programming languages, which improves its ability to generate code or understand code-related questions.\n","* **Mathematics**: Training on mathematical text helps the model learn logical reasoning, symbolic representation, and to address mathematical queries.\n","\n","To learn more about Gemma 2, follow this link: [Gemma 2 Model Card](https://www.kaggle.com/models/google/gemma-2).\n","\n","\n"]},{"cell_type":"markdown","id":"34cbbd4e","metadata":{"papermill":{"duration":0.011192,"end_time":"2024-11-05T17:23:21.384098","exception":false,"start_time":"2024-11-05T17:23:21.372906","status":"completed"},"tags":[]},"source":["# What is LoRA?  \n","\n","**LoRA** stands for **Low-Rank Adaptation**. It is a method used to fine-tune large language models (LLMs) by freezing the weights of the LLM and injecting trainable rank-decomposition matrices. The number of trainable parameters during fine-tunning will decrease therefore considerably. According to **LoRA** paper, this number decreases **10,000 times**, and the computational resources size decreases 3 times. "]},{"cell_type":"markdown","id":"439af649","metadata":{"papermill":{"duration":0.011135,"end_time":"2024-11-05T17:23:21.407325","exception":false,"start_time":"2024-11-05T17:23:21.39619","status":"completed"},"tags":[]},"source":["# How we proceed?\n","\n","For fine-tunning with LoRA, we will follow the steps:\n","\n","1. Install prerequisites\n","2. Load and process the data for fine-tuning\n","3. Initialize the code for Gemma causal language model (Gemma Causal LM)\n","4. Perform fine-tuning\n","5. Test the fine-tunned model with questions from the data used for fine-tuning and with aditional questions"]},{"cell_type":"markdown","id":"820b88eb","metadata":{"papermill":{"duration":0.010919,"end_time":"2024-11-05T17:23:21.429364","exception":false,"start_time":"2024-11-05T17:23:21.418445","status":"completed"},"tags":[]},"source":["# Prerequisites\n","\n","\n","## Install packages\n","\n","We start by installing `keras-nlp` and `keras` packages."]},{"cell_type":"code","execution_count":1,"id":"380310d3","metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2024-11-05T17:23:21.45338Z","iopub.status.busy":"2024-11-05T17:23:21.453016Z","iopub.status.idle":"2024-11-05T17:24:05.176857Z","shell.execute_reply":"2024-11-05T17:24:05.175942Z"},"papermill":{"duration":43.738707,"end_time":"2024-11-05T17:24:05.179216","exception":false,"start_time":"2024-11-05T17:23:21.440509","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n","tensorflow-decision-forests 1.8.1 requires wurlitzer, which is not installed.\u001b[0m\u001b[31m\r\n","\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n","tensorflow-decision-forests 1.8.1 requires wurlitzer, which is not installed.\r\n","tensorflow 2.15.0 requires keras<2.16,>=2.15.0, but you have keras 3.6.0 which is incompatible.\u001b[0m\u001b[31m\r\n","\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n","keras-cv 0.8.2 requires keras-core, which is not installed.\u001b[0m\u001b[31m\r\n","\u001b[0m"]}],"source":["# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n","!pip install -q -U keras-nlp\n","!pip install -q -U keras>=3\n","!pip install -q -U kagglehub --upgrade"]},{"cell_type":"markdown","id":"655c997a","metadata":{"papermill":{"duration":0.013429,"end_time":"2024-11-05T17:24:05.204723","exception":false,"start_time":"2024-11-05T17:24:05.191294","status":"completed"},"tags":[]},"source":["## Import packages\n","\n","Now we can import the packages we just installed. We will also install `os`, so that we can set the environment variables needed for keras backend. We will use `jax` as `KERAS_BACKEND`.\n","\n","Because we want to publish the Model from the Notebook, we also include `kagglehub` and import secrets from `Kaggle App`."]},{"cell_type":"code","execution_count":2,"id":"dcf9db90","metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2024-11-05T17:24:05.230296Z","iopub.status.busy":"2024-11-05T17:24:05.229557Z","iopub.status.idle":"2024-11-05T17:24:20.226199Z","shell.execute_reply":"2024-11-05T17:24:20.225392Z"},"papermill":{"duration":15.011949,"end_time":"2024-11-05T17:24:20.228648","exception":false,"start_time":"2024-11-05T17:24:05.216699","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["2024-11-05 17:24:09.650971: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n","2024-11-05 17:24:09.651102: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n","2024-11-05 17:24:09.781847: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"]}],"source":["import os\n","os.environ[\"KERAS_BACKEND\"] = \"jax\" # you can also use tensorflow or torch\n","os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\" # avoid memory fragmentation on JAX backend.\n","os.environ[\"JAX_PLATFORMS\"] = \"\"\n","import keras\n","import keras_nlp\n","import kagglehub\n","\n","\n","from kaggle_secrets import UserSecretsClient\n","user_secrets = UserSecretsClient()\n","os.environ[\"KAGGLE_USERNAME\"] = user_secrets.get_secret(\"kaggle_username\")\n","os.environ[\"KAGGLE_KEY\"] = user_secrets.get_secret(\"kaggle_key\")\n","\n","import numpy as np\n","import pandas as pd\n","from tqdm.notebook import tqdm\n","tqdm.pandas() # progress bar for pandas\n","\n","import matplotlib.pyplot as plt\n","import seaborn as sns\n","from IPython.display import display, Markdown"]},{"cell_type":"markdown","id":"715044e3","metadata":{"papermill":{"duration":0.011434,"end_time":"2024-11-05T17:24:20.252213","exception":false,"start_time":"2024-11-05T17:24:20.240779","status":"completed"},"tags":[]},"source":["## Configurations\n","\n","\n","We use a `Config` class to group the information needed to control the fine-tuning process:\n","* random seed \n","* dataset path\n","* preset - name of pretrained Gemma 2\n","* sequence length - this is the maximum size of input sequence for training\n","* batch size - size of the input batch in training, x 2 as two GPUs\n","* lora rank - rank for LoRA, higher means more trainable parameters \n","* learning rate used in the train\n","* epochs - number of epochs for train"]},{"cell_type":"code","execution_count":3,"id":"1e6b70b0","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.277465Z","iopub.status.busy":"2024-11-05T17:24:20.276854Z","iopub.status.idle":"2024-11-05T17:24:20.28208Z","shell.execute_reply":"2024-11-05T17:24:20.281264Z"},"papermill":{"duration":0.019729,"end_time":"2024-11-05T17:24:20.283929","exception":false,"start_time":"2024-11-05T17:24:20.2642","status":"completed"},"tags":[]},"outputs":[],"source":["class Config:\n","    seed = 42\n","    dataset_path = \"/kaggle/input/kaggle-docs/questions_answers\"\n","    preset = \"gemma2_2b_en\" # name of pretrained Gemma 2\n","    sequence_length = 512 # max size of input sequence for training\n","    batch_size = 1 # size of the input batch in training\n","    lora_rank = 5 # rank for LoRA, higher means more trainable parameters\n","    learning_rate=8e-5 # learning rate used in train\n","    epochs = 20 # number of epochs to train"]},{"cell_type":"markdown","id":"5be81f2a","metadata":{"papermill":{"duration":0.011322,"end_time":"2024-11-05T17:24:20.306834","exception":false,"start_time":"2024-11-05T17:24:20.295512","status":"completed"},"tags":[]},"source":["Set a random seed for results reproducibility."]},{"cell_type":"code","execution_count":4,"id":"9c1dd2e2","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.332538Z","iopub.status.busy":"2024-11-05T17:24:20.332251Z","iopub.status.idle":"2024-11-05T17:24:20.336768Z","shell.execute_reply":"2024-11-05T17:24:20.335927Z"},"papermill":{"duration":0.019488,"end_time":"2024-11-05T17:24:20.338638","exception":false,"start_time":"2024-11-05T17:24:20.31915","status":"completed"},"tags":[]},"outputs":[],"source":["keras.utils.set_random_seed(Config.seed)"]},{"cell_type":"markdown","id":"6f3d3fd9","metadata":{"papermill":{"duration":0.011221,"end_time":"2024-11-05T17:24:20.36156","exception":false,"start_time":"2024-11-05T17:24:20.350339","status":"completed"},"tags":[]},"source":["# Load the data\n","\n","\n","We load the data we will use for fine-tunining."]},{"cell_type":"code","execution_count":5,"id":"9f5775bc","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.385699Z","iopub.status.busy":"2024-11-05T17:24:20.385431Z","iopub.status.idle":"2024-11-05T17:24:20.416116Z","shell.execute_reply":"2024-11-05T17:24:20.41526Z"},"papermill":{"duration":0.045077,"end_time":"2024-11-05T17:24:20.41818","exception":false,"start_time":"2024-11-05T17:24:20.373103","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Question</th>\n","      <th>Answer</th>\n","      <th>Category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>What are the different types of competitions a...</td>\n","      <td># Types of Competitions\\n\\nKaggle Competitions...</td>\n","      <td>competition</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>What are the different competition formats on ...</td>\n","      <td>There are handful of different formats competi...</td>\n","      <td>competition</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>How to join a competition?</td>\n","      <td>Before you start, navigate to the [Competition...</td>\n","      <td>competition</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>How to form, manage, and disband teams in a co...</td>\n","      <td>Everyone that competes in a Competition does s...</td>\n","      <td>competition</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>How do I make a submission in a competition?</td>\n","      <td>You will need to submit your model predictions...</td>\n","      <td>competition</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            Question  \\\n","0  What are the different types of competitions a...   \n","1  What are the different competition formats on ...   \n","2                         How to join a competition?   \n","3  How to form, manage, and disband teams in a co...   \n","4       How do I make a submission in a competition?   \n","\n","                                              Answer     Category  \n","0  # Types of Competitions\\n\\nKaggle Competitions...  competition  \n","1  There are handful of different formats competi...  competition  \n","2  Before you start, navigate to the [Competition...  competition  \n","3  Everyone that competes in a Competition does s...  competition  \n","4  You will need to submit your model predictions...  competition  "]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["df = pd.read_csv(f\"{Config.dataset_path}/data.csv\")\n","df.head()"]},{"cell_type":"markdown","id":"05a87fb4","metadata":{"papermill":{"duration":0.011803,"end_time":"2024-11-05T17:24:20.443016","exception":false,"start_time":"2024-11-05T17:24:20.431213","status":"completed"},"tags":[]},"source":["Let's check the total number of rows in this dataset."]},{"cell_type":"code","execution_count":6,"id":"08afdf0a","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.467951Z","iopub.status.busy":"2024-11-05T17:24:20.467666Z","iopub.status.idle":"2024-11-05T17:24:20.473327Z","shell.execute_reply":"2024-11-05T17:24:20.472485Z"},"papermill":{"duration":0.02046,"end_time":"2024-11-05T17:24:20.475309","exception":false,"start_time":"2024-11-05T17:24:20.454849","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["60"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["df.shape[0]"]},{"cell_type":"markdown","id":"559dbf25","metadata":{"papermill":{"duration":0.011587,"end_time":"2024-11-05T17:24:20.498725","exception":false,"start_time":"2024-11-05T17:24:20.487138","status":"completed"},"tags":[]},"source":["For easiness, we will create the following template for QA: "]},{"cell_type":"code","execution_count":7,"id":"0980c397","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.523555Z","iopub.status.busy":"2024-11-05T17:24:20.523303Z","iopub.status.idle":"2024-11-05T17:24:20.533112Z","shell.execute_reply":"2024-11-05T17:24:20.532254Z"},"papermill":{"duration":0.024561,"end_time":"2024-11-05T17:24:20.535099","exception":false,"start_time":"2024-11-05T17:24:20.510538","status":"completed"},"tags":[]},"outputs":[],"source":["template = \"\\n\\nCategory:\\nkaggle-{Category}\\n\\nQuestion:\\n{Question}\\n\\nAnswer:\\n{Answer}\"\n","df[\"prompt\"] = df.apply(lambda row: template.format(Category=row.Category,\n","                                                             Question=row.Question,\n","                                                             Answer=row.Answer), axis=1)\n","data = df.prompt.tolist()"]},{"cell_type":"markdown","id":"a569b806","metadata":{"papermill":{"duration":0.017395,"end_time":"2024-11-05T17:24:20.564399","exception":false,"start_time":"2024-11-05T17:24:20.547004","status":"completed"},"tags":[]},"source":["## Template utility function"]},{"cell_type":"code","execution_count":8,"id":"ee3935e1","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.590478Z","iopub.status.busy":"2024-11-05T17:24:20.589857Z","iopub.status.idle":"2024-11-05T17:24:20.594852Z","shell.execute_reply":"2024-11-05T17:24:20.593978Z"},"papermill":{"duration":0.019962,"end_time":"2024-11-05T17:24:20.596772","exception":false,"start_time":"2024-11-05T17:24:20.57681","status":"completed"},"tags":[]},"outputs":[],"source":["def colorize_text(text):\n","    for word, color in zip([\"Category\", \"Question\", \"Answer\"], [\"blue\", \"red\", \"green\"]):\n","        text = text.replace(f\"\\n\\n{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n","    return text"]},{"cell_type":"markdown","id":"180b370f","metadata":{"papermill":{"duration":0.011704,"end_time":"2024-11-05T17:24:20.620532","exception":false,"start_time":"2024-11-05T17:24:20.608828","status":"completed"},"tags":[]},"source":["# Specialized class to query Gemma\n","\n","\n","We define a specialized class to query Gemma. But first, we need to initialize an object of GemmaCausalLM class."]},{"cell_type":"markdown","id":"ee31b38f","metadata":{"papermill":{"duration":0.012128,"end_time":"2024-11-05T17:24:20.689261","exception":false,"start_time":"2024-11-05T17:24:20.677133","status":"completed"},"tags":[]},"source":["## Initialize the code for Gemma Causal LM"]},{"cell_type":"code","execution_count":9,"id":"bc2cbb55","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:24:20.715163Z","iopub.status.busy":"2024-11-05T17:24:20.714356Z","iopub.status.idle":"2024-11-05T17:25:05.411325Z","shell.execute_reply":"2024-11-05T17:25:05.410244Z"},"papermill":{"duration":44.713073,"end_time":"2024-11-05T17:25:05.414342","exception":false,"start_time":"2024-11-05T17:24:20.701269","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.\n"]},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n","</pre>\n"],"text/plain":["\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n","└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n","</pre>\n"],"text/plain":["┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n","└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n","</pre>\n"],"text/plain":["\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n","│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n","│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n","└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n","</pre>\n"],"text/plain":["┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,614,341,888\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n","│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n","│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n","└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n","</pre>\n"],"text/plain":["\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n","</pre>\n"],"text/plain":["\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n","</pre>\n"],"text/plain":["\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"]},"metadata":{},"output_type":"display_data"}],"source":["gemma_causal_lm = keras_nlp.models.GemmaCausalLM.from_preset(Config.preset)\n","gemma_causal_lm.summary()"]},{"cell_type":"markdown","id":"bebd4de2","metadata":{"papermill":{"duration":0.014621,"end_time":"2024-11-05T17:25:05.444361","exception":false,"start_time":"2024-11-05T17:25:05.42974","status":"completed"},"tags":[]},"source":["## Define the specialized class\n","\n","Here we define the special class `GemmaQA`. \n","in the `__init__` we pass the `GemmaCausalLM` object created before.\n","The `query` member function uses `GemmaCausalLM` member function `generate` to generate the answer, based on a prompt that includes the category and the question."]},{"cell_type":"code","execution_count":10,"id":"c22f82ca","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:25:05.472765Z","iopub.status.busy":"2024-11-05T17:25:05.472445Z","iopub.status.idle":"2024-11-05T17:25:05.478936Z","shell.execute_reply":"2024-11-05T17:25:05.477894Z"},"papermill":{"duration":0.02268,"end_time":"2024-11-05T17:25:05.48097","exception":false,"start_time":"2024-11-05T17:25:05.45829","status":"completed"},"tags":[]},"outputs":[],"source":["class GemmaQA:\n","    def __init__(self, max_length=512):\n","        self.max_length = max_length\n","        self.prompt = template\n","        self.gemma_causal_lm = gemma_causal_lm\n","        \n","    def query(self, category, question):\n","        response = self.gemma_causal_lm.generate(\n","            self.prompt.format(\n","                Category=category,\n","                Question=question,\n","                Answer=\"\"), \n","            max_length=self.max_length)\n","        display(Markdown(colorize_text(response)))\n","        "]},{"cell_type":"markdown","id":"c4c3952b","metadata":{"papermill":{"duration":0.012898,"end_time":"2024-11-05T17:25:05.507625","exception":false,"start_time":"2024-11-05T17:25:05.494727","status":"completed"},"tags":[]},"source":["## Gemma preprocessor\n","\n","\n","This preprocessing layer will take in batches of strings, and return outputs in a ```(x, y, sample_weight)``` format, where the y label is the next token id in the x sequence.\n","\n","From the code below, we can see that, after the preprocessor, the data shape is ```(num_samples, sequence_length)```."]},{"cell_type":"code","execution_count":11,"id":"bed9e7df","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:25:05.539318Z","iopub.status.busy":"2024-11-05T17:25:05.538937Z","iopub.status.idle":"2024-11-05T17:25:05.669821Z","shell.execute_reply":"2024-11-05T17:25:05.669031Z"},"papermill":{"duration":0.149773,"end_time":"2024-11-05T17:25:05.672058","exception":false,"start_time":"2024-11-05T17:25:05.522285","status":"completed"},"tags":[]},"outputs":[],"source":["x, y, sample_weight = gemma_causal_lm.preprocessor(data[0:2])"]},{"cell_type":"code","execution_count":12,"id":"b7411721","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:25:05.701651Z","iopub.status.busy":"2024-11-05T17:25:05.700866Z","iopub.status.idle":"2024-11-05T17:25:05.707338Z","shell.execute_reply":"2024-11-05T17:25:05.706335Z"},"papermill":{"duration":0.023361,"end_time":"2024-11-05T17:25:05.709214","exception":false,"start_time":"2024-11-05T17:25:05.685853","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["{'token_ids': Array([[     2,    109,   8606, ...,  25688, 235290,  75676],\n","       [     2,    109,   8606, ...,    109,    688,   2299]],      dtype=int32), 'padding_mask': Array([[ True,  True,  True, ...,  True,  True,  True],\n","       [ True,  True,  True, ...,  True,  True,  True]], dtype=bool)} [[   109   8606 235292 ... 235290  75676      1]\n"," [   109   8606 235292 ...    688   2299      1]]\n"]}],"source":["print(x, y)"]},{"cell_type":"markdown","id":"13cadebf","metadata":{"papermill":{"duration":0.013477,"end_time":"2024-11-05T17:25:05.736601","exception":false,"start_time":"2024-11-05T17:25:05.723124","status":"completed"},"tags":[]},"source":["# Perform fine-tuning with LoRA"]},{"cell_type":"markdown","id":"180b5c49","metadata":{"papermill":{"duration":0.013661,"end_time":"2024-11-05T17:25:05.763813","exception":false,"start_time":"2024-11-05T17:25:05.750152","status":"completed"},"tags":[]},"source":["## Enable LoRA for the model\n","\n","LoRA rank is setting the number of trainable parameters. A larger rank will result in a larger number of parameters to train."]},{"cell_type":"code","execution_count":13,"id":"95a84c49","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:25:05.793102Z","iopub.status.busy":"2024-11-05T17:25:05.791946Z","iopub.status.idle":"2024-11-05T17:25:06.300077Z","shell.execute_reply":"2024-11-05T17:25:06.29924Z"},"papermill":{"duration":0.524745,"end_time":"2024-11-05T17:25:06.302121","exception":false,"start_time":"2024-11-05T17:25:05.777376","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n","</pre>\n"],"text/plain":["\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n","└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n","</pre>\n"],"text/plain":["┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n","└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n","</pre>\n"],"text/plain":["\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,618,002,688</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n","│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n","│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n","└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n","</pre>\n"],"text/plain":["┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n","┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n","│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,618,002,688\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n","│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n","├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n","│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n","│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n","└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,618,002,688</span> (9.75 GB)\n","</pre>\n"],"text/plain":["\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,618,002,688\u001b[0m (9.75 GB)\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,660,800</span> (13.96 MB)\n","</pre>\n"],"text/plain":["\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m3,660,800\u001b[0m (13.96 MB)\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n","</pre>\n"],"text/plain":["\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"]},"metadata":{},"output_type":"display_data"}],"source":["# Enable LoRA for the model and set the LoRA rank to the lora_rank as set in Config (4).\n","gemma_causal_lm.backbone.enable_lora(rank=Config.lora_rank)\n","gemma_causal_lm.summary()"]},{"cell_type":"markdown","id":"5196cae7","metadata":{"papermill":{"duration":0.014369,"end_time":"2024-11-05T17:25:06.331261","exception":false,"start_time":"2024-11-05T17:25:06.316892","status":"completed"},"tags":[]},"source":["We see that only a small part of the parameters are trainable. 2.6 billions parameters total, and only 2.9 Millions parameters trainable."]},{"cell_type":"markdown","id":"196609e6","metadata":{"papermill":{"duration":0.014237,"end_time":"2024-11-05T17:25:06.360325","exception":false,"start_time":"2024-11-05T17:25:06.346088","status":"completed"},"tags":[]},"source":["## Run the training sequence\n","\n","We set the `sequence_length` for the `GemmaCausalLM` (from configuration, will be 512).\n","We compile the model, with the loss, optimizer and metric.\n","For the metric, it is used `SparseCategoricalAccuracy`. This metric calculates how often predictions match integer labels."]},{"cell_type":"code","execution_count":14,"id":"ec66fb35","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:25:06.390768Z","iopub.status.busy":"2024-11-05T17:25:06.390413Z","iopub.status.idle":"2024-11-05T17:44:00.647751Z","shell.execute_reply":"2024-11-05T17:44:00.646794Z"},"papermill":{"duration":1134.274837,"end_time":"2024-11-05T17:44:00.649679","exception":false,"start_time":"2024-11-05T17:25:06.374842","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Epoch 1/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m88s\u001b[0m 828ms/step - loss: 1.6834 - sparse_categorical_accuracy: 0.5354\n","Epoch 2/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 1.5878 - sparse_categorical_accuracy: 0.5486\n","Epoch 3/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 1.5177 - sparse_categorical_accuracy: 0.5575\n","Epoch 4/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 1.4742 - sparse_categorical_accuracy: 0.5677\n","Epoch 5/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m83s\u001b[0m 1s/step - loss: 1.4314 - sparse_categorical_accuracy: 0.5768\n","Epoch 6/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m83s\u001b[0m 1s/step - loss: 1.3843 - sparse_categorical_accuracy: 0.5854\n","Epoch 7/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 1.3279 - sparse_categorical_accuracy: 0.5973\n","Epoch 8/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 1.2613 - sparse_categorical_accuracy: 0.6115\n","Epoch 9/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 1.1836 - sparse_categorical_accuracy: 0.6291\n","Epoch 10/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 1.0991 - sparse_categorical_accuracy: 0.6519\n","Epoch 11/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 1.0191 - sparse_categorical_accuracy: 0.6757\n","Epoch 12/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.9281 - sparse_categorical_accuracy: 0.7005\n","Epoch 13/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.8552 - sparse_categorical_accuracy: 0.7196\n","Epoch 14/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.7909 - sparse_categorical_accuracy: 0.7423\n","Epoch 15/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 0.7449 - sparse_categorical_accuracy: 0.7560\n","Epoch 16/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.6686 - sparse_categorical_accuracy: 0.7773\n","Epoch 17/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.5975 - sparse_categorical_accuracy: 0.7981\n","Epoch 18/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m83s\u001b[0m 1s/step - loss: 0.5333 - sparse_categorical_accuracy: 0.8190\n","Epoch 19/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 826ms/step - loss: 0.4764 - sparse_categorical_accuracy: 0.8381\n","Epoch 20/20\n","\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m50s\u001b[0m 825ms/step - loss: 0.4541 - sparse_categorical_accuracy: 0.8440\n"]},{"data":{"text/plain":["<keras.src.callbacks.history.History at 0x7fadf8518ee0>"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["#set sequence length cf. config (512)\n","gemma_causal_lm.preprocessor.sequence_length = Config.sequence_length \n","\n","# Compile the model with loss, optimizer, and metric\n","gemma_causal_lm.compile(\n","    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n","    optimizer=keras.optimizers.Adam(learning_rate=Config.learning_rate),\n","    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",")\n","\n","# Train model\n","gemma_causal_lm.fit(data, epochs=Config.epochs, batch_size=Config.batch_size)"]},{"cell_type":"markdown","id":"3bbc44e4","metadata":{"papermill":{"duration":0.109677,"end_time":"2024-11-05T17:44:00.871183","exception":false,"start_time":"2024-11-05T17:44:00.761506","status":"completed"},"tags":[]},"source":["# Test the fine-tuned model\n","\n","We instantiate an object of class GemmaQA. Because `gemma_causal_lm` was fine-tuned using LoRA, `gemma_qa` defined here will use the fine-tuned model."]},{"cell_type":"code","execution_count":15,"id":"442caadc","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:44:01.093808Z","iopub.status.busy":"2024-11-05T17:44:01.092925Z","iopub.status.idle":"2024-11-05T17:44:01.097315Z","shell.execute_reply":"2024-11-05T17:44:01.096457Z"},"papermill":{"duration":0.117538,"end_time":"2024-11-05T17:44:01.099135","exception":false,"start_time":"2024-11-05T17:44:00.981597","status":"completed"},"tags":[]},"outputs":[],"source":["gemma_qa = GemmaQA()"]},{"cell_type":"markdown","id":"a5bc0e2f","metadata":{"papermill":{"duration":0.110991,"end_time":"2024-11-05T17:44:01.320153","exception":false,"start_time":"2024-11-05T17:44:01.209162","status":"completed"},"tags":[]},"source":["For start, we are testing the model with some of the data from the training set itself."]},{"cell_type":"markdown","id":"4a3f6af2","metadata":{"papermill":{"duration":0.110523,"end_time":"2024-11-05T17:44:01.540975","exception":false,"start_time":"2024-11-05T17:44:01.430452","status":"completed"},"tags":[]},"source":["## Sample 1"]},{"cell_type":"code","execution_count":16,"id":"ae8eb6e2","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:44:01.765545Z","iopub.status.busy":"2024-11-05T17:44:01.765199Z","iopub.status.idle":"2024-11-05T17:44:42.256048Z","shell.execute_reply":"2024-11-05T17:44:42.255088Z"},"papermill":{"duration":40.717305,"end_time":"2024-11-05T17:44:42.370625","exception":false,"start_time":"2024-11-05T17:44:01.65332","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-competition\n","\n","**<font color='red'>Question:</font>**\n","What are the different types of competitions available on Kaggle?\n","\n","**<font color='green'>Answer:</font>**\n","# Types of Competitions\n","\n","Kaggle Competitions are a central part of the Kaggle community’s collaborative data science problem-solving ethos. Every week, a new competition is launched with the goal of tackling a real-world data science or machine learning problem in just one week.\n","\n","Currently, Kaggle Competitions are launched on a weekly basis. In the future, we plan to increase the cadence to launch a new competition every other day or even more frequently.\n","\n","Currently, Kaggle Competitions are launched in two formats:\n","\n","# 1) Community Competitions\n","\n","Community Competitions are launched with a relatively open problem statement and encourage collaborative problem-solving. Community Competition participants have the freedom to explore any approach they’d like and may or may not use the public data or ground truth solution provided. At the end of the competition period, the team with the highest overall accuracy on the test set will be declared the winner.\n","\n","If you’d like to encourage collaboration and foster a friendly problem-solving atmosphere, Community Competitions are a great choice.\n","\n","# 2) Restricted Competitions\n","\n","Restricted Competitions are launched with a more specific problem statement and are restricted to teams that register and accept the competition’s terms and conditions. This type of competition is a good fit for problems that require restricted data or an exclusive solution set.\n","\n","## Examples\n","\n","Here are some examples of the different types of Kaggle Competitions we launch:\n","\n","### Community Competitions\n","\n","[COVID-19 Hospitalizations Prediction Competition](https://www.kaggle.com/c/covid19-hospitalizations-prediction)\n","\n","[Titanic: Machine Learning from Disaster Competition](https://www.kaggle.com/c/the-titanic-machine-learning-from-disaster)\n","\n","[2020 Super Mario Maker Competition](https://www.kaggle.com/c/super-mario-maker-dataset)\n","\n","### Restricted Competitions\n","\n","[2019 Leaderboard Competition](https://www.kaggle.com/c/2019-leaderboard-competition)\n","\n","[2018 Leaderboard Competition](https://www.kaggle.com/c/2018-leaderboard-competition)\n","\n","[2017 Leaderboard Competition](https://www.kaggle.com/c/2017-leaderboard-competition)\n","\n","[2016 Leaderboard Competition](https"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["row = df.iloc[0]\n","gemma_qa.query(row.Category,row.Question)"]},{"cell_type":"markdown","id":"bc60bfc6","metadata":{"papermill":{"duration":0.111435,"end_time":"2024-11-05T17:44:42.593312","exception":false,"start_time":"2024-11-05T17:44:42.481877","status":"completed"},"tags":[]},"source":["## Sample 2"]},{"cell_type":"code","execution_count":17,"id":"2f568d3a","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:44:42.823378Z","iopub.status.busy":"2024-11-05T17:44:42.822959Z","iopub.status.idle":"2024-11-05T17:44:49.552079Z","shell.execute_reply":"2024-11-05T17:44:49.55117Z"},"papermill":{"duration":6.849332,"end_time":"2024-11-05T17:44:49.554115","exception":false,"start_time":"2024-11-05T17:44:42.704783","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-tpu\n","\n","**<font color='red'>Question:</font>**\n","How to load and save model on TPU?\n","\n","**<font color='green'>Answer:</font>**\n","When saving and loading a model to/from disk on TPU, we need to use special functionality due to the way TPUs are organized.\n","\n","First, we need to create a TPUPostV2. This is a special VM that TPUs run on. It has a lot of extra features, including saving and loading functionality for models in TensorFlow’s model.json format. To create a TPUPostV2, run the command `tpu-proxmox-image`.\n","\n","Now that we have a TPUPostV2, we can create a TPUSession from it. The arguments for creating a TPUSession are:\n","\n","```\n","TPUSession(tpu-proxmox-image) # creates a TPUPostV2 VM from Proxmox\n","TPUSession(p"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["row = df.iloc[15]\n","gemma_qa.query(row.Category,row.Question)"]},{"cell_type":"markdown","id":"1f45b2fc","metadata":{"papermill":{"duration":0.110781,"end_time":"2024-11-05T17:44:49.791955","exception":false,"start_time":"2024-11-05T17:44:49.681174","status":"completed"},"tags":[]},"source":["## Sample 3"]},{"cell_type":"code","execution_count":18,"id":"0a8972c3","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:44:50.018365Z","iopub.status.busy":"2024-11-05T17:44:50.017542Z","iopub.status.idle":"2024-11-05T17:44:57.097316Z","shell.execute_reply":"2024-11-05T17:44:57.096348Z"},"papermill":{"duration":7.195114,"end_time":"2024-11-05T17:44:57.099251","exception":false,"start_time":"2024-11-05T17:44:49.904137","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-noteboook\n","\n","**<font color='red'>Question:</font>**\n","What are the different types of notebooks available on Kaggle?\n","\n","**<font color='green'>Answer:</font>**\n","There are two different types of notebooks available on Kaggle: Jupiter Notebooks and Python Scripts.\n","\n","## Jupiter Notebooks\n","\n","Jupiter Notebooks are a special type of notebook that Kaggle provides. They are based on the open source data science platform Jupiter from Google. Jupiter Notebooks allow seamless collaboration with comments and versioning, and they integrate with Kaggle competitions and other features of the Kaggle platform.\n","\n","Jupiter Notebooks are written in Python and use the Jupiter Notebook extension for TensorFlow (v1.13 or later) or Keras (v2.1.0 or later) libraries for deep learning. For other machine learning libraries, Jupiter Notebooks typically use common libraries like pandas, scikit-learn, or TensorFlow Probability.\n","\n","When creating a Notebook from the New Notebook button, Kaggle allows you to select templates for common use cases like data exploration, model training, or"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["row = df.iloc[25]\n","gemma_qa.query(row.Category,row.Question)"]},{"cell_type":"markdown","id":"f78f6af5","metadata":{"papermill":{"duration":0.111945,"end_time":"2024-11-05T17:44:57.323823","exception":false,"start_time":"2024-11-05T17:44:57.211878","status":"completed"},"tags":[]},"source":["## Not seen question(s)"]},{"cell_type":"code","execution_count":19,"id":"b0446c22","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:44:57.595469Z","iopub.status.busy":"2024-11-05T17:44:57.595119Z","iopub.status.idle":"2024-11-05T17:45:07.530749Z","shell.execute_reply":"2024-11-05T17:45:07.529782Z"},"papermill":{"duration":10.05102,"end_time":"2024-11-05T17:45:07.53274","exception":false,"start_time":"2024-11-05T17:44:57.48172","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-notebook\n","\n","**<font color='red'>Question:</font>**\n","How to run a notebook?\n","\n","**<font color='green'>Answer:</font>**\n","When you run a notebook, its cells are executed one after another, from top to\n","bottom. The execution state of a cell (i.e., whether it succeeded or failed)\n","communicates the execution state of the cells below it using \"warming pan\"\n","syntax. For example, a cell that throws an error will make all the cells\n","below it invalid (the \"cold\" state). Conversely, a successful cell makes\n","below it valid, with the possible exception of the last cell which validity\n","depends on the specific execution strategy.\n","\n","When you run a notebook from the Notebook interface, you can pause a\n","execution at any time using the keyboard's pause key (spacebar). You can\n","resume from the same position by pressing the spacebar again.\n","\n","If you prefer to control execution manually from the command line, KubeShell\n","provides an `execute` command for executing notebook commands. For example:\n","\n","```\n","kagub `execute notebook --name kaggle-samples --command execute --name-to-name-mapping {'Run': 'run'} syntax to execute lines one after another, another time to execute the entire notebook\n","```\n","\n","If you prefer to control execution manually one"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["category = \"notebook\"\n","question = \"How to run a notebook?\"\n","gemma_qa.query(category,question)"]},{"cell_type":"code","execution_count":20,"id":"d2ff9d54","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:45:07.760085Z","iopub.status.busy":"2024-11-05T17:45:07.759243Z","iopub.status.idle":"2024-11-05T17:45:24.703733Z","shell.execute_reply":"2024-11-05T17:45:24.702805Z"},"papermill":{"duration":17.059175,"end_time":"2024-11-05T17:45:24.705928","exception":false,"start_time":"2024-11-05T17:45:07.646753","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-discussions\n","\n","**<font color='red'>Question:</font>**\n","How to create a discussion topic?\n","\n","**<font color='green'>Answer:</font>**\n","Discussions is a threaded view of interesting conversations in machine learning. To start a conversation, head to [Discussions > Community](https://www.kaggle.com/discussions/community) or any of the other notebooks view formats and click on New Discussion in the upper-right corner.\n","\n","First provide a title that describes the topic of your discussion. Then jump into the text editor to share your thoughts.\n","\n","If you are posting a question, consider breaking it up into smaller questions if it gets too long. It is better to have multiple shorter questions that people can answer than one long question that no one can.\n","\n","When you are finished with your post, mark it up or down the community posts that are interesting to you. This helps identify the types of conversations that are most valuable to have on the platform.\n","\n","When you hover over a community post, you will see reaction buttons just like on an puzzle.\n","\n","If you click on “Report Post”, a modal will open where you can select the reason for your report.\n","\n","If you have any feedback or issues with a discussion, you can report the post in the same way as with individual puzzle files.\n","\n","When creating a discussion you will also notice an “subscribe” button. Subscribing to a discussion will add it to the right-hand side of the page in the “Recently Added” view format. This way you can quickly review new discussions that have been posted.\n","\n","## Viewing Replies\n","\n","By default, when you click on a new discussion, you will view the main post and the discussions replies on the same page. To view replies one after another, click on “Next” at the bottom of the page after each reply.\n","\n","If you want to view replies in a separate window, from the same page you can also click on “Discussion Replies List”.\n","\n","## Discussion Replies List\n","\n","From the same page as before, you can also access a separate window for discussion replies list view. This is an option if you prefer to view replies in a separate window from the main discussion page or if you plan to jump between discussions frequently.\n","\n","##"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["category = \"discussions\"\n","question = \"How to create a discussion topic?\"\n","gemma_qa.query(category,question)"]},{"cell_type":"code","execution_count":21,"id":"4ed642fc","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:45:24.930659Z","iopub.status.busy":"2024-11-05T17:45:24.930331Z","iopub.status.idle":"2024-11-05T17:45:30.523988Z","shell.execute_reply":"2024-11-05T17:45:30.523044Z"},"papermill":{"duration":5.707758,"end_time":"2024-11-05T17:45:30.526004","exception":false,"start_time":"2024-11-05T17:45:24.818246","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-competitions\n","\n","**<font color='red'>Question:</font>**\n","What is a code competition?\n","\n","**<font color='green'>Answer:</font>**\n","Code competitions are events where participants are challenged to solve a series of problems using a programming language of their choice. The problems are often open-ended, allowing for creative solutions, but they may also be more structured, with specific requirements or constraints.\n","\n","At the end of the competition period, participants submit their solutions and the organizers will provide a scoring or evaluation mechanism. The top-performing solutions will be recognized and awarded prizes or other recognition.\n","\n","Code competitions can be a great way to improve your programming skills, meet other programmers, and work on challenging problems in a competitive environment. They are a popular form of programming engagement and are offered by Kaggle and other organizations."],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["category = \"competitions\"\n","question = \"What is a code competition?\"\n","gemma_qa.query(category,question)"]},{"cell_type":"code","execution_count":22,"id":"7ccba25b","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:45:30.753189Z","iopub.status.busy":"2024-11-05T17:45:30.75283Z","iopub.status.idle":"2024-11-05T17:45:40.141015Z","shell.execute_reply":"2024-11-05T17:45:40.140073Z"},"papermill":{"duration":9.504302,"end_time":"2024-11-05T17:45:40.142984","exception":false,"start_time":"2024-11-05T17:45:30.638682","status":"completed"},"tags":[]},"outputs":[{"data":{"text/markdown":["\n","\n","**<font color='blue'>Category:</font>**\n","kaggle-datasets\n","\n","**<font color='red'>Question:</font>**\n","What are the steps to create a Kaggle dataset?\n","\n","**<font color='green'>Answer:</font>**\n","You can create a new Kaggle dataset in two ways:\n","\n","# Using the web interface\n","Click the `Create new content...` button at the top of any dataset's page. You will be taken to a modal that allows you to enter the details of your dataset.\n","\n","- Enter a descriptive title and subtitle for your dataset in the top fields.\n","- Select the `Dataset` content type from the menu options.\n","- Check the \"Share with public\" box to make the dataset publicly viewable.\n","- Upload a README file from your computer (or paste in the URL) to a folder on your desktop. The folder will be your dataset's public folder.\n","- Optionally, upload more files from your computer into the \"Files\" section below the file upload field. These files will be from your desktop and will be visible to anyone with a link. If you need to upload more files that are not from your desktop, use the technique described in the next section.\n","- Select the \"Permissions\" tab and, for the users listed here, select the \"View\" option. Datasets created with public permissions will not"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["category = \"datasets\"\n","question = \"What are the steps to create a Kaggle dataset?\"\n","gemma_qa.query(category,question)"]},{"cell_type":"markdown","id":"893903aa","metadata":{"papermill":{"duration":0.111914,"end_time":"2024-11-05T17:45:40.367571","exception":false,"start_time":"2024-11-05T17:45:40.255657","status":"completed"},"tags":[]},"source":["# Save the model"]},{"cell_type":"code","execution_count":23,"id":"50a36b1f","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:45:40.593882Z","iopub.status.busy":"2024-11-05T17:45:40.593183Z","iopub.status.idle":"2024-11-05T17:46:21.878985Z","shell.execute_reply":"2024-11-05T17:46:21.878086Z"},"papermill":{"duration":41.402152,"end_time":"2024-11-05T17:46:21.881337","exception":false,"start_time":"2024-11-05T17:45:40.479185","status":"completed"},"tags":[]},"outputs":[],"source":["preset_dir = \".\\gemma2_2b_en_kaggle_docs\"\n","gemma_causal_lm.save_to_preset(preset_dir)"]},{"cell_type":"markdown","id":"fb9a34d9","metadata":{"papermill":{"duration":0.111973,"end_time":"2024-11-05T17:46:22.10848","exception":false,"start_time":"2024-11-05T17:46:21.996507","status":"completed"},"tags":[]},"source":["# Publish Model on Kaggle as a Kaggle Model\n","\n","We are publishing now the saved model as a Kaggle Model."]},{"cell_type":"code","execution_count":24,"id":"208dc8f0","metadata":{"execution":{"iopub.execute_input":"2024-11-05T17:46:22.337853Z","iopub.status.busy":"2024-11-05T17:46:22.336735Z","iopub.status.idle":"2024-11-05T17:49:39.100384Z","shell.execute_reply":"2024-11-05T17:49:39.099428Z"},"papermill":{"duration":196.880309,"end_time":"2024-11-05T17:49:39.102962","exception":false,"start_time":"2024-11-05T17:46:22.222653","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Uploading Model https://www.kaggle.com/models/gpreda/gemma2-kaggle-docs/keras/gemma2_2b_en_kaggle_docs ...\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/tokenizer.json\n"]},{"name":"stderr","output_type":"stream","text":["Uploading: 100%|██████████| 591/591 [00:00<00:00, 873B/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/tokenizer.json (591B)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/metadata.json\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 143/143 [00:00<00:00, 215B/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/metadata.json (143B)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/preprocessor.json\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 1.41k/1.41k [00:00<00:00, 2.21kB/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/preprocessor.json (1KB)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/task.json\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 2.98k/2.98k [00:00<00:00, 4.60kB/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/task.json (3KB)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/config.json\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 782/782 [00:00<00:00, 1.28kB/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/config.json (782B)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/model.weights.h5\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 10.5G/10.5G [03:07<00:00, 55.9MB/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/model.weights.h5 (10GB)\n","Starting upload for file .\\gemma2_2b_en_kaggle_docs/assets/tokenizer/vocabulary.spm\n"]},{"name":"stderr","output_type":"stream","text":["\n","Uploading: 100%|██████████| 4.24M/4.24M [00:00<00:00, 5.85MB/s]"]},{"name":"stdout","output_type":"stream","text":["Upload successful: .\\gemma2_2b_en_kaggle_docs/assets/tokenizer/vocabulary.spm (4MB)\n"]},{"name":"stderr","output_type":"stream","text":["\n"]},{"name":"stdout","output_type":"stream","text":["Your model instance version has been created.\n","Files are being processed...\n","See at: https://www.kaggle.com/models/gpreda/gemma2-kaggle-docs/keras/gemma2_2b_en_kaggle_docs\n"]}],"source":["kaggle_username = os.environ[\"KAGGLE_USERNAME\"]\n","\n","kaggle_uri = f\"kaggle://{kaggle_username}/gemma2-kaggle-docs/keras/gemma2_2b_en_kaggle_docs\"\n","keras_nlp.upload_preset(kaggle_uri, preset_dir)"]},{"cell_type":"markdown","id":"48f52d71","metadata":{"papermill":{"duration":0.20093,"end_time":"2024-11-05T17:49:39.528794","exception":false,"start_time":"2024-11-05T17:49:39.327864","status":"completed"},"tags":[]},"source":["# Conclusions\n","\n"]},{"cell_type":"markdown","id":"e86741c2","metadata":{"papermill":{"duration":0.198373,"end_time":"2024-11-05T17:49:39.923841","exception":false,"start_time":"2024-11-05T17:49:39.725468","status":"completed"},"tags":[]},"source":["We demonstated how to fine-tune a **Gemma 2** model using LoRA.   \n","We also created a class to run queries to the **Gemma 2** model and tested it with some examples from the existing training data but also with some new, not seen questions.   \n","We also saved the models as a Keras model. \n","Then we published the model as a Kaggle Model on Kaggle Models platform."]}],"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"datasetId":4484051,"sourceId":7711309,"sourceType":"datasetVersion"},{"isSourceIdPinned":true,"modelId":78150,"modelInstanceId":72244,"sourceId":85984,"sourceType":"modelInstanceVersion"}],"dockerImageVersionId":30674,"isGpuEnabled":true,"isInternetEnabled":true,"language":"python","sourceType":"notebook"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":1586.034427,"end_time":"2024-11-05T17:49:44.525214","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-11-05T17:23:18.490787","version":"2.5.0"}},"nbformat":4,"nbformat_minor":5}