Data_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "73fa1ead-6e29-4078-8ed9-e6485faed726",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ! pip install ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec461120",
   "metadata": {},
   "source": [
    "# Creating required folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "609342ab-438f-4838-a200-38e008bcbedf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"DASK_DATAFRAME__QUERY_PLANNING\"] = \"False\"  # Needed for running Curator on the GPU\n",
    "\n",
    "NOTEBOOK_DIR = os.path.abspath(\"\")\n",
    "DATA_DIR = os.path.join(NOTEBOOK_DIR, \"data\")\n",
    "TEMP_DIR = os.path.join(NOTEBOOK_DIR, \".temp\")\n",
    "os.makedirs(DATA_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac609117",
   "metadata": {},
   "source": [
    "# Importing all required libraries and packages\n",
    "# Importing helper functions for data processing from the python scripts in $'helpers'$ folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2cde47a1-3b77-4388-8bc6-70b79db67861",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from nemo_curator.utils.distributed_utils import get_client\n",
    "from nemo_curator.datasets import DocumentDataset\n",
    "from nemo_curator.filters import WordCountFilter\n",
    "from nemo_curator.modifiers import UnicodeReformatter\n",
    "from nemo_curator.utils.file_utils import expand_outdir_and_mkdir\n",
    "from nemo_curator import ScoreFilter, Sequential\n",
    "from nemo_curator.modules.modify import Modify\n",
    "from functools import partial\n",
    "from nemo_curator.filters import RepeatingTopNGramsFilter\n",
    "\n",
    "\n",
    "# # Importing helper functions\n",
    "# from helpers.filters import FilterLowScores\n",
    "# from helpers.docbuilder import download_and_convert_dataset\n",
    "from helpers.filters import FilterLowScores\n",
    "from helpers.docbuilder import download_and_convert_dataset\n",
    "from helpers.modifiers import SectionNumberFormatter, redact_pii\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd51862",
   "metadata": {},
   "source": [
    "# Setting System Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bccd3c3c-a123-4815-8085-62d469a66b32",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuDF Spilling is enabled\n"
     ]
    }
   ],
   "source": [
    "device = \"gpu\"\n",
    "n_workers = 16\n",
    "client = get_client(device, n_workers=n_workers, set_torch_to_use_rmm=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5e96b0f",
   "metadata": {},
   "source": [
    "# Download the data from huggingface and processing it\n",
    "* Each question and title pair has multiple answers. The answer with highest score is picked and the rest of the answers are discarded.\n",
    "* The dataset is converted into $DocumentDataset$ format provided by NeMo Curator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "36c77d8a-c353-4e81-a080-2a4f1e730a12",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Download directory:  /root/ODSC-Hackathon-Repository/data/raw\n",
      "Downloading Law QA dataset from 'https://huggingface.co/datasets/ymoslem/Law-StackExchange/resolve/main/law-stackexchange-questions-answers.json'...\n"
     ]
    }
   ],
   "source": [
    "dataset_df = download_and_convert_dataset(DATA_DIR)\n",
    "raw_dataset = DocumentDataset.from_pandas(dataset_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75af59d2",
   "metadata": {},
   "source": [
    "# Running Data Curation pipeline:\n",
    "### Steps in the curation process:\n",
    "\n",
    "1. Unify text encoding to Unicode for the \"title\" and \"question\" fields.\n",
    "    * We used the $UnicodeReformatter$ under $DocumentModifier$ provided in $NeMo Curator$.\n",
    "\n",
    "\n",
    "\n",
    "2. Apply formatting for section numbers in the \"question\" field. \n",
    "    * We have wrapped custom text cleaning using regex into $DocumentModifier$ provided in $NeMo Curator$.\n",
    "\n",
    "\n",
    "\n",
    "3. Remove personally identifiable information (PII) like $name of the person, address, phone number, email address, location$ from the \"title\" and \"question\" fields.\n",
    "    * We have used the $PiiModifier$ provided in $NeMo Curator$.\n",
    "\n",
    "\n",
    "\n",
    "4. Exclude questions with fewer than 15 words.\n",
    "    * We have manually investigated questions with 10, 15, 20, 30, 40, 50 words and found that questions with 15 or lesser number of words are not contextually complete and meaningful enough.\n",
    "    * So we have discarded all the questions with 15 or lesser words. We have used $WordCountFilter$ under $ScoreFilter$ provided in $NeMo Curator$.\n",
    "\n",
    "\n",
    "\n",
    "5. Remove questions with a score below zero.\n",
    "    * Every question-title pair in the dataset has a score which denotes the upvotes/downvotes received. We have discarded the questions with score less than 0 i.e. questions which have more downvotes than the upvotes.\n",
    "    * For this, we have utilized $FilterLowScores$ under $ScoreFilter$ provided in $NeMo Curator$.\n",
    "\n",
    "\n",
    "\n",
    "6. Eliminate questions with excessive repeating n-grams.\n",
    "    * We decided to remove the repeating n grams as it improves the quality of the data and reduces bias in model training as repetitive language introduces bias in language models by over-emphasizing certain phrases.\n",
    "    * We have used n=5 (i.e. 5 grams) with the repeating ratio of 0.15.\n",
    "    * We leveraged $RepeatingTopNGramsFilter$ under $ScoreFilter$ provided in $NeMo Curator$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9dc92f5-7a99-491c-9b8b-e235257889fe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def run_curation_pipeline(dataset: DocumentDataset, device: str) -> DocumentDataset:\n",
    "    print(f\"Running curation pipeline on '{device}'...\")\n",
    "    orig_dataset = dataset\n",
    "\n",
    "    cpu_curation_steps = Sequential(\n",
    "        [\n",
    "            # Unify the text encoding to Unicode.\n",
    "            Modify(UnicodeReformatter(), text_field=\"title\"),\n",
    "            Modify(UnicodeReformatter(), text_field=\"question\"),\n",
    "            # Modify(UnicodeReformatter(), text_field=\"answer\"),\n",
    "\n",
    "            Modify(SectionNumberFormatter(), text_field=\"question\"),\n",
    "            # Modify(SectionNumberFormatter(), text_field=\"answer\"),\n",
    "\n",
    "            partial(redact_pii, text_field=\"title\"),\n",
    "            partial(redact_pii, text_field=\"question\"),\n",
    "            # partial(redact_pii, text_field=\"answer\"),\n",
    "\n",
    "            ScoreFilter(\n",
    "                WordCountFilter(min_words=15),\n",
    "                text_field=\"question\",\n",
    "                score_type=int,\n",
    "            ),\n",
    "            \n",
    "            # ScoreFilter(\n",
    "            #     WordCountFilter(min_words=15),\n",
    "            #     text_field=\"answer\",\n",
    "            #     score_type=int,\n",
    "            # ),\n",
    "\n",
    "            ScoreFilter(\n",
    "                FilterLowScores(score_threshold=0),\n",
    "                text_field=\"question_score\",\n",
    "                score_type=bool,\n",
    "            ),\n",
    "\n",
    "            ScoreFilter(\n",
    "                RepeatingTopNGramsFilter(n=5, max_repeating_ngram_ratio=0.15),\n",
    "                text_field=\"question\",\n",
    "            ),\n",
    "\n",
    "            # ScoreFilter(\n",
    "            #     RepeatingTopNGramsFilter(n=5, max_repeating_ngram_ratio=0.15),\n",
    "            #     text_field=\"answer\",\n",
    "            # ),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    # Run the CPU curation steps.\n",
    "    dataset = cpu_curation_steps(dataset)\n",
    "    dataset = dataset.persist()\n",
    "    # Drop the columns that are no longer needed.\n",
    "    dataset.df = dataset.df.drop(columns=[\"answer\", \"answer_score\", \"question_score\"])\n",
    "    orig_len = len(orig_dataset.df)\n",
    "    new_len = len(dataset.df)\n",
    "\n",
    "    print(f\"Original dataset length: {orig_len}\")\n",
    "    print(f\"New dataset length: {new_len}\")\n",
    "\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0f721fc2-51b0-4567-8aed-f29d56e1b7bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running curation pipeline on 'gpu'...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/dask/dataframe/core.py:4448: UserWarning: \n",
      "You did not provide metadata, so Dask is running your function on a small dataset to guess output types. It is possible that Dask will guess incorrectly.\n",
      "To provide an explicit output types or to silence this message, please provide the `meta=` keyword, as described in the map or apply function that you are using.\n",
      "  Before: .apply(func)\n",
      "  After:  .apply(func, meta=('question', 'float64'))\n",
      "\n",
      "  warnings.warn(meta_warning(meta))\n",
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 59.17 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 59.17 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: EmailRecognizer\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: PhoneRecognizer\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: SpacyRecognizer\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: UsSsnRecognizer\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: CreditCardRecognizer\n",
      "2024-10-28 00:40:00 INFO:Loaded recognizer: IpRecognizer\n",
      "2024-10-28 00:40:00 WARNING:model_to_presidio_entity_mapping is missing from configuration, using default\n",
      "2024-10-28 00:40:00 WARNING:low_score_entity_names is missing from configuration, using default\n",
      "2024-10-28 00:40:00 WARNING:Model en_core_web_lg is not installed. Downloading...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting en-core-web-lg==3.7.1\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.7.1/en_core_web_lg-3.7.1-py3-none-any.whl (587.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m587.7/587.7 MB\u001b[0m \u001b[31m262.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.2 in /usr/local/lib/python3.10/dist-packages (from en-core-web-lg==3.7.1) (3.7.5)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.0.10)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.0.8)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (8.2.5)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.1.3)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.4.1)\n",
      "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.12.3)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (4.66.5)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.32.3)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.8.2)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.1.4)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (75.2.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.4.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.26.4)\n",
      "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.2.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.20.1)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (4.12.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.4.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.26.20)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2024.7.4)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.1.5)\n",
      "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (8.1.7)\n",
      "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.5.4)\n",
      "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (13.7.1)\n",
      "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.18.1)\n",
      "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (7.0.4)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.1.5)\n",
      "Requirement already satisfied: marisa-trie>=0.7.7 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.2.0)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (2.18.0)\n",
      "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (1.16.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-lg==3.7.1) (0.1.2)\n",
      "Installing collected packages: en-core-web-lg\n",
      "Successfully installed en-core-web-lg-3.7.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n",
      "2024-10-28 00:40:13 INFO:Finished downloading model en_core_web_lg\n",
      "2024-10-28 00:40:15 INFO:Loaded recognizer: AddressRecognizer\n",
      "2024-10-28 00:40:15 WARNING:Entity URL doesn't have the corresponding recognizer in language : en\n",
      "2024-10-28 00:40:50 WARNING:Entity URL doesn't have the corresponding recognizer in language : en\n",
      "2024-10-28 00:47:04 WARNING:Entity URL doesn't have the corresponding recognizer in language : en\n",
      "2024-10-28 00:47:37 WARNING:Entity URL doesn't have the corresponding recognizer in language : en\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original dataset length: 24343\n",
      "New dataset length: 21069\n"
     ]
    }
   ],
   "source": [
    "curated_dataset = run_curation_pipeline(raw_dataset, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1ff2b3",
   "metadata": {},
   "source": [
    "# Creating System Prompt\n",
    "* We created each question-title input in such a way that, there is a system prompt that instructs the LLM what the task is provided the question, title and the list of tags.\n",
    "* The training data will include the tags associated with the question-title pair as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4e35ed42-6b66-44a2-a1aa-1add7e411357",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "SYSTEM_PROMPT = '''\n",
    "You are an expert in Law and also in tagging legal questions.\n",
    "You are provided with a question enclosed in +++++ and it's corresponding title enclosed in >>>>> from the law domain.\n",
    "\n",
    "You are also provided with a list of all the tags, enclosed in ^^^^^.\n",
    "\n",
    "Your task is to:\n",
    "i. Understand the question and it's title.\n",
    "ii. Pick up the tags that are most appropriate and relevant to the question, strictly from the tags provided to you.\n",
    "iii. Make sure you return the tags alone without their description.\n",
    "\n",
    "\n",
    "```\n",
    "NOTES: All tags must be in lowercase, ordered lexicographically and separated by commas.\n",
    "```\n",
    "\n",
    "Your output should be a JSON with the below format:\n",
    "```\n",
    "tags : <Put your relevant tags here>\n",
    "```\n",
    "\n",
    "\n",
    ">>>>>\n",
    "Title: {title}\n",
    ">>>>>\n",
    "\n",
    "\n",
    "+++++\n",
    "Question: {question}\n",
    "+++++\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbf65a7",
   "metadata": {},
   "source": [
    "# Colecting the unique list of tags from the entire train data.\n",
    "* Since each question can have multiple tags associated with it, we have collected all the list of unique tags and passed them into the prompt for the LLM to choose from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0da8ae46-0cb2-43da-9838-2b633df9dcd8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "tags_df = pd.read_csv(\"Tags and QA.csv\")\n",
    "tag_desc = list(tags_df['Tag'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1670ef36",
   "metadata": {},
   "source": [
    "# Formatting the data with the system prompt to be passed during training process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9d178bc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import ast\n",
    "from tqdm import tqdm\n",
    "\n",
    "def format_dataset(dataset: DocumentDataset, filename: str, tag_desc, SYSTEM_PROMPT) -> DocumentDataset:\n",
    "\n",
    "    df = dataset.df.compute()\n",
    "    has_tags = \"tags\" in df.columns\n",
    "\n",
    "    df.reset_index(inplace=True, drop=True)\n",
    "    # df = add_examples(df, tags_df)\n",
    "\n",
    "    inputs = []\n",
    "    outputs = []\n",
    "    for i in tqdm(range(len(df))):\n",
    "        try:\n",
    "            # inputs.append(SYSTEM_PROMPT.format(tag_desc=tag_desc, question=df['question'][i], title=df['title'][i]))\n",
    "            inputs.append(SYSTEM_PROMPT.format(question=df['question'][i], title=df['title'][i]))\n",
    "            outputs.append(df[\"tags\"][i] if has_tags else \"\")\n",
    "        except:\n",
    "            print(i)\n",
    "            pass\n",
    "\n",
    "    df['input'] = inputs\n",
    "    df['output'] = outputs\n",
    "\n",
    "    df[\"filename\"] = filename\n",
    "\n",
    "    df = df.drop(columns=[\"title\", \"question\"])\n",
    "\n",
    "    if has_tags:\n",
    "        df = df.drop(columns=[\"tags\"])\n",
    "    \n",
    "    return DocumentDataset.from_pandas(df)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "051b5ffa-f0d0-4af0-a184-56af8ad983e0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 21069/21069 [00:02<00:00, 7107.03it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original dataset columns: Index(['filename', 'id', 'title', 'question', 'tags'], dtype='object')\n",
      "Formatted dataset columns: Index(['filename', 'id', 'input', 'output'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "formatted_dataset = format_dataset(curated_dataset, \"law-stackexchange-curated.jsonl\", tag_desc, SYSTEM_PROMPT)\n",
    "print(f\"Original dataset columns: {curated_dataset.df.columns}\")\n",
    "print(f\"Formatted dataset columns: {formatted_dataset.df.columns}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7a27c2bb-3259-4c2d-a2f6-1b9f8b359e7e",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 344.89 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "You are an expert in Law and also in tagging legal questions.\n",
      "You are provided with a question enclosed in +++++ and it's corresponding title enclosed in >>>>> from the law domain.\n",
      "\n",
      "You are also provided with a list of all the tags, enclosed in ^^^^^.\n",
      "\n",
      "Your task is to:\n",
      "i. Understand the question and it's title.\n",
      "ii. Pick up the tags that are most appropriate and relevant to the question, strictly from the tags provided to you.\n",
      "iii. Make sure you return the tags alone without their description.\n",
      "\n",
      "\n",
      "```\n",
      "NOTES: All tags must be in lowercase, ordered lexicographically and separated by commas.\n",
      "```\n",
      "\n",
      "Your output should be a JSON with the below format:\n",
      "```\n",
      "tags : <Put your relevant tags here>\n",
      "```\n",
      "\n",
      "\n",
      "List of Tags to choose from:\n",
      "^^^^^\n",
      "['united-states', 'copyright', 'contract-law', 'criminal-law', 'united-kingdom', 'intellectual-property', 'gdpr', 'england-and-wales', 'internet', 'privacy', 'california', 'employment', 'licensing', 'software', 'international', 'canada', 'european-union', 'constitutional-law', 'trademark', 'legal-terms', 'civil-law', 'tax-law', 'business', 'us-constitution', 'police', 'liability', 'rental-property', 'germany', 'property', 'real-estate', 'fraud', 'landlord', 'consumer-protection', 'terms-of-service', 'corporate-law', 'traffic', 'india', 'patents', 'immigration', 'evidence', 'discrimination', 'new-york-state', 'common-law', 'open-source-software', 'texas', 'data-protection', 'tenant', 'fair-use', 'civil-procedure', 'medical', 'australia', 'residential-lease', 'freedom-of-speech', 'florida', 'theft', 'driving', 'jurisdiction', 'defamation', 'insurance', 'rules-of-court', 'rent', 'us-supreme-court', 'court', 'is-x-legal', 'legal-history', 'lawyer', 'human-rights', 'music', 'minor', 'small-claims-court', 'family-law', 'marriage', 'finance', 'ownership', 'education', 'creative-commons', 'first-amendment', 'breach-of-contract', 'jury', 'wills', 'vehicle', 'public-domain', 'divorce', 'labor-law', 'us-federal-government', 'trial', 'tort', 'citizenship', 'search-and-seizure', 'covid-19', 'commerce', 'definition', 'oregon', 'murder', 'criminal-procedure', 'new-jersey', 'children', 'self-defense', 'france', 'damages', 'legal-research', 'regulations', 'debt', 'workplace', 'banking', 'recording', 'data-ownership', 'british-columbia', 'hacking', 'washington', 'massachusetts', 'assault', 'health', 'currency', 'witnesses', 'email', 'online', 'federal-courts', 'payment', 'religion', 'trusts-and-estates', 'trespass', 'firearms', 'consent', 'non-disclosure', 'judge', 'elections', 'gpl', 'social-media', 'president', 'libel', 'photography', 'pennsylvania', 'any-jurisdiction', 'lease', 'litigation', 'appeal', 'legal-concepts', 'harassment', 'military', 'interpretation', 'limited-liability-company', 'signature', 'advertisements', 'ontario', 'dmca', 'arrest', 'inheritance', 'bankruptcy', 'youtube', 'arbitration', 'website', 'ohio', 'probate', 'rules-of-evidence', 'colorado', 'professional-ethics', 'non-profit', 'name', 'motor-vehicle', 'eula', 'new-zealand', 'fifth-amendment', 'death', 'settlement', 'illinois', 'gambling', 'civil-legal-system', 'negligence', 'derivative-work', 'travel', 'hoa', 'laws-of-war', 'abortion', 'visa', 'mail', 'voting', 'congress', 'cryptocurrency', 'process', 'fees', 'sexual-assault', 'case-law', 'precedent', 'online-piracy', 'alcohol', 'fourth-amendment', 'speeding', 'georgia', 'weapons', 'law-in-fiction', 'sentencing', 'michigan', 'attorney-client-privilege', 'tenancy-rules', 'security', 'perjury', 'incorporation', 'statutes', 'nevada', 'school', 'personal-information', 'securities', 'extradition', 'copyright-notice', 'shareholders', 'non-compete', 'employer', 'air-travel', 'double-jeopardy', 'ireland', 'virginia', 'encryption', 'data-storage', 'sanctions', 'hypothetical', 'publishing', 'confidentiality', 'sexual-offences', 'comparative-law', 'rape', 'data', 'utah', 'parental-rights', 'drugs', 'new-york-city', 'treaty', 'indian-penal-code', 'homeownership', 'arizona', 'testimony', 'animals', 'wisconsin', 'warranties', 'netherlands', 'judiciary', 'passport', 'copyright-transfer', 'pardon', 'maryland', 'legislature', 'politics', 'branding', 'identification', 'language', 'food', 'parking', 'disabilities', 'academia', 'disclaimers', 'discovery', 'pornography', 'power-of-attorney', 'switzerland', 'federalism', 'court-records', 'domain-name', 'competition', 'north-carolina', 'legal-writing', 'civil', 'statute-of-limitations', 'mental-health', 'civil-rights', 'due-process', 'indian-constitutional-law', 'accident', 'salary', 'child-abuse', 'dual-citizenship', 'hipaa', 'prosecutorial-discretion', 'real-property', 'forgery', 'subpoena', 'antitrust-law', 'breaking-the-lease', 'prosecution', 'unlicensed-practice', 'eviction', 'class-action', 'prison', 'digital-contract', 'child-support', 'homicide', 'judicial-review', 'administrative-law', 'research', 'slander', 'borders', 'health-insurance', 'investigation', 'impeachment', 'corporate-tax-laws', 'right-of-publicity', 'criminal-record', 'pro-se', 'google', 'fair-housing-acts', 'service', 'foia', 'us-president', 'manslaughter', 'loan', 'legal-citation', 'sec', 'multiple-jurisdiction', 'safety', 'russia', 'money-laundering', 'compensation', 'translation', 'mortgage', 'law-of-the-sea', 'computer-misuse-act', 'equity', 'punishment', 'spain', 'sex-discrimination', 'age', 'legal-representation', 'austria', 'scotland', 'second-amendment', 'gift', 'transportation', 'judging', 'water', 'artificial-intelligence', 'japan', 'federal-tax-law', 'law-enforcement', 'pets', 'claims', 'investment', 'authorship', 'plea', 'legal-education', 'missouri', 'us-senate', 'jurisprudence', 'china', 'conflict-of-interest', 'bribery', 'postal-service', 'customer-service', 'attorney-fees', 'restaurants', 'customs-law', 'environmental', 'battery', 'estate-planning', 'attribution', 'burden-of-proof', 'fourteenth-amendment', 'italy', 'easement', 'sweden', 'cryptography', 'product-liability', 'warrant', 'sixth-amendment', 'medical-marijuana', 'public-records', 'sport', 'equal-protection', 'suicide', 'sublease', 'history', 'new-hampshire', 'executive', 'reference-request', 'phonecall', 'religious-law', 'parliamentary-procedure', 'ccpa', 'irs', 'bills', 'civil-judgment', 'shipping', 'unions', 'cookies', 'public-law', 'statutory-construction', 'extraterritorial-jurisdiction', 'blackmail', 'naturalization', 'brexit', 'charity', 'aviation', 'local-authorities', 'standing', 'contempt-of-court', 'costs', 'minnesota', 'state-university', 'indonesia', 'malpractice', 'residency', 'adhesion-contracts', 'vaccination', 'barristers', 'separation-of-powers', 'income-tax', 'tennessee', 'deposit', 'extortion', 'notaries', 'collections', 'self-incrimination', 'sale-of-goods', 'injunctions', 'radio', 'emergencies', 'fines', 'conflict-of-laws', 'car', 'advice', 'delaware', 'phone', 'jury-nullification', 'noise', 'new-mexico', 'partnership', 'sovereignty', 'video-games', 'crime', 'birth-certificate', 'binding', 'alberta', 'refugee-law', 'governmental-liability', 'south-africa', 'capital-punishment', 'insider-trading', 'trees', 'theory-of-law', 'injury', 'court-order', 'media', 'contractor', 'waivers', 'construction-law', 'legislative-intent', 'database', 'transgender', 'texas-sb8', 'license-plates', 'indiana', 'protest', 'journalism', 'kentucky', 'enforcing-judgment', 'legal-tender', 'government', 'municipal-bylaws', 'fbi', 'americans-with-disabilities-act', 'child-porn', 'accounting', 'freedom-of-religion', 'freedom-of-information', 'official-immunity', 'work-for-hire', 'connecticut', 'dog', 'mexico', 'trade-name', 'students', 'fcc', 'false-advertising', 'prostitution', 'coppa', 'asylum', 'government-officers', 'diplomatic-immunity', 'wiretapping', 'export', 'restraining-orders', 'guardianship', 'bail', 'kidnapping', 'succession', 'belgium', 'notice', 'lotteries', 'secret', 'spam', 'hardware', 'ferpa', 'local-ordinance', 'fda', 'immunity', 'remedies', 'business-lease', 'slavery', 'never-talk-to-police', 'digital-forensics', 'choice-of-law', 'fiduciary-duties', 'drones', 'admissibility', 'family', 'miranda-warning', 'identity-theft', 'foreign-relations', 'hong-kong', 'money-service', 'detention', 'agency', 'innocent-infringement', 'misconduct', 'citizens-arrest', 'disclosure', 'unemployment', 'assured-shorthold-tenancy', 'arkansas', 'alabama', 'towing', 'use-of-force', 'israel', 'computer', 'schengen', 'interrogation', 'south-carolina', 'nationality', 'cross-examination', 'communications-law', 'cfaa', 'ethical-hacking', 'juvenile-law', 'quebec', 'entertainment-law', 'indictment', 'distribution-terms', 'grand-jury', 'credit-report', 'demand-letter', 'mens-rea', 'car-insurance', 'us-house-of-reps', 'repossession', 'copyfraud', 'bill-of-rights', 'name-title', 'equality-act-2010', 'space', 'invoice', 'ukraine', 'vandalism', 'frivolous-arguments', 'political', 'penalty', 'liens', 'public-office', 'hearsay', 'kansas', 'treason', 'natural-person', 'illegal-immigration', 'norway', 'lemon-law', 'parody', 'from-the-latin', 'labor', 'mediation', 'conversion', 'texting', 'interstate-travel', 'whistleblower', 'damage-mitigation', 'judicial-immunity', 'terrorism', 'surveillance', 'subject-access-request', 'receipts', 'private-property', 'voir-dire', 'obstruction', 'legal-fees', 'food-processing', 'f1-visa', 'common-property', 'custody', 'duty-of-care', 'police-power', 'louisiana', 'community-property', 'entrapment', 'singapore', 'census', 'brazil', 'penal-law', 'uae', 'portugal', 'philippines', 'indemnity', 'trespassing', 'expungement', 'billing', 'solicitor', 'idaho', 'telemarketing', 'employee', 'data-protection-act', 'advocacy', 'trade-secret', 'government-contracts', 'adverse-possession', 'ucc', 'agriculture', 'judgment', 'iowa', 'social-security', 'european-court-of-human-rights', 'poland', 'abuse', 'tenth-amendment', 'car-title', 'motion', 'west-virginia', 'denmark', 'digital-content', 'eminent-domain', 'deposition', 'bench-trial', 'monarchy', 'finland', 'parliamentary-law', 'nuisance', 'writ-of-mandamus', 'recusal', 'public-health', 'automotive', 'barter', 'credit-card', 'amateur-radio', 'pacer', 'anonymization', 'charge-dismissed', 'qualified-immunity', 'vehicle-registration', 'legislative-power', 'statement', 'argentina', 'conspiracy', 'rhode-island', 'pandemic', 'loan-guarantee', 'restitution', 'stalking', 'executor', 'undercover', 'mississippi', 'secession', 'adoption', 'good-samaritan', 'victoria', 'cybersquatting', 'title', 'freelance', 'acquittal', 'pleadings', 'false-arrest', 'defense', 'checks', 'ex-post-facto', 'import', 'bfoq', 'acknowledgement', 'tfeu', 'alternative-dispute-resolution', 'stateless', 'juridical-person', 'autonomous', 'constitutional-rights', 'refund', 'right-to-counsel', 'email-marketing', 'warranty', 'uk-supreme-court', 'freedom-of-press', 'sweepstakes', 'foreclosure', 'business-expense', 'counterfeiting', 'escrow', 'civil-damages', 'tenancy', 'turkey', 'saudi-arabia', 'unfair-terms', 'hungary', 'equitable-relief', 'district-of-columbia', 'bar-association', 'paperwork', 'eu-law', 'eighth-amendment', 'green-card', 'sole-propietorship', 'insanity', 'unjust-enrichment', 'appearance', 'punitive-damages', 'subsidiaries', 'school-law', 'retail', 'oklahoma', 'borrowing', 'cftc', 'fair-dealing', 'statistics', 'vermont', 'czech-republic', 'incitement', 'pakistan', 'robot', 'natural-law', 'human-resources', 'burglary', 'campaign-finance', 'northern-ireland', 'vice-president', 'preemption', 'united-nations', 'moral-duty', 'accident-insurance', 'iran', 'wrongful-death', 'opca', 'time-periods', 'war', 'attainder', 'censorship', 'legislation', 'hawaii', 'community-law', 'magna-carta', 'criminal-damage', 'work-permit', 'internet-scam', 'government-benefits', 'prohibition', 'blockchain', 'puerto-rico', 'authority', 'consideration', 'montana', 'condominium', 'nebraska', 'non-citizen', 'return', 'morality', 'tribunal', 'attorney-client-relationship', 'united-nations-charter', 'campaign', 'wrongful-conviction', 'vatican-city-state', 'right-to-repair', 'us-states', 'cctv', 'city-of-london', 'law-review', 'itar', 'medicaid', 'public-order', 'egypt', 'mineral-rights', 'jan-6', 'watershed', 'parliament', 'maine', 'bar-admission', 'can-spam-act-of-2003', 'threats', 'state-power', 'geneva-convention', 'vaccine', 'malaysia', 'res-judicata', 'prenuptial-agreement', 'fingerprint', 'motorcycle', 'summary-judgment', 'legislative-drafting', 'adult', 'zoning', 'munition', 'textualism', 'treelaw', 'fs718', 'azerbaijan', 'tribunal-procedure', 'force-majeure', 'council-tax', 'municipal-law', 'reasonableness', 'dmv', 'negotiations', 'statutory-license', 'prima-facie-speed-limits', 'reasonable', 'hidden', 'croatia', 'romania', 'deception', 'lease-renewal', 'kenya', 'electronic-procedure', 'public-relations', 'smart-contracts', 'spousal-immunity', 'civil-penalty', 'lebanon', 'trolley-problem', 'blasphemy', 'service-of-process', 'wrongful-dismissal', 'contingency-fee', 'shell-companies', 'mutual-mistake', 'electoral-commission', 'breaking-and-entering', 'iraq', 'thailand', 'prior-art', 'continental-law', 'jury-instruction', 'theft-act', 'justification', 'taiwan', 'international-court-of-justice', 'clean-hands', 'flag', 'public-transport', 'games', 'osha', 'enforcement', 'freedom-of-assembly', 'defense-of-others', 'hunting', 'european-council', 'civil-claims', '9th-circuit', 'proxy', 'knowledge', 'crown-court', 'biosafety', 'faa', 'european-convention-on-human-rights', 'drm', 'arraignment', 'standard-of-review', 'f2-visa', 'shoplifting', 'ecommerce', 'share-alike', 'marketing', 'strike', 'attempt', 'steve-bannon', 'hate-crime', 'possession', 'prejudice', 'seventh-amendment', 'privy-council', 'houston', 'in-rem', 'espionage', 'legal-aid', 'garnishment', 'professional-licensing', 'oath', 'contributory-negligence', 'domestic-abuse', 'admission-of-new-states', 'great-britain', 'cyprus', 'sri-lanka', 'slovakia', 'strata-property', 'dismissal', 'parole', 'unconscionability', 'the-hague', 'human-trafficking', 'open-carry', 'current-events', 'commerce-clause', 'county-court', 'rights-of-audience', 'remedy', 'epo', 'new-south-wales', 'utility-bills', 'bylaw', 'suggestions', 'cannabis', 'tcpa', 'cease-and-desist', 'lines-of-case-law', 'interim-possession-order', 'taa', 'meet-bob', 'enforceability', 'mercenary', 'courtroom-strategy', 'full-faith-and-credit', 'national-security', 'bangladesh', 'victims-rights', 'social-security-card', 'north-korea', 'squatting', 'third-amendment', 'section-230', 'roberts-rules-of-order', 'presumption-of-innocence', 'emotional-distress', 'legal-doctrine', 'directive', 'silks', 'amendment', 'european-court-of-justice', 'litigant-in-person', 'prima-facie', 'commercial-lease', 'perimeter-search', 'legal-advice', 'first-impression', 'disrepair', 'self-defence', 'recap', 'vietnam', 'free-law-project', 'hmrc', 'un-security-council', 'sui-juris', 'data-retention', 'ohrc', 'myanmar', 'swatting', 'living', 'dhs', 'distributed-ledger-technology', 'ethereum', 'aboriginal-law', 'msa', 'declaration-of-incompatibility', 'venezuela', 'iceland', 'alimony', 'counterterrorism', 'belarus', 'library', 'tonga', 'bigamy', 'split-estate', 'fare-evasion', 'last-generation', 'suppression', 'marital-law', 'local-authority', 'ecb', 'case-reports', 'legal-person', 'building-code', 'cloudact', 'initiative', 'coast', 'interpol', 'bailiff', 'tfl', 'notation', 'panhandling', 'lithuania', 'abolishment', '2nd-amendment', 'caste', 'death-certificate', 'circuit-splits', 'admiralty', 'knowing', 'inchoate-offences', 'stay-order', 'pre-action-protocol', 'nigeria', 'teenager', 'energy', 'nobility-clause', 'euro', 'first-instance', 'vexatious-litigants', 'calculation', 'twenty-seventh-amendment', 'adultery', 'intent', 'house-of-lords', 'robbery', 'usury', 'toyota', 'rico', 'sentence', 'code-of-federal-regulations', 'court-forms', 'life-sentence', 'nominal-damages', 'slayer-rule', 'naming-conventions', 'civil-right-act', 'peaceful-enjoyment', 'finan', 'commonwealth', 'inquisitorial-system', 'adversarial-system', 'non-fungible-token', 'inquisitorial', 'converture', 'public-accommodations', 'common-scold', 'k9']\n",
      "^^^^^\n",
      "\n",
      "\n",
      ">>>>>\n",
      "Title: Why is drunk driving causing accident punished so much worse than just drunk driving?\n",
      ">>>>>\n",
      "\n",
      "\n",
      "+++++\n",
      "Question: When people drink and drive and then cause an accident especially where if someone dies they get years and years in prison but just the act of drunk driving is punished way more lenient. Shouldn't the 2, drunk driving and drunk driving then causing accident be similarly punished? I feel like a lot of times it's luck whether an accident happens.\n",
      "+++++\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(formatted_dataset.df.head()['input'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "462d236e-5c0a-4e87-93c0-77adbc3484d1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Curated dataset columns: Index(['filename', 'id', 'input', 'output'], dtype='object')\n",
      "\n",
      "Saving curated dataset to '/root/ODSC-Hackathon-Repository/data/curated_dataset'...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 344.89 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to disk complete for 1 partitions\n"
     ]
    }
   ],
   "source": [
    "print(f\"Curated dataset columns: {formatted_dataset.df.columns}\")\n",
    "result_fp = os.path.join(DATA_DIR, \"curated_dataset\")\n",
    "print()\n",
    "print(f\"Saving curated dataset to '{result_fp}'...\")\n",
    "formatted_dataset.to_json(result_fp, write_to_filename=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3936831e",
   "metadata": {},
   "source": [
    "# Splitting the data into Train and Validation sets with 95% and 5% ratios respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4eaf3172-46f8-4db2-b93a-55e0b7fba7be",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 344.89 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original size: 21069\n",
      "After splitting:\n",
      "    Train size: 20016\n",
      "    Validation size: 1053\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 327.89 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to disk complete for 1 partitions\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 17.22 MiB.\n",
      "This may cause some slowdown.\n",
      "Consider loading the data with Dask directly\n",
      " or using futures or delayed objects to embed the data into the graph without repetition.\n",
      "See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to disk complete for 1 partitions\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "VAL_RATIO = 0.05\n",
    "\n",
    "df = formatted_dataset.df.compute()\n",
    "\n",
    "# Some sanity checks\n",
    "assert len(df) > 0, \"The dataset is empty.\"\n",
    "assert VAL_RATIO >= 0 and VAL_RATIO <= 1, \"VAL_RATIO must be between 0 and 1.\"\n",
    "val_size = int(len(df) * VAL_RATIO)\n",
    "output_dir = f\"{DATA_DIR}/split\"\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "\n",
    "# Split the data into training and temporary sets\n",
    "train_df, val_df = train_test_split(df, test_size=val_size, random_state=42)\n",
    "\n",
    "print(f\"Original size: {len(df)}\")\n",
    "print(\"After splitting:\")\n",
    "print(f\"    Train size: {len(train_df)}\")\n",
    "print(f\"    Validation size: {len(val_df)}\")\n",
    "\n",
    "train_df[\"filename\"] = \"train.jsonl\"\n",
    "val_df[\"filename\"] = \"val.jsonl\"\n",
    "\n",
    "DocumentDataset.from_pandas(train_df).to_json(output_dir, write_to_filename=True)\n",
    "DocumentDataset.from_pandas(val_df).to_json(output_dir, write_to_filename=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "49c33b0b-b42e-42f3-b659-07ef5ef9a5e0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "file_path = f\"{DATA_DIR}/split/train.jsonl\"\n",
    "import json\n",
    "data = []\n",
    "\n",
    "with open(file_path, 'r') as file:\n",
    "    for line in file:\n",
    "        data.append(json.loads(line))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "198f9c33-858a-4fd1-87f6-032075716ddc",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'filename': 'train.jsonl',\n",
       " 'id': 'law-stackexchange-qa-94691',\n",
       " 'input': '\\nYou are an expert in Law and also in tagging legal questions.\\nYou are provided with a question enclosed in +++++ and it\\'s corresponding title enclosed in >>>>> from the law domain.\\n\\nYou are also provided with a list of all the tags, enclosed in ^^^^^.\\n\\nYour task is to:\\ni. Understand the question and it\\'s title.\\nii. Pick up the tags that are most appropriate and relevant to the question, strictly from the tags provided to you.\\niii. Make sure you return the tags alone without their description.\\n\\n\\n```\\nNOTES: All tags must be in lowercase, ordered lexicographically and separated by commas.\\n```\\n\\nYour output should be a JSON with the below format:\\n```\\ntags : <Put your relevant tags here>\\n```\\n\\n\\nList of Tags to choose from:\\n^^^^^\\n[\\'united-states\\', \\'copyright\\', \\'contract-law\\', \\'criminal-law\\', \\'united-kingdom\\', \\'intellectual-property\\', \\'gdpr\\', \\'england-and-wales\\', \\'internet\\', \\'privacy\\', \\'california\\', \\'employment\\', \\'licensing\\', \\'software\\', \\'international\\', \\'canada\\', \\'european-union\\', \\'constitutional-law\\', \\'trademark\\', \\'legal-terms\\', \\'civil-law\\', \\'tax-law\\', \\'business\\', \\'us-constitution\\', \\'police\\', \\'liability\\', \\'rental-property\\', \\'germany\\', \\'property\\', \\'real-estate\\', \\'fraud\\', \\'landlord\\', \\'consumer-protection\\', \\'terms-of-service\\', \\'corporate-law\\', \\'traffic\\', \\'india\\', \\'patents\\', \\'immigration\\', \\'evidence\\', \\'discrimination\\', \\'new-york-state\\', \\'common-law\\', \\'open-source-software\\', \\'texas\\', \\'data-protection\\', \\'tenant\\', \\'fair-use\\', \\'civil-procedure\\', \\'medical\\', \\'australia\\', \\'residential-lease\\', \\'freedom-of-speech\\', \\'florida\\', \\'theft\\', \\'driving\\', \\'jurisdiction\\', \\'defamation\\', \\'insurance\\', \\'rules-of-court\\', \\'rent\\', \\'us-supreme-court\\', \\'court\\', \\'is-x-legal\\', \\'legal-history\\', \\'lawyer\\', \\'human-rights\\', \\'music\\', \\'minor\\', \\'small-claims-court\\', \\'family-law\\', \\'marriage\\', \\'finance\\', \\'ownership\\', \\'education\\', \\'creative-commons\\', \\'first-amendment\\', \\'breach-of-contract\\', \\'jury\\', \\'wills\\', \\'vehicle\\', \\'public-domain\\', \\'divorce\\', \\'labor-law\\', \\'us-federal-government\\', \\'trial\\', \\'tort\\', \\'citizenship\\', \\'search-and-seizure\\', \\'covid-19\\', \\'commerce\\', \\'definition\\', \\'oregon\\', \\'murder\\', \\'criminal-procedure\\', \\'new-jersey\\', \\'children\\', \\'self-defense\\', \\'france\\', \\'damages\\', \\'legal-research\\', \\'regulations\\', \\'debt\\', \\'workplace\\', \\'banking\\', \\'recording\\', \\'data-ownership\\', \\'british-columbia\\', \\'hacking\\', \\'washington\\', \\'massachusetts\\', \\'assault\\', \\'health\\', \\'currency\\', \\'witnesses\\', \\'email\\', \\'online\\', \\'federal-courts\\', \\'payment\\', \\'religion\\', \\'trusts-and-estates\\', \\'trespass\\', \\'firearms\\', \\'consent\\', \\'non-disclosure\\', \\'judge\\', \\'elections\\', \\'gpl\\', \\'social-media\\', \\'president\\', \\'libel\\', \\'photography\\', \\'pennsylvania\\', \\'any-jurisdiction\\', \\'lease\\', \\'litigation\\', \\'appeal\\', \\'legal-concepts\\', \\'harassment\\', \\'military\\', \\'interpretation\\', \\'limited-liability-company\\', \\'signature\\', \\'advertisements\\', \\'ontario\\', \\'dmca\\', \\'arrest\\', \\'inheritance\\', \\'bankruptcy\\', \\'youtube\\', \\'arbitration\\', \\'website\\', \\'ohio\\', \\'probate\\', \\'rules-of-evidence\\', \\'colorado\\', \\'professional-ethics\\', \\'non-profit\\', \\'name\\', \\'motor-vehicle\\', \\'eula\\', \\'new-zealand\\', \\'fifth-amendment\\', \\'death\\', \\'settlement\\', \\'illinois\\', \\'gambling\\', \\'civil-legal-system\\', \\'negligence\\', \\'derivative-work\\', \\'travel\\', \\'hoa\\', \\'laws-of-war\\', \\'abortion\\', \\'visa\\', \\'mail\\', \\'voting\\', \\'congress\\', \\'cryptocurrency\\', \\'process\\', \\'fees\\', \\'sexual-assault\\', \\'case-law\\', \\'precedent\\', \\'online-piracy\\', \\'alcohol\\', \\'fourth-amendment\\', \\'speeding\\', \\'georgia\\', \\'weapons\\', \\'law-in-fiction\\', \\'sentencing\\', \\'michigan\\', \\'attorney-client-privilege\\', \\'tenancy-rules\\', \\'security\\', \\'perjury\\', \\'incorporation\\', \\'statutes\\', \\'nevada\\', \\'school\\', \\'personal-information\\', \\'securities\\', \\'extradition\\', \\'copyright-notice\\', \\'shareholders\\', \\'non-compete\\', \\'employer\\', \\'air-travel\\', \\'double-jeopardy\\', \\'ireland\\', \\'virginia\\', \\'encryption\\', \\'data-storage\\', \\'sanctions\\', \\'hypothetical\\', \\'publishing\\', \\'confidentiality\\', \\'sexual-offences\\', \\'comparative-law\\', \\'rape\\', \\'data\\', \\'utah\\', \\'parental-rights\\', \\'drugs\\', \\'new-york-city\\', \\'treaty\\', \\'indian-penal-code\\', \\'homeownership\\', \\'arizona\\', \\'testimony\\', \\'animals\\', \\'wisconsin\\', \\'warranties\\', \\'netherlands\\', \\'judiciary\\', \\'passport\\', \\'copyright-transfer\\', \\'pardon\\', \\'maryland\\', \\'legislature\\', \\'politics\\', \\'branding\\', \\'identification\\', \\'language\\', \\'food\\', \\'parking\\', \\'disabilities\\', \\'academia\\', \\'disclaimers\\', \\'discovery\\', \\'pornography\\', \\'power-of-attorney\\', \\'switzerland\\', \\'federalism\\', \\'court-records\\', \\'domain-name\\', \\'competition\\', \\'north-carolina\\', \\'legal-writing\\', \\'civil\\', \\'statute-of-limitations\\', \\'mental-health\\', \\'civil-rights\\', \\'due-process\\', \\'indian-constitutional-law\\', \\'accident\\', \\'salary\\', \\'child-abuse\\', \\'dual-citizenship\\', \\'hipaa\\', \\'prosecutorial-discretion\\', \\'real-property\\', \\'forgery\\', \\'subpoena\\', \\'antitrust-law\\', \\'breaking-the-lease\\', \\'prosecution\\', \\'unlicensed-practice\\', \\'eviction\\', \\'class-action\\', \\'prison\\', \\'digital-contract\\', \\'child-support\\', \\'homicide\\', \\'judicial-review\\', \\'administrative-law\\', \\'research\\', \\'slander\\', \\'borders\\', \\'health-insurance\\', \\'investigation\\', \\'impeachment\\', \\'corporate-tax-laws\\', \\'right-of-publicity\\', \\'criminal-record\\', \\'pro-se\\', \\'google\\', \\'fair-housing-acts\\', \\'service\\', \\'foia\\', \\'us-president\\', \\'manslaughter\\', \\'loan\\', \\'legal-citation\\', \\'sec\\', \\'multiple-jurisdiction\\', \\'safety\\', \\'russia\\', \\'money-laundering\\', \\'compensation\\', \\'translation\\', \\'mortgage\\', \\'law-of-the-sea\\', \\'computer-misuse-act\\', \\'equity\\', \\'punishment\\', \\'spain\\', \\'sex-discrimination\\', \\'age\\', \\'legal-representation\\', \\'austria\\', \\'scotland\\', \\'second-amendment\\', \\'gift\\', \\'transportation\\', \\'judging\\', \\'water\\', \\'artificial-intelligence\\', \\'japan\\', \\'federal-tax-law\\', \\'law-enforcement\\', \\'pets\\', \\'claims\\', \\'investment\\', \\'authorship\\', \\'plea\\', \\'legal-education\\', \\'missouri\\', \\'us-senate\\', \\'jurisprudence\\', \\'china\\', \\'conflict-of-interest\\', \\'bribery\\', \\'postal-service\\', \\'customer-service\\', \\'attorney-fees\\', \\'restaurants\\', \\'customs-law\\', \\'environmental\\', \\'battery\\', \\'estate-planning\\', \\'attribution\\', \\'burden-of-proof\\', \\'fourteenth-amendment\\', \\'italy\\', \\'easement\\', \\'sweden\\', \\'cryptography\\', \\'product-liability\\', \\'warrant\\', \\'sixth-amendment\\', \\'medical-marijuana\\', \\'public-records\\', \\'sport\\', \\'equal-protection\\', \\'suicide\\', \\'sublease\\', \\'history\\', \\'new-hampshire\\', \\'executive\\', \\'reference-request\\', \\'phonecall\\', \\'religious-law\\', \\'parliamentary-procedure\\', \\'ccpa\\', \\'irs\\', \\'bills\\', \\'civil-judgment\\', \\'shipping\\', \\'unions\\', \\'cookies\\', \\'public-law\\', \\'statutory-construction\\', \\'extraterritorial-jurisdiction\\', \\'blackmail\\', \\'naturalization\\', \\'brexit\\', \\'charity\\', \\'aviation\\', \\'local-authorities\\', \\'standing\\', \\'contempt-of-court\\', \\'costs\\', \\'minnesota\\', \\'state-university\\', \\'indonesia\\', \\'malpractice\\', \\'residency\\', \\'adhesion-contracts\\', \\'vaccination\\', \\'barristers\\', \\'separation-of-powers\\', \\'income-tax\\', \\'tennessee\\', \\'deposit\\', \\'extortion\\', \\'notaries\\', \\'collections\\', \\'self-incrimination\\', \\'sale-of-goods\\', \\'injunctions\\', \\'radio\\', \\'emergencies\\', \\'fines\\', \\'conflict-of-laws\\', \\'car\\', \\'advice\\', \\'delaware\\', \\'phone\\', \\'jury-nullification\\', \\'noise\\', \\'new-mexico\\', \\'partnership\\', \\'sovereignty\\', \\'video-games\\', \\'crime\\', \\'birth-certificate\\', \\'binding\\', \\'alberta\\', \\'refugee-law\\', \\'governmental-liability\\', \\'south-africa\\', \\'capital-punishment\\', \\'insider-trading\\', \\'trees\\', \\'theory-of-law\\', \\'injury\\', \\'court-order\\', \\'media\\', \\'contractor\\', \\'waivers\\', \\'construction-law\\', \\'legislative-intent\\', \\'database\\', \\'transgender\\', \\'texas-sb8\\', \\'license-plates\\', \\'indiana\\', \\'protest\\', \\'journalism\\', \\'kentucky\\', \\'enforcing-judgment\\', \\'legal-tender\\', \\'government\\', \\'municipal-bylaws\\', \\'fbi\\', \\'americans-with-disabilities-act\\', \\'child-porn\\', \\'accounting\\', \\'freedom-of-religion\\', \\'freedom-of-information\\', \\'official-immunity\\', \\'work-for-hire\\', \\'connecticut\\', \\'dog\\', \\'mexico\\', \\'trade-name\\', \\'students\\', \\'fcc\\', \\'false-advertising\\', \\'prostitution\\', \\'coppa\\', \\'asylum\\', \\'government-officers\\', \\'diplomatic-immunity\\', \\'wiretapping\\', \\'export\\', \\'restraining-orders\\', \\'guardianship\\', \\'bail\\', \\'kidnapping\\', \\'succession\\', \\'belgium\\', \\'notice\\', \\'lotteries\\', \\'secret\\', \\'spam\\', \\'hardware\\', \\'ferpa\\', \\'local-ordinance\\', \\'fda\\', \\'immunity\\', \\'remedies\\', \\'business-lease\\', \\'slavery\\', \\'never-talk-to-police\\', \\'digital-forensics\\', \\'choice-of-law\\', \\'fiduciary-duties\\', \\'drones\\', \\'admissibility\\', \\'family\\', \\'miranda-warning\\', \\'identity-theft\\', \\'foreign-relations\\', \\'hong-kong\\', \\'money-service\\', \\'detention\\', \\'agency\\', \\'innocent-infringement\\', \\'misconduct\\', \\'citizens-arrest\\', \\'disclosure\\', \\'unemployment\\', \\'assured-shorthold-tenancy\\', \\'arkansas\\', \\'alabama\\', \\'towing\\', \\'use-of-force\\', \\'israel\\', \\'computer\\', \\'schengen\\', \\'interrogation\\', \\'south-carolina\\', \\'nationality\\', \\'cross-examination\\', \\'communications-law\\', \\'cfaa\\', \\'ethical-hacking\\', \\'juvenile-law\\', \\'quebec\\', \\'entertainment-law\\', \\'indictment\\', \\'distribution-terms\\', \\'grand-jury\\', \\'credit-report\\', \\'demand-letter\\', \\'mens-rea\\', \\'car-insurance\\', \\'us-house-of-reps\\', \\'repossession\\', \\'copyfraud\\', \\'bill-of-rights\\', \\'name-title\\', \\'equality-act-2010\\', \\'space\\', \\'invoice\\', \\'ukraine\\', \\'vandalism\\', \\'frivolous-arguments\\', \\'political\\', \\'penalty\\', \\'liens\\', \\'public-office\\', \\'hearsay\\', \\'kansas\\', \\'treason\\', \\'natural-person\\', \\'illegal-immigration\\', \\'norway\\', \\'lemon-law\\', \\'parody\\', \\'from-the-latin\\', \\'labor\\', \\'mediation\\', \\'conversion\\', \\'texting\\', \\'interstate-travel\\', \\'whistleblower\\', \\'damage-mitigation\\', \\'judicial-immunity\\', \\'terrorism\\', \\'surveillance\\', \\'subject-access-request\\', \\'receipts\\', \\'private-property\\', \\'voir-dire\\', \\'obstruction\\', \\'legal-fees\\', \\'food-processing\\', \\'f1-visa\\', \\'common-property\\', \\'custody\\', \\'duty-of-care\\', \\'police-power\\', \\'louisiana\\', \\'community-property\\', \\'entrapment\\', \\'singapore\\', \\'census\\', \\'brazil\\', \\'penal-law\\', \\'uae\\', \\'portugal\\', \\'philippines\\', \\'indemnity\\', \\'trespassing\\', \\'expungement\\', \\'billing\\', \\'solicitor\\', \\'idaho\\', \\'telemarketing\\', \\'employee\\', \\'data-protection-act\\', \\'advocacy\\', \\'trade-secret\\', \\'government-contracts\\', \\'adverse-possession\\', \\'ucc\\', \\'agriculture\\', \\'judgment\\', \\'iowa\\', \\'social-security\\', \\'european-court-of-human-rights\\', \\'poland\\', \\'abuse\\', \\'tenth-amendment\\', \\'car-title\\', \\'motion\\', \\'west-virginia\\', \\'denmark\\', \\'digital-content\\', \\'eminent-domain\\', \\'deposition\\', \\'bench-trial\\', \\'monarchy\\', \\'finland\\', \\'parliamentary-law\\', \\'nuisance\\', \\'writ-of-mandamus\\', \\'recusal\\', \\'public-health\\', \\'automotive\\', \\'barter\\', \\'credit-card\\', \\'amateur-radio\\', \\'pacer\\', \\'anonymization\\', \\'charge-dismissed\\', \\'qualified-immunity\\', \\'vehicle-registration\\', \\'legislative-power\\', \\'statement\\', \\'argentina\\', \\'conspiracy\\', \\'rhode-island\\', \\'pandemic\\', \\'loan-guarantee\\', \\'restitution\\', \\'stalking\\', \\'executor\\', \\'undercover\\', \\'mississippi\\', \\'secession\\', \\'adoption\\', \\'good-samaritan\\', \\'victoria\\', \\'cybersquatting\\', \\'title\\', \\'freelance\\', \\'acquittal\\', \\'pleadings\\', \\'false-arrest\\', \\'defense\\', \\'checks\\', \\'ex-post-facto\\', \\'import\\', \\'bfoq\\', \\'acknowledgement\\', \\'tfeu\\', \\'alternative-dispute-resolution\\', \\'stateless\\', \\'juridical-person\\', \\'autonomous\\', \\'constitutional-rights\\', \\'refund\\', \\'right-to-counsel\\', \\'email-marketing\\', \\'warranty\\', \\'uk-supreme-court\\', \\'freedom-of-press\\', \\'sweepstakes\\', \\'foreclosure\\', \\'business-expense\\', \\'counterfeiting\\', \\'escrow\\', \\'civil-damages\\', \\'tenancy\\', \\'turkey\\', \\'saudi-arabia\\', \\'unfair-terms\\', \\'hungary\\', \\'equitable-relief\\', \\'district-of-columbia\\', \\'bar-association\\', \\'paperwork\\', \\'eu-law\\', \\'eighth-amendment\\', \\'green-card\\', \\'sole-propietorship\\', \\'insanity\\', \\'unjust-enrichment\\', \\'appearance\\', \\'punitive-damages\\', \\'subsidiaries\\', \\'school-law\\', \\'retail\\', \\'oklahoma\\', \\'borrowing\\', \\'cftc\\', \\'fair-dealing\\', \\'statistics\\', \\'vermont\\', \\'czech-republic\\', \\'incitement\\', \\'pakistan\\', \\'robot\\', \\'natural-law\\', \\'human-resources\\', \\'burglary\\', \\'campaign-finance\\', \\'northern-ireland\\', \\'vice-president\\', \\'preemption\\', \\'united-nations\\', \\'moral-duty\\', \\'accident-insurance\\', \\'iran\\', \\'wrongful-death\\', \\'opca\\', \\'time-periods\\', \\'war\\', \\'attainder\\', \\'censorship\\', \\'legislation\\', \\'hawaii\\', \\'community-law\\', \\'magna-carta\\', \\'criminal-damage\\', \\'work-permit\\', \\'internet-scam\\', \\'government-benefits\\', \\'prohibition\\', \\'blockchain\\', \\'puerto-rico\\', \\'authority\\', \\'consideration\\', \\'montana\\', \\'condominium\\', \\'nebraska\\', \\'non-citizen\\', \\'return\\', \\'morality\\', \\'tribunal\\', \\'attorney-client-relationship\\', \\'united-nations-charter\\', \\'campaign\\', \\'wrongful-conviction\\', \\'vatican-city-state\\', \\'right-to-repair\\', \\'us-states\\', \\'cctv\\', \\'city-of-london\\', \\'law-review\\', \\'itar\\', \\'medicaid\\', \\'public-order\\', \\'egypt\\', \\'mineral-rights\\', \\'jan-6\\', \\'watershed\\', \\'parliament\\', \\'maine\\', \\'bar-admission\\', \\'can-spam-act-of-2003\\', \\'threats\\', \\'state-power\\', \\'geneva-convention\\', \\'vaccine\\', \\'malaysia\\', \\'res-judicata\\', \\'prenuptial-agreement\\', \\'fingerprint\\', \\'motorcycle\\', \\'summary-judgment\\', \\'legislative-drafting\\', \\'adult\\', \\'zoning\\', \\'munition\\', \\'textualism\\', \\'treelaw\\', \\'fs718\\', \\'azerbaijan\\', \\'tribunal-procedure\\', \\'force-majeure\\', \\'council-tax\\', \\'municipal-law\\', \\'reasonableness\\', \\'dmv\\', \\'negotiations\\', \\'statutory-license\\', \\'prima-facie-speed-limits\\', \\'reasonable\\', \\'hidden\\', \\'croatia\\', \\'romania\\', \\'deception\\', \\'lease-renewal\\', \\'kenya\\', \\'electronic-procedure\\', \\'public-relations\\', \\'smart-contracts\\', \\'spousal-immunity\\', \\'civil-penalty\\', \\'lebanon\\', \\'trolley-problem\\', \\'blasphemy\\', \\'service-of-process\\', \\'wrongful-dismissal\\', \\'contingency-fee\\', \\'shell-companies\\', \\'mutual-mistake\\', \\'electoral-commission\\', \\'breaking-and-entering\\', \\'iraq\\', \\'thailand\\', \\'prior-art\\', \\'continental-law\\', \\'jury-instruction\\', \\'theft-act\\', \\'justification\\', \\'taiwan\\', \\'international-court-of-justice\\', \\'clean-hands\\', \\'flag\\', \\'public-transport\\', \\'games\\', \\'osha\\', \\'enforcement\\', \\'freedom-of-assembly\\', \\'defense-of-others\\', \\'hunting\\', \\'european-council\\', \\'civil-claims\\', \\'9th-circuit\\', \\'proxy\\', \\'knowledge\\', \\'crown-court\\', \\'biosafety\\', \\'faa\\', \\'european-convention-on-human-rights\\', \\'drm\\', \\'arraignment\\', \\'standard-of-review\\', \\'f2-visa\\', \\'shoplifting\\', \\'ecommerce\\', \\'share-alike\\', \\'marketing\\', \\'strike\\', \\'attempt\\', \\'steve-bannon\\', \\'hate-crime\\', \\'possession\\', \\'prejudice\\', \\'seventh-amendment\\', \\'privy-council\\', \\'houston\\', \\'in-rem\\', \\'espionage\\', \\'legal-aid\\', \\'garnishment\\', \\'professional-licensing\\', \\'oath\\', \\'contributory-negligence\\', \\'domestic-abuse\\', \\'admission-of-new-states\\', \\'great-britain\\', \\'cyprus\\', \\'sri-lanka\\', \\'slovakia\\', \\'strata-property\\', \\'dismissal\\', \\'parole\\', \\'unconscionability\\', \\'the-hague\\', \\'human-trafficking\\', \\'open-carry\\', \\'current-events\\', \\'commerce-clause\\', \\'county-court\\', \\'rights-of-audience\\', \\'remedy\\', \\'epo\\', \\'new-south-wales\\', \\'utility-bills\\', \\'bylaw\\', \\'suggestions\\', \\'cannabis\\', \\'tcpa\\', \\'cease-and-desist\\', \\'lines-of-case-law\\', \\'interim-possession-order\\', \\'taa\\', \\'meet-bob\\', \\'enforceability\\', \\'mercenary\\', \\'courtroom-strategy\\', \\'full-faith-and-credit\\', \\'national-security\\', \\'bangladesh\\', \\'victims-rights\\', \\'social-security-card\\', \\'north-korea\\', \\'squatting\\', \\'third-amendment\\', \\'section-230\\', \\'roberts-rules-of-order\\', \\'presumption-of-innocence\\', \\'emotional-distress\\', \\'legal-doctrine\\', \\'directive\\', \\'silks\\', \\'amendment\\', \\'european-court-of-justice\\', \\'litigant-in-person\\', \\'prima-facie\\', \\'commercial-lease\\', \\'perimeter-search\\', \\'legal-advice\\', \\'first-impression\\', \\'disrepair\\', \\'self-defence\\', \\'recap\\', \\'vietnam\\', \\'free-law-project\\', \\'hmrc\\', \\'un-security-council\\', \\'sui-juris\\', \\'data-retention\\', \\'ohrc\\', \\'myanmar\\', \\'swatting\\', \\'living\\', \\'dhs\\', \\'distributed-ledger-technology\\', \\'ethereum\\', \\'aboriginal-law\\', \\'msa\\', \\'declaration-of-incompatibility\\', \\'venezuela\\', \\'iceland\\', \\'alimony\\', \\'counterterrorism\\', \\'belarus\\', \\'library\\', \\'tonga\\', \\'bigamy\\', \\'split-estate\\', \\'fare-evasion\\', \\'last-generation\\', \\'suppression\\', \\'marital-law\\', \\'local-authority\\', \\'ecb\\', \\'case-reports\\', \\'legal-person\\', \\'building-code\\', \\'cloudact\\', \\'initiative\\', \\'coast\\', \\'interpol\\', \\'bailiff\\', \\'tfl\\', \\'notation\\', \\'panhandling\\', \\'lithuania\\', \\'abolishment\\', \\'2nd-amendment\\', \\'caste\\', \\'death-certificate\\', \\'circuit-splits\\', \\'admiralty\\', \\'knowing\\', \\'inchoate-offences\\', \\'stay-order\\', \\'pre-action-protocol\\', \\'nigeria\\', \\'teenager\\', \\'energy\\', \\'nobility-clause\\', \\'euro\\', \\'first-instance\\', \\'vexatious-litigants\\', \\'calculation\\', \\'twenty-seventh-amendment\\', \\'adultery\\', \\'intent\\', \\'house-of-lords\\', \\'robbery\\', \\'usury\\', \\'toyota\\', \\'rico\\', \\'sentence\\', \\'code-of-federal-regulations\\', \\'court-forms\\', \\'life-sentence\\', \\'nominal-damages\\', \\'slayer-rule\\', \\'naming-conventions\\', \\'civil-right-act\\', \\'peaceful-enjoyment\\', \\'finan\\', \\'commonwealth\\', \\'inquisitorial-system\\', \\'adversarial-system\\', \\'non-fungible-token\\', \\'inquisitorial\\', \\'converture\\', \\'public-accommodations\\', \\'common-scold\\', \\'k9\\']\\n^^^^^\\n\\n\\n>>>>>\\nTitle: Can defendants arraigned in federal court sometimes be \"out on bail\" secretly with no way for the public to know about or verify the bail?\\n>>>>>\\n\\n\\n+++++\\nQuestion: In this answer to my Politics SE question *Is former president <PERSON> \"out on bail\" as <PERSON> asserts? If so, were campaign funds used? which ends: Bail is a particular type of bond in which the defendant submits an upfront payment that will be held until he returns to court, but there\\'s no indication <PERSON> was asked to post bail. there is a discussion about what \"but there\\'s no indication\" means, including: Isn\\'t posting bail generally a matter of public knowledge? We often hear about person X was released wrt case Y for Z amount of money. Which would bolster this answer. and Generally yes, it\\'s disclosed to the public. I just don\\'t know if it\\'s legally required to be disclosed, or whether the judge has discretion. <PERSON> is weird... Chuckles that Politics SE would think <PERSON> is weird aside, this has piqued my curiosity. <PERSON> is a former US attorney with extensive experience in arraignments for corruption and similar crimes, and would be keenly aware of the difference between bond and bail. Further, the \"out on bail\" statement was made on national television amidst a discussion on truth and honesty in politics. And yet I can\\'t tell if the \"out on bail\" assertion is true, false, or currently unknowable with any certainty. Question: Can defendants arraigned in federal court sometimes be \"out on bail\" secretly with no way for the public to know about or verify the bail?\\n+++++\\n\\n',\n",
       " 'output': 'arraignment,arrest,bail,federal-courts,politics'}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06395e78",
   "metadata": {},
   "source": [
    "# Data processing for the submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "82807e35-71b8-4490-877b-31f2d059a157",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 1 files\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5000/5000 [00:00<00:00, 61458.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing the formatted submission dataset to disk...\n",
      "Writing to disk complete for 1 partitions\n"
     ]
    }
   ],
   "source": [
    "# SUBMISSION DATA PREPARATION\n",
    "output_dir = f\"{DATA_DIR}/split\"\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "\n",
    "submission_ds = \"data/submission/evaluation-dataset-verified-for-participants.jsonl\"\n",
    "assert os.path.exists(submission_ds), f\"The submission dataset does not exist at '{submission_ds}'\"\n",
    "submission_ds = DocumentDataset.read_json(submission_ds)\n",
    "submission_ds = format_dataset(submission_ds, \"submission.jsonl\", tag_desc, SYSTEM_PROMPT)\n",
    "print(\"Writing the formatted submission dataset to disk...\")\n",
    "submission_ds.to_json(output_dir, write_to_filename=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "395bc297-35b5-4109-8c77-ab7965d85b44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "file_path = f\"{DATA_DIR}/split/submission.jsonl\"\n",
    "import json\n",
    "data = []\n",
    "\n",
    "with open(file_path, 'r') as file:\n",
    "    for line in file:\n",
    "        data.append(json.loads(line))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6cdbbfe3-16af-4166-afa7-0c38b870a8f0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'filename': 'submission.jsonl',\n",
       " 'input': \"\\nYou are an expert in Law and also in tagging legal questions.\\nYou are provided with a question enclosed in +++++ and it's corresponding title enclosed in >>>>> from the law domain.\\n\\nYou are also provided with a list of all the tags, enclosed in ^^^^^.\\n\\nYour task is to:\\ni. Understand the question and it's title.\\nii. Pick up the tags that are most appropriate and relevant to the question, strictly from the tags provided to you.\\niii. Make sure you return the tags alone without their description.\\n\\n\\n```\\nNOTES: All tags must be in lowercase, ordered lexicographically and separated by commas.\\n```\\n\\nYour output should be a JSON with the below format:\\n```\\ntags : <Put your relevant tags here>\\n```\\n\\n\\n>>>>>\\nTitle: Navigating UK Residency Requirements for Foreign Nationals\\n>>>>>\\n\\n\\n+++++\\nQuestion: What are the implications of switching from a work-sponsored visa to a family-based visa on the path to obtaining permanent residency in the UK, particularly with regards to meeting the residency requirements and potential tax implications?\\n+++++\\n\\n\",\n",
       " 'output': ''}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[678]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "eef816dc-4641-4b23-a4a4-03aaf6e88257",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client.close()\n",
    "exit(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8651cbbe-a43b-4ffa-b620-3e671c118b58",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}