2022-hw4-strong.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C_jdZ5vHJ4A9",
    "papermill": {
     "duration": 0.017173,
     "end_time": "2022-03-27T14:28:37.328547",
     "exception": false,
     "start_time": "2022-03-27T14:28:37.311374",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Task description\n",
    "- Classify the speakers of given features.\n",
    "- Main goal: Learn how to use transformer.\n",
    "- Baselines:\n",
    "  - Easy: Run sample code and know how to use transformer.\n",
    "  - Medium: Know how to adjust parameters of transformer.\n",
    "  - Strong: Construct [conformer](https://arxiv.org/abs/2005.08100) which is a variety of transformer. \n",
    "  - Boss: Implement [Self-Attention Pooling](https://arxiv.org/pdf/2008.01077v1.pdf) & [Additive Margin Softmax](https://arxiv.org/pdf/1801.05599.pdf) to further boost the performance.\n",
    "\n",
    "- Other links\n",
    "  - Kaggle: [link](https://www.kaggle.com/t/ac77388c90204a4c8daebeddd40ff916)\n",
    "  - Slide: [link](https://docs.google.com/presentation/d/1HLAj7UUIjZOycDe7DaVLSwJfXVd3bXPOyzSb6Zk3hYU/edit?usp=sharing)\n",
    "  - Data: [link](https://github.com/MachineLearningHW/ML_HW4_Dataset)\n",
    "\n",
    "# Download dataset\n",
    "- Data is [here](https://github.com/MachineLearningHW/ML_HW4_Dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LhLNWB-AK2Z5",
    "papermill": {
     "duration": 0.015649,
     "end_time": "2022-03-27T14:28:37.362058",
     "exception": false,
     "start_time": "2022-03-27T14:28:37.346409",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "\"\"\"\n",
    "If the links below become inaccessible, please connect TAs.\n",
    "\"\"\"\n",
    "\n",
    "!wget https://github.com/MachineLearningHW/ML_HW4_Dataset/raw/0.0.1/Dataset.tar.gz.partaa\n",
    "!wget https://github.com/MachineLearningHW/ML_HW4_Dataset/raw/0.0.1/Dataset.tar.gz.partab\n",
    "!wget https://github.com/MachineLearningHW/ML_HW4_Dataset/raw/0.0.1/Dataset.tar.gz.partac\n",
    "!wget https://github.com/MachineLearningHW/ML_HW4_Dataset/raw/0.0.1/Dataset.tar.gz.partad\n",
    "\n",
    "!cat Dataset.tar.gz.parta* > Dataset.tar.gz\n",
    "\n",
    "!tar zxvf Dataset.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ENWVAUDVJtVY",
    "papermill": {
     "duration": 0.0157,
     "end_time": "2022-03-27T14:28:37.393691",
     "exception": false,
     "start_time": "2022-03-27T14:28:37.377991",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Fix Random Seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:37.436940Z",
     "iopub.status.busy": "2022-03-27T14:28:37.434025Z",
     "iopub.status.idle": "2022-03-27T14:28:38.940962Z",
     "shell.execute_reply": "2022-03-27T14:28:38.940324Z",
     "shell.execute_reply.started": "2022-03-27T13:26:27.195440Z"
    },
    "id": "E6burzCXIyuA",
    "papermill": {
     "duration": 1.531288,
     "end_time": "2022-03-27T14:28:38.941097",
     "exception": false,
     "start_time": "2022-03-27T14:28:37.409809",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import random\n",
    "\n",
    "def set_seed(seed):\n",
    "    np.random.seed(seed)\n",
    "    random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if torch.cuda.is_available():\n",
    "        torch.cuda.manual_seed(seed)\n",
    "        torch.cuda.manual_seed_all(seed)\n",
    "    torch.backends.cudnn.benchmark = False\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "\n",
    "set_seed(87)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k7dVbxW2LASN",
    "papermill": {
     "duration": 0.015927,
     "end_time": "2022-03-27T14:28:38.973462",
     "exception": false,
     "start_time": "2022-03-27T14:28:38.957535",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Data\n",
    "\n",
    "## Dataset\n",
    "- Original dataset is [Voxceleb2](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html).\n",
    "- The [license](https://creativecommons.org/licenses/by/4.0/) and [complete version](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/files/license.txt) of Voxceleb2.\n",
    "- We randomly select 600 speakers from Voxceleb2.\n",
    "- Then preprocess the raw waveforms into mel-spectrograms.\n",
    "\n",
    "- Args:\n",
    "  - data_dir: The path to the data directory.\n",
    "  - metadata_path: The path to the metadata.\n",
    "  - segment_len: The length of audio segment for training. \n",
    "- The architecture of data directory \\\\\n",
    "  - data directory \\\\\n",
    "  |---- metadata.json \\\\\n",
    "  |---- testdata.json \\\\\n",
    "  |---- mapping.json \\\\\n",
    "  |---- uttr-{random string}.pt \\\\\n",
    "\n",
    "- The information in metadata\n",
    "  - \"n_mels\": The dimention of mel-spectrogram.\n",
    "  - \"speakers\": A dictionary. \n",
    "    - Key: speaker ids.\n",
    "    - value: \"feature_path\" and \"mel_len\"\n",
    "\n",
    "\n",
    "For efficiency, we segment the mel-spectrograms into segments in the traing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:39.016822Z",
     "iopub.status.busy": "2022-03-27T14:28:39.011395Z",
     "iopub.status.idle": "2022-03-27T14:28:39.018639Z",
     "shell.execute_reply": "2022-03-27T14:28:39.019028Z",
     "shell.execute_reply.started": "2022-03-27T13:26:28.646447Z"
    },
    "id": "KpuGxl4CI2pr",
    "papermill": {
     "duration": 0.030045,
     "end_time": "2022-03-27T14:28:39.019146",
     "exception": false,
     "start_time": "2022-03-27T14:28:38.989101",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import torch\n",
    "import random\n",
    "from pathlib import Path\n",
    "from torch.utils.data import Dataset\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    " \n",
    " \n",
    "class myDataset(Dataset):\n",
    "\tdef __init__(self, data_dir, segment_len=128):\n",
    "\t\tself.data_dir = data_dir\n",
    "\t\tself.segment_len = segment_len\n",
    "\t\n",
    "\t\t# Load the mapping from speaker neme to their corresponding id. \n",
    "\t\tmapping_path = Path(data_dir) / \"mapping.json\"\n",
    "\t\tmapping = json.load(mapping_path.open())\n",
    "\t\tself.speaker2id = mapping[\"speaker2id\"]\n",
    "\t\n",
    "\t\t# Load metadata of training data.\n",
    "\t\tmetadata_path = Path(data_dir) / \"metadata.json\"\n",
    "\t\tmetadata = json.load(open(metadata_path))[\"speakers\"]\n",
    "\t\n",
    "\t\t# Get the total number of speaker.\n",
    "\t\tself.speaker_num = len(metadata.keys())\n",
    "\t\tself.data = []\n",
    "\t\tfor speaker in metadata.keys():\n",
    "\t\t\tfor utterances in metadata[speaker]:\n",
    "\t\t\t\tself.data.append([utterances[\"feature_path\"], self.speaker2id[speaker]])\n",
    " \n",
    "\tdef __len__(self):\n",
    "\t\t\treturn len(self.data)\n",
    " \n",
    "\tdef __getitem__(self, index):\n",
    "\t\tfeat_path, speaker = self.data[index]\n",
    "\t\t# Load preprocessed mel-spectrogram.\n",
    "\t\tmel = torch.load(os.path.join(self.data_dir, feat_path))\n",
    "\n",
    "\t\t# Segmemt mel-spectrogram into \"segment_len\" frames.\n",
    "\t\tif len(mel) > self.segment_len:\n",
    "\t\t\t# Randomly get the starting point of the segment.\n",
    "\t\t\tstart = random.randint(0, len(mel) - self.segment_len)\n",
    "\t\t\t# Get a segment with \"segment_len\" frames.\n",
    "\t\t\tmel = torch.FloatTensor(mel[start:start+self.segment_len])\n",
    "\t\telse:\n",
    "\t\t\tmel = torch.FloatTensor(mel)\n",
    "\t\t# Turn the speaker id into long for computing loss later.\n",
    "\t\tspeaker = torch.FloatTensor([speaker]).long()\n",
    "\t\treturn mel, speaker\n",
    " \n",
    "\tdef get_speaker_number(self):\n",
    "\t\treturn self.speaker_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "668hverTMlGN",
    "papermill": {
     "duration": 0.01546,
     "end_time": "2022-03-27T14:28:39.050920",
     "exception": false,
     "start_time": "2022-03-27T14:28:39.035460",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Dataloader\n",
    "- Split dataset into training dataset(90%) and validation dataset(10%).\n",
    "- Create dataloader to iterate the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:39.091501Z",
     "iopub.status.busy": "2022-03-27T14:28:39.090731Z",
     "iopub.status.idle": "2022-03-27T14:28:39.093358Z",
     "shell.execute_reply": "2022-03-27T14:28:39.092887Z",
     "shell.execute_reply.started": "2022-03-27T13:26:28.659383Z"
    },
    "id": "B7c2gZYoJDRS",
    "papermill": {
     "duration": 0.026515,
     "end_time": "2022-03-27T14:28:39.093452",
     "exception": false,
     "start_time": "2022-03-27T14:28:39.066937",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader, random_split\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "\n",
    "\n",
    "def collate_batch(batch):\n",
    "\t# Process features within a batch.\n",
    "\t\"\"\"Collate a batch of data.\"\"\"\n",
    "\tmel, speaker = zip(*batch)\n",
    "\t# Because we train the model batch by batch, we need to pad the features in the same batch to make their lengths the same.\n",
    "\tmel = pad_sequence(mel, batch_first=True, padding_value=-20)    # pad log 10^(-20) which is very small value.\n",
    "\t# mel: (batch size, length, 40)\n",
    "\treturn mel, torch.FloatTensor(speaker).long()\n",
    "\n",
    "\n",
    "def get_dataloader(data_dir, batch_size, n_workers):\n",
    "\t\"\"\"Generate dataloader\"\"\"\n",
    "\tdataset = myDataset(data_dir)\n",
    "\tspeaker_num = dataset.get_speaker_number()\n",
    "\t# Split dataset into training dataset and validation dataset\n",
    "\ttrainlen = int(0.9 * len(dataset))\n",
    "\tlengths = [trainlen, len(dataset) - trainlen]\n",
    "\ttrainset, validset = random_split(dataset, lengths)\n",
    "\n",
    "\ttrain_loader = DataLoader(\n",
    "\t\ttrainset,\n",
    "\t\tbatch_size=batch_size,\n",
    "\t\tshuffle=True,\n",
    "\t\tdrop_last=True,\n",
    "\t\tnum_workers=n_workers,\n",
    "\t\tpin_memory=True,\n",
    "\t\tcollate_fn=collate_batch,\n",
    "\t)\n",
    "\tvalid_loader = DataLoader(\n",
    "\t\tvalidset,\n",
    "\t\tbatch_size=batch_size,\n",
    "\t\tnum_workers=n_workers,\n",
    "\t\tdrop_last=True,\n",
    "\t\tpin_memory=True,\n",
    "\t\tcollate_fn=collate_batch,\n",
    "\t)\n",
    "\n",
    "\treturn train_loader, valid_loader, speaker_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5FOSZYxrMqhc",
    "papermill": {
     "duration": 0.015804,
     "end_time": "2022-03-27T14:28:39.127248",
     "exception": false,
     "start_time": "2022-03-27T14:28:39.111444",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Model\n",
    "- TransformerEncoderLayer:\n",
    "  - Base transformer encoder layer in [Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n",
    "  - Parameters:\n",
    "    - d_model: the number of expected features of the input (required).\n",
    "\n",
    "    - nhead: the number of heads of the multiheadattention models (required).\n",
    "\n",
    "    - dim_feedforward: the dimension of the feedforward network model (default=2048).\n",
    "\n",
    "    - dropout: the dropout value (default=0.1).\n",
    "\n",
    "    - activation: the activation function of intermediate layer, relu or gelu (default=relu).\n",
    "\n",
    "- TransformerEncoder:\n",
    "  - TransformerEncoder is a stack of N transformer encoder layers\n",
    "  - Parameters:\n",
    "    - encoder_layer: an instance of the TransformerEncoderLayer() class (required).\n",
    "\n",
    "    - num_layers: the number of sub-encoder-layers in the encoder (required).\n",
    "\n",
    "    - norm: the layer normalization component (optional)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:39.172063Z",
     "iopub.status.busy": "2022-03-27T14:28:39.171467Z",
     "iopub.status.idle": "2022-03-27T14:28:49.162334Z",
     "shell.execute_reply": "2022-03-27T14:28:49.161864Z",
     "shell.execute_reply.started": "2022-03-27T13:26:28.673586Z"
    },
    "id": "iXZ5B0EKJGs8",
    "papermill": {
     "duration": 10.01904,
     "end_time": "2022-03-27T14:28:49.162464",
     "exception": false,
     "start_time": "2022-03-27T14:28:39.143424",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting conformer\r\n",
      "  Downloading conformer-0.2.5-py3-none-any.whl (4.1 kB)\r\n",
      "Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from conformer) (1.9.1)\r\n",
      "Collecting einops\r\n",
      "  Downloading einops-0.4.1-py3-none-any.whl (28 kB)\r\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->conformer) (4.1.1)\r\n",
      "Installing collected packages: einops, conformer\r\n",
      "Successfully installed conformer-0.2.5 einops-0.4.1\r\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "!pip install conformer\n",
    "from conformer import ConformerBlock\n",
    "\n",
    "class Classifier(nn.Module):\n",
    "\tdef __init__(self, d_model=224, n_spks=600, dropout=0.25):\n",
    "\t\tsuper().__init__()\n",
    "\t\t# Project the dimension of features from that of input into d_model.\n",
    "\t\tself.prenet = nn.Linear(40, d_model)\n",
    "\t\t# TODO:\n",
    "\t\t#   Change Transformer to Conformer.\n",
    "\t\t#   https://arxiv.org/abs/2005.08100\n",
    "\t\t#self.encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, dim_feedforward=d_model*2, nhead=2, dropout=dropout)\n",
    "\t\t#self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=3)\n",
    "\t\tself.encoder = ConformerBlock(\n",
    "\t\t\t\tdim = d_model,\n",
    "\t\t\t\tdim_head = 4,\n",
    "\t\t\t\theads = 4,\n",
    "\t\t\t\tff_mult = 4,\n",
    "\t\t\t\tconv_expansion_factor = 2,\n",
    "\t\t\t\tconv_kernel_size = 20,\n",
    "\t\t\t\tattn_dropout = dropout,\n",
    "\t\t\t\tff_dropout = dropout,\n",
    "\t\t\t\tconv_dropout = dropout,\n",
    "\t\t)\n",
    "\n",
    "\n",
    "\t\t# Project the the dimension of features from d_model into speaker nums.\n",
    "\t\tself.pred_layer = nn.Sequential(\n",
    "\t\t\tnn.BatchNorm1d(d_model),\n",
    "\t\t\t#nn.Linear(d_model, d_model),\n",
    "\t\t\t#nn.ReLU(),\n",
    "\t\t\tnn.Linear(d_model, n_spks),\n",
    "\t\t)\n",
    "\n",
    "\tdef forward(self, mels):\n",
    "\t\t\"\"\"\n",
    "\t\targs:\n",
    "\t\t\tmels: (batch size, length, 40)\n",
    "\t\treturn:\n",
    "\t\t\tout: (batch size, n_spks)\n",
    "\t\t\"\"\"\n",
    "\t\t# out: (batch size, length, d_model)\n",
    "\t\tout = self.prenet(mels)\n",
    "\t\t# out: (length, batch size, d_model)\n",
    "\t\tout = out.permute(1, 0, 2)\n",
    "\t\t# The encoder layer expect features in the shape of (length, batch size, d_model).\n",
    "\t\tout = self.encoder(out)\n",
    "\t\t# out: (batch size, length, d_model)\n",
    "\t\tout = out.transpose(0, 1) \n",
    "\t\t# mean pooling\n",
    "\t\tstats = out.mean(dim=1)\n",
    "\n",
    "\t\t# out: (batch, n_spks)\n",
    "\t\tout = self.pred_layer(stats)\n",
    "\t\treturn out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W7yX8JinM5Ly",
    "papermill": {
     "duration": 0.018016,
     "end_time": "2022-03-27T14:28:49.199579",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.181563",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Learning rate schedule\n",
    "- For transformer architecture, the design of learning rate schedule is different from that of CNN.\n",
    "- Previous works show that the warmup of learning rate is useful for training models with transformer architectures.\n",
    "- The warmup schedule\n",
    "  - Set learning rate to 0 in the beginning.\n",
    "  - The learning rate increases linearly from 0 to initial learning rate during warmup period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:49.246478Z",
     "iopub.status.busy": "2022-03-27T14:28:49.245805Z",
     "iopub.status.idle": "2022-03-27T14:28:49.248379Z",
     "shell.execute_reply": "2022-03-27T14:28:49.247989Z",
     "shell.execute_reply.started": "2022-03-27T13:26:39.094070Z"
    },
    "id": "ykt0N1nVJJi2",
    "papermill": {
     "duration": 0.030404,
     "end_time": "2022-03-27T14:28:49.248483",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.218079",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "import torch\n",
    "from torch.optim import Optimizer\n",
    "from torch.optim.lr_scheduler import LambdaLR\n",
    "\n",
    "\n",
    "def get_cosine_schedule_with_warmup(\n",
    "\toptimizer: Optimizer,\n",
    "\tnum_warmup_steps: int,\n",
    "\tnum_training_steps: int,\n",
    "\tnum_cycles: float = 0.5,\n",
    "\tlast_epoch: int = -1,\n",
    "):\n",
    "\t\"\"\"\n",
    "\tCreate a schedule with a learning rate that decreases following the values of the cosine function between the\n",
    "\tinitial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the\n",
    "\tinitial lr set in the optimizer.\n",
    "\n",
    "\tArgs:\n",
    "\t\toptimizer (:class:`~torch.optim.Optimizer`):\n",
    "\t\tThe optimizer for which to schedule the learning rate.\n",
    "\t\tnum_warmup_steps (:obj:`int`):\n",
    "\t\tThe number of steps for the warmup phase.\n",
    "\t\tnum_training_steps (:obj:`int`):\n",
    "\t\tThe total number of training steps.\n",
    "\t\tnum_cycles (:obj:`float`, `optional`, defaults to 0.5):\n",
    "\t\tThe number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0\n",
    "\t\tfollowing a half-cosine).\n",
    "\t\tlast_epoch (:obj:`int`, `optional`, defaults to -1):\n",
    "\t\tThe index of the last epoch when resuming training.\n",
    "\n",
    "\tReturn:\n",
    "\t\t:obj:`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.\n",
    "\t\"\"\"\n",
    "\tdef lr_lambda(current_step):\n",
    "\t\t# Warmup\n",
    "\t\tif current_step < num_warmup_steps:\n",
    "\t\t\treturn float(current_step) / float(max(1, num_warmup_steps))\n",
    "\t\t# decadence\n",
    "\t\tprogress = float(current_step - num_warmup_steps) / float(\n",
    "\t\t\tmax(1, num_training_steps - num_warmup_steps)\n",
    "\t\t)\n",
    "\t\treturn max(\n",
    "\t\t\t0.0, 0.5 * (1.0 + math.cos(math.pi * float(num_cycles) * 2.0 * progress))\n",
    "\t\t)\n",
    "\n",
    "\treturn LambdaLR(optimizer, lr_lambda, last_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-LN2XkteM_uH",
    "papermill": {
     "duration": 0.018513,
     "end_time": "2022-03-27T14:28:49.285406",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.266893",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Model Function\n",
    "- Model forward function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:49.327719Z",
     "iopub.status.busy": "2022-03-27T14:28:49.326989Z",
     "iopub.status.idle": "2022-03-27T14:28:49.328930Z",
     "shell.execute_reply": "2022-03-27T14:28:49.329367Z",
     "shell.execute_reply.started": "2022-03-27T13:26:39.103803Z"
    },
    "id": "N-rr8529JMz0",
    "papermill": {
     "duration": 0.025655,
     "end_time": "2022-03-27T14:28:49.329492",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.303837",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "def model_fn(batch, model, criterion, device):\n",
    "\t\"\"\"Forward a batch through the model.\"\"\"\n",
    "\n",
    "\tmels, labels = batch\n",
    "\tmels = mels.to(device)\n",
    "\tlabels = labels.to(device)\n",
    "\n",
    "\touts = model(mels)\n",
    "\n",
    "\tloss = criterion(outs, labels)\n",
    "\n",
    "\t# Get the speaker id with highest probability.\n",
    "\tpreds = outs.argmax(1)\n",
    "\t# Compute accuracy.\n",
    "\taccuracy = torch.mean((preds == labels).float())\n",
    "\n",
    "\treturn loss, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cwM_xyOtNCI2",
    "papermill": {
     "duration": 0.018064,
     "end_time": "2022-03-27T14:28:49.365906",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.347842",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Validate\n",
    "- Calculate accuracy of the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:49.409837Z",
     "iopub.status.busy": "2022-03-27T14:28:49.409077Z",
     "iopub.status.idle": "2022-03-27T14:28:49.411390Z",
     "shell.execute_reply": "2022-03-27T14:28:49.411006Z",
     "shell.execute_reply.started": "2022-03-27T13:26:39.118561Z"
    },
    "id": "YAiv6kpdJRTJ",
    "papermill": {
     "duration": 0.027536,
     "end_time": "2022-03-27T14:28:49.411489",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.383953",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "import torch\n",
    "\n",
    "\n",
    "def valid(dataloader, model, criterion, device): \n",
    "\t\"\"\"Validate on validation set.\"\"\"\n",
    "\n",
    "\tmodel.eval()\n",
    "\trunning_loss = 0.0\n",
    "\trunning_accuracy = 0.0\n",
    "\tpbar = tqdm(total=len(dataloader.dataset), ncols=0, desc=\"Valid\", unit=\" uttr\")\n",
    "\n",
    "\tfor i, batch in enumerate(dataloader):\n",
    "\t\twith torch.no_grad():\n",
    "\t\t\tloss, accuracy = model_fn(batch, model, criterion, device)\n",
    "\t\t\trunning_loss += loss.item()\n",
    "\t\t\trunning_accuracy += accuracy.item()\n",
    "\n",
    "\t\tpbar.update(dataloader.batch_size)\n",
    "\t\tpbar.set_postfix(\n",
    "\t\t\tloss=f\"{running_loss / (i+1):.2f}\",\n",
    "\t\t\taccuracy=f\"{running_accuracy / (i+1):.2f}\",\n",
    "\t\t)\n",
    "\n",
    "\tpbar.close()\n",
    "\tmodel.train()\n",
    "\n",
    "\treturn running_accuracy / len(dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g6ne9G-eNEdG",
    "papermill": {
     "duration": 0.018262,
     "end_time": "2022-03-27T14:28:49.447716",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.429454",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Main function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T14:28:49.501464Z",
     "iopub.status.busy": "2022-03-27T14:28:49.500932Z",
     "iopub.status.idle": "2022-03-27T15:16:15.711137Z",
     "shell.execute_reply": "2022-03-27T15:16:15.710692Z",
     "shell.execute_reply.started": "2022-03-27T13:26:39.130770Z"
    },
    "id": "Usv9s-CuJSG7",
    "papermill": {
     "duration": 2846.24543,
     "end_time": "2022-03-27T15:16:15.711270",
     "exception": false,
     "start_time": "2022-03-27T14:28:49.465840",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Info]: Use cuda now!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "  cpuset_checked))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Info]: Finish loading data!\n",
      "Layer: 0 parameter num: 8960\n",
      "Layer: 1 parameter num: 224\n",
      "Layer: 2 parameter num: 200704\n",
      "Layer: 3 parameter num: 896\n",
      "Layer: 4 parameter num: 200704\n",
      "Layer: 5 parameter num: 224\n",
      "Layer: 6 parameter num: 224\n",
      "Layer: 7 parameter num: 224\n",
      "Layer: 8 parameter num: 3584\n",
      "Layer: 9 parameter num: 7168\n",
      "Layer: 10 parameter num: 3584\n",
      "Layer: 11 parameter num: 224\n",
      "Layer: 12 parameter num: 4100\n",
      "Layer: 13 parameter num: 224\n",
      "Layer: 14 parameter num: 224\n",
      "Layer: 15 parameter num: 224\n",
      "Layer: 16 parameter num: 224\n",
      "Layer: 17 parameter num: 200704\n",
      "Layer: 18 parameter num: 896\n",
      "Layer: 19 parameter num: 8960\n",
      "Layer: 20 parameter num: 448\n",
      "Layer: 21 parameter num: 448\n",
      "Layer: 22 parameter num: 448\n",
      "Layer: 23 parameter num: 100352\n",
      "Layer: 24 parameter num: 224\n",
      "Layer: 25 parameter num: 200704\n",
      "Layer: 26 parameter num: 896\n",
      "Layer: 27 parameter num: 200704\n",
      "Layer: 28 parameter num: 224\n",
      "Layer: 29 parameter num: 224\n",
      "Layer: 30 parameter num: 224\n",
      "Layer: 31 parameter num: 224\n",
      "Layer: 32 parameter num: 224\n",
      "Layer: 33 parameter num: 224\n",
      "Layer: 34 parameter num: 224\n",
      "Layer: 35 parameter num: 134400\n",
      "Layer: 36 parameter num: 600\n",
      "Total params: 1283068\n",
      "[Info]: Finish creating model!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [01:08<00:00, 29.20 step/s, accuracy=0.34, loss=3.35, step=2000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1017.19 uttr/s, accuracy=0.30, loss=3.42]\n",
      "Train: 100% 2000/2000 [00:52<00:00, 38.14 step/s, accuracy=0.50, loss=1.88, step=4000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1661.99 uttr/s, accuracy=0.50, loss=2.28]\n",
      "Train: 100% 2000/2000 [00:51<00:00, 38.52 step/s, accuracy=0.72, loss=1.65, step=6000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1749.11 uttr/s, accuracy=0.59, loss=1.88]\n",
      "Train: 100% 2000/2000 [00:55<00:00, 36.29 step/s, accuracy=0.59, loss=1.63, step=8000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1702.86 uttr/s, accuracy=0.64, loss=1.66]\n",
      "Train: 100% 2000/2000 [00:54<00:00, 36.98 step/s, accuracy=0.62, loss=1.56, step=1e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1786.18 uttr/s, accuracy=0.66, loss=1.57]\n",
      "Train:   0% 8/2000 [00:00<00:54, 36.37 step/s, accuracy=0.69, loss=1.21, step=1e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 10000, best model saved. (accuracy=0.6552)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:53<00:00, 37.16 step/s, accuracy=0.72, loss=1.13, step=12000]\n",
      "Valid: 100% 5664/5667 [00:04<00:00, 1392.39 uttr/s, accuracy=0.71, loss=1.34]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.69 step/s, accuracy=0.69, loss=1.33, step=14000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1121.53 uttr/s, accuracy=0.71, loss=1.32]\n",
      "Train: 100% 2000/2000 [00:54<00:00, 36.90 step/s, accuracy=0.78, loss=0.90, step=16000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1713.65 uttr/s, accuracy=0.72, loss=1.25]\n",
      "Train: 100% 2000/2000 [00:56<00:00, 35.63 step/s, accuracy=0.72, loss=1.00, step=18000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1818.57 uttr/s, accuracy=0.74, loss=1.18]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.55 step/s, accuracy=0.81, loss=0.81, step=2e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1648.01 uttr/s, accuracy=0.73, loss=1.20]\n",
      "Train:   0% 8/2000 [00:00<00:54, 36.28 step/s, accuracy=0.78, loss=0.78, step=2e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 20000, best model saved. (accuracy=0.7380)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:55<00:00, 35.75 step/s, accuracy=0.84, loss=0.61, step=22000]\n",
      "Valid: 100% 5664/5667 [00:04<00:00, 1270.76 uttr/s, accuracy=0.76, loss=1.11]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.71 step/s, accuracy=0.88, loss=0.31, step=24000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1788.12 uttr/s, accuracy=0.77, loss=1.09]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.59 step/s, accuracy=0.81, loss=0.69, step=26000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1800.13 uttr/s, accuracy=0.77, loss=1.05]\n",
      "Train: 100% 2000/2000 [00:58<00:00, 34.09 step/s, accuracy=0.94, loss=0.39, step=28000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1841.67 uttr/s, accuracy=0.78, loss=1.06]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.10 step/s, accuracy=0.75, loss=1.43, step=3e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1830.22 uttr/s, accuracy=0.78, loss=1.01]\n",
      "Train:   0% 9/2000 [00:00<00:47, 41.56 step/s, accuracy=0.69, loss=1.24, step=3e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 30000, best model saved. (accuracy=0.7828)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:52<00:00, 37.82 step/s, accuracy=0.88, loss=0.45, step=32000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1121.29 uttr/s, accuracy=0.79, loss=0.98]\n",
      "Train: 100% 2000/2000 [00:55<00:00, 35.80 step/s, accuracy=0.84, loss=0.46, step=34000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1846.97 uttr/s, accuracy=0.80, loss=0.98]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.04 step/s, accuracy=0.94, loss=0.26, step=36000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1683.57 uttr/s, accuracy=0.81, loss=0.90]\n",
      "Train: 100% 2000/2000 [01:00<00:00, 32.97 step/s, accuracy=0.81, loss=0.50, step=38000]\n",
      "Valid: 100% 5664/5667 [00:04<00:00, 1193.53 uttr/s, accuracy=0.80, loss=0.95]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.92 step/s, accuracy=0.91, loss=0.47, step=4e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1680.17 uttr/s, accuracy=0.80, loss=1.00]\n",
      "Train:   0% 7/2000 [00:00<01:00, 32.74 step/s, accuracy=0.97, loss=0.34, step=4e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 40000, best model saved. (accuracy=0.8114)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:49<00:00, 40.68 step/s, accuracy=0.97, loss=0.16, step=42000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1674.98 uttr/s, accuracy=0.81, loss=0.93]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.19 step/s, accuracy=0.88, loss=0.52, step=44000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1856.62 uttr/s, accuracy=0.82, loss=0.90]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.53 step/s, accuracy=0.88, loss=0.52, step=46000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1834.73 uttr/s, accuracy=0.83, loss=0.85]\n",
      "Train: 100% 2000/2000 [01:02<00:00, 31.85 step/s, accuracy=0.94, loss=0.18, step=48000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1039.85 uttr/s, accuracy=0.83, loss=0.90]\n",
      "Train: 100% 2000/2000 [00:52<00:00, 38.23 step/s, accuracy=0.91, loss=0.35, step=5e+4]\n",
      "Valid: 100% 5664/5667 [00:04<00:00, 1219.33 uttr/s, accuracy=0.83, loss=0.86]\n",
      "Train:   0% 8/2000 [00:00<00:58, 34.02 step/s, accuracy=0.94, loss=0.23, step=5e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 50000, best model saved. (accuracy=0.8275)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:50<00:00, 39.22 step/s, accuracy=0.97, loss=0.29, step=52000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1836.86 uttr/s, accuracy=0.83, loss=0.85]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.89 step/s, accuracy=0.91, loss=0.28, step=54000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1644.25 uttr/s, accuracy=0.83, loss=0.83]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.73 step/s, accuracy=0.97, loss=0.25, step=56000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1797.82 uttr/s, accuracy=0.84, loss=0.85]\n",
      "Train: 100% 2000/2000 [01:00<00:00, 32.87 step/s, accuracy=0.97, loss=0.15, step=58000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1057.05 uttr/s, accuracy=0.84, loss=0.80]\n",
      "Train: 100% 2000/2000 [00:56<00:00, 35.65 step/s, accuracy=0.94, loss=0.12, step=6e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1820.90 uttr/s, accuracy=0.84, loss=0.81]\n",
      "Train:   0% 8/2000 [00:00<00:58, 34.04 step/s, accuracy=0.94, loss=0.30, step=6e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 60000, best model saved. (accuracy=0.8438)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:49<00:00, 40.08 step/s, accuracy=1.00, loss=0.06, step=62000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1524.09 uttr/s, accuracy=0.85, loss=0.78]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.79 step/s, accuracy=0.94, loss=0.17, step=64000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1814.74 uttr/s, accuracy=0.85, loss=0.76]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.01 step/s, accuracy=0.97, loss=0.11, step=66000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1809.76 uttr/s, accuracy=0.85, loss=0.77]\n",
      "Train: 100% 2000/2000 [00:55<00:00, 36.02 step/s, accuracy=0.91, loss=0.21, step=68000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 1019.51 uttr/s, accuracy=0.86, loss=0.71]\n",
      "Train: 100% 2000/2000 [01:05<00:00, 30.51 step/s, accuracy=0.97, loss=0.08, step=7e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1673.65 uttr/s, accuracy=0.86, loss=0.74]\n",
      "Train:   0% 8/2000 [00:00<00:52, 37.63 step/s, accuracy=1.00, loss=0.15, step=7e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 70000, best model saved. (accuracy=0.8626)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:49<00:00, 40.22 step/s, accuracy=0.97, loss=0.19, step=72000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1665.87 uttr/s, accuracy=0.86, loss=0.74]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.93 step/s, accuracy=0.91, loss=0.37, step=74000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1819.36 uttr/s, accuracy=0.87, loss=0.71]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.31 step/s, accuracy=0.97, loss=0.12, step=76000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1542.15 uttr/s, accuracy=0.86, loss=0.74]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.94 step/s, accuracy=1.00, loss=0.03, step=78000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1792.61 uttr/s, accuracy=0.87, loss=0.67]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.71 step/s, accuracy=1.00, loss=0.02, step=8e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1827.39 uttr/s, accuracy=0.87, loss=0.69]\n",
      "Train:   0% 8/2000 [00:00<00:53, 37.31 step/s, accuracy=0.97, loss=0.17, step=8e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 80000, best model saved. (accuracy=0.8702)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:50<00:00, 39.88 step/s, accuracy=1.00, loss=0.07, step=82000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1691.35 uttr/s, accuracy=0.88, loss=0.66]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.18 step/s, accuracy=0.97, loss=0.09, step=84000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1667.29 uttr/s, accuracy=0.87, loss=0.67]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 40.00 step/s, accuracy=0.97, loss=0.13, step=86000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1831.24 uttr/s, accuracy=0.88, loss=0.68]\n",
      "Train: 100% 2000/2000 [01:00<00:00, 32.94 step/s, accuracy=1.00, loss=0.03, step=88000]\n",
      "Valid: 100% 5664/5667 [00:05<00:00, 950.75 uttr/s, accuracy=0.87, loss=0.68] \n",
      "Train: 100% 2000/2000 [01:06<00:00, 29.95 step/s, accuracy=0.97, loss=0.14, step=9e+4]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1806.07 uttr/s, accuracy=0.87, loss=0.68]\n",
      "Train:   0% 8/2000 [00:00<00:52, 37.79 step/s, accuracy=0.97, loss=0.07, step=9e+4]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 90000, best model saved. (accuracy=0.8773)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train: 100% 2000/2000 [00:49<00:00, 40.14 step/s, accuracy=1.00, loss=0.04, step=92000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1640.08 uttr/s, accuracy=0.88, loss=0.65]\n",
      "Train: 100% 2000/2000 [00:50<00:00, 39.75 step/s, accuracy=1.00, loss=0.01, step=94000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1632.95 uttr/s, accuracy=0.87, loss=0.67]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.27 step/s, accuracy=0.97, loss=0.08, step=96000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1673.38 uttr/s, accuracy=0.88, loss=0.65]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.31 step/s, accuracy=1.00, loss=0.02, step=98000]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1811.95 uttr/s, accuracy=0.87, loss=0.66]\n",
      "Train: 100% 2000/2000 [00:49<00:00, 40.08 step/s, accuracy=1.00, loss=0.02, step=1e+5]\n",
      "Valid: 100% 5664/5667 [00:03<00:00, 1831.78 uttr/s, accuracy=0.88, loss=0.66]\n",
      "Train:   0% 0/2000 [00:00<?, ? step/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 100000, best model saved. (accuracy=0.8787)\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.optim import AdamW\n",
    "from torch.utils.data import DataLoader, random_split\n",
    "\n",
    "\n",
    "def parse_args():\n",
    "\t\"\"\"arguments\"\"\"\n",
    "\tconfig = {\n",
    "\t\t\"data_dir\": \"../input/ml2022spring-hw4/Dataset\",\n",
    "\t\t\"save_path\": \"model.ckpt\",\n",
    "\t\t\"batch_size\": 32,\n",
    "\t\t\"n_workers\": 8,\n",
    "\t\t\"valid_steps\": 2000,\n",
    "\t\t\"warmup_steps\": 1000,\n",
    "\t\t\"save_steps\": 10000,\n",
    "\t\t\"total_steps\": 100000,\n",
    "\t}\n",
    "\n",
    "\treturn config\n",
    "\n",
    "\n",
    "def main(\n",
    "\tdata_dir,\n",
    "\tsave_path,\n",
    "\tbatch_size,\n",
    "\tn_workers,\n",
    "\tvalid_steps,\n",
    "\twarmup_steps,\n",
    "\ttotal_steps,\n",
    "\tsave_steps,\n",
    "):\n",
    "\t\"\"\"Main function.\"\"\"\n",
    "\tdevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\tprint(f\"[Info]: Use {device} now!\")\n",
    "\n",
    "\ttrain_loader, valid_loader, speaker_num = get_dataloader(data_dir, batch_size, n_workers)\n",
    "\ttrain_iterator = iter(train_loader)\n",
    "\tprint(f\"[Info]: Finish loading data!\",flush = True)\n",
    "\n",
    "\tmodel = Classifier(n_spks=speaker_num).to(device)\n",
    "\ttotal_params = 0\n",
    "\tfor i, param in enumerate(model.parameters()):\n",
    "\t\tprint('Layer:', i, 'parameter num:', param.numel())\n",
    "\t\ttotal_params += param.numel()\n",
    "\tprint(f'Total params: {total_params}')\n",
    "\tcriterion = nn.CrossEntropyLoss()\n",
    "\toptimizer = AdamW(model.parameters(), lr=1e-3)\n",
    "\tscheduler = get_cosine_schedule_with_warmup(optimizer, warmup_steps, total_steps)\n",
    "\tprint(f\"[Info]: Finish creating model!\",flush = True)\n",
    "\n",
    "\tbest_accuracy = -1.0\n",
    "\tbest_state_dict = None\n",
    "\n",
    "\tpbar = tqdm(total=valid_steps, ncols=0, desc=\"Train\", unit=\" step\")\n",
    "\n",
    "\tfor step in range(total_steps):\n",
    "\t\t# Get data\n",
    "\t\ttry:\n",
    "\t\t\tbatch = next(train_iterator)\n",
    "\t\texcept StopIteration:\n",
    "\t\t\ttrain_iterator = iter(train_loader)\n",
    "\t\t\tbatch = next(train_iterator)\n",
    "\n",
    "\t\tloss, accuracy = model_fn(batch, model, criterion, device)\n",
    "\t\tbatch_loss = loss.item()\n",
    "\t\tbatch_accuracy = accuracy.item()\n",
    "\n",
    "\t\t# Updata model\n",
    "\t\tloss.backward()\n",
    "\t\toptimizer.step()\n",
    "\t\tscheduler.step()\n",
    "\t\toptimizer.zero_grad()\n",
    "\n",
    "\t\t# Log\n",
    "\t\tpbar.update()\n",
    "\t\tpbar.set_postfix(\n",
    "\t\t\tloss=f\"{batch_loss:.2f}\",\n",
    "\t\t\taccuracy=f\"{batch_accuracy:.2f}\",\n",
    "\t\t\tstep=step + 1,\n",
    "\t\t)\n",
    "\n",
    "\t\t# Do validation\n",
    "\t\tif (step + 1) % valid_steps == 0:\n",
    "\t\t\tpbar.close()\n",
    "\n",
    "\t\t\tvalid_accuracy = valid(valid_loader, model, criterion, device)\n",
    "\n",
    "\t\t\t# keep the best model\n",
    "\t\t\tif valid_accuracy > best_accuracy:\n",
    "\t\t\t\tbest_accuracy = valid_accuracy\n",
    "\t\t\t\tbest_state_dict = model.state_dict()\n",
    "\n",
    "\t\t\tpbar = tqdm(total=valid_steps, ncols=0, desc=\"Train\", unit=\" step\")\n",
    "\n",
    "\t\t# Save the best model so far.\n",
    "\t\tif (step + 1) % save_steps == 0 and best_state_dict is not None:\n",
    "\t\t\ttorch.save(best_state_dict, save_path)\n",
    "\t\t\tpbar.write(f\"Step {step + 1}, best model saved. (accuracy={best_accuracy:.4f})\")\n",
    "\n",
    "\tpbar.close()\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\tmain(**parse_args())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NLatBYAhNNMx",
    "papermill": {
     "duration": 43.614448,
     "end_time": "2022-03-27T15:17:42.842760",
     "exception": false,
     "start_time": "2022-03-27T15:16:59.228312",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Inference\n",
    "\n",
    "## Dataset of inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T15:19:10.547938Z",
     "iopub.status.busy": "2022-03-27T15:19:10.546272Z",
     "iopub.status.idle": "2022-03-27T15:19:10.548509Z",
     "shell.execute_reply": "2022-03-27T15:19:10.548936Z",
     "shell.execute_reply.started": "2022-03-27T14:11:49.943771Z"
    },
    "id": "efS4pCmAJXJH",
    "papermill": {
     "duration": 43.598517,
     "end_time": "2022-03-27T15:19:10.549097",
     "exception": false,
     "start_time": "2022-03-27T15:18:26.950580",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import torch\n",
    "from pathlib import Path\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class InferenceDataset(Dataset):\n",
    "\tdef __init__(self, data_dir):\n",
    "\t\ttestdata_path = Path(data_dir) / \"testdata.json\"\n",
    "\t\tmetadata = json.load(testdata_path.open())\n",
    "\t\tself.data_dir = data_dir\n",
    "\t\tself.data = metadata[\"utterances\"]\n",
    "\n",
    "\tdef __len__(self):\n",
    "\t\treturn len(self.data)\n",
    "\n",
    "\tdef __getitem__(self, index):\n",
    "\t\tutterance = self.data[index]\n",
    "\t\tfeat_path = utterance[\"feature_path\"]\n",
    "\t\tmel = torch.load(os.path.join(self.data_dir, feat_path))\n",
    "\n",
    "\t\treturn feat_path, mel\n",
    "\n",
    "\n",
    "def inference_collate_batch(batch):\n",
    "\t\"\"\"Collate a batch of data.\"\"\"\n",
    "\tfeat_paths, mels = zip(*batch)\n",
    "\n",
    "\treturn feat_paths, torch.stack(mels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tl0WnYwxNK_S",
    "papermill": {
     "duration": 44.208959,
     "end_time": "2022-03-27T15:20:38.235378",
     "exception": false,
     "start_time": "2022-03-27T15:19:54.026419",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Main funcrion of Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-27T15:22:05.936935Z",
     "iopub.status.busy": "2022-03-27T15:22:05.936295Z",
     "iopub.status.idle": "2022-03-27T15:22:40.992254Z",
     "shell.execute_reply": "2022-03-27T15:22:40.992666Z",
     "shell.execute_reply.started": "2022-03-27T14:11:49.957089Z"
    },
    "id": "i8SAbuXEJb2A",
    "papermill": {
     "duration": 79.198144,
     "end_time": "2022-03-27T15:22:40.992828",
     "exception": false,
     "start_time": "2022-03-27T15:21:21.794684",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Info]: Use cuda now!\n",
      "[Info]: Finish loading data!\n",
      "[Info]: Finish creating model!\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54a15c138b254845898a7693a350f111",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/8000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json\n",
    "import csv\n",
    "from pathlib import Path\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "def parse_args():\n",
    "\t\"\"\"arguments\"\"\"  \n",
    "\tconfig = {\n",
    "\t\t\"data_dir\": \"../input/ml2022spring-hw4/Dataset\",\n",
    "\t\t\"model_path\": \"model.ckpt\",\n",
    "\t\t\"output_path\": \"output.csv\",\n",
    "\t}\n",
    "\n",
    "\treturn config\n",
    "\n",
    "\n",
    "def main(\n",
    "\tdata_dir,\n",
    "\tmodel_path,\n",
    "\toutput_path,\n",
    "):\n",
    "\t\"\"\"Main function.\"\"\"\n",
    "\tdevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\tprint(f\"[Info]: Use {device} now!\")\n",
    "\n",
    "\tmapping_path = Path(data_dir) / \"mapping.json\"\n",
    "\tmapping = json.load(mapping_path.open())\n",
    "\n",
    "\tdataset = InferenceDataset(data_dir)\n",
    "\tdataloader = DataLoader(\n",
    "\t\tdataset,\n",
    "\t\tbatch_size=1,\n",
    "\t\tshuffle=False,\n",
    "\t\tdrop_last=False,\n",
    "\t\tnum_workers=8,\n",
    "\t\tcollate_fn=inference_collate_batch,\n",
    "\t)\n",
    "\tprint(f\"[Info]: Finish loading data!\",flush = True)\n",
    "\n",
    "\tspeaker_num = len(mapping[\"id2speaker\"])\n",
    "\tmodel = Classifier(n_spks=speaker_num).to(device)\n",
    "\tmodel.load_state_dict(torch.load(model_path))\n",
    "\tmodel.eval()\n",
    "\tprint(f\"[Info]: Finish creating model!\",flush = True)\n",
    "\n",
    "\tresults = [[\"Id\", \"Category\"]]\n",
    "\tfor feat_paths, mels in tqdm(dataloader):\n",
    "\t\twith torch.no_grad():\n",
    "\t\t\tmels = mels.to(device)\n",
    "\t\t\touts = model(mels)\n",
    "\t\t\tpreds = outs.argmax(1).cpu().numpy()\n",
    "\t\t\tfor feat_path, pred in zip(feat_paths, preds):\n",
    "\t\t\t\tresults.append([feat_path, mapping[\"id2speaker\"][str(pred)]])\n",
    "\n",
    "\twith open(output_path, 'w', newline='') as csvfile:\n",
    "\t\twriter = csv.writer(csvfile)\n",
    "\t\twriter.writerows(results)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\tmain(**parse_args())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 3296.542885,
   "end_time": "2022-03-27T15:23:25.938177",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-03-27T14:28:29.395292",
   "version": "2.3.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "40267d1484924b0987239a659d1210ee": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "41c18e6e682d454a9eb5e733a537a017": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_7dd8955bf652405b96f46979fd2c8729",
       "placeholder": "​",
       "style": "IPY_MODEL_65b3f9dc23f24a0c90fa9c2ca8cc89a4",
       "value": " 8000/8000 [00:34&lt;00:00, 229.20it/s]"
      }
     },
     "4f09ae36f6714b1aab82769424e39ed8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "54a15c138b254845898a7693a350f111": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_7aca5757c182420fb6be5ec46d6f71cd",
        "IPY_MODEL_ee7a76d6bf994e3ca2404ddf07382fe3",
        "IPY_MODEL_41c18e6e682d454a9eb5e733a537a017"
       ],
       "layout": "IPY_MODEL_4f09ae36f6714b1aab82769424e39ed8"
      }
     },
     "65b3f9dc23f24a0c90fa9c2ca8cc89a4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "7aca5757c182420fb6be5ec46d6f71cd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_87f69a62c3a04fa58341cf56443e60d7",
       "placeholder": "​",
       "style": "IPY_MODEL_40267d1484924b0987239a659d1210ee",
       "value": "100%"
      }
     },
     "7dd8955bf652405b96f46979fd2c8729": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "80c0da948e4a4f6185f2df8e9f631a6b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8757236d98814823a53386f7843a752e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "87f69a62c3a04fa58341cf56443e60d7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ee7a76d6bf994e3ca2404ddf07382fe3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_80c0da948e4a4f6185f2df8e9f631a6b",
       "max": 8000,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_8757236d98814823a53386f7843a752e",
       "value": 8000
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}