04-evaluation/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3fb05700",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "IPython.notebook.set_autosave_interval(0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autosave disabled\n"
     ]
    }
   ],
   "source": [
    "%autosave 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "464b9b4a",
   "metadata": {},
   "source": [
    "# 4. Evaluation Metrics for Classification\n",
    "\n",
    "In the previous session we trained a model for predicting churn. How do we know if it's good?\n",
    "\n",
    "\n",
    "## 4.1 Evaluation metrics: session overview \n",
    "\n",
    "* Dataset: https://www.kaggle.com/blastchar/telco-customer-churn\n",
    "* https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-03-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv\n",
    "\n",
    "\n",
    "*Metric* - function that compares the predictions with the actual values and outputs a single number that tells how good the predictions are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3ff7c44",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff66e768",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "81ee3ecc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data-week-3.csv')\n",
    "\n",
    "df.columns = df.columns.str.lower().str.replace(' ', '_')\n",
    "\n",
    "categorical_columns = list(df.dtypes[df.dtypes == 'object'].index)\n",
    "\n",
    "for c in categorical_columns:\n",
    "    df[c] = df[c].str.lower().str.replace(' ', '_')\n",
    "\n",
    "df.totalcharges = pd.to_numeric(df.totalcharges, errors='coerce')\n",
    "df.totalcharges = df.totalcharges.fillna(0)\n",
    "\n",
    "df.churn = (df.churn == 'yes').astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0b1903b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=1)\n",
    "df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=1)\n",
    "\n",
    "df_train = df_train.reset_index(drop=True)\n",
    "df_val = df_val.reset_index(drop=True)\n",
    "df_test = df_test.reset_index(drop=True)\n",
    "\n",
    "y_train = df_train.churn.values\n",
    "y_val = df_val.churn.values\n",
    "y_test = df_test.churn.values\n",
    "\n",
    "del df_train['churn']\n",
    "del df_val['churn']\n",
    "del df_test['churn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ab4132a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical = ['tenure', 'monthlycharges', 'totalcharges']\n",
    "\n",
    "categorical = [\n",
    "    'gender',\n",
    "    'seniorcitizen',\n",
    "    'partner',\n",
    "    'dependents',\n",
    "    'phoneservice',\n",
    "    'multiplelines',\n",
    "    'internetservice',\n",
    "    'onlinesecurity',\n",
    "    'onlinebackup',\n",
    "    'deviceprotection',\n",
    "    'techsupport',\n",
    "    'streamingtv',\n",
    "    'streamingmovies',\n",
    "    'contract',\n",
    "    'paperlessbilling',\n",
    "    'paymentmethod',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cc4583b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv = DictVectorizer(sparse=False)\n",
    "\n",
    "train_dict = df_train[categorical + numerical].to_dict(orient='records')\n",
    "X_train = dv.fit_transform(train_dict)\n",
    "\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4ea51f38",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_dict = df_val[categorical + numerical].to_dict(orient='records')\n",
    "X_val = dv.transform(val_dict)\n",
    "\n",
    "y_pred = model.predict_proba(X_val)[:, 1]\n",
    "churn_decision = (y_pred >= 0.5)\n",
    "(y_val == churn_decision).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d8a7559",
   "metadata": {},
   "source": [
    "## 4.2 Accuracy and dummy model\n",
    "\n",
    "* Evaluate the model on different thresholds\n",
    "* Check the accuracy of dummy baselines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "948eda88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1409"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fc59dff6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y_val == churn_decision).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bcb4def",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1132/ 1409"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "adf1012b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ffb75fc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_val, y_pred >= 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "27840301",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.00 0.274\n",
      "0.05 0.509\n",
      "0.10 0.591\n",
      "0.15 0.666\n",
      "0.20 0.710\n",
      "0.25 0.739\n",
      "0.30 0.760\n",
      "0.35 0.772\n",
      "0.40 0.785\n",
      "0.45 0.793\n",
      "0.50 0.803\n",
      "0.55 0.801\n",
      "0.60 0.795\n",
      "0.65 0.786\n",
      "0.70 0.766\n",
      "0.75 0.744\n",
      "0.80 0.735\n",
      "0.85 0.726\n",
      "0.90 0.726\n",
      "0.95 0.726\n",
      "1.00 0.726\n"
     ]
    }
   ],
   "source": [
    "thresholds = np.linspace(0, 1, 21)\n",
    "\n",
    "scores = []\n",
    "\n",
    "for t in thresholds:\n",
    "    score = accuracy_score(y_val, y_pred >= t)\n",
    "    print('%.2f %.3f' % (t, score))\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "11f492a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xff7b24eb48e0>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfOElEQVR4nO3de3xVZ53v8c8vVyABQkgINAkk3Etb6CVS206tvVF0tJ2ZOk7rXatVa7Xjbaa+nJfj6Zxz5jgzWnWGmbFTa723VWccHNFeaGsVLZBaoIUmIRAgAXIhae7k/jt/7F1M00A2sHdW9trf9+uVV/da+2nWb5HkmyfPs9azzN0REZHklxZ0ASIiEh8KdBGRkFCgi4iEhAJdRCQkFOgiIiGREdSBCwoKvKysLKjDi4gkpeeee+6YuxeO915ggV5WVkZlZWVQhxcRSUpmdvBk72nIRUQkJBToIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEjEFupmtN7NqM6s1s7vHeX+hmT1lZs+b2S4ze3P8SxURkVOZ8Dp0M0sHNgDXAw3AdjPb6O57RjX7G+ARd/83M1sFbALKElCvSMJ0HB/kuYNt7DnSSWZ6GrnTMsjNziAnK+MPr7Mj/83NzmBaZhpmFnTZIifEcmPRWqDW3fcDmNlDwE3A6EB3YFb09WzgSDyLFEmEtp4BttW1sbWulW11bew52snpPB4gPc3IyUr/Q9BHQ79kzgyWFOawZF4uSwtzKc6bTlqagl8SL5ZALwbqR203AJeOafNF4DEz+ziQA1wXl+pE4qips4+tdW1sq2tl6/429jZ3AzAtM42LF87hrmuXsbY8nwtL8xhx6Okfort/iO6+oT+87n/l9TDd/YP09A//oc3AEJ3HB/nF4aO09w6eOG52RhqLC3MjIV+Yy5J5kdeLC3KZnpUe1D+HhFC8bv2/FXjQ3b9sZpcB3zWz8919ZHQjM7sduB1g4cKFcTq0yPjq23pf1QM/0NoLQG52BpcsmsOfXlzMpeX5XFCcR1bGa6eTcrMzKDrDY7f1DLCvpZt9zd3sa+mmtrmbXQ0d/PyFo6/6K6A4b/qJgF86L5eV82dx3jmzmJapoJfTF0ugHwZKR22XRPeNdhuwHsDdf2dm04ACoHl0I3e/D7gPoKKiQs++k7jpHRjixcOd7KxvZ2dDO88faudw+3EAZk/PZG15Pu96/SLWluezasEsMtITe4FXfk4W+Tn5vK4s/1X7+waHOdDaw77mnkjgRz+217VxfHAYgIw0Y8X8mawuyePC0tmsLslj2bzchNcsyS+WQN8OLDOzciJBfgvwjjFtDgHXAg+a2bnANKAlnoWKvGJweITqxi521Lezq6GdXQ0d1DR1MRLtIhTnTefChXl8+KrFrC3PZ/m8mVNmDHtaZjor589i5fxZr9o/MuIc7exj9+EOdja0s7O+g5/vOsIPtx0CYHpmOucXz2JNSR6rS/O4sCSP0vzpmpSVV7FYHhIdvQzxq0A68IC7/x8zuweodPeN0Stb/gPIJTJB+lfu/tipPmdFRYVrtUWZyMiIU9faw65oyO1saGf3kU4GhiKjefk5Wawumc2akjzWRHuzBbnZAVcdHyMjzoHWHnY1dJz45fXiqHOfMyOT1SV5rCmJnHdF2RzyZmQFXLUkmpk95+4V474XS6AnggJdxuPu1DR189juRp6ta2VXQwddfUNApJd6Qcls1pTMZk1pHmtK8iiZk1q91Ff+OtnZ0M6u6C+4V/46mZaZxrtfv4gPX7UkNL/U5LUU6DKlDY84vz/0Mo/tbuSxPU0cbO3FDFYtmMWa6PDCmtI8ls7LJX2KDJ1MJb0DQ7zQ0MHDlfX89PnDZGek857LFnH7GxYzV8EeOgp0mXL6Bof57b5jPLa7iSdeauJY9wBZ6WlcvnQu61bN57pz5zFv1rSgy0w6+1q6+efNe/nvnUeYnpnOey8v4/YrFzMnR0MxYaFAlymh4/ggT1U189ieRp6ubqF3YJjc7AyuXjmPdauKeOOKQmZOywy6zFCobe7m65v38rNdR5iRmc77rijjQ1cu1hh7CCjQJTCNHX08vicylPK7fa0MjTiFM7O5flUR61YVcdmSuWRn6JrrRKlp6uJrm/fy811Hyc3O4ANXlHHbHy1m9gz94kxWCnSZNO7O7iOdbH6pmSermtjZ0AFAeUEO684r4obz5nNhSd6UuYwwVVQ1dvK1J/byixcbmTktgw9cUc4H/qic2dMV7MlGgS4J1TswxG/2HuPJqmaerGqmuasfM7iwNI/rzi3ihvOKWFKYm1JXo0xVe4508rXNNTy6u4lZ0zL44JWLef8VZRrqSiIKdIm7+rZenqxqZnNVM8/ub2VgaITc7AzesLyAa1ZGxsN16dzU9eLhDr76xF6eeKmJ2dMzuf0Ni3nf5WXkZMdrNRBJFAW6nLWh4RGer28/MZRS0xRZ2Kq8IIdrVs7j2pXzqCjLH3dNFJm6Xmjo4KtP1LC5qpm5OVl87OqlvOPShVpLZgpToMsZGRoe4Ze7G3l8TxNPV7fQcXyQjDRjbXk+16ycxzUr57G4MDfoMiUOfn/oZb78WDVbaltZMHsad127jLddUqL1Y6YgBbqclpER539eOMq9j9dQd6yH/Jwsrl4RCfArlxcwS+OtobWl9hj/+Gg1O+rbKS/I4ZPXL+ctFyzQJPYUokCXmLg7m19q5p8eq6aqsYsVRTP59LrlXHtuke7QTCFjvw9Wzp/JZ9at4Npz52liewpQoMuEfrsv0jN7/lA7ZXNn8Mnrl/PW1eeoZ5bCRkacn+06wr2P13CgtZeLFubx2RtWcPmSgqBLS2kKdDmp5w+9zD9Fx07nz5rGXddFxk4zNXYqUYPDI/zkuQa+tnkvRzv6uGLpXD6zbgUXLZwTdGkpSYEur1HV2MmXH6vh8T1NzM3J4o6rl/JOXd0gp9A3OMz3tx7iX5+qpbVngOtXFfHpdctfs7a7JJYCXU44cKyHe5+oYePOI+RmZ/DhNyzm/VeU6/pjiVlP/xDf2lLHN57ZT3f/EG9dfQ5vWb2A15XlaxGwSaBAF460H+efn9zLI5UNZKYb77+inA+/QYs1yZlr7x3gG8/s59u/PUDvQOTxeSvnz2RteT6Xls/ldeVzmDdTK2bGmwI9hXUcH+Trm/fy3WcP4u6889JF3HH1Ev2gSdz0Dw2zq6GDrftb2VrXxnMHXz4R8IsLcrh0cf6JkD8nb3rA1SY/BXqK2lbXxicf3sHRjuPcfHEJn7h2GaX5M4IuS0JucHiE3Uc62VbXytb9bWw70HbiqVMlc6aztjyf15fPZW15PovmztClkKdJgZ5iBodH+PrmvWx4qpbS/Bl87ZaLuLA0L+iyJEUNjzhVjZ1sq2s7EfBtPQMAFM3K5uKFcyLPRi2dzQXFs7VQ2AQU6CnkYGsPdz20gx317bztkhK+eON55GrCU6YQd6e2uZutdW1sq2tjR307h9p6ATCDJYW5rC6ZzYWleawuyePcBTO1Zv4oCvQU4O785+8P84X/fpH0NOP//tkFvGX1OUGXJRKTtp4BdjW0s6uhg5317exsaOdYd6QXn5lunLtgFmtK8lgdfUD4ksLUfb6sAj3kOo4P8vn/eoH/2XWUteX53PsXF1KsySdJYu7OkY4+dtW3s6OhnV31HbxwuIPu/shYfE5WOucXz6Z4znSM5Av2my8pPuM7bk8V6PpbPMm9MvHZ1NnHZ29YwUeuWpKyPRcJDzOjOG86xXnTedMFC4DIUgT7j3Wzs76DnQ3t7GzoYOv+toArPTNXLkvM8gkK9CQ1duLzxx+9XBOfEmppacbSeTNZOm8mN19SEnQ5U5ICPQmNnvj880tK+FtNfIoICvSk4u785PeH+dvoxOeGd1zMH69eEHRZIjJFKNCTREfvIJ//qSY+ReTkFOhJYEd9O3d87zmau/o18SkiJ6VAn+JePNzBu+/fSl5OJj/56OWs0cSniJxETE8xMLP1ZlZtZrVmdvc4799rZjuiHzVm1h73SlNQbXM373lgG7OmZ/Lw7ZcpzEXklCbsoZtZOrABuB5oALab2UZ33/NKG3f/5Kj2HwcuSkCtKaW+rZd33b+VNDO+98FLtUqdiEwolh76WqDW3fe7+wDwEHDTKdrfCvwwHsWlqubOPt71za30Dgzx3dvWUl6QE3RJIpIEYgn0YqB+1HZDdN9rmNkioBx48iTv325mlWZW2dLScrq1poT23gHe/c1ttHT18+AH1nLuAj3eS0RiE+8nAd8C/Njdh8d7093vc/cKd68oLCyM86GTX3f/EO/91nbqWnu4/z0VXKyH8IrIaYgl0A8DpaO2S6L7xnMLGm45I32Dw3zw29t58XAHG95xMZcvTcxaDyISXrEE+nZgmZmVm1kWkdDeOLaRma0E5gC/i2+J4Tc4PMLHvv97tta18ZW3r+H6VUVBlyQiSWjCQHf3IeBO4FHgJeARd99tZveY2Y2jmt4CPORBrcebpIZHnE89spPNVc387z85n5suHHd6QkRkQjHdWOTum4BNY/Z9Ycz2F+NXVmpwd/7mpy/ws51HuPtNK3nnpYuCLklEkli8J0UlRu7O3/+iih9uq+djVy/hI1ctCbokEUlyCvSA/MuTtdz3zH7ee9kiPrNuRdDliEgIKNAD8K0tdXz58Rr+7OJi/vat52GmhbZE5Owp0CfZjyrr+V8/28MN5xXxDzevJk2rJopInCjQJ9GmF47y1z/ZxZXLCvj6rReRka5/fhGJHyXKJPlVTQt3PfQ8Fy2cwzfefQnZGelBlyQiIaNAnwRdfYN88uEdLCnM5YH3vY4ZWVqGXkTiT8kyCe57Zj9tPQM8+P7XMXt6ZtDliEhIqYeeYM2dfdz/6zresnoBq0vygi5HREJMgZ5g9z6xl6GRET57g641F5HEUqAnUG1zN49U1vPOSxexaK4eUiEiiaVAT6B/fLSKaRlp3HnN0qBLEZEUoEBPkOcOtvHo7iY+fNUSCnKzgy5HRFKAAj0B3J2/31RF4cxsPnhledDliEiKUKAnwBMvNVN58GX+8rpluuZcRCaNAj3OhoZH+NIvq1hckMPbK0on/h9EROJEgR5nP36ugdrmbv5q/QoytVaLiEwiJU4cHR8Y5t4narhoYR43nDc/6HJEJMUo0OPogS11NHX287k3nas1zkVk0inQ46StZ4B/f3of1507j7Xl+UGXIyIpSIEeJ//yZC09A0P89fqVQZciIilKgR4H9W29fPfZA/z5JaUsK5oZdDkikqIU6HHw5ceqSTPjL69fFnQpIpLCFOhn6cXDHfx0xxE+8EflLJg9PehyRCSFKdDP0pd+WUXejEw+ctWSoEsRkRSnQD8Lv97bwq/3HuPOq5fqSUQiEjgF+hkaGXH+3y+qKM6bzrsvWxR0OSIiCvQz9bNdR9h9pJPP3LCc7Iz0oMsREYkt0M1svZlVm1mtmd19kjZvN7M9ZrbbzH4Q3zKnlv6hYf7x0WpWLZjFTWuKgy5HRASACdd2NbN0YANwPdAAbDezje6+Z1SbZcDngCvc/WUzm5eogqeC7z97iIaXj/OdD1xAWppu8ReRqSGWHvpaoNbd97v7APAQcNOYNh8CNrj7ywDu3hzfMqeOzr5B/vnJvVyxdC5XLisIuhwRkRNiCfRioH7UdkN032jLgeVmtsXMnjWz9eN9IjO73cwqzayypaXlzCoO2Dd+tY+Xewe5e70W4BKRqSVek6IZwDLgjcCtwH+YWd7YRu5+n7tXuHtFYWFhnA49eZo6+/jmb+q4cc05XFAyO+hyREReJZZAPwyMfvROSXTfaA3ARncfdPc6oIZIwIfKV5+oYXjE+cy6FUGXIiLyGrEE+nZgmZmVm1kWcAuwcUybnxLpnWNmBUSGYPbHr8zgHTjWwyOVDbzz0kUsnDsj6HJERF5jwkB39yHgTuBR4CXgEXffbWb3mNmN0WaPAq1mtgd4Cvisu7cmqugg/MtTtWSkGXdcrVv8RWRqiumR9O6+Cdg0Zt8XRr124FPRj9A52NrDfz1/mPdeVsa8mdOCLkdEZFy6UzQGG56qJT3N+PBVi4MuRUTkpBToE6hv6+U/f3+Yd6xdSNEs9c5FZOpSoE/gX5+uJc1My+OKyJSnQD+Fhpd7+VFlA3/xulLmz1bvXESmNgX6Kfzr0/swg4++Ub1zEZn6FOgncaT9OD+qrOftFaWck6dHy4nI1KdAP4l/e3ofAHdcvTTgSkREYqNAH8fRjuM8vL2et11SQrF65yKSJBTo4/jGr/Yz4s4db1TvXESShwJ9jKbOPn6w7RA3X1xCab7WbBGR5KFAH+Pff7WP4RHnYxo7F5Eko0Afpbmzjx9sPcSfXlSsFRVFJOko0Ee575n9DI04d6p3LiJJSIEe1dLVz/e2HuSmC8+hrCAn6HJERE6bAj3qP369n4GhEfXORSRpKdCBY939fPd3B7lxzTksLswNuhwRkTOiQAfu/3UdfUPD3HlN6B6DKiIpJOUDva1ngO/87gBvXX0OS+epdy4iySvlA/3+X+/n+OAwH79GY+ciktxSOtDbewf49m8P8OYLFrCsaGbQ5YiInJWUDvRv/qaOnoFhPqGxcxEJgZQN9I7eQR7ccoA3XzCfFfPVOxeR5Jeygf7NLXV09Q/xcfXORSQkUjLQO44P8q0tddxwXhHnLpgVdDkiInGRkoH+4JYDdPUN8Ylr1TsXkfBIuUDv7Bvkm7/Zz/WrijjvnNlBlyMiEjcpF+jf3nKAzr4h7lLvXERCJqUCvbt/iPt/U8e1K+dxfrF65yISLjEFupmtN7NqM6s1s7vHef99ZtZiZjuiHx+Mf6ln79l9rXQcH+S2K8uDLkVEJO4yJmpgZunABuB6oAHYbmYb3X3PmKYPu/udCagxbqqbugBYXZIXbCEiIgkQSw99LVDr7vvdfQB4CLgpsWUlRnVjFyVzppObPeHvMRGRpBNLoBcD9aO2G6L7xrrZzHaZ2Y/NrDQu1cVZdWMXK7Rmi4iEVLwmRX8GlLn7auBx4NvjNTKz282s0swqW1pa4nTo2AwMjbCvpZvlus1fREIqlkA/DIzucZdE953g7q3u3h/dvB+4ZLxP5O73uXuFu1cUFhaeSb1n7EBrD0MjzkoFuoiEVCyBvh1YZmblZpYF3AJsHN3AzBaM2rwReCl+JcZHVWNkQnS5hlxEJKQmnB109yEzuxN4FEgHHnD33WZ2D1Dp7huBT5jZjcAQ0Aa8L4E1n5Gaxi4y0owlemaoiIRUTJd7uPsmYNOYfV8Y9fpzwOfiW1p8VTV2UV6QQ1ZGSt1LJSIpJGXSraapSxOiIhJqKRHoPf1DHGrrZaXGz0UkxFIi0Pc2dwOohy4ioZYSgV4TvcJFlyyKSJilRKBXNXYxPTOd0jkzgi5FRCRhUiLQa5q6WF6US1qaBV2KiEjCpESgVzV26YYiEQm90Ad6a3c/x7r7WaHxcxEJudAHek1T5AoXBbqIhF3oA726sRNAy+aKSOiFP9CbupkzI5PCmdlBlyIiklDhD/TGTpYXzcRMV7iISLiFOtDdnZqmbt1QJCIpIdSBfrj9ON39Q7rlX0RSQqgDvaYpcsu/JkRFJBWEOtCrG7Uol4ikjpAHeifnzJ7GrGmZQZciIpJw4Q70pm7dUCQiKSO0gT44PMK+5m4Nt4hIyghtoB9s7WFgeEQToiKSMkIb6FXRh1poyEVEUkVoA72msYv0NGNJYW7QpYiITIrQBnp1Uxdlc2cwLTM96FJERCZFeAO9sUvDLSKSUkIZ6McHhjnY1suKollBlyIiMmlCGeh7m7twhxXzNX4uIqkjlIFeHb3CRc8RFZFUEtpAz85IY9HcnKBLERGZNOEM9KYulhXlkp6mh1qISOqIKdDNbL2ZVZtZrZndfYp2N5uZm1lF/Eo8fTVNXZoQFZGUM2Ggm1k6sAF4E7AKuNXMVo3TbiZwF7A13kWejvbeAZo6+zUhKiIpJ5Ye+lqg1t33u/sA8BBw0zjt/g74EtAXx/pOmyZERSRVxRLoxUD9qO2G6L4TzOxioNTdf36qT2Rmt5tZpZlVtrS0nHaxsaiOPqVo5XwNuYhIajnrSVEzSwO+Anx6orbufp+7V7h7RWFh4dkeelzVjV3MmpZB0azshHx+EZGpKpZAPwyUjtouie57xUzgfOBpMzsAvB7YGNTEaE1TFyvnz8JMV7iISGqJJdC3A8vMrNzMsoBbgI2vvOnuHe5e4O5l7l4GPAvc6O6VCan4FNydqsYulmtCVERS0ISB7u5DwJ3Ao8BLwCPuvtvM7jGzGxNd4Olo7Oyjq2+IFRo/F5EUlBFLI3ffBGwas+8LJ2n7xrMv68yceKiFrnARkRQUqjtFaxToIpLCQhXo1Y1dzJ81jdkzMoMuRURk0oUr0Ju6WK6HWohIigpNoA+POHubu1mpQBeRFBWaQD/Q2sPA0Ihu+ReRlBWaQH9lQlQ9dBFJVaEJ9KrGLsxg6TzdVCQiqSk0gV7T1EXZ3BymZaYHXYqISCBCE+jVjV26/lxEUlooAr1vcJgDrT26ZFFEUlooAr22uZsR14SoiKS2UAS6nlIkIhKSQK9p6iIrI42yuTOCLkVEJDChCPSqxi6WFuaSkR6K0xEROSOhSMCapi5WaPxcRFJc0gd6R+8gRzv6FOgikvKSPtBrmrUGuogIhCDQX7nCRT10EUl1oQj0mdkZLJg9LehSREQClfyBHn2ohZkFXYqISKCSOtDdPbKGi4ZbRESSO9Cbu/rpOD6oCVEREZI80DUhKiLyB6EIdK3hIiKS7IHe1EXhzGzyc7KCLkVEJHDJHeiNXVoyV0QkKmkDfXjE2dvcpeEWEZGopA30Q2299A2OaEJURCQqpkA3s/VmVm1mtWZ29zjvf8TMXjCzHWb2GzNbFf9SX+3EFS7qoYuIADEEupmlAxuANwGrgFvHCewfuPsF7n4h8A/AV+Jd6Fg1TV2YwbKi3EQfSkQkKcTSQ18L1Lr7fncfAB4CbhrdwN07R23mAB6/EsdX3djFwvwZzMjKSPShRESSQixpWAzUj9puAC4d28jMPgZ8CsgCrhnvE5nZ7cDtAAsXLjzdWl+lukkToiIio8VtUtTdN7j7EuCvgb85SZv73L3C3SsKCwvP+Fj9Q8PUHevRJYsiIqPEEuiHgdJR2yXRfSfzEPAnZ1HThPY19zA84uqhi4iMEkugbweWmVm5mWUBtwAbRzcws2WjNv8Y2Bu/El+ruikyZK8euojIH0w4hu7uQ2Z2J/AokA484O67zeweoNLdNwJ3mtl1wCDwMvDeRBZd3dhNZrpRVpCTyMOIiCSVmC4RcfdNwKYx+74w6vVdca7rlGqaulhSmEtmetLeFyUiEndJmYh6qIWIyGslXaB39Q1yuP24JkRFRMZIukCvaYrc8q8JURGRV0u6QK9u7Ab0UAsRkbGSLtALcrO4flURJXOmB12KiMiUknQLoaw7bz7rzpsfdBkiIlNO0vXQRURkfAp0EZGQUKCLiISEAl1EJCQU6CIiIaFAFxEJCQW6iEhIKNBFRELC3BP+POfxD2zWAhw8w/+9ADgWx3KSgc45NeicU8PZnPMidx/3GZ6BBfrZMLNKd68Iuo7JpHNODTrn1JCoc9aQi4hISCjQRURCIlkD/b6gCwiAzjk16JxTQ0LOOSnH0EVE5LWStYcuIiJjKNBFREJiSge6ma03s2ozqzWzu8d5P9vMHo6+v9XMygIoM65iOOdPmdkeM9tlZpvNbFEQdcbTROc8qt3NZuZmlvSXuMVyzmb29ujXereZ/WCya4y3GL63F5rZU2b2fPT7+81B1BkvZvaAmTWb2Ysned/M7OvRf49dZnbxWR/U3afkB5AO7AMWA1nATmDVmDZ3AP8efX0L8HDQdU/COV8NzIi+/mgqnHO03UzgGeBZoCLouifh67wMeB6YE92eF3Tdk3DO9wEfjb5eBRwIuu6zPOc3ABcDL57k/TcDvwAMeD2w9WyPOZV76GuBWnff7+4DwEPATWPa3AR8O/r6x8C1ZmaTWGO8TXjO7v6Uu/dGN58FSia5xniL5esM8HfAl4C+ySwuQWI55w8BG9z9ZQB3b57kGuMtlnN2YFb09WzgyCTWF3fu/gzQdoomNwHf8YhngTwzW3A2x5zKgV4M1I/abojuG7eNuw8BHcDcSakuMWI559FuI/IbPplNeM7RP0VL3f3nk1lYAsXydV4OLDezLWb2rJmtn7TqEiOWc/4i8C4zawA2AR+fnNICc7o/7xNKuodES4SZvQuoAK4KupZEMrM04CvA+wIuZbJlEBl2eSORv8KeMbML3L09yKIS7FbgQXf/spldBnzXzM5395GgC0sWU7mHfhgoHbVdEt03bhszyyDyZ1rrpFSXGLGcM2Z2HfB54EZ375+k2hJlonOeCZwPPG1mB4iMNW5M8onRWL7ODcBGdx909zqghkjAJ6tYzvk24BEAd/8dMI3IIlZhFdPP++mYyoG+HVhmZuVmlkVk0nPjmDYbgfdGX78NeNKjsw1JasJzNrOLgG8QCfNkH1eFCc7Z3TvcvcDdy9y9jMi8wY3uXhlMuXERy/f2T4n0zjGzAiJDMPsnscZ4i+WcDwHXApjZuUQCvWVSq5xcG4H3RK92eT3Q4e5Hz+ozBj0TPMEs8ZuJ9Ez2AZ+P7ruHyA80RL7gPwJqgW3A4qBrnoRzfgJoAnZEPzYGXXOiz3lM26dJ8qtcYvw6G5Ghpj3AC8AtQdc8Cee8CthC5AqYHcC6oGs+y/P9IXAUGCTyF9dtwEeAj4z6Gm+I/nu8EI/va936LyISElN5yEVERE6DAl1EJCQU6CIiIaFAFxEJCQW6iEhIKNBFREJCgS4iEhL/H+y+fEay39iaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thresholds, scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "174b048b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e0897055",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 1409})"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(y_pred >= 1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8b8ece7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7260468417317246"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - y_val.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1600912",
   "metadata": {},
   "source": [
    "## 4.3 Confusion table\n",
    "\n",
    "* Different types of errors and correct decisions\n",
    "* Arranging them in a table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "501711e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "actual_positive = (y_val == 1)\n",
    "actual_negative = (y_val == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d898b9ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "t = 0.5\n",
    "predict_positive = (y_pred >= t)\n",
    "predict_negative = (y_pred < t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ce46cfef",
   "metadata": {},
   "outputs": [],
   "source": [
    "tp = (predict_positive & actual_positive).sum()\n",
    "tn = (predict_negative & actual_negative).sum()\n",
    "\n",
    "fp = (predict_positive & actual_negative).sum()\n",
    "fn = (predict_negative & actual_positive).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "778f4ef6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[922, 101],\n",
       "       [176, 210]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix = np.array([\n",
    "    [tn, fp],\n",
    "    [fn, tp]\n",
    "])\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4a7aab64",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.65, 0.07],\n",
       "       [0.12, 0.15]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(confusion_matrix / confusion_matrix.sum()).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a37b1d4c",
   "metadata": {},
   "source": [
    "## 4.4 Precision and Recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7dfcf399",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6752411575562701"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = tp / (tp + fp)\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4e180eec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5440414507772021"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = tp / (tp + fn)\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5088b30e",
   "metadata": {},
   "source": [
    "## 4.5 ROC Curves\n",
    "\n",
    "### TPR and FRP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f8b9350d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5440414507772021"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tpr = tp / (tp + fn)\n",
    "tpr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "0a8b418e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09872922776148582"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fpr = fp / (fp + tn)\n",
    "fpr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "fc8e1021",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "\n",
    "thresholds = np.linspace(0, 1, 101)\n",
    "\n",
    "for t in thresholds:\n",
    "    actual_positive = (y_val == 1)\n",
    "    actual_negative = (y_val == 0)\n",
    "    \n",
    "    predict_positive = (y_pred >= t)\n",
    "    predict_negative = (y_pred < t)\n",
    "\n",
    "    tp = (predict_positive & actual_positive).sum()\n",
    "    tn = (predict_negative & actual_negative).sum()\n",
    "\n",
    "    fp = (predict_positive & actual_negative).sum()\n",
    "    fn = (predict_negative & actual_positive).sum()\n",
    "    \n",
    "    scores.append((t, tp, fp, fn, tn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "76beba9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ['threshold', 'tp', 'fp', 'fn', 'tn']\n",
    "df_scores = pd.DataFrame(scores, columns=columns)\n",
    "\n",
    "df_scores['tpr'] = df_scores.tp / (df_scores.tp + df_scores.fn)\n",
    "df_scores['fpr'] = df_scores.fp / (df_scores.fp + df_scores.tn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "e125f2fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca7c6f9a0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxY0lEQVR4nO3dd3xUVfrH8c+ZSSeNFAiQCiRACD30IohIUxAsSFMUQVFE17Kyi7uruLqWVX+roAiiiAUUCyJSVBSpAUKvgUACBEgIAUINaef3xw0aEchAJrmZmef9euWV5M7NzHMJ+ebm3HOfo7TWCCGEcHwWswsQQghhHxLoQgjhJCTQhRDCSUigCyGEk5BAF0IIJ+Fm1guHhITo6Ohos15eCCEc0vr1649prUMv95hpgR4dHU1ycrJZLy+EEA5JKbX/So/JkIsQQjgJCXQhhHASEuhCCOEkTBtDF0IIeygoKCAjI4O8vDyzS7ErLy8vwsPDcXd3t/lrJNCFEA4tIyMDPz8/oqOjUUqZXY5daK3JyckhIyODmJgYm7+uzCEXpdQHSqmjSqltV3hcKaXeUkqlKqW2KKVaXkPdQghRLnl5eQQHBztNmAMopQgODr7mvzpsGUOfAfS6yuO9gdiSt9HAu9dUgRBClJMzhflF13NMZQa61noZcPwqu/QHZmpDEhColKp1zZXYaNfaH0h+/zEWbT3CvuwzFBVL+18hhAD7jKHXAQ6W+jyjZNuRS3dUSo3GOIsnMjLyul4sZ89aOmbMoFVqW3IIwMPNQr1QX+Jq+hJX04/YGsb7iCAfrBbn+60thKhacnJy6N69OwCZmZlYrVZCQ40bOTdv3kyzZs0oLCykUaNGfPTRR/j4+GC1WmnSpAmFhYXExMTw8ccfExgYWO5aKvWiqNZ6KjAVIDEx8bpOrTu2bgN7YPYdoWyiIXuOnmF31mmS00/w7abDv+1XzcPKjY1q0rdJLbo2CMXL3WqfgxBCiFKCg4PZtGkTAM899xy+vr489dRTAPj6+v722NChQ5kyZQpPPPEE3t7ev22/9957mTx5MhMmTCh3LfYI9ENARKnPw0u2VYzgegDEWo8S26LHHx46c6GQPVmn2ZN1ho0HT7B4exbfbT5MNQ8rPeJr0rdpbTrHhki4CyEqXefOndmyZcuftrdv3/6y26+HPQJ9HjBWKTUbaAvkaq3/NNxiN4GRoKxwfN+fHvL1dKNFZHVaRFbnrtYRvNC/mKR9x5m/5TCLtmcyd9Nh/DzdSsK9Fp1iQ/B0k3AXwlk8/912dhw+ZdfnjK/tz79ubVyu5ygsLGThwoX06vXH+SVFRUUsWbKEkSNHluv5Lyoz0JVSs4CuQIhSKgP4F+AOoLWeAiwA+gCpwDngPrtUdiVWdyPUj+8tc1c3q4VOsSF0ig3hhdsSWLU3h/mbD7N4eyZfbzyEv5cbLw5owq3NaldoyUII13T+/HmaN28OGGfoF4P74vZDhw7RqFEjevTocZVnsV2Zga61HlzG4xp4xC7V2Cq4HuSUHeiluVst3BAXyg1xobw4oAkrU4/x9s97eHTWRvZln2Vc9/pOOfVJCFdS3jNpeys9Vn657efOnaNnz55MnjyZcePGlfv1HLOXS1A9OJ4G+vqmLHq4WejWsAazRrdjYMs6vPnTbsbN3sTWjFzO5xfZuVghhLg8Hx8f3nrrLV5//XUKCwvL/XyOeet/UF3IPw1ns8G3xnU/jaebldfvbEb9Gr68uiiF7zYfRimIDPLhhrhQ+japRWJ0kEx/FEJUmBYtWtC0aVNmzZrF8OHDy/VcjhnoJTNdyNlbrkAH426sh7vW59amtdl2KJfdWWfYdjiXL5IPMnP1fmr4eXJP+yhGdIzB19Mx/7mEEJXjueee+8PnZ86cuex+l27/7rvv7PL6jplQQXWN98f3QVR7uzxlRJAPEUE+9G5ifH72QiE/7zrKVxsy+O8Pu3l/RRqjOteld0IYUcHV5KxdCFHlOGag/zZ18doujF6Lap5u3NqsNrc2q83mgyf5v59289riFF5bnIKHm4W6IdXw9zbaWipgYMs6DGp9fXe/CiGEPThmoFvdoXrUNc90uV7NIgL58L427M46zeaDJ9mddZrUo2c4X2BcQD1xtoBnvtrKziOnebZvI9ysjnmtWQjh2Bwz0KFkpsufby6qSHE1/Yir6fen7UXFmv8s2Mn7K9JIO3aWt4e0wN/L9qb0QghhD457KhlU1wj065y6aE9Wi+LZW+L5z0Bjfvsd767i4PFzZpclhHAxjhvowfUg/wycOWp2Jb8Z3CaSmfe3ITM3j/6TV5KcfrWuw0IIYV+OG+hBJVMXK3nYpSwd6ocw95GOBHi7M2TaGmatPYCuAn9FCCEqjtVqpXnz5r+9paens3TpUgICAmjevDmNGjXi+eefB/jD9oYNG/7WmdEeHDjQS9bZq8CZLterbqgv3zzcgTYxQfzt663cN2MdmbnOtYCtEOJ3F2/lv/gWHR0NGP1bNm3aRHJyMp988gkbNmz4w/aNGzcyf/58Vq5caZc6HDfQA6PA4lZpM12uVaCPBzPvb8Nzt8aTtC+HHm/+ylfrM+RsXQgXVK1aNVq1akVqauoftnt7e//WpMseHHeWi9XNCPUqNuRSmsWiGNExhq4NavDUnM08OWczC7dl8tLABGr4eZldnhDOZ+F4yNxq3+cMawK9X77qLqW7KsbExPDNN9/84fGcnBySkpL4xz/+QXZ29m/bT5w4wZ49e+jSpYtdSnXcQIeSmS5V8wy9tOiQanz+YHs+XJnGa4tTuPnNZTzQKYb42v7E1vCjTqA3FrnzVAiHdaWuisuXL6dFixZYLBbGjx9P48aNWbp0KcuXL6dZs2bs2bOHxx9/nLCwMLvU4diBHlwP9q8ypi5W8da3Vovigc516dqgBuO/2sJ/f9j922Ohfp70SQjjlma1aRVZXcJdiOtVxpl0ZevcuTPz58+/4va0tDTatWvHXXfd9dsZfnk4eKDXh4KzcPoI+DvGIhX1a/jy5ZgO5J4rYM/R06RknWb57mPMXneQj1bvp0mdAGbe34bq1TzMLlUIUcFiYmIYP348r7zyCrNmzSr38znuRVGAmiXN7DO3mVvHdQjwcScxOoihbaOYMrwV6//Rg1dub0JK1mmGvr+Gk+fyzS5RCFEJHnroIZYtW0Z6enq5n8vBAz3BeJ+52dw67MDX041BrSOZdk8iqdlnGPr+GnLPFZhdlhDCBpdrk9u1a9fLDrdcut3b25tDhw79NtWxPBw70L38oXqM/a9qm+iGuFDeG96KPVlnGDwtSeavCyFs5tiBDlCrKRzZYnYVdtWtQQ2m3tOK/Tln6T95BVszcs0uSQjhABw/0MOawIk0yHOu0OvaoAZfjumAm8XCne+t4ttNh+SmJCGuwBl/Nq7nmJwg0JsZ77O2m1tHBWhUy5+5j3SkUS1/Hpu9iQHvrOLX3dlO+Z9XiOvl5eVFTk6OU/1caK3JycnBy+vabkB07GmLYJyhgzHsEtXB3FoqQKifJ1882J4v12cw6edU7v1gLR3rBzPtnkR8PBz/2ydEeYWHh5ORkfGHOzCdgZeXF+Hh4df0NY6fCH5hUC3UqS6MXsrdamFwm0hubxnOp2v2M3H+Dp74fDPvDG0pNyEJl+fu7k5MTIzZZVQJjj/kohSENXWKqYtl8XCzcF/HGCb0acSi7Zm8ujjF7JKEEFWI4wc6GMMuR3dBoWvcjDOyUwxD20Yy5de9zF57wOxyhBBVhHMEeq2mUFwA2bvMrqRSKKV4rl9jOseGMP7rrdz34Vq2ZJw0uywhhMmcI9DDmhrvnXgc/VLuVgtThyfydM8GbDx4kn6TVjJqZjKHT543uzQhhEmcI9CD6oF7Nch0rhuMyuLtYeWRbvVZ/tduPHVzHCtTj9HzzWV8kXzQqaZwCSFs4xyBbrFAWILT3TFqKz8vd8beGMuix7oQX9ufv365hQc+SuZ8fpHZpQkhKpFzBDoYF0Yzt0JxsdmVmCYy2IdZo9rxj1vi+TnlKE/O2URxsZypC+EqbAp0pVQvpVSKUipVKTX+Mo9HKqV+UUptVEptUUr1sX+pZajVDPJPV+kl6SqDxaIY2SmGv/duxIKtmbz+o0xtFMJVlBnoSikrMBnoDcQDg5VS8Zfs9izwhda6BXA38I69Cy1TRDvj/X77rJ7t6B7oHMPgNhFM/mUvc5IPml2OEKIS2HKG3gZI1Vrv01rnA7OB/pfsowH/ko8DgMP2K9FGIbFQrYYEegmlFBP7J9CxfjB//2YrX67PMLskIUQFsyXQ6wClT/EySraV9hwwTCmVASwAHr3cEymlRiulkpVSyXbvu6CU0cslfaWxxqjA3Wrh3WGtaB0dxFNzNvPKol0ypi6EE7PXRdHBwAytdTjQB/hYKfWn59ZaT9VaJ2qtE0NDQ+300qVEdYRTGXByv/2f20H5e7nz0f1tGNwmkneX7mXUzGS+3XSInUdOcaFQZsEI4Uxsac51CIgo9Xl4ybbSRgK9ALTWq5VSXkAIcNQeRdosuqPxPn0lVI+u1JeuytytFl4akED9Gr68snAXS3YZ35ZqHlZm3N+G1tFBJlcohLAHW87Q1wGxSqkYpZQHxkXPeZfscwDoDqCUagR4AZXfyzK0EXhXl3H0y1DKmP2y9fmbWfR4Z94a3IJQP08e/Wwjx8+6Rg8cIZxdmYGutS4ExgKLgZ0Ys1m2K6UmKqX6lez2JDBKKbUZmAWM0GbcqmixGMMu6Ssq/aUdhaeblYZh/vRrVptJQ1py/Gw+T83ZLGPrQjgBm8bQtdYLtNZxWut6WusXS7b9U2s9r+TjHVrrjlrrZlrr5lrrHyqy6KuK6miMoefKrI6yJNQJ4NlbGvHzrqNMX5FmdjlCiHJynjtFLyo9ji7KNLxdFL0ah/HKol18tuaA9IARwoE5X6DXTADPANgvwy62UErxyh1NaVs3iL9/s5V7PlgrHRuFcFDOF+gWK0S1lzP0axDg7c7H97flhf6NSU4/Qc83l7Eu/bjZZQkhrpHzBToY4+jH98KpI2ZX4jAsFsXw9tEsfrwLoX6ejJ6ZzP6cs2aXJYS4Bs4Z6PW6Ge/3LDa3DgcUGezDByNao4H7Z6wj91yB2SUJIWzknIFeMwECImHXArMrcUjRIdV4b1grDhw/x5hP15Nf6LotiYVwJM4Z6EpBwz6wbylcOGN2NQ6pbd1gXh7YlFV7cxjwzkp2ZZ4yuyQhRBmcM9ABGvaFoguw92ezK3FYt7cK573hrcg6lcetb69g8i+p5BVI/xchqirnDfTIDuAVCCky7FIePRuHsfjxLtzcOIzXFqfQ6oUfeXz2Rn7ckSXhLkQVY0tzLsdkdYO4nrB7ERQVGp+L6xLs68nkIS0Z2vYY8zYdZtH2TOZuOoyfpxs94mvSt2ktOsWG4OlmNbtUIVyaMuvOwMTERJ2cnFyxL7J9Lsy5F0Z8D9GdKva1XEhBUTErU4+xYOsRFm3L5FReIX5ebtwcH8YtTWvRsX4IHm7O+8efEGZSSq3XWide9jGnDvQLp+HVutB6FPR6qWJfy0XlFxrh/v3WIyzensnpvEICvN3p2bgmfZvWpmO9YNysEu5C2IvrBjrAJ3dAzh4Yt8mY/SIqzIXCIlbsOcb3W47ww44szlwopHV0dd4bnkhQNQ+zyxPCKVwt0J3/1KlhHziRDoc3ml2J0/N0s9K9UU3eGNSc5Gdv4pXbm7AlI5f+k1ewJ+u02eUJ4fScP9AbDzRmuyx92exKXIqXu5VBrSOZPbod5/OLGfjOKj5O2i+LaQhRgZw/0L0DodPjRhuAA0lmV+NyWkRWZ97YjtQNrcY/5m6j9Ys/MXz6GlbsOWZ2aUI4HecPdIA2o8G3Jvz0PEi/70pXO9CbuY905PtxnXiwS132ZZ/lng/W8MGKNOm/LoQduUage1SDLk/DgVWwd4nZ1bgkpRSNawfw114N+fGJLtzUqCYT5+/g2bnbKCiSXjFC2INrBDpAy3shMBKWTIRiCRAz+Xi4MWVYK8Z0rcenaw4wbtZGOVMXwg5cJ9DdPKDr3+HIZtg8y+xqXJ7FonimV0PG927Iwm2ZzFiVbnZJQjg81wl0gKaDIKIdLP4bnM40uxoBPNilLjc1qsFLC3ayJeOk2eUI4dBcK9AtFug/GQovwPwn5AJpFaCU4r93NiPU15Oxn23kVJ4sqCHE9XKtQAcIqQ/d/g4p38P2r82uRgCBPh68NbgFh06e574P17HhwAmzSxLCIbleoAO0ewRqt4QFT8OZbLOrEUBidBCv3dGUtGNnGfjOKkZ8uJbNB0+aXZYQDsU1A93qZgy95J+Fr+432usK0w1sGc7yv3bjmV4N2XzwJP0nr2TkjHVszcg1uzQhHIJrBjpAzXjo+wakLYMlz5tdjShRzdONMV3rsfyZG3m6ZwOS95/g1kkreOCjdWw7JMEuxNU4f7fFssx/ApKnw50zoPEAs6sRlzidV8CMlelMW76PU3mF3Bxfk0e61adpeABKumcKF+Ta7XPLUpgPM/pA1g548FcIiTW7InEZp/IK+GBFGtNXpHE6r5DoYB/6Nq3F7S3DqRvqa3Z5QlQaCfSynDoMk9tBZFsYOsfsasRV5J4vYOHWI8zfcoRVe4/h4WZhzoMdaBIeYHZpQlQK1+6Hbgv/2tDlKdjzA6RKr5eqLMDbnbvbRPLJA21Z8cyNBFfzZORH6ziSe97s0oQwnU2BrpTqpZRKUUqlKqXGX2Gfu5RSO5RS25VSn9m3zErQ9kEIjIIfnoViWc3eEdQO9Gb6iETO5RcxckYyZy/IbCXh2soMdKWUFZgM9AbigcFKqfhL9okF/gZ01Fo3Bh63f6kVzM0TekyEoztg48dmVyNs1DDMn7eHtGBX5inGfraBvAL5ZSxcly1n6G2AVK31Pq11PjAb6H/JPqOAyVrrEwBa66P2LbOSxPc3er38/G9jgWnhELo1qMELtyWwdHc2d09N4ujpPLNLEsIUtgR6HeBgqc8zSraVFgfEKaVWKqWSlFK9LvdESqnRSqlkpVRydnYVvENTKej1EpzNhqR3za5GXIOhbaN4d2grUjJPc9uklew4fMrskoSodPa6KOoGxAJdgcHANKVU4KU7aa2naq0TtdaJoaGhdnppO6vTCuJ6QdI7cOGM2dWIa9ArIYw5D7WnWMMdU1bx044ss0sSolLZEuiHgIhSn4eXbCstA5intS7QWqcBuzEC3jF1fhLOn4ANH5ldibhGCXUC+HZsR+rX8GXUx8lMXbZXFs8QLsOWQF8HxCqlYpRSHsDdwLxL9pmLcXaOUioEYwhmn/3KrGQRbSC6M6x622i1KxxKTX8vPh/dnt4JYby0YBfjv9pKoSxzJ1xAmYGutS4ExgKLgZ3AF1rr7UqpiUqpfiW7LQZylFI7gF+Ap7XWORVVdKXo/CScPiKrGzkobw8rkwa3ZGy3+nyefJCn5mymqFjO1IVzkztFr0RrmHajMfQyNtno0Cgc0uRfUnltcQoDW9ThtTubYbVIDxjhuK52p6ik1JUoZZylfz4UtnwOLYaaXZG4To90q09xseb1H3dzoaiY21vWIa6mH3UCvaXBl3AqEuhX06APhLeBHyZAbA/wrWF2ReI6PdrduEb/xk+7+X7LEQCCqnkwoU8jBrasI8EunIIMuZQlezdM6QQNesFdM82uRpRT7rkC9hw9TUrWab7ecIj1+09wU6OavDQwgRp+XmaXJ0SZpDlXeYTGQdfxsONb2D7X7GpEOQX4uJMYHcTQtlF88WB7nu3biOV7sunxxjLeXbpX+sEIhyaBbosO46BWc1jwFJw7bnY1wk6sFsUDneuy4LHONI8I5JVFu+j86i+89+tezuVLsAvHI4Fui4trkJ4/AT89Z3Y1ws7qhfry0f1t+GpMBxrX9uc/C3fR5dVfeH/5Ps7nS7Mv4Tgk0G0VlgBtRhudGDO3mV2NqACtoqrz8ci2fDWmPQ3D/Pn39zvp8tovfLAiTbo4CocgF0Wvxbnj8FYLqN0chs81pjYKp7U27Thv/rib1ftyqOHnycNd6zG8fbTMYxemkoui9uITZFwg3bcU9vxodjWigrWJCWLW6HbMHt2OmJBqPPfdDl5dvMvssoS4Ign0a5U4EoLqGXPTiwrMrkZUgnZ1g/n8wfYMaRvJe7/u45cUx2z3L5yfBPq1cvOAm1+AY7ulZ7qL+ect8TQM8+PJLzaTmSuLaIiqRwL9ejToAw1vMWa87P3Z7GpEJfFytzJpSEvyCooYN3sj+YXSwVFULRLo10MpGDAFQhvAnBFwLNXsikQlqV/Dl3/flsDatOO0fvEn/vrlZn7dnS0910WVIIF+vTz9YPBssLjBrEHGHHXhEga2DGfm/W3o3rAGC7Zmcu8Ha/nfkj1mlyWEBHq5VI+CQZ/Cif2w+FmzqxGVqEtcKG8Mak7yszdxa7PaTPo5le2Hc80uS7g4CfTyimoPbUYZC2GcSDe7GlHJvNytvNC/MYE+Hjw9ZwsFsjKSMJEEuj10GGcMvSx/w+xKhAkCfTz4920J7DhyiilL95pdjnBhEuj24F8LWt0Lmz6DkwfMrkaYoFdCGLc0rcVbP+9hacpRuUgqTCGBbi8dHzdmv6z4P7MrESZ5vl9jwgK8GPHhOgZNTWL1XsdeVlc4Hgl0ewmoAy2GGc27cg+ZXY0wQbCvJz/+5QaeuzWe9GNnGTwtiX99u03O1kWlkUC3p05/AV0Mv75sdiXCJF7uVkZ0jGHZX7txX8doPlq9n+e/2yGhLiqFrClqT4GR0G4MrHobEu6AujeYXZEwiZe7lX/eEo9VKd5fkYZSRusAWbtUVCQ5Q7e3bhOM5l3zHoX8s2ZXI0yklGJC30bc1zGaD1emM2HuNpnWKCqUBLq9uXtD/0lwcj8secHsaoTJlFL885Z4xnStx2drDjDiw7XknpMunaJiSKBXhKgOxupGa6bAgSSzqxEmU0rxTK+GvHZHU9amHWfAuyvZnyN/vQn7k0CvKN3/BYERMOc+mfUiALgzMYJPRrbl+Nl8HvgoWRaiFnYngV5RPH2N5l35Z+DTOyFP+nwIaFs3mEmDW5KafYbn5m03uxzhZCTQK1LNxnDXTDiWAl/cA4X5ZlckqoBOsSGM7VafL5Iz+GZjhtnlCCcigV7R6nWDfm8b65B++wgUy+rxAh7rHkub6CAmfLONvdlnzC5HOAkJ9MrQfIgxpr71C2M6Y7FMXXN1blYL/xvcHA83C89/t8PscoSTkECvLJ2fgK5/h02fwnfjJNQFtQK8GdW5Lst2Z7PzyCmzyxFOwKZAV0r1UkqlKKVSlVLjr7Lf7UoprZRKtF+JTqTrM9DlaaPfy4KnQG4Hd3nD2kbh42Fl2rJ9ZpcinECZga6UsgKTgd5APDBYKRV/mf38gMeANfYu0ql0mwAdHoXk6bDiTbOrESYL8HFnUOsI5m0+zOGT580uRzg4W87Q2wCpWut9Wut8YDbQ/zL7vQC8AuTZsT7noxTcNBESboclz8PWL82uSJhsZKcYNPDhyjSzSxEOzpZArwMcLPV5Rsm23yilWgIRWuvvr/ZESqnRSqlkpVRydnb2NRfrNCwWuO1diOoIc8dA+gqzKxImCq/uQ98mtZi19iCn8qQtgLh+5b4oqpSyAG8AT5a1r9Z6qtY6UWudGBoaWt6XdmxunjDoE6geDbOGQOZWsysSJhrdpS5nLhTKWLooF1sC/RAQUerz8JJtF/kBCcBSpVQ60A6YJxdGbeATBMO+Nu4q/Xgg5Mh6lK4qoU4AfZvW4u2fU3lpwU6KiuWCubh2tgT6OiBWKRWjlPIA7gbmXXxQa52rtQ7RWkdrraOBJKCf1jq5Qip2NoERMHwu6CL4+DY4ddjsioRJ/jeoOfe0j2Lqsn08+PF6zl6QXi/i2pQZ6FrrQmAssBjYCXyhtd6ulJqolOpX0QW6hNA4GPolnDsOM/tLqLsoN6uFif0TmNi/Mb+kHOXWSSvYcOCE2WUJB6LMWhorMTFRJyfLSfwf7F9lNPKqFgL3zIPqUWZXJEyyKvUYT83ZTOapPEZ3qcfjN8Xi5W41uyxRBSil1mutLzukLXeKViVRHYwgP38SPugFx/aYXZEwSYf6ISz6SxfuSoxgyq97ueeDtRTKakeiDBLoVU14KxjxPRQXwPSbZUqjC/P3cufl25vyasnCGG/+tNvskkQVJ4FeFYUlwP2LwSfYGFNP/tDsioSJ7kqMYFBiBO8s3cuy3S58/4YokwR6VRVcDx74Cep2hfmPw8JnpKGXC3uuX2Nia/jyxBebOHpKbsYWlyeBXpV5B8KQL6Ddw8b6pHPHQJFMZXNF3h5WJg9pyZkLhTw2e5PMUxeXJYFe1Vms0PMluPFZ2DIbvrxPVj5yUbE1/ZjYP4HV+3KY9HOq2eWIKkgC3REoZbTd7fkf2DkPPh8GRdLzwxXd2SqcAS3q8L8lu1m9N8fsckQVI4HuSNo/DH3fgD2LYfHfza5GmEApxQu3JRAdXI3HZm8k58wFs0sSVYgEuqNpPRLaj4W1U2HddLOrESbw9XRj0pCWnDxfwLjZG8krkHVqhUEC3RH1mAixN8PCv0LaMrOrESaIr+3PfwY0YWVqDmM+Wc+FQgl1IYHumCxWuH06BNeHWYNh5f+gUP70djW3twrnPwOb8EtKNg9/skFCXUigOywvfxj2FUR3gh//CZPbws75sk6pixncJpIXBySwZNdRxnyyQTo0ujgJdEcWEA5DPjd6qrt5wudD4aNbIXOb2ZWJSjS0bRQvDkhgacpR7piyWtYmdWES6M6gfnd4aCX0+S9kbYf3OsP8v0D+ObMrE5VkaNsopo9ozcHj5+g/eSWbDp40uyRhAgl0Z2F1gzajYNwGaPuQ0f/lw16Qm2F2ZaKSdGtQg68f7oCnm4Xh76+RM3UXJIHubLyrQ6//GC0DcvbB1G5wcJ3ZVYlKElfTj08faEthsebv32zFrPUOhDkk0J1V3M1Gcy+PavBhb1j6ssyEcRFRwdV4umcDlqZk89WGQ2V/gXAaEujOrEZDGPUzxPeHpf+BKZ1h/2qzqxKVYESHaBKjqjPxu+1kSXdGlyGB7ux8guCO6caapQXnjbP1xROgQH7InZnFonj1jqZcKCzm2bky68lVSKC7itge8PBqSLwfVk+CaTfK9EYnVzfUl7/0iOPHHVmsTD1mdjmiEkiguxJPX7jlDRgyB85mw7RusGWO2VWJCjSiQzR1Ar15eeEuiqWHutOTQHdFcTcbZ+vhbeDrB4zWATIbwil5uVt58uY4th7K5futR8wuR1QwCXRXVS0Ehn8NjQcYrQMWjYdi6QXijPo3r0PDMD/++0MK+YWyjKEzk0B3ZW6ecPsHRjveNVPg4wFwRhYhdjZWi+KZ3g3Zn3OO2esOmF2OqEAS6K7OYoGeL0L/yXBwDUzpBPtXmV2VsLOucaG0qxvEmz/uJu3YWbPLERVEAl0YWgz7/UakGbfAd4/ByYNmVyXsRCnFv29rgkUpBk9NIl1C3SlJoIvfhTWB0UuNqY0bP4W3W8L3T8IxWZDYGdSv4cuno9qSX1TM4GlJ7M+RUHc2Eujij7z8oe9/YdxGaD4U1s+ASa1gek/YMBPOHDW7QlEODcP8+fSBtuQVFDFk2hrpn+5klFnNexITE3VycrIpry2uweks2DIbNn4Cx3Yb20IbQnRnYxm8ujcYF1eFQ1mXfpw7p6xmfO+GPHRDPbPLEddAKbVea5142cck0IVNtIYjm2DfUkhbDgeSoOAsePgZ89rbjoGI1mZXKa7BPR+sZfuhXJY/0w0fDzezyxE2ulqg2zTkopTqpZRKUUqlKqXGX+bxJ5RSO5RSW5RSS5RSUeUtWlQxSkHtFtDpL8b89WfSjDtOG98Ge3+B6TfBV6Ok/7oDeax7LDln8/k0SaYyOosyA10pZQUmA72BeGCwUir+kt02Aola66bAl8Cr9i5UVDFunsaZef9J8PhW6PI07JwHbyfCt2Nh9w/SrreKaxVVnc6xIby3bC/n8+WmMmdgyxl6GyBVa71Pa50PzAb6l95Ba/2L1vriemdJQLh9yxRVmqcv3PgsjF0HCQNh+1z47E54tR588xCkr5DWAlXUuO6xHDuTz6dr9ptdirADWwK9DlB6QnJGybYrGQksvNwDSqnRSqlkpVRydrbckeh0AiPhtnfgr3uNdr2Nb4Nd38OMvvBWc/jhH7DnR7hw2uxKRYnW0UF0qBfMe8v2kVcgZ+mOzq7TFpVSw4BE4LXLPa61nqq1TtRaJ4aGhtrzpUVV4uZptOvtPwmeTIEBUyEwCpLehU/vgJej4JsxcOGM2ZUK4NEbY8k+fYGvNsj1D0dny6XtQ0BEqc/DS7b9gVLqJmACcIPWWgZPhcHDB5oNMt7yzxntBXYvgrVT4dB6uGumsbKSME27ukE0DQ/g/eVp3N06EqtFmV2SuE62nKGvA2KVUjFKKQ/gbmBe6R2UUi2A94B+Wmu580RcnocP1OsGvV+B4XPh/HGjJ3vSFMiXuxbNopRidJe6pB07y487sswuR5RDmYGutS4ExgKLgZ3AF1rr7UqpiUqpfiW7vQb4AnOUUpuUUvOu8HRCGOreAA8uh/BEWPQMvNkYfnreuJFJVLpejcOICPJm6rK9ZpciykFuLBLm0tq4SSlpMuycD97VYeA0iL3J7Mpczker0vnXvO18+VB7EqODzC5HXEG5bywSosIoBVHtYdAn8Mga8K9tXDj9+d+y4EYluzMxnOo+7kz5dZ/ZpYjrJIEuqo7QBkYL3xZDYdlr8H532P4NFEkDqcrg4+HG8PbR/LQzi0XbMs0uR1wHCXRRtbh7G4ttDJwG50/CnBHwVgtjfH3Dx7DvVzgrK9hXlAe71KVlZCBjP9vAD9sl1B2NjKGLqqu4CFIWQtI7cGA16JL1MC3ukHA7tH8YajUzt0YndCqvgHumr2X74VzeHdqKm+Jrml2SKEW6LQrHV1QApw7Bif2QssA4Wy84azQMq9MKwppCdCcIllaw9nAqr4Dh769hx5FTfPtIJ+Jr+5tdkighgS6cz/mTxoIbKQshcyvknwaLG/T5LyTeZ3Z1TuHE2Xy6v/Er9UKr8cWD7VFKbjiqCmSWi3A+3oHQcRzcvxDGH4BHN0DdbjD/cVjwtFxItYPq1Tx4plcD1qWfYO6mP90cLqogCXTh+CwWY6hlyOfQfqzRVuDj2+DYHrMrc3h3toqgWXgALy3Yxem8ArPLEWWQQBfOw2KFni/Cbe/C4U3wTjtY9Dc4f8LsyhyWxaKY2D+BY2cu8L+f5BdkVSeBLpxP8yEwbgO0GAZrpsCbTeDLkUafdunweM2aRQQyKDGCD1els2ZfjtnliKuQi6LCuWVuNUI9ZSGcywE3b4jvZ4R9VCdjuEaU6cTZfO6YsoojuXl8dH8bWktrANPILBchigrhYBJs/RK2fQUXTkFAhNH9MboLxHQGvzCzq6zSjp7K4+6pSWSdymPmyDa0ipJQN4MEuhClFZw3GoFt/xrSV8KFXFAWaHkvdJsAvrL4ypVklYR69ukLzHmoPY1qyfz0yiaBLsSVFBdB5hbYNAuSp4O7D3T6C0R1NJbU860pwzKXyMzNo9+kFfh6ufHd2E5U87RlnRxhLxLoQtgiezf8MAH2/PD7NncfaNjXGHOP7iLhXmL13hyGvp/EbS3q8MZdzc0ux6VcLdDlV6sQF4XGwdA5kLMXju+DE+mQtc3o+Lh1DviHQ62mxpl79WiI6QI14o0WwC6mfb1gxnWP5f9+2kP7usHcmRhR9heJCieBLsSlguv9sSdMr1dg13zYMRdy9kHaMsgvmf5YPQYa3Qot74GQWFPKNcujN8aStC+Hf367HXerhVub1Zb1SE0mQy5CXCut4fQRY7HrnfONgC8ugNieRgdIFxqaOXoqj3s/XMfOI6eoF1qNcd1jubVpbSwS7BVGxtCFqEhnso0Lqmunwblj4F4NwhKMDpDNBxvdIJ1YcbFm4bZM/rdkN7uzztA7IYzX72qGj4cMAFQECXQhKkNBHuz8Dg4lw5EtcGSz0eK32WDo/i/wr2V2hRWquFgzfUUaLy3cSePa/rx/T2vCArzMLsvpSKALYYa8U7D8dWOBDoubMc+9xTDj7N2JLdmZxbhZG/H1cmPGfW1krrqdSaALYabjacai1zu+NcbaazU37kwNjDLewhPBx7nuutx55BT3fbiO/KJiPhvVloZhEur2IoEuRFVwNseY/rhlNmTtgKILxnarBzToDc2HQb0bweocY8/px85y99QkCoqKmTW6HXE1/cwuySlIoAtR1RQXw9mjxnz3nd/Bls+N5mGe/hDVAaI7Q2Q7Y567h4/Z1V63tGNnGfTeaoq15oMRrWkaHmh2SQ5PAl2Iqq4wH/YshtSfIG05HN9rbFcWCIkz1kttPgRqt3S4G5n2Zp9h2PtrOHr6Ao90q8/YbvXxcHONaZ0VQQJdCEdz6jAcWv/7bJm0X6Ewzzhjb9QPajUz7lr1r+MQAZ97roDn52/n6w2HaFTLnyd6xNElLgRPN6vZpTkcCXQhHN35k0Z3yI2fGkFPyc+tTzCENTHmvIc1NUI+uL6xelMV9OOOLCZ8s5Wjpy/g5+lGj/iajL2xPnVDfc0uzWFIoAvhTC6chqztxtl7Zsnb0Z1QlG887uYNNRqWzKKJhKC6RtjXjAd3b3NrBwqKilmZeozvtxxh4bZM/L3cmDu2IzX8ZM66LSTQhXB2RQWQnWKE+5EtkL0LTh6A3IO/B72yGk3FvKuDVwD41YK4m6H+TeBRzZSytx3K5c4pq4kL82P2qHZ4e1TNvyyqEgl0IVxVcTGc3G8sxZe5xegkmXcS8nKNGTbnT4CblxHqzQZDXE+wuldqiT9sz+TBT9bTOyGMSYNbSh+YMkigCyH+rKgQDqwyGoztmAtnssAnBJrcAeGtjQuvQfUqpdHYtGX7eHHBTuJq+hJfy5+4MD/axgTRIqK6BPwlyh3oSqlewP8AK/C+1vrlSx73BGYCrYAcYJDWOv1qzymBLkQVUlQIe5fAxk+MLpKlx+Orl4zFB0QYd7R6BZS8Bf7+sXfJx57+13VBVmvNjFXpLE3JZnfWaY7k5gFQK8CLPk1qcVdiBA3C5MYkKGegK6WswG6gB5ABrAMGa613lNrnYaCp1vohpdTdwACt9aCrPa8EuhBVVGG+MQafuRWO7jAW+ji5H3IzjNk2lHES6On/e+AH1DF+GQRGGmf/pcP/4j4e1f409TL3XAE/p2Tx/ZYjLNt9jPyiYvo2rcXj3WOJdfE7Tsu7YlEbIFVrva/kyWYD/YEdpfbpDzxX8vGXwCSllNJmjecIIa6fm4cx/bFW0z8/VlxsLO5xcRw+L9cI+Qunfv/4t+0njF8C+1cZj1+Jsv4e9FYPAAKAASVvRWGaE+fyOZlSQPEuTboT3JSU0+pxWvV9wO7Pa0ug1wEOlvo8A2h7pX201oVKqVwgGDhWeiel1GhgNEBkZOR1liyEMI3FAl7+xtu1OH8Szh//c+hf+ouhuOBPX2oFQgD/wmLSc85y/EJhuQ/DbB6+FdOMrVK7AGmtpwJTwRhyqczXFkKYyDvQeCsHDyDOHrU4MVv+djkElF4BNrxk22X3UUq5YfzFlGOPAoUQQtjGlkBfB8QqpWKUUh7A3cC8S/aZB9xb8vEdwM8yfi6EEJWrzCGXkjHxscBijOGsD7TW25VSE4FkrfU8YDrwsVIqFTiOEfpCCCEqkU1j6FrrBcCCS7b9s9THecCd9i1NCCHEtXD8+T9CCCEACXQhhHAaEuhCCOEkJNCFEMJJmNZtUSmVDey/zi8P4ZK7UF2AHLNrkGN2DeU55iitdejlHjAt0MtDKZV8peY0zkqO2TXIMbuGijpmGXIRQggnIYEuhBBOwlEDfarZBZhAjtk1yDG7hgo5ZoccQxdCCPFnjnqGLoQQ4hIS6EII4SSqdKArpXoppVKUUqlKqfGXedxTKfV5yeNrlFLRJpRpVzYc8xNKqR1KqS1KqSVKqSgz6rSnso651H63K6W0Usrhp7jZcsxKqbtKvtfblVKfVXaN9mbD/+1IpdQvSqmNJf+/+5hRp70opT5QSh1VSm27wuNKKfVWyb/HFqVUy3K/qNa6Sr5htOrdC9TFWKxkMxB/yT4PA1NKPr4b+NzsuivhmLsBPiUfj3GFYy7Zzw9YBiQBiWbXXQnf51hgI1C95PMaZtddCcc8FRhT8nE8kG523eU85i5AS2DbFR7vAywEFNAOWFPe16zKZ+i/LU6ttc4HLi5OXVp/4KOSj78Euit1yfLhjqXMY9Za/6K1PlfyaRLGClKOzJbvM8ALwCtAXmUWV0FsOeZRwGSt9QkArfXRSq7R3mw5Zg1cXKw0ADhcifXZndZ6Gcb6EFfSH5ipDUlAoFKqVnlesyoH+uUWp65zpX201oXAxcWpHZUtx1zaSIzf8I6szGMu+VM0Qmv9fWUWVoFs+T7HAXFKqZVKqSSlVK9Kq65i2HLMzwHDlFIZGOsvPFo5pZnmWn/ey1Spi0QL+1FKDQMSgRvMrqUiKaUswBvACJNLqWxuGMMuXTH+ClumlGqitT5pZlEVbDAwQ2v9ulKqPcYqaAla62KzC3MUVfkM3RUXp7blmFFK3QRMAPpprS9UUm0Vpaxj9gMSgKVKqXSMscZ5Dn5h1JbvcwYwT2tdoLVOA3ZjBLyjsuWYRwJfAGitVwNeGE2snJVNP+/XoioHuisuTl3mMSulWgDvYYS5o4+rQhnHrLXO1VqHaK2jtdbRGNcN+mmtk80p1y5s+b89F+PsHKVUCMYQzL5KrNHebDnmA0B3AKVUI4xAz67UKivXPOCektku7YBcrfWRcj2j2VeCy7hK3AfjzGQvMKFk20SMH2gwvuFzgFRgLVDX7Jor4Zh/ArKATSVv88yuuaKP+ZJ9l+Lgs1xs/D4rjKGmHcBW4G6za66EY44HVmLMgNkE3Gx2zeU83lnAEaAA4y+ukcBDwEOlvseTS/49ttrj/7Xc+i+EEE6iKg+5CCGEuAYS6EII4SQk0IUQwklIoAshhJOQQBdCCCchgS6EEE5CAl0IIZzE/wMPg5r1WtqrpQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_scores.threshold, df_scores['tpr'], label='TPR')\n",
    "plt.plot(df_scores.threshold, df_scores['fpr'], label='FPR')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f09dc644",
   "metadata": {},
   "source": [
    "### Random model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b3a4b466",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "y_rand = np.random.uniform(0, 1, size=len(y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c6c24296",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5017743080198722"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((y_rand >= 0.5) == y_val).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ba13786f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tpr_fpr_dataframe(y_val, y_pred):\n",
    "    scores = []\n",
    "\n",
    "    thresholds = np.linspace(0, 1, 101)\n",
    "\n",
    "    for t in thresholds:\n",
    "        actual_positive = (y_val == 1)\n",
    "        actual_negative = (y_val == 0)\n",
    "\n",
    "        predict_positive = (y_pred >= t)\n",
    "        predict_negative = (y_pred < t)\n",
    "\n",
    "        tp = (predict_positive & actual_positive).sum()\n",
    "        tn = (predict_negative & actual_negative).sum()\n",
    "\n",
    "        fp = (predict_positive & actual_negative).sum()\n",
    "        fn = (predict_negative & actual_positive).sum()\n",
    "\n",
    "        scores.append((t, tp, fp, fn, tn))\n",
    "\n",
    "    columns = ['threshold', 'tp', 'fp', 'fn', 'tn']\n",
    "    df_scores = pd.DataFrame(scores, columns=columns)\n",
    "\n",
    "    df_scores['tpr'] = df_scores.tp / (df_scores.tp + df_scores.fn)\n",
    "    df_scores['fpr'] = df_scores.fp / (df_scores.fp + df_scores.tn)\n",
    "    \n",
    "    return df_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "f7af4bd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_rand = tpr_fpr_dataframe(y_val, y_rand)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "6ea930c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca7bec8b0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsIElEQVR4nO3dd3QV1drH8e9zUkmoIYEAofceMCBFmoA0ARVBUBQ0gDRBULzgvRbw1SsgqCiIIE1EROkdBGnSA4TeewAhhF5CErLfP070IgIJ5CSTc87zWStr5cxMZp5t4OewZ8/eYoxBKaWU87NZXYBSSinH0EBXSikXoYGulFIuQgNdKaVchAa6Ukq5CE+rLhwYGGgKFSpk1eWVUsopbdmy5bwxJuhe+ywL9EKFChEREWHV5ZVSyimJyPH77dMuF6WUchEa6Eop5SI00JVSykVY1oeulFKOEB8fT1RUFLGxsVaX4lC+vr6EhITg5eWV4p/RQFdKObWoqCiyZMlCoUKFEBGry3EIYwwxMTFERUVRuHDhFP9csl0uIjJeRM6JyK777BcRGSEih0Rkh4hUfoi6lVIqVWJjY8mZM6fLhDmAiJAzZ86H/ldHSvrQJwKNH7C/CVA86asL8M1DVaCUUqnkSmH+p0dpU7KBboxZDVx4wCEtge+N3QYgu4jkeehKUujEgUjWj+3DrdjraXUJpZRySo4Y5ZIPOHnH56ikbf8gIl1EJEJEIqKjox/pYqc3zqT6qfFEDwnj6ObFj3QOpZRylJiYGEJDQwkNDSU4OJh8+fL99VlECA0NpVy5crRu3ZobN24A4OHh8df25s2bc+nSJYfUkq7DFo0xY4wxYcaYsKCge765mqxqLw8isu4ESLxN4QUvsOObDly/6VpPt5VSziNnzpxERkYSGRlJ165d6dOnz1+f/f39iYyMZNeuXXh7ezN69GgAMmXK9Nf2gIAARo4c6ZBaHBHop4D8d3wOSdqWZkLrPkeWPptZEdCWCmdnM2NIJ75ddZgbcQlpeVmllHpktWrV4tChQ//YXr16dU6dckxkOmLY4lygp4j8BDwOXDbGnHHAeR8oW7Zs1Ov1LeemefPK3u/puaQQtdfU5aOWZWlSPs268JVSGdjAebvZc/qKQ89ZJm9WPmheNlXnSEhIYNGiRTRu/PfxJbdv32b58uWEh4en6vx/SsmwxanAeqCkiESJSLiIdBWRrkmHLASOAIeAsUB3h1SWQrlaDYP8j/Ol3zge9z9Dtylb6TV1Gxevx6VnGUop9Q83b94kNDSUsLAwChQo8Fdw/7k9ODiYs2fP0rBhQ4dcL9k7dGNMu2T2G6CHQ6p5FJ7e0HoSHmPq8LXHcCrVGcWna86w7vB5etYrRtuqBfD18rCsPKVU+kntnbSj/dlXfr/tN27coFGjRowcOZJevXql+nquMZdL1jzQZjJy9Syd9nVhUfvcFAnMzIfz9lB36EombzhOwu1Eq6tUSqm/8fPzY8SIEQwbNoyEhNQ/A3SNQAco8Dh0nA8JNyk+rxXTmsCUTo8TkiMT783eRavR6zl07qrVVSql1N9UqlSJChUqMHXq1FSfS+w9JukvLCzMpMkCFxeOwg+t4HIU1BuAqdaDebvP8/6cXdyIu02/p0oS/kRhbDbXe7NMKXe0d+9eSpcubXUZaeJebRORLcaYsHsd7zp36H8KKAzhv0LxhrDsQ2Tsk7QIPMPSPrWpUyKIjxfuZejS/VZXqZRSDueasy3654S2U2DvPFjwNox9klxZ8jImuDy/FczLwFVnKRzoT5uw/MmfSymlnIRrBvqfSjeHwrVh2xQ4E4mc2cGT53+lno9h2dwwdsW/TblqjcAFJ/ZRSrkf1w50AN9sUP1/Q+Pl6h/cWvct1TaMJeuSF4jdVhHfWr2gTEvwSPlE8kopldG4Xh96crIE49voAy6/Hskn0plz0dEwIxy+rAgnNlhdnVJKPTL3C/Qk+YMDearDuzSM+4zBOQZiPHxgShv4Y6fVpSml1CNx20AHCCsUwJDWoXxzpjgfBw7G+GSGyc9BzGGrS1NKOZE/p8P98+vYsWOsXLmSbNmyERoaSunSpRk4cCDA37aXKlWKt99+22F1uHWgA7QMzUfv+sX5bmc8TS705fKNWGJGN2Xr5rVYNUZfKeVc/nyV/8+vQoUKAfYZFiMjI4mIiOCHH35g69atf9u+bds25s+fz9q1ax1Sh9sHOsCbDYoz8sXK1K5Rk89zf4JX/FVC5zdj7eCWbNu60erylFJOzt/fn8cee+wf0+dmypSJ0NDQDDV9rtMTEZpVyEOzCnmA0ty60pjds/5L5aNT8J3TiAOLSuJfsh75Kj0FBZ+wTwimlMp4FvV3/HOw4PLQ5NMHHvLn7IkAhQsXZtasWX/bHxMTw4YNG3jvvfe4c7W2ixcvcvDgQWrXru2QUjXQ78EnaxDlOwwn9tK/2D53OB5HV1B45xjY9Q3x/sF4VXsdHusIfgFWl6qUygDuN6vimjVrqFSpEjabjf79+1O2bFlWrlzJmjVrqFixIgcPHuTNN98kODjYIXVooD+Ab/bcVHplMLHxt5m6dh+Rq2fz/LXF1Fg+EFYPhXKtoNLLkL+qvpykVEaQzJ10eqtVqxbz58+/7/ajR49SrVo12rRp89cdfmpoH3oK+Hp58ErdsvTr3Zf/C/iEpnGfsj/oKcyumTD+Kfi6iv1tVH2IqpR6CIULF6Z///4MHjzYIefTQH8IebJlYnq36oSUqkKjI23oFjyVqNqfgU9mmNMdvm+hQx6VUg+la9eurF69mmPHjqX6XK43fW46SEw0jF1zhFErD3P5ZjwNSwXxScEtBG34BG7HQf33oVp37YZRKh3o9Ln/o3foj8BmE16vU5Q1/6pH34Yl2HDsIk+uKsLGpoug6JOw5F2Y1h5uXrK6VKWUG9FAT4Wsvl70ql+cRb1rkS97Jl6cdoLJhf4LT30MBxbDmDpwZrvVZSql3IQGugOE5PBjerca1CkRxHtzdjPsWkPouBBux8O4RrB7VvInUUo9Mld8q/tR2qSB7iCZfTwZ+0oYbcJC+Oq3Q/x8Li90WWl/KeGXjrDiv5CoC1Ur5Wi+vr7ExMS4VKgbY4iJicHX1/ehfk7HoTuQh034+NnynLkcy7szd5I//HGqd5wP8/vAqk8hahM0HgxBJawuVSmXERISQlRU1N/ewHQFvr6+hISEPNTP6CiXNHD5ZjytvllH9NVbzOpegyKB/rD5O1g+COJvQJVOUOdf+qapUuqh6SiXdJYtkxfjO1TBwyaET4rg4o14qNoZ3tgKlV+BTWPg83KwsJ+OW1dKOYwGehopkNOPMS8/xqmLN+n6wxbiEhIhcxA8/Tl0W2df8i5iAnz1GMzqCreuWl2yUsrJaaCnobBCAQxtXYGNRy/w7qyd/3tok6s0PPsN9NkFNXvBjmkwph6c3WNtwUopp6YPRdNYy9B8HIm+zpfLD3Ii5gb+Ph4AFAjwo2n5PITVH4hH8adg+msw9kl4ejiEvmhx1UopZ6SBng7ebFCc2ITbrD8cQ2zCbYyB9UdimLT+OLmy+NC0fB5aNp1H6Ka3kNnd4Pg6aDoUvDJZXbpSyonoKBeLXL+VwG/7zjF/x2lW7o/mVkIiebN4MTz3IqpFjYfc5aHNJMhZ1OpSlVIZyINGuWigZwDXbiWwfO9Z5kaeZvm+c7yUYx8DE0fgiYHW46FYA6tLVEplEKketigijUVkv4gcEpH+99hfQERWiMg2EdkhIk1TW7Q7yezjScvQfIzrWIUJr1Zh+e1Q6l0dxDnPXJgprWH9SJ1rXSmVrGQDXUQ8gJFAE6AM0E5Eytx12H+An40xlYC2wChHF+ou6pXMxZI+talSsSJ1YwYQ6VfDPnvjjHCIPmB1eUqpDCwlD0WrAoeMMUcAROQnoCVw5xg7A2RN+j4bcNqRRbqbbJm8GNamImXzZeP5Bd4Myl6AF/fOQHbNgOKNoEZPKFRL51tXSv1NSgI9H3Dyjs9RwON3HfMhsFRE3gD8gXt2+opIF6ALQIECBR62VrciIoQ/UZjCgX688aMXE70aMLrsNooenQqTmkPBmlDvXSj0hNWlKqUyCEe9WNQOmGiMCQGaApNF5B/nNsaMMcaEGWPCgoKCHHRp1/ZkqdzM6lETr6y5qL+lOu/k/5Gb9f8LMYdgYjOY/Bxcc61JiZRSjyYlgX4KyH/H55CkbXcKB34GMMasB3yBQEcUqKBE7izM7lGT3vWLM3NnDLVWFWdC2GziG3xkH7M+th78sdPqMpVSFktJoG8GiotIYRHxxv7Qc+5dx5wA6gOISGnsga63jQ7k7WmjT8MSzO5Rk5LBWRi4+Cg1VpVhXth4TOJtGPcU7Ln716KUcifJBroxJgHoCSwB9mIfzbJbRAaJSIukw94COovIdmAq0NG40mzzGUi5fNmY0qka07pUo2iQP2+sNHTyGUpsQEn4+RX7hF9KKbekLxY5MWMMcyJP88Hc3ZBwkzlBYyh04Xdu1htEpjq9rS5PKZUGdD50FyUiPFMpH0v71CasaF4anu7C/NuPk2nF+0z6uAsr9+roUaXcid6huwhjDFEXb3LgzCXyrXmHUn/M43BiHjYV6EyzF3uQ1e/h1iZUSmVMeofuBkSE/AF+1C+bl1KvTybu+clkyexPu6hBnB8axo7f51ldolIqjWmguyIRvMu1INfbmzlSbxS+xFNhWXsiRrzE9Us6+EgpV6WB7spsNorUeYmAfhGsDX6Z0JiFxH4RxrSZMzh07prV1SmlHEz70N3Iri2/k3NhZ3IkRPNG/BuczFWPpyvkoVmFvBQO9Le6PKVUCuh86Op/rp8nbvLzeP6xnbFZuvPf6OqAUDZvVjrXKkLzinnxsOmkX0plVPpQVP2PfyDery3AVrwhr1/9mgMFhzIh7DgmIZ43p0Xy1OermBN5isREfS9MKWejge6OvP2h7Y/QbDjeCdeot2sAC+jJorBtZJXr9P4pkvbjNnLywg2rK1VKPQTtcnF3iYlwcCms/xqOrcF4+XMgb0teP1aHaJOdfzcrQ7uq+RGde12pDOFBXS4pmQ9duTKbDUo2tn+d2YFsGEXJnb/wm/dsfsnUhvdn3WTPmct81LKchrpSGZx2uaj/yVMBnh0NPTZiK/okL1yZyMasA/DZPJohM35H51tTKmPTQFf/lLMotJ0Cr8wlW64Q3vP6gb47W7LvyxaYs7utrk4pdR8a6Or+itRBOi3DdN/A5uAXCL64lYRvanN07mD7HOxKqQxFA10lS3KVpnrXUSyvP4/1UonCWz9h16dPsmX3PqtLU0rdQQNdpYiI8HztylQdsJi1ZT6gWNxeMk9rRafRS9l09ILV5Sml0GGL6hHFHVyBx4+t2W0K80JsfwoGB9G8Yl6alc9DIZ1GQKk0o2+KKofzLl4PjzYTKS+HWJ5vLNm8Exm6ZD91P1vJ178dtLo8pdySBrp6dKWfRlp8Rd6Y9UxL6Mu2Zy7SpEwQny87yK5Tl62uTim3o4GuUqdSe3jxZ/DKRI7FPfj6UncaZ9rL279sJy4h0erqlHIrGugq9Uo0gtfXQOtJeIjh69sf0Sh6It+s2G91ZUq5FQ105Rg2G5R9Bl5fjVR4gT5eM3hsTWd27j9gdWVKuQ0NdOVY3v7w7GiuPzWcKrZ9hPxYjynf/pcj565aXZlSLk+HLao0c+nYDq5N707ItZ2sTSzLd36dOOJRGID6pXIzoGkpvDz0nkKph6ErFinrJCZybd0YvFYMwuf2dU74lmC1XyMGn65IhWL5GfXiY2Tz87K6SqWcho5DV9ax2cj8RFd83toFjQdTILsP7S98xcbs/8bj2GqeHbWWo+evW12lUi5BA12lD78AqNYVuv4O4b/ilzk733t+zGvXxvDsiOVM3nBcl71TKpU00FX6y18VXl8FVV+nPQv41ettdsz7mg7frdVl75RKBQ10ZQ2vTNB0CLwyl8Dc+RjqNYaPT73G518MZuqmE7qYhlKPQANdWatIHaTzb9BuGsFBQQy3fYHfvNfpNu43Tl+6aXV1SjmVFAW6iDQWkf0ickhE+t/nmDYiskdEdovIj44tU7k0ESjZGO9uq0is+2+ae27ig5OdGTDsKwbN28O5q7FWV6iUU0g20EXEAxgJNAHKAO1EpMxdxxQHBgA1jTFlgTcdX6pyeR6e2Oq+g63TrwQGZGeSx8eU3tSfFkPmMmTxPmLjdZUkpR4kJXfoVYFDxpgjxpg44Ceg5V3HdAZGGmMuAhhjzjm2TOVW8lXGq/taeKIvz3ut5VfvfpxePYmnR6xh+8lLVlenVIaVkkDPB5y843NU0rY7lQBKiMhaEdkgIo3vdSIR6SIiESISER0d/WgVK/fglQkafIB0WUWW4KJ84T2Kz6/1Y+Do7/li2QF9aKrUPTjqoagnUByoC7QDxopI9rsPMsaMMcaEGWPCgoKCHHRp5dKCy0H4Mmg5irJ+l5jp9R65V77Dp/N3aqgrdZeUBPopIP8dn0OStt0pCphrjIk3xhwFDmAPeKVSz2aDSi9h67UVU6MX7TxXELapN0MXbNdQV+oOKQn0zUBxESksIt5AW2DuXcfMxn53jogEYu+COeK4MpUCfLIgT32EaTqMhh5bqbGxO8Pmb9VQVypJsoFujEkAegJLgL3Az8aY3SIySERaJB22BIgRkT3ACqCfMSYmrYpW7k2qdiKx5TfU8NhL082vMm7sCGJvxVldllKW09kWldMy+xZwdfbbZI09zR+2YPye7EvWGp3tXTRKuSidbVG5JCnVjKz9drKt2gjO3M5K1mXvsPeLp9m4+zAJt3U9U+V+9A5duYQ9py6zZ85QWp4dxR8mgH62PiTkrkTx3FmoGJKNZyrlw9fLw+oylUo1XeBCuY3Yoxsxv3TE8+Z5Ps3+IdMvFefyzXhyZfGhW92itKtaQINdOTUNdOVerp+HSS3gwhHMi9PYSDm+WHaADUcukDurD93rFuOFKvk12JVT0j505V78A6HDXMhRCJnalmq2ffzUpTpTO1ejYIA/H8zdTd2hK5m07hiXbujoGOU69A5dua5r52BiM7h0EuoNgGo9MDYP1h2O4fNfDxBx/CKeNuGJ4oE8E5qPFhXzYrOJ1VUr9UDa5aLc17VzMK837F8IwRWgxQjIWwljDDtPXWbBjjPM33GGU5du0r5aAT5qWQ4RDXWVcWmgK/dmDOydCwv72QO+XCuo8y8IKpG02/Dp4n18u+oIr1QvyMAWZTXUVYb1oED3TO9ilEp3IlCmJRSuA79/DpvGwO6ZUOEFaPAhkiWY/o1LkZhoGLvmKDYRPmheRkNdOR19KKrcR6bs0HAg9N4B1XvArpnwdVWImIAYw7tNSxP+RGEmrjvGm9MidUEN5XQ00JX7yRwET/0fdFsHeSrA/DdhYjPk7C7+06w0/RqVZE7kadqN3UD01VtWV6tUimmgK/cVWAw6zIMWX0P0XhhdC5nXmx5VsvHNS5XZe+YKz4xcy4mYG1ZXqlSKaKAr9yYClV+GXtvg8a4QOQVGVKLJmZHMfqkg1+MSeHXiJi7fjLe6UqWSpYGuFECmHNDkU+i2Hoo3hPWjKDXtCZYVmIS5cJTuU7YQrxN+qQxOA12pOwWVgNYToPd2qN6dwNMrWerTn7JHJ/Lh7EhdTENlaBroSt1L9vz2B6c9NuFZ/Ene9ZrKi9s78tPM6VZXptR9aaAr9SDZ8kHbH0ls/T0h3tdpt7MT+0a2sU8noFQGo4GuVHJEsJVtSZa3I/k1qAOFzv1G/IjHYM0wuK0PS1XGoYGuVArZfLPwZLcvGVr8B5bFV4Dlg+DbOhClU1iojEEDXamH4GET3n2xEUvLfUbnuL5cuxQN3zWApf+BBH0JSVlLA12ph+RhEz5rXRG/Ci2oduUT9uR9DtZ9BWPqwdndVpen3JgGulKPwMMmDGtdkXoVi9H0SCsGBwzk5qUzmDF1IXKq1eUpN6WBrtQj8vSw8XmbivRrVJKl8aHUvPIJ6+KLw+yuDHyvD6XfW8yAmTu4GqsPTlX60PnQlXIAYwz7z15lxa4T1N/VnxKX1jA/dzd6nahFcFZfhjxfkSeKB1pdpnIBusCFUunpdjzM7AK7Z3KpQEPej36SuRcL8NLjBXm3aWn8fXQZAvXodJFopdKThxe0+g7qvkv26AhG3BzA2oD/4/DmxTT+cjUbjsRYXaFyURroSqUFmwfU/Rf02QPNhpPP+wZTfT6hQ/x02o1Zx4dzd3MjLsHqKpWL0UBXKi15+0GVcOi2Din7HJ3ip7A010hmr9tJ0y/XEHHsgtUVKheiga5UevDJbO+GaTaM4te3sCHw/8ifcJzW365nwMydnLp00+oKlQvQQFcqvYhAlU7w6iJ8ieN78y4fl45i+paT1B26gv/M3smOqEvcjNO1TNWj0VEuSlnh8in46UU4s51LtT9kyOX6/Lz5JAmJBhHIn8OPxuWC6dOgBJm8PayuVmUgqR7lIiKNRWS/iBwSkf4POK6ViBgRuefFlFJJsuWDVxdB6eZkX/0Bn+RYwO/v1GPUS5XpXb84JXJnYczqIzQdsYYtx7WfXaVMsnfoIuIBHAAaAlHAZqCdMWbPXcdlARYA3kBPY8wDb7/1Dl0p4HYCzH0Dtv8INd6Ahh/Zu2aAdYfO02/6Ds5cvkn3usXo27AENptYXLCyWmrv0KsCh4wxR4wxccBPQMt7HPcRMBiIfeRKlXI3Hp7QciRU6Wyf4Gteb3vIAzWKBbKkT21aVQ7h6xWH6PrDFq7f0qGO6v5SEuj5gDuXZ4lK2vYXEakM5DfGLHjQiUSki4hEiEhEdHT0QxerlEuy2aDpUKj1FmydBNPaQ9wNADL7eDLk+Qq8/3QZlu09S+vR6zmtI2LUfaR6lIuI2IDhwFvJHWuMGWOMCTPGhAUFBaX20kq5DhGo/z40/QwOLIbvW8D1mKRdwmtPFGZchyqcuHCDZiPWMH/HaYsLVhlRSgL9FJD/js8hSdv+lAUoB6wUkWNANWCuPhhV6hFU7QwvTIY/dsL4RnDlf8Fdr1QuZveoSYEAP3r+uI0eU7YSc00X1VD/k5JA3wwUF5HCIuINtAXm/rnTGHPZGBNojClkjCkEbABaJPdQVCl1H6Wbw8uz4OofML4xXDz2165iuTIzo1sN+5S9e/6g3mcr+XLZQa7oFL2KFAS6MSYB6AksAfYCPxtjdovIIBFpkdYFKuWWCtaADnPg1hV7qEfv/2uXp4eNHvWKsaBXLaoVycnnyw5Qa/AK/m/+HmZti2LXqcvExuvLSe5IXyxSKiM7uxu+fwbib0CTwRD60l/DGv+069Rlvlh2kJX7z5GQaP/7nNXXk061ivBqzUJk8fWyoHCVVnQ+dKWc2aWTMKsrHP8dSj0NzUeAf85/HBZ/O5Fj56+z/+xVZm87zbK9Z8mWyYte9YsT/kRhCwpXaUEDXSlnl3gb1o+E3z4Cn6z2u/Vyrf5xt36nHVGXGLpkP2sOnmd4m4o8VzkkHQtWaUUXuFDK2dk8oGYv6LISchSEGeEw5Xm4cPS+P1IhJDsTOlahaqEA/jN7F4ejr6VfvcoSGuhKOZPcZSH8V2g8GI6vh68eg+nhcGrrPQ/39LDxZbtQfDxt9JiyVR+WujgNdKWcjc0DqnWFnpuhWjc4sATG1oNxT8HWyXDr6t8Oz5MtE8PbhLLvj6u8P2cXcQmJFhWu0pr2oSvl7GKvwLbJEDEBYg6Clz+Ue84e9rnL/nXY4MX7+GblYfJlz0TPJ4vx/GMheHnoPZ2z0YeiSrkDY+DkJnu475phH+pYpC7UfBOK1sMYw6oD0Xy+7CDbT16iSKA/v3StTs7MPlZXrh6CBrpS7ubGBdgyETaNgatnoNLL0PhT8MmMMYZle8/R48etVC+Skwkdq+i0vE5ER7ko5W78AqBWX+i9A57oC9t+gNE14eQmRISGZXLz3tNlWHUgmjFrjlhdrXIQDXSlXJmnNzT4wL46kkm0TyOwaSwA7R8vQNPywQxdsl9XRXIRGuhKuYOC1aHr71CsASx8G+b3RRIT+LRVBfJm96Xnj9vYGXXZ6ipVKmmgK+UufLNBu6lQszdEjIPJz5I18SrfvPQYicbwzKi1DP/1gA5rdGIa6Eq5E5sHNBwEz35rHxEzth7lvE6z9M06tAzNy4jlB3lm5Fr2nL5idaXqEWigK+WOKraFVxdC/E34riHZTi5jeJtQxrz8GOeu3qLlyN8Zsfwg8bf1bt2ZaKAr5a5CwqDzCshZBKa2henhPBWSwK99atO4XB6G/3qAZ0etZd2h81g1vFk9HB2HrpS7i7sBvw+HdV8BAjV6QpVOLDxmGDhvN2ev3KJq4QD6NChB9aL/nLZXpS99sUgplbxLJ+DXD2D3TBAPKNaAuAovMfVyWUauOsa5q7eoXiQnfRqWoGrhAKurdVsa6EqplDt/CCKnQOSPcO0PCCpFXK13mHw5lG9WHeX8tVvUKJqTtlULUL9ULvx9PK2u2K1ooCulHt7tBNg7B1YOhvP7IVdZ4iq9yk83qzJyQzRnr9zCx9NGg9K5+bBFWYKy6Jww6UEDXSn16BJvw66ZsPYLOLsLPH1JLNWcyFJ9mH3YMG3zSZ4oFsh3HcKQB6ygpBxD53JRSj06mwdUaG1/07TLKqj0Mrb9C6m8+DkGPXaLfo1KsnzfOWZHnrK6Urenga6UShkRyBsKzT6DTsvs88RMaMJrWTdTuUB2Ppy7h3NXY62u0q1poCulHl7uMtB5JYRUwTarC98VWMbN+ATem71Lx6xbSANdKfVo/HPCy7Mg9CUCIoYzJ/80lu0+TbuxG5iy8Tgx125ZXaHb0YeiSqnUMQZWfAyrh3Is4Al63OrB7hiDh02oXiQnzSrkoXHZYHL4e1tdqUvQh6JKqbQjAk/+B57+nEIX1zHfewArX8hE1zpFiLp4gwEzd1Ll42VM3nDc6kpdnga6Usoxwl6DjgsRk0ihua3o5/kLK/rUZEGvJ6hZLJAP5uxixb5zVlfp0jTQlVKOU7A6dF0LFV+ENZ8h09pTNsibb9pXpnSerLwxdRv7/tCpedOKBrpSyrF8s8IzI6HZcDi4FKa0xs/EMq5DFfx9PAifGKHDG9OIBrpSKm1UCYdnR8PxtTD5WYJtlxnXoQoXrsfR+fstxMbftrpCl5OiQBeRxiKyX0QOiUj/e+zvKyJ7RGSHiCwXkYKOL1Up5XQqtoXWk+D0NviiHOUi3mVcEz92RF3irZ+3k5ioY9YdKdlAFxEPYCTQBCgDtBORMncdtg0IM8ZUAKYDQxxdqFLKSZVpAT02QuVXYOcMaixtzprgEVzbvZhhS/dZXZ1LSckdelXgkDHmiDEmDvgJaHnnAcaYFcaYG0kfNwAhji1TKeXUchaFZsOg7x6o/z75Eo4zyXswz6xrxdI5U6yuzmWkJNDzASfv+ByVtO1+woFF99ohIl1EJEJEIqKjo1NepVLKNfgFQK23kN47SGg5Gn8fD+pv7cGSsf8mIUH71FPLoQ9FRaQ9EAYMvdd+Y8wYY0yYMSYsKCjIkZdWSjkTT288K7Uj91vr2Z+jLo1Ofc36z9ty8fJVqytzaikJ9FNA/js+hyRt+xsRaQD8G2hhjNFJHJRSyfLwzUyZXjPZWawbta4vJXZ4BX758m3mb9rD9VsJVpfndJKdy0VEPIEDQH3sQb4ZeNEYs/uOYyphfxja2BhzMCUX1rlclFJ3OrF5Pomrh1Po6hauGx/G257Ds/ZbvFKjsC5zd4dUr1gkIk2BLwAPYLwx5mMRGQREGGPmisgyoDxwJulHThhjWjzonBroSql7STy9nUuLPyHgxGJW3a7AQK/ePFGxFCWDs1AidxbK5c1GJm8Pq8u0jC5Bp5RyLsbAlokkLnqHS2RlQHwnlsSVB4QAf29er12El6sXxM/b/e7cdbZFpZRzEYGwV7GF/0pA1ix8a/uUgyH/x4Jax6mUNxP/XbSP2kNW8P36Y/py0h000JVSGVfeUPtLSS1H4WWzUXbzAMbdfo9ZnSpQIncW3p+zm3ZjN3Ai5kayp3IHGuhKqYzN0wcqvQTd1kKrcXBmO5V+78aUjhUZ8nwF9py+QuMvVzPu96NuPz+MBrpSyjmIQPnn7RN+HVuDTA+nTaU8LOlTm7BCAXw0fw91h65k8obj3HLTl5Q00JVSzqVCG2gyBPYvgF86kNfzGpNercKUTo8TkiMT783eRYuv1hJ91f1eh9FAV0o5n8dfh0afwIElMLIKsn0qNYvm5Jeu1fnulTBOXLjBS99t4LybLVStga6Uck7Ve0DX3yGwJMzuBhOaIsfX0aBMbsZ3rMKJCzdo/91GYtwo1DXQlVLOK1cpeHURPP0FXDgME5vC9y2p7nuccR2qcPT8ddqN3cDJC+4xCkYDXSnl3Gw2CHsVem+3d8Oc3Q3jG1Hz5komvFqFM5djeXbUWrYcv2h1pWlOA10p5Rq8Mtm7YXpuhpCqMCOcGmenMqtbDfx9PGk3dgNzIv8xr6BL0UBXSrmWTDng5ZlQ9llY+h+KbXqP2eHlqZQ/O29Oi+TniJPJn8NJud9ECEop1+fpA63GQ7b8sO4rcuydxw913iXcozT/mrEDmwjPP+Z6C6vpHbpSyjXZbPDUR9BlBQQWx2tRXybGvUXvvPvoNz2SmVujrK7Q4TTQlVKuLW8l+0iY1hOx3Y7jzZhB/Jb5A+bNmMTsba7Vp66BrpRyfSL2PvXuG+GZ0RT0T2CC1xAuzejD3K1Hra7OYTTQlVLuw8MTQtth67mZ+Cpd6ei5hAKzW7Fs/WarK3MIDXSllPvx9Mar2WBuPTeBEh5nqLq4BUfmDoGEOKsrSxUNdKWU2/Kp8ByJXVZywKskRbZ+TNzXj9vnh3FSGuhKKbeWOU9JgrsvoLdtAH9cjoUf28CsbhB7xerSHpoGulLK7YUE+NOh4+s0jR/Cz37tMDt+gtE14fg6q0t7KBroSikFVC6Qg6EvhPGfKy3pwCCux4OZ0BQW9nOau3UNdKWUStKkfB4W9qrF5cDKVIn5kGVZnsFsGsutL6twJXK21eUlSwNdKaXuUCxXZmZ0rU6PRhV55/qLPHtrIEeue5F1dgeOf9MKrpyxusT7EmOMJRcOCwszERERllxbKaVSwhhD9LVbHD5zkaiFQ2l+8Xvw9MG70SBsYR3B5pHuNYnIFmNM2L326R26Ukrdh4iQK4sv1Uvk4blew/iu3BS2xhXAtrAv8V9VgZ3TITHjLEitga6UUingYRN6tm7E0WZT6ZnQhyMX4mBGOPFfPw5rhkH0AatL1C4XpZR6WKcv3eSbFQe4vGU6r9kWEmo7ZN8RVBpajYXg8ml27Qd1uWigK6XUI4q6eIORKw6zJiKShh5beNN3Ppm9wCN8CeQsmibX1D50pZRKAyE5/Pjvc+WZ+nYrboa+Ruvr/+Ly9VguftuM86fSfxZHDXSllEql/AF+fNqqAt+99RLfFx2O561LXBzzNP2/X8ainWe4GZc+D061y0UppRzsj8hfCZjzEpeNHz1u9WS3Vzk+bVWB5hXzpvrcqe5yEZHGIrJfRA6JSP977PcRkWlJ+zeKSKFU1qyUUk4rOLQh3l1XEBiQk2m+nzAg62L6TtvK/B2n0/S6yQa6iHgAI4EmQBmgnYiUueuwcOCiMaYY8Dkw2NGFKqWUU8ldFnl9JVKmJe2vTWB1pn5E/DyYJdsOpdklU3KHXhU4ZIw5YoyJA34CWt51TEtgUtL304H6IiKOK1MppZyQTxZ4fjy0nkSu3Hn40HMi1WbXZvvC79LkcikJ9HzAyTs+RyVtu+cxxpgE4DKQ8+4TiUgXEYkQkYjo6OhHq1gppZyJCJR9Bo8uv3Hj5cUc8H8M/+AiaXIpzzQ5630YY8YAY8D+UDQ9r62UUlbzK1qdKu/MS7Pzp+QO/RSQ/47PIUnb7nmMiHgC2YAYRxSolFIqZVIS6JuB4iJSWES8gbbA3LuOmQt0SPr+eeA3Y9V4SKWUclPJdrkYYxJEpCewBPAAxhtjdovIICDCGDMXGAdMFpFDwAXsoa+UUiodpagP3RizEFh417b37/g+Fmjt2NKUUko9DH31XymlXIQGulJKuQgNdKWUchEa6Eop5SIsm21RRKKB44/444HAeQeW4wy0ze5B2+weUtPmgsaYoHvtsCzQU0NEIu43faSr0ja7B22ze0irNmuXi1JKuQgNdKWUchHOGuhjrC7AAtpm96Btdg9p0man7ENXSin1T856h66UUuouGuhKKeUiMnSgu+Pi1Cloc18R2SMiO0RkuYgUtKJOR0quzXcc10pEjIg4/RC3lLRZRNok/a53i8iP6V2jo6Xgz3YBEVkhItuS/nw3taJORxGR8SJyTkR23We/iMiIpP8eO0SkcqovaozJkF/Yp+o9DBQBvIHtQJm7jukOjE76vi0wzeq606HN9QC/pO+7uUObk47LAqwGNgBhVtedDr/n4sA2IEfS51xW150ObR4DdEv6vgxwzOq6U9nm2kBlYNd99jcFFgECVAM2pvaaGfkO3R0Xp062zcaYFcaYG0kfN2BfQcqZpeT3DPARMBiITc/i0khK2twZGGmMuQhgjDmXzjU6WkrabICsSd9nA06nY30OZ4xZjX19iPtpCXxv7DYA2UUkT2qumZED3WGLUzuRlLT5TuHY/w/vzJJtc9I/RfMbYxakZ2FpKCW/5xJACRFZKyIbRKRxulWXNlLS5g+B9iIShX39hTfSpzTLPOzf92Sl6yLRynFEpD0QBtSxupa0JCI2YDjQ0eJS0psn9m6Xutj/FbZaRMobYy5ZWVQaawdMNMYME5Hq2FdBK2eMSbS6MGeRke/Q3XFx6pS0GRFpAPwbaGGMuZVOtaWV5NqcBSgHrBSRY9j7Guc6+YPRlPyeo4C5xph4Y8xR4AD2gHdWKWlzOPAzgDFmPeCLfRIrV5Wiv+8PIyMHujsuTp1sm0WkEvAt9jB39n5VSKbNxpjLxphAY0whY0wh7M8NWhhjIqwp1yFS8md7Nva7c0QkEHsXzJF0rNHRUtLmE0B9ABEpjT3Qo9O1yvQ1F3glabRLNeCyMeZMqs5o9ZPgZJ4SN8V+Z3IY+HfStkHY/0KD/Rf+C3AI2AQUsbrmdGjzMuAsEJn0NdfqmtO6zXcduxInH+WSwt+zYO9q2gPsBNpaXXM6tLkMsBb7CJhI4Cmra05le6cCZ4B47P/iCge6Al3v+B2PTPrvsdMRf6711X+llHIRGbnLRSml1EPQQFdKKRehga6UUi5CA10ppVyEBrpSSrkIDXSllHIRGuhKKeUi/h9DvnvQtfHjhwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_rand.threshold, df_rand['tpr'], label='TPR')\n",
    "plt.plot(df_rand.threshold, df_rand['fpr'], label='FPR')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1958bf4d",
   "metadata": {},
   "source": [
    "### Ideal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "b56b21df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1023, 386)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_neg = (y_val == 0).sum()\n",
    "num_pos = (y_val == 1).sum()\n",
    "num_neg, num_pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7f051585",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "y_ideal = np.repeat([0, 1], [num_neg, num_pos])\n",
    "y_ideal\n",
    "\n",
    "y_ideal_pred = np.linspace(0, 1, len(y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "f7f9c21f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7260468417317246"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - y_val.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c85adf63",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_ideal, y_ideal_pred >= 0.726)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b30738fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>tp</th>\n",
       "      <th>fp</th>\n",
       "      <th>fn</th>\n",
       "      <th>tn</th>\n",
       "      <th>tpr</th>\n",
       "      <th>fpr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>386</td>\n",
       "      <td>1023</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.1</td>\n",
       "      <td>386</td>\n",
       "      <td>882</td>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.862170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.2</td>\n",
       "      <td>386</td>\n",
       "      <td>741</td>\n",
       "      <td>0</td>\n",
       "      <td>282</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.724340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0.3</td>\n",
       "      <td>386</td>\n",
       "      <td>600</td>\n",
       "      <td>0</td>\n",
       "      <td>423</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.586510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>0.4</td>\n",
       "      <td>386</td>\n",
       "      <td>459</td>\n",
       "      <td>0</td>\n",
       "      <td>564</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.448680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>0.5</td>\n",
       "      <td>386</td>\n",
       "      <td>319</td>\n",
       "      <td>0</td>\n",
       "      <td>704</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.311828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>0.6</td>\n",
       "      <td>386</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>845</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.173998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.7</td>\n",
       "      <td>386</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>986</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.036168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>0.8</td>\n",
       "      <td>282</td>\n",
       "      <td>0</td>\n",
       "      <td>104</td>\n",
       "      <td>1023</td>\n",
       "      <td>0.730570</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.9</td>\n",
       "      <td>141</td>\n",
       "      <td>0</td>\n",
       "      <td>245</td>\n",
       "      <td>1023</td>\n",
       "      <td>0.365285</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>385</td>\n",
       "      <td>1023</td>\n",
       "      <td>0.002591</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     threshold   tp    fp   fn    tn       tpr       fpr\n",
       "0          0.0  386  1023    0     0  1.000000  1.000000\n",
       "10         0.1  386   882    0   141  1.000000  0.862170\n",
       "20         0.2  386   741    0   282  1.000000  0.724340\n",
       "30         0.3  386   600    0   423  1.000000  0.586510\n",
       "40         0.4  386   459    0   564  1.000000  0.448680\n",
       "50         0.5  386   319    0   704  1.000000  0.311828\n",
       "60         0.6  386   178    0   845  1.000000  0.173998\n",
       "70         0.7  386    37    0   986  1.000000  0.036168\n",
       "80         0.8  282     0  104  1023  0.730570  0.000000\n",
       "90         0.9  141     0  245  1023  0.365285  0.000000\n",
       "100        1.0    1     0  385  1023  0.002591  0.000000"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ideal = tpr_fpr_dataframe(y_ideal, y_ideal_pred)\n",
    "df_ideal[::10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "60cb9f81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca7af4a90>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmQ0lEQVR4nO3deXxV9bnv8c+TiQQIc5gyKyCDzGFIrFYrImAVrcosUalWPdrT47H30GsH29p7am1P7+2p7al1YFBAcATFecLKIEEQBRwQAyTIYGSQIUDC7/6xd2gIgeyQvffaw/f9evFq9t6LvZ5lwrcraz/r+ZlzDhERiX4JXhcgIiLBoUAXEYkRCnQRkRihQBcRiREKdBGRGJHk1Y47dOjg8vLyvNq9iEhUWrVq1VfOuYz6XvMs0PPy8igpKfFq9yIiUcnMNp/qNV1yERGJEQp0EZEYoUAXEYkRnl1DFxEJhqNHj1JWVkZlZaXXpQRVamoqWVlZJCcnB/x3FOgiEtXKyspIT08nLy8PM/O6nKBwzlFRUUFZWRn5+fkB/70GL7mY2SNmttPMPjrF62ZmfzKzjWa21swGNaJuEZEmqayspH379jET5gBmRvv27Rv9W0cg19BnAKNO8/pooLv/z83AXxtVgYhIE8VSmNc4k2Nq8JKLc26JmeWdZpOxwCznm8O73MzamFkX59yXja4mAGvXvs+hFTN5N+cWMH2mKxJK6anJTB6eQ/MUXZ2NBsH4LmUCW2s9LvM/d1Kgm9nN+M7iycnJOaOdHfjgOQrLZ1C+ZSP/q+oHVJN4Ru8jIg1zDp77oJyHpg6hc+tUr8uJSBUVFVx88cUAbN++ncTERDIyfDdyfvDBB/Tv35+qqip69erFzJkzad68OYmJifTt25eqqiry8/OZPXs2bdq0aXItFsgCF/4z9Oedc+fW89rzwG+dc//wP34d+A/n3GlvAy0oKHBnfKfokt/DG7+GXpfD1Q9DUrMzex8ROa03Pt7BHXNW0zI1iYemDqFvVmuvSzrJhg0b6NWrl9dlAHDPPffQsmVL7rrrLgBatmzJ/v37AZg8eTKDBw/mzjvvPOH54uJievTowd13333S+9V3bGa2yjlXUN/+g3HNohzIrvU4y/9c6FxwF4z6LWxYBPMmwZEDId2dSLz6Ts9OPHVbEUkJCVz7t6V8vmu/1yVFrfPPP5+NGzee9HxhYSHl5cGJzGBcclkI3G5m84BhwN5QXT8/wfBbIaUFLPpXmPFdmDQfWtY7r0ZEmqBn51Y8fVsR5//uTR599wvuvbKv1yWd0i8XrWP9tn1Bfc/eXVvxi8v7NOk9qqqqePHFFxk16sT+kurqal5//XWmTZvWpPevEUjb4lxgGXCOmZWZ2TQzu8XMbvFvshjYBGwE/g7cFpTKAjFoKox/DHZugIcvgYrPw7ZrkXjSqVUqY/t35en3y9lXedTrcqLGoUOHGDBgAAUFBeTk5BwP7prnO3fuzI4dO7jkkkuCsr9AulwmNvC6A/4lKNWciZ6XQfFCmDMeHh7pO1PPGuxZOSKxqrgojwWrylhQUsa0bwV+s0s4NfVMOtjS0tJYs2bNKZ8/ePAgl156KQ888AA//OEPm7y/2Oj7yx4K0171XYKZ+V349GWvKxKJOedmtmZwbltmLyvl2LGGmymkYc2bN+dPf/oTf/jDH6iqqmry+8VGoAN06OYL9Q7dYe5EeH+W1xWJxJziojxKKw7y9me7vC4lZgwcOJB+/foxd+7cJr9XbN0tkN4Jrn8B5hfDwjtg3zb49n9ADN5FJuKFUX06k5HejJlLS7nonI5elxNx7rnnnhMe17Qm1lX3+UWLFgVl/7Fzhl6jWTpMegIGTIa3/hMW/RCqm/6rjIhASlICk4fl8NYnuyj9Su3CkSb2Ah0gMRnGPgAX/Nh36UW96iJBM2lYDsmJxqxlp1wJTTwSm4EOvsss3/kpfPePsPFVX6/6fl33E2mqjumpjD63CwtKtnLgsH77jSSxG+g1Cm6E8Y+rV10kiIqL8vjmcBVPrw7tTeHSOLEf6AA9x/h61Sv3+nrVy1d5XZFIVBuU04ZzM1sxa2kpgcyDkvCIj0CHE3vVZ6hXXaQpzIziwjw+27mfZZ9XeF2O+MVPoIN61UWC6PL+XWnbPJkZS0u9LsVziYmJDBgw4Pif0tJS3nrrLVq3bs2AAQPo1asXv/zlLwFOeL5nz57HJzMGQ2z1oQdCveoiQZGanMiEoTn87e3PKdt9kKy2zb0uyTP13eJfWlrK+eefz/PPP8+BAwcYMGAAl19+OcDx5w8dOsTAgQO56qqrOO+885pcR3ydodeo6VXvP0m96iJNMGV4Lg6Y997WBreNZy1atGDw4MEnjc9NS0tjwIABETU+NzolJsOVf4HWmbDkfvhmO1w7w3eNXUQCktkmjQt7ZDC/ZCs/GtGdpESPzxFfnA7bPwzue3buC6N/e9pNaqYnAuTn5/PMM8+c8HpFRQXLly/nZz/7Gbt2/bN9evfu3Xz22WdccMEFQSk1Ps/Qa5zQq/6aetVFzsCEoTns/OYwb3y80+tSPFNzyWXNmjUnhPk777zDwIEDGTlyJNOnT6dPnz7Hn+/fvz+ZmZlceumldO7cOSh1xO8Zem0FN0LLzvDkjb5e9SlPQfuzva5KJCp8p2dHOqY3Y97KrYzsE5xgOmMNnEmHW8218lM9/8UXXzB8+HDGjRt3/Ay/KeL7DL22nmOgeJF61UUaKTkxgWsLsnjrk51s23PI63KiSn5+PtOnT+e+++4Lyvsp0GvLHqJedZEzMGFIDscczC/Rh6ONdcstt7BkyRJKS0ub/F4K9Lo6dIPvvwYdeqhXXSRA2e2ac373DsxfuZXqOFz8or4xuRdeeGG9l1vqPp+WlkZ5eTl5eXlNrkOBXp+WHX296mdf5OtVf+u3oNubRU5rwpActu2t5O1P4/fDUa8p0E+lWUuYOE9z1UUCdEnvTmSkN9NYXQ8p0E9Hc9VFApaSlMCkob7FL74I8+IXsTgg7EyOSYHeEM1VFwnY5GE5JCUYs5aVhm2fqampVFRUxFSoO+eoqKggNTW1UX9PfeiBUq+6SIM6tkplTN8uPFlSxl0jz6FFs9BHTFZWFmVlZSfcgRkLUlNTycrKatTfUaA3Rk2v+pxxvl71yfMhc7DXVYlElOKiXBZ+sI2nV5dz3fDckO8vOTmZ/Pz8kO8nGuiSS2OpV13ktAbltNXiFx5RoJ8JzVUXOSUtfuEdBfqZqpmrftaFvl71t+9Xr7qI3+X9u9KuRYoWvwgzBXpT1MxV7zce3rwXXvuFQl0E/+IXQ7J5bcMOynYf9LqcuKFAb6rEZLjyf3xdMO/+P1j8Yzh2zOuqRDw32f+B6GPLt3hcSfxQoAdDQgJc9l9QdAes/Ds8ewtUHfG6KhFPZbZJY2TvzsxbuYXKo9VelxMXFOjBYgaX/Np3E9LaJ2DOtVC5z+uqRDxVXJTHnoNHWbhmm9elxIWAAt3MRpnZJ2a20cym1/N6jpm9aWarzWytmY0JfqlRwMw3JmDsA/DFOzBjjG9pO5E4NfysdpzTKZ0ZamEMiwYD3cwSgQeA0UBvYKKZ9a6z2U+B+c65gcAE4C/BLjSqDJwCk+ZDxSZ46BLY9anXFYl4wsyYWpTL+i/3sWrzbq/LiXmBnKEPBTY65zY5544A84CxdbZxQCv/160B/X7VfQRc/zxUHYJHRsKW5V5XJOKJqwZm0io1SS2MYRBIoGcCtZchKfM/V9s9wBQzKwMWA3fU90ZmdrOZlZhZSazNXahX5iDfDUhp7WDWWNiwyOuKRMKueUoS4wqyeemj7ezYV+l1OTEtWB+KTgRmOOeygDHAbDM76b2dcw865wqccwUZGRlB2nWEa5fvC/XOfeGJ6+C9v3tdkUjYXVeYS7VzPL5CLYyhFEiglwPZtR5n+Z+rbRowH8A5twxIBToEo8CY0KI9TF0I54yGxXfBa/foBiSJK7ntW3DROR2Zs2ILR6p0n0aoBBLoK4HuZpZvZin4PvRcWGebLcDFAGbWC1+gx8E1lUZIaQ7jZsPgG+Aff4RnfqBedYkrxUV5fLX/MC9+9KXXpcSsBgPdOVcF3A68DGzA182yzsx+ZWZX+Df7d+AmM/sAmAtc79SjdLLEJN9CGepVlzh0frcO5HdooQ9HQyigeejOucX4Puys/dzPa329HjgvuKXFqJpe9fSuvqFeM8bApAXQqovXlYmEVEKCMbUwl18uWs/asj30y2rjdUkxR3eKemXg5H/2qj98Cez6xOuKRELumsFZtEhJZOZSLSQdCgp0L3UfATe8AFWVvhWQ1KsuMS49NZnvDcpi0dptVOw/7HU5MUeB7rWuA31tjc3bq1dd4kJxUS5Hqo4xb+XWhjeWRlGgR4K6veorHvS6IpGQ6dYxnfO6tefx5ZupqlYLYzAp0CNF7V71F38Mr/5Cc9UlZhUX5rFtbyWvbdjhdSkxRYEeSWp61QtuhHf/r+aqS8y6uFcnMtukqYUxyBTokSYxybdYxnd+pl51iVmJCcZ1hbks3/Q1H2/Xz3ewKNAjkRlccBeM/QuU/sPXq75Pd9dJbBlfkE2zpAS1MAaRAj2SDZzsW4RaveoSg9q2SOHKAZk8u7qcvQePel1OTFCgR7puI+CGxVB1WL3qEnOmFuVy6Gg1C1aphTEYFOjRoOsA+P6r0KKDetUlpvTp2poheW2ZtWwzx45p/FNTKdCjRds8uPEVzVWXmFNclMeWrw/y1qc7vS4l6inQo0nduerqVZcYcGmfznRq1YwZ+nC0yRTo0ab2XHX1qksMSE5MYPKwXJZ8uovPd+33upyopkCPRpqrLjFm4tAcUhITmL1MZ+lNoUCPVjVz1cf+Bb54Bx5Vr7pEr4z0ZlzWrwtPripj/+Eqr8uJWgr0aFczV/1r9apLdJtamMv+w1U8/X6Z16VELQV6LNBcdYkBA3Pa0j+rNTOXlqIVLM+MAj1W1J6rPvMKWF93HW+RyFdclMfnuw7w7sYKr0uJSgr0WFIzV71LP5g/VXPVJepc1q8L7VukaArjGVKgxxrNVZco1iwpkYlDc3j94x1s/fqg1+VEHQV6LKpvrnq1hh9JdJg8PIcEMx5brhbGxlKgx6rjc9X9vepPXAdHK72uSqRBXVqnMapPZ+at3MqhI9VelxNVFOixrKZXfczv4dMXYc44OKw78STyTS3MZe+hoyz8oNzrUqKKAj0eDL0JrvwrlL4Dj30PDu32uiKR0xqa346endOZsXSzWhgbQYEeLwZMgmsehfL34ZFRsFc3b0jkMjOuL8pjw5f7WFmqE5BAKdDjSZ8rYcpTsG8bPHQJ7FjndUUipzR2QCat05KZqRbGgCnQ481Z34YbXgSc70z9iyVeVyRSr7SURMYPyealddvZvlcf6AdCgR6POp/ruwGpVVd47Gr48EmvKxKp15RhuRxzjsdXqIUxEAr0eNUmG258CTIL4KlpsPTPXlckcpKc9s25uGdH5r63hcNVamFsSECBbmajzOwTM9toZtNPsc04M1tvZuvMbE5wy5SQSGsL1z0DvcfCK3fDSz/RXaUScYqL8vhq/xEWf6jx0A1pMNDNLBF4ABgN9AYmmlnvOtt0B34CnOec6wP8KPilSkgkp8I1M2DYrbD8L/DkDboBSSLKeWd34KyMFlqiLgCBnKEPBTY65zY5544A84Cxdba5CXjAObcbwDmn1V6jSUICjPpPGHkvrH9WveoSURISjOLCPD7Yuoc1W/d4XU5ECyTQM4GttR6X+Z+rrQfQw8zeNbPlZjaqvjcys5vNrMTMSnbt2nVmFUtomEHRHXD1w7D1PfWqS0S5enAWLZslMUstjKcVrA9Fk4DuwIXARODvZtam7kbOuQedcwXOuYKMjIwg7VqCqu81tXrVR8D2j7yuSISWzZK4elAmz6/9kq/2H/a6nIgVSKCXA9m1Hmf5n6utDFjonDvqnPsC+BRfwEs0Ot6rbvDoaPWqS0SYWpTHkepjzF2xxetSIlYggb4S6G5m+WaWAkwA6i6H8yy+s3PMrAO+SzCbglemhF3nc+H7r0KrTJj9PfWqi+fOzmjJ+d078PiKLRytVjdWfRoMdOdcFXA78DKwAZjvnFtnZr8ysyv8m70MVJjZeuBN4MfOOa0hFe1aZ8GNL0L2UH+v+n+DBiWJh4oL89i+r5JX1u3wupSIZF5NMisoKHAlJSWe7Fsa6WglPPMDXwfMsFvh0v/j64wRCbPqY44Lf/8mXVqlMf+WQq/L8YSZrXLOFdT3mv5VSsOSU32TGofdCiv+ql518UxigjF1eB7vlX7N+m37vC4n4ijQJTAJCTD6tzDyN+pVF09dW5BFanICs5aVel1KxFGgS+MU3e7rVS9bqV518USb5ilcNTCTZ9eUs+fgEa/LiSgKdGm8E3rVNVddwm9qYR6VR48xv2RrwxvHEQW6nJn8C3zTGkFz1SXsenVpxdD8dsxatpnqY+q8qqFAlzPXqY+/V11z1SX8ri/Ko2z3Id78WKOjaijQpWlaZ/nO1LOGqFddwmpk7050aZ3KTH04epwCXZourS1MeRp6Xwmv/FRz1SUskhITmDwsh3c++4qNO/d7XU5EUKBLcNT0qg+/Tb3qEjYThuaQkqgWxhoKdAme43PV1asu4dGhZTO+278LT60q45vKo16X4zkFugRfTa+65qpLGFxflMeBI9U8tUo/Zwp0CY2+18B1T2uuuoRcv6w2DMhuw6xlmzkW5y2MCnQJneO96v656pve9roiiVHXF+Wx6asDvLPxK69L8ZQCXULreK96pnrVJWTG9O1Ch5bN4n6JOgW6hF5Nr7rmqkuIpCQlMGloNm98spMtFQe9LsczCnQJj7Q2J/aqv/JThboE1eThuSSaxXULowJdwqemV33oD2DZn2HRv8Kxaq+rkhjRqVUqo87tzPySrRw8UuV1OZ5QoEt4JSTA6Pvg/H+H92f6VkKqVv+wBEdxUR77Kqt4ZnXddezjgwJdws8MLv45XPwL+HABPHEdHInf654SPAW5bendpRWzl23Gq+U1vaRAF++cfyeM+T18+hLMugIOaF1xaRozY8rwXD7e/g2rt+7xupywU6CLt4beBONnw/YP4eFL4OsvvK5IotwVA7rSPCWRee9t8bqUsFOgi/d6XQ5Tn4ODFb5Q37ba64okirVslsTl/bqy6IMv426+iwJdIkPOcJj2KiSlwaOXwcbXvK5IotjEYTkcOlrNc2u2eV1KWCnQJXJk9PDdVdr+LJgzHtbM8boiiVL9s1rTs3M6c+PssosCXSJLeme4fjHkfQuevRWW3K8bkKTRzIxJw3JYt20fH5bt9bqcsFGgS+RJbQWTFkC/8fDGvfD8v0F1fN4oImdu7IBMUpMTmLsyfs7SFegSmZJS4Kq/wbf+DVY9Ck9MUa+6NErrtGQu69uV51aXx82Howp0iVxmMOKef/aqz7xcverSKFMLc+Nq8QsFukS+ml71HR+pV10apX92fC1+oUCX6KBedTlDxUW5cbP4hQJdokfdXvXP1KsuDfMtfpESF4tfBBToZjbKzD4xs41mNv00211tZs7MCoJXokgttXvV546H1Y97XZFEuGZJiUwamhMXi180GOhmlgg8AIwGegMTzax3PdulA/8KrAh2kSInqN2r/txt6lWXBsXL4heBnKEPBTY65zY5544A84Cx9Wz3a+A+oDKI9YnUT73q0gidWqVyaRwsfhFIoGcCW2s9LvM/d5yZDQKynXMvnO6NzOxmMysxs5Jdu3Y1uliRE9TtVZ+vuepyatf7F794dnXszndp8oeiZpYA/Bfw7w1t65x70DlX4JwryMjIaOquRU7sVf/kRc1Vl1OqWfxi5tLSmF38IpBALweyaz3O8j9XIx04F3jLzEqB4cBCfTAqYVV3rvruUq8rkghjZhQX5fLJjm9Y8cXXXpcTEoEE+kqgu5nlm1kKMAFYWPOic26vc66Dcy7POZcHLAeucM6VhKRikVOp3av+kHrV5WRjB2TSpnkyM2O0hbHBQHfOVQG3Ay8DG4D5zrl1ZvYrM7si1AWKNErOcJj2CiSlqlddTpKanMj4Idm8sn4H2/Yc8rqcoAvoGrpzbrFzrodz7mzn3G/8z/3cObewnm0v1Nm5eCrjnFpz1cepV11OMGVYLs45Hlu+2etSgk53ikpsqulVzz9fvepygux2zbm4VyfmrdxK5dFqr8sJKgW6xC71qsspXF+Ux9cHjvDC2i+9LiWoFOgS2zRXXepRdHZ7unVsycxlsdXCqECX2FfvXPXYn7wnp2ZmFBfmsrZsL6u37vG6nKBRoEv8OGGu+kjNVY9z3xuURXqzpJiawqhAl/iiueri16JZEtcUZPHCh1+y85vYGEGlQJf4o7nq4nfd8FyOVjvmrtja8MZRQIEu8Ulz1QU4K6Ml3+6RweMrNnOk6pjX5TSZAl3iV9256m+rVz0eXV+Ux85vDvPyuu1el9JkCnSJb7V71d+8F57/kXrV48y3e2SQ2755TMx3UaCLnNCrPkO96nEmIcG4bnguJZt381H5Xq/LaRIFugic3Kv++LVw+Buvq5IwubYgm7TkxKhfok6BLlLb0Jvg6odgyzKYNRYOxubcbDlR67RkrhqUyXNrtrH7wBGvyzljCnSRuvpe88/FMmZeDt/s8LoiCYPiwjwOVx1j3srobWFUoIvUp+dlMOkJ+HqT7wakrzZ6XZGE2Dmd0yk8qz2PLd9M9bHo7HZSoIucytnfgeufhyMHfKG+daXXFUmIFRflUr7nEK9tiM7fyhToIqeTOdi3AlJqa9/ll48Xe12RhNCIXp3o2jo1alsYFegiDWl/tm9UQMde8MRkKHnE64okRJISE5g8PJeln1fw2Y7o63JSoIsEomWG7/JLtxG+hTJe/7XuKo1RE4fmkJKUwMwobGFUoIsEKqUFTJgLg6bCO7+HZ2+D6qNeVyVB1q5FClf078rT75ezrzK6vr8KdJHGSEyCy/8EF/5v+GAOzBmvG5BiUHFhHgePVLOgpMzrUhpFgS7SWGZw4X/AFf8Nm96CGZepVz3G9M1qzaCcNsxeVsqxKGphVKCLnKlBU2HiPPjqM/Wqx6DiojxKKw7y9me7vC4lYAp0kaboMbJOr/p7XlckQTL63C5kpDeLqhZGBbpIU9X0qqe1Ua96DElJSmDS0Bze+mQXpV8d8LqcgCjQRYKhple9Ux/1qseQycNySEowZi3b7HUpAVGgiwRLiw5QvAi6XeLrVX/jXvWqR7mOrVIZ07cLC0q2cuBw5C98okAXCaaUFjBhju8D0yX3w3P/ol71KFdclMs3h6t4enW516U0SIEuEmzHe9V/Amse9/eq7/e6KjlDg3Lacm5mK2YtLcVF+G9cCnSRUDCDC6fX6lUfo171KGVmFBfm8dnO/Sz7vMLrck4roEA3s1Fm9omZbTSz6fW8fqeZrTeztWb2upnlBr9UkSh0Qq/6CN//StS5vH9X2rVIYUaEtzA2GOhmlgg8AIwGegMTzax3nc1WAwXOuX7Ak8Dvgl2oSNQ63qt+EB4eqV71KJSanMiEIdm8tmEHZbsjdwHxQM7QhwIbnXObnHNHgHnA2NobOOfedM7VHOVyICu4ZYpEuZPmqr/gdUXSSJOH+y48PLZ8i8eVnFoggZ4J1F5kr8z/3KlMA16s7wUzu9nMSsysZNeu6LmdViQojs9V7w1PTIGVD3tdkTRCZps0RvbuzLyVW6g8Wu11OfUK6oeiZjYFKADur+9159yDzrkC51xBRkZGMHctEh1qz1V/4U7NVY8yxUV57Dl4lIUfbPO6lHoFEujlQHatx1n+505gZiOAu4ErnHOHg1OeSAzSXPWoNfysdpzTKZ2ZEdrCGEigrwS6m1m+maUAE4CFtTcws4HA3/CF+c7glykSY2r3qmuuetQwM6YW5bJu2z5Wbd7tdTknaTDQnXNVwO3Ay8AGYL5zbp2Z/crMrvBvdj/QElhgZmvMbOEp3k5EapzUq6656tHgqoGZpKcmMTMC57skBbKRc24xsLjOcz+v9fWIINclEj8GTYWWnWDB9b4RvFOehg7dvK5KTqF5ShLjCrKZubSUHZf1olOrVK9LOk53iopEgh6Xaq56FJlamEu1czy+IrJaGBXoIpEiczB8/1X1qkeB3PYtuOicjsxZsYUjVce8Luc4BbpIJGl3lnrVo8TUwly+2n+YFz/60utSjlOgi0Sa473ql6hXPYJd0D2D/A4tImq+iwJdJBLVnqv+zu/huduhOvIXWIgnCQnG1MJcVm/Zw9qyPV6XAyjQRSJXTa/6t6fDmsfgqWlQdcTrqqSWqwdn0TwlkZlLI6OFUYEuEsnM4KKfwMh7Yf2zvuvqRw95XZX4tUpN5upBWSxau42K/d7fIK9AF4kGRXfAd/8In70Cj18Lh/Z4XZH4FRflcqTqGPNWbm144xBToItEi4Ib4Xt/hy3L4NExsC8yB0TFm24d0zmvW3seX76ZqmpvWxgV6CLRpN+1MHkB7NkCD10COzd4XZEAxYV5bNtbyWsbvB3doEAXiTZnfwduWAzHjsIjl0Lpu15XFPcu7tWJzDZpnrcwKtBFolGXfr4bkFp2gtlXwrpnvK4oriUmGNcV5rJ809d8vH2fZ3Uo0EWiVdtcuPFl6DoIFtwAy/7idUVxbcKQbJolJTDLwymMCnSRaNa8HUx9Fnp9F17+Cbx8NxyLnNki8aRN8xSuHJDJM++Xs/egNwuWKNBFol1yGlw7E4beDMv+DE9/H6q874mOR1OLcjl0tJoFq7xpYVSgi8SChEQY/TsY8Uv46Cl47Gqo3Ot1VXGnT9fWDMlry6xlm6k+Fv75Owp0kVhhBt/6EVz1oK9X/ZHR6lX3wNTCPLZ8fZC3Pw3/apwKdJFY0388TH7S36s+Qr3qYTbq3M50atWMGR7Md1Ggi8Sisy/y96pXw8PqVQ+n5MQEJg/LZcmnu/h81/6w7luBLhKruvTzrYCUrl71cJswNJvkRGN2mFsYFegisaxNzom96sv/6nVFcaFjeiqX9e3Ck6vK2H84fHPsFegisa6mV73nZfDSdHjlp+pVD4Piojz2H67i6ffLwrZPBbpIPEhOg3GzYMhNsPS/1aseBgOy29AvqzUzl5biwrSEoAJdJF4kJMKY+2HEPf/sVddc9ZAxM4oL8/h81wHe3VgRln0q0EXiiRl869/8c9WXw6OjYW+511XFrMv6daF9i5SwTWFUoIvEo37j/HPVt8LDmqseKqnJiUwYms3rH+9g69cHQ74/BbpIvDreq17l71X/h9cVxaQpw3NJMOOx5aFvYVSgi8SzLv3g+6/5e9Wvgo+e9rqimNOldRqX9unEvJVbOXSkOqT7UqCLxLvavepP3qi56iEwtTCPvYeO8tya0H5eoUAXEc1VD7Fh+e3o2Tmdmcs2h7SFUYEuIj6aqx4yZkZxUR4bvtzHytLdIdtPQIFuZqPM7BMz22hm0+t5vZmZPeF/fYWZ5QW9UhEJvfrmqu/xZrGGWDN2QFdapSYxM4QtjA0GupklAg8Ao4HewEQz611ns2nAbudcN+CPwH3BLlREwqRmrvr3/g5lK+HPBfDGb+DIAa8ri2rNU5IYPySbl9Zt58u9h0Kyj6QAthkKbHTObQIws3nAWGB9rW3GAvf4v34S+LOZmQvX/a4iEnz9xkFOIbx2Dyz5Hax8CFp29LqqqPbj6mNcm3SAj16+nS7jbgv6+wcS6JlA7d+5yoBhp9rGOVdlZnuB9sBXtTcys5uBmwFycnLOsGQRCZs22XDNw77r6iUPQ1Wl1xVFtRSgY+ujdBt4TkjeP5BADxrn3IPAgwAFBQU6exeJFjnDfH+kydqE8L0D+VC0HMiu9TjL/1y925hZEtAaCM80GhERAQIL9JVAdzPLN7MUYAKwsM42C4Fi/9fXAG/o+rmISHg1eMnFf038duBlIBF4xDm3zsx+BZQ45xYCDwOzzWwj8DW+0BcRkTAK6Bq6c24xsLjOcz+v9XUlcG1wSxMRkcbQnaIiIjFCgS4iEiMU6CIiMUKBLiISI8yr7kIz2wWc6RIeHahzF2oc0DHHBx1zfGjKMec65zLqe8GzQG8KMytxzhV4XUc46Zjjg445PoTqmHXJRUQkRijQRURiRLQG+oNeF+ABHXN80DHHh5Acc1ReQxcRkZNF6xm6iIjUoUAXEYkRER3o8bg4dQDHfKeZrTeztWb2upnlelFnMDV0zLW2u9rMnJlFfYtbIMdsZuP83+t1ZjYn3DUGWwA/2zlm9qaZrfb/fI/xos5gMbNHzGynmX10itfNzP7k/++x1swGNXmnzrmI/INvVO/nwFn4Vm76AOhdZ5vbgP/xfz0BeMLrusNwzBcBzf1f3xoPx+zfLh1YAiwHCryuOwzf5+7AaqCt/3FHr+sOwzE/CNzq/7o3UOp13U085guAQcBHp3h9DPAiYMBwYEVT9xnJZ+jHF6d2zh0Bahanrm0sMNP/9ZPAxWZmYawx2Bo8Zufcm865g/6Hy/GtIBXNAvk+A/wauA+IhUUtAznmm4AHnHO7AZxzO8NcY7AFcswOaOX/ujWwLYz1BZ1zbgm+9SFOZSwwy/ksB9qYWZem7DOSA72+xakzT7WNc64KqFmcOloFcsy1TcP3//DRrMFj9v8qmu2ceyGchYVQIN/nHkAPM3vXzJab2aiwVRcagRzzPcAUMyvDt/7CHeEpzTON/ffeoLAuEi3BY2ZTgALg217XEkpmlgD8F3C9x6WEWxK+yy4X4vstbImZ9XXO7fGyqBCbCMxwzv3BzArxrYJ2rnPumNeFRYtIPkOPx8WpAzlmzGwEcDdwhXPucJhqC5WGjjkdOBd4y8xK8V1rXBjlH4wG8n0uAxY65446574APsUX8NEqkGOeBswHcM4tA1LxDbGKVQH9e2+MSA70eFycusFjNrOBwN/whXm0X1eFBo7ZObfXOdfBOZfnnMvD97nBFc65Em/KDYpAfrafxXd2jpl1wHcJZlMYawy2QI55C3AxgJn1whfou8JaZXgtBKb6u12GA3udc1826R29/iS4gU+Jx+A7M/kcuNv/3K/w/YMG3zd8AbAReA84y+uaw3DMrwE7gDX+Pwu9rjnUx1xn27eI8i6XAL/Phu9S03rgQ2CC1zWH4Zh7A+/i64BZA4z0uuYmHu9c4EvgKL7fuKYBtwC31PoeP+D/7/FhMH6udeu/iEiMiORLLiIi0ggKdBGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRihQBcRiRH/H7RYPQHpGD/OAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_ideal.threshold, df_ideal['tpr'], label='TPR')\n",
    "plt.plot(df_ideal.threshold, df_ideal['fpr'], label='FPR')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "434d98c9",
   "metadata": {},
   "source": [
    "### Putting everything together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "d7a28486",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca7a695b0>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABLNUlEQVR4nO3dd3xN9//A8dcnm4jYQkRi7wgihNp7htorMapUUdVqKb9SHWip0iq1aWvvldqhFA2VmrVjj9SOmfH5/XES39CQSO69556bz/PRPJJ77uk975Pxdu7nvD/vj5BSoiiKohifnd4BKIqiKKahErqiKIqNUAldURTFRqiEriiKYiNUQlcURbERDnodOFeuXNLHx0evwyuKohjSgQMH/pVS5k7uOd0Suo+PD/v379fr8IqiKIYkhDj/sufUkIuiKIqNUAldURTFRqiEriiKYiNUQlcURbERKqEriqLYiBQTuhBithDihhDiyEueF0KIyUKI00KIQ0KIiqYPU1EURUlJaq7Q5wKNX/F8E6BYwsfbwNT0h6UoiqK8rhTr0KWUO4UQPq/YJQiYL7U+vHuFENmEEPmklFdNFWRSEyaswv38av4sEkTuvHnJnj07dkKY41CKkuG5uTjSpWpBMjvpNmVFeQ2m+Cl5AheTPL6UsO0/CV0I8TbaVTwFCxZM08GcLmzmrRwryB51gyFXehPHv2jpXCT+R5IvFEVJBylh9d+XmRlcGQ93F73DUVJg0X92pZTTgekA/v7+aVpZo0ijKXwyphBf1fucSlnvMS2qMoeO/sPRo0e5cOHCs/2yZMlC8+bNad++PU2aNMHFRf0yKsrr2vbPdQYsOEjQlF3MDK5MuQLueoekvIIpqlwuA15JHhdI2GYWxYrBmF0fsi/HWHweHmJs+fNsWLWU8+fPc+/ePfbu3cusWbPo1KkTW7Zs4c033yR37tx07dqVNWvW8PjxY3OFpig2p27JvCzvVw0HOzva/fQHZ6Ki9Q5JeQVTJPQ1QHBCtUtV4K65xs8BfHzA3h7W3ngHWn4PZ7bB3OYQHYWbmxtVqlShZ8+eTJ8+natXr7J582Y6duxIaGgoQUFB5M2bl+DgYNatW8eTJ0/MFaai2IySHllZ0a8a8RLm7D6ndzjKK6SmbHEhsAcoIYS4JIToJYToK4Tom7DLBuAscBqYAfQzW7SAo6OW1E+dAioGQ4df4MZxmNUAbp55bl8HBwfq16/PjBkzuHbtGr/99htt2rRh7dq1tGjRAg8PDxYvXmzOcBXFJuTN6kJQ+fys+Osy9x7H6B2O8hIpJnQpZScpZT4ppaOUsoCUcpaUcpqUclrC81JK+a6UsoiUspyU0uwtFIsVS0joACWbQcgaeHwXZjWESweS/X8cHR1p1KgRs2fP5vr166xfv55SpUrRsWNHRo8ejVosW1FeLaSaDw+fxrF0/yW9Q1FewpAzRYsVg9OntTvwAHgFQK/N4OQK85rDyY2v/P+dnJxo2rQp27dvJzg4mJEjR9K5c2cOHDjAw4cPzX8CimJAZT3dqeSdnZ/3RBIfry6ArJEhE3rRonD/Pty4kWRjrqJaUs9VDBZ2gr/mp/g6zs7OzJ07lzFjxrBo0SL8/f3JkiULRYsWpX///uzcuZO4uDjznYiiGExINR8ibz5kx6kovUNRkmHIhF6smPb52bBLIre80H09FK4NawZA2Ngkl/HJE0IwdOhQzp49y7Jlyxg1ahRly5Zl9uzZ1KpVCy8vL7788kvu379vlnNRFCNpXMaD3G7OzPsjUu9QlGQYOqGfPp3Mk85u0Hkx+HWBsDGwdiDExab4moUKFaJNmzZ8+umnrFq1ihs3brBo0SL8/PwYMWIEPj4+jBkzhpMnT6qrdiXDcnKwo0uVgoSdiCLy3wd6h6O8wJAJ3dtbK138zxV6IntHCJoCNYdoQy+LOsPT1/vly5IlCx06dGDDhg38+eefVK1alU8++YQSJUrg6upK+fLlqVWrFrVq1aJ27drMmjUr/SemKAbQuUpBHO0F8/e8dCU0RSeGTOiOjlCo0CsSOoAQUHcENJ8Ipzc/q1VPi8qVK7N+/XqOHDnCnDlz6N+/PwUKFMDOzg47Oztu3rzJW2+9xXvvvUdsbMrvBhTFyPK4udCkbD6W7r/Igyfq992aGLbjTmKlS4r8e0IWD1jWU6tV77occhZJ0zHLlClDmTJl/rM9Li6Ojz76iG+//ZaTJ0+yaNEi3N3VFGnFdoVU82HN31dYcfAy3ap66x2OksCQV+igVbqcOpXiPU9NyabP16pfTr5WPa3s7e2ZMGEC06dPZ8uWLVSvXp1z59SMOsV2VSyYjbKeWZn/R6Saw2FFDJvQixWD6Gi4fj2V/0PSWvW5Kdeqp0Xv3r3ZuHEjly9fJiAggN27d5v8GIpiDYQQhAT6cOpGNHvO3NQ7HCWBoRM6pHLYJVEaatVfV926ddm3bx/Zs2enbt26zJgxQ13BKDapRfn8ZM/syFxVwmg1DJvQixbVPr/yxmhy0lCr/rqKFy/O3r17qVmzJm+//TbNmjXj8mWzNaBUFF24ONrTMaAgW45f59JtNcPaGhg2ofv4gINDGhI6/K9WvXzn16pVfx05cuRg48aNTJ48mbCwMMqUKcP8+fPV1bpiU7pW9UYCi/68mOK+ivkZNqE7OGili6815JKUvSO0+jFJrXqn165VT4mdnR0DBgzg0KFDlC1blpCQEFq1asW1a9dMehxF0YtntkzULp6bJfsvEhsXr3c4GZ5hEzr8r9IlzZ6rVd+Srlr1VylatCg7duxgwoQJbNq0iTJlyvDll1+yfv16IiMjiY9XfwiKcXUMKMiN+0/Y9s+NlHdWzMrQCT2xjW66RzH8e0KHX1/aV90U7O3tGTx4MAcPHqRUqVKMGDGC5s2bU6hQITw9PRkwYAC7du1SyV0xnLol85DHzZlF4WrYRW+GTujFi8ODB3DliglerGRTCFlrtlr1Z4cpWZJdu3Zx69Ytdu3axbRp06hWrRozZ86kRo0aBAQEcPOmKgNTjMPR3o52/gUIO3GDK3ce6R1OhmbohO7rq33++28TvaBXZbPXqifKnj071atXp0+fPixfvpwbN24wc+ZMjhw5Qv369bl165bZjq0optaxckHiJSzZr67S9WTohF6+vPb54EETvmiuovDWFshV3Gy16slxc3OjV69erF69muPHj1O/fn1u375tkWMrSnp55chMjWK5WBJ+kTi1+IVuDJ3Qs2aFIkUgIsLEL5wlj1arXqSO2WrVX6ZRo0asXLmSo0ePUrduXVW/rhhGx8oFuXL3MTtOqpujejF0QgeoUMHEV+iJnLNAp0Wv3VfdFJo0acLq1as5ffo0AQEBHDhgnvF8RTGlBqXzktvNWbXV1ZHhE7qfH5w5A3fvmuHFTdBXPa0aN27M7t27cXBwoEaNGixcuFBNSlKsmpODHZ0DtMUvzqnFL3Rh+IReoYL2+dAhMx3AhH3VX5evry9//vkn5cuXp3PnzgQGBrJx40aV2BWr1aVKQRzsBPP3ROodSoZk+ITu56d9NsuwS1IWqFVPTt68edm5cyfTp0/n6tWrNG7cmAYNGvDggboCUqxPnqwuNC2Xj2X7L6nFL3Rg+ISeLx/kyWOGG6PJsVCt+oscHR3p3bs3p06dYtKkSWzbto3g4GA1CUmxSiHVvLn/JJYVB9UNfUszfEIXwow3RpNjwVr1Fzk5OTFw4EAmTJjAihUr+OSTTyx2bEVJrYoFs6vFL3Ri+IQO2rDL0aPw9KmFDmiBvuqvMmjQIPr27cu4ceOYOXOmRY+tKClRi1/oxyYSeoUKEBMDx45Z8KAv9lXf8Y3FatWFEEyePJmGDRvSu3dvmjVrxv79+y1ybEVJjRbl85PD1UktfmFhNpHQE2+MWmQcPanEvuq+HWD7F7BlpMWSuqOjIytXruSrr75i7969VK5cmVatWnHxopp6rejPxdGejpW91OIXFmYTCb1YMXB1teA4elL2jtBqmlYFs3sSbBgCFrpZmTlzZoYNG8a5c+f44osv2LJlC2XLlmXOnDlq7FLRXZeq3gD8sveCzpFkHDaR0O3stL4uuiT0xACafQvVBkD4DFjVF2ItNaAPWbNmZfjw4Rw6dIgKFSrQs2dPWrZsycOH6spI0Y9ntkw0LO3BovALPI6J0zucDMEmEjpowy4RERa7OP4vIaDB59okpEOLYUE7eHzPoiEULlyYbdu2MXHiRNavX09ISIgqbVR0FVLNhzsPY1gTYYoe10pKUpXQhRCNhRAnhBCnhRBDk3m+oBBiuxDioBDikBCiqelDfbWKFeH+/XQsSWcKQmhtAoKmwLnfYW5TuG/Z5ebs7OwYNGgQ33zzDcuWLeP//u//LHp8RUmqauEclMjrxlxVwmgRKSZ0IYQ9MAVoApQGOgkhSr+w2whgiZSyAtAR+NHUgaakenXt886dlj5yMip0hc5L4OZZmNkAok5aPITBgwfTu3dvvvrqK+bOnWvx4ysKaBVZwdW8OXb1HgfOq3bQ5paaK/QA4LSU8qyU8imwCAh6YR8JZE342h2w+PurEiUgb17YscPSR36JYvWh+zqIfQSzG8KFvRY9vBCCKVOmUK9ePd5++23mzZtn0eMrSqLWFTzJ6uKgShgtIDUJ3RNIWgt3KWFbUqOArkKIS8AGYEByLySEeFsIsV8IsT8qyrQNroSAmjW1hG417+w8K2oTkDLlgPlBcHytRQ/v6OjI8uXLqVGjBt27d2fYsGFqTF2xuMxODrT39+K3I9e4fu+x3uHYNFPdFO0EzJVSFgCaAj8LIf7z2lLK6VJKfymlf+7cuU106P+pWRMuXoTISJO/dNrlKKQldY9ysLgb/DnDood3d3fnt99+4+2332bs2LEEBQWxcOFCDh06xJMnTywai5JxdQv0Jk5Kft2nShjNKTUJ/TLgleRxgYRtSfUClgBIKfcALkAuUwT4OmrV0j5bzbBLItecELwGSjSBDR/CllEWfRvh6OjItGnTmDhxIps2baJz586UL1+eXLlysWvXLovFoWRc3jldqVMiDwv2XeBprHqXaC6pSejhQDEhRCEhhBPaTc81L+xzAagHIIQohZbQLdM0PIkyZSBHDitM6ABOmaH9z1CpB+yaCCv7WLRWXQjBoEGDuHfvHocOHWLhwoV4eHjQsWNH/v33X4vFoWRcIdV8+Df6CaFHruodis1KMaFLKWOB/sBG4DhaNctRIcRoIUTLhN0+AHoLIf4GFgLdpQ41SnZ2/xtHt0r2DtpCGTrWqjs7O1OuXDk6duzIkiVLiIqKonv37mpsXTG7GkVzUSiXq7o5akapGkOXUm6QUhaXUhaRUn6ZsO1TKeWahK+PSSmrSynLSyn9pJSbzBn0q9SqBefOaWPpVulZrfqP/6tVv6fPFUuFChX49ttvWb9+PRMnTtQlBiXjsLMTBAd6c/DCHQ5duqN3ODbJZmaKJrLacfQXVejyv1r1WQ0g6oQuYfTr148333yToUOHMn36dDX5QzGrtpUK4Opkz7w/1ELS5mBzCd3XF9zdDZDQQatV77EeYh9rKyBZuFYdtLH1WbNmUatWLfr06UPjxo1Vx0bFbNxcHHmzYgHWHrrCzWhVZWVqNpfQ7e2hRg2DJHSA/BW0ssbMOXWpVQfIli0bmzZtYsqUKezatYuyZcuq6hfFbEKqefM0Np5F4erCwdRsLqGDNuxy6hRcMUo/oBdr1fdNt3gIdnZ29OvXj8OHD+Ph4UGrVq04c8YyC2ErGUvRPG5UL5qTX/eeJzZO3Yw3JZtM6A0aaJ/Xr9c3jteStFY9dAhsHqlL68jChQuzfv16pJQ0b96c27dV/w3F9EICfbhy9zFbjl/XOxSbYpMJ3dcXvL1h9Wq9I3lNibXq/j1h93cW76ueqGjRoqxcuZIzZ87Qtm1bnlpssVYlo6hXKi+e2TKpEkYTs8mELgQEBcGWLRAdrXc0r8neQVsso+7/6VarDlCzZk1mzpzJtm3bqFq1KocPH7Z4DIrtsrcTdAv0Zu/ZW/xzzfK/37bKJhM6QKtW8OQJbNKtIj4dhICaH2q16pG7dKtVDw4OZuXKlVy+fJlKlSrx1Vdf8ejRI4vHodimDv5eODvYqRJGE7LZhF6jBmTPbsBhl6QqdNEWodaxVr1Vq1YcOXKE1q1bM3z4cPLkyUPXrl1Zs2YNjx+rznlK2mV3daKVnyerDl7m7sMYvcOxCTab0B0coFkzWLcOYmP1jiYditaHHhsg9oluteq5c+dm8eLFbNu2jY4dOxIaGkpQUBB58+YlODiYdevWqc6NSpoEV/PmUUwcSw+oEkZTsNmEDto4+q1bsHu33pGkU34/eGszuObSrVYdoE6dOsyYMYNr164RGhpK27ZtWbt2LS1atCBv3rx0796dDRs2qJuoSqqVye9OZZ/szN9znvh4NUs5vWw6oTdqBE5OBh92SZTdB3pu0q2velKOjo40btyYWbNmcf36ddavX0/r1q1ZtWoVzZo1w8PDg169erFx40ZiDf32SLGEkGo+XLj1kLCTN/QOxfBsOqG7uUG9elpCt4kWJS/2VdepVj0pJycnmjZtypw5c7h+/Tpr166lefPmLF26lMaNG1OnTh3Vnld5pUZlPMib1Zm56uZoutl0Qgdt2OXsWThwQO9ITCRpX3Uda9WT4+zsTPPmzZk/fz43btxg5syZ7N+/n4CAAI4dO6Z3eIqVcrS3o0sVb3aejOJMlNHqjK2LzSf0Dh20apdRo/SOxISsoK96SlxcXOjVqxdhYWE8fPiQwMBApk6dqq7WlWR1CiiIk70dP+9RV+npYfMJPVs2+PhjrQ2A4W+OJvViX/U5+vVVf5UqVaoQHh5OiRIl6NevHx4eHjRq1IgtW7boHZpiRXK7OdPMNx/LDlwi+om675JWNp/QAfr3Bw8PGDbMRsbSk0rsq35L377qr+Ll5cW+ffs4ePAgH330ESdOnKBRo0ZMmjRJ9V9XngkO9Cb6SSwr/rqkdyiGlSESuqsrjBgBv/9u0JmjKbGCvuopEULg5+fHV199xdGjR2nZsiWDBg2iX79+xMSoSSUKVCiYnfIF3Jn3R6T6hz6NMkRCB+jdG3x84JNPdC8MMY+kfdXntYRjL67jbT1cXV1Zvnw5Q4cOZdq0aXTq1En9ASuAVsJ4JuoBu0/f1DsUQ8owCd3JCT77DP76C+bP1zsaM0nsq57PF5YE69JXPbXs7OwYM2YM48aNY/ny5Xz//fd6h6RYgWa++cjp6qS6MKZRhknoAF27QvXq8P77cNX67h+ahpX0VU+tIUOG0KJFCz788EP279+vdziKzpwd7OkUUJCt/1zn4q2HeodjOBkqodvZwezZ8PgxvPOODd4gTZRcX/U46xynFkIwd+5cPDw86NChA3fv3tU7JEVnXaoWxE4IftmrShhfV4ZK6ADFi8Po0drs0SVL9I7GjJ71VU+oVV/cDWKssztijhw5WLhwIefPn6dp06bs3Wt9N3UVy8nnnonGZTxYFH6RR0/j9A7HUDJcQgdtyKVyZa2c8YYtt49IrFVvOh5OhsKC9vDEOmfiVa9enTlz5nDy5EkCAwNp2rQp4eHheoel6CQ40Ju7j2JY8/dlvUMxlAyZ0B0ctKGX6Gjo1Mng7XVTI6A3tJoKkb/DL2/CI+tcJ7Rbt26cO3eOsWPH8ueffxIQEECLFi04YDN9G5TUCiiUg5Iebsz947yqgHoNGTKhA5QtC1OnwrZtWimjzfPrDG3nwOW/YHZjuGudkzeyZMnCxx9/zLlz5/jqq6/YvXs3/v7+tGzZkoMHD+odnmIhQgi6V/Ph+NV7hEda5wWINcqwCR2ge3ft5ug338DSpXpHYwFlWkHX5XDvCsxsANeP6h3RS7m5uTFs2DAiIyP5/PPP+f3336lYsSKtW7cmPDxcXbVlAEF+nrhncmSeKmFMtQyd0AG++w6qVoUePeCE9c2aN73CtaBHKCC1K/VzO/WO6JWyZs3KiBEjiIyMZNSoUWzfvp2AgACKFy/O8OHDOXnypN4hKmaSycmeDpW9+O3oNa7dtc4b+tYmwyd0JydYtkwbVx88WO9oLMSjrDYBKWt++KUNHF6md0Qpcnd3Z+TIkURGRjJjxgx8fHwYO3Ysfn5+aozdhnWt4k28lPy6T5UwpkaGT+gAnp5ar5cNG2y010tysnlBz9/A0x+W94I/ftA7olTJli0bb731Fps3byYyMpLcuXPTokULLl2yznsCSvoUzJmZeiXzsPDPCzyJVSWMKUlVQhdCNBZCnBBCnBZCDH3JPu2FEMeEEEeFEAtMG6b5DRgAhQrBBx9AXEb5vcmUHbqthNJBsGk4/DbMqmeVvsjLy4t169YRHR1NixYtiI62zpJMJX1Cqvnwb/RTNhy21endppNiQhdC2ANTgCZAaaCTEKL0C/sUA4YB1aWUZYBBpg/VvJyd4euv4cgRraQxw3B0gbZzoco7sPdHWNbDaicgJadcuXIsXryYQ4cO0aFDBx49eqR3SIqJVS+Si8K5XdUSdamQmiv0AOC0lPKslPIpsAgIemGf3sAUKeVtACmlIafrtGmj9XoZMQLu39c7Gguys4PGY6DhF3BslVXXqienSZMm/Pjjj4SGhlKnTh2uXbumd0iKCdnZCUICffj74h0iLt7ROxyrlpqE7glcTPL4UsK2pIoDxYUQu4UQe4UQjZN7ISHE20KI/UKI/VFRUWmL2IyEgIkTtdmj332ndzQWJgRUGwBtZsHFP626Vj05ffr0Yfny5Rw+fJiAgAD+/vtvvUNSTKhNpQJkcXZgviphfCVT3RR1AIoBtYFOwAwhRLYXd5JSTpdS+ksp/XPnzm2iQ5tW5crQvLmW0DPkkGy5tklq1evDtSN6R5RqrVu3ZteuXcTHx1O9enXWrl2rd0iKiWRxdqBNRU/WHbrKv9FP9A7HaqUmoV8GvJI8LpCwLalLwBopZYyU8hxwEi3BG9Inn8CtWzBjht6R6ORZrbqAOU2svlY9qQoVKvDnn39SqlQpgoKCGD9+vJqEZCOCq/nwNC6ehfsu6B2K1UpNQg8HigkhCgkhnICOwIvL4axCuzpHCJELbQjmrOnCtKzAQKhTB8aPhycZ9WLAoyy8tRmyesLPbxqiVj1R/vz52bFjB23atGHIkCH07t2bWJtv2GP7iuTOQo1iufh13wVi4oxTjWVJKSZ0KWUs0B/YCBwHlkgpjwohRgshWibsthG4KYQ4BmwHhkgpDb2G1CefwJUrNry6UWq4F4CeoeAVkFCr/r1hmshnzpyZxYsXM3z4cGbNmkX37t2JyzD1qLYrJNCHa/ces+nodb1DsUpCr7ej/v7+0ppXqJESqlTRhl7++UebSZphxTyGlX20Cpgq70Cjr7TKGIMYM2YMn3zyCd26dWPOnDnY29vrHZKSRnHxktrjt5MvayaW9A3UOxxdCCEOSCn9k3vOOH+VFiaEdpV+5gz88ove0ejM0UXr1FjlHdg31XC16sOGDePzzz/n559/pnPnzqxfv57z51VbViOytxMEV/Xhz8hbHLtyT+9wrI5K6K/QsqU2nv7BB3A9o7/Ds7ODJmOh4ZeGrFUfMWIEn3/+OUuXLqV58+b4+PiQJ08e5s+frxK7wbTzL4CLox3z90TqHYrVUQn9FRLXIH3wQFvdSAGq9ddq1S+FG65WfcSIEdy8eZNdu3Yxbdo0ihcvTkhICK1atVKTkQwkW2YnWlfwZFXEZe48fKp3OFZFJfQUlCwJo0ZpHRmXGafQw7yeq1W37r7qL8qePTvVq1enT58+7Ny5kwkTJrBp0yZKly7NuHHjVD8YgwgO9OFxTDxL9l9MeecMRCX0VPjwQ6hUCd59F24aunbHhArV1Lo1giH6qifH3t6ewYMHExERQZUqVRg6dCiFChXim2++4cGDB3qHp7xCqXxZCSiUg/l7zhMXr4bMEqmEngqJa5DeugXDhukdjRXJWyahVt04fdWTU6JECUJDQ/njjz+oWLEiH330EYULF+bbb7/l4cOHeoenvET3aj5cuv2I7f8YsnWUWaiEnkq+vto4+qxZcOiQ3tFYEfcC2pV6gcqGq1V/UWBgIBs3bmT37t34+vrywQcfUKRIESZNmqS6OFqhhqXzks/dhXnq5ugzKqG/hv/7P3B316peDJqzzCNTdui6Akq3gk0jDNdX/UXVqlVj8+bN7Ny5k1KlSjFo0CCKFCnC999/ryYnWREHezu6VCnI76f+5fQNde8DVEJ/LTlywMiRsGULhIbqHY2VSaxVr9rPkLXqyalRowbbtm0jLCyM4sWLM3DgQD755BO9w1KS6BhQECd7VcKYSCX01/TOO1CsmHaVHhOjdzRW5llfdWPWqr9MrVq1CAsLo0+fPnz99deEqn/NrUauLM40L5+P5Qcucf+x+oNUCf01OTnBN99o7QAmTdI7GiuVWKtuwL7qrzJx4kR8fX0JDg7m8uUXG44qeulezYcHT+NYfsA2fs/SQyX0NGjZElq1gqFDYfNmvaOxUuXaQrcVhuyr/jKZMmVi8eLFPHr0iM6dO/P0qZrUYg18C2TDzysb8/ecJz6DlzCqhJ4GQmhdGEuVgvbt4eRJvSOyUs9q1RP6qp/doXdE6VayZEmmTp3Kzp078fDwoFevXmzcuFG1D9BZ92o+nP33Ab+f/lfvUHSlEnoaubnB2rVajXqLFnDb+EPF5vGsVt3T0LXqSXXr1o2NGzfSvHlzli5dSuPGjRk9erTeYWVoTcvlI1cW5wy/RJ1K6Ong4wMrV8K5c9pNUuUlEmvVDdhX/WUaNmzI/PnzuXHjBh07duSLL74gIiJC77AyLCcHOzoHeLHtxA0u3My4k8FUQk+nN97QWgLMn68lduUlMmV7vlZ90wjDJ3UAFxcXpkyZQs6cOenRowcxqvRJN12qemMvRIYuYVQJ3QSGDNGGXsaM0TsSK5dYqx7QB/b8AGvfg3jjT9TJkSMHU6dOJSIignHjxukdToaVN6sLjct6sGT/RR4+zZhLDqqEbgL580Pv3jB3Lpw/r3c0Vs7ODpqMgxofwF/ztJWQ4ox/Vdu6dWs6dOjA6NGj+e2339RNUp2EVPPh3uNYVh7MmGWlKqGbyMcfa9Uv6gItFYSAep9CvZFweCks7gZPjT/u+f3331OgQAGaNGlC7dq1CQsL0zukDMffOzul82Xl5z0Zc0UqldBNpEAB6NlTa951Sc1vSJ0ag6HpeDj5G8xvCQ+M3Zs4d+7cHDt2jMmTJ3Pq1Cnq1KnDgAEDMmRi0YsQgq5Vvfnn2n0OXryjdzgWpxK6CQ0dqvWk+uwzvSMxkIDe0OFnuHYYZjWAW8a+s+zi4sKAAQM4c+YM7733Hj/88APvvfeeSuoW1NIvP5md7Fn05wW9Q7E4ldBNyNsbBg2CmTNh2za9ozGQUi0geDU8vKkl9SsH9Y4o3TJlysTEiRMZPHgw33//Pe+//75K6haSxdmBFr75Wfv31QzX30UldBMbPVpr3vXWW9papEoqFawKvTaDQyaY0wxOb9E7onQTQjB+/Hjee+89Jk2axDvvvKPKGi2kU5WCPIqJY3XEFb1DsSiV0E0sUyZtHP3cORg+XO9oDCZ3cW1Wac7CsKADRCzQO6J0E0IwceJEhg4dyk8//USTJk24raYVm135Au6U9HBjYQYbdlEJ3Qxq1NBWN5o8GXbv1jsag3HzgO4bwOcNWPUO7PzG8BOQhBCMGTOGOXPmsHPnTgIDAzlz5ozeYdk0IQSdqxTk6JV7HL50V+9wLEYldDMZM0YbU+/QQVW9vDaXrNB5Kfh2gG1fwLr3Ic74E0W6d+/Oli1biIqKomXLlmohajML8vPExdGOheEZ5ypdJXQzyZIF1qyB+/ehaVO4m3EuEkzDwQla/wRvvA8H5sDirjZRq16zZk0WL17M8ePHGThwoN7h2DT3TI40K5ef1QcvZ5iboyqhm1G5crB8ORw/Dm3bgmqf/ZqEgPqj/lerPq+F4WvVAerXr8/w4cOZPXs2v/zyi97h2LTgQO8MtfiFSuhmVr++Vsa4ZYs28UitMZwGibXq14/YRK06wMiRI6lRowZ9+/blxIkTeodjs8p7ZazFL1RCt4CQEG1M/ddftXLG+Hi9IzIgG6tVd3BwYMGCBTg7O/Pee+/pHY5NC6nmnWEWv1AJ3UKGDtVmkM6dC2+/rZJ6mrxYq37K2LXqBQoU4IMPPmDjxo0cOnRI73Bslrb4hVOGWPwiVQldCNFYCHFCCHFaCDH0Ffu1EUJIIYS/6UK0HZ9+CiNGaHXq/fsbvhpPH0lr1Rd2gIO/6h1Rurzzzju4uroyfvx4vUOxWc4O9nQOKJghFr9IMaELIeyBKUAToDTQSQhROpn93ID3gH2mDtKWjB4NH34IU6eqzoxplrRWfXU/Q9eqZ8+enbfeeouFCxdy8eJFvcOxWRll8YvUXKEHAKellGellE+BRUBQMvt9DowDHpswPpuT2GK3Y0cYNgwWLtQ7IoOyoVr1xD4vkyZN0jsUm5U3qwuNMsDiF6lJ6J5A0kuHSwnbnhFCVAS8pJTrX/VCQoi3hRD7hRD7o6KiXjtYW2Fnp42l16wJ3bvDjh16R2RQL9aqLzFmX3Vvb2/at2/P9OnTuasmLJhN94TFL1YdtN3+Lum+KSqEsAO+BVJcJllKOV1K6S+l9M+dO3d6D21ozs7aAtOFC0NQEPz9t94RGVTSWvUToYbtqz5kyBDu37+vxtLNKHHxi3l/RNps58vUJPTLgFeSxwUStiVyA8oCYUKISKAqsEbdGE1ZjhywcSO4uUGjRnD6tN4RGdiLfdVvR+od0WupUKEC7du354svvmDIkCHEqQkLJieEIKSaNyeu32ffuVt6h2MWqUno4UAxIUQhIYQT0BFYk/iklPKulDKXlNJHSukD7AVaSin3myViG1OwIGzerE04atAALmfMpRBNI2mt+kzj1ar/+uuvvPvuu4wfP54333yT6OhovUOyOUF+nmTL7Mg8Gy1hTDGhSyljgf7ARuA4sERKeVQIMVoI0dLcAWYEJUtCaCj8+682s1Ql9XQoWBV6bQIHF8PVqjs4OPDDDz/www8/sH79evz9/dm7d6/eYdkUF0d7OlT2YtOx61y580jvcEwuVWPoUsoNUsriUsoiUsovE7Z9KqVck8y+tdXV+evz94cNG7TOjDVrQmSk3hEZWO4SSfqqtzdcrfq7777Lpk2bePjwIdWrV2fo0KE8fqyKx0ylaxVvpJT8sve83qGYnJopakVq1ICtW+H2bXjjDVAtPtIhsVa9UA1D1qrXrVuXw4cP07NnT8aNG0ejRo2IjbXdcjtL8sqRmXql8rIo/CKPY2zrXoVK6FYmIADCwiAmBqpXVyWN6WLwWnV3d3dmzJjB7Nmz2blzJyNHjtQ7JJvRvZoPtx48Zf2hq3qHYlIqoVshX19tpaNcubQx9enT9Y7IwGygr3qPHj3o1asXY8aMYdOmTXqHYxOqFclJ0TxZmLfHtkoYVUK3UkWLwt69WkLv0wfee0819EqzZPuqG6vz3uTJkyldujTdunXj6lXbuqrUgxCCkEBvDl26y8GLd/QOx2RUQrdi2bLBunUwaJC2Pmn37qCGUdPhub7qDQ3VVz1z5swsWbKE+/fv06VLF1WnbgJvViyAm7ODTXVhVAndytnbw7ffwhdfwM8/az1g1MpH6WDgvuqlS5dmypQpbN++nS+//FLvcAzP1dmBtv4FWH/4Kjfu20YVkUroBiAEDB8OEydqS9q9+aZ201RJIwP3Ve/evTtdu3bls88+IywsTO9wDK9bVW9i4iQL99lGp0uV0A1k0CCt7e769TB4sN7RGJxB+6oLIfjxxx8pWrQonTt3JiM3uTOFwrmzUKt4bn7dd56nsca/SaUSusH07QsffAA//ADTpukdjcG92Fd9hzFq1d3c3FiyZAm3bt2iU6dOatJROnWv5sON+0/YePSa3qGkm0roBjRuHDRtCgMGwPbtekdjcElr1bd/AesGGaJWvXz58kyfPp2tW7fSpk0bnjx5ondIhlWreG68c2a2if4uKqEbkL29tjBG8eLQsiV88w2ov+d0eK5Wfa5hatWDg4OZPn06GzZsoG3btiqpp5GdnaBbVW/2n7/NkcvG7kevErpBZc0Kv/0GtWvDRx9BmTKwapUhRgys04u16r+2gyf39Y4qRb1792batGmsW7eONm3aqA6NadTO34tMjvaGX6JOJXQD8/KCtWu1nurOztC6NdSrB2oB+XQI6A1tZsKFPTA/CB5af9/sPn36MG3aNEJDQ3njjTfU2qRp4J7JkdYVPVkdcYXbD4xbF6wSug1o2FBb8WjKFC2ZV6gA77wDD61/1MA6lWv7v8Uy5rWA+9f1jihFffr0Yd26dZw9e5aAgAD+/PNPvUMynJBAH57ExrMo3Lj/IKqEbiMcHKBfPzh1CgYOhJ9+0ro3qou1NCrZDDovhltntQlI/1r/clJNmjRhz549uLi40KBBA3Wl/ppKeLgRWDgnv+w9T1y8MccuhV6Nafz9/eX+/c+3TY+JieHSpUuqDCsdXFxcKFCgAJs3O9KxI2TOrI2tV62qd2QGdfkA/NoeZDx0XgJelfWOKEVnzpzB19eXWrVqsX79eoQQeodkGL8duUrfX/7ip26VaFTGQ+9wkiWEOCClTH6JTymlLh+VKlWSLzp79qyMioqS8fHx/3lOSVl8fLyMioqSZ8+elVJKefSolEWKSOnoKOWoUVI+fqxzgEb172kpvysv5ed5pTy+Xu9oUuW7776TgJw7d67eoRhKTGycDPxqi+w0fY/eobwUsF++JK9a1ZDL48ePyZkzp7qiSCMhBDlz5nz2Dqd0adi3D9q2hVGjtLH1Xbv0jdGQchbRWgXkKQWLu8D+2XpHlKIBAwZQvXp1Bg0axJUrV/QOxzAc7O3oUtWbP87c5NR1669yepFVJXRAJfN0evH7lzMnLFigLW/38KG2vN2HH4Ia1XpNWXJD93VQtL62UMbWz626RtTOzo7Zs2fz+PFj+vXrp3c4htIpoCBODnbMM2AJo9UldMU8mjSBI0e01gETJmgrI6nyxtfk5AodF0LFYPh9PKzqB3HW2yWtePHifPbZZ6xevZqtW7fqHY5h5HB1omX5/Kz46zL3Hlvvzzc5KqEncfPmTfz8/PDz88PDwwNPT89nj4UQ+Pn5UbZsWdq1a8fDhJpAe3v7Z9tbtGjBnTt39D2JV8iSBX78UWvudeMGVK6sXb0rr8HeAVpMhtqfwN8LYEEHq56ANHDgQAoWLMjHH39MvFohJdVCAn14+DSOpfsv6R3Ka1EJPYmcOXMSERFBREQEffv25f3333/22NXVlYiICI4cOYKTkxPTEjpjZcqU6dn2HDlyMGXKFJ3PImVNm8LhwxAYCF26aK0DrHj0wPoIAbU/hpbfw9kwmNvMamvVXVxc+OKLLzhw4ABLly7VOxzDKFfAnYoFs/HznkjiDVTC6KB3AC8zaNAgIiIiTPqafn5+fPfdd+l+nRo1anAomfGKwMDAZLdbo9y5tRmmwcFa64BLl7SFNOzt9Y7MQCoGQxYPWBqi1ap3XQG5iuod1X907tyZ8ePHM3z4cFq3bo2Tk5PeIRlCSDUf3lsUwY5TUdQpkUfvcFJFXaG/ptjYWEJDQylXrtxz2+Pi4ti6dSstW7bUKbLX5+ysNfkaPFhb4q5RI20oRnkNxRtqN0ufPtCS+kXrm6Fpb2/P2LFjOXPmDDNmzNA7HMNoUjYfud2cDdWF0Wqv0E1xJW1Kjx49ws/PD9Cu0Hv16vXc9suXL1OqVCkaNGigY5Svz85Ou0latqw209TPDxYv1maZKqnkWQl6bYJf22qtAtrOgZJN9Y7qOY0bN6Z27dqMHDmShg0bUqxYMb1DsnpODnZ0DijIpK2niPz3AT65XPUOKUXqCj2VEsfKIyIi+P7775+9bU3cfv78eaSUhhhDT06PHrB3r3bjtE4d6NMHLlzQOyoDSaxVz1vGKmvVhRBMnToVOzs76tSpw+nT1t/KwBp0qVIQBzvB/D3n9Q4lVVRCN5HMmTMzefJkJkyYQGys9S+QkJzy5WH/fq20cc4cKFYM3n0XTp7UOzKDcM0FIWuhaAOtVn3bF1Z1t7lkyZJs3bqVJ0+eUKdOHc6cOaN3SFYvT1YXmpbLx9L9F3nwxPr/rlVCN6EKFSrg6+vLwoUL9Q4lzbJm1Za3O31au2qfPh1KlIA33oBZs+C6dRZzWA8nV+i4QLthuvMbWP2uVdWqlytXjq1bt/Lo0SPq1q2r+qenQkg1b+4/iWXFwct6h5Iiq2rOdfz4cUqVKqVLPLbElN/Ha9fg559h9mz45x9tW+nS2rBM06Za/3VnZ5McyrZICTvGQdgYKFIP2s8H5yx6R/XMrl27qFGjBuPGjeOjjz7SOxyrJqWkxQ+7eBITz6b3a+o+m/1VzbnUFbrySh4eMGQIHDumDceMHastrDF3LjRrppU/duqkjb8rSQgBtYcmqVVvalW16m+88QaNGjVi/PjxPHjwQO9wrJoQgpBAH07diGbPmZt6h/NKqUroQojGQogTQojTQoihyTw/WAhxTAhxSAixVQjhbfpQFT0JAZUqwccfa0vf3bypzTht3x42b9YmKXXtqvqv/0fFYOi0CP49BbPqa5+txMiRI4mKino2SU55uRbl85PD1Ym5Vl7CmGJCF0LYA1OAJkBpoJMQovQLux0E/KWUvsAy4GtTB6pYF2dnbchl5kyIjIQRI2D5cm28/a23tGZgas3iBM9q1R/CrIZWU6seGBhIgwYN+Prrr5+1slCS5+JoT8fKXmw5fp1Lt633e5WaK/QA4LSU8qyU8imwCAhKuoOUcruUMvEs9wIFTBumYs2yZIHPP9fG2Dt0gCVL/jccExICO3ZYVbGHPhJr1V3ctVr1f9brHREAn376KTdu3OCnn37SOxSr16WqNvDwy17rredNTUL3BJK+kb6UsO1legGhyT0hhHhbCLFfCLE/Kioq9VEqhuDtrZU7RkVpV+jt22urJdWuDUWLai0GQkPhvvX2sjKvZ33VS8PirhA+S++IeOONN6hbty5ff/01jx490jscq+aZLRMNS3uwKPwCj2Pi9A4nWSa9KSqE6Ar4A98k97yUcrqU0l9K6Z87d25THlqxIs7OWrvemTPh6lWtSqZQIfjuO22YJnt26N4dMmTFXNK+6usHW0Vf9f/7v//j2rVrzJ8/X9c4jCCkmg93Hsaw5m/rXDQkNQn9MuCV5HGBhG3PEULUB4YDLaWUhh09TWyHm/gRGRlJWFgY7u7u+Pn5UapUKT777DOA57aXLFmSDz/8UOforU/mzNrN0i1b4M4d7Qbqu+9qST4gQKueyXCsrK96rVq18Pf3Z8KECcTFWeeVp7WoWjgHJfK6Me+PSPQq+X6ll61Nl/iB1u/lLFAIcAL+Bsq8sE8F4AxQLKXXS/xIbk3RY8eOmWzdvbRydXX9z7bt27fLZs2aSSmljI6OlkWLFpUHDhx4bvvDhw9liRIl5K5duywab3Ks4fuYkq1bpcyTR8rMmaWcNEnK6Gi9I9JBfLyU28dIOTKrlPNbS/n4nm6hLF68WAJyxYoVusVgFL/sjZTeH6+T4edu6nJ8XrGmaIrNuaSUsUKI/sBGwB6YLaU8KoQYnfDCa9CGWLIASxOK7i9IKdPVdnDQIDBx91z8/LS3/enh6upKpUqVOH36NHny/K+lZqZMmZ416VJSVrcuHDwI3brBe+/BZ59p/WMGDtRq3zOExFr1rPlh7SCtr3rnpeCW1+KhvPnmmxQqVIhvvvmG1q1bW/z4RtK6gidjQ/9h3p7z+Pvk0Duc56RqDF1KuUFKWVxKWURK+WXCtk8TkjlSyvpSyrxSSr+ED+P0kH1BYvdEPz+/ZH+xb968yd69eylTpsxz22/fvs2pU6eoWbOmpUI1vPz5taGY33/XbpyOHat1ffztN70js7CKwdBpYUKtegP41/KNsxwcHBg8eDB79uxh9+7dFj++kWR2cqC9vxehh69y/Z6VLc77skt3c38Ybcgla9as0s/PT1asWFFOnTr1ue2+vr4yU6ZMctiwYZYON1nW8H1Mi2PHpPT1lVIIKUeMkDI2Vu+ILOzSfinHFZZyrI+UF/ZZ/PDR0dEyZ86csmXLlhY/ttFE/hstfYaukxM2nbD4sXnFkIua+p9KNWrU4ODBgxw4cIC+ffs+t/3vv//m6NGjzJo1y+SrLGUkpUppLQR69IAvvoCqVWHpUjBo88rX51kJ3tqsW626q6sr7777LmvWrGHlypUWPbbReOd0pU6JPCzYd4GnsdazVqtK6CZSqFAhhg4dyrhx4/QOxdAyZdK6Ov7yC9y+rdWyFy0Kn3yiNQjbtk2rc7dZOQrrWqs+ZMgQAgMDad++PatXr7bosY0mONCbf6OfEHrkqt6hPKMSugn17duXnTt3EhkZqXcohtelC5w4AStXahOWxo2DXr207o6entpaqAcP6h2lmTyrVW9g8Vr1LFmyEBoaSqVKlWjXrh1r1661yHGNqGax3BTK5WpV/V1U+1wbZIvfx5gYbSHrc+dg9WrtKv7BA/D31+rZK1SAWrW0RTlsRlwsrH8f/poPfl2hxSSwt8yqkXfv3qVBgwZEREQQHh5O+fLlLXJco5mz+xyfrT3Gmv7V8S2QzSLHVO1zFcNzdNRmm9atC5Mmacn9m2+0IZqff4bevbU+7dOn6x2pCdk7QIvJUGsoRPwCy3tB7FOLHNrd3Z3Q0FDc3d3p37+/dU6isQJtKhUgs5M98/6wjiXqVEJXDClbNvjwQ9i5U5uBevIkNGig1bIPGGBDN1KFgDrDoOEXcGyVNq4eY5meKzlz5mTs2LHs2rWLX3/91SLHNJqsLo60qViAtYeucDNa/wnyKqErhmdnpw21rF0LH3ygLaHXoIE2Bm8zqg2A5hPh1Cb4tR08umORw/bo0YPKlSszZMgQ7t27Z5FjGk1INW+exsazKFz/xQBUQldshr09jB+vraZ04IA2Sen997VqGZvg3xPenAEX9sCcpnDP/A2i7OzsmDJlCtevX3/Ww0h5XtE8blQvmpNf954nNk7fEkaV0BWbExICp05Bz54webJWJdO5MyxbZgMdHn3bQZelcOcCzGwAN46b/ZCVK1emV69eTJo0iZ07d5r9eEYUEujDlbuP2XJc32UGVUJXbFLevPDTT/DXX9CundblsV07yJNH6x+zfTvEW898kNdTpC702ADxMTC7EUSaf6r+2LFjKVasGE2bNmXXrl1mP57R1CuVF89smXQvYVQJPYmbN28+6+Pi4eGBp6fns8dCCPz8/Chbtizt2rV7tmRXYrvdsmXL0qJFC+7cuZPsa1erVi3Z7d27d2fZsmVpinfUqFGMHz8+Tf9vRlG+vFbiePUqhIVp9etr1mjVMoUKadUxCxZozxtKPl9tAlKWvPBzKzhq3pmdOXPmZNu2bXh6etKkSRP++OMPsx7PaOztBN0Cvdl79hb/XNPvXoNK6EnkzJmTiIgIIiIi6Nu3L++///6zx66urkRERHDkyBGcnJyeLaybKVOmZ9tz5MjBlClTkn1t9QegLwcHrU592jS4dg1+/VXrvrl0qTaJqUAB6NsXbtzQO9LXkN0bem6E/BVhaQ/Y86NZD5cvXz62b99Ovnz5aNy4MYcOHTLr8YymY2UvnB3smL9HvxJGy8xSSIPP1h7l2BXT/ktXOn9WRrYok/KOKahRo0ayv8yBgYEv/SXPkiUL0dHRSCkZMGAAmzdvxsvLCycnp2f7HDhwgMGDBxMdHU2uXLmYO3cu+fLlY8aMGUyfPp2nT59StGhRfv75ZzJnzpzu88ioMmXSxtQ7d4a4OK1N87x5MHUqLFwIw4ZBzZrg46O18rWz5suezDkgeBWs6A0bh8G9y9Dgc7MFnT9/frZv307lypVp3749+/fvJ0uWLGY5ltFky+xEKz9PVv51mY8blcQ9s6PFY7DmX1WrFBsbS2hoKOXKlXtue1xcHFu3bqVly1d3Dl65ciUnTpzg2LFjzJ8//9mVe0xMDAMGDGDZsmUcOHCAnj17Mnz4cEDrVR0eHs7ff/9NqVKlmDVL/7UobYW9PVSqpN08PXwY3nhDS+jVq2stBtzctCv4rVuteMzdMRO0mwcBb8OeH2DFWxBrvppoT09PFixYwKlTp3j33XfNdhwjCq7mzaOYOJYe0KeE0Wqv0E1xJW1KiX3SQbtC79Wr13PbL1++TKlSpWjQoMErX2fnzp106tQJe3t78ufPT926dQE4ceIER44cefb/x8XFkS9fPgCOHDnCiBEjuHPnDtHR0TRq1MhMZ5mxlSwJ69drFTKnT2ttBv7+G5Ys0cbZvby0FgM+PlC4sDYOX7asNvdHd3b20ORryOoJW0ZC9A3o+KvWudEMateuzaeffsqoUaOoU6cO3bt3N8txjKZMfncq+2Rn/p7z9KheCHs7y/5yWG1CtzaJY+Uv2/7w4UMaNWrElClTGDhw4Gu/vpSSMmXKsGfPnv881717d1atWkX58uWZO3cuYWFhaTgDJbWKFXu+J8ykSbBqlTbefuqU1vExsfyxSBF4802tcViJErqE+z9CwBuDwC0frO4Hs5tA12XaikhmMGLECMLCwnj33XdxdHSkY8eO2Nvbm+VYRhIc6MOAhQfZcfIGdUtadvUpNeRiIpkzZ2by5MlMmDCB2FfMO69ZsyaLFy8mLi6Oq1evsn37dgBKlChBVFTUs4QeExPD0aNHAbh//z758uUjJiZGTcHWgYsLdOwIy5fDoUNw757WS2baNK2173ffaVf3zZtbydBM+Q7QZVlCrXp9s9Wq29vbs2DBAooVK0bXrl0pW7YsCxcuJF73b4C+Gpf1IG9WZ+bq0N9FJXQTqlChAr6+vixcuPCl+7Ru3ZpixYpRunRpgoODCQwMBMDJyYlly5bx8ccfU758efz8/J6Nr3/++edUqVKF6tWrU7JkSYuci/JyQmjj6336aMvlXboEo0bBn39C/fqQNas2Bt+/P4SH6xRkkToJtepxMMt8ter58uXjr7/+YsmSJdjb29O5c2fat2/PgwcPzHI8I3C0t6NLFW92noziTJSFZ7K9bCkjc39Y6xJ0tkB9H/Xx6JGUv/4q5cCBUtaoIaWrq5QgZXCwlJcv6xTU7fNSfu8v5ehcUh5ZYdZDxcXFyfHjx0shhKxYsaK8dOmSWY9nza7feySLfrJejlx9xOSvjVqCTlHMz8VFK4WcNEnrAnnlCnz8MSxapI3JDxqkDdlYVLaCz9eq751qtkPZ2dnxwQcfsGbNGk6ePElAQECGrVXP4+ZCs3L5WHbgEtFPLNf6UyV0RTGTrFlh7Fg4fhyCguDHH7WZq/7+MGSI9njDBrh508yBJNaql2wGvw2FTSPMOtDfvHlzdu/ejRCCevXqcfjwYbMdy5qFVPMh+kksK/66ZLFjqoSuKGZWuLBW9njlinb1LgR8/z28+y40awb582t9ZjZsMGMfd8dM0H4+VO4Nf3xv9lp1X19fwsLCcHZ2pl69es9u8Gckfl7Z8C3gzrw/Ii22QIhK6IpiIblywcCB2o3Shw+1BL9zJ/Trp/WZadYMcuaEFi3g229h3z5tP5Oxs4em30D9UXBkOfzSxqx91YsWLcr27dtxcHCgbt26vLjkpK0TQhAS6MOZqAfsPm3ut2EaldAVRQd2dpAvH9SoARMnwuXLsGKFVh554oS2UEfVqtpM1TJltKv58HATrBUtBLzxfkJf9b0wpwncvWySc0pOsWLF2L59O05OTlStWpWRI0fy9KllltGzBs1885HT1cliXRhVQlcUK+DkBK1bay1/T57USiFXrIDhw7WukLNna4th+/pqJZJr1sDFi+lI8L7tE/qqX4RZ5u2rXqJECQ4dOkSXLl0YPXo0AQEBrFmzhidP9F+yzdxcHO3pGODF1n+uc/GWKd9uJU8l9BcktsNN/IiMjCQsLAx3d3f8/PwoVarUs5Vbkm4vWbIkH374YbKvuX///pfOHvXx8eHff/9NU6y1a9fOcG9jMwpPTy3Bjx4N69Zp7X2nTQNXV21bUBAULKj1d2/QAD76SBunP35caziWKs9q1WMTatXN1+c8e/bszJs3j9WrV3Pjxg2CgoLIkycPwcHBnDx50mzHtQZdq3pjJwS/7LXARKOX1TOa+8Na69BdXV3/s2379u2yWbNmUkopo6OjZdGiReWBAwee2/7w4UNZokQJuWvXrtc6nre3t4yKikpTrLVq1ZLh4eH/2W4N30fFfO7dk3LXLim//17KXr2krFhRSicnreYdpMyUSUp/fynbtZNyyBApf/pJyn37pHz48CUvmLRW/fBys8f/9OlTGRoaKnv06CHd3Nykl5eXvHr1qtmPq6d3ftkvfUdtlA+fxKb7tXhFHbr19nIJHQrXTFzu5FEOmoxN10u4urpSqVIlTp8+TZ48eZ5tz5Qp07MmXS8KCwtj/PjxrFu3jps3b9KpUycuX75MYGDgc3e/f/nlFyZPnszTp0+pUqUKP/74I/b29rzzzjuEh4fz6NEj2rZtq9Z2zODc3LSZqNWr/29bTIx2dX7woPZx7JjWFnj1akgcsra31ypucuSAbNm06ppmzaBx44K49twICzvBsp5w/xoE9jNb/I6OjjRu3JjGjRvTv39/atSoQVBQENu3b7fZttDBgT5sOHyN1RGX6RhQ0GzHUUMuL0jsnujn50fr1q3/8/zNmzfZu3cvZco83w3y9u3bnDp1ipo1a77y9T/77DPeeOMNjh49SuvWrblw4QIAx48fZ/HixezevZuIiAjs7e2f9W358ssv2b9/P4cOHWLHjh0ZdrKG8nKOjtr4ekiI1ltm0yZtLP7RIzhzRutDM2wYVKwI7u7awtmrV0Pbtlr1TesuOViTfRXxJZprfdU3DrdIU5qKFSuyYMECwsPDCQkJsdk+MFUK5aCkhxvz9pw3awmj9V6hp/NKOq1e1lXx999/p0KFCtjZ2TF06FDKlClDWFgYv//+O+XLl+fUqVMMGjQIDw+PV77+zp07WbFiBQDNmjUje/bsAGzdupUDBw5QuXJlQPuHJfEdwJIlS5g+fTqxsbFcvXqVY8eO4evra8KzVmyVnZ12VV64sNYVMqnYWPj9d1i5UuskuWpVJvLmmceSHkOpuecH7l2+SpauU7FzcjZrjEFBQXzzzTd8+OGH+Pr6PlvSsWbNmlStWhU7q15hJHWEEIRU82HYisOER94moFAOsxwnVQldCNEYmATYAzOllGNfeN4ZmA9UAm4CHaSUkaYNVV81atRg3bp1L91+7tw5qlatSvv27Z/1TX8dUkpCQkIYM2bMc9vPnTvH+PHjCQ8PJ3v27HTv3p3Hjx+n9TQU5RkHB6hTR/v49lvYuBHmzLGnwcSvec/fk68bjGRH3xt8fmgqznm98PbW6uSzZdM+smf/79dZs2pDO69r8ODBODo6Ehoayo4dO569Oy1QoADt2rWjZ8+elC1b1nQnr4Mgv/yM2XCceX9E6pfQhRD2wBSgAXAJCBdCrJFSHkuyWy/gtpSyqBCiIzAO6GCOgK1VoUKFGDp0KOPGjXtlt8WaNWuyYMECRowYQWhoKLdv3wagXr16BAUF8f7775MnTx5u3brF/fv3uXfvHq6urri7u3P9+nVCQ0OpXbu2hc5KySgcHLTx9GbN4OlTwbFjg9i5Kz/V5ADWF/Bn3qmBfL58EJejXFMslcya9X9J3stLWxDExwdy507+HwJXV+0KduDAgc+qwW7fvs369etZsmQJU6ZMYeLEibRv356RI0dSunRps34vzCWzkwMdKnsxe3ckV+8+Ip97JpMfIzVX6AHAaSnlWQAhxCIgCEia0IOAUQlfLwN+EEIIac7BIivUt29fxo8fT2RkJD4+PsnuM3LkSDp16kSZMmWoVq0aBQtqN0hKly7NF198QcOGDYmPj8fR0ZEpU6ZQtWpVKlSoQMmSJfHy8qJ60jthimIGTk7aAtr4tYc7gbBlFG/bfc3bfjORWfIQH6+VRsbHaZ/j4p//OulzMTHwNAbizwBnnj/OvYQPBNjbaVf2SVd/qgpULQKyX3ni4uKIizuB/KEzx6xiiai0aW0vaZftKfO+6sjQcZ+a/PVFSjlXCNEWaCylfCvhcTegipSyf5J9jiTscynh8ZmEff594bXeBt4GKFiwYKXz55+vyzx+/DilSpVK90lldOr7qJjchX2wfxbEpm2472kMPH2SkOCf/i/RxzxN+Jzw9avuicbHxxMdHU1MTEwaT8JKOEnu+7Sk04i+afrfhRAHpJT+yT1n0ZuiUsrpwHQAf3//DHX1riiGVrCK9pFGTgkfinml5vbxZcAryeMCCduS3UcI4QC4o90cVRRFUSwkNQk9HCgmhCgkhHACOgJrXthnDRCS8HVbYFtax88z2LC7yanvn6JkXCkmdCllLNAf2AgcB5ZIKY8KIUYLIVom7DYLyCmEOA0MBoamJRgXFxdu3rypklIaSSm5efMmLi4ueoeiKIoOUrwpai7+/v7yxcZSMTExXLp0SdVZp4OLiwsFChTA0dFR71AURTEDq7kpmhJHR0cKFSqkdxiKoiiGZPw5tYqiKAqgErqiKIrNUAldURTFRuh2U1QIEQWkdQmPXEDalvkxLnXOGYM654whPefsLaXMndwTuiX09BBC7H/ZXV5bpc45Y1DnnDGY65zVkIuiKIqNUAldURTFRhg1oU/XOwAdqHPOGNQ5ZwxmOWdDjqEriqIo/2XUK3RFURTlBSqhK4qi2AirTuhCiMZCiBNCiNNCiP90cBRCOAshFic8v08I4aNDmCaVinMeLIQ4JoQ4JITYKoTw1iNOU0rpnJPs10YIIYUQhi9xS805CyHaJ/ysjwohFlg6RlNLxe92QSHEdiHEwYTf76Z6xGkqQojZQogbCSu6Jfe8EEJMTvh+HBJCVEz3QaWUVvkB2KOtQlgYbbGTv4HSL+zTD5iW8HVHYLHecVvgnOsAmRO+ficjnHPCfm7ATmAv4K933Bb4ORcDDgLZEx7n0TtuC5zzdOCdhK9LA5F6x53Oc64JVASOvOT5pkAoINCWUN2X3mNa8xX6s8WppZRPgcTFqZMKAuYlfL0MqCeEgVeQTcU5Sym3SykfJjzci7aClJGl5ucM8DkwDrCF3sqpOefewBQp5W0AKeUNC8doaqk5ZwlkTfjaHbhiwfhMTkq5E7j1il2CgPlSsxfIJoTIl55jWnNC9wQuJnl8KWFbsvtIbSGOu0BOi0RnHqk556R6of0Lb2QpnnPCW1EvKeV6SwZmRqn5ORcHigshdgsh9gohGlssOvNIzTmPAroKIS4BG4ABlglNN6/7954iq+qHrqSeEKIr4A/U0jsWcxJC2AHfAt11DsXSHNCGXWqjvQvbKYQoJ6W8o2dQZtYJmCulnCCECAR+FkKUlVLG6x2YUVjzFXpGXJw6NeeMEKI+MBxoKaV8YqHYzCWlc3YDygJhQohItLHGNQa/MZqan/MlYI2UMkZKeQ44iZbgjSo159wLWAIgpdwDuKA1sbJVqfp7fx3WnNAtuji1lUjxnIUQFYCf0JK50cdVIYVzllLelVLmklL6SCl90O4btJRS7k/+5QwhNb/bq9CuzhFC5EIbgjlrwRhNLTXnfAGoByCEKIWW0KMsGqVlrQGCE6pdqgJ3pZRX0/WKet8JTuEucVO0K5MzwPCEbaPR/qBB+4EvBU4DfwKF9Y7ZAue8BbgORCR8rNE7ZnOf8wv7hmHwKpdU/pwF2lDTMeAw0FHvmC1wzqWB3WgVMBFAQ71jTuf5LgSuAjFo77h6AX2Bvkl+xlMSvh+HTfF7rab+K4qi2AhrHnJRFEVRXoNK6IqiKDZCJXRFURQboRK6oiiKjVAJXVEUxUaohK4oimIjVEJXFEWxEf8PX8nF+zQWLj4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_scores.threshold, df_scores['tpr'], label='TPR', color='black')\n",
    "plt.plot(df_scores.threshold, df_scores['fpr'], label='FPR', color='blue')\n",
    "\n",
    "plt.plot(df_ideal.threshold, df_ideal['tpr'], label='TPR ideal')\n",
    "plt.plot(df_ideal.threshold, df_ideal['fpr'], label='FPR ideal')\n",
    "\n",
    "# plt.plot(df_rand.threshold, df_rand['tpr'], label='TPR random', color='grey')\n",
    "# plt.plot(df_rand.threshold, df_rand['fpr'], label='FPR random', color='grey')\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "bc605391",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca6b72e50>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAE9CAYAAABtDit8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0wklEQVR4nO3dd3xUVd7H8c9J7wmk0AIEpHcwVEVBlFWkqYigdBVX17K6rrrrWtZ1H13d9uji4+JKlapYUEFFAWFZBIL0nkCAUNJ7n8x5/riRjUgKZGbuzNzf+/XKi7kzNzO/m8CXc+659xyltUYIIUTtfMwuQAgh3J0EpRBC1EOCUggh6iFBKYQQ9ZCgFEKIekhQCiFEPfzMLuByxcTE6ISEBLPLEEJ4mZ07d2ZprWMv9ZrHBWVCQgJJSUlmlyGE8DJKqZO1vSZdbyGEqIcEpRBC1EOCUggh6uFx5ygvpbKykrS0NMrKyswuxSMEBQURHx+Pv7+/2aUI4RG8IijT0tIIDw8nISEBpZTZ5bg1rTXZ2dmkpaXRrl07s8sRwiN4Rde7rKyM6OhoCckGUEoRHR0trW8hLoNXBCUgIXkZ5GclxOVxWlAqpeYppTKUUvtreV0ppd5QSiUrpfYqpfo5qxZXUEoxZcqUC9s2m43Y2FhGjx59We+TkJBAVlZWo/cRQjiOM1uUC4Cb63j9FqBj9dds4P+cWIvThYaGsn//fkpLSwFYt24drVq1MrkqIYQjOG0wR2u9SSmVUMcu44BF2phi/TulVJRSqoXW+pyzanK2UaNG8fnnnzNhwgSWLVvG5MmT2bx5MwA5OTnMmjWL48ePExISwty5c+nVqxfZ2dlMnjyZM2fOMHjwYGrOOP/ee+/xxhtvUFFRwcCBA3nrrbfw9fU16/CEcAmtNZVVmsoqO5VVdiqq7Ma27b/bFTb7hX0qquzVrxnbVRUllGl/xvZpRXiQY67sMHPUuxVwusZ2WvVzPwlKpdRsjFYnbdq0cUlxV2LSpEm89NJLjB49mr179zJr1qwLQfnCCy/Qt29fPv74Y9avX8+0adPYvXs3v//977n22mt5/vnn+fzzz3n33XcBOHToECtWrGDLli34+/vz0EMPsWTJEqZNm2bmIQovprWm3GantKKK0soqym12ym1VlFfaL/3YZqf8wn61vf7fxxUXgu6/IVhpu2i7OhSvVDglLAx4le/s3cjq8LZXBGWDaa3nAnMBEhMT6/wp/v7TAxw8W+DQz+/WMoIXxnSvd79evXqRmprKsmXLGDVq1I9e+/e//82qVasAuOGGG8jOzqagoIBNmzbx4YcfAnDrrbfSpEkTAL755ht27txJ//79ASgtLSUuLs6RhyW8hNaawnIbGQVlpBeUk1FYRkZBOUXltguhd+HPyipKKqooq37uwuPqr8YsoeXnowj08yHQ39f408+HQD9fAqofB/j5EBbkh7+vDwG+Pvj7Kvx9ffD3u2jb19j3R9u+Pvj7XbT9w/dUf39gZT6t10whMOsk7W99lrAmwQ77GZsZlGeA1jW246uf82hjx47lySefZOPGjWRnZ1/x+2itmT59Oq+88ooDqxOeRmvN2fwyTueUkF5gBGB6QRnpheXV20Y4llZW/eR7lYIQf1+CA3wJ8vclJMCXYH/jcXRoAMFNfvx8cICf8ae/z4Xv+SHsjAD0IcDXl0B/n0s874Ofr4kX0VQUw7y7IfsI3PUeUZ3rGh65fGYG5WrgYaXUcmAgkO+I85MNafk506xZs4iKiqJnz55s3LjxwvNDhw5lyZIlPPfcc2zcuJGYmBgiIiK47rrrWLp0Kb/73e9Yu3Ytubm5AIwYMYJx48bx+OOPExcXR05ODoWFhbRt29akIxPOllVUztHzhRxJL+RoeiFHzhdyLL2IwnLbj/YL9veleWQQseGB9IyP4sbwQJpFBBEXYfzZLMJ4LTTA1zqXgvmHQMeRcOOL0OFGh7+904JSKbUMGAbEKKXSgBcAfwCt9dvAGmAUkAyUADOdVYsrxcfH8+ijj/7k+RdffJFZs2bRq1cvQkJCWLhwIWCcu5w8eTLdu3dnyJAhF87BduvWjZdffpmRI0dit9vx9/dnzpw5EpReIr+0kr1peew6lcfu03nsTcsjq6jiwutNQvzp1Cyc2/u1olPzcBKiQ6tDMJCwQD/rBGB9Cs5BeQHEdoYRzzvtY5SnreudmJioL56P8tChQ3Tt2tWkijyT/Mxc73x+GW+sP8a249mkZBZfeL5DXBi946Po1jKCzs3C6dQ8jNiwQAnD+uSnwcIxgIJfbAffxrX7lFI7tdaJl3rNIwZzhPBkdrtm2Y5TvLrmMBVVdq7tEMNtfVvRp3UTerWOJMJBI7OWknsSFo6G0jyYsqrRIVkfCUohnOhEVjHPrNrLthM5DG4fzat39KRtdKjZZXm27BRYOBYqimDaJ9DK+Tf1SVAK4QBaa4rKbWQWlpNVVEFmYTmHzxcwd9NxAvx8+NMdPZmY2Fq6046w8VWwlcL0T6FFL5d8pASlEA1kq7KTlltKSmYRKZlFJGcUcTyzmPMFZWQVlVNWaf/J9/ysezNeGteDZhFBJlTspcb8HQrOQkxHl32kBKUQl6C1Zk9aPusPZ3AsvZCUzCJSs0qoqPpvGMaEBdA+JozEtk2IDQ8kNjyQmLDAC4/jwoNoGhpg4lF4kfP7YP0f4Y53IDDcpSEJEpRCXFBl1+xIzeGL/ef58sB5zuWX4aMgITqU9rFhDO8Sx1WxYdVfoUSFSAi6xNldsGg8BIRCcZYRlC4mQekgvr6+9OzZE5vNRrt27Vi8eDFRUVGNft8FCxaQlJTEP/7xj8YXKSitqCKrqJzMonIyC8urzymWk5ZbysYjGWQVVRDo58N1nWJ5cmRnbuzajMgQGZU2zekd8N4dEBxpnJNskmBKGRKUDhIcHMzu3bsBmD59OnPmzOHZZ581tygLyi+p5MNdaazYcZpTOSU/es2u9SXPIwJEhwYw+Kpobu7RnOGd4wgNlH8apju1Dd67HUJjjZCMal3/9ziJ/G1wgsGDB7N3714Atm/fzmOPPUZZWRnBwcHMnz+fzp07s2DBAlavXk1JSQkpKSncdtttvPbaawDMnz+fV155haioKHr37k1gYCAAqampzJo1i6ysLGJjY5k/fz5t2rRhxowZBAcHs2vXLjIyMpg3bx6LFi1i69atDBw4kAULFpj1o3AJrTW7TuexdNspPtt7lrJKO73jI7l7QBtqDjIrpYgM9r9wDjG2+nxi09AA/M28T1lcWlgstLoabnsbIlqaW4vW2qO+rr76an2xgwcP/uQ5VwsNDdVaa22z2fSECRP02rVrtdZa5+fn68rKSq211uvWrdO333671lrr+fPn63bt2um8vDxdWlqq27Rpo0+dOqXPnj2rW7durTMyMnR5ebkeMmSI/sUvfqG11nr06NF6wYIFWmut3333XT1u3DittdbTp0/Xd911l7bb7frjjz/W4eHheu/evbqqqkr369dP79q16yf1usPPrLEKSiv0ov+c0D/727e67dOf6W7PrdW/+XCv3peWZ3ZpojHSD2ptt7v8Y4EkXUvueGeLcv6tP32u+3gYcD9UlMCSO3/6ep+7oe89UJwNKy+a83Hm5/V+ZGlpKX369OHMmTN07dqVm266CYD8/HymT5/OsWPHUEpRWVl54XtGjBhBZGQkYNzbffLkSbKyshg2bBixsbEA3HXXXRw9ehSArVu3XpiSberUqTz11FMX3mvMmDEopejZsyfNmjWjZ8+exmF3705qaip9+vSp9xg8RWZhOXM2JLMy6TQlFVV0axHBH2/rwbg+rQiTLrNnO/oVrJgCw38L1/7S7GoukL9VDvLDOcqSkhJ+9rOfMWfOHB599FGee+45hg8fzkcffURqairDhg278D0/dKnBGAyy2WyXeOeG+eG9fHx8fvS+Pj4+jXpfd1JUbmPupuP8a/Nxym12xvdpxdTBbekdHykXcnuDQ5/B+zOgWXfo514TVHtnUNbVAgwIqfv10OgGtSBrExISwhtvvMH48eN56KGHyM/Pv7B2TkPOFQ4cOJDHHnuM7OxsIiIieP/99+nduzcAQ4YMYfny5UydOpUlS5YwdOjQK67Tk1TY7CzddpI31yeTXVzBqJ7NeXJkZ9rHhpldmnCUAx/BqvugRR/j3u3gKLMr+hHvDEqT9e3bl169erFs2TKeeuoppk+fzssvv8ytt17ilMBFWrRowYsvvsjgwYOJior6UZf5zTffZObMmbz++usXBnO8mdaaz/ae4/Uvj3Aqp4RB7Zvy7i1d6dM6yuzShCMVZcBHD0J8f7h7JQRFmF3RT8g0axbl7j+zvWl5vPTpQZJO5tK1RQRP39yZ6zvFShfbW6VugRa9IdC8XoJMsyY8RkZBGa99eYQPdqYRExbAq7f35M7E1vj6SEB6naT5xszkve+ChGvMrqZOEpTCLdjtmsXfneS1Lw5TWaV54Pr2PDy8g8NW0RNuZttcWPtr6DwKek0EN+8pSFAK0yVnFPL0qn3sPJnL0I4x/GFcDxJiZM5Gr/WfN+Gr30GX0TBhvtuHJHhRUGqt5fxVA5l9XtpWZedsXhknsovZfiKbdzadICTQl79O7M1tfVvJ79Gbbf4LfPMSdBsPd/wLfD2jx+AVQRkUFER2djbR0dHyj6weWmuys7MJCnLd/IjF5TY2H8vi60PpfH8yl9O5JT9a5H50rxa8OLY7MWGBdbyL8AqVpdBzIoz/P6cv3+BInlNpHeLj40lLSyMzM9PsUjxCUFAQ8fHxTv+c9YfTWbz1JFtSsqmw2YkI8mNQ+2h+1qM57aJDaRsdQrvYUOLCZVJbr6Y1FJ4z7tce/qyx7eNZ99Z7RVD6+/vTrl07s8sQNXyy+wyPr9hNi8hgpg5qy4iucfRPaCqTT1iN1sb5yN1L4eebITLeI85JXswrglK4ly/2n+OJlXvon9CUBTMHEBzga3ZJwgxaw9qnYPtcGPAARLQyu6IrJkEpHOqbQ+k8smwXfVpHMW9GfwlJq7Lb4fPHYecCGPwwjHzZI1uSP5CgFA6z6WgmD773PV1bRDB/Zn+Z/NbKts81QvLaJ2DE8x4dkiBBKRzk26OZ3L8oiaviwlg0awARcqG4tV09w5jYotddHh+SAHJmXTTahiMZ3L8oiQ6xYSy9b6AsumVVVZXwzR+gNA/8g6D3JK8ISZCgFI2gtebTPWd5YNFOOsaFsfT+gTSR5VmtyVZuzCW5+c9w7Cuzq3E46XqLy6a1ZktyNn/+6gi7T+fRKz6SRbMGSEvSqirLYOVUIyBved24d9vLSFCKy5KSWcRvP9zHthM5tIgM4pXbezLh6ni5PtKqKkpg+d1wfCOM/jskzjS7IqeQoBQNduBsPlPf3Y7WmhfHdGPSgDYE+cvlP5ZWlge5J2DcHGPNKS8lQSka5PtTucyYt52wQD+W3D+IdjK7j7VVFINfsHFb4kPbjMEbLyb9JVGv/yRnMeVf22gaGsDKnw+WkLS60jxYOBbW/MrY9vKQBAlKUY/1h9OZsWAH8U2CWfnAYOKbhJhdkjBTSQ4sGgfn9kCHG82uxmWk6y1q9X7SaX7z4T66tohg0awBcumP1RVnGSGZdQwmLYVOI82uyGUkKMVPaK3567qjvLk+mWs7xPDWlH5yp43V2e2wZAJkJ8PkZdBhhNkVuZQEpfiJ/1lziHc2n+CuxNa8fFsPufRHGPNH3vCcMSN5u+vMrsblJCjFjyz8TyrvbD7BtMFt+f3Y7jJjvNXlnYbT26DnBMu1ImuSoBQXfHMond9/eoAbuzbjhTESkpaXmwoLx0BZgRGSwU3Mrsg0EpQCgLN5pTyxcg/dWkbwxuQ+so621WWnGCFZWQLTPrZ0SIKTLw9SSt2slDqilEpWSj1zidfbKKU2KKV2KaX2KqVGObMecWlVds3jK3ZTWWXnH5P7ERIg/39aWuYRmD8KbGUw/VNo2dfsikzntKBUSvkCc4BbgG7AZKVUt4t2+x2wUmvdF5gEvOWsekTt3v42hW0ncnhJ1tMWACnrQdthxufQvKfZ1bgFZzYdBgDJWuvjAEqp5cA44GCNfTQQUf04EjjrxHrEJXx/Kpe/rjvKmN4tuaOf565pIhygqtIY1R70oDHhbkhTsytyG87sercCTtfYTqt+rqYXgSlKqTRgDfCIE+sRF8kvqeTRZbtoHhHEy+N7yOCNlZ35Hv6RCGd3G9sSkj9i9gVyk4EFWut4YBSwWCn1k5qUUrOVUklKqSRZu9sx7HbN4yt3k15Qxpt39yUyWC4ot6zT2407brS2/KBNbZwZlGeA1jW246ufq+leYCWA1norEATEXPxGWuu5WutErXVibGysk8q1ljkbkll/OIPnRnejXxv5x2FZJ/8Di2+D0BiYuQaatDW7IrfkzHOUO4COSql2GAE5Cbj7on1OASOABUqprhhBKU1GJzlwNp+NRzLZkZrDt0czGd+nJVMHyT8Myzq3B967AyLjYdpqiGhhdkVuy2lBqbW2KaUeBr4EfIF5WusDSqmXgCSt9WrgV8A7SqnHMQZ2ZmittbNqsrKk1Bwm/nMrdg0d48KYPjiBp27uLOclrSy2C/SbDkOfgLA4s6txa8rTcikxMVEnJSWZXYZHKamwccv/bqbKrvnooWuIDQ80uyRhppQN0LwXhEabXYlbUUrt1FonXuo1swdzhAv8ae1hTmaX8Oc7e0tIWt2hT2HJnbDuebMr8SgSlF5uS3IWC7eeZNY17RjUXloQlrZ/Faycbtxpc/P/mF2NR5Gg9GKFZZU89cFe2seE8tTNnc0uR5hpzwpYdR+0HghTP4SgSLMr8ihyU6+Xyi+p5Ofv7eRcfikfPDhEVku0Mls5bHod2l4Dd6+AALlN9XJJUHqhU9klzFywnVM5xnlJuU7SwrQGv0BjcougSAiQNY+uhASllzl0roAp/9pGZZWdRbMGMvgqOS9pWd+9DWe/h/H/J9dINpKco/Qyr6w9DMCHD10jIWllW96AL5421t+2V5ldjceToPQix9IL2XQ0kxlDEugQF2Z2OcIsm16Hdc9B99vhzgXgJ6tnNpYEpReZt+UEgX4+3CO3JVrXpj/D+peNadJuf8eYNk00mgSll8guKufD789we794msr629bVZhD0v884L+krQxCOIj9JLzFvywnKbXbuvTbB7FKEq2kNp76DtoMh4VrjSziUtCi9wKnsEt7ZfILxfVrSIS7c7HKEK9ntsObXMP9mIyyFU0iL0sNprXn584P4KsUzt3Q1uxzhSnY7fPYYfL8Ihjxi3HUjnEJalB5u+Y7TfHUwnUdGdKB5ZJDZ5QhXsVfBJ78wQnLok3DTH0CmzHMaaVF6sPWH03n+k/0M7RjDA9ddZXY5wpVS1sOepTD8Wbj+KbOr8XoSlB6otKKKf25K4a0NKXRpHsE/7u6Hr4+0Jiyl401w3zcQf8npE4WDSdfbw+xIzeGGv2zk718f46buzVh87wBZGMwqbOWw6n44vcPYlpB0GWlRepBdp3KZMW87cRFBrHxgMAPayZKillFZCiumQvI6aDsEWvc3uyJLkaD0EHa75ulVe4kKCWD57EE0i5CBG8uoKIHlk+H4tzDmDbh6utkVWY50vT3EhiMZHE0v4lcjO0lIWklFsbF0w4lNMP4tCUmTSIvSQ7z9bQqtooIZ07ul2aUIV/INMFZIvP0d6DnB7GosS4LSA3y65yw7UnN5fnQ3/H2lE2AJpblgq4DwZjBhnlwjaTIJSje3/0w+T6/aS2LbJkwdLLMCWUJJDiwaZzyevRF8ZBkPs0lQurHkjCKmzdtOk5AA5tzTT1qTVlCUaYRkdjJMWioh6SYkKN3UmbxSpr27DR8F7903UAZwrKDwPCwcC3mn4J6V0H6Y2RWJahKUbqi0ooqp726jsNzG8tmDaBcjq+ZZwmdPQH4aTPlApkpzMxKUbuibw+kczyzmX9MS6d5S1l+2jNF/NYJS7rhxO3LSyw19tuccseGBDO8SZ3YpwtlyTsCap6DKBuHNJSTdlASlmykqt7HhSAajejSXiS68XXYKzB8F+1ZC3kmzqxF1kK63m/n6YDrlNjuj5cJy75Z5BBaOMeaVnP4ZRMs0ee5MgtKNlNuqWLHjNM0jgri6TROzyxHOkn7AGN1WPjDjc4jrYnZFoh7S9XYTSak5jPrfzWw9ns2917bDR7rd3quiGEKawsw1EpIeQlqUJtNa8+oXh5m76TgtI4NZOGsA13eKNbss4QxFGcZ9260HwEPfycXkHkRalCb729fH+Oe3x5nUvzVfPX6dhKS3OrUN3rzaWOMGJCQ9jLQoTfTJ7jO88c0xJibG8z+39UTJxAfeKXWLMVVaeHO4aoTZ1YgrIC1KE61MOs1VsaESkt7s+EZ47w6IbGWck4xsZXZF4gpIUJpk/5l8klJzub5THH4y2YV3KjgLSydB03bG6HZ4c7MrEldIut4mWLUzjWc+3EtMWCD3DGpjdjnCWSJawtg34aobIDTa7GpEI0hQmuBvXx+la4sIFs4cQJPQALPLEY52cLVx+U/CtdDrTrOrEQ4gfT4XK6mwkZZbyk1dm0lIeqN9H8D7M2DzX0Brs6sRDuLUoFRK3ayUOqKUSlZKPVPLPhOVUgeVUgeUUkudWY87SM4oAqBjszCTKxEOt3sZfHg/tBkEExfJ8g1exGldb6WULzAHuAlIA3YopVZrrQ/W2Kcj8BvgGq11rlLK66fLOZb+Q1CGm1yJcKjvF8HqR6HdUJi8HAJkDlFv4swW5QAgWWt9XGtdASwHxl20z/3AHK11LoDWOsOJ9Ziuwmbn491nCPTzoW3TELPLEY6iNaT+GzqMgLtXSkh6IWcO5rQCTtfYTgMGXrRPJwCl1BbAF3hRa/2FE2syTYXNzkNLvmfzsSz+ML6HXBLkLSqKjWAc9xboKvALNLsi4QRm/2v1AzoCw4DJwDtKqaiLd1JKzVZKJSmlkjIzM11boQOU26p4aMlOvj6Uzh/GdWfqIFlN0Sv8++/w9lAozgJfPwlJL+bMoDwDtK6xHV/9XE1pwGqtdaXW+gRwFCM4f0RrPVdrnai1ToyN9ax7obXWPL5iN18fyuDl8T2YOjjB7JKEI3z7Gnz9ArTsA0GyXIe3c2ZQ7gA6KqXaKaUCgEnA6ov2+RijNYlSKgajK37ciTW53Mqk06zZd55nbunCFGlJej6tYf3LsOGP0GsS3P4O+PqbXZVwMqcFpdbaBjwMfAkcAlZqrQ8opV5SSo2t3u1LIFspdRDYAPxaa53trJpc7WR2Mb//9CCD20cze2h7s8sRjrDtn7Dpdeg7Fca/JbMAWYRT78zRWq8B1lz03PM1Hmvgieovr6K15sn39+Dro/jzxN4yEa+36DkBKorg2ifAx+xT/MJV5DftJAfPFbAjNZdf3dSJVlHBZpcjGsNuh6R5YKuA0Bi47kkJSYuRe72d5Mv95/FRyCJhns5eBZ8+BrsWg38o9L7L7IqECSQonWTt/vP0T2hKTJhcMuKx7FXw8UOwdzlc/zT0mmh2RcIk0n9wsLySCv667ijHMoq4pYfMP+ixqiqN+7b3Lofhv4Phv5V7ty1MWpQONnvxTrafyGF451huvzre7HLElco9Cclfw00vwTWPmV2NMJkEpQMlZxSy/UQOT9/chQeHyYL2HsleZVzyE9MBHt4JYZ51g4NwDul6O4jWmnlbUvHzUdyZKC1Jj1RZaiwC9u+/GdsSkqKaBKUD2KrsvLD6AEu3neKu/q1lAMcTVRTD0omQsh5CYsyuRrgZ6Xo7wKPLd7Fm33keuK49T9/cxexyxOUqL4QlE+H0d3Db29B7ktkVCTcjQdlI5/JLWbPvPD+//iqeuUVC0uPYq4zu9untcMe/oMcdZlck3JAEZSN9c8iYa3hM7xYmVyKuiI8v9LkbBj0E3cbWv7+wJAnKRjiRVcyf1h6md+soujaPMLsccTmKsyHrCLQdAv2mmV2NcHOXPZijlPJRSt3jjGI8zS9X7MbPV/HWPf1k0gtPUpQJC8fAsslQVmB2NcID1BqUSqkIpdRvlFL/UEqNVIZHMOaLtPy9XLYqO3vT8pgyqK1MeuFJCs/Dglsh5zhMXAhB0hMQ9aur670YyAW2AvcBvwUUMF5rvdv5pbm37OIKtIZmEUFmlyIaKv+M0ZIsPA9TPoCEa82uSHiIuoKyvda6J4BS6l/AOaCN1rrMJZW5uYyCcgBiw+WaSY+RNA+KM2HqR9Dm4nXuhKhdXUFZ+cMDrXWVUipNQvK/MgqNH0WcBKX709qY0GL4b40R7mi5vVRcnroGc3orpQqUUoVKqUKgV41ty58BP5FVDECcdL3dW9YxmH8L5J02LgWSkBRXoNYWpdZaFgOpxX9SsnjtyyP0jo+kuQSl+8o4bJyT1Hbj7hshrlCtQamUCgJ+DnQA9gLzqhcMs7zXvjhC84ggFswcgK9cFuSezu+HReOMVuSMNRDb2eyKhAerq+u9EEgE9gGjgL+4pCI3l1lYzp60PO7oF0+T0ACzyxGXkn4AFo4G3wAJSeEQdQ3mdKsx6v0usN01Jbm3DYcz0Bpu7BZndimiNhEtoe01MPJlaNrO7GqEF2joqLdNyTT4AHx9KJ2WkUF0ayEXKrudc3shphMEN4FJS8yuRniRurrefapHuQtk1NtQVlnF5mNZ3NA1DvmPw82c2AzzboavnjW7EuGF6mpR7tFa93VZJR5ga0o2pZVVjOjazOxSRE0pG4z7tpu0het+bXY1wgvV1aLULqvCQ3y06wwRQX4Mbh9tdiniB0e/gqV3QdP2MP0zCJeVL4Xj1dWijFNKPVHbi1rrvzqhHreVXVTOFwfOM7l/a4L85RJTt1BZCqsfgbguMPVjCGlqdkXCS9UVlL5AGMZEGJb3l3VHqbJrpg5OMLsU8QP/YOO+7YiWEBxldjXCi9UVlOe01i+5rBI3duBsPsu2n2LGkAQ6xIWZXY7Y9wFkp8Cwp6FZN7OrERZQ1zlKaUliLEP7+9UHaRISwC9v7GR2OWL3UvjwfjixCWwVZlcjLKKuoBzhsirc2JcHzrM9NYcnR3YmMtjf7HKsbedC+PghaHcd3PM++MmdUcI1ag1KrXWOKwtxR1pr5mxIISE6hLv6tza7HGvb/g58+ih0uBEmr4CAELMrEhZy2WvmWMl/UrLZdyaf2dddJZNfmC0wArqMNu648ZcZm4RrySqMdZi76Tix4YHc3q+V2aVYV85x4xrJ3ndBr4nGBLxCuJi0KOuwJy2Pkd2ayXWTZtAaNv4J5gyEc3uM5yQkhUkkKGtRbqsir6RSJuY1g9aw/mXY+D/QYwI062F2RcLipOtdi8xCWTzMFFrDuufgP29Cv+kw+u/gI/+fC3PJ38BaZFQHZVyEBKVLHVpthGT/+yUkhduQFmUtfmhRxoVL19uluoyBOxdCt3FyTlK4DfnvuhansksAaBkVbHIlFmCvgnXPGyPcPj7QfbyEpHArEpS1OHS+gGYRgTSVdXGcq8oGHz8IW/4Xjqw1uxohLsmpQamUulkpdUQplayUeqaO/e5QSmmlVKIz67kch88V0rm5LPfgVFWV8OF9sHcF3PAcDP6F2RUJcUlOC0qllC8wB7gF6AZMVkr9ZKoXpVQ48BiwzVm1XK7KKjvJGUV0bR5udiney1YB78+AAx8Zi4Bd96TZFQlRK2e2KAcAyVrr41rrCmA5MO4S+/0B+BNQ5sRaLktqVjEVVXa6tJCgdJqqcihKh1tegyGPmF2NEHVy5qh3K+B0je00YGDNHZRS/YDWWuvPlVJus9jJofOFAHRuJl1vh6ssBW2HwHCY+QX4yoUXwv2ZNpijlPIB/gr8qgH7zlZKJSmlkjIzM51e2+FzBfj5KK6KC3X6Z1lKRTEsuROW32NcWC4hKTyEM4PyDFBzbrL46ud+EA70ADYqpVKBQcDqSw3oaK3naq0TtdaJsbGxTizZcPh8IVfFhhHoJ/d4O0x5Ibw3AU5ugd6T5fIf4VGcGZQ7gI5KqXZKqQBgErD6hxe11vla6xitdYLWOgH4DhirtU5yYk31sts1e9Py6NZSut0OU5YPi2+D09vgjneNmYCE8CBOC0qttQ14GPgSOASs1FofUEq9pJQa66zPbaxdp/PIKqpgWGfnt1wt48PZcHY3TFwIPW43uxohLptTTxJprdcAay567vla9h3mzFoa6quD5/HzUQzrHGd2Kd5jxAuQeC90Gml2JUJcEbkz5yJfH0xnUPtoWR+nsYoyYOscY9CmWTcJSeHRJChrOJVdQkpmMTd0kdZkoxScgwW3GnNK5p4wuxohGk2uz6hhw5EMAAnKxshPg4VjjBbllFXGMg5CeDgJyhrWH84gITqEhBi5fvKK5KYaIVmaB1M/htb9TS5ICMeQrne10ooqth7PZri0Jq9c+gGoKIFpn0hICq8iLcpqW49nUWGzS7f7SlSWGUvIdrkV2l1n3J4ohBeRFmW19YczCPb3ZUC7pmaX4lkyDsGb/eDIF8a2hKTwQhKUgNaaDYczuaZDjNy2eDnO7zNGt+1VMmgjvJoEJXC+oIwzeaUM7Rhjdime4+wuWDAa/IJg5hqI7WR2RUI4jZyjBE5kFgPQIS7M5Eo8RN4pWDgOgiJhxqfQJMHsioRwKglK4HiWEZTt5LKgholsDdc8Cr3ugqjW9e8vhIeToMSY0TzQz4fmEbI0bZ1S/w2hcUY3W5ZuEBYi5yiB1OxiEqJD8fGRORJrlbLemE9y7VNmVyKEy0lQAimZxdLtrsvRr2DpJIi+Cu74l9nVCOFylg/KwrJKTmQVy0S9tTn0GSy/G+K6wPRPIVSuDBDWY/mgPHTOWEisRysJyp/QGrbPhRa9YdpqCJGL8YU1WX4w58DZfAC6t4w0uRI3Y7eDjw9MWmIEZpD8RyKsy/Ityj2n84gNDyQuPNDsUtzHriXGLEDlRcYtiRKSwuIsH5RJJ3Ppn9AEJasCGpLmwycPga8/KMv/9RACsHhQnssvJS23lKvbyrk3ALbNhc9+CR1/BpOXQ0CI2RUJ4RYsfY5yX5pxfrJvmyhzC3EHSfNg7a+hy2iYMB/8AsyuSAi3YekWZWZROQAtI4NNrsQNJFwH/e+DOxdISApxEUsHZXZRBQBNQy0aDFrDsXXGnzEd4Na/GOcmhRA/YvGgLCciyI8APwv+GLSGb16CJRPgwIdmVyOEW7P0Ocqs4gpiwix4WZDW8NXvYOs/4OoZ0O02sysSwq1ZOiizi8qJDrNYt9tuhy+eNu64GTAbbnkN5NIoIepkwT7nf2UWlhMdarEWZfp+Y4R78MMSkkI0kKVblOkF5QztGGt2Ga7Vohc8sBniukpICtFAlm1RFpXbKCq30TzSApP1Vtngo5/D/lXGdrNuEpJCXAbLBuX5/DIA75/VvKoSVt0Le5YZa90IIS6bZbve6QVGUDbz5qC0lcP7M+HI5zDyjzDkYbMrEsIjWTYoL7QovbXrXVUJK6bAsa/gltdh4GyzKxLCY1k3KAu8vOvt4wfNukPnUZA40+xqhPBolg3KrKJywgP9CA7wNbsUxyovgsJzENMRbnzR7GqE8AqWHczJL60kItjL7msuK4D37oAFo6Gi2OxqhPAalm1RFpTaiPSmoCzNM0Ly3G5jpcQAWVVSCEexcFBWEhHsJYdfkgOLb4P0AzBxEXS51eyKhPAqlu16nysoJTbcSwZyNr4KGYdg0lIJSSGcwJJBWVpRRVpuKR3jwswuxTFufBFmfAadRppdiRBeyZJBmZJZhNbQwZODsuAsrLofyvKNtW1aDzC7IiG8llODUil1s1LqiFIqWSn1zCVef0IpdVAptVcp9Y1Sqq0z6/lBSmYR4MFBmXca5o+CI2sgO8XsaoTwek4LSqWULzAHuAXoBkxWSnW7aLddQKLWuhfwAfCas+qp6Vh6Eb4+ioRoDxwZzk2FBaOMAZypH0OrfmZXJITXc2aLcgCQrLU+rrWuAJYD42ruoLXeoLUuqd78Doh3Yj0XJGcU0TY6xPOWgMhOMVqSZQUw/RNo3d/sioSwBGcmRSvgdI3ttOrnanMvsNaJ9VyQnFlEh1gP7Hb7+EJojDFw07Kv2dUIYRlucSGhUmoKkAhcX8vrs4HZAG3atGnUZ1XY7KRmFTOyW7NGvY9L5adBeEtokgCzv5W5JIVwMWe2KM8ArWtsx1c/9yNKqRuBZ4GxWuvyS72R1nqu1jpRa50YG9u4GcmPZxVhs2s6Nw9v1Pu4zLm98PZQ2PBHY1tCUgiXc2ZQ7gA6KqXaKaUCgEnA6po7KKX6Av/ECMkMJ9ZywZHzhQB0bRHhio9rnDPfw8Ix4B8Cfe42uxohLMtpQam1tgEPA18Ch4CVWusDSqmXlFJjq3d7HQgD3ldK7VZKra7l7Rzm0LlC/H0V7WLcfMT79HZYNA6CImDmGoi+yuyKhLAsp56j1FqvAdZc9NzzNR7f6MzPv5TD5wvoEBeOv68bj3iXF8GyScbAzfRPIdIlFwMIIWrhFoM5rnT4XCFDroo2u4y6BYbBHe9CbBeIaGF2NUJYnqWCMq+kgvMFZe47kJP8NRRnQe9JcNVws6sRQlRz4/6n4x2uHsjp4o4DOUe+gGWT4bv/M5aXFUK4DWsF5bkCALq6W4vy0KfGQmDNusPUj8DXUg19IdyepYLy0LlCmoYGEBseaHYp/7V/FaycbtxpM+0TCGlqdkVCiItYqumy/2w+3VtGoNzpou3sFGg9EO5ZCYFu1tIVQgAWalGW26o4ml5I95aRZpdiKM0z/rzu10ZLUkJSCLdlmaA8ll5EZZWmRys3GMjZ8S680Reyjhm3JPoFmF2REKIOlgnKM3mlALRtavIdOd+9DZ8/YcxIHtm6/v2FEKazTFDmFlcA0DTMxNbbljfgi6ehy2iYuBj8vWRxMyG8nHWCsqQSgKYhJgXl/g9h3XPQ/Xa4c4F0t4XwIJYZ9c4tqSDI34fgAF9zCug8Cka+DAMflOskhfAwlmlR5hRX0MTVrUmtjXOSJTlGN3vIIxKSQnggywRlrquDUmv48lnjnOSuxa77XCGEw1mmeZNTUkGTUH/XfJjdDmufgh3vGF3tIY+65nOFEE5hmRZlQWklkcEuCEq7HT77pRGSQx6Bm1+R5RuE8HCWaVHml9pcE5SlOXB8Iwx9Em74nYSkEF7AMkFZUFZJhDODsspmhGJoDDywCYKjnPdZQgiXskTXu9xWRYXNTkSQk4KyqhI+mAmrHzEGcSQkhfAqlgjKkvIqAEKdcQ2lrRxWToNDq435JKWrLYTXsUTXu6jcmDE8JNDBh1tZCiumQvI6GPVnGHC/Y99fCOEWLBGUJRVGizLM0UH5wb3GOjdj/heunuHY9xZCuA1LBGVxRXWL0tFd74GzoesY6DPZse8rhHArlgjK/OoJMRwy6l1WACe+NQKy/bDGv58Qwu1ZYjDndG4JAPFRwY17o9JcWDwePpgFeacbX5gQwiNYokV5Lr8Mf19FTFgjFhUryYFF4yDjEExcCFEy6a4QVmGJoPxhQgwfnyu8dKco0wjJ7GSYvAw63uTYAoUQbs0SQZlTXEHT0EbMHHR0LeQch7tXwFXDHVeYEMIjWCIos4rKrywotTYuIO83DdoPl+62EBZlkcGcUlpd7kBO3imYez2c+d7YlpAUwrK8vkVZVllFZmE5rZuGNPybck7AwjHGpUDa7rzihBAeweuDMq360qDWTRvYosxKNkLSVgrTV0PLPs4rTgjhEbw+KM/klQHQKqoBLcrck7BgFNirYPpn0LyHk6sTQngCrw/KjAIjKJtFNOAayvAW0OlnMOgXENfFyZUJITyF9wdlYTkAseF1BOX5fRDWHMJiYeybLqpMCOEpvH7U+1h6IS0igwgJqOX/hLSdsOBWWP2wawsTQngMrw/K07mltIsJvfSLp7YZd9wERcGo111alxDCc3h9UBaV2S69BETqFlh8G4TFwcy1ENXG9cUJITyC15+jLCq3ERZ00WHa7fDlbyAy3rgEKLy5OcUJITyC1wdlYVnlT2c29/GBycvBx98YwBFCiDp4fde7uKKK0MDqmc2PrIWPHjSuk4xoKSEphGgQpwalUupmpdQRpVSyUuqZS7weqJRaUf36NqVUgqNrsGuNr1Jw8BNYMQUyD0NFsaM/RgjhxZwWlEopX2AOcAvQDZislOp20W73Arla6w7A34A/OaOWzllfwfszodXVMO1jCIpwxscIIbyUM1uUA4BkrfVxrXUFsBwYd9E+44CF1Y8/AEYo5diFscf5/Jtbjj4PbQbBlFUQFOnItxdCWIAzg7IVUHNhmbTq5y65j9baBuQD0Re/kVJqtlIqSSmVlJmZeVlFnNUxHG9yLdzzPgSGX9b3CiEEeMhgjtZ6rtY6UWudGBt7eQMwz//iPsJnvg8BtVx0LoQQ9XDm5UFngJqz3cZXP3epfdKUUn5AJJDtyCJ6tJKuthCicZzZotwBdFRKtVNKBQCTgNUX7bMamF79eAKwXmutnViTEEJcNqe1KLXWNqXUw8CXgC8wT2t9QCn1EpCktV4NvAssVkolAzkYYSqEEG7FqXfmaK3XAGsueu75Go/LgDudWYMQQjSWRwzmCCGEmSQohRCiHhKUQghRDwlKIYSohwSlEELUQ4JSCCHqIUEphBD1UJ52I4xSKhM4eZnfFgNkOaEcV/OW4wA5FnflLcdyJcfRVmt9yckkPC4or4RSKklrnWh2HY3lLccBcizuyluOxdHHIV1vIYSohwSlEELUwypBOdfsAhzEW44D5Fjclbcci0OPwxLnKIUQojGs0qIUQogr5lVB6Q7L4zpCA47jCaXUQaXUXqXUN0qptmbU2RD1HUuN/e5QSmmllNuOuDbkWJRSE6t/NweUUktdXWNDNODvVxul1Aal1K7qv2OjzKizIZRS85RSGUqp/bW8rpRSb1Qf616lVL8r+iCttVd8YUwOnAK0BwKAPUC3i/Z5CHi7+vEkYIXZdV/hcQwHQqofP+iOx9HQY6neLxzYBHwHJJpddyN+Lx2BXUCT6u04s+u+wuOYCzxY/bgbkGp23XUcz3VAP2B/La+PAtYCChgEbLuSz/GmFqVbLI/rAPUeh9Z6g9a6pHrzO4z1iNxRQ34nAH/AWNO9zJXFXaaGHMv9wBytdS6A1jrDxTU2REOOQwMR1Y8jgbMurO+yaK03YayOUJtxwCJt+A6IUkq1uNzP8aagdNjyuCZryHHUdC/G/5juqN5jqe4KtdZaf+7Kwq5AQ34vnYBOSqktSqnvlFI3u6y6hmvIcbwITFFKpWGsUPCIa0pzisv993RJTl0KQjiXUmoKkAhcb3YtV0Ip5QP8FZhhcimO4ofR/R6G0crfpJTqqbXOM7OoKzAZWKC1/otSajDGulY9tNZ2swszize1KC9neVyctTyuAzTkOFBK3Qg8C4zVWpe7qLbLVd+xhAM9gI1KqVSMc0ir3XRApyG/lzRgtda6Umt9AjiKEZzupCHHcS+wEkBrvRUIwrh32hM16N9TfbwpKL1ledx6j0Mp1Rf4J0ZIuuN5sB/UeSxa63ytdYzWOkFrnYBxvnWs1jrJnHLr1JC/Xx9jtCZRSsVgdMWPu7DGhmjIcZwCRgAopbpiBGWmS6t0nNXAtOrR70FAvtb63GW/i9mjVg4eARuF8b94CvBs9XMvYfzjA+MX/j6QDGwH2ptd8xUex9dAOrC7+mu12TVf6bFctO9G3HTUu4G/F4VxKuEgsA+YZHbNV3gc3YAtGCPiu4GRZtdcx7EsA84BlRgt+nuBnwM/r/E7mVN9rPuu9O+X3JkjhBD18KautxBCOIUEpRBC1EOCUggh6iFBKYQQ9ZCgFEKIekhQCo+llKpSSu2u8ZWglBqmlMqv3j6klHqhet+azx9WSv3Z7PqF55BbGIUnK9Va96n5RPXUeZu11qOVUqHAbqXUp9Uv//B8MLBLKfWR1nqLa0sWnkhalMJraa2LgZ1Ah4ueL8W4kPqyJ0cQ1iRBKTxZcI1u90cXv6iUisa4f/zARc83wbgHe5NryhSeTrrewpP9pOtdbahSahdgB17VWh9QSg2rfn4PRkj+XWt93mWVCo8mQSm80Wat9ejanldKtQO+U0qt1FrvdnFtwgNJ11tYjjamQHsVeNrsWoRnkKAUVvU2cJ27LjAn3IvMHiSEEPWQFqUQQtRDglIIIeohQSmEEPWQoBRCiHpIUAohRD0kKIUQoh4SlEIIUQ8JSiGEqMf/A94ku1mvQAyCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 5))\n",
    "\n",
    "plt.plot(df_scores.fpr, df_scores.tpr, label='Model')\n",
    "plt.plot([0, 1], [0, 1], label='Random', linestyle='--')\n",
    "\n",
    "plt.xlabel('FPR')\n",
    "plt.ylabel('TPR')\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "4a689523",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "37c94946",
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = roc_curve(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "5863648f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffaca5d3eaf0>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAE9CAYAAABtDit8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoGklEQVR4nO3deXhV1b3/8feXMASQQRkUAQUVRZBJowhVC6JWkUlBBAUCOLR6RVuv1V7nWjtcve3t1WItWiZlVESj4lSVihTRIIgiRQERgighTAphCFm/P3bC7xCSnIScnX3O3p/X8+Qx55ydc76b4Ie19tprLXPOISIiZasRdAEiIslOQSkiEoeCUkQkDgWliEgcCkoRkTgUlCIicdQMuoDKatq0qWvTpk3QZYhIyCxZsmSLc65Zaa+lXFC2adOG7OzsoMsQkZAxs6/Lek1dbxGROBSUIiJxKChFROJIuWuUpdm/fz85OTns2bMn6FJSQnp6Oq1ataJWrVpBlyKSEkIRlDk5OTRo0IA2bdpgZkGXk9Scc+Tl5ZGTk0Pbtm2DLkckJYSi671nzx6aNGmikKwAM6NJkyZqfYtUQiiCElBIVoL+rEQqx7egNLOJZrbZzD4r43Uzs8fMbLWZLTezM/2qpTqYGSNGjDj4uKCggGbNmtGvX79KvU+bNm3YsmVLlY8RkcTxs0U5Gbi0nNcvA9oVfd0I/NXHWnxXv359PvvsM/Lz8wF46623aNmyZcBViUgi+DaY45x7z8zalHPIQGCq85ZY/8DMGptZC+fcJr9q8lvfvn159dVXGTJkCDNmzGD48OEsWLAAgK1btzJ27FjWrl1LvXr1mDBhAp07dyYvL4/hw4ezceNGevToQeyK888++yyPPfYY+/bto3v37jzxxBOkpaUFdXoiVTZ98XpeWrbR18+o5fbS7vimPDDgjIS9Z5DXKFsCG2Ie5xQ9dxgzu9HMss0sOzc3t1qKOxLDhg1j5syZ7Nmzh+XLl9O9e/eDrz3wwAN069aN5cuX87vf/Y5Ro0YB8Otf/5rzzjuPFStWcMUVV7B+/XoAVq5cyaxZs1i4cCHLli0jLS2NadOmBXJeIpUxffF6rv7bolK/7p77KYu/2urbZ9ct3MX9eXfR55snE/q+KXF7kHNuAjABICMjo9xNfn798go+/2ZnQj+/w/ENeaB/x7jHde7cmXXr1jFjxgz69u17yGvvv/8+c+bMAeDCCy8kLy+PnTt38t577/HCCy8AcPnll3P00UcD8Pbbb7NkyRLOPvtsAPLz82nevHkiT0tSVHW0yqqiOAi7tz3msNe6tz2GgV1bck33ExL/wbu3wrNXwoE1nPrjexP61kEG5UagdczjVkXPpbQBAwZwxx13MH/+fPLy8o74fZxzZGZm8vvf/z6B1Umqig3H8oIoGfgahmXZtwumDoDcVXD1s3BaecMjlRdkUGYBt5jZTKA7sCMR1ycr0vLz09ixY2ncuDGdOnVi/vz5B58///zzmTZtGvfddx/z58+nadOmNGzYkAsuuIDp06dz77338tprr7Ft2zYA+vTpw8CBA/nFL35B8+bN2bp1K99//z0nnnhiQGcm1alkqzE2HAMJomRXqx60uwQuehBOuSjhb+9bUJrZDKAX0NTMcoAHgFoAzrkngXlAX2A1sBsY41ct1alVq1bceuuthz3/4IMPMnbsWDp37ky9evWYMmUK4F27HD58OB07dqRnz56ccIL3l79Dhw48/PDDXHLJJRQWFlKrVi3Gjx+voAyBinSdS7YaFY5l2LkJ9u6EZqdBn/t9+xhLtX29MzIyXMn1KFeuXMnpp58eUEWpSX9m1etIus4Kxjh25MCU/oDBf3wIaVVr95nZEudcRmmvpcRgjkiqe2nZRj7ftJMOLRqqdZgI276GKf0gfzuMmFPlkIxHQSnio+KWZHFIzvppj6BLSn15a2DKANj3A4x6CVr6P6lPQSnig+KAjO1mD+yqmVoJMf8PUJAPmS9Di87V8pEKSpFKqOg9jCUDUt3sBOr/Z9j5DTRtV20fqaAUKUVZgVjRgRgFZIJ9+ym881sY/BTUaVCtIQkKSpHDTF+8nrvnfgocHogKwAB8sxSmDoLa9WHXFi8oq5mCMkHS0tLo1KkTBQUFtG3blmeeeYbGjRtX+X0nT55MdnY2f/nLX6pepFTqHsbfXdFJgRi0DR/Bs4OhbiPvmuTRbQIpQ0GZIHXr1mXZsmUAZGZmMn78eO65555gi4qwqnSd1WpMEusXe3O36zfzQrJx6/g/4xMFpQ969OjB8uXLAfjwww+57bbb2LNnD3Xr1mXSpEmcdtppTJ48maysLHbv3s2aNWu44ooreOSRRwCYNGkSv//972ncuDFdunShTp06AKxbt46xY8eyZcsWmjVrxqRJkzjhhBMYPXo0devWZenSpWzevJmJEycydepUFi1aRPfu3Zk8eXJQfxS+q2wgKgRTyFHNoOVZcMWT0PD4QEtRUCbYgQMHePvtt7nuuusAaN++PQsWLKBmzZr84x//4O677z64itCyZctYunQpderU4bTTTmPcuHHUrFmTBx54gCVLltCoUSN69+5Nt27dABg3bhyZmZlkZmYyceJEbr31Vl588UUAtm3bxqJFi8jKymLAgAEsXLiQp59+mrPPPptly5bRtWvXIP44fFPa7TexFIgpbPNKaNYejjkJMrOCrgYIa1BOuvzw5zoOgnNugH27YdpVh7/e9Rrodi3syoPZow59bcyrcT8yPz+frl27snHjRk4//XQuvvhiAHbs2EFmZiZffvklZsb+/fsP/kyfPn1o1KgR4M3t/vrrr9myZQu9evWiWbNmAFx99dV88cUXACxatOjgkmwjR47kzjvvPPhe/fv3x8zo1KkTxx57LJ06dfJOu2NH1q1bF7qgLL6JW4EYMl+8CbNGQO+74byfB13NQeEMygAUX6PcvXs3P/nJTxg/fjy33nor9913H71792bu3LmsW7eOXr16HfyZ4i41eINBBQUFR/z5xe9Vo0aNQ963Ro0aVXrfZFGyi62ZLiG08hV4bjQc2xHOHBX38OoUzqAsrwVYu175r9dvUqEWZFnq1avHY489xqBBg7j55pvZsWPHwb1zKnKtsHv37tx2223k5eXRsGFDnnvuObp06QJAz549mTlzJiNHjmTatGmcf/75R1xnKintdp0OLRpqpkuYrJgLc66HFl29udt1Gwdd0SHCGZQB69atG507d2bGjBnceeedZGZm8vDDD3P55aVcEiihRYsWPPjgg/To0YPGjRsf0mV+/PHHGTNmDI8++ujBwZywiw1J3a4TUj9shrk3Qauz4ZrZkN4w6IoOo2XWIioV/swUkhGybiG06AJ1jgqsBC2zJimjtHUbFZIhlT3JW5m8y9XQ5kdBV1OuIHdhFDlEcQsy9pYfhWRILZ4Ar/wcPn8RUqBXqxalJAV1syPkX4/Dm/dC+34wZBKYBV1RXKEJSucclgJ/4MkgyOvS8WbSKCRDbsEf4e2HoMMgGPw0pNUKuqIKCUVQpqenk5eXR5MmTRSWcTjnyMvLIz09vVo/VzNpBID9+dBpKAz6q+/bNyRS6lRajlatWpGTk0Nubm7QpaSE9PR0WrVq5fvnlLWhlgIxYpyD7zd587V73+M9rpFawyOhCMpatWrRtm3boMuQGCVvEldARpRz3vXIZdPhZwugUauUuCZZUiiCUpKLBmYE8ELytTvhwwlwzk+hYerOpFJQSkIpJAWAwkJ49RewZDL0uAUueTglW5LFFJSSMApJOejDCV5Innc79Lk/pUMSFJSSIApJOcRZo72FLTpfnfIhCZqZIwlSPLqtkIywA/vh7d9A/naolQ5dhoUiJEEtSqmC2Nt/ihfRVUhGVMFeeH4s/PsVaHYadB4adEUJpaCUSivt5nGtDxlh+/fA7JHw5Ztw2aOhC0lQUEollbw/UvdGRty+3TDzGlg7H/r9GTLGBF2RLxSUUmEasJHD7NkO276CgeO9PadCSkEpFaYBGzlo3y6oWdeblnjzYm/wJsQUlBJX8TVJDdgI4I1qPzsYWnSGfv8b+pAEBaXEUdo1SYmw3VvhmSvguxVw/u1BV1NtFJRSqpIj2+puC7u2wNSBsOVLGDYdTr0k6IqqjYJSDqORbTlMYSFMGwJ5q2H4DDilT9AVVSsFpRxGgzZymBo14ML7vBXJ214QdDXVTkEppdKgjQCwfQNsWAydhkSuFRlLQSkHxY5ud2iRfJvQSzXbtg6m9Ic9O72QrHt00BUFRkEpgEa3pYS8NV5I7t8No16MdEiCz0FpZpcC/wekAU875/5Q4vUTgClA46JjfuWcm+dnTVI6XZeUg3JXwZQBULgfMl+G4zoFXVHgfFtmzczSgPHAZUAHYLiZdShx2L3AbOdcN2AY8IRf9UjZpi9ez+Kvtuq6pHjWvAOuEEa/qpAs4meL8hxgtXNuLYCZzQQGAp/HHOOA4othjYBvfKxHylDcmlR3O+IO7PdGtc+9yVtwt94x8X8mIvxcuLclsCHmcU7Rc7EeBEaYWQ4wDxjnYz1SCrUmBYCNH8NfMuCbZd5jheQhgl7hfDgw2TnXCugLPGNmh9VkZjeaWbaZZWvv7sSJHcBRazLCNnzozbhxLvKDNmXxs+u9EWgd87hV0XOxrgMuBXDOLTKzdKApsDn2IOfcBGACQEZGhvOr4KjQ9EQ56Ot/wbSr4Kjm3sBNo1ZBV5SU/AzKj4B2ZtYWLyCHAdeUOGY90AeYbGanA+mAmow+KW1lck1PjLBNn3irADVqBaOyoGGLoCtKWr4FpXOuwMxuAd7Au/VnonNuhZk9BGQ757KA/wSeMrNf4A3sjHbOqcXoA83flsM0aw9nZnqrAB3VPOhqkpqlWi5lZGS47OzsoMtIKVqZXA6x5l04rjPUbxJ0JUnFzJY45zJKey3owRzxmUJSDrHyZe+a5Fv3B11JSlFQhphCUg7x2RyYnQnHd4NLfxd0NSlFQRlimpYoB30yC+ZcD627w8gXIL1R0BWlFC2KEULa40YOUbAX3nsUTvwRXDMLatcPuqKUo6AModil0nQjecQ5BzXrePdIpjeC2vWCriglKShDJnZK4qyf9gi6HAnSB0/CNx/DoL/qHskq0jXKkNECFwLAwsfg9bu8/bcLDwRdTcpTizJEtMCFAN71yHceho5XwpUTvBWBpErUogwJLXAhALz3P15Idr4arnxKIZkgCsqQ0K1AAsAJ58LZ13vXJdPUYUwU/UmGiLrcEeUcrP8ATuwBbc7zviSh1KIUSWWFhTDvlzDpUi8sxRcKyhAoHsSRiCkshFdug4+egp7jvFk34gt1vVNYyfUlNYgTIYUH4KVb4JPpcP4dcOG9YBZ0VaGloExhsdMUtb5kxKx5xwvJ3vfAj+8MuprQU1CmoNi53B1aNNQMnChqdzFc/za0KnX5REkwXaNMMcX3Sy7+aqvmckdNwV6YcwNs+Mh7rJCsNmpRphCtLxlh+/Nh1khY/Rac2BNanx10RZGioEwhuqk8ovbthpnDYe0/of9jcFZm0BVFjoIyRWged0Tt2wXThsL6f8GgJ6BryY1MpTooKFOEVgWKqLTa3g6JVz4FnYYEXU1kKShTiFqTEZK/DQr2QYNjYchE3SMZMAVlkit5K5BEwO6tMHWg9/2N86FGWqDliIIy6Wlbh4j5IdcLybzVMGy6QjJJKCiTmLZ1iJjvv4UpA2D7erh2NpzUK+iKpIiCMklpId4IeuV22JEDI57XUmlJRkGZhHRjeUT1+5MXlJpxk3Q0hTEJ6cbyCNn6Fcy7Ew4UQIPjFJJJSkGZZHRjeYTkrYFJfeHT2bD966CrkXKo651kdGN5ROSugin9vXUlM1+BJicHXZGUQ0GZRNSajIjvVnij21YDRr8KzdsHXZHEoa53ktAod4Ts2wX1joEx8xSSKUItyoCV3M5BAzgh9sNmb95263Pg5g90M3kKUYsyYLHbOSgkQ2z9Ynj8LPh4qvdYIZlS1KIMkGbeRMS6hTDtKu/2n5P7BF2NHAG1KAOkEe4IWDsfnh0MjVp61yQb6XeditSiDIhGuCNg5zcwfRgc0xZGveRdn5SUpKAMgEa4I6Lh8TDgcTj5QqjfJOhqpAoUlAHQFMWQ+zzLu/2nzXnQ+aqgq5EE0DXKgKjLHVKfPg/PjYYFfwTngq5GEsTXoDSzS81slZmtNrNflXHMUDP73MxWmNl0P+tJBsXXJiWEls2AF26AE86FoVO1fUOI+Nb1NrM0YDxwMZADfGRmWc65z2OOaQf8F/Aj59w2Mwv91W6NdIfUx1Mh61Zoez4Mnwm16wddkSSQny3Kc4DVzrm1zrl9wExgYIljbgDGO+e2ATjnNvtYT+A00h1SzsG69+GUPnDNbIVkCPk5mNMS2BDzOAfoXuKYUwHMbCGQBjzonHvdx5oCUXKaolqTIbJvlxeMA58AdwBq1gm6IvFB0IM5NYF2QC9gOPCUmTUueZCZ3Whm2WaWnZubW70VJoCmKYbU+3+GJ8+HXVsgraZCMsT8bFFuBFrHPG5V9FysHGCxc24/8JWZfYEXnB/FHuScmwBMAMjIyEjJocQOLRpqmmKY/PMRePe3cMZgSG8UdDXiMz9blB8B7cysrZnVBoYBWSWOeRGvNYmZNcXriq/1saZqNX3xeq7+2yI+37Qz6FIkUZyDdx72QrLzMLjyKUirFXRV4jPfgtI5VwDcArwBrARmO+dWmNlDZjag6LA3gDwz+xx4F/ilcy7Pr5qqm/bkDqHFf4P3HoVuI2HQE1oFKCJ8nZnjnJsHzCvx3P0x3zvg9qKvUFKXO2Q6DYF9P8B5t0ONoC/xS3XRb9onurE8RAoLIXsiFOyD+k3hgjsUkhGjud4+0Y3lIVF4AF6+DZY+A7XqQ5erg65IAqCg9JFuLE9xhQfgxZth+Uz48V3QeWjQFUlA1H/wgbrdIXBgvzdve/lM6H0v9L5bc7cjTC3KBNNakyGx7WtY/Q+4+CH40W1BVyMBU1AmUGxIagZOiio84N3y0/QUuGUJHNUs6IokCajrnSAKyRDYn+9tAvb+/3qPFZJSREGZAArJENi3C6YPhTXvQL2mQVcjSUZd7wTQ1g4pbu/3MG0obPgArngSugwLuiJJMgrKKtIakymu8IDX3d7wIQx+2lvkQqQEBWUV6cbyFFcjDbpeA+feDB0GxD9eIklBWQVqTaawXXmwZRWc2BPOHBV0NZLkKj2YY2Y1zOxaP4pJNWpNpqgfcmFKf5gxHPZoCTyJr8ygNLOGZvZfZvYXM7vEPOPw1ovUXK4iak2mmO+/hcmXw9a1MHQKpDcMuiJJAeV1vZ8BtgGLgOuBuwEDBjnnlvlfmkiC7djotSS//xZGPA9tzgu6IkkR5QXlSc65TgBm9jSwCTjBObenWioTSbTsibArF0bOhRNK7nMnUrbyrlHuL/7GOXcAyFFI/n9a+CKFuKJtlnrfDTfOV0hKpZUXlF3MbKeZfW9m3wOdYx5H/gq4BnJSxJYvYdJlsH2DdytQk5ODrkhSUJldb+ecNgMpRfEe3cXbz2ogJ4lt/rd3TdIVerNvRI5QmUFpZunAz4BTgOXAxKINwyJNG4aliG8/g6kDvVbk6HnQ7LSgK5IUVt5gzhS865QLgL5AR0AL86ENw5LedytgSj+oWRcyX/aWTBOpgvKCskPMqPffgQ+rpySRKmp4PJz4I7jkYTimbdDVSAiUF5Sxo94FpmXwJdltWg5NT4W6R8OwaUFXIyFS3qh316JR7p0a9Zak99UCmHgpvHlP0JVICJUXlJ845xoWfTVwztWM+T6S875072SSWvOut1Ra49ZwwS+DrkZCqLygdNVWRYrQvZNJ6Is3YfrVcMxJkPkKNDgu6IokhMq7RtnczG4v60Xn3J98qCdpaUm1JLQ/H7LGQfP2MPJFqHdM0BVJSJUXlGnAUXgLYUSatqBNUrXqevO2Gx4PdRsHXY2EWHlBuck591C1VZLEtCdOkvn0echbA73ugmM7BF2NREB51ygj35KMpS53klg2HV64Ab56Dwr2BV2NRER5Lco+1VZFkoqd192hRSQH+pPLkinw8m1w0o9h2AyoWTvoiiQiymxROucifx+M5nUnkQ+fgpdvhVMuguGzoHa9oCuSCNHmYnFoXneSqNMQ2veDIROhZp2gq5GIUVBKctu61rtHssvV0HkoaCqtBKDSuzCKVAvnYP5/w/jusOkT7zmFpAREQSnJxzl452GY/zs4Ywgce0bQFUnEqestycU5eOs++NfjcGYm9Psz1NC/5xIs/Q0sgxbACMjKLC8kz75BISlJQy3KMmgBjIC07w9XTYEOA3VNUpKG/rkuh2bjVJPCA/DW/d4Id40a0HGQQlKSioJSgnWgAF68CRb+H6x6LehqRErla1Ca2aVmtsrMVpvZr8o5brCZOTPL8LMeSTIH9sML18PyWXDhfdDjP4KuSKRUvgWlmaUB44HLgA7AcDM7bKkXM2uAt7vjYr9qkSRUsA+eGw0r5nqbgF1wR9AViZTJzxblOcBq59xa59w+YCYwsJTjfgP8N7DHx1ok2RzYCz98B5c9Aj3HBV2NSLn8HPVuCWyIeZwDdI89wMzOBFo75141M212EgX788EVQp0GMOZ1SNONF5L8AhvMMbMawJ+A/6zAsTeaWbaZZefm5vpem+6h9Mm+Xd4mYDOv9W4sV0hKivAzKDcCrWMetyp6rlgD4AxgvpmtA84Fskob0HHOTXDOZTjnMpo1a+Zjydr2wTd7v4dnh8DXC6HLcN3+IynFz3/SPwLamVlbvIAcBlxT/KJzbgfQtPixmc0H7nDOZftYU1za9sEHe3bAs4Nh48cw+O9wxpVBVyRSKb61KJ1zBcAtwBvASmC2c26FmT1kZgP8+txE0I3mCfbCjfDNMhg6RSEpKcnXi0TOuXnAvBLP3V/Gsb38rEUC1OcByLgOTr0k6EpEjohm5og/ftgMi8Z7gzbHdlBISkrTsKMk3s5NMHUA7MiB0y7zVigXSWFqUcbQbUEJsCMHJveFnd/AiDkKSQkFtShjaGm1Ktq2Dqb0h/ztMPJFaH12wAWJJIaCskhxa1Ij3lXw3QrYtxtGvQQtzwy6GpGEUVAWUWuyCvbvgVrp0P5yaHuBNz1RJER0jTKGWpNHYPNKePxMWPW691ghKSGkoJQj9+2nMPlyb4VyDdpIiCko0Wj3EflmKUzuBzXTYcw8aHZq0BWJ+EbXKNH1yUrbvh6mDIT0RjD6ZTi6TdAVifhKQVlE1ycroVFr+NGt0PlqaNw6/vEiKU5BKRW37n2o39zrZmvrBokQXaOUilnzjree5Gt3Bl2JSLVTUEp8X7wJ04dBk5Nh8NNBVyNS7SIflBrxjmPlKzDzGmjeHjJfhvpN4/+MSMhEPig14l0O5+DDCdCiC4zKgnrHBF2RSCA0mINGvEtVWAg1asCwaV5gpjcMuiKRwES+RSmlWDrNWwVo7w/elESFpEScglIOlT0JXroZ0mqB6a+HCEQ8KDWQU8LiCfDKz6HdT2D4TKhdL+iKRJJCpK9RaiAnRvZEeO2X0L4fDJkENWsHXZFI0oh0ixI0kHNQmwvg7OvhqskKSZESIh+UkeYcfPmW99+mp8Dlf/SuTYrIIRSUUeUcvP0QTBsCK14IuhqRpBbpa5SR5Ry8eS8s+gucNRo6XBF0RSJJTUEZNYWF8Ppd3oybc26Eyx4Bs6CrEklq6npHzXefeSPcPW5RSIpUkFqUUdOiM/x0ATQ/XSEpUkFqUUbBgQKY+zP4bI73+NgOCkmRSlBQht2B/TDnOvhkhrfXjYhUmrreYVawF54bA6tehUt+Cz1vCboikZQU2RZl6Od5H9gPs0Z4IXnZowpJkSqIbIsy9PO8a9SEYzvCaX0hY0zQ1YiktMgGJYR0nvfeH+D7TdC0HVz0YNDViIRCZLveobRnJzw7GCb3g327gq5GJDQi3aIMlfztXkhuWubtlFi7ftAViYSGgjIMdm+FZ66A71bA0KnQ/vKgKxIJFQVlGMz/A2xeCcOmw6mXBF2NSOgoKMPgogeh0xBofU7QlYiEkgZzUtXOb2DODbBnh7e3jUJSxDe+BqWZXWpmq8xstZn9qpTXbzezz81suZm9bWYn+llPsZS/2Xz7BpjUF1bNg7w1QVcjEnq+BaWZpQHjgcuADsBwM+tQ4rClQIZzrjPwPPCIX/XESumbzbetg8l9vQGckS9CyzODrkgk9PxsUZ4DrHbOrXXO7QNmAgNjD3DOveuc21308AOglY/1HCIlbzbPW+O1JPfshMyXoPXZQVckEgl+BmVLYEPM45yi58pyHfCaj/WkvhppUL8pjH4Fju8WdDUikZEUgzlmNgLIAB4t4/UbzSzbzLJzc3Or9FkpeX1yR463hcPRbeDGf8JxnYKuSCRS/AzKjUDrmMetip47hJldBNwDDHDO7S3tjZxzE5xzGc65jGbNmlWpqJS7PrlpOTx5Prz7W++xFtwVqXZ+BuVHQDsza2tmtYFhQFbsAWbWDfgbXkhu9rGWQ6TM9cmNH8OU/lCrHnS9JuhqRCLLt6B0zhUAtwBvACuB2c65FWb2kJkNKDrsUeAo4DkzW2ZmWWW8XfRs+BCmDoT0hjBmHjQ5OeiKRCLL15k5zrl5wLwSz90f8/1Ffn5+ScXXJ7u3PaY6P7by9v4AM4Z5AzeZL0OjarsZQERKEakpjClzfbLOUTD479CsPTRsEXQ1IpEXqaCEJL8+ufofsGsLdBkGJ/cOuhoRKZIUtwcJsOp1mDEcPvirt72siCQNBWUyWPmytxHYsR1h5FxIi1xDXySpRSYok/ZG88/mwOxMb6bNqJegXpIPNIlEUGSaLkk7kJO3Blp3h2tnQ50GQVcjIqWITIsSkmwgJ3+7998Lfum1JBWSIkkrUkGZND76OzzWDbZ86U1JrFk76IpEpBwKyur2wZPw6u3eiuSNWsc/XkQCp6CsTgsfg9fvgvb9YOgzUCs96IpEpAIUlNXlsxfgrfug45Vw1WR1t0VSSGRGvQN3Wl+45GHofpPukxRJMWpR+sk575rk7q1eN7vnOIWkSApSUPrFOXjjHu+a5NJngq5GRKpAzRs/FBbCa3fCR095Xe2etwZdkYhUgYIy0QoL4ZWfw8dTvK72xb/R9g0iKU5BmWj5W2HtfDj/DrjwXoWkSAgoKBPlQIEXivWbwk/fg7qNg65IRBJEgzmJcGA/PD8GssZ5gzgKSZFQUVBWVcFemD0KVmZ560mqqy0SOup6V8X+fJg1Ela/BX3/B865IeiKRMQHCsqqeP46b5+b/v8HZ40OuhoR8Ukkut6+rW7e/UYY9FeFpEjIRaJFmdDVzffshK/+Caf3h5N6Vf39RCTpRaJFCQla3Tx/GzwzCJ4fC9s3JKQuEUl+kWhRJsTurTB1IGxeCUOnQGMtuisSFQrKivgh1wvJvNUwfAa0uzjoikSkGikoK+KL12DrWrhmFpzcO+hqRKSaKSjL45x3A/mZo+Ck3upui0RUZAZzKm37epjwY9j4sfdYISkSWWpRlmbrVzClv3crkCsMuhoRCZiCsqQtq72QLMiHzCw4vmvQFYlIwELf9a7UrJxtX8PkvnBgH2S+opAUESACLcpKzcpp0AJO/Qmc+x/QvL3PlYlIqgh9UEIFZuV8+ykcdRwc1QwGPF59hYlISgh91zuunCUw+XLIuiXoSkQkSUU7KNcv9mbcpDeGvo8GXY2IJKnoBuW6hfDMFXBUcxjzGjSu4oIZIhJakbhGeZjCQnjjv6BRK+8WoAbHBV2RiCSxaAZljRowfCbUqOUN4IiIlCNaXe9Vr8Hcm6DwADQ8XiEpIhXia1Ca2aVmtsrMVpvZr0p5vY6ZzSp6fbGZtfGtmM9fglkjIPffsG+Xbx8jIuHjW1CaWRowHrgM6AAMN7MOJQ67DtjmnDsF+F/gv/2opWf+fHhuDLQ8C0a9COkN/fgYEQkpP1uU5wCrnXNrnXP7gJnAwBLHDASmFH3/PNDHLLEbY5+X/w7jtj8CJ5wLI+ZAeqNEvr2IRICfQdkSiN1YJqfouVKPcc4VADuAJiXfyMxuNLNsM8vOzc2tVBFHNW/DF416wrXPQZ0GlfpZERFIkVFv59wEYAJARkaGq8zPjrnmWuBaP8oSkYjws0W5EYhd7bZV0XOlHmNmNYFGQJ6PNYmIVJqfQfkR0M7M2ppZbWAYkFXimCwgs+j7IcA7zrlKtRhFRPzmW9fbOVdgZrcAbwBpwETn3AozewjIds5lAX8HnjGz1cBWvDAVEUkqvl6jdM7NA+aVeO7+mO/3AFf5WYOISFVFa2aOiMgRUFCKiMShoBQRiUNBKSISh4JSRCQOBaWISBwKShGROCzVJsKYWS7wdSV/rCmwxYdyqltYzgN0LskqLOdyJOdxonOu1NW8Uy4oj4SZZTvnMoKuo6rCch6gc0lWYTmXRJ+Hut4iInEoKEVE4ohKUE4IuoAECct5gM4lWYXlXBJ6HpG4RikiUhVRaVGKiByxUAVlUm2PWwUVOI/bzexzM1tuZm+b2YlB1FkR8c4l5rjBZubMLGlHXCtyLmY2tOh3s8LMpld3jRVRgb9fJ5jZu2a2tOjvWN8g6qwIM5toZpvN7LMyXjcze6zoXJeb2ZlH9EHOuVB84S0OvAY4CagNfAJ0KHHMzcCTRd8PA2YFXfcRnkdvoF7R9zcl43lU9FyKjmsAvAd8AGQEXXcVfi/tgKXA0UWPmwdd9xGexwTgpqLvOwDrgq67nPO5ADgT+KyM1/sCrwEGnAssPpLPCVOLMim2x02AuOfhnHvXObe76OEHePsRJaOK/E4AfoO3p/ue6iyukipyLjcA451z2wCcc5urucaKqMh5OKBh0feNgG+qsb5Kcc69h7c7QlkGAlOd5wOgsZm1qOznhCkoE7Y9bsAqch6xrsP7FzMZxT2Xoq5Qa+fcq9VZ2BGoyO/lVOBUM1toZh+Y2aXVVl3FVeQ8HgRGmFkO3g4F46qnNF9U9v+nUqXEdrVSOjMbAWQAPw66liNhZjWAPwGjAy4lUWridb974bXy3zOzTs657UEWdQSGA5Odc380sx54+1qd4ZwrDLqwoISpRRmW7XErch6Y2UXAPcAA59zeaqqtsuKdSwPgDGC+ma3Du4aUlaQDOhX5veQAWc65/c65r4Av8IIzmVTkPK4DZgM45xYB6Xhzp1NRhf5/iidMQRmW7XHjnoeZdQP+hheSyXgdrFi55+Kc2+Gca+qca+Oca4N3vXWAcy47mHLLVZG/Xy/itSYxs6Z4XfG11VhjRVTkPNYDfQDM7HS8oMyt1ioTJwsYVTT6fS6wwzm3qdLvEvSoVYJHwPri/Su+Brin6LmH8P7nA+8X/hywGvgQOCnomo/wPP4BfAcsK/rKCrrmIz2XEsfOJ0lHvSv4ezG8SwmfA58Cw4Ku+QjPowOwEG9EfBlwSdA1l3MuM4BNwH68Fv11wM+An8X8TsYXneunR/r3SzNzRETiCFPXW0TEFwpKEZE4FJQiInEoKEVE4lBQiojEoaCUlGVmB8xsWcxXGzPrZWY7ih6vNLMHio6Nff7fZvY/QdcvqUNTGCWV5TvnusY+UbR03gLnXD8zqw8sM7OXi14ufr4usNTM5jrnFlZvyZKK1KKU0HLO7QKWAKeUeD4f70bqSi+OINGkoJRUVjem2z235Itm1gRv/viKEs8fjTcH+73qKVNSnbreksoO63oXOd/MlgKFwB+ccyvMrFfR85/gheSfnXPfVlulktIUlBJGC5xz/cp63szaAh+Y2Wzn3LJqrk1SkLreEjnOWwLtD8BdQdciqUFBKVH1JHBBsm4wJ8lFqweJiMShFqWISBwKShGROBSUIiJxKChFROJQUIqIxKGgFBGJQ0EpIhKHglJEJI7/B4sBUe4syVN/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 5))\n",
    "\n",
    "plt.plot(fpr, tpr, label='Model')\n",
    "plt.plot([0, 1], [0, 1], label='Random', linestyle='--')\n",
    "\n",
    "plt.xlabel('FPR')\n",
    "plt.ylabel('TPR')\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886f8028",
   "metadata": {},
   "source": [
    "## 4.6 ROC AUC\n",
    "\n",
    "* Area under the ROC curve - useful metric\n",
    "* Interpretation of AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "7509a4b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "6643a6e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.843850505725819"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e1c0c11b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8438796286447967"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc(df_scores.fpr, df_scores.tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "0774697c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999430203759136"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc(df_ideal.fpr, df_ideal.tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "9f8658f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.843850505725819"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fpr, tpr, thresholds = roc_curve(y_val, y_pred)\n",
    "auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "29cbdd64",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "9fd31898",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.843850505725819"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "02bb73e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "neg = y_pred[y_val == 0]\n",
    "pos = y_pred[y_val == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "cfeb22b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "ee7da8e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8434"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 100000\n",
    "success = 0 \n",
    "\n",
    "for i in range(n):\n",
    "    pos_ind = random.randint(0, len(pos) - 1)\n",
    "    neg_ind = random.randint(0, len(neg) - 1)\n",
    "\n",
    "    if pos[pos_ind] > neg[neg_ind]:\n",
    "        success = success + 1\n",
    "\n",
    "success / n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "2b9e19e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84646"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 50000\n",
    "\n",
    "np.random.seed(1)\n",
    "pos_ind = np.random.randint(0, len(pos), size=n)\n",
    "neg_ind = np.random.randint(0, len(neg), size=n)\n",
    "\n",
    "(pos[pos_ind] > neg[neg_ind]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6336a636",
   "metadata": {},
   "source": [
    "## 4.7 Cross-Validation\n",
    "\n",
    "* Evaluating the same model on different subsets of data\n",
    "* Getting the average prediction and the spread within predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "92708443",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(df_train, y_train, C=1.0):\n",
    "    dicts = df_train[categorical + numerical].to_dict(orient='records')\n",
    "\n",
    "    dv = DictVectorizer(sparse=False)\n",
    "    X_train = dv.fit_transform(dicts)\n",
    "\n",
    "    model = LogisticRegression(C=C, max_iter=1000)\n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    return dv, model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "4689044d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dv, model = train(df_train, y_train, C=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "08ac302c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(df, dv, model):\n",
    "    dicts = df[categorical + numerical].to_dict(orient='records')\n",
    "\n",
    "    X = dv.transform(dicts)\n",
    "    y_pred = model.predict_proba(X)[:, 1]\n",
    "\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "95d88ee1",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict(df_val, dv, model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "e3c7e607",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "e1d8a427",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "d5c8e07d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tqdm in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (4.61.2)\n",
      "\u001b[33mWARNING: You are using pip version 21.2.2; however, version 21.2.4 is available.\n",
      "You should consider upgrading via the '/home/alexey/.pyenv/versions/3.8.11/bin/python3.8 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "a8fe7363",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "ce936aca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bdfe85df0c7415582b9e25df38dc9c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C=0.001 0.825 +- 0.009\n",
      "C=0.01 0.840 +- 0.009\n",
      "C=0.1 0.841 +- 0.008\n",
      "C=0.5 0.840 +- 0.007\n",
      "C=1 0.841 +- 0.008\n",
      "C=5 0.841 +- 0.008\n",
      "C=10 0.841 +- 0.008\n"
     ]
    }
   ],
   "source": [
    "n_splits = 5\n",
    "\n",
    "for C in tqdm([0.001, 0.01, 0.1, 0.5, 1, 5, 10]):\n",
    "    kfold = KFold(n_splits=n_splits, shuffle=True, random_state=1)\n",
    "\n",
    "    scores = []\n",
    "\n",
    "    for train_idx, val_idx in kfold.split(df_full_train):\n",
    "        df_train = df_full_train.iloc[train_idx]\n",
    "        df_val = df_full_train.iloc[val_idx]\n",
    "\n",
    "        y_train = df_train.churn.values\n",
    "        y_val = df_val.churn.values\n",
    "\n",
    "        dv, model = train(df_train, y_train, C=C)\n",
    "        y_pred = predict(df_val, dv, model)\n",
    "\n",
    "        auc = roc_auc_score(y_val, y_pred)\n",
    "        scores.append(auc)\n",
    "\n",
    "    print('C=%s %.3f +- %.3f' % (C, np.mean(scores), np.std(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "2f72b194",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8419433083969826,\n",
       " 0.8458047775129122,\n",
       " 0.8325145494681918,\n",
       " 0.8325466042079682,\n",
       " 0.8525462018763139]"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "03e81326",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8572386167896259"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv, model = train(df_full_train, df_full_train.churn.values, C=1.0)\n",
    "y_pred = predict(df_test, dv, model)\n",
    "\n",
    "auc = roc_auc_score(y_test, y_pred)\n",
    "auc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db242dff",
   "metadata": {},
   "source": [
    "## 4.8 Summary\n",
    "\n",
    "* Metric - a single number that describes the performance of a model\n",
    "* Accuracy - fraction of correct answers; sometimes misleading \n",
    "* Precision and recall are less misleading when we have class inbalance\n",
    "* ROC Curve - a way to evaluate the performance at all thresholds; okay to use with imbalance\n",
    "* K-Fold CV - more reliable estimate for performance (mean + std)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4941b0ca",
   "metadata": {},
   "source": [
    "## 4.9 Explore more\n",
    "\n",
    "* Check the precision and recall of the dummy classifier that always predict \"FALSE\"\n",
    "* F1 score = 2 * P * R / (P + R)\n",
    "* Evaluate precision and recall at different thresholds, plot P vs R - this way you'll get the precision/recall curve (similar to ROC curve)\n",
    "* Area under the PR curve is also a useful metric\n",
    "\n",
    "Other projects:\n",
    "\n",
    "* Calculate the metrics for datasets from the previous week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ade60b8d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}