Naive Bayes - Gaussian.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Dependências"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/arnaldo/miniconda3/envs/ml/lib/python3.5/site-packages/sklearn/utils/fixes.py:313: FutureWarning: numpy not_equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.\n",
      "  _nan_object_mask = _nan_object_array != _nan_object_array\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "%matplotlib inline\n",
    "pd.options.display.max_rows = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introdução "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Naive Bayes**, técnicamente conhecido como *Posterior Probability*, é um técnica simples de construção de classificadores. Não é um único algoritmo para treinamento de classificadores, mas uma família de algoritmos baseada num princípio em comum: todos os classificadores Naive Bayes assumem que o valor de um atributo em particular é *independente* do valor dos outros atributos, dada a variável da classe. Isso é o que torna o Naive Bayes *\"Naive\"*: ele não considera a dependência entre os atributos. Se pensarmos no caso de classificação textual - onde cada palavra representa um atributo -, por exemplo, a ordem das palavras são importantes para a classificação.\n",
    "\n",
    "**Vantagens**:\n",
    "- Requer um baixo número de dados de treinamento para estimar parâmetros do modelo\n",
    "- Facilmente escalável\n",
    "- Performance comparável com outros classificadores\n",
    "\n",
    "**Desvantagens**:\n",
    "- Pode ser superado por outros classificadores, como árvores de decisão e Random Forests.\n",
    "\n",
    "## Teorema de Bayes\n",
    "\n",
    "A famosa equação de Bayes nos permite fazer predições a partir dos dados. Tal equação é definida pela seguinte fórmula:\n",
    "\n",
    "$$P(A\\mid B)={\\frac {P(B\\mid A)\\,P(A)}{P(B)}}$$\n",
    "\n",
    "Para torná-la menos abstrata, vamos substituir as variáveis $A$ e $B$ por nomes mais tangíveis. Dessa forma, é melhor pensar na equação acima da seguinte forma:\n",
    "\n",
    "<img src=\"images/teorema-bayes.jpg\" width=\"600px\">\n",
    "\n",
    "[Fonte](https://github.com/odubno/GaussNaiveBayes)\n",
    "\n",
    "Onde:\n",
    "- **Posterior Probability**: essa é resposta da predição do nosso Naive Bayes para uma nova amostra, onde cada valor representa a probabilidade da amostra pertencer a cada classe.\n",
    "- **Class Prior Probability**: a probabilidade a priori de uma determinada classe.\n",
    "- **Likelihood**: a verossimilhança é calculada pelo produtório de todas as *Funções de Densidade de Probabilidade Normal* (**Normal Probability Density Functions**). A FDP Normal é calculada usando a distribuição Gaussiana. Daí o nome Gaussian Naive Bayes. Nós utilizaremos a FDP Normal para calcular o valor da probabilidade normal para cada atributo dado uma classe. A FDP Normal é dada pela seguinte fórmula:\n",
    "\n",
    "$$\\Large f(x \\mid \\mu,\\sigma^2) = \\frac{1}{\\sqrt{2\\pi \\sigma^2}} e^{-\\frac{(x-\\mu)^2}{2\\sigma^2}}$$\n",
    "\n",
    "**É importante não confundir verossimilhança com probabilidade**:\n",
    "    - A probabilidade normal é calculada para cada atributo dada uma classe e seu valor está sempre entre 0 e 1.\n",
    "    - A verossimilhança é o produto de todos os valores de probabilidade normal.\n",
    "    \n",
    "Veja [esse link](https://github.com/odubno/GaussNaiveBayes) para entender como a *likelihood* é calculada.\n",
    "\n",
    "- **Predictor Prior Probability**: é o mesmo que **Probabilidade Marginal** (*Marginal Probability*). Representa a probabilidade dados os novos dados sob todos os valores possíveis de atributos para cada classe. Repare que não é necessário calcular esse valor (eles apenas normaliza as nossas probabilidades). Sem esse termo, temos as predições. Com ele, nós temos a probabilidade exata. Entretanto, não normalizar as predições (para gerar as probabilidades) não altera o resultado final."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Dados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
       "0                  5.1               3.5                1.4               0.2   \n",
       "1                  4.9               3.0                1.4               0.2   \n",
       "2                  4.7               3.2                1.3               0.2   \n",
       "3                  4.6               3.1                1.5               0.2   \n",
       "4                  5.0               3.6                1.4               0.2   \n",
       "..                 ...               ...                ...               ...   \n",
       "145                6.7               3.0                5.2               2.3   \n",
       "146                6.3               2.5                5.0               1.9   \n",
       "147                6.5               3.0                5.2               2.0   \n",
       "148                6.2               3.4                5.4               2.3   \n",
       "149                5.9               3.0                5.1               1.8   \n",
       "\n",
       "     class  \n",
       "0        0  \n",
       "1        0  \n",
       "2        0  \n",
       "3        0  \n",
       "4        0  \n",
       "..     ...  \n",
       "145      2  \n",
       "146      2  \n",
       "147      2  \n",
       "148      2  \n",
       "149      2  \n",
       "\n",
       "[150 rows x 5 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris = load_iris()\n",
    "\n",
    "df = pd.DataFrame(data=iris.data, columns=iris.feature_names)\n",
    "df['class'] = iris.target\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.843333</td>\n",
       "      <td>3.057333</td>\n",
       "      <td>3.758000</td>\n",
       "      <td>1.199333</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.828066</td>\n",
       "      <td>0.435866</td>\n",
       "      <td>1.765298</td>\n",
       "      <td>0.762238</td>\n",
       "      <td>0.819232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.300000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5.100000</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.800000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.350000</td>\n",
       "      <td>1.300000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.400000</td>\n",
       "      <td>3.300000</td>\n",
       "      <td>5.100000</td>\n",
       "      <td>1.800000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.900000</td>\n",
       "      <td>4.400000</td>\n",
       "      <td>6.900000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sepal length (cm)  sepal width (cm)  petal length (cm)  \\\n",
       "count         150.000000        150.000000         150.000000   \n",
       "mean            5.843333          3.057333           3.758000   \n",
       "std             0.828066          0.435866           1.765298   \n",
       "min             4.300000          2.000000           1.000000   \n",
       "25%             5.100000          2.800000           1.600000   \n",
       "50%             5.800000          3.000000           4.350000   \n",
       "75%             6.400000          3.300000           5.100000   \n",
       "max             7.900000          4.400000           6.900000   \n",
       "\n",
       "       petal width (cm)       class  \n",
       "count        150.000000  150.000000  \n",
       "mean           1.199333    1.000000  \n",
       "std            0.762238    0.819232  \n",
       "min            0.100000    0.000000  \n",
       "25%            0.300000    0.000000  \n",
       "50%            1.300000    1.000000  \n",
       "75%            1.800000    2.000000  \n",
       "max            2.500000    2.000000  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4) (150,)\n"
     ]
    }
   ],
   "source": [
    "x = df.drop(labels='class', axis=1).values\n",
    "y = df['class'].values\n",
    "\n",
    "print(x.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Implementação "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GaussianNaiveBayes():\n",
    "    def __init__(self, priors=None):\n",
    "        self.priors = priors\n",
    "        self.theta_ = 0.0\n",
    "        self.sigma_ = 0.0\n",
    "\n",
    "    def fit(self, x, y):\n",
    "        classes, counts = np.unique(y, return_counts=True)\n",
    "        self.priors = counts / counts.sum() if self.priors is None else self.priors\n",
    "\n",
    "        self.theta_ = np.array([np.mean(x[y == c], axis=0) for c in classes])\n",
    "        self.sigma_ = np.array([np.var(x[y == c], axis=0) for c in classes])\n",
    "\n",
    "    def predict(self, x):\n",
    "        return np.argmax(self.predict_proba(x), axis=1)\n",
    "\n",
    "    def predict_proba(self, x):\n",
    "        y_pred = []\n",
    "        for sample in x:\n",
    "            joint_prob = self.__joint_prob(sample)\n",
    "            posterior_prob = self.__posterior_prob(joint_prob)\n",
    "            y_pred.append(posterior_prob)\n",
    "        return np.array(y_pred)\n",
    "\n",
    "    def __normal_pdf(self, x, mean_c, var_c):\n",
    "        exponent = ((x - mean_c)**2) / (2 * var_c)\n",
    "        f = (1.0 / np.sqrt(2.0 * np.pi * var_c)) * np.exp(-exponent)\n",
    "        return np.prod(f)\n",
    "\n",
    "    def __joint_prob(self, x):\n",
    "        joint_prob = []\n",
    "        for p, t, s in zip(self.priors, self.theta_, self.sigma_):\n",
    "            joint_prob.append(p * self.__normal_pdf(x, t, s))\n",
    "        return joint_prob\n",
    "\n",
    "    def __posterior_prob(self, joint_prob):\n",
    "        marginal_prob = np.sum(joint_prob)\n",
    "        return joint_prob / marginal_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Teste "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 5.006  3.428  1.462  0.246]\n",
      " [ 5.936  2.77   4.26   1.326]\n",
      " [ 6.588  2.974  5.552  2.026]]\n",
      "[[ 0.121764  0.140816  0.029556  0.010884]\n",
      " [ 0.261104  0.0965    0.2164    0.038324]\n",
      " [ 0.396256  0.101924  0.298496  0.073924]]\n",
      "[0 0 0 0 1 1 1 2 2 2]\n",
      "[[  1.00000000e+000   2.71568036e-018   4.97897739e-025]\n",
      " [  1.00000000e+000   6.08147999e-017   1.16347854e-022]\n",
      " [  1.00000000e+000   2.15419961e-016   1.67026397e-023]\n",
      " [  1.00000000e+000   3.90567500e-016   1.38142807e-023]\n",
      " [  5.37488397e-042   9.99999177e-001   8.22738054e-007]\n",
      " [  1.60315864e-094   9.57480597e-001   4.25194033e-002]\n",
      " [  5.65738810e-083   9.99558019e-001   4.41981441e-004]\n",
      " [  6.65703592e-274   6.88505128e-011   1.00000000e+000]\n",
      " [  1.12727751e-222   3.34619351e-009   9.99999997e-001]\n",
      " [  8.68653305e-255   2.56736467e-011   1.00000000e+000]]\n"
     ]
    }
   ],
   "source": [
    "clf = GaussianNaiveBayes(priors=[0.1, 0.2, 0.7])\n",
    "clf.fit(x, y)\n",
    "\n",
    "print(clf.theta_)\n",
    "print(clf.sigma_)\n",
    "print(clf.predict(x[::15]))\n",
    "print(clf.predict_proba(x[::15]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.953333333333\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf.predict(x)\n",
    "print(accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparação com o Scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 5.006  3.428  1.462  0.246]\n",
      " [ 5.936  2.77   4.26   1.326]\n",
      " [ 6.588  2.974  5.552  2.026]]\n",
      "[[ 0.121764  0.140816  0.029556  0.010884]\n",
      " [ 0.261104  0.0965    0.2164    0.038324]\n",
      " [ 0.396256  0.101924  0.298496  0.073924]]\n",
      "[0 0 0 0 1 1 1 2 2 2]\n",
      "[[  1.00000000e+000   2.71568036e-018   4.97897739e-025]\n",
      " [  1.00000000e+000   6.08147999e-017   1.16347854e-022]\n",
      " [  1.00000000e+000   2.15419961e-016   1.67026397e-023]\n",
      " [  1.00000000e+000   3.90567500e-016   1.38142807e-023]\n",
      " [  5.37488397e-042   9.99999177e-001   8.22738054e-007]\n",
      " [  1.60315864e-094   9.57480597e-001   4.25194033e-002]\n",
      " [  5.65738810e-083   9.99558019e-001   4.41981441e-004]\n",
      " [  6.65703592e-274   6.88505128e-011   1.00000000e+000]\n",
      " [  1.12727751e-222   3.34619351e-009   9.99999997e-001]\n",
      " [  8.68653305e-255   2.56736467e-011   1.00000000e+000]]\n"
     ]
    }
   ],
   "source": [
    "clf_sk = GaussianNB(priors=[0.1, 0.2, 0.7], var_smoothing=0.0)\n",
    "clf_sk.fit(x, y)\n",
    "\n",
    "print(clf_sk.theta_)\n",
    "print(clf_sk.sigma_)\n",
    "print(clf_sk.predict(x[::15]))\n",
    "print(clf_sk.predict_proba(x[::15]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.953333333333\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf_sk.predict(x)\n",
    "print(accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Referências"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Repositório do GitHub](https://github.com/odubno/GaussNaiveBayes)\n",
    "- [Naive Bayes Classifier From Scratch](https://chrisalbon.com/machine_learning/naive_bayes/naive_bayes_classifier_from_scratch/)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}