C1_W1_Assignment (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1: Logistic Regression\n",
    "Welcome to week one of this specialization. You will learn about logistic regression. Concretely, you will be implementing logistic regression for sentiment analysis on tweets. Given a tweet, you will decide if it has a positive sentiment or a negative one. Specifically you will: \n",
    "\n",
    "* Learn how to extract features for logistic regression given some text\n",
    "* Implement logistic regression from scratch\n",
    "* Apply logistic regression on a natural language processing task\n",
    "* Test using your logistic regression\n",
    "* Perform error analysis\n",
    "\n",
    "## Important Note on Submission to the AutoGrader\n",
    "\n",
    "Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:\n",
    "\n",
    "1. You have not added any _extra_ `print` statement(s) in the assignment.\n",
    "2. You have not added any _extra_ code cell(s) in the assignment.\n",
    "3. You have not changed any of the function parameters.\n",
    "4. You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.\n",
    "5. You are not changing the assignment code where it is not required, like creating _extra_ variables.\n",
    "\n",
    "If you do any of the following, you will get something like, `Grader Error: Grader feedback not found` (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don't remember the changes you have made, you can get a fresh copy of the assignment by following these [instructions](https://www.coursera.org/learn/classification-vector-spaces-in-nlp/supplement/YLuAg/h-ow-to-refresh-your-workspace).\n",
    "\n",
    "Lets get started!\n",
    "\n",
    "We will be using a data set of tweets. Hopefully you will get more than 99% accuracy.  \n",
    "Run the cell below to load in the packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [Import Functions and Data](#0)\n",
    "- [1 - Logistic Regression](#1)\n",
    "    - [1.1 - Sigmoid](#1-1)\n",
    "        - [Exercise 1 - sigmoid (UNQ_C1)](#ex-1)\n",
    "    - [1.2 - Cost function and Gradient](#1-2)\n",
    "        - [Exercise 2 - gradientDescent (UNQ_C2)](#ex-2)\n",
    "- [2 - Extracting the Features](#2)\n",
    "    - [Exercise 3 - extract_features (UNQ_C3)](#ex-3)\n",
    "- [3 - Training Your Model](#3)\n",
    "- [4 - Test your Logistic Regression](#4)\n",
    "    - [Exercise 4 - predict_tweet (UNQ_C4)](#ex-4)\n",
    "    - [4.1 - Check the Performance using the Test Set](#4-1)\n",
    "        - [Exercise 5 - test_logistic_regression (UNQ_C5)](#ex-5)\n",
    "- [5 - Error Analysis](#5)\n",
    "- [6 - Predict with your own Tweet](#6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Import Functions and Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package twitter_samples to\n",
      "[nltk_data]     /home/jovyan/nltk_data...\n",
      "[nltk_data]   Package twitter_samples is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run this cell to import nltk\n",
    "import nltk\n",
    "from os import getcwd\n",
    "import w1_unittest\n",
    "\n",
    "nltk.download('twitter_samples')\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imported Functions\n",
    "\n",
    "Download the data needed for this assignment. Check out the [documentation for the twitter_samples dataset](http://www.nltk.org/howto/twitter.html).\n",
    "\n",
    "* twitter_samples: if you're running this notebook on your local computer, you will need to download it using:\n",
    "```Python\n",
    "nltk.download('twitter_samples')\n",
    "```\n",
    "\n",
    "* stopwords: if you're running this notebook on your local computer, you will need to download it using:\n",
    "```python\n",
    "nltk.download('stopwords')\n",
    "```\n",
    "\n",
    "#### Import some helper functions that we provided in the utils.py file:\n",
    "* process_tweet: cleans the text, tokenizes it into separate words, removes stopwords, and converts words to stems.\n",
    "* build_freqs: this counts how often a word in the 'corpus' (the entire set of tweets) was associated with a positive label '1' or a negative label '0', then builds the 'freqs' dictionary, where each key is the (word,label) tuple, and the value is the count of its frequency within the corpus of tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "filePath = f\"{getcwd()}/../tmp2/\"\n",
    "nltk.data.path.append(filePath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from nltk.corpus import twitter_samples \n",
    "\n",
    "from utils import process_tweet, build_freqs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the Data\n",
    "* The `twitter_samples` contains subsets of five thousand positive_tweets, five thousand negative_tweets, and the full set of 10,000 tweets.  \n",
    "    * If you used all three datasets, we would introduce duplicates of the positive tweets and negative tweets.  \n",
    "    * You will select just the five thousand positive tweets and five thousand negative tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the set of positive and negative tweets\n",
    "all_positive_tweets = twitter_samples.strings('positive_tweets.json')\n",
    "all_negative_tweets = twitter_samples.strings('negative_tweets.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Train test split: 20% will be in the test set, and 80% in the training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the data into two pieces, one for training and one for testing (validation set) \n",
    "test_pos = all_positive_tweets[4000:]\n",
    "train_pos = all_positive_tweets[:4000]\n",
    "test_neg = all_negative_tweets[4000:]\n",
    "train_neg = all_negative_tweets[:4000]\n",
    "\n",
    "train_x = train_pos + train_neg \n",
    "test_x = test_pos + test_neg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Create the numpy array of positive labels and negative labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine positive and negative labels\n",
    "train_y = np.append(np.ones((len(train_pos), 1)), np.zeros((len(train_neg), 1)), axis=0)\n",
    "test_y = np.append(np.ones((len(test_pos), 1)), np.zeros((len(test_neg), 1)), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_y.shape = (8000, 1)\n",
      "test_y.shape = (2000, 1)\n"
     ]
    }
   ],
   "source": [
    "# Print the shape train and test sets\n",
    "print(\"train_y.shape = \" + str(train_y.shape))\n",
    "print(\"test_y.shape = \" + str(test_y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Create the frequency dictionary using the imported build_freqs function.  \n",
    "    * We highly recommend that you open utils.py and read the build_freqs function to understand what it is doing.\n",
    "    * To view the file directory, go to the menu and click File->Open.\n",
    "\n",
    "```Python\n",
    "    for y,tweet in zip(ys, tweets):\n",
    "        for word in process_tweet(tweet):\n",
    "            pair = (word, y)\n",
    "            if pair in freqs:\n",
    "                freqs[pair] += 1\n",
    "            else:\n",
    "                freqs[pair] = 1\n",
    "```\n",
    "* Notice how the outer for loop goes through each tweet, and the inner for loop steps through each word in a tweet.\n",
    "* The 'freqs' dictionary is the frequency dictionary that's being built. \n",
    "* The key is the tuple (word, label), such as (\"happy\",1) or (\"happy\",0).  The value stored for each key is the count of how many times the word \"happy\" was associated with a positive label, or how many times \"happy\" was associated with a negative label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(freqs) = <class 'dict'>\n",
      "len(freqs) = 11436\n"
     ]
    }
   ],
   "source": [
    "# create frequency dictionary\n",
    "freqs = build_freqs(train_x, train_y)\n",
    "\n",
    "# check the output\n",
    "print(\"type(freqs) = \" + str(type(freqs)))\n",
    "print(\"len(freqs) = \" + str(len(freqs.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```\n",
    "type(freqs) = <class 'dict'>\n",
    "len(freqs) = 11436\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process Tweet\n",
    "The given function 'process_tweet' tokenizes the tweet into individual words, removes stop words and applies stemming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is an example of a positive tweet: \n",
      " #FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)\n",
      "\n",
      "This is an example of the processed version of the tweet: \n",
      " ['followfriday', 'top', 'engag', 'member', 'commun', 'week', ':)']\n"
     ]
    }
   ],
   "source": [
    "# test the function below\n",
    "print('This is an example of a positive tweet: \\n', train_x[0])\n",
    "print('\\nThis is an example of the processed version of the tweet: \\n', process_tweet(train_x[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```\n",
    "This is an example of a positive tweet: \n",
    " #FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)\n",
    " \n",
    "This is an example of the processes version: \n",
    " ['followfriday', 'top', 'engag', 'member', 'commun', 'week', ':)']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Logistic Regression \n",
    "\n",
    "<a name='1-1'></a>\n",
    "### 1.1 - Sigmoid\n",
    "You will learn to use logistic regression for text classification. \n",
    "* The sigmoid function is defined as: \n",
    "\n",
    "$$ h(z) = \\frac{1}{1+\\exp^{-z}} \\tag{1}$$\n",
    "\n",
    "It maps the input 'z' to a value that ranges between 0 and 1, and so it can be treated as a probability. \n",
    "\n",
    "<div style=\"width:image width px; font-size:100%; text-align:center;\"><img src='./images/sigmoid_plot.jpg' alt=\"alternate text\" width=\"width\" height=\"height\" style=\"width:300px;height:200px;\" /> Figure 1 </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-1'></a>\n",
    "### Exercise 1 -  sigmoid\n",
    "Implement the sigmoid function. \n",
    "* You will want this function to work if z is a scalar as well as if it is an array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li><a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.exp.html\" > numpy.exp </a> </li>\n",
    "\n",
    "</ul>\n",
    "</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C1 GRADED FUNCTION: sigmoid\n",
    "def sigmoid(z): \n",
    "    '''\n",
    "    Input:\n",
    "        z: is the input (can be a scalar or an array)\n",
    "    Output:\n",
    "        h: the sigmoid of z\n",
    "    '''\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # calculate the sigmoid of z\n",
    "    h = 1 / (1+ np.exp(-1*z))\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESS!\n",
      "CORRECT!\n"
     ]
    }
   ],
   "source": [
    "# Testing your function \n",
    "if (sigmoid(0) == 0.5):\n",
    "    print('SUCCESS!')\n",
    "else:\n",
    "    print('Oops!')\n",
    "\n",
    "if (sigmoid(4.92) == 0.9927537604041685):\n",
    "    print('CORRECT!')\n",
    "else:\n",
    "    print('Oops again!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w1_unittest.test_sigmoid(sigmoid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression: Regression and a Sigmoid\n",
    "\n",
    "Logistic regression takes a regular linear regression, and applies a sigmoid to the output of the linear regression.\n",
    "\n",
    "Regression:\n",
    "$$z = \\theta_0 x_0 + \\theta_1 x_1 + \\theta_2 x_2 + ... \\theta_N x_N$$\n",
    "Note that the $\\theta$ values are \"weights\". If you took the deep learning specialization, we referred to the weights with the 'w' vector.  In this course, we're using a different variable $\\theta$ to refer to the weights.\n",
    "\n",
    "Logistic regression\n",
    "$$ h(z) = \\frac{1}{1+\\exp^{-z}}$$\n",
    "$$z = \\theta_0 x_0 + \\theta_1 x_1 + \\theta_2 x_2 + ... \\theta_N x_N$$\n",
    "We will refer to 'z' as the 'logits'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-2'></a>\n",
    "### 1.2 - Cost function and Gradient\n",
    "\n",
    "The cost function used for logistic regression is the average of the log loss across all training examples:\n",
    "\n",
    "$$J(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m y^{(i)}\\log (h(z(\\theta)^{(i)})) + (1-y^{(i)})\\log (1-h(z(\\theta)^{(i)}))\\tag{5} $$\n",
    "* $m$ is the number of training examples\n",
    "* $y^{(i)}$ is the actual label of training example 'i'.\n",
    "* $h(z^{(i)})$ is the model's prediction for the training example 'i'.\n",
    "\n",
    "The loss function for a single training example is\n",
    "$$ Loss = -1 \\times \\left( y^{(i)}\\log (h(z(\\theta)^{(i)})) + (1-y^{(i)})\\log (1-h(z(\\theta)^{(i)})) \\right)$$\n",
    "\n",
    "* All the $h$ values are between 0 and 1, so the logs will be negative. That is the reason for the factor of -1 applied to the sum of the two loss terms.\n",
    "* Note that when the model predicts 1 ($h(z(\\theta)) = 1$) and the label 'y' is also 1, the loss for that training example is 0. \n",
    "* Similarly, when the model predicts 0 ($h(z(\\theta)) = 0$) and the actual label is also 0, the loss for that training example is 0. \n",
    "* However, when the model prediction is close to 1 ($h(z(\\theta)) = 0.9999$) and the label is 0, the second term of the log loss becomes a large negative number, which is then multiplied by the overall factor of -1 to convert it to a positive loss value. $-1 \\times (1 - 0) \\times log(1 - 0.9999) \\approx 9.2$ The closer the model prediction gets to 1, the larger the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.210340371976294"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify that when the model predicts close to 1, but the actual label is 0, the loss is a large positive value\n",
    "-1 * (1 - 0) * np.log(1 - 0.9999) # loss is about 9.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Likewise, if the model predicts close to 0 ($h(z) = 0.0001$) but the actual label is 1, the first term in the loss function becomes a large number: $-1 \\times log(0.0001) \\approx 9.2$.  The closer the prediction is to zero, the larger the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.210340371976182"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify that when the model predicts close to 0 but the actual label is 1, the loss is a large positive value\n",
    "-1 * np.log(0.0001) # loss is about 9.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Update the weights\n",
    "\n",
    "To update your weight vector $\\theta$, you will apply gradient descent to iteratively improve your model's predictions.  \n",
    "The gradient of the cost function $J$ with respect to one of the weights $\\theta_j$ is:\n",
    "\n",
    "$$\\nabla_{\\theta_j}J(\\theta) = \\frac{1}{m} \\sum_{i=1}^m(h^{(i)}-y^{(i)})x^{(i)}_j \\tag{5}$$\n",
    "* 'i' is the index across all 'm' training examples.\n",
    "* 'j' is the index of the weight $\\theta_j$, so $x^{(i)}_j$ is the feature associated with weight $\\theta_j$\n",
    "\n",
    "* To update the weight $\\theta_j$, we adjust it by subtracting a fraction of the gradient determined by $\\alpha$:\n",
    "$$\\theta_j = \\theta_j - \\alpha \\times \\nabla_{\\theta_j}J(\\theta) $$\n",
    "* The learning rate $\\alpha$ is a value that we choose to control how big a single update will be.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-2'></a>\n",
    "### Exercise 2 - gradientDescent\n",
    "Implement gradient descent function.\n",
    "* The number of iterations 'num_iters\" is the number of times that you'll use the entire training set.\n",
    "* For each iteration, you'll calculate the cost function using all training examples (there are 'm' training examples), and for all features.\n",
    "* Instead of updating a single weight $\\theta_i$ at a time, we can update all the weights in the column vector:  \n",
    "$$\\mathbf{\\theta} = \\begin{pmatrix}\n",
    "\\theta_0\n",
    "\\\\\n",
    "\\theta_1\n",
    "\\\\ \n",
    "\\theta_2 \n",
    "\\\\ \n",
    "\\vdots\n",
    "\\\\ \n",
    "\\theta_n\n",
    "\\end{pmatrix}$$\n",
    "* $\\mathbf{\\theta}$ has dimensions (n+1, 1), where 'n' is the number of features, and there is one more element for the bias term $\\theta_0$ (note that the corresponding feature value $\\mathbf{x_0}$ is 1).\n",
    "* The 'logits', 'z', are calculated by multiplying the feature matrix 'x' with the weight vector 'theta'.  $z = \\mathbf{x}\\mathbf{\\theta}$\n",
    "    * $\\mathbf{x}$ has dimensions (m, n+1) \n",
    "    * $\\mathbf{\\theta}$: has dimensions (n+1, 1)\n",
    "    * $\\mathbf{z}$: has dimensions (m, 1)\n",
    "* The prediction 'h', is calculated by applying the sigmoid to each element in 'z': $h(z) = sigmoid(z)$, and has dimensions (m,1).\n",
    "* The cost function $J$ is calculated by taking the dot product of the vectors 'y' and 'log(h)'.  Since both 'y' and 'h' are column vectors (m,1), transpose the vector to the left, so that matrix multiplication of a row vector with column vector performs the dot product.\n",
    "$$J = \\frac{-1}{m} \\times \\left(\\mathbf{y}^T \\cdot log(\\mathbf{h}) + \\mathbf{(1-y)}^T \\cdot log(\\mathbf{1-h}) \\right)$$\n",
    "* The update of theta is also vectorized.  Because the dimensions of $\\mathbf{x}$ are (m, n+1), and both $\\mathbf{h}$ and $\\mathbf{y}$ are (m, 1), we need to transpose the $\\mathbf{x}$ and place it on the left in order to perform matrix multiplication, which then yields the (n+1, 1) answer we need:\n",
    "$$\\mathbf{\\theta} = \\mathbf{\\theta} - \\frac{\\alpha}{m} \\times \\left( \\mathbf{x}^T \\cdot \\left( \\mathbf{h-y} \\right) \\right)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>use numpy.dot for matrix multiplication.</li>\n",
    "    <li>To ensure that the fraction -1/m is a decimal value, cast either the numerator or denominator (or both), like `float(1)`, or write `1.` for the float version of 1. </li>\n",
    "</ul>\n",
    "</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C2 GRADED FUNCTION: gradientDescent\n",
    "def gradientDescent(x, y, theta, alpha, num_iters):\n",
    "    '''\n",
    "    Input:\n",
    "        x: matrix of features which is (m,n+1)\n",
    "        y: corresponding labels of the input matrix x, dimensions (m,1)\n",
    "        theta: weight vector of dimension (n+1,1)\n",
    "        alpha: learning rate\n",
    "        num_iters: number of iterations you want to train your model for\n",
    "    Output:\n",
    "        J: the final cost\n",
    "        theta: your final weight vector\n",
    "    Hint: you might want to print the cost to make sure that it is going down.\n",
    "    '''\n",
    "    m = x.shape[0]\n",
    "    \n",
    "    for i in range(0, num_iters):\n",
    "        \n",
    "        # get z, the dot product of x and theta\n",
    "        z = np.dot(x , theta)\n",
    "        \n",
    "        # get the sigmoid of z\n",
    "        h = sigmoid(z)\n",
    "        \n",
    "        # calculate the cost function\n",
    "        J = -1 * np.sum((y*np.log(h) + (1-y)*np.log(1-h))) / m\n",
    "\n",
    "        # update the weights theta\n",
    "        theta = theta - (alpha * np.dot(np.transpose(x), (h-y))) / m\n",
    "        \n",
    "    J = float(J)\n",
    "    return J, theta\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cost after training is 0.67094970.\n",
      "The resulting vector of weights is [4.1e-07, 0.00035658, 7.309e-05]\n"
     ]
    }
   ],
   "source": [
    "# Check the function\n",
    "# Construct a synthetic test case using numpy PRNG functions\n",
    "np.random.seed(1)\n",
    "# X input is 10 x 3 with ones for the bias terms\n",
    "tmp_X = np.append(np.ones((10, 1)), np.random.rand(10, 2) * 2000, axis=1)\n",
    "# Y Labels are 10 x 1\n",
    "tmp_Y = (np.random.rand(10, 1) > 0.35).astype(float)\n",
    "\n",
    "# Apply gradient descent\n",
    "tmp_J, tmp_theta = gradientDescent(tmp_X, tmp_Y, np.zeros((3, 1)), 1e-8, 700)\n",
    "print(f\"The cost after training is {tmp_J:.8f}.\")\n",
    "print(f\"The resulting vector of weights is {[round(t, 8) for t in np.squeeze(tmp_theta)]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```\n",
    "The cost after training is 0.67094970.\n",
    "The resulting vector of weights is [4.1e-07, 0.00035658, 7.309e-05]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w1_unittest.test_gradientDescent(gradientDescent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Extracting the Features\n",
    "\n",
    "* Given a list of tweets, extract the features and store them in a matrix. You will extract two features.\n",
    "    * The first feature is the number of positive words in a tweet.\n",
    "    * The second feature is the number of negative words in a tweet. \n",
    "* Then train your logistic regression classifier on these features.\n",
    "* Test the classifier on a validation set. \n",
    "\n",
    "<a name='ex-3'></a>\n",
    "### Exercise 3 - extract_features\n",
    "Implement the extract_features function. \n",
    "* This function takes in a single tweet.\n",
    "* Process the tweet using the imported `process_tweet` function and save the list of tweet words.\n",
    "* Loop through each word in the list of processed words\n",
    "    * For each word, check the 'freqs' dictionary for the count when that word has a positive '1' label. (Check for the key (word, 1.0)\n",
    "    * Do the same for the count for when the word is associated with the negative label '0'. (Check for the key (word, 0.0).)\n",
    "\n",
    "**Note:** In the implementation instructions provided above, the prediction of being positive or negative depends on feature vector which counts-in duplicate words - this is different from what you have seen in the lecture videos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Make sure you handle cases when the (word, label) key is not found in the dictionary. </li>\n",
    "    <li> Search the web for hints about using the 'get' function of a Python dictionary.  Here is an <a href=\"https://www.programiz.com/python-programming/methods/dictionary/get\" > example </a> </li>\n",
    "</ul>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C3 GRADED FUNCTION: extract_features\n",
    "def extract_features(tweet, freqs, process_tweet=process_tweet):\n",
    "    '''\n",
    "    Input: \n",
    "        tweet: a string containing one tweet\n",
    "        freqs: a dictionary corresponding to the frequencies of each tuple (word, label)\n",
    "    Output: \n",
    "        x: a feature vector of dimension (1,3)\n",
    "    '''\n",
    "    # process_tweet tokenizes, stems, and removes stopwords\n",
    "    word_l = process_tweet(tweet)\n",
    "    \n",
    "    # 3 elements for [bias, positive, negative] counts\n",
    "    x = np.zeros(3) \n",
    "    \n",
    "    # bias term is set to 1\n",
    "    x[0] = 1 \n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    \n",
    "    # loop through each word in the list of words\n",
    "    for word in word_l:\n",
    "        \n",
    "        # increment the word count for the positive label 1\n",
    "        if ((word ,1.0) in freqs) : \n",
    "            x[1] += freqs[(word , 1.0)]\n",
    "        else : \n",
    "            x[1] +=0\n",
    "        \n",
    "        # increment the word count for the negative label 0\n",
    "        if ((word , 0) in freqs) : \n",
    "            x[2] += freqs[(word , 0)]\n",
    "        else : \n",
    "            x[2] +=0\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    x = x.reshape(1, 3)  # reshape to add batch dimension\n",
    "    assert(x.shape == (1, 3))\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.000e+00 3.133e+03 6.100e+01]]\n"
     ]
    }
   ],
   "source": [
    "# Check your function\n",
    "# test 1\n",
    "# test on training data\n",
    "tmp1 = extract_features(train_x[0], freqs)\n",
    "print(tmp1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```\n",
    "[[1.000e+00 3.133e+03 6.100e+01]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "# test 2:\n",
    "# check for when the words are not in the freqs dictionary\n",
    "tmp2 = extract_features('blorb bleeeeb bloooob', freqs)\n",
    "print(tmp2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```\n",
    "[[1. 0. 0.]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w1_unittest.test_extract_features(extract_features, freqs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Training Your Model\n",
    "\n",
    "To train the model:\n",
    "* Stack the features for all training examples into a matrix X. \n",
    "* Call `gradientDescent`, which you've implemented above.\n",
    "\n",
    "This section is given to you.  Please read it for understanding and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cost after training is 0.22522315.\n",
      "The resulting vector of weights is [6e-08, 0.00053818, -0.0005583]\n"
     ]
    }
   ],
   "source": [
    "# collect the features 'x' and stack them into a matrix 'X'\n",
    "X = np.zeros((len(train_x), 3))\n",
    "for i in range(len(train_x)):\n",
    "    X[i, :]= extract_features(train_x[i], freqs)\n",
    "\n",
    "# training labels corresponding to X\n",
    "Y = train_y\n",
    "\n",
    "# Apply gradient descent\n",
    "J, theta = gradientDescent(X, Y, np.zeros((3, 1)), 1e-9, 1500)\n",
    "print(f\"The cost after training is {J:.8f}.\")\n",
    "print(f\"The resulting vector of weights is {[round(t, 8) for t in np.squeeze(theta)]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "\n",
    "```\n",
    "The cost after training is 0.22522315.\n",
    "The resulting vector of weights is [6e-08, 0.00053818, -0.0005583]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4'></a>\n",
    "## 4 -  Test your Logistic Regression\n",
    "\n",
    "It is time for you to test your logistic regression function on some new input that your model has not seen before. \n",
    "<a name='ex-4'></a>\n",
    "### Exercise 4 - predict_tweet\n",
    "Implement `predict_tweet`.\n",
    "Predict whether a tweet is positive or negative.\n",
    "\n",
    "* Given a tweet, process it, then extract the features.\n",
    "* Apply the model's learned weights on the features to get the logits.\n",
    "* Apply the sigmoid to the logits to get the prediction (a value between 0 and 1).\n",
    "\n",
    "$$y_{pred} = sigmoid(\\mathbf{x} \\cdot \\theta)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C4 GRADED FUNCTION: predict_tweet\n",
    "def predict_tweet(tweet, freqs, theta):\n",
    "    '''\n",
    "    Input: \n",
    "        tweet: a string\n",
    "        freqs: a dictionary corresponding to the frequencies of each tuple (word, label)\n",
    "        theta: (3,1) vector of weights\n",
    "    Output: \n",
    "        y_pred: the probability of a tweet being positive or negative\n",
    "    '''\n",
    "    ### START CODE HERE ###\n",
    "    \n",
    "    # extract the features of the tweet and store it into x\n",
    "    x = extract_features(tweet , freqs)\n",
    "    \n",
    "    # make the prediction using x and theta\n",
    "    y_pred = sigmoid(np.dot(x , theta))\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am happy -> 0.519275\n",
      "I am bad -> 0.494347\n",
      "this movie should have been great. -> 0.515979\n",
      "great -> 0.516065\n",
      "great great -> 0.532096\n",
      "great great great -> 0.548062\n",
      "great great great great -> 0.563929\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to test your function\n",
    "for tweet in ['I am happy', 'I am bad', 'this movie should have been great.', 'great', 'great great', 'great great great', 'great great great great']:\n",
    "    print( '%s -> %f' % (tweet, predict_tweet(tweet, freqs, theta)))    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "```\n",
    "I am happy -> 0.519275\n",
    "I am bad -> 0.494347\n",
    "this movie should have been great. -> 0.515979\n",
    "great -> 0.516065\n",
    "great great -> 0.532096\n",
    "great great great -> 0.548062\n",
    "great great great great -> 0.563929\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.83110307]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feel free to check the sentiment of your own tweet below\n",
    "my_tweet = 'I am learning :)'\n",
    "predict_tweet(my_tweet, freqs, theta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w1_unittest.test_predict_tweet(predict_tweet, freqs, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4-1'></a>\n",
    "### 4.1 -  Check the Performance using the Test Set\n",
    "After training your model using the training set above, check how your model might perform on real, unseen data, by testing it against the test set.\n",
    "\n",
    "<a name='ex-5'></a>\n",
    "### Exercise 5 - test_logistic_regression\n",
    "Implement `test_logistic_regression`. \n",
    "* Given the test data and the weights of your trained model, calculate the accuracy of your logistic regression model. \n",
    "* Use your 'predict_tweet' function to make predictions on each tweet in the test set.\n",
    "* If the prediction is > 0.5, set the model's classification 'y_hat' to 1, otherwise set the model's classification 'y_hat' to 0.\n",
    "* A prediction is accurate when the y_hat equals the test_y.  Sum up all the instances when they are equal and divide by m.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use np.asarray() to convert a list to a numpy array</li>\n",
    "    <li>Use numpy.squeeze() to make an (m,1) dimensional array into an (m,) array </li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C5 GRADED FUNCTION: test_logistic_regression\n",
    "def test_logistic_regression(test_x, test_y, freqs, theta, predict_tweet=predict_tweet):\n",
    "    \"\"\"\n",
    "    Input: \n",
    "        test_x: a list of tweets\n",
    "        test_y: (m, 1) vector with the corresponding labels for the list of tweets\n",
    "        freqs: a dictionary with the frequency of each pair (or tuple)\n",
    "        theta: weight vector of dimension (3, 1)\n",
    "    Output: \n",
    "        accuracy: (# of tweets classified correctly) / (total # of tweets)\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    \n",
    "    # the list for storing predictions\n",
    "    y_hat = []\n",
    "\n",
    "    for tweet in test_x:\n",
    "        # get the label prediction for the tweet\n",
    "        y_pred = sigmoid(np.dot(extract_features(tweet, freqs), theta))\n",
    "        \n",
    "        if y_pred > 0.5 :\n",
    "            # append 1.0 to the list\n",
    "            y_hat.append(1.0)\n",
    "        else:\n",
    "            # append 0 to the list\n",
    "            y_hat.append(0.0)\n",
    "\n",
    "    # Convert y_hat to a NumPy array for comparison with test_y\n",
    "    y_hat = np.array(y_hat)\n",
    "    \n",
    "    # Calculate accuracy\n",
    "    correct_predictions = (y_hat == np.squeeze(test_y)).sum()\n",
    "    total_predictions = len(test_y)\n",
    "    accuracy = correct_predictions / total_predictions\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic regression model's accuracy = 0.9950\n"
     ]
    }
   ],
   "source": [
    "tmp_accuracy = test_logistic_regression(test_x, test_y, freqs, theta)\n",
    "print(f\"Logistic regression model's accuracy = {tmp_accuracy:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Output: \n",
    "```0.9950```  \n",
    "Pretty good!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w1_unittest.unittest_test_logistic_regression(test_logistic_regression, freqs, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='5'></a>\n",
    "## 5 -  Error Analysis\n",
    "\n",
    "In this part you will see some tweets that your model misclassified. Why do you think the misclassifications happened? Specifically what kind of tweets does your model misclassify?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label Predicted Tweet\n",
      "THE TWEET IS: @MarkBreech Not sure it would be good thing 4 my bottom daring 2 say 2 Miss B but Im gonna be so stubborn on mouth soaping ! #NotHavingit :p\n",
      "THE PROCESSED TWEET IS: ['sure', 'would', 'good', 'thing', '4', 'bottom', 'dare', '2', 'say', '2', 'miss', 'b', 'im', 'gonna', 'stubborn', 'mouth', 'soap', 'nothavingit', ':p']\n",
      "1\t0.48901497\tb'sure would good thing 4 bottom dare 2 say 2 miss b im gonna stubborn mouth soap nothavingit :p'\n",
      "THE TWEET IS: I'm playing Brain Dots : ) #BrainDots\n",
      "http://t.co/UGQzOx0huu\n",
      "THE PROCESSED TWEET IS: [\"i'm\", 'play', 'brain', 'dot', 'braindot']\n",
      "1\t0.48418949\tb\"i'm play brain dot braindot\"\n",
      "THE TWEET IS: I'm playing Brain Dots : ) #BrainDots http://t.co/aOKldo3GMj http://t.co/xWCM9qyRG5\n",
      "THE PROCESSED TWEET IS: [\"i'm\", 'play', 'brain', 'dot', 'braindot']\n",
      "1\t0.48418949\tb\"i'm play brain dot braindot\"\n",
      "THE TWEET IS: I'm playing Brain Dots : ) #BrainDots http://t.co/R2JBO8iNww http://t.co/ow5BBwdEMY\n",
      "THE PROCESSED TWEET IS: [\"i'm\", 'play', 'brain', 'dot', 'braindot']\n",
      "1\t0.48418949\tb\"i'm play brain dot braindot\"\n",
      "THE TWEET IS: off to the park to get some sunlight : )\n",
      "THE PROCESSED TWEET IS: ['park', 'get', 'sunlight']\n",
      "1\t0.49636374\tb'park get sunlight'\n",
      "THE TWEET IS: @msarosh Uff Itna Miss karhy thy ap :p\n",
      "THE PROCESSED TWEET IS: ['uff', 'itna', 'miss', 'karhi', 'thi', 'ap', ':p']\n",
      "1\t0.48237069\tb'uff itna miss karhi thi ap :p'\n",
      "THE TWEET IS: @phenomyoutube u probs had more fun with david than me : (\n",
      "THE PROCESSED TWEET IS: ['u', 'prob', 'fun', 'david']\n",
      "0\t0.50988239\tb'u prob fun david'\n",
      "THE TWEET IS: pats jay : (\n",
      "THE PROCESSED TWEET IS: ['pat', 'jay']\n",
      "0\t0.50040365\tb'pat jay'\n",
      "THE TWEET IS: my beloved grandmother : ( https://t.co/wt4oXq5xCf\n",
      "THE PROCESSED TWEET IS: ['belov', 'grandmoth']\n",
      "0\t0.50000002\tb'belov grandmoth'\n",
      "THE TWEET IS: Sr. Financial Analyst - Expedia, Inc.: (#Bellevue, WA) http://t.co/ktknMhvwCI #Finance #ExpediaJobs #Job #Jobs #Hiring\n",
      "THE PROCESSED TWEET IS: ['sr', 'financi', 'analyst', 'expedia', 'inc', 'bellevu', 'wa', 'financ', 'expediajob', 'job', 'job', 'hire']\n",
      "0\t0.50648681\tb'sr financi analyst expedia inc bellevu wa financ expediajob job job hire'\n"
     ]
    }
   ],
   "source": [
    "# Some error analysis done for you\n",
    "print('Label Predicted Tweet')\n",
    "for x,y in zip(test_x,test_y):\n",
    "    y_hat = predict_tweet(x, freqs, theta)\n",
    "\n",
    "    if np.abs(y - (y_hat > 0.5)) > 0:\n",
    "        print('THE TWEET IS:', x)\n",
    "        print('THE PROCESSED TWEET IS:', process_tweet(x))\n",
    "        print('%d\\t%0.8f\\t%s' % (y, y_hat, ' '.join(process_tweet(x)).encode('ascii', 'ignore')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later in this specialization, we will see how we can use deeplearning to improve the prediction performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='6'></a>\n",
    "## 6 - Predict with your own Tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ridicul', 'bright', 'movi', 'plot', 'terribl', 'sad', 'end']\n",
      "[[0.48125423]]\n",
      "Negative sentiment\n"
     ]
    }
   ],
   "source": [
    "# Feel free to change the tweet below\n",
    "my_tweet = 'This is a ridiculously bright movie. The plot was terrible and I was sad until the ending!'\n",
    "print(process_tweet(my_tweet))\n",
    "y_hat = predict_tweet(my_tweet, freqs, theta)\n",
    "print(y_hat)\n",
    "if y_hat > 0.5:\n",
    "    print('Positive sentiment')\n",
    "else: \n",
    "    print('Negative sentiment')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}