NLP for Text Classification (Jupyter Notebook).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing for Text Classification with NLTK and Scikit-learn\n",
    "\n",
    "### Presented by Eduonix!\n",
    "\n",
    "In the project, Getting Started With Natural Language Processing in Python, we learned the basics of tokenizing, part-of-speech tagging, stemming, chunking, and named entity recognition; furthermore, we dove into machine learning and text classification using a simple support vector classifier and a dataset of positive and negative movie reviews. \n",
    "\n",
    "In this tutorial, we will expand on this foundation and explore different ways to improve our text classification results. We will cover and use:\n",
    "\n",
    "* Regular Expressions\n",
    "* Feature Engineering\n",
    "* Multiple scikit-learn Classifiers\n",
    "* Ensemble Methods\n",
    "\n",
    "### 1. Import Necessary Libraries\n",
    "\n",
    "To ensure the necessary libraries are installed correctly and up-to-date, print the version numbers for each library.  This will also improve the reproducibility of our project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python: 2.7.13 |Continuum Analytics, Inc.| (default, May 11 2017, 13:17:26) [MSC v.1500 64 bit (AMD64)]\n",
      "NLTK: 3.2.5\n",
      "Scikit-learn: 0.19.1\n",
      "Pandas: 0.21.0\n",
      "Numpy: 1.14.1\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import nltk\n",
    "import sklearn\n",
    "import pandas\n",
    "import numpy\n",
    "\n",
    "print('Python: {}'.format(sys.version))\n",
    "print('NLTK: {}'.format(nltk.__version__))\n",
    "print('Scikit-learn: {}'.format(sklearn.__version__))\n",
    "print('Pandas: {}'.format(pandas.__version__))\n",
    "print('Numpy: {}'.format(numpy.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load the Dataset\n",
    "\n",
    "Now that we have ensured that our libraries are installed correctly, let's load the data set as a Pandas DataFrame. Furthermore, let's extract some useful information such as the column information and class distributions. \n",
    "\n",
    "The data set we will be using comes from the UCI Machine Learning Repository.  It contains over 5000 SMS labeled messages that have been collected for mobile phone spam research. It can be downloaded from the following URL:\n",
    "\n",
    "https://archive.ics.uci.edu/ml/datasets/sms+spam+collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# load the dataset of SMS messages\n",
    "df = pd.read_table('SMSSPamCollection', header=None, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 5572 entries, 0 to 5571\n",
      "Data columns (total 2 columns):\n",
      "0    5572 non-null object\n",
      "1    5572 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 87.1+ KB\n",
      "None\n",
      "      0                                                  1\n",
      "0   ham  Go until jurong point, crazy.. Available only ...\n",
      "1   ham                      Ok lar... Joking wif u oni...\n",
      "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
      "3   ham  U dun say so early hor... U c already then say...\n",
      "4   ham  Nah I don't think he goes to usf, he lives aro...\n"
     ]
    }
   ],
   "source": [
    "# print useful information about the dataset\n",
    "print(df.info())\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ham     4825\n",
      "spam     747\n",
      "Name: 0, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# check class distribution\n",
    "classes = df[0]\n",
    "print(classes.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Preprocess the Data\n",
    "\n",
    "Preprocessing the data is an essential step in natural language process. In the following cells, we will convert our class labels to binary values using the LabelEncoder from sklearn, replace email addresses, URLs, phone numbers, and other symbols by using regular expressions, remove stop words, and extract word stems.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 1 0 0 1 0 0 1 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# convert class labels to binary values, 0 = ham and 1 = spam\n",
    "encoder = LabelEncoder()\n",
    "Y = encoder.fit_transform(classes)\n",
    "\n",
    "print(Y[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    Go until jurong point, crazy.. Available only ...\n",
      "1                        Ok lar... Joking wif u oni...\n",
      "2    Free entry in 2 a wkly comp to win FA Cup fina...\n",
      "3    U dun say so early hor... U c already then say...\n",
      "4    Nah I don't think he goes to usf, he lives aro...\n",
      "5    FreeMsg Hey there darling it's been 3 week's n...\n",
      "6    Even my brother is not like to speak with me. ...\n",
      "7    As per your request 'Melle Melle (Oru Minnamin...\n",
      "8    WINNER!! As a valued network customer you have...\n",
      "9    Had your mobile 11 months or more? U R entitle...\n",
      "Name: 1, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# store the SMS message data\n",
    "text_messages = df[1]\n",
    "print(text_messages[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Regular Expressions\n",
    "\n",
    "Some common regular expression metacharacters - copied from wikipedia\n",
    "\n",
    "**^**     Matches the starting position within the string. In line-based tools, it matches the starting position of any line.\n",
    "\n",
    "**.**     Matches any single character (many applications exclude newlines, and exactly which characters are considered newlines is flavor-, character-encoding-, and platform-specific, but it is safe to assume that the line feed character is included). Within POSIX bracket expressions, the dot character matches a literal dot. For example, a.c matches \"abc\", etc., but [a.c] matches only \"a\", \".\", or \"c\".\n",
    "\n",
    "**[ ]**    A bracket expression. Matches a single character that is contained within the brackets. For example, [abc] matches \"a\", \"b\", or \"c\". [a-z] specifies a range which matches any lowercase letter from \"a\" to \"z\". These forms can be mixed: [abcx-z] matches \"a\", \"b\", \"c\", \"x\", \"y\", or \"z\", as does [a-cx-z].\n",
    "The - character is treated as a literal character if it is the last or the first (after the ^, if present) character within the brackets: [abc-], [-abc]. Note that backslash escapes are not allowed. The ] character can be included in a bracket expression if it is the first (after the ^) character: []abc].\n",
    "\n",
    "**[^ ]**   Matches a single character that is not contained within the brackets. For example, [^abc] matches any character other than \"a\", \"b\", or \"c\". [^a-z] matches any single character that is not a lowercase letter from \"a\" to \"z\". Likewise, literal characters and ranges can be mixed.\n",
    "\n",
    "**$**      Matches the ending position of the string or the position just before a string-ending newline. In line-based tools, it matches the ending position of any line.\n",
    "\n",
    "**( )**    Defines a marked subexpression. The string matched within the parentheses can be recalled later (see the next entry, \\n). A marked subexpression is also called a block or capturing group. BRE mode requires \\( \\).\n",
    "\n",
    "**\\n**     Matches what the nth marked subexpression matched, where n is a digit from 1 to 9. This construct is vaguely defined in the POSIX.2 standard. Some tools allow referencing more than nine capturing groups.\n",
    "\n",
    "**\\***     Matches the preceding element zero or more times. For example, ab*c matches \"ac\", \"abc\", \"abbbc\", etc. [xyz]* matches \"\", \"x\", \"y\", \"z\", \"zx\", \"zyx\", \"xyzzy\", and so on. (ab)* matches \"\", \"ab\", \"abab\", \"ababab\", and so on.\n",
    "\n",
    "**{m,n}**  Matches the preceding element at least m and not more than n times. For example, a{3,5} matches only \"aaa\", \"aaaa\", and \"aaaaa\". This is not found in a few older instances of regexes. BRE mode requires \\{m,n\\}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use regular expressions to replace email addresses, URLs, phone numbers, other numbers\n",
    "\n",
    "# Replace email addresses with 'email'\n",
    "processed = text_messages.str.replace(r'^.+@[^\\.].*\\.[a-z]{2,}$',\n",
    "                                 'emailaddress')\n",
    "\n",
    "# Replace URLs with 'webaddress'\n",
    "processed = processed.str.replace(r'^http\\://[a-zA-Z0-9\\-\\.]+\\.[a-zA-Z]{2,3}(/\\S*)?$',\n",
    "                                  'webaddress')\n",
    "\n",
    "# Replace money symbols with 'moneysymb' (£ can by typed with ALT key + 156)\n",
    "processed = processed.str.replace(r'£|\\$', 'moneysymb')\n",
    "    \n",
    "# Replace 10 digit phone numbers (formats include paranthesis, spaces, no spaces, dashes) with 'phonenumber'\n",
    "processed = processed.str.replace(r'^\\(?[\\d]{3}\\)?[\\s-]?[\\d]{3}[\\s-]?[\\d]{4}$',\n",
    "                                  'phonenumbr')\n",
    "    \n",
    "# Replace numbers with 'numbr'\n",
    "processed = processed.str.replace(r'\\d+(\\.\\d+)?', 'numbr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove punctuation\n",
    "processed = processed.str.replace(r'[^\\w\\d\\s]', ' ')\n",
    "\n",
    "# Replace whitespace between terms with a single space\n",
    "processed = processed.str.replace(r'\\s+', ' ')\n",
    "\n",
    "# Remove leading and trailing whitespace\n",
    "processed = processed.str.replace(r'^\\s+|\\s+?$', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0       go until jurong point crazy available only in ...\n",
      "1                                 ok lar joking wif u oni\n",
      "2       free entry in numbr a wkly comp to win fa cup ...\n",
      "3             u dun say so early hor u c already then say\n",
      "4       nah i don t think he goes to usf he lives arou...\n",
      "5       freemsg hey there darling it s been numbr week...\n",
      "6       even my brother is not like to speak with me t...\n",
      "7       as per your request melle melle oru minnaminun...\n",
      "8       winner as a valued network customer you have b...\n",
      "9       had your mobile numbr months or more u r entit...\n",
      "10      i m gonna be home soon and i don t want to tal...\n",
      "11      six chances to win cash from numbr to numbr nu...\n",
      "12      urgent you have won a numbr week free membersh...\n",
      "13      i ve been searching for the right words to tha...\n",
      "14                      i have a date on sunday with will\n",
      "15      xxxmobilemovieclub to use your credit click th...\n",
      "16                                 oh k i m watching here\n",
      "17      eh u remember how numbr spell his name yes i d...\n",
      "18      fine if that s the way u feel that s the way i...\n",
      "19      england v macedonia dont miss the goals team n...\n",
      "20               is that seriously how you spell his name\n",
      "21      i m going to try for numbr months ha ha only j...\n",
      "22           so pay first lar then when is da stock comin\n",
      "23      aft i finish my lunch then i go str down lor a...\n",
      "24      ffffffffff alright no way i can meet up with y...\n",
      "25      just forced myself to eat a slice i m really n...\n",
      "26                          lol your always so convincing\n",
      "27      did you catch the bus are you frying an egg di...\n",
      "28      i m back amp we re packing the car now i ll le...\n",
      "29      ahhh work i vaguely remember that what does it...\n",
      "                              ...                        \n",
      "5542             armand says get your ass over to epsilon\n",
      "5543                u still havent got urself a jacket ah\n",
      "5544    i m taking derek amp taylor to walmart if i m ...\n",
      "5545        hi its in durban are you still on this number\n",
      "5546             ic there are a lotta childporn cars then\n",
      "5547    had your contract mobile numbr mnths latest mo...\n",
      "5548                     no i was trying it all weekend v\n",
      "5549    you know wot people wear t shirts jumpers hat ...\n",
      "5550            cool what time you think you can get here\n",
      "5551    wen did you get so spiritual and deep that s g...\n",
      "5552    have a safe trip to nigeria wish you happiness...\n",
      "5553                           hahaha use your brain dear\n",
      "5554    well keep in mind i ve only got enough gas for...\n",
      "5555    yeh indians was nice tho it did kane me off a ...\n",
      "5556    yes i have so that s why u texted pshew missin...\n",
      "5557    no i meant the calculation is the same that lt...\n",
      "5558                                sorry i ll call later\n",
      "5559    if you aren t here in the next lt gt hours imm...\n",
      "5560                      anything lor juz both of us lor\n",
      "5561    get me out of this dump heap my mom decided to...\n",
      "5562    ok lor sony ericsson salesman i ask shuhui the...\n",
      "5563                               ard numbr like dat lor\n",
      "5564    why don t you wait til at least wednesday to s...\n",
      "5565                                            huh y lei\n",
      "5566    reminder from onumbr to get numbr pounds free ...\n",
      "5567    this is the numbrnd time we have tried numbr c...\n",
      "5568                    will b going to esplanade fr home\n",
      "5569    pity was in mood for that so any other suggest...\n",
      "5570    the guy did some bitching but i acted like i d...\n",
      "5571                            rofl its true to its name\n",
      "Name: 1, Length: 5572, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# change words to lower case - Hello, HELLO, hello are all the same word\n",
    "processed = processed.str.lower()\n",
    "print(processed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "# remove stop words from text messages\n",
    "\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "processed = processed.apply(lambda x: ' '.join(\n",
    "    term for term in x.split() if term not in stop_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove word stems using a Porter stemmer\n",
    "ps = nltk.PorterStemmer()\n",
    "\n",
    "processed = processed.apply(lambda x: ' '.join(\n",
    "    ps.stem(term) for term in x.split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Generating Features\n",
    "\n",
    "Feature engineering is the process of using domain knowledge of the data to create features for machine learning algorithms. In this project, the words in each text message will be our features.  For this purpose, it will be necessary to tokenize each word.  We will use the 1500 most common words as features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# create bag-of-words\n",
    "all_words = []\n",
    "\n",
    "for message in processed:\n",
    "    words = word_tokenize(message)\n",
    "    for w in words:\n",
    "        all_words.append(w)\n",
    "        \n",
    "all_words = nltk.FreqDist(all_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words: 6562\n",
      "Most common words: [(u'numbr', 2961), (u'u', 1207), (u'call', 679), (u'go', 456), (u'get', 451), (u'ur', 391), (u'gt', 318), (u'lt', 316), (u'come', 304), (u'ok', 293), (u'free', 284), (u'day', 276), (u'know', 275), (u'love', 266), (u'like', 261)]\n"
     ]
    }
   ],
   "source": [
    "# print the total number of words and the 15 most common words\n",
    "print('Number of words: {}'.format(len(all_words)))\n",
    "print('Most common words: {}'.format(all_words.most_common(15)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the 1500 most common words as features\n",
    "word_features = list(all_words.keys())[:1500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "avail\n",
      "buffet\n",
      "world\n",
      "great\n"
     ]
    }
   ],
   "source": [
    "# The find_features function will determine which of the 1500 word features are contained in the review\n",
    "def find_features(message):\n",
    "    words = word_tokenize(message)\n",
    "    features = {}\n",
    "    for word in word_features:\n",
    "        features[word] = (word in words)\n",
    "\n",
    "    return features\n",
    "\n",
    "# Lets see an example!\n",
    "features = find_features(processed[0])\n",
    "for key, value in features.items():\n",
    "    if value == True:\n",
    "        print key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now lets do it for all the messages\n",
    "messages = zip(processed, Y)\n",
    "\n",
    "# define a seed for reproducibility\n",
    "seed = 1\n",
    "np.random.seed = seed\n",
    "np.random.shuffle(messages)\n",
    "\n",
    "# call find_features function for each SMS message\n",
    "featuresets = [(find_features(text), label) for (text, label) in messages]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can split the featuresets into training and testing datasets using sklearn\n",
    "from sklearn import model_selection\n",
    "\n",
    "# split the data into training and testing datasets\n",
    "training, testing = model_selection.train_test_split(featuresets, test_size = 0.25, random_state=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4179\n",
      "1393\n"
     ]
    }
   ],
   "source": [
    "print(len(training))\n",
    "print(len(testing))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Scikit-Learn Classifiers with NLTK\n",
    "\n",
    "Now that we have our dataset, we can start building algorithms! Let's start with a simple linear support vector classifier, then expand to other algorithms. We'll need to import each algorithm we plan on using from sklearn.  We also need to import some performance metrics, such as accuracy_score and classification_report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVC Accuracy: 96.1234745154\n"
     ]
    }
   ],
   "source": [
    "# We can use sklearn algorithms in NLTK\n",
    "from nltk.classify.scikitlearn import SklearnClassifier\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "model = SklearnClassifier(SVC(kernel = 'linear'))\n",
    "\n",
    "# train the model on the training data\n",
    "model.train(training)\n",
    "\n",
    "# and test on the testing dataset!\n",
    "accuracy = nltk.classify.accuracy(model, testing)*100\n",
    "print(\"SVC Accuracy: {}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K Nearest Neighbors Accuracy: 94.0416367552\n",
      "Decision Tree Accuracy: 95.2620244078\n",
      "Random Forest Accuracy: 95.6927494616\n",
      "Logistic Regression Accuracy: 95.9798994975\n",
      "SGD Classifier Accuracy: 95.9798994975\n",
      "Naive Bayes Accuracy: 96.2670495334\n",
      "SVM Linear Accuracy: 96.1234745154\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression, SGDClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import classification_report, accuracy_score, confusion_matrix\n",
    "\n",
    "# Define models to train\n",
    "names = [\"K Nearest Neighbors\", \"Decision Tree\", \"Random Forest\", \"Logistic Regression\", \"SGD Classifier\",\n",
    "         \"Naive Bayes\", \"SVM Linear\"]\n",
    "\n",
    "classifiers = [\n",
    "    KNeighborsClassifier(),\n",
    "    DecisionTreeClassifier(),\n",
    "    RandomForestClassifier(),\n",
    "    LogisticRegression(),\n",
    "    SGDClassifier(max_iter = 100),\n",
    "    MultinomialNB(),\n",
    "    SVC(kernel = 'linear')\n",
    "]\n",
    "\n",
    "models = zip(names, classifiers)\n",
    "\n",
    "for name, model in models:\n",
    "    nltk_model = SklearnClassifier(model)\n",
    "    nltk_model.train(training)\n",
    "    accuracy = nltk.classify.accuracy(nltk_model, testing)*100\n",
    "    print(\"{} Accuracy: {}\".format(name, accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Voting Classifier: Accuracy: 96.1234745154\n"
     ]
    }
   ],
   "source": [
    "# Ensemble methods - Voting classifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "names = [\"K Nearest Neighbors\", \"Decision Tree\", \"Random Forest\", \"Logistic Regression\", \"SGD Classifier\",\n",
    "         \"Naive Bayes\", \"SVM Linear\"]\n",
    "\n",
    "classifiers = [\n",
    "    KNeighborsClassifier(),\n",
    "    DecisionTreeClassifier(),\n",
    "    RandomForestClassifier(),\n",
    "    LogisticRegression(),\n",
    "    SGDClassifier(max_iter = 100),\n",
    "    MultinomialNB(),\n",
    "    SVC(kernel = 'linear')\n",
    "]\n",
    "\n",
    "models = zip(names, classifiers)\n",
    "\n",
    "nltk_ensemble = SklearnClassifier(VotingClassifier(estimators = models, voting = 'hard', n_jobs = -1))\n",
    "nltk_ensemble.train(training)\n",
    "accuracy = nltk.classify.accuracy(nltk_model, testing)*100\n",
    "print(\"Voting Classifier: Accuracy: {}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make class label prediction for testing set\n",
    "txt_features, labels = zip(*testing)\n",
    "\n",
    "prediction = nltk_ensemble.classify_many(txt_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.96      0.99      0.98      1214\n",
      "          1       0.92      0.75      0.83       179\n",
      "\n",
      "avg / total       0.96      0.96      0.96      1393\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">predicted</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>ham</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">actual</th>\n",
       "      <th>ham</th>\n",
       "      <td>1203</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spam</th>\n",
       "      <td>45</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            predicted     \n",
       "                  ham spam\n",
       "actual ham       1203   11\n",
       "       spam        45  134"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print a confusion matrix and a classification report\n",
    "print(classification_report(labels, prediction))\n",
    "\n",
    "pd.DataFrame(\n",
    "    confusion_matrix(labels, prediction),\n",
    "    index = [['actual', 'actual'], ['ham', 'spam']],\n",
    "    columns = [['predicted', 'predicted'], ['ham', 'spam']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}