1.2-introduction-to-sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hi sklearn!\n",
    "\n",
    "Diving into machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First challenge is [restaurants reviews](https://inclass.kaggle.com/c/restaurant-reviews) \n",
    "The goal of this competition is to learn predicting whether restaurant review is positive or negative.  \n",
    "\n",
    "### Reading restaurant  collection\n",
    "\n",
    "each line is a grade and text of review separated by tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\tThank you thank you thank you !! I  want to thank the people that made this place happen ....you have made all my dreams come true. Imagine a delicious yogurt shop with super fun flavors like peanut butter, chocolate mint, cake batter and so many more.  I used to have to travel to  Yogurtland or Jujuberry  but not anymore , now we have one right in the hood!! Guess what?   instead of eating a normal lunch I can pig out with a healthy peanut butter yogurt smothered in chocolate chips.   Could be the perfect lunch!!  See you there!\r\n",
      "5\tA Humane Society store at the Biltmore?  Interesting.  I had seen an adorable chihuahua mix at the Humane Society's webpage, and headed over to check out the little dog and the store.  They sell a variety of toys, treats, leashes, collars, and foods- in short, they sell a little bit of everything.  There's no tax, and if you adopt a pet from them they will give you 10% off the price as a thank you.  When I went there were 4 dogs up for adoption, a rabbit, and a wall of cats.  The little chihuahua mix was hanging out in the store with another little dog, so we got to play with them for a little bit.  The staff were friendly, and weren't pushy or hovering.  \\n\\nIn the end, we did end up adopting that little chihuahua mix from here.  He's the sweetest little dog ever.  Thank you Petique!\r\n",
      "1\tDon't buy Nike sneakers if you want to return or exchange them.. I've bought many from them. I went this week  and bought 2 pairs of Nike SB's that were already over retail price as it is.. I tried on one side then bought them... When I got home I realized that one of the shoes were damaged.. It's only been 2 days and they said they can't take it back.. I've bought many shoes there and this will be the last time... I'm from Nor Cal and people are crazy about sneakers out there and all the shops will accept a return or exchanged within 14- 30 days.. This was the first place that does not do that.. Why? Because sneakers and SB's are not big in AZ... They don't sell here.. I wouldn't buy them there if I were you.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 ./data/1-restaurant-train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import codecs\n",
    "# python kung fu.\n",
    "with codecs.open('./data/1-restaurant-train.csv') as f:\n",
    "    labels, reviews = zip(*[line.split('\\t') for line in f.readlines()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('4', '5', '1', '3', '5', '4', '5', '3', '5', '5')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Thank you thank you thank you !! I  want to thank the people that made this place happen ....you have made all my dreams come true. Imagine a delicious yogurt shop with super fun flavors like peanut butter, chocolate mint, cake batter and so many more.  I used to have to travel to  Yogurtland or Jujuberry  but not anymore , now we have one right in the hood!! Guess what?   instead of eating a normal lunch I can pig out with a healthy peanut butter yogurt smothered in chocolate chips.   Could be the perfect lunch!!  See you there!\\n',\n",
       " \"A Humane Society store at the Biltmore?  Interesting.  I had seen an adorable chihuahua mix at the Humane Society's webpage, and headed over to check out the little dog and the store.  They sell a variety of toys, treats, leashes, collars, and foods- in short, they sell a little bit of everything.  There's no tax, and if you adopt a pet from them they will give you 10% off the price as a thank you.  When I went there were 4 dogs up for adoption, a rabbit, and a wall of cats.  The little chihuahua mix was hanging out in the store with another little dog, so we got to play with them for a little bit.  The staff were friendly, and weren't pushy or hovering.  \\\\n\\\\nIn the end, we did end up adopting that little chihuahua mix from here.  He's the sweetest little dog ever.  Thank you Petique!\\n\")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### read test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with codecs.open('./data/1-restaurant-test.csv') as f:\n",
    "    kaggle_test_reviews = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"My son just loves this place.  Weird that he'd ask to come here everytime we go grocery shopping (bribe) and not even care to go to Toys R Us.  Not complaining.   I'm not into little knick knacks, but they have quite a selection on little travel toys, educational materials for kids and holiday stuff.  I bought a couple of red bows with brass jingles on it and wreaths to put on my porch lights for $4!   Why is it that it's so cheap, but you can end up spending $50?\\n\",\n",
       " '\"We gave it a 9, so we will make that 5-, 4,5 stars. \\\\n   To start with it\\'s just beautiful and we lucked out to be outside, under the heater next to the roaring fireplace. The service could not have been better and thanks to our YELPing friends, we hardly needed a menu. \\\\n  The Portugese clam soup was \\\\souper\\\\\"\" though on the salty side. The pork belly was \\\\\"\"off the hook\\\\\"\" and the steak tacos were tops too! We gave the dishes, 9, 9.5 and 9.5 respectivley.\\\\n  Margaritas were awesome, Smokehouse and Pomogranite but it was a short glass.    \\\\n  It\\'s been a long time since we had a solid 9. Thanks guys! Oh and the table guac is a must. PS if you get he bread pudding, get the sauce on the side. It\\'s over the top sweet. My pancreas is running laps right now!\"\"\"\\n']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kaggle_test_reviews[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### prepare solution\n",
    "\n",
    "define useful supplementary function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "from IPython.display import FileLink\n",
    "\n",
    "def create_solution(predictions, filename='1-restaurant-predictions.csv'):\n",
    "    result = pandas.DataFrame({'Id': numpy.arange(len(predictions)), 'Solution': predictions})\n",
    "    result.to_csv('data/{}'.format(filename), index=False)\n",
    "    return FileLink('data/{}'.format(filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute simple statistics in the reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compute_data_expressions(reviews):\n",
    "    features = []\n",
    "    # length of each string\n",
    "    features.append(map(len, reviews))\n",
    "    \n",
    "    # number of letters, digits, spaces = words\n",
    "    for pattern in [str.isalpha, str.isdigit, str.isspace]:\n",
    "        features.append(map(lambda review: sum(map(pattern, review)), reviews))\n",
    "        \n",
    "    features = numpy.array(features).T\n",
    "    return features\n",
    "\n",
    "features = compute_data_expressions(reviews)\n",
    "kaggle_test_features = compute_data_expressions(kaggle_test_reviews)\n",
    "\n",
    "# convert labels to int values\n",
    "labels = map(int, labels)\n",
    "\n",
    "# Making problem simpler: convert to positive/negative reviews. \n",
    "answers = numpy.array(labels) >= 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True, False, ...,  True, False,  True], dtype=bool)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 536,  411,    0,  107],\n",
       "       [ 796,  603,    3,  159],\n",
       "       [ 720,  534,    6,  143],\n",
       "       ..., \n",
       "       [1092,  835,    9,  214],\n",
       "       [ 574,  437,    2,  116],\n",
       "       [ 637,  488,    4,  105]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification quality measure — ROC curve and area under the curve (AUC)\n",
    "[Explanation of the ROC curve](http://arogozhnikov.github.io/RocCurve.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# area under the roc curve\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99376260608836542"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=1)\n",
    "knn_clf.fit(features, answers) # train an algorithm\n",
    "roc_auc_score(answers, knn_clf.predict_proba(features)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/1-restaurant-predictions.csv' target='_blank'>data/1-restaurant-predictions.csv</a><br>"
      ],
      "text/plain": [
       "/Users/antares/Yandex.Disk.localized/2017-01-Imperial/data/1-restaurant-predictions.csv"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_solution(knn_clf.predict_proba(kaggle_test_features)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send it to kaggle and check that its score differs from the above value _significantly_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross validation / overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/antares/.virtualenvs/rep/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "trainX, testX, trainY, testY = train_test_split(features, answers, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kaggle challenges \n",
    "\n",
    "You'll participate in challenges, where your model performance will be assessed by kaggle using the predictions on the unlabeled data.\n",
    "\n",
    "It is an important practice, first challenge is [here](https://inclass.kaggle.com/c/restaurant-reviews). \n",
    "\n",
    "First kaggle is due to 30 Jan."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Knn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test 0.512878045192\n",
      "train 0.995269637289\n"
     ]
    }
   ],
   "source": [
    "knn_clf = KNeighborsClassifier(n_neighbors=1)\n",
    "knn_clf.fit(trainX, trainY)\n",
    "print 'test', roc_auc_score(testY, knn_clf.predict_proba(testX)[:, 1])\n",
    "print 'train', roc_auc_score(trainY, knn_clf.predict_proba(trainX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "situation above is called 'overfitting'\n",
    "\n",
    "## Sidenote: sklearn interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.33333333,  0.66666667],\n",
       "       [ 0.66666667,  0.33333333],\n",
       "       [ 0.33333333,  0.66666667],\n",
       "       ..., \n",
       "       [ 1.        ,  0.        ],\n",
       "       [ 0.66666667,  0.33333333],\n",
       "       [ 0.33333333,  0.66666667]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# work with scikit-learn models in simplest case consists of \n",
    "# 1. defining model with parameters\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=3)\n",
    "# 2. training (method fit, X of shape [n_samples, n_features], target y of shape [n_samples])\n",
    "knn_clf.fit(trainX, trainY)\n",
    "# 3. predicting (predict probabilities for classification)\n",
    "knn_clf.predict_proba(testX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding optimal number of neighbours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 0.512878045192\n",
      "2 0.523998599836\n",
      "4 0.53293147584\n",
      "8 0.541474487911\n",
      "16 0.552905570616\n",
      "32 0.56343027432\n",
      "64 0.572396762402\n"
     ]
    }
   ],
   "source": [
    "for n_neighbors in [1, 2, 4, 8, 16, 32, 64]:\n",
    "    knn_clf = KNeighborsClassifier(n_neighbors=n_neighbors)\n",
    "    knn_clf.fit(trainX, trainY)\n",
    "    print n_neighbors, roc_auc_score(testY, knn_clf.predict_proba(testX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Bag of words](https://en.wikipedia.org/wiki/Bag-of-words_model)\n",
    "\n",
    "Simplest way to represent the text is to count number of times each word is met"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# take the 100 the most frequent words\n",
    "vectorizer = CountVectorizer(max_features=100)\n",
    "vectorizer.fit(reviews)\n",
    "counts = vectorizer.transform(reviews).toarray()\n",
    "kaggle_test_counts = vectorizer.transform(kaggle_test_reviews).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(82065, 100)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 1, ..., 0, 5, 0],\n",
       "       [0, 0, 0, ..., 0, 4, 0],\n",
       "       [1, 0, 1, ..., 0, 2, 0],\n",
       "       ..., \n",
       "       [1, 0, 1, ..., 0, 0, 0],\n",
       "       [1, 0, 1, ..., 0, 1, 0],\n",
       "       [0, 0, 2, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vocabulary is a dictionary which keeps correspondence between columns and words\n",
    "# vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_counts, test_counts, train_labels, test_labels = train_test_split(counts, answers, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "\n",
    "Naive Bayes will be explained later in the lectures. It is a generative model.\n",
    "\n",
    "Naive explanation: we compute for each word probability to appear in positive and negative review. To assess a new review, this information is combined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bernoulli\n",
    "uses only information about presence of words in the text (count is zero or not) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.69277446873488746"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import BernoulliNB\n",
    "nb_clf = BernoulliNB()\n",
    "nb_clf.fit(train_counts, train_labels)\n",
    "roc_auc_score(test_labels, nb_clf.predict_proba(test_counts)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### multinomial\n",
    "\n",
    "... also pays attention to counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73529071403854918"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "nb_clf = MultinomialNB()\n",
    "nb_clf.fit(train_counts, train_labels)\n",
    "roc_auc_score(test_labels, nb_clf.predict_proba(test_counts)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61548, 100)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_counts.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/1-restaurant-predictions-nb.csv' target='_blank'>data/1-restaurant-predictions-nb.csv</a><br>"
      ],
      "text/plain": [
       "/Users/antares/Yandex.Disk.localized/2017-01-Imperial/data/1-restaurant-predictions-nb.csv"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_solution(nb_clf.predict_proba(kaggle_test_counts)[:, 1], filename='1-restaurant-predictions-nb.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regression + Ridge regularization\n",
    "\n",
    "Linear model is a very simple approximation:\n",
    "$$\\hat{y_i}= \\theta_0 + \\sum_j \\theta_j x_i^j $$\n",
    "where $x_i^j$ is a value for $j$-th feature for $i$-th sample, $\\theta_j$ — is a parameter to find for $j$-th feature, $\\hat{y_i}$ — prediction of linear model for $i$-th sample.\n",
    "\n",
    "And we can introduce the loss function (how our approximation is far from the true values). For example:\n",
    "$$\\mathcal{L} = \\sum_i (y_i - \\hat{y}_i)^2 \\to \\min$$\n",
    "(widely known as ordinary least squares)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.787431742938\n",
      "0.792064902203\n"
     ]
    }
   ],
   "source": [
    "ridge_clf = Ridge()\n",
    "ridge_clf.fit(train_counts, train_labels)\n",
    "# use `predict` method for regression methods to evaluate function for new data\n",
    "print roc_auc_score(test_labels,  ridge_clf.predict(test_counts))\n",
    "print roc_auc_score(train_labels, ridge_clf.predict(train_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework (due to 26 Jan, 9AM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #0. (1 point)**. Play with regularization parameter of RidgeRegression, see how it affects quality on train and test.\n",
    "Check quality of best model by submitting to kaggle.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# take the 30000 the most frequent words, and use 5000 samples in training\n",
    "vectorizer_reg = CountVectorizer(max_features=30000)\n",
    "vectorizer_reg.fit(reviews)\n",
    "counts_reg = vectorizer_reg.transform(reviews)\n",
    "train_counts_reg, test_counts_reg, train_labels_reg, test_labels_reg = \\\n",
    "    train_test_split(counts_reg, answers, random_state=42, train_size=5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.834342755226\n"
     ]
    }
   ],
   "source": [
    "ridge_regularization = Ridge(alpha=0.01)\n",
    "ridge_regularization.fit(train_counts_reg, train_labels_reg)\n",
    "print roc_auc_score(test_labels_reg,  ridge_regularization.predict(test_counts_reg))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# play with regularization here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise #1. (1 point)** Let's write the correspondence between columns and words (done below). Which words are most popular?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dictionary = numpy.empty(len(vectorizer.vocabulary_), dtype='O')\n",
    "for word, index in vectorizer.vocabulary_.iteritems():\n",
    "    dictionary[index] = word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([u'about', u'after', u'all', u'also', u'always', u'an', u'and',\n",
       "       u'are', u'as', u'at', u'back', u'be', u'because', u'been', u'but',\n",
       "       u'by', u'can', u'chicken', u'could', u'do', u'don', u'even',\n",
       "       u'food', u'for', u'from', u'get', u'go', u'good', u'got', u'great',\n",
       "       u'had', u'has', u'have', u'he', u'here', u'if', u'in', u'is', u'it',\n",
       "       u'just', u'like', u'little', u'love', u'me', u'menu', u'more',\n",
       "       u'much', u'my', u'ni', u'nice', u'no', u'not', u'nthe', u'of',\n",
       "       u'on', u'one', u'only', u'or', u'order', u'ordered', u'other',\n",
       "       u'our', u'out', u'people', u'place', u'pretty', u'really',\n",
       "       u'restaurant', u'service', u'she', u'so', u'some', u'than', u'that',\n",
       "       u'the', u'their', u'them', u'there', u'they', u'this', u'time',\n",
       "       u'to', u'too', u'try', u'up', u'us', u've', u'very', u'was', u'we',\n",
       "       u'well', u'were', u'what', u'when', u'which', u'will', u'with',\n",
       "       u'would', u'you', u'your'], dtype=object)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #2. (1 point) ** By analyzing coefficients in `ridge_clf.coef_`, determine which words have the highest impact on decision (= have the largest modulus of `coef_`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #3. (2 points) **  Does combining features and counts improve quality? Use `numpy.hstack` to concatenate arrays.\n",
    "Explain the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #4. (2 points)** Print examples on which your classifier makes mistakes (both false positive and false negative).\n",
    "\n",
    "This is important step to understand what can be done to improve the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #5. (optional, just for fun)**  Write a restaurant review, which is misunderstood by your best model. \n",
    "Something like \"Hate each time I'm not eating here\".\n",
    "\n",
    "Use your knowledge about the structure of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Major Goal in kaggle competition (not in this homework). ** Provide best classification model for the problem. \n",
    "\n",
    "You can start with computing new features:\n",
    "1. Computing occurrences of symbols (like \"!\"; \":)\", etc.)\n",
    "2. Ignoring the stop-words, words with digits, etc.\n",
    "3. Adding more words in the model bag of words \n",
    "4. words \"likes\", \"liked\", \"like\", \"likely\" all have \"like\" in it. You can use this information!\n",
    "5. test your ideas\n",
    "\n",
    "Or start with changing parameters of classifiers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Completed? \n",
    "\n",
    "\n",
    "Rename the notebook to `1.2-Surname-sklearn.ipynb`, download (`File > Download as .ipynb`) and send to `icl.ml@yandex.ru`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}