9_Assignment_DT_Instructions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jKMeoHjWN-3k"
   },
   "source": [
    "# Assignment 8: DT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ACUkHex3N-3m"
   },
   "source": [
    "<ol>\n",
    "    <li><strong>Apply Decision Tree Classifier(DecisionTreeClassifier) on these feature sets</strong>\n",
    "        <ul>\n",
    "            <li><font color='red'>Set 1</font>: categorical, numerical features + project_title(TFIDF)+  preprocessed_eassay (TFIDF)</li>\n",
    "            <li><font color='red'>Set 2</font>: categorical, numerical features + project_title(TFIDF W2V)+  preprocessed_eassay (TFIDF W2V)</li>        </ul>\n",
    "    </li>\n",
    "    <li><strong>The hyper paramter tuning (best `depth` in range [1, 5, 10, 50], and the best `min_samples_split` in range [5, 10, 100, 500])</strong>\n",
    "        <ul>\n",
    "    <li>Find the best hyper parameter which will give the maximum <a href='https://www.appliedaicourse.com/course/applied-ai-course-online/lessons/receiver-operating-characteristic-curve-roc-curve-and-auc-1/'>AUC</a> value</li>\n",
    "    <li>find the best hyper paramter using k-fold cross validation(use gridsearch cv or randomsearch cv)/simple cross validation data(you can write your own for loops refer sample solution)</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    <li>\n",
    "    <strong>Representation of results</strong>\n",
    "        <ul>\n",
    "    <li>You need to plot the performance of model both on train data and cross validation data for each hyper parameter, like shown in the figure\n",
    "    <img src='https://i.imgur.com/Gp2DQmh.jpg' width=500px> with X-axis as <strong>min_sample_split</strong>, Y-axis as <strong>max_depth</strong>, and Z-axis as <strong>AUC Score</strong> , we have given the notebook which explains how to plot this 3d plot, you can find it in the same drive <i>3d_scatter_plot.ipynb</i></li>\n",
    "            <p style=\"text-align:center;font-size:30px;color:red;\"><strong>or</strong></p> <br>\n",
    "    <li>You need to plot the performance of model both on train data and cross validation data for each hyper parameter, like shown in the figure\n",
    "    <img src='https://i.imgur.com/fgN9aUP.jpg' width=300px> <a href='https://seaborn.pydata.org/generated/seaborn.heatmap.html'>seaborn heat maps</a> with rows as <strong>n_estimators</strong>, columns as <strong>max_depth</strong>, and values inside the cell representing <strong>AUC Score</strong> </li>\n",
    "    <li>You choose either of the plotting techniques out of 3d plot or heat map</li>\n",
    "    <li>Once after you found the best hyper parameter, you need to train your model with it, and find the AUC on test data and plot the ROC curve on both train and test.\n",
    "    <img src='https://i.imgur.com/wMQDTFe.jpg' width=300px></li>\n",
    "    <li>Along with plotting ROC curve, you need to print the <a href='https://www.appliedaicourse.com/course/applied-ai-course-online/lessons/confusion-matrix-tpr-fpr-fnr-tnr-1/'>confusion matrix</a> with predicted and original labels of test data points\n",
    "    <img src='https://i.imgur.com/IdN5Ctv.png' width=300px></li>\n",
    "    <li>Once after you plot the confusion matrix with the test data, get all the `false positive data points`\n",
    "        <ul>\n",
    "            <li> Plot the WordCloud(https://www.geeksforgeeks.org/generating-word-cloud-python/) with the words of essay text of these `false positive data points`</li>\n",
    "            <li> Plot the box plot with the `price` of these `false positive data points`</li>\n",
    "            <li> Plot the pdf with the `teacher_number_of_previously_posted_projects` of these `false positive data points`</li>\n",
    "        </ul>\n",
    "        </ul>\n",
    "    </li>\n",
    "   <li><b>Task 2: </b>For this task consider set-1 features. Select all the features which are having non-zero feature importance.You can get the feature importance using  'feature_importances_` \n",
    "   (https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html), discard the all other remaining features and then apply any of the model of you choice i.e. (Dession tree, Logistic Regression, Linear SVM), you need to do hyperparameter tuning corresponding to the model you selected and procedure in step 2 and step 3<br>\n",
    "  Note: when you want to find the feature importance make sure you don't use max_depth parameter keep it None.\n",
    "  </li>\n",
    "    <br>\n",
    "    <li>You need to summarize the results at the end of the notebook, summarize it in the table format\n",
    "        <img src='http://i.imgur.com/YVpIGGE.jpg' width=400px>\n",
    "    </li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oVQNrUY8N-3o",
    "outputId": "e2c98ba7-26f5-43b0-88b3-3a11ae4daec1"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\installed\\Anaconda3\\lib\\site-packages\\nltk\\twitter\\__init__.py:20: UserWarning:\n",
      "\n",
      "The twython library has not been installed. Some functionality from the twitter package will not be available.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neg: 0.01, neu: 0.745, pos: 0.245, compound: 0.9975, "
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "# import nltk\n",
    "# nltk.download('vader_lexicon')\n",
    "\n",
    "sid = SentimentIntensityAnalyzer()\n",
    "\n",
    "for_sentiment = 'a person is a person no matter how small dr seuss i teach the smallest students with the biggest enthusiasm \\\n",
    "for learning my students learn in many different ways using all of our senses and multiple intelligences i use a wide range\\\n",
    "of techniques to help all my students succeed students in my class come from a variety of different backgrounds which makes\\\n",
    "for wonderful sharing of experiences and cultures including native americans our school is a caring community of successful \\\n",
    "learners which can be seen through collaborative student project based learning in and out of the classroom kindergarteners \\\n",
    "in my class love to work with hands on materials and have many different opportunities to practice a skill before it is\\\n",
    "mastered having the social skills to work cooperatively with friends is a crucial aspect of the kindergarten curriculum\\\n",
    "montana is the perfect place to learn about agriculture and nutrition my students love to role play in our pretend kitchen\\\n",
    "in the early childhood classroom i have had several kids ask me can we try cooking with real food i will take their idea \\\n",
    "and create common core cooking lessons where we learn important math and writing concepts while cooking delicious healthy \\\n",
    "food for snack time my students will have a grounded appreciation for the work that went into making the food and knowledge \\\n",
    "of where the ingredients came from as well as how it is healthy for their bodies this project would expand our learning of \\\n",
    "nutrition and agricultural cooking recipes by having us peel our own apples to make homemade applesauce make our own bread \\\n",
    "and mix up healthy plants from our classroom garden in the spring we will also create our own cookbooks to be printed and \\\n",
    "shared with families students will gain math and literature skills as well as a life long enjoyment for healthy cooking \\\n",
    "nannan'\n",
    "ss = sid.polarity_scores(for_sentiment)\n",
    "\n",
    "for k in ss:\n",
    "    print('{0}: {1}, '.format(k, ss[k]), end='')\n",
    "\n",
    "# we can use these 4 things as features/attributes (neg, neu, pos, compound)\n",
    "# neg: 0.0, neu: 0.753, pos: 0.247, compound: 0.93"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r6FUMj5TN-3y"
   },
   "source": [
    "<h1>1. Decision Tree </h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QQmid3VAN-31"
   },
   "source": [
    "## 1.1 Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NqY4ES_3N-33"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>school_state</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "      <th>clean_categories</th>\n",
       "      <th>clean_subcategories</th>\n",
       "      <th>essay</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ca</td>\n",
       "      <td>mrs</td>\n",
       "      <td>grades_prek_2</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>math_science</td>\n",
       "      <td>appliedsciences health_lifescience</td>\n",
       "      <td>i fortunate enough use fairy tale stem kits cl...</td>\n",
       "      <td>725.05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  school_state teacher_prefix project_grade_category  \\\n",
       "0           ca            mrs          grades_prek_2   \n",
       "\n",
       "   teacher_number_of_previously_posted_projects  project_is_approved  \\\n",
       "0                                            53                    1   \n",
       "\n",
       "  clean_categories                 clean_subcategories  \\\n",
       "0     math_science  appliedsciences health_lifescience   \n",
       "\n",
       "                                               essay   price  \n",
       "0  i fortunate enough use fairy tale stem kits cl...  725.05  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "data = pandas.read_csv('preprocessed_data.csv')\n",
    "data.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_lXoccSzN-37"
   },
   "source": [
    "<h2>1.2 Splitting data into Train and cross validation(or test): Stratified Sampling</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rSw1_vFcN-38"
   },
   "outputs": [],
   "source": [
    "# please write all the code with proper documentation, and proper titles for each subsection\n",
    "# go through documentations and blogs before you start coding\n",
    "# first figure out what to do, and then think about how to do.\n",
    "# reading and understanding error messages will be very much helpfull in debugging your code\n",
    "# when you plot any graph make sure you use \n",
    "    # a. Title, that describes your plot, this will be very helpful to the reader\n",
    "    # b. Legends if needed\n",
    "    # c. X-axis label\n",
    "    # d. Y-axis label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TNXtHNGVN-4B"
   },
   "source": [
    "<h2>1.3 Make Data Model Ready: encoding eassay, and project_title</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nEevUQOcN-4C"
   },
   "outputs": [],
   "source": [
    "# please write all the code with proper documentation, and proper titles for each subsection\n",
    "# go through documentations and blogs before you start coding\n",
    "# first figure out what to do, and then think about how to do.\n",
    "# reading and understanding error messages will be very much helpfull in debugging your code\n",
    "# make sure you featurize train and test data separatly\n",
    "\n",
    "# when you plot any graph make sure you use \n",
    "    # a. Title, that describes your plot, this will be very helpful to the reader\n",
    "    # b. Legends if needed\n",
    "    # c. X-axis label\n",
    "    # d. Y-axis label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9B9fYus1N-4I"
   },
   "source": [
    "<h2>1.4 Make Data Model Ready: encoding numerical, categorical features</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L-yI_TVhN-4J"
   },
   "outputs": [],
   "source": [
    "# please write all the code with proper documentation, and proper titles for each subsection\n",
    "# go through documentations and blogs before you start coding \n",
    "# first figure out what to do, and then think about how to do.\n",
    "# reading and understanding error messages will be very much helpfull in debugging your code\n",
    "# make sure you featurize train and test data separatly\n",
    "\n",
    "# when you plot any graph make sure you use \n",
    "    # a. Title, that describes your plot, this will be very helpful to the reader\n",
    "    # b. Legends if needed\n",
    "    # c. X-axis label\n",
    "    # d. Y-axis label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oFZ2BOVKN-4N"
   },
   "source": [
    "<h2>1.5 Appling  Decision Tree on different kind of featurization as mentioned in the instructions</h2>\n",
    "\n",
    "<br>Apply  Decision Tree on different kind of featurization as mentioned in the instructions\n",
    "<br> For Every model that you work on make sure you do the step 2 and step 3 of instrucations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ag95KYbvN-4O"
   },
   "outputs": [],
   "source": [
    "# please write all the code with proper documentation, and proper titles for each subsection\n",
    "# go through documentations and blogs before you start coding\n",
    "# first figure out what to do, and then think about how to do.\n",
    "# reading and understanding error messages will be very much helpfull in debugging your code\n",
    "# when you plot any graph make sure you use \n",
    "    # a. Title, that describes your plot, this will be very helpful to the reader\n",
    "    # b. Legends if needed\n",
    "    # c. X-axis label\n",
    "    # d. Y-axis label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2zqgP8b0N-4U"
   },
   "source": [
    "<h2>1.6 Getting top features using `feature_importances_`</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mHvBlI_9N-4X"
   },
   "outputs": [],
   "source": [
    "# please write all the code with proper documentation, and proper titles for each subsection\n",
    "# go through documentations and blogs before you start coding\n",
    "# first figure out what to do, and then think about how to do.\n",
    "# reading and understanding error messages will be very much helpfull in debugging your code\n",
    "# when you plot any graph make sure you use \n",
    "    # a. Title, that describes your plot, this will be very helpful to the reader\n",
    "    # b. Legends if needed\n",
    "    # c. X-axis label\n",
    "    # d. Y-axis label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vfyyOdUaN-4e"
   },
   "source": [
    "<h1>2. Summary</h1>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "9_Assignment_DT_Instructions.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}