2. preprocessing of movie profile .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Content Based Filtering - preprocessing \n",
    "They suggest similar items based on a particular item. This system uses item metadata, such as genre, director, actors, etc. for movies, to make these recommendations. The general idea behind these recommender systems is that if a person liked a particular item, he or she will also like an item that is similar to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 build countvectorizer model for the combined information column (genres, cast, director) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "import scipy.spatial.distance as dist\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined = pd.read_csv('data/combined_info.csv',encoding='iso-8859-1',index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(input='content', encoding='iso-8859-1', decode_error='ignore', analyzer='word',\n",
    "                      ngram_range=(1,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### the dataset is too large and here we first use a sample of the combined data to do the modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = combined.iloc[:2000,:].copy()\n",
    "cv_model = cv.fit_transform(sample['info'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000, 4200)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_model.get_shape()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cv = pd.DataFrame(cv_model.toarray(), index=sample.index,columns=sorted(cv.vocabulary_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>popularity</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>info</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MovieID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>0</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "      <td>tomhanks timallen donrickles johnlasseter anim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8844</th>\n",
       "      <td>0</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "      <td>robinwilliams jonathanhyde kirstendunst  adven...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         adult  popularity  vote_average  vote_count  \\\n",
       "MovieID                                                \n",
       "862          0   21.946943           7.7      5415.0   \n",
       "8844         0   17.015539           6.9      2413.0   \n",
       "\n",
       "                                                      info  \n",
       "MovieID                                                     \n",
       "862      tomhanks timallen donrickles johnlasseter anim...  \n",
       "8844     robinwilliams jonathanhyde kirstendunst  adven...  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 concatenate the numerial columns (MovieID, popularity,vote_average,vote_count,adult)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaroneckhart</th>\n",
       "      <th>aaronkimjohnston</th>\n",
       "      <th>aaronschwartz</th>\n",
       "      <th>abbaskiarostami</th>\n",
       "      <th>abbassayah</th>\n",
       "      <th>abdolrahmanbagheri</th>\n",
       "      <th>abelferrara</th>\n",
       "      <th>abo</th>\n",
       "      <th>abrahampolonsky</th>\n",
       "      <th>acasares</th>\n",
       "      <th>...</th>\n",
       "      <th>ðº</th>\n",
       "      <th>ðºð</th>\n",
       "      <th>ð½</th>\n",
       "      <th>ð½ð</th>\n",
       "      <th>ð¾ð</th>\n",
       "      <th>ð¾ð½ñ</th>\n",
       "      <th>popularity</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>adult</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MovieID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8844</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 4204 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         aaroneckhart  aaronkimjohnston  aaronschwartz  abbaskiarostami  \\\n",
       "MovieID                                                                   \n",
       "862                 0                 0              0                0   \n",
       "8844                0                 0              0                0   \n",
       "\n",
       "         abbassayah  abdolrahmanbagheri  abelferrara  abo  abrahampolonsky  \\\n",
       "MovieID                                                                      \n",
       "862               0                   0            0    0                0   \n",
       "8844              0                   0            0    0                0   \n",
       "\n",
       "         acasares  ...  ðº  ðºð  ð½  ð½ð  ð¾ð  ð¾ð½ñ  popularity  \\\n",
       "MovieID            ...                                             \n",
       "862             0  ...   0    0   0    0    0      0   21.946943   \n",
       "8844            0  ...   0    0   0    0    0      0   17.015539   \n",
       "\n",
       "         vote_average  vote_count  adult  \n",
       "MovieID                                   \n",
       "862               7.7      5415.0      0  \n",
       "8844              6.9      2413.0      0  \n",
       "\n",
       "[2 rows x 4204 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['popularity','vote_average','vote_count','adult']\n",
    "sample2 = sample.loc[:,cols].copy()\n",
    "featureMatrix = pd.concat([df_cv,sample2],axis=1)\n",
    "df_cv2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 normalize the column values to the same scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The movie profile has some components as Boolean and others are real-valued or integer-valued. We can compute the cosine distance between vectors, but before that, we should apply appropriate scaling of the nonBoolean components, so that they neither dominate the calculation nor are they irrelevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = MinMaxScaler()\n",
    "featureMatrix_norm = scaler.fit_transform(featureMatrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "featureMatrix_norm = pd.DataFrame(featureMatrix_norm,index=featureMatrix.index, columns=featureMatrix.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "featureMatrix_norm.to_csv('data/movie_norm_featureMatrix_sample.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:bdata] *",
   "language": "python",
   "name": "conda-env-bdata-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}