LDA_Topic_Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                                Topic Modeling by LDA approach\n",
    "First We have corpus or set of documents which will help us to prepare dictionary so,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc1 = \"Sugar is bad to consume. My sister likes to have sugar, but not my father.\"\n",
    "doc2 = \"My father spends a lot of time driving my sister around to dance practice.\"\n",
    "doc3 = \"Doctors suggest that driving may cause increased stress and blood pressure.\"\n",
    "doc4 = \"Sometimes I feel pressure to perform well at school, but my father never seems to drive my sister to do better.\"\n",
    "doc5 = \"Health experts say that Sugar is not good for your lifestyle.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of taking such short documents we can also take big data like newsgrous data or wikipedia data that is also exactly same fomat like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_complete = [doc1, doc2, doc3, doc4, doc5]  #corpus or compiled documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now first thing we need to do is cleaning documents like removing stopwords,punctuations, and lemmatization\n",
    "import string\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "stop = set(stopwords.words('english'))\n",
    "exclude = set(string.punctuation)\n",
    "lemma = WordNetLemmatizer()\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "sbEng = SnowballStemmer('english')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to separate sentences in document thwn u won't be removing punctuation marks first,then we will use sentence tokenizer to separate into sentences and then will clean each of sentence but here we need list of words tokens so we can do remove punctuation first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean(doc):#to clean document\n",
    "    doc=' '.join([item.strip('!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~') for item in (doc).lower().split(' ')]) #removing punctuation marks\n",
    "    doc=' '.join([item for item in (doc).split(' ') if item not in stop])#stopwords removed\n",
    "    doc=' '.join([sbEng.stem(item) for item in (doc).split(' ')]) #stemming,can slso perform lemmatizing\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_clean=[clean(doc).split() for doc in doc_complete]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['sugar', 'bad', 'consum', 'sister', 'like', 'sugar', 'father'],\n",
       " ['father',\n",
       "  'spend',\n",
       "  'lot',\n",
       "  'time',\n",
       "  'drive',\n",
       "  'sister',\n",
       "  'around',\n",
       "  'danc',\n",
       "  'practic'],\n",
       " ['doctor',\n",
       "  'suggest',\n",
       "  'drive',\n",
       "  'may',\n",
       "  'caus',\n",
       "  'increas',\n",
       "  'stress',\n",
       "  'blood',\n",
       "  'pressur'],\n",
       " ['sometim',\n",
       "  'feel',\n",
       "  'pressur',\n",
       "  'perform',\n",
       "  'well',\n",
       "  'school',\n",
       "  'father',\n",
       "  'never',\n",
       "  'seem',\n",
       "  'drive',\n",
       "  'sister',\n",
       "  'better'],\n",
       " ['health', 'expert', 'say', 'sugar', 'good', 'lifestyl']]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_clean #documents cleaned and converted into list of words/tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preparing dictionary from documents\n",
    "import gensim\n",
    "from gensim import corpora\n",
    "dictionary = corpora.Dictionary(doc_clean) #basically each unique word now given id \n",
    "#now we gonna make document term matrix or bag of words representation of each document that is list of tuples having two values one is id of word and other is frequency of that word in that document\n",
    "doc_term_matrix=[dictionary.doc2bow(doc) for doc in doc_clean]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 2)]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_term_matrix[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sugar'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sugar', 'bad', 'consum', 'sister', 'like', 'sugar', 'father']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_clean[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can see word sugar has id 5 and it has arrived twice in sentence so (5,2).\n",
    "Next step is to create an object for LDA model and train it on Document-Term matrix. The gensim module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the object for LDA model using gensim library\n",
    "Lda = gensim.models.ldamodel.LdaModel\n",
    "\n",
    "# Running and Trainign LDA model on the document term matrix.\n",
    "ldamodel = Lda(doc_term_matrix, num_topics=3, id2word = dictionary, passes=50)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.029*\"father\" + 0.029*\"sister\" + 0.029*\"drive\"'), (1, '0.073*\"sister\" + 0.073*\"father\" + 0.072*\"drive\"'), (2, '0.100*\"sugar\" + 0.040*\"stress\" + 0.040*\"doctor\"')]\n"
     ]
    }
   ],
   "source": [
    "print(ldamodel.print_topics(num_topics=3, num_words=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Latent semantic indexing model\n",
    "from gensim.models import LsiModel\n",
    "\n",
    "model = LsiModel(doc_term_matrix, id2word=dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gensim.models.lsimodel.LsiModel at 0x7fae799ae550>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, '0.401*\"father\" + 0.401*\"sister\" + 0.377*\"drive\"'),\n",
       " (1, '-0.561*\"sugar\" + 0.260*\"pressur\" + 0.225*\"drive\"'),\n",
       " (2, '0.354*\"sugar\" + 0.275*\"doctor\" + 0.275*\"blood\"')]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.print_topics(num_topics=3, num_words=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "hert=ldamodel[doc_term_matrix[4]] #to implement trained model on new document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.04854002), (1, 0.0479168), (2, 0.9035432)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hert #probability of each topic for this document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model perplexity and topic coherence provide a convenient measure to judge how good a given topic model is. In my experience, topic coherence score, in particular, has been more helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Perplexity:  -4.230102421239365\n",
      "\n",
      "Coherence Score:  0.32324888110161115\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import CoherenceModel\n",
    "# Compute Perplexity\n",
    "print('\\nPerplexity: ', ldamodel.log_perplexity(doc_term_matrix))  # a measure of how good the model is. lower the better.\n",
    "\n",
    "# Compute Coherence Score\n",
    "coherence_model_lda = CoherenceModel(model=ldamodel, texts=doc_clean, dictionary=dictionary, coherence='c_v')\n",
    "coherence_lda = coherence_model_lda.get_coherence()\n",
    "print('\\nCoherence Score: ', coherence_lda)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el282061403872054410004165317034\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el282061403872054410004165317034_data = {\"lambda.step\": 0.01, \"mdsDat\": {\"topics\": [1, 2, 3], \"x\": [-0.08431018488349759, 0.08564203732145007, -0.0013318524379525081], \"Freq\": [48.35808181762695, 48.1235466003418, 3.518367290496826], \"y\": [0.0, 0.0, 0.0], \"cluster\": [1, 1, 1]}, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"Term\": [\"sugar\", \"around\", \"lot\", \"time\", \"spend\", \"practic\", \"danc\", \"seem\", \"perform\", \"school\", \"better\", \"feel\", \"sometim\", \"well\", \"never\", \"health\", \"say\", \"good\", \"lifestyl\", \"expert\", \"like\", \"consum\", \"bad\", \"doctor\", \"increas\", \"may\", \"suggest\", \"blood\", \"caus\", \"stress\", \"sugar\", \"may\", \"caus\", \"suggest\", \"doctor\", \"increas\", \"blood\", \"stress\", \"good\", \"lifestyl\", \"expert\", \"health\", \"say\", \"like\", \"consum\", \"bad\", \"pressur\", \"drive\", \"sister\", \"father\", \"around\", \"danc\", \"practic\", \"spend\", \"lot\", \"time\", \"perform\", \"seem\", \"never\", \"well\", \"feel\", \"sometim\", \"better\", \"never\", \"well\", \"school\", \"sometim\", \"seem\", \"feel\", \"perform\", \"practic\", \"time\", \"spend\", \"danc\", \"lot\", \"around\", \"father\", \"sister\", \"drive\", \"pressur\", \"like\", \"consum\", \"bad\", \"say\", \"health\", \"good\", \"lifestyl\", \"expert\", \"suggest\", \"doctor\", \"caus\", \"stress\", \"may\", \"blood\", \"sugar\", \"increas\", \"health\", \"good\", \"say\", \"expert\", \"lifestyl\", \"like\", \"consum\", \"bad\", \"may\", \"stress\", \"doctor\", \"increas\", \"caus\", \"blood\", \"suggest\", \"around\", \"time\", \"lot\", \"practic\", \"danc\", \"spend\", \"seem\", \"perform\", \"school\", \"never\", \"well\", \"better\", \"feel\", \"sometim\", \"pressur\", \"father\", \"sister\", \"drive\", \"sugar\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -2.3010001182556152, -3.2177000045776367, -3.2177000045776367, -3.2177000045776367, -3.2177000045776367, -3.2177000045776367, -3.2177000045776367, -3.2177000045776367, -3.218100070953369, -3.218100070953369, -3.218100070953369, -3.218100070953369, -3.218100070953369, -3.2179999351501465, -3.2179999351501465, -3.2179999351501465, -3.2191998958587646, -3.2242000102996826, -3.2302000522613525, -3.2302000522613525, -4.602399826049805, -4.602399826049805, -4.602399826049805, -4.602399826049805, -4.602399826049805, -4.602399826049805, -4.60260009765625, -4.60260009765625, -4.60260009765625, -4.60260009765625, -4.60260009765625, -4.60260009765625, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1900999546051025, -3.1902999877929688, -3.1902999877929688, -3.1902999877929688, -3.1902999877929688, -3.1902999877929688, -3.1902999877929688, -2.6231000423431396, -2.6231000423431396, -2.626300096511841, -3.188199996948242, -4.5742998123168945, -4.5742998123168945, -4.5742998123168945, -4.5746002197265625, -4.5746002197265625, -4.5746002197265625, -4.5746002197265625, -4.5746002197265625, -4.574900150299072, -4.574900150299072, -4.574900150299072, -4.574900150299072, -4.574900150299072, -4.574900150299072, -4.57450008392334, -4.574900150299072, -3.5255000591278076, -3.5255000591278076, -3.5255000591278076, -3.5255000591278076, -3.5255000591278076, -3.5260000228881836, -3.5260000228881836, -3.5260000228881836, -3.526700019836426, -3.526700019836426, -3.526700019836426, -3.526700019836426, -3.526700019836426, -3.526700019836426, -3.526700019836426, -3.5267999172210693, -3.5267999172210693, -3.5267999172210693, -3.5267999172210693, -3.5267999172210693, -3.5267999172210693, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.527400016784668, -3.5250000953674316, -3.5241000652313232, -3.5241000652313232, -3.5248000621795654, -3.526099920272827], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.6097999811172485, 0.45680001378059387, 0.45680001378059387, 0.45680001378059387, 0.45680001378059387, 0.45680001378059387, 0.45680001378059387, 0.45680001378059387, 0.45669999718666077, 0.45669999718666077, 0.45669999718666077, 0.45669999718666077, 0.45669999718666077, 0.45660001039505005, 0.45660001039505005, 0.45660001039505005, -0.005799999926239252, -0.3255000114440918, -0.33149999380111694, -0.33149999380111694, -0.9408000111579895, -0.9408000111579895, -0.9408000111579895, -0.9408000111579895, -0.9408000111579895, -0.9408000111579895, -0.9409999847412109, -0.9409999847412109, -0.9409999847412109, -0.9409999847412109, -0.9409999847412109, -0.9409999847412109, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.4713999927043915, 0.47130000591278076, 0.47130000591278076, 0.47130000591278076, 0.47130000591278076, 0.47130000591278076, 0.47130000591278076, 0.27570000290870667, 0.27570000290870667, 0.27250000834465027, 0.025100000202655792, -0.8996999859809875, -0.8996999859809875, -0.8996999859809875, -0.8999000191688538, -0.8999000191688538, -0.8999000191688538, -0.8999000191688538, -0.8999000191688538, -0.9003000259399414, -0.9003000259399414, -0.9003000259399414, -0.9003000259399414, -0.9003000259399414, -0.9003000259399414, -1.663699984550476, -0.9003000259399414, 0.1492999941110611, 0.1492999941110611, 0.1492999941110611, 0.1492999941110611, 0.1492999941110611, 0.1485999971628189, 0.1485999971628189, 0.1485999971628189, 0.14790000021457672, 0.14790000021457672, 0.14790000021457672, 0.14790000021457672, 0.14790000021457672, 0.14790000021457672, 0.14790000021457672, 0.1348000019788742, 0.1348000019788742, 0.1348000019788742, 0.1348000019788742, 0.1348000019788742, 0.1348000019788742, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, 0.13410000503063202, -0.311599999666214, -0.6252999901771545, -0.6252999901771545, -0.6259999871253967, -0.6152999997138977], \"Freq\": [2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0826914310455322, 0.8327353596687317, 0.8327353596687317, 0.8327353596687317, 0.8327353596687317, 0.8327353596687317, 0.8327353596687317, 0.8327353596687317, 0.8324315547943115, 0.8324315547943115, 0.8324315547943115, 0.8324315547943115, 0.8324315547943115, 0.8324814438819885, 0.8324815034866333, 0.8324816226959229, 0.8314950466156006, 0.8273035287857056, 0.8223651051521301, 0.8223649263381958, 0.20852278172969818, 0.2085227072238922, 0.20852269232273102, 0.20852266252040863, 0.20852264761924744, 0.20852264761924744, 0.20848360657691956, 0.20848360657691956, 0.20848360657691956, 0.20848360657691956, 0.20848360657691956, 0.20848360657691956, 0.8518848419189453, 0.8518848419189453, 0.8518848419189453, 0.8518847823143005, 0.8518849611282349, 0.8518847227096558, 0.8518849611282349, 0.8518847227096558, 0.8517206907272339, 0.8517206907272339, 0.8517206907272339, 0.8517206907272339, 0.8517206907272339, 0.8517205119132996, 1.5019184350967407, 1.5019184350967407, 1.4970134496688843, 0.8534795641899109, 0.2134094089269638, 0.21340937912464142, 0.21340937912464142, 0.21334627270698547, 0.21334627270698547, 0.21334624290466309, 0.21334624290466309, 0.21334624290466309, 0.213295578956604, 0.213295578956604, 0.213295578956604, 0.213295578956604, 0.213295578956604, 0.213295578956604, 0.21336571872234344, 0.213295578956604, 0.04453614726662636, 0.04453614726662636, 0.04453614726662636, 0.04453613981604576, 0.04453613981604576, 0.04451238736510277, 0.044512368738651276, 0.04451233893632889, 0.044482000172138214, 0.044482000172138214, 0.044482000172138214, 0.044482000172138214, 0.044482000172138214, 0.044482000172138214, 0.044482000172138214, 0.04447736218571663, 0.044477351009845734, 0.044477351009845734, 0.04447733983397484, 0.04447733983397484, 0.04447734355926514, 0.044451504945755005, 0.04445149004459381, 0.04445148631930351, 0.044451478868722916, 0.044451478868722916, 0.044451478868722916, 0.04445146769285202, 0.04445146769285202, 0.0445590578019619, 0.044597528874874115, 0.04459748789668083, 0.04456710070371628, 0.04450910910964012], \"Total\": [2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.340566396713257, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0904031991958618, 1.0904033184051514, 1.090403437614441, 1.7295336723327637, 2.3688840866088867, 2.3688809871673584, 2.3688809871673584, 1.1047205924987793, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1048197746276855, 1.1048197746276855, 1.104819893836975, 1.104819893836975, 1.1048200130462646, 1.1048200130462646, 1.104819893836975, 1.104819893836975, 1.104819893836975, 1.104819893836975, 1.1048200130462646, 1.1048197746276855, 1.1048200130462646, 1.1048197746276855, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047205924987793, 2.3688809871673584, 2.3688809871673584, 2.3688840866088867, 1.7295336723327637, 1.0904031991958618, 1.0904033184051514, 1.090403437614441, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 2.340566396713257, 1.090512990951538, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0903139114379883, 1.0904031991958618, 1.0904033184051514, 1.090403437614441, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.090512990951538, 1.1047205924987793, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1047207117080688, 1.1048197746276855, 1.1048197746276855, 1.104819893836975, 1.104819893836975, 1.104819893836975, 1.104819893836975, 1.1048200130462646, 1.1048200130462646, 1.7295336723327637, 2.3688809871673584, 2.3688809871673584, 2.3688840866088867, 2.340566396713257]}, \"token.table\": {\"Term\": [\"around\", \"bad\", \"better\", \"blood\", \"caus\", \"consum\", \"danc\", \"doctor\", \"drive\", \"drive\", \"expert\", \"father\", \"father\", \"feel\", \"good\", \"health\", \"increas\", \"lifestyl\", \"like\", \"lot\", \"may\", \"never\", \"perform\", \"practic\", \"pressur\", \"pressur\", \"say\", \"school\", \"seem\", \"sister\", \"sister\", \"sometim\", \"spend\", \"stress\", \"sugar\", \"suggest\", \"time\", \"well\"], \"Topic\": [2, 1, 2, 1, 1, 1, 2, 1, 1, 2, 1, 1, 2, 2, 1, 1, 1, 1, 1, 2, 1, 2, 2, 2, 1, 2, 1, 2, 2, 1, 2, 2, 2, 1, 1, 1, 2, 2], \"Freq\": [0.9052062630653381, 0.9170917272567749, 0.9051249027252197, 0.9169996380805969, 0.9169996380805969, 0.9170918464660645, 0.9052061438560486, 0.9169996380805969, 0.4221397042274475, 0.4221397042274475, 0.917167067527771, 0.4221402406692505, 0.844280481338501, 0.9051247835159302, 0.917167067527771, 0.917167067527771, 0.9169996380805969, 0.917167067527771, 0.917091965675354, 0.9052061438560486, 0.9169996380805969, 0.9051249027252197, 0.9051250219345093, 0.9052061438560486, 0.5781905651092529, 0.5781905651092529, 0.917167067527771, 0.9051249027252197, 0.9051250219345093, 0.4221402406692505, 0.844280481338501, 0.9051247835159302, 0.9052061438560486, 0.9169996380805969, 0.854494035243988, 0.9169996380805969, 0.9052061438560486, 0.9051249027252197]}, \"plot.opts\": {\"ylab\": \"PC2\", \"xlab\": \"PC1\"}, \"topic.order\": [3, 2, 1], \"R\": 30};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el282061403872054410004165317034\", ldavis_el282061403872054410004165317034_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el282061403872054410004165317034\", ldavis_el282061403872054410004165317034_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el282061403872054410004165317034\", ldavis_el282061403872054410004165317034_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=            Freq  cluster  topics         x    y\n",
       "topic                                           \n",
       "2      48.358082        1       1 -0.084310  0.0\n",
       "1      48.123547        1       2  0.085642  0.0\n",
       "0       3.518367        1       3 -0.001332  0.0, topic_info=     Category      Freq      Term     Total  loglift  logprob\n",
       "term                                                         \n",
       "26    Default  2.000000     sugar  2.000000  30.0000  30.0000\n",
       "12    Default  1.000000    around  1.000000  29.0000  29.0000\n",
       "24    Default  1.000000       lot  1.000000  28.0000  28.0000\n",
       "3     Default  1.000000      time  1.000000  27.0000  27.0000\n",
       "6     Default  1.000000     spend  1.000000  26.0000  26.0000\n",
       "32    Default  1.000000   practic  1.000000  25.0000  25.0000\n",
       "11    Default  1.000000      danc  1.000000  24.0000  24.0000\n",
       "21    Default  1.000000      seem  1.000000  23.0000  23.0000\n",
       "18    Default  1.000000   perform  1.000000  22.0000  22.0000\n",
       "2     Default  1.000000    school  1.000000  21.0000  21.0000\n",
       "25    Default  1.000000    better  1.000000  20.0000  20.0000\n",
       "15    Default  1.000000      feel  1.000000  19.0000  19.0000\n",
       "13    Default  1.000000   sometim  1.000000  18.0000  18.0000\n",
       "22    Default  1.000000      well  1.000000  17.0000  17.0000\n",
       "14    Default  1.000000     never  1.000000  16.0000  16.0000\n",
       "28    Default  1.000000    health  1.000000  15.0000  15.0000\n",
       "1     Default  1.000000       say  1.000000  14.0000  14.0000\n",
       "8     Default  1.000000      good  1.000000  13.0000  13.0000\n",
       "7     Default  1.000000  lifestyl  1.000000  12.0000  12.0000\n",
       "33    Default  1.000000    expert  1.000000  11.0000  11.0000\n",
       "31    Default  1.000000      like  1.000000  10.0000  10.0000\n",
       "23    Default  1.000000    consum  1.000000   9.0000   9.0000\n",
       "9     Default  1.000000       bad  1.000000   8.0000   8.0000\n",
       "29    Default  1.000000    doctor  1.000000   7.0000   7.0000\n",
       "27    Default  1.000000   increas  1.000000   6.0000   6.0000\n",
       "17    Default  1.000000       may  1.000000   5.0000   5.0000\n",
       "19    Default  1.000000   suggest  1.000000   4.0000   4.0000\n",
       "10    Default  1.000000     blood  1.000000   3.0000   3.0000\n",
       "5     Default  1.000000      caus  1.000000   2.0000   2.0000\n",
       "4     Default  1.000000    stress  1.000000   1.0000   1.0000\n",
       "...       ...       ...       ...       ...      ...      ...\n",
       "7      Topic3  0.044536  lifestyl  1.090314   0.1493  -3.5255\n",
       "31     Topic3  0.044512      like  1.090403   0.1486  -3.5260\n",
       "23     Topic3  0.044512    consum  1.090403   0.1486  -3.5260\n",
       "9      Topic3  0.044512       bad  1.090403   0.1486  -3.5260\n",
       "17     Topic3  0.044482       may  1.090513   0.1479  -3.5267\n",
       "4      Topic3  0.044482    stress  1.090513   0.1479  -3.5267\n",
       "29     Topic3  0.044482    doctor  1.090513   0.1479  -3.5267\n",
       "27     Topic3  0.044482   increas  1.090513   0.1479  -3.5267\n",
       "5      Topic3  0.044482      caus  1.090513   0.1479  -3.5267\n",
       "10     Topic3  0.044482     blood  1.090513   0.1479  -3.5267\n",
       "19     Topic3  0.044482   suggest  1.090513   0.1479  -3.5267\n",
       "12     Topic3  0.044477    around  1.104721   0.1348  -3.5268\n",
       "3      Topic3  0.044477      time  1.104721   0.1348  -3.5268\n",
       "24     Topic3  0.044477       lot  1.104721   0.1348  -3.5268\n",
       "32     Topic3  0.044477   practic  1.104721   0.1348  -3.5268\n",
       "11     Topic3  0.044477      danc  1.104721   0.1348  -3.5268\n",
       "6      Topic3  0.044477     spend  1.104721   0.1348  -3.5268\n",
       "21     Topic3  0.044452      seem  1.104820   0.1341  -3.5274\n",
       "18     Topic3  0.044451   perform  1.104820   0.1341  -3.5274\n",
       "2      Topic3  0.044451    school  1.104820   0.1341  -3.5274\n",
       "14     Topic3  0.044451     never  1.104820   0.1341  -3.5274\n",
       "22     Topic3  0.044451      well  1.104820   0.1341  -3.5274\n",
       "25     Topic3  0.044451    better  1.104820   0.1341  -3.5274\n",
       "15     Topic3  0.044451      feel  1.104820   0.1341  -3.5274\n",
       "13     Topic3  0.044451   sometim  1.104820   0.1341  -3.5274\n",
       "30     Topic3  0.044559   pressur  1.729534  -0.3116  -3.5250\n",
       "16     Topic3  0.044598    father  2.368881  -0.6253  -3.5241\n",
       "20     Topic3  0.044597    sister  2.368881  -0.6253  -3.5241\n",
       "0      Topic3  0.044567     drive  2.368884  -0.6260  -3.5248\n",
       "26     Topic3  0.044509     sugar  2.340566  -0.6153  -3.5261\n",
       "\n",
       "[130 rows x 6 columns], token_table=      Topic      Freq      Term\n",
       "term                           \n",
       "12        2  0.905206    around\n",
       "9         1  0.917092       bad\n",
       "25        2  0.905125    better\n",
       "10        1  0.917000     blood\n",
       "5         1  0.917000      caus\n",
       "23        1  0.917092    consum\n",
       "11        2  0.905206      danc\n",
       "29        1  0.917000    doctor\n",
       "0         1  0.422140     drive\n",
       "0         2  0.422140     drive\n",
       "33        1  0.917167    expert\n",
       "16        1  0.422140    father\n",
       "16        2  0.844280    father\n",
       "15        2  0.905125      feel\n",
       "8         1  0.917167      good\n",
       "28        1  0.917167    health\n",
       "27        1  0.917000   increas\n",
       "7         1  0.917167  lifestyl\n",
       "31        1  0.917092      like\n",
       "24        2  0.905206       lot\n",
       "17        1  0.917000       may\n",
       "14        2  0.905125     never\n",
       "18        2  0.905125   perform\n",
       "32        2  0.905206   practic\n",
       "30        1  0.578191   pressur\n",
       "30        2  0.578191   pressur\n",
       "1         1  0.917167       say\n",
       "2         2  0.905125    school\n",
       "21        2  0.905125      seem\n",
       "20        1  0.422140    sister\n",
       "20        2  0.844280    sister\n",
       "13        2  0.905125   sometim\n",
       "6         2  0.905206     spend\n",
       "4         1  0.917000    stress\n",
       "26        1  0.854494     sugar\n",
       "19        1  0.917000   suggest\n",
       "3         2  0.905206      time\n",
       "22        2  0.905125      well, R=30, lambda_step=0.01, plot_opts={'ylab': 'PC2', 'xlab': 'PC1'}, topic_order=[3, 2, 1])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyLDAvis\n",
    "import pyLDAvis.gensim\n",
    "# Visualize the topics\n",
    "pyLDAvis.enable_notebook()\n",
    "vis = pyLDAvis.gensim.prepare(ldamodel, doc_term_matrix, dictionary)\n",
    "vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " So how to infer pyLDAvis’s output?\n",
    "\n",
    "Each bubble on the left-hand side plot represents a topic. The larger the bubble, the more prevalent is that topic.\n",
    "\n",
    "A good topic model will have fairly big, non-overlapping bubbles scattered throughout the chart instead of being clustered in one quadrant.\n",
    "\n",
    "A model with too many topics, will typically have many overlaps, small sized bubbles clustered in one region of the chart.\n",
    "\n",
    "Alright, if you move the cursor over one of the bubbles, the words and bars on the right-hand side will update. These words are the salient keywords that form the selected topic.\n",
    "\n",
    "We have successfully built a good looking topic model.\n",
    "\n",
    "Given our prior knowledge of the number of natural topics in the document, finding the best model was fairly straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for new document using trained model\n",
    "new_document='stress daughter to school and his blood pressure is high'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleandoc=clean(new_document)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleandoc=cleandoc.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix=dictionary.doc2bow(cleandoc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(13, 1), (18, 1), (19, 1), (25, 1)]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.06938401), (1, 0.31519938), (2, 0.61541665)]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ldamodel[matrix]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, '0.029*\"father\" + 0.029*\"sister\" + 0.029*\"drive\"'),\n",
       " (1, '0.073*\"sister\" + 0.073*\"father\" + 0.072*\"drive\"'),\n",
       " (2, '0.100*\"sugar\" + 0.040*\"stress\" + 0.040*\"doctor\"')]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ldamodel.print_topics(num_topics=3,num_words=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                        Using big dataset newsgroup dataset to train our lda model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['rec.autos' 'comp.sys.mac.hardware' 'rec.motorcycles' 'misc.forsale'\n",
      " 'comp.os.ms-windows.misc' 'alt.atheism' 'comp.graphics'\n",
      " 'rec.sport.baseball' 'rec.sport.hockey' 'sci.electronics' 'sci.space'\n",
      " 'talk.politics.misc' 'sci.med' 'talk.politics.mideast'\n",
      " 'soc.religion.christian' 'comp.windows.x' 'comp.sys.ibm.pc.hardware'\n",
      " 'talk.politics.guns' 'talk.religion.misc' 'sci.crypt']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>target</th>\n",
       "      <th>target_names</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: lerxst@wam.umd.edu (where's my thing)\\nS...</td>\n",
       "      <td>7</td>\n",
       "      <td>rec.autos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: guykuo@carson.u.washington.edu (Guy Kuo)...</td>\n",
       "      <td>4</td>\n",
       "      <td>comp.sys.mac.hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>From: irwin@cmptrc.lonestar.org (Irwin Arnstei...</td>\n",
       "      <td>8</td>\n",
       "      <td>rec.motorcycles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>From: tchen@magnus.acs.ohio-state.edu (Tsung-K...</td>\n",
       "      <td>6</td>\n",
       "      <td>misc.forsale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>From: dabl2@nlm.nih.gov (Don A.B. Lindbergh)\\n...</td>\n",
       "      <td>2</td>\n",
       "      <td>comp.os.ms-windows.misc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                content  target  \\\n",
       "0     From: lerxst@wam.umd.edu (where's my thing)\\nS...       7   \n",
       "1     From: guykuo@carson.u.washington.edu (Guy Kuo)...       4   \n",
       "10    From: irwin@cmptrc.lonestar.org (Irwin Arnstei...       8   \n",
       "100   From: tchen@magnus.acs.ohio-state.edu (Tsung-K...       6   \n",
       "1000  From: dabl2@nlm.nih.gov (Don A.B. Lindbergh)\\n...       2   \n",
       "\n",
       "                 target_names  \n",
       "0                   rec.autos  \n",
       "1       comp.sys.mac.hardware  \n",
       "10            rec.motorcycles  \n",
       "100              misc.forsale  \n",
       "1000  comp.os.ms-windows.misc  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_json('https://raw.githubusercontent.com/selva86/datasets/master/newsgroups.json')\n",
    "print(df.target_names.unique())\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'From: twillis@ec.ecn.purdue.edu (Thomas E Willis)\\nSubject: PB questions...\\nOrganization: Purdue University Engineering Computer Network\\nDistribution: usa\\nLines: 36\\n\\nwell folks, my mac plus finally gave up the ghost this weekend after\\nstarting life as a 512k way back in 1985.  sooo, i\\'m in the market for a\\nnew machine a bit sooner than i intended to be...\\n\\ni\\'m looking into picking up a powerbook 160 or maybe 180 and have a bunch\\nof questions that (hopefully) somebody can answer:\\n\\n* does anybody know any dirt on when the next round of powerbook\\nintroductions are expected?  i\\'d heard the 185c was supposed to make an\\nappearence \"this summer\" but haven\\'t heard anymore on it - and since i\\ndon\\'t have access to macleak, i was wondering if anybody out there had\\nmore info...\\n\\n* has anybody heard rumors about price drops to the powerbook line like the\\nones the duo\\'s just went through recently?\\n\\n* what\\'s the impression of the display on the 180?  i could probably swing\\na 180 if i got the 80Mb disk rather than the 120, but i don\\'t really have\\na feel for how much \"better\" the display is (yea, it looks great in the\\nstore, but is that all \"wow\" or is it really that good?).  could i solicit\\nsome opinions of people who use the 160 and 180 day-to-day on if its worth\\ntaking the disk size and money hit to get the active display?  (i realize\\nthis is a real subjective question, but i\\'ve only played around with the\\nmachines in a computer store breifly and figured the opinions of somebody\\nwho actually uses the machine daily might prove helpful).\\n\\n* how well does hellcats perform?  ;)\\n\\nthanks a bunch in advance for any info - if you could email, i\\'ll post a\\nsummary (news reading time is at a premium with finals just around the\\ncorner... :( )\\n--\\nTom Willis  \\\\  twillis@ecn.purdue.edu    \\\\    Purdue Electrical Engineering\\n---------------------------------------------------------------------------\\n\"Convictions are more dangerous enemies of truth than lies.\"  - F. W.\\nNietzsche\\n'"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.content[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "#so we goona use each content as document \n",
    "documents=[df.content[i] for i in range(len(df))] #corpus ready"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents_clean=[clean(document).split() for document in documents ] #cleaned corpus and converted to list of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_dictionary=corpora.Dictionary(documents_clean) #dictionary prepared "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_matrix=[new_dictionary.doc2bow(doc) for doc in documents_clean] #matrix generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 1),\n",
       " (1, 1),\n",
       " (2, 1),\n",
       " (3, 1),\n",
       " (4, 1),\n",
       " (5, 1),\n",
       " (6, 2),\n",
       " (7, 1),\n",
       " (8, 1),\n",
       " (9, 1),\n",
       " (10, 1),\n",
       " (11, 1),\n",
       " (12, 5),\n",
       " (13, 1),\n",
       " (14, 1),\n",
       " (15, 1),\n",
       " (16, 1),\n",
       " (17, 1),\n",
       " (18, 1),\n",
       " (19, 1),\n",
       " (20, 1),\n",
       " (21, 1),\n",
       " (22, 1),\n",
       " (23, 1),\n",
       " (24, 1),\n",
       " (25, 1),\n",
       " (26, 1),\n",
       " (27, 1),\n",
       " (28, 1),\n",
       " (29, 1),\n",
       " (30, 1),\n",
       " (31, 1),\n",
       " (32, 2),\n",
       " (33, 1),\n",
       " (34, 1),\n",
       " (35, 1),\n",
       " (36, 1),\n",
       " (37, 1),\n",
       " (38, 1),\n",
       " (39, 1),\n",
       " (40, 1),\n",
       " (41, 1),\n",
       " (42, 1),\n",
       " (43, 1),\n",
       " (44, 1),\n",
       " (45, 1),\n",
       " (46, 1),\n",
       " (47, 1),\n",
       " (48, 1),\n",
       " (49, 1),\n",
       " (50, 1),\n",
       " (51, 1),\n",
       " (52, 1),\n",
       " (53, 1),\n",
       " (54, 2),\n",
       " (55, 1),\n",
       " (56, 1),\n",
       " (57, 1),\n",
       " (58, 1),\n",
       " (59, 1),\n",
       " (60, 1),\n",
       " (61, 1),\n",
       " (62, 1),\n",
       " (63, 1)]"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_matrix[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'newldamode' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-115-d736b6009fe3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnewldamodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc_matrix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_topics\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mid2word\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_dictionary\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpasses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnewldamode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_topics\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_topics\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mnum_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'newldamode' is not defined"
     ]
    }
   ],
   "source": [
    "newldamodel = Lda(doc_matrix, num_topics=10, id2word = new_dictionary, passes=50)\n",
    "#new ldamodel trained on big data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.009*\"israel\" + 0.008*\"isra\" + 0.006*\"db\" + 0.004*\"arab\" + 0.003*\"p\"'), (1, '0.007*\"the\" + 0.006*\"peopl\" + 0.006*\"would\" + 0.005*\"in\" + 0.005*\"one\"'), (2, '0.062*\"max>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'ax>\\'\" + 0.002*\"dog\" + 0.002*\"ra\" + 0.001*\"rock\" + 0.001*\"counterst\"'), (3, '0.011*\">\" + 0.010*\"organ\" + 0.010*\"subject\" + 0.010*\"writes:\" + 0.009*\"lin\"'), (4, '0.041*\"x\" + 0.029*\"1\" + 0.020*\"0\" + 0.016*\"2\" + 0.009*\"3\"'), (5, '0.011*\"organ\" + 0.011*\"subject\" + 0.010*\"lin\" + 0.008*\"nntp-posting-host\" + 0.005*\"univers\"'), (6, '0.011*\"use\" + 0.010*\"subject\" + 0.009*\"organ\" + 0.008*\"lin\" + 0.007*\"i\"'), (7, '0.008*\"space\" + 0.008*\"armenian\" + 0.006*\"the\" + 0.005*\"turkish\" + 0.004*\"of\"'), (8, '0.006*\"use\" + 0.004*\"list\" + 0.004*\"inform\" + 0.004*\"mail\" + 0.003*\"research\"'), (9, '0.008*\"god\" + 0.007*\"the\" + 0.007*\"one\" + 0.006*\"would\" + 0.006*\"i\"')]\n"
     ]
    }
   ],
   "source": [
    "print(newldamodel.print_topics(num_topics=10,num_words=5))#new lda model trained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}