spaCy_tuTorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Introduction to Natural Language in Python using spaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This tutorial provides a brief introduction to working with natural language (sometimes called \"text analytics\") in Pytho, using [spaCy](https://spacy.io/) and related libraries.\n",
    "Data science teams in industry must work with lots of text, one of the top four categories of data used in machine learning.\n",
    "Usually that's human-generated text, although not always.\n",
    "\n",
    "Think about it: how does the \"operating system\" for business work? Typically, there are contracts (sales contracts, work agreements, partnerships), there are invoices, there are insurance policies, there are regulations and other laws, and so on.\n",
    "All of those are represented as text.\n",
    "\n",
    "You may run across a few acronyms: _natural language processing_ (NLP), _natural language understanding_ (NLU), _natural language generation_ (NLG) — which are roughly speaking \"read text\", \"understand meaning\", \"write text\" respectively.\n",
    "Increasingly these tasks overlap and it becomes difficult to categorize any given feature.\n",
    "\n",
    "The _spaCy_ framework — along with a wide and growing range of plug-ins and other integrations — provides features for a wide range of natural language tasks.\n",
    "It's become one of the most widely used natural language libraries in Python for industry use cases, and has quite a large community — and with that, much support for commercialization of research advances as this area continues to evolve rapidly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "\n",
    "Check out the excellent _spaCy_ [installation notes](https://spacy.io/usage) for a \"configurator\" which generates installation commands based on which platforms and natural languages you need to support.\n",
    "\n",
    "Some people tend to use `pip` while others use `conda`, and there are instructions for both.  For example, to get started with _spaCy_ working with text in English and installed via `conda` on a Linux system:\n",
    "```\n",
    "conda install -c conda-forge spacy\n",
    "python -m spacy download en_core_web_sm\n",
    "```\n",
    "\n",
    "BTW, the second line above is a download for language resources (models, etc.) and the `_sm` at the end of the download's name indicates a \"small\" model. There's also \"medium\" and \"large\", albeit those are quite large. Some of the more advanced features depend on the latter, although we won't quite be diving to the bottom of that ocean in this (brief) tutorial.\n",
    "\n",
    "Now let's load _spaCy_ and run some code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That `nlp` variable is now your gateway to all things _spaCy_ and loaded with the `en_core_web_sm` small model for English.\n",
    "Next, let's run a small \"document\" through the natural language parser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"The rain in Spain falls mainly on the plain.\"\n",
    "doc = nlp(text)\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.is_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we created a [doc](https://spacy.io/api/doc) from the text, which is a container for a document and all of its annotations. Then we iterated through the document to see what _spaCy_ had parsed.\n",
    "\n",
    "Good, but it's a lot of info and a bit difficult to read. Let's reformat the _spaCy_ parse of that sentence as a [pandas](https://pandas.pydata.org/) dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "cols = (\"text\", \"lemma\", \"POS\", \"explain\", \"stopword\")\n",
    "rows = []\n",
    "\n",
    "for t in doc:\n",
    "    row = [t.text, t.lemma_, t.pos_, spacy.explain(t.pos_), t.is_stop]\n",
    "    rows.append(row)\n",
    "\n",
    "df = pd.DataFrame(rows, columns=cols)\n",
    "    \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much more readable!\n",
    "In this simple case, the entire document is merely one short sentence.\n",
    "For each word in that sentence _spaCy_ has created a [token](https://spacy.io/api/token), and we accessed fields in each token to show:\n",
    "\n",
    " - raw text\n",
    " - [lemma](https://en.wikipedia.org/wiki/Lemma_(morphology)) – a root form of the word\n",
    " - [part of speech](https://en.wikipedia.org/wiki/Part_of_speech)\n",
    " - a flag for whether the word is a _stopword_ – i.e., a common word that may be filtered out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's use the [displaCy](https://ines.io/blog/developing-displacy) library to visualize the parse tree for that sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "displacy.render(doc, style=\"dep\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does that bring back memories of grade school? Frankly, for those of us coming from more of a computational linguistics background, that diagram sparks joy.\n",
    "\n",
    "But let's backup for a moment. How do you handle multiple sentences?\n",
    "\n",
    "There are features for _sentence boundary detection_ (SBD) – also known as _sentence segmentation_ – based on the builtin/default [sentencizer](https://spacy.io/api/sentencizer):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"We were all out at the zoo one day, I was doing some acting, walking on the railing of the gorilla exhibit. I fell in. Everyone screamed and Tommy jumped in after me, forgetting that he had blueberries in his front pocket. The gorillas just went wild.\"\n",
    "\n",
    "doc = nlp(text)\n",
    "\n",
    "for sent in doc.sents:\n",
    "    print(\">\", sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When _spaCy_ creates a document, it uses a principle of _non-destructive tokenization_ meaning that the tokens, sentences, etc., are simply indexes into a long array. In other words, they don't carve the text stream into little pieces. So each sentence is a [span](https://spacy.io/api/span) with a _start_ and an _end_ index into the document array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sent in doc.sents:\n",
    "    print(\">\", sent.start, sent.end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can index into the document array to pull out the tokens for one sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc[48:54]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or simply index into a specific token, such as the verb `went` in the last sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token = doc[51]\n",
    "print(token.text, token.lemma_, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we can parse a document, segment that document into sentences, then look at annotations about the tokens in each sentence. That's a good start."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acquiring Text\n",
    "\n",
    "Now that we can parse texts, where do we get texts?\n",
    "One quick source is to leverage the interwebs.\n",
    "Of course when we download web pages we'll get HTML, and then need to extract text from them.\n",
    "[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a popular package for that.\n",
    "\n",
    "First, a little housekeeping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Character Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following shows examples of how to use [codecs](https://docs.python.org/3/library/codecs.html) and [normalize unicode](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize). NB: the example text comes from the article \"[Metal umlat](https://en.wikipedia.org/wiki/Metal_umlaut)\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = \"Rinôçérôse screams ﬂow not unlike an encyclopædia, \\\n",
    "'TECHNICIÄNS ÖF SPÅCE SHIP EÅRTH THIS IS YÖÜR CÄPTÅIN SPEÄKING YÖÜR ØÅPTÅIN IS DEA̋D' to Spın̈al Tap.\"\n",
    "\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable `x` is a *string* in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "repr(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its translation into [ASCII](http://www.asciitable.com/) is unusable by parsers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ascii(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoding as [UTF-8](http://unicode.org/faq/utf_bom.html) doesn't help much:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.encode('utf8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignoring difficult characters is perhaps an even worse strategy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.encode('ascii', 'ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, one can *normalize* text, then encode…"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "\n",
    "unicodedata.normalize('NFKD', x).encode('ascii','ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even before this normalization and encoding, you may need to convert some characters explicitly **before** parsing. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = \"The sky “above” the port … was the color of ‘cable television’ – tuned to the Weather Channel®\"\n",
    "\n",
    "ascii(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the results for that line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unicodedata.normalize('NFKD', x).encode('ascii', 'ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...which still drops characters that may be important for parsing a sentence.\n",
    "\n",
    "So a more advanced approach could be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = x.replace('“', '\"').replace('”', '\"')\n",
    "x = x.replace(\"‘\", \"'\").replace(\"’\", \"'\")\n",
    "x = x.replace('…', '...').replace('–', '-')\n",
    "\n",
    "x = unicodedata.normalize('NFKD', x).encode('ascii', 'ignore').decode('utf-8')\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following function `get_text()` we'll parse the HTML to find all of the `<p/>` tags, then extract the text for those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import traceback\n",
    "\n",
    "def get_text (url):\n",
    "    buf = []\n",
    "    \n",
    "    try:\n",
    "        soup = BeautifulSoup(requests.get(url).text, \"html.parser\")\n",
    "        \n",
    "        for p in soup.find_all(\"p\"):\n",
    "            buf.append(p.get_text())\n",
    "\n",
    "        return \"\\n\".join(buf)\n",
    "    except:\n",
    "        print(traceback.format_exc())\n",
    "        sys.exit(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's grab some text from online sources.\n",
    "We can compare open source licenses hosted on the [Open Source Initiative](https://opensource.org/licenses/) site:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lic = {}\n",
    "lic[\"mit\"] = nlp(get_text(\"https://opensource.org/licenses/MIT\"))\n",
    "lic[\"asl\"] = nlp(get_text(\"https://opensource.org/licenses/Apache-2.0\"))\n",
    "lic[\"bsd\"] = nlp(get_text(\"https://opensource.org/licenses/BSD-3-Clause\"))\n",
    "\n",
    "for sent in lic[\"bsd\"].sents:\n",
    "    print(\">\", sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common use case for natural language work is to compare texts. For example, with those open source licenses we can download their text, parse, then compare [similarity](https://spacy.io/api/doc#similarity) metrics among them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pairs = [\n",
    "    [\"mit\", \"asl\"],\n",
    "    [\"asl\", \"bsd\"],\n",
    "    [\"bsd\", \"mit\"]\n",
    "]\n",
    "\n",
    "for a, b in pairs:\n",
    "    print(a, b, lic[a].similarity(lic[b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is interesting, since the [BSD](https://opensource.org/licenses/BSD-3-Clause) and [MIT](https://opensource.org/licenses/MIT) licenses appear to be the most similar documents.\n",
    "In fact they are closely related.\n",
    "\n",
    "Admittedly, there was some extra text included in each document due to the OSI disclaimer in the footer – but this provides a reasonable approximation for comparing the licenses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Understanding\n",
    "\n",
    "Now let's dive into some of the _spaCy_ features for NLU.\n",
    "Given that we have a parse of a document, from a purely grammatical standpoint we can pull the [noun chunks](https://spacy.io/usage/linguistic-features#noun-chunks), i.e., each of the noun phrases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Steve Jobs and Steve Wozniak incorporated Apple Computer on January 3, 1977, in Cupertino, California.\"\n",
    "doc = nlp(text)\n",
    "\n",
    "for chunk in doc.noun_chunks:\n",
    "    print(chunk.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad. The noun phrases in a sentence generally provide more information content – as a simple filter used to reduce a long document into a more \"distilled\" representation.\n",
    "\n",
    "We can take this approach further and identify [named entities](https://spacy.io/usage/linguistic-features#named-entities) within the text, i.e., the proper nouns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The _displaCy_ library provides an excellent way to visualize named entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"ent\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're working with [knowledge graph](https://www.akbc.ws/) applications and other [linked data](http://linkeddata.org/), your challenge is to construct links between the named entities in a document and other related information for the entities – which is called [entity linking](http://nlpprogress.com/english/entity_linking.html).\n",
    "Identifying the named entities in a document is the first step in this particular kind of AI work.\n",
    "For example, given the text above, one might link the `Steve Wozniak` named entity to a [lookup in DBpedia](http://dbpedia.org/page/Steve_Wozniak)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In more general terms, one can also link _lemmas_ to resources that describe their meanings.\n",
    "For example, in an early section we parsed the sentence `The gorillas just went wild` and were able to show that the lemma for the word `went` is the verb `go`. At this point we can use a venerable project called [WordNet](https://wordnet.princeton.edu/) which provides a lexical database for English – in other words, it's a computable thesaurus.\n",
    "\n",
    "There's a _spaCy_ integration for WordNet called\n",
    "[spacy-wordnet](https://github.com/recognai/spacy-wordnet) by [Daniel Vila Suero](https://twitter.com/dvilasuero), an expert in natural language and knowledge graph work.\n",
    "\n",
    "Then we'll load the WordNet data via NLTK (these things happen):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download(\"wordnet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that _spaCy_ runs as a \"pipeline\" and allows means for customizing parts of the pipeline in use.\n",
    "That's excellent for supporting really interesting workflow integrations in data science work.\n",
    "Here we'll add the `WordnetAnnotator` from the _spacy-wordnet_ project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install spacy-wordnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy_wordnet.wordnet_annotator import WordnetAnnotator\n",
    "\n",
    "print(\"before\", nlp.pipe_names)\n",
    "\n",
    "if \"WordnetAnnotator\" not in nlp.pipe_names:\n",
    "    nlp.add_pipe(WordnetAnnotator(nlp.lang), after=\"tagger\")\n",
    "    \n",
    "print(\"after\", nlp.pipe_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within the English language, some words are infamous for having many possible meanings. For example, click through the results online in a [WordNet](http://wordnetweb.princeton.edu/perl/webwn?s=star&sub=Search+WordNet&o2=&o0=1&o8=1&o1=1&o7=&o5=&o9=&o6=&o3=&o4=&h=) search to find the meanings related to the word `withdraw`.\n",
    "\n",
    "Now let's use _spaCy_ to perform that lookup automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token = nlp(\"withdraw\")[0]\n",
    "token._.wordnet.synsets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token._.wordnet.lemmas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token._.wordnet.wordnet_domains()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, if you're working with knowledge graphs, those \"word sense\" links from WordNet could be used along with graph algorithms to help identify the meanings for a particular word. That can also be used to develop summaries for larger sections of text through a technique called _summarization_.  It's beyond the scope of this tutorial, but an interesting application currently for natural language in industry."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going in the other direction, if you know _a priori_ that a document was about a particular domain or set of topics, then you can constrain the meanings returned from _WordNet_. In the following example, we want to consider NLU results that are within Finance and Banking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "domains = [\"finance\", \"banking\"]\n",
    "sentence = nlp(\"I want to withdraw 5,000 euros.\")\n",
    "\n",
    "enriched_sent = []\n",
    "\n",
    "for token in sentence:\n",
    "    # get synsets within the desired domains\n",
    "    synsets = token._.wordnet.wordnet_synsets_for_domain(domains)\n",
    "    \n",
    "    if synsets:\n",
    "        lemmas_for_synset = []\n",
    "        \n",
    "        for s in synsets:\n",
    "            # get synset variants and add to the enriched sentence\n",
    "            lemmas_for_synset.extend(s.lemma_names())\n",
    "            enriched_sent.append(\"({})\".format(\"|\".join(set(lemmas_for_synset))))\n",
    "    else:\n",
    "        enriched_sent.append(token.text)\n",
    "\n",
    "print(\" \".join(enriched_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That example may look simple but, if you play with the `domains` list, you'll find that the results have a kind of combinatorial explosion when run without reasonable constraints.\n",
    "Imagine having a knowledge graph with millions of elements: you'd want to constrain searches where possible to avoid having every query take days/weeks/months/years to compute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes the problems encountered when trying to understand a text – or better yet when trying to understand a _corpus_ (a dataset with many related texts) – become so complex that you need to visualize it first.\n",
    "Here's an interactive visualization for understanding texts: [scattertext](https://spacy.io/universe/project/scattertext), a product of the genius of [Jason Kessler](https://twitter.com/jasonkessler).\n",
    "To install:\n",
    "\n",
    "```\n",
    "conda install -c conda-forge scattertext\n",
    "```\n",
    "\n",
    "Let's analyze text data from the party conventions during the 2012 US Presidential elections. It may take a minute or two to run, but the results from all that number crunching is worth the wait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scattertext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scattertext as st\n",
    "\n",
    "if \"merge_entities\" not in nlp.pipe_names:\n",
    "    nlp.add_pipe(nlp.create_pipe(\"merge_entities\"))\n",
    "\n",
    "if \"merge_noun_chunks\" not in nlp.pipe_names:\n",
    "    nlp.add_pipe(nlp.create_pipe(\"merge_noun_chunks\"))\n",
    "\n",
    "convention_df = st.SampleCorpora.ConventionData2012.get_data() \n",
    "corpus = st.CorpusFromPandas(convention_df,\n",
    "                             category_col=\"party\",\n",
    "                             text_col=\"text\",\n",
    "                             nlp=nlp).build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have the `corpus` ready, generate an interactive visualization in HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = st.produce_scattertext_explorer(\n",
    "    corpus,\n",
    "    category=\"democrat\",\n",
    "    category_name=\"Democratic\",\n",
    "    not_category_name=\"Republican\",\n",
    "    width_in_pixels=1000,\n",
    "    metadata=convention_df[\"speaker\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll render the HTML – give it a minute or two to load, it's worth the wait..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "from IPython.core.display import display, HTML\n",
    "import sys\n",
    "\n",
    "IN_COLAB = \"google.colab\" in sys.modules\n",
    "print(IN_COLAB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB: use the following cell on Google Colab:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if IN_COLAB:\n",
    "    display(HTML(\"<style>.container { width:98% !important; }</style>\"))\n",
    "    display(HTML(html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB: use the following cell instead on Jupyter in general:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"foo.html\"\n",
    "\n",
    "with open(file_name, \"wb\") as f:\n",
    "    f.write(html.encode(\"utf-8\"))\n",
    "\n",
    "IFrame(src=file_name, width = 1200, height=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine if you had text from the past three years of customer support for a particular product in your organization. Suppose your team needed to understand how customers have been talking about the product? This _scattertext_ library might come in quite handy! You could cluster (k=2) on _NPS scores_ (a customer evaluation metric) then replace the Democrat/Republican dimension with the top two components from the clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "Five years ago, if you’d asked about open source in Python for natural language, a default answer from many people working in data science would've been [NLTK](https://www.nltk.org/).\n",
    "That project includes just about everything but the kitchen sink and has components which are relatively academic.\n",
    "Another popular natural language project is [CoreNLP](https://stanfordnlp.github.io/CoreNLP/) from Stanford.\n",
    "Also quite academic, albeit powerful, though _CoreNLP_ can be challenging to integrate with other software for production use.\n",
    "\n",
    "Then a few years ago everything in this natural language corner of the world began to change.\n",
    "The two principal authors for _spaCy_ -- [Matthew Honnibal](https://twitter.com/honnibal) and [Ines Montani](https://twitter.com/_inesmontani) -- launched the project in 2015 and industry adoption was rapid.\n",
    "They focused on an _opinionated_ approach (do what's needed, do it well, no more, no less) which provided simple, rapid integration into data science workflows in Python, as well as faster execution and better accuracy than the alternatives.\n",
    "Based on those priorities, _spaCy_ become sort of the opposite of _NLTK_.\n",
    "Since 2015, _spaCy_ has consistently focused on being an open source project (i.e., depending on its community for directions, integrations, etc.) and being commercial-grade software (not academic research).\n",
    "That said, _spaCy_ has been quick to incorporate the SOTA advances in machine learning, effectively becoming a conduit for moving research into industry.\n",
    "\n",
    "It's important to note that machine learning for natural language got a big boost during the mid-2000's as Google began to win international language translation competitions.\n",
    "Another big change occurred during 2017-2018 when, following the many successes of _deep learning_, those approaches began to out-perform previous machine learning models.\n",
    "For example, see the [ELMo](https://arxiv.org/abs/1802.05365) work on _language embedding_ by Allen AI, followed by [BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html) from Google, and more recently [ERNIE](https://medium.com/syncedreview/baidus-ernie-tops-google-s-bert-in-chinese-nlp-tasks-d6a42b49223d) by Baidu -- in other words, the search engine giants of the world have gifted the rest of us with a Sesame Street repertoire of open source embedded language models based on deep learning, which is now _state of the art_ (SOTA).\n",
    "Speaking of which, to keep track of SOTA for natural language, keep an eye on [NLP-Progress](http://nlpprogress.com/) and [Papers with Code](https://paperswithcode.com/sota).\n",
    "\n",
    "The use cases for natural language have shifted dramatically over the past two years, after deep learning techniques arose to the fore.\n",
    "Circa 2014, a natural language tutorial in Python might have shown _word count_ or _keyword search_ or _sentiment detection_ where the target use cases were relatively underwhelming.\n",
    "Circa 2019 we're talking about analyzing thousands of documents for vendor contracts in an industrial supply chain optimization ... or hundreds of millions of documents for policy holders of an insurance company, or gazillions of documents regarding financial disclosures.\n",
    "More contemporary natural language work tends to be in NLU, often to support construction of _knowledge graphs,_ and increasingly in NLG where large numbers of similar documents can be summarized at human scale.\n",
    "\n",
    "The [spaCy Universe](https://spacy.io/universe) is a great place to check for deep-dives into particular use cases, and to see how this field is evolving. Some selections from this \"universe\" include:\n",
    "\n",
    " - [Blackstone](https://spacy.io/universe/project/blackstone) – parsing unstructured legal texts\n",
    " - [Kindred](https://spacy.io/universe/project/kindred) – extracting entities from biomedical texts (e.g., Pharma)\n",
    " - [mordecai](https://spacy.io/universe/project/mordecai) – parsing geographic information\n",
    " - [Prodigy](https://spacy.io/universe/project/prodigy) – human-in-the-loop annotation to label datasets\n",
    " - [spacy-raspberry](https://spacy.io/universe/project/spacy-raspberry) – Raspberry PI image for running _spaCy_ and deep learning on edge devices\n",
    " - [Rasa NLU](https://spacy.io/universe/project/rasa) – Rasa integration for voice apps\n",
    "\n",
    "Also, a couple super new items to mention:\n",
    "\n",
    "  - [spacy-pytorch-transformers](https://explosion.ai/blog/spacy-pytorch-transformers) to fine tune (i.e., use _transfer learning_ with) the Sesame Street characters and friends: BERT, GPT-2, XLNet, etc.\n",
    "  - [spaCy IRL 2019](https://irl.spacy.io/2019/) conference – check out videos from the talks!\n",
    "\n",
    "There's so much more that can be done with _spaCy_ – hopefully this tutorial provides an introduction. We wish you all the best in your natural language work."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}