05_dictionaries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printed copies of *Elements of Data Science* are available now, with a **full color interior**, from [Lulu.com](https://www.lulu.com/shop/allen-downey/elements-of-data-science/paperback/product-9dyrwn.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "[Click here to run this notebook on Colab](https://colab.research.google.com/github/AllenDowney/ElementsOfDataScience/blob/v1/05_dictionaries.ipynb) or\n",
    "[click here to download it](https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/05_dictionaries.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print(\"Downloaded \" + str(local))\n",
    "    return filename\n",
    "\n",
    "download('https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/utils.py')\n",
    "\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous chapter we used a `for` loop to read a file and count the words.\n",
    "In this chapter we'll count the number of *unique* words and the number of times each one appears.\n",
    "To do that, we'll use one of Python's most useful features, a **dictionary**.\n",
    "\n",
    "You will also see how to select an element from a sequence (tuple, list, or array).  And you will learn a little about Unicode, which is used to represent letters, numbers, and punctuation for almost every language in the world."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing\n",
    "\n",
    "Suppose you have a variable named `t` that refers to a list or tuple.\n",
    "You can select an element using the **bracket operator**, `[]`.\n",
    "For example, here's a tuple of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = ('zero', 'one', 'two')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select the first element, we put `0` in brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "t[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select the second element, we put `1` in brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "t[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "To select the third element, we put `2` in brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "t[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number in brackets is called an **index** because it indicates which element we want.\n",
    "Tuples and lists use zero-based numbering -- that is, the index of the first element is 0.  Some other programming languages use one-based numbering.\n",
    "\n",
    "The index in brackets can also be a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 1\n",
    "t[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or an expression with variables, values, and operators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "t[i+1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if the index goes past the end of the sequence, you get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect IndexError\n",
    "\n",
    "t[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, the index has to be an integer -- if it is any other type, you get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "t[1.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "t['1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** You can use negative integers as indices.  Try using `-1` and `-2` as indices, and see if you can figure out what they do. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionaries\n",
    "\n",
    "A dictionary is similar to a tuple or list, but in a dictionary, the index can be almost any type, not just an integer.\n",
    "We can create an empty dictionary like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can add elements like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['one'] = 1\n",
    "d['two'] = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the indices are the strings, `'one'` and `'two'`.\n",
    "If you display the dictionary, it shows each index and the corresponding value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of creating an empty dictionary and then adding elements, you can create a dictionary and specify the elements at the same time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {'one': 1, 'two': 2, 'three': 3}\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are talking about dictionaries, an index is usually called a **key**.  In this example, the keys are strings and the corresponding values are integers.\n",
    "A dictionary is also called a **map**, because it represents a correspondence or \"mapping\", between keys and values.\n",
    "So we might say that this dictionary maps from English number names to the corresponding integers.\n",
    "\n",
    "You can use the bracket operator to select an element from a dictionary, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['two']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But don't forget the quotation marks.\n",
    "Without them, Python looks for a variable named `two` and doesn't find one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect NameError\n",
    "\n",
    "d[two]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check whether a particular key is in a dictionary, you can use the `in` operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "'one' in d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "'zero' in d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the word `in` is an operator in Python, you can't use it as a variable name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect SyntaxError\n",
    "\n",
    "in = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each key in a dictionary can only appear once.\n",
    "Adding the same key again has no effect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['one'] = 1\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you can change the value associated with a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['one'] = 100\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can loop through the keys in a dictionary like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in d:\n",
    "    print(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want the keys and the values, one way to get them is to loop through the keys and look up the values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in d:\n",
    "    print(key, d[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can loop through both at the same time, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key, value in d.items():\n",
    "    print(key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `items` method loops through the key-value pairs in the dictionary.\n",
    "Each time through the loop, they are assigned to `key` and `value`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Make a dictionary with the integers `1`, `2`, and `3` as keys and strings as values.  The strings should be the words \"one\", \"two\", and \"three\" or their equivalents in any language you know.\n",
    "\n",
    "Write a loop that prints just the values from the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting Unique Words\n",
    "\n",
    "In the previous chapter we downloaded *War and Peace* from Project Gutenberg and counted the number of lines and words.\n",
    "Now that we have dictionaries, we can also count the number of unique words and the number of times each one appears."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "When you run the following cell, it checks to see whether you already have a file named `2600-0.txt`, which is the name of the file that contains the text of *War and Peace*.\n",
    "If not, it copies the file to your computer.\n",
    "The file is [originally from Project Gutenberg](https://www.gutenberg.org/files/2600/2600-0.txt), but I put a slightly cleaned up version of it in the repository for this book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "download('https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/data/2600-0.txt');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did in the previous chapter, we can read the text of *War and Peace* and count the number of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "fp = open('2600-0.txt')\n",
    "count = 0\n",
    "for line in fp:\n",
    "    count += len(line.split())\n",
    "    \n",
    "count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To count the number of unique words, we'll loop through the words in each line and add them as keys in a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "fp = open('2600-0.txt')\n",
    "unique_words = {}\n",
    "for line in fp:\n",
    "    for word in line.split():\n",
    "        unique_words[word] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the first example we've seen with one loop **nested** inside another.\n",
    "\n",
    "* The outer loop runs through the lines in the file.\n",
    "\n",
    "* The inner loops runs through the words in each line.\n",
    "\n",
    "Each time through the inner loop, we add a word as a key in the dictionary, with the value `1`.\n",
    "If a word that is already in the dictionary appears again, adding it to the dictionary again has no effect.\n",
    "So the dictionary gets only one copy of each unique word in the file.\n",
    "At the end of the loop, we can display the first eight keys like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(unique_words)[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `list` function puts the keys from the dictionary in a list.\n",
    "In the bracket operator, `:8` is a special index called a *slice* that selects the first eight elements.\n",
    "\n",
    "Each word only appears once, so the number of keys is the number of unique words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(unique_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are about 42,000 different words in the book, which is substantially less than the total number of words, about 560,000.\n",
    "But this count is not correct yet, because we have not taken into account capitalization and punctuation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Before we deal with those problems, let's practice with nested loops -- that is, one loop inside another.\n",
    "Suppose you have a list of words, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = ['War', 'and', 'Peace']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a nested loop that iterates through each word in the list, and each letter in each word, and prints the letters on separate lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with Capitalization\n",
    "\n",
    "When we count unique words, we probably want to treat `The` and `the` as the same word.  We can do that by converting all words to lower case, using the `lower` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = 'The'\n",
    "word.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`lower` creates a new string; it does not modify the original string.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can assign the new string back to the existing variable, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = word.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if we can display the new value of `word`, we get the lowercase version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Modify the previous loop so it makes a lowercase version of each word before adding it to the dictionary.  How many unique words are there, if we ignore the difference between uppercase and lowercase?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing Punctuation\n",
    "\n",
    "To remove punctuation from the words, we can use `strip`, which removes characters from the beginning and end of a string.\n",
    "Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = 'abracadabra'\n",
    "word.strip('ab')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, `strip` removes all instances of `a` and `b` from the beginning and end of the word, but not from the middle.\n",
    "Like `lower`, this function makes a new word -- it doesn't modify the original:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To remove punctuation, we can use the `string` library, which provides a variable named `punctuation`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "string.punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`string.punctuation` contains the most common punctuation marks, but as we'll see, not all of them.\n",
    "Nevertheless, we can use it to handle most cases.  Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = \"It's not given to people to judge what's right or wrong.\"\n",
    "\n",
    "for word in line.split():\n",
    "    word = word.strip(string.punctuation)\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`strip` removes the period at the end of `wrong`, but not the apostrophes in `It's`, `don't` and `what's`.\n",
    "That's good, because we want to treat an apostrophe as part of a word.\n",
    "But we have one more problem to solve.\n",
    "Here's another line from the book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = 'anyone, and so you don’t deserve to have them.”'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what happens when we try to remove the punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in line.split():\n",
    "    word = word.strip(string.punctuation)\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The comma after `anyone` is removed, but not the quotation mark at the end of the last word.\n",
    "The problem is that this kind of quotation mark is not in `string.punctuation`, so `strip` doesn't remove it.\n",
    "To fix this problem, we'll use the following loop, which\n",
    "\n",
    "1. Reads the file and builds a dictionary that contains all punctuation marks that appear in the book, then\n",
    "\n",
    "2. It uses the `join` function to concatenate the keys of the dictionary in a single string.\n",
    "\n",
    "You don't have to understand everything about how it works, but I suggest you read it and see how much you can figure out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "\n",
    "fp = open('2600-0.txt')\n",
    "punc_marks = {}\n",
    "for line in fp:\n",
    "    for x in line:\n",
    "        category = unicodedata.category(x)\n",
    "        if category[0] == 'P':\n",
    "            punc_marks[x] = 1\n",
    "        \n",
    "all_punctuation = ''.join(punc_marks)\n",
    "print(all_punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a string containing all of the punctuation characters that appear in the document, in the order they first appear."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Modify the word-counting loop from the previous section to convert words to lower case *and* strip punctuation before adding them to the dictionary.  Now how many unique words are there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "To get you started, here's the code from the previous chapter that skips over the front matter at the beginning and the license at the end, as we did in the previous chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fp = open('2600-0.txt')\n",
    "for line in fp:\n",
    "    if line.startswith('***'):\n",
    "        break\n",
    "\n",
    "count = 0\n",
    "for line in fp:\n",
    "    if line.startswith('***'):\n",
    "        break\n",
    "    count += len(line.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting Word Frequencies\n",
    "\n",
    "In the previous section we counted the number of unique words, but we might also want to know how often each word appears.  Then we can find the most common and least common words in the book.\n",
    "To count the frequency of each word, we'll make a dictionary that maps from each word to the number of times it appears.\n",
    "\n",
    "Here's an example that loops through a string and counts the number of times each letter appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = 'Mississippi'\n",
    "\n",
    "letter_counts = {}\n",
    "for x in word:\n",
    "    if x in letter_counts:\n",
    "        letter_counts[x] += 1\n",
    "    else:\n",
    "        letter_counts[x] = 1\n",
    "        \n",
    "letter_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `if` statement includes a feature we have not seen before, an `else` clause.\n",
    "Here's how it works.\n",
    "\n",
    "1. First, it checks whether the letter, `x`, is already a key in the dictionary, `letter_counts`.\n",
    "\n",
    "2. If so, it runs the first statement, `letter_counts[x] += 1`, which increments the value associated with the letter.\n",
    "\n",
    "3. Otherwise, it runs the second statement, `letter_counts[x] = 1`, which adds `x` as a new key, with the value `1` indicating that we have seen the new letter once.\n",
    "\n",
    "The result is a dictionary that maps from each letter to the number of times it appears.\n",
    "To get the most common letters, we can use a `Counter`, which is similar to a dictionary.  To use it, we have to import a library called `collections`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use `collections.Counter` to convert the dictionary to a `Counter`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = collections.Counter(letter_counts)\n",
    "type(counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Counter` provides a function called `most_common` we can use to get the most common characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter.most_common(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a list of tuples, where each tuple contains a character and count, sorted by count."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Modify the loop from the previous exercise to count the frequency of the words in *War and Peace*. Then print the 20 most common words and the number of times each one appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** You can run `most_common` with no value in parentheses, like this:\n",
    "\n",
    "```\n",
    "word_freq_pairs = counter.most_common()\n",
    "```\n",
    "\n",
    "The result is a list of tuples, with one tuple for every unique word in the book.\n",
    "Use it to answer the following questions:\n",
    "\n",
    "1. How many times does the #1 ranked word appear (that is, the first element of the list)?\n",
    "\n",
    "2. How many times does the #10 ranked word appear?\n",
    "\n",
    "3. How many times does the #100 ranked word appear?\n",
    "\n",
    "4. How many times does the #1000 ranked word appear?\n",
    "\n",
    "5. How many times does the #10000 ranked word appear?\n",
    "\n",
    "Do you see a pattern in the results?  We will explore this pattern more in the next chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Write a loop that counts how many words appear 200 times.  What are they?  How many words appear 100 times, 50 times, and 20 times?\n",
    "\n",
    "**Optional:** If you know how to define a function, write a function that takes a `Counter` and a frequency as arguments, prints all words with that frequency, and returns the number of words with that frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This chapter introduces dictionaries, which are collections of keys and corresponding values.\n",
    "We used a dictionary to count the number of unique words in a file and the number of times each one appears.\n",
    "\n",
    "It also introduces the bracket operator, which selects an element from a list or tuple, or looks up a key in a dictionary and finds the corresponding value.\n",
    "\n",
    "We saw some new methods for working with strings, including `lower` and `strip`.\n",
    "Also, we used the `unicodedata` library to identify characters that are considered punctuation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "*Elements of Data Science*\n",
    "\n",
    "Copyright 2021 [Allen B. Downey](https://allendowney.com)\n",
    "\n",
    "License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}