06_plotting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printed copies of *Elements of Data Science* are available now, with a **full color interior**, from [Lulu.com](https://www.lulu.com/shop/allen-downey/elements-of-data-science/paperback/product-9dyrwn.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "[Click here to run this notebook on Colab](https://colab.research.google.com/github/AllenDowney/ElementsOfDataScience/blob/v1/06_plotting.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print(\"Downloaded \" + str(local))\n",
    "    return filename\n",
    "\n",
    "download('https://raw.githubusercontent.com/AllenDowney/ElementsOfDataScience/v1/utils.py')\n",
    "\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chapter presents ways to create figures and graphs, more generally called **data visualizations**.  As examples, we'll generate three figures:\n",
    "\n",
    "* We'll replicate a figure from the Pew Research Center that shows changes in religious affiliation in the United States over time.\n",
    "\n",
    "* We'll replicate a figure from *The Economist* that shows the prices of sandwiches in Boston and London (we saw this data back in Chapter 3).\n",
    "\n",
    "* We'll make a plot to test Zipf's law, which describes the relationship between word frequencies and their ranks.\n",
    "\n",
    "With the tools in this chapter, you can generate a variety of simple graphs.  We will see more visualization tools in later chapters.\n",
    "But before we get started with plotting, we need a new feature: keyword arguments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keyword Arguments\n",
    "\n",
    "When you call most functions, you have to provide values.\n",
    "For example, when you call `np.exp`, you provide a number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.exp(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you call `np.power`, you provide two numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.power(10, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values you provide are called **arguments**.\n",
    "Specifically, the values in these examples are **positional arguments** because their position determines how they are used.\n",
    "In the second example, `power` computes `10` to the sixth power, not `6` to the tenth power -- because of the order of the arguments.\n",
    "\n",
    "Many functions also take **keyword arguments**, which are identified by name.\n",
    "For example, we have previously used `int` to convert a string to an integer.\n",
    "Here's how we use it with a string as a positional argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "int('21')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `int` assumes that the number is in base 10.  But you can provide a keyword argument that specifies a different base.\n",
    "For example, the string `'21'`, interpreted in base 8, represents the number `2 * 8 + 1 = 17`.  Here's how we do this conversion using `int`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "int('21', base=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The integer value `8` is a keyword argument, with the keyword `base`.\n",
    "Specifying a keyword argument looks like an assignment statement, but it does not create a new variable.\n",
    "And when you provide a keyword argument, you don't choose the variable name -- it is specified by the function.\n",
    "If you provide a name that is not specified by the function, you get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "int('123', bass=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** The `print` function takes a keyword argument called `end` that specifies the character it prints at the end of the line.\n",
    "By default, `end` is the newline character, `\\n`.\n",
    "So if you call `print` more than once, the outputs normally appear on separate lines, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in [1, 2, 3]:\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify the previous example so the outputs appear on one line with spaces between them.\n",
    "Then modify it to print an open bracket at the beginning and a close bracket and newline at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphing Religious Affiliation\n",
    "\n",
    "Now we're ready to make some graphs.\n",
    "In October 2019 the Pew Research Center published \"In U.S., Decline of Christianity Continues at Rapid Pace\".\n",
    "It includes this figure, which shows changes in religious affiliation among adults in the U.S. over the previous 10 years.\n",
    "\n",
    "<img src=\"https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/figs/pew_religion_figure1.png\" width=\"350\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "You can read the article at <https://www.pewresearch.org/religion/2019/10/17/in-u-s-decline-of-christianity-continues-at-rapid-pace/>.\n",
    "The data are from a table in [this PDF document](https://www.pewresearch.org/religion/wp-content/uploads/sites/7/2019/11/Detailed-tables-for-upload-11.11.19.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an exercise, we'll replicate this figure.\n",
    "It shows results from two sources, Religious Landscape Studies and Pew Research Political Surveys.\n",
    "The political surveys provide data from more years, so we'll focus on that.\n",
    "\n",
    "The data from the figure are available from Pew Research, but they are in a PDF document.  It is sometimes possible to extract data from PDF documents, but for now we'll enter the data by hand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "You can download the data from <https://www.pewforum.org/wp-content/uploads/sites/7/2019/10/Detailed-Tables-v1-FOR-WEB.pdf>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "year = [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "christian = [77, 76, 75, 73, 73, 71, 69, 68, 67, 65]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "unaffiliated = [17, 17, 19, 19, 20, 21, 24, 23, 25, 26]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The library we'll use for plotting is Matplotlib -- more specifically, we'll use a part of it called Pyplot, which we'll import with the shortened name `plt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pyplot provides a function called `plot` that makes a line plot.  It takes two sequences as arguments, the `x` values and the `y` values.  The sequences can be tuples, lists, or arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(year, christian);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The semi-colon at the end of the line prevents the return value from `plot`, which is an object representing the line, from being displayed.\n",
    "\n",
    "If you plot multiple lines in a single cell, they appear on the same axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(year, christian)\n",
    "plt.plot(year, unaffiliated);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting them on the same axes makes it possible to compare them directly.\n",
    "However, notice that Pyplot chooses the range for the axes automatically.\n",
    "In this example the y-axis starts around 15, not zero.\n",
    "\n",
    "As a result, it provides a misleading picture, making the ratio of the two lines look bigger than it really is.\n",
    "We can set the limits of the y-axis using the function `plt.ylim` -- the arguments are the lower bound and the upper bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(year, christian)\n",
    "plt.plot(year, unaffiliated)\n",
    "\n",
    "plt.ylim(0, 80);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better, but this graph is missing some important elements: labels for the axes, a title, and a legend. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decorating the Axes\n",
    "\n",
    "To label the axes and add a title, we'll use Pyplot functions `xlabel`, `ylabel`, and `title`.  All of them take strings as arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(year, christian)\n",
    "plt.plot(year, unaffiliated)\n",
    "\n",
    "plt.ylim(0, 80)\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('% of adults')\n",
    "plt.title('Religious affiliation of U.S adults');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add a legend, first we add a label to each line, using the keyword argument `label`.\n",
    "Then we call `plt.legend` to create the legend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(year, christian, label='Christian')\n",
    "plt.plot(year, unaffiliated, label='Unaffiliated')\n",
    "\n",
    "plt.ylim(0, 80)\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('% of adults')\n",
    "plt.title('Religious affiliation of U.S adults')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The legend shows the labels we provided when we created the lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** The original figure plots lines between the data points, but it also plots markers showing the location of each data point.  It is good practice to include these markers, especially if data are not available for every year.\n",
    "\n",
    "Modify the previous example to include a keyword argument `marker` with the string value `'.'`, which indicates that you want to plot small circles as markers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** In the original figure, the line labeled `'Christian'` is red and the line labeled `'Unaffiliated'` is gray.\n",
    "\n",
    "Find the online documentation of `plt.plot`, or ask a virtual assistant like ChatGPT, and figure out how to use keyword arguments to specify colors.\n",
    "Choose colors to (roughly) match the original figure.\n",
    "\n",
    "The `legend` function takes a keyword argument that specifies the location of the legend.  Read the documentation of this function and move the legend to the center left of the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting Sandwich Prices\n",
    "\n",
    "In Chapter 3 we used data from an article in *The Economist* comparing sandwich prices in Boston and London: \"Why Americans pay more for lunch than Britons do\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "You can read the article at <https://www.economist.com/finance-and-economics/2019/09/07/why-americans-pay-more-for-lunch-than-britons-do>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The article includes this graph showing prices of several sandwiches in the two cities:\n",
    "\n",
    "<img src=\"https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/figs/20190907_FNC941.png\"  width=\"300\">\n",
    "\n",
    "As an exercise, let's see if we can replicate this figure.\n",
    "Here's the data from the article again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "name_list = [\n",
    "    'Lobster roll',\n",
    "    'Chicken caesar',\n",
    "    'Bang bang chicken',\n",
    "    'Ham and cheese',\n",
    "    'Tuna and cucumber',\n",
    "    'Egg'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_price_list = [9.99, 7.99, 7.49, 7, 6.29, 4.99]\n",
    "london_price_list = [7.5, 5, 4.4, 5, 3.75, 2.25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous section we plotted percentages on the y-axis versus time on the x-axis.\n",
    "Now we want to plot the sandwich names on the y-axis and the prices on the x-axis.\n",
    "Here's how:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(boston_price_list, name_list)\n",
    "plt.xlabel('Price in USD');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default Pyplot connects the points with lines, but in this example the lines don't make sense because the sandwich names are discrete -- that is, there are no intermediate points between an egg sandwich and a tuna sandwich.\n",
    "We can remove the lines and add markers with the keywords `linestyle` and `marker`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(boston_price_list, name_list, linestyle='', marker='o')\n",
    "plt.xlabel('Price in USD');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can do the same thing more concisely by providing a **format string** as a positional argument.\n",
    "In the following examples, `'o'` indicates a circle marker and `'s'` indicates a square.\n",
    "You can read the documentation of `plt.plot` to learn more about format strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(boston_price_list, name_list, 'o')\n",
    "plt.plot(london_price_list, name_list, 's')\n",
    "\n",
    "plt.xlabel('Price in USD')\n",
    "plt.title('Pret a Manger prices in Boston and London');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to approximate the colors in the original figure, we can use the strings `'C3'` and `'C0'`, which specify colors from the default color sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(boston_price_list, name_list, 'o', color='C3')\n",
    "plt.plot(london_price_list, name_list, 's', color='C0')\n",
    "\n",
    "plt.xlabel('Price in USD')\n",
    "plt.title('Pret a Manger prices in Boston and London');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To connect the dots with lines, we'll use `plt.hlines`, which draws horizontal lines.  It takes three arguments: a sequence of values on the y-axis, which are the sandwich names in this example, and two sequences of values on the x-axis, which are the London prices and Boston prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hlines(name_list, london_price_list, boston_price_list, color='gray')\n",
    "\n",
    "plt.plot(boston_price_list, name_list, 'o', color='C3')\n",
    "plt.plot(london_price_list, name_list, 's', color='C0')\n",
    "\n",
    "plt.xlabel('Price in USD')\n",
    "plt.title('Pret a Manger prices in Boston and London');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** To finish off this example, add a legend that identifies the London and Boston prices.  Remember that you have to add a `label` keyword each time you call `plt.plot`, and then call `plt.legend`.\n",
    "\n",
    "Notice that the sandwiches in our figure are in the opposite order of the sandwiches in the original figure.  There is a Pyplot function that inverts the y-axis; see if you can find it and use it to reverse the order of the sandwich list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zipf's Law\n",
    "\n",
    "In almost any book, in almost any language, if you count the number of unique words and the number of times each word appears, you will find a remarkable pattern: the most common word appears twice as often as the second most common -- at least approximately -- three times as often as the third most common, and so on.\n",
    "\n",
    "In general, if we sort the words in descending order of frequency, there is an inverse relationship between the rank of the words -- first, second, third, etc. -- and the number of times they appear.\n",
    "This observation was most famously made by George Kingsley Zipf, so it is called Zipf's law.\n",
    "\n",
    "To see if this law holds for the words in *War and Peace*, we'll make a Zipf plot, which shows:\n",
    "\n",
    "* The frequency of each word on the y-axis, and\n",
    "\n",
    "* The rank of each word on the x-axis, starting from 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "When you run the following cell, it checks to see whether you already have a file named `2600-0.txt`, which is the name of the file that contains the text of *War and Peace*.\n",
    "If not, it copies the file to your computer.\n",
    "The file is [originally from Project Gutenberg](https://www.gutenberg.org/files/2600/2600-0.txt), but I put a slightly cleaned up version of it in the repository for this book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "download('https://github.com/AllenDowney/ElementsOfDataScience/raw/v1/data/2600-0.txt');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous chapter, we looped through the book and made a string that contains all punctuation characters.\n",
    "Here are the results, which we will need again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_punctuation = ',.-:[#]*/“’—‘!?”;()%@'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following program reads through the book and makes a dictionary that maps from each word to the number of times it appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# On Windows, it might be necessary to specify the encoding \n",
    "\n",
    "# fp = open('2600-0.txt', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "fp = open('2600-0.txt')\n",
    "for line in fp:\n",
    "    if line.startswith('***'):\n",
    "        break\n",
    "\n",
    "unique_words = {}\n",
    "for line in fp:\n",
    "    if line.startswith('***'):\n",
    "        break\n",
    "        \n",
    "    for word in line.split():\n",
    "        word = word.lower()\n",
    "        word = word.strip(all_punctuation)\n",
    "        if word in unique_words:\n",
    "            unique_words[word] += 1\n",
    "        else:\n",
    "            unique_words[word] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `unique_words`, the keys are words and the values are their frequencies.  We can use the `values` function to get the values from the dictionary.  The result has the type `dict_values`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "freqs = unique_words.values()\n",
    "type(freqs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we plot them, we have to sort them, but the `sort` function doesn't work with `dict_values`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%expect AttributeError\n",
    "\n",
    "freqs.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `list` to make a list of frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq_list = list(unique_words.values())\n",
    "type(freq_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can use `sort`.  By default it sorts in ascending order, but we can pass a keyword argument to reverse the order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq_list.sort(reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, for the ranks, we need a sequence that counts from 1 to `n`, where `n` is the number of elements in `freq_list`.  We can use the `range` function, which returns a value with type `range`.\n",
    "As a small example, here's the range from 1 to 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "range(1, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there's a catch.  If we use the range to make a list, we see that \"the range from 1 to 5\" includes 1, but it doesn't include 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(range(1, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That might seem strange, but it is often more convenient to use `range` when it is defined this way, rather than what might seem like the more natural way.\n",
    "Anyway, we can get what we want by increasing the second argument by one:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Read about the different ways to define `range`, and their pros and cons, at <https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(range(1, 6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, finally, we can make a range that represents the ranks from `1` to `n`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = len(freq_list)\n",
    "ranks = range(1, n+1)\n",
    "ranks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can plot the frequencies versus the ranks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(ranks, freq_list)\n",
    "\n",
    "plt.xlabel('Rank')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title(\"War and Peace and Zipf's law\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to Zipf's law, these frequencies should be inversely proportional to the ranks.\n",
    "If that's true, we can write:\n",
    "\n",
    "$f = k / r$\n",
    "\n",
    "where $r$ is the rank of a word, $f$ is its frequency, and $k$ is an unknown constant of proportionality.\n",
    "If we take the logarithm of both sides, we get\n",
    "\n",
    "$\\log f = \\log k - \\log r$\n",
    "\n",
    "This equation implies that if we plot $f$ versus $r$ on a log-log scale, we expect to see a straight line with intercept at $\\log k$ and slope $-1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logarithmic Scales\n",
    "\n",
    "We can use `plt.xscale` to plot the x-axis on a log scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(ranks, freq_list)\n",
    "\n",
    "plt.xlabel('Rank')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title(\"War and Peace and Zipf's law\")\n",
    "plt.xscale('log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `plt.yscale` to plot the y-axis on a log scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(ranks, freq_list)\n",
    "\n",
    "plt.xlabel('Rank')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title(\"War and Peace and Zipf's law\")\n",
    "plt.xscale('log')\n",
    "plt.yscale('log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is not quite a straight line, but it is close.  We can get a sense of the slope by connecting the end points with a line.\n",
    "First, we'll select the first and last elements from `xs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = ranks[0], ranks[-1]\n",
    "xs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the first and last elements from `ys`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "ys = freq_list[0], freq_list[-1]\n",
    "ys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And plot a line between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(xs, ys, color='gray')\n",
    "plt.plot(ranks, freq_list)\n",
    "\n",
    "plt.xlabel('Rank')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title(\"War and Peace and Zipf's law\")\n",
    "plt.xscale('log')\n",
    "plt.yscale('log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The slope of this line is the \"rise over run\", that is, the difference on the y-axis divided by the difference on the x-axis.\n",
    "We can compute the rise using `np.log10` to compute the log base 10 of the first and last values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log10(ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can use `np.diff` to compute the difference between the elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "rise = np.diff(np.log10(ys))\n",
    "rise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Use `log10` and `diff` to compute the run, that is, the difference on the x-axis.  Then divide the rise by the run to get the slope of the grey line.\n",
    "Is it close to $-1$, as Zipf's law predicts?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This chapter introduces Pyplot, which is part of the Matplotlib library.\n",
    "We used to replicate two figures and make a Zipf plot.\n",
    "These examples demonstrate the most common elements of data visualization, including lines and markers, values and labels on the axes, a legend and a title.\n",
    "The Zipf plot also shows the power of plotting data on logarithmic scales."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "*Elements of Data Science*\n",
    "\n",
    "Copyright 2021 [Allen B. Downey](https://allendowney.com)\n",
    "\n",
    "License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}