FullAccessingDataFrames.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing elements in Pandas DataFrames\n",
    "\n",
    "For me, even though I have used Pandas off and on over many years, I still occasionally get confused about the syntax for accessing and modifying elements in Pandas DataFrames!\n",
    "\n",
    "**I feel like the basic problem is that some of the Pandas syntax is too flexible**\n",
    "\n",
    "- Similar notation lets you do multiple things!\n",
    "- There are multiple ways of doing the same thing!\n",
    "\n",
    "**So, what I'll try to do is point out the confusion points, or ambiguities in the syntax, and advise on what methods to avoid.**\n",
    "\n",
    "For a very complete reference, see the\n",
    "[Indexing and Selecting Data](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html)\n",
    "section of the Pandas User's Guide.\n",
    "\n",
    "*As Alex Olteanu does in his blog post \n",
    "[How to make 538 plots](https://www.dataquest.io/blog/making-538-plots/), \n",
    "\"We’ll work with data describing the percentages of Bachelors conferred to women in the US \n",
    "from 1970 to 2011. We’ll use a dataset compiled by data scientist \n",
    "[Randal Olson](http://www.randalolson.com/2014/06/14/percentage-of-bachelors-degrees-conferred-to-women-by-major-1970-2012/), \n",
    "who collected the data from the \n",
    "[National Center for Education Statistics](https://nces.ed.gov/about/).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load DataFrame from CSV – integer default index\n",
    "\n",
    "If we load data into a DataFrame from a CSV (comma-separated value) text file, Pandas will automatically create rows labels, called an `index`, for the DataFrame, consisting of sequential integers.\n",
    "\n",
    "The `.head()` DataFrame (and Series) function by default returns the first five rows. You can put a number in the parentheses to specify the number of rows to return. You can also use the `.tail()` function to see the last rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1970</td>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1971</td>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1972</td>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1973</td>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1974</td>\n",
       "      <td>14.074623</td>\n",
       "      <td>16.204850</td>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "      <td>52.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Agriculture   Business  Engineering  Health  Psychology\n",
       "0  1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1  1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "2  1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "3  1973     9.653602  12.804602          1.6    77.4        50.4\n",
       "4  1974    14.074623  16.204850          2.2    77.9        52.6"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/women_percent_deg_usa_subset.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All you have to do is put an integer in the parentheses to see that many rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>2009</td>\n",
       "      <td>48.667224</td>\n",
       "      <td>48.840474</td>\n",
       "      <td>16.8</td>\n",
       "      <td>85.1</td>\n",
       "      <td>77.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>2010</td>\n",
       "      <td>48.730042</td>\n",
       "      <td>48.757988</td>\n",
       "      <td>17.2</td>\n",
       "      <td>85.0</td>\n",
       "      <td>77.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>2011</td>\n",
       "      <td>50.037182</td>\n",
       "      <td>48.180418</td>\n",
       "      <td>17.5</td>\n",
       "      <td>84.8</td>\n",
       "      <td>76.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Year  Agriculture   Business  Engineering  Health  Psychology\n",
       "39  2009    48.667224  48.840474         16.8    85.1        77.1\n",
       "40  2010    48.730042  48.757988         17.2    85.0        77.0\n",
       "41  2011    50.037182  48.180418         17.5    84.8        76.7"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And even though the `index` looks like integers, it's really a special type of `Index` class under the hood. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=42, step=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read CSV specifying index column\n",
    "\n",
    "For this tutorial, though, **I want it to be clear how Pandas behaves when we use integers versus row labels for selecting DataFrame elements**, so I'm going to explicitly tell Pandas to use the Year column from the data as the `index`.\n",
    "\n",
    "*Note: I could have just used the `df.set_index('Year')` method on the previous DataFrame, but I wanted to show you how to do it all in one step during the load.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>14.074623</td>\n",
       "      <td>16.204850</td>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "      <td>52.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4\n",
       "1974    14.074623  16.204850          2.2    77.9        52.6"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/women_percent_deg_usa_subset.csv', index_col='Year')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index values\n",
    "\n",
    "This time the Index has a name, and is not just a sequential range of integers, but a list of the specific integers present in the Year column of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980,\n",
       "            1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991,\n",
       "            1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,\n",
       "            2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011],\n",
       "           dtype='int64', name='Year')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column names\n",
    "\n",
    "- We can access the column names with the `df.columns` method\n",
    "- That also happens to return a Pandas `Index` – this time with a data type (dtype) of `'object'`, which is how Pandas deals with strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Agriculture', 'Business', 'Engineering', 'Health', 'Psychology'], dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Single-axis indexing/selecting\n",
    "\n",
    "## Square brackets notation: `df[]`\n",
    "\n",
    "**One of the most confusing things about Pandas is how many different things you can do with a set of square brackets after the DataFrame variable name!**\n",
    "\n",
    "From what I have gathered, this single square brackets `df[]` notation was the original Pandas syntax, but in trying to make it flexible, it became confusing to write and read, and they eventually added the more explicit `df.loc[]` and `df.iloc[]` methods you'll see in a bit. \n",
    "\n",
    "*It's still quite handy to use the single brackets for grabbing a single DataFrame column, but for other uses I would recommend the more explicit methods!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `df[]` with name inside for a single column\n",
    "\n",
    "Like with a Python dictionary, you can use the (quoted) name of a column in square brackets to grab a single column out of a DataFrame. The column is returned as a Pandas `Series`, which is the standard 1D data structure, consisting of an `index` and the column's values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Year\n",
       "1970    77.1\n",
       "1971    75.5\n",
       "1972    76.9\n",
       "1973    77.4\n",
       "1974    77.9\n",
       "Name: Health, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Health'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `df[]` with list inside for multiple columns\n",
    "\n",
    "If you instead put a list of column names inside the square brackets, Pandas will return an ordered set of those columns in a `DataFrame`. \n",
    "\n",
    "**This looks a little weird because a list also has square brackets around it!** \n",
    "\n",
    "And, if you try to use this notation to assign just those columns to another variable, there is a problem you'll see below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Business</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>9.064439</td>\n",
       "      <td>77.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>9.503187</td>\n",
       "      <td>75.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>10.558962</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>12.804602</td>\n",
       "      <td>77.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>16.204850</td>\n",
       "      <td>77.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Business  Health\n",
       "Year                   \n",
       "1970   9.064439    77.1\n",
       "1971   9.503187    75.5\n",
       "1972  10.558962    76.9\n",
       "1973  12.804602    77.4\n",
       "1974  16.204850    77.9"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['Business','Health']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This format isn't great for assignment – **SettingWithCopyWarning**\n",
    "\n",
    "If you use this \"list inside of square brackets\" notation and assign to a new variable as a way to just keep a couple columns of your DataFrame, you will run into a **SettingWithCopyWarning** if you try to change that new DataFrame. \n",
    "\n",
    "*Because of some Pandas inner-workings, when you slice or index into a DataFrame it's not actually clear whether it will create a \"view\" into the original DataFrame or a copy of the data!* \n",
    "\n",
    "- **If you are assigning a subset of a DataFrame to a new variable**, with the intention of creating a copy that you'll work on independently of the original, **create an explicit copy by chaining the `.copy()` function to the end!**\n",
    "- If you're trying to assign values to a subset of your original DataFrame, **use the `df.loc[,]` notation shown below.**\n",
    "\n",
    "- Additional learning resources:\n",
    "    - [The best explanation of the SettingWithCopyWarning I've seen](https://www.dataquest.io/blog/settingwithcopywarning/)\n",
    "    - [A complicated explanation in the Pandas documentation](https://pandas.pydata.org/pandas-docs/version/0.22/indexing.html#indexing-view-versus-copy)\n",
    "\n",
    "Here's an example of when the problem arises. We try to put a couple columns into a new DataFrame, and then change the new DataFrame by adding an additional column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-10-2f3f93fea47e>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_temp['ratio'] = df['Business']/df['Health']\n"
     ]
    }
   ],
   "source": [
    "df_temp = df[['Business','Health']]\n",
    "df_temp['ratio'] = df['Business']/df['Health']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we make an explicit `.copy()` of the data, we don't have a problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_temp = df[['Business','Health']].copy()\n",
    "df_temp['ratio'] = df['Business']/df['Health']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `df[]` with integer \"slice\" notation\n",
    "\n",
    "**Here's one place where things get screwy!**\n",
    "\n",
    "If you put a range of ordered integers into the square brackets, using Python's \"slice notation\" with a colon between the integers, Pandas will return a set of **rows** from the DataFrame, including all the columns!!\n",
    "\n",
    "Note that, just like with a Python list, what's returned is inclusive of the initial integer index, but excludes the final value.\n",
    "\n",
    "### *I would advise against using this notation!*\n",
    "\n",
    "*There are other ways of doing the same thing, and if you don't ever use it, maybe you'll forget it exists!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `df[]` with boolean series for multiple rows, all columns\n",
    "\n",
    "*Another potentially confusing notation, but in this case, convenient enough that you'll see it quite often. Note that you can easily use the `df.loc[,:]` notation below to do the same thing, which is somewhat more clear and readable.*\n",
    "\n",
    "**It's very common to want all the rows from your DataFrame which pass a certain test**, or set of criteria. You can write the test itself in a very straightforward way, returning a `Series` of True/False boolean values.\n",
    "\n",
    "Here we'll test which rows, or years, of the Business category had less than 15% women."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Year\n",
       "1970     True\n",
       "1971     True\n",
       "1972     True\n",
       "1973     True\n",
       "1974    False\n",
       "1975    False\n",
       "1976    False\n",
       "1977    False\n",
       "1978    False\n",
       "1979    False\n",
       "1980    False\n",
       "1981    False\n",
       "1982    False\n",
       "1983    False\n",
       "1984    False\n",
       "1985    False\n",
       "1986    False\n",
       "1987    False\n",
       "1988    False\n",
       "1989    False\n",
       "1990    False\n",
       "1991    False\n",
       "1992    False\n",
       "1993    False\n",
       "1994    False\n",
       "1995    False\n",
       "1996    False\n",
       "1997    False\n",
       "1998    False\n",
       "1999    False\n",
       "2000    False\n",
       "2001    False\n",
       "2002    False\n",
       "2003    False\n",
       "2004    False\n",
       "2005    False\n",
       "2006    False\n",
       "2007    False\n",
       "2008    False\n",
       "2009    False\n",
       "2010    False\n",
       "2011    False\n",
       "Name: Business, dtype: bool"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Business'] < 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matching rows from conditional (boolean Series) in square brackets\n",
    "\n",
    "If you put that conditional test statement in square brackets after the DataFrame variable name, Pandas will return all the columns in all the rows that returned True in the test's boolean Series. As we saw above, only four rows pass the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['Business'] < 15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More complicated conditionals\n",
    "\n",
    "You can combine multiple conditions if you put them in parentheses and put a logical operator between.\n",
    "\n",
    "- `&` = \"and\"\n",
    "- `|` = \"or\"  *(It's called a \"pipe\" character, and it's `shift-\\` above the Enter/Return key)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>50.037182</td>\n",
       "      <td>48.180418</td>\n",
       "      <td>17.5</td>\n",
       "      <td>84.8</td>\n",
       "      <td>76.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4\n",
       "2011    50.037182  48.180418         17.5    84.8        76.7"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[(df['Agriculture'] <= 10) | (df['Agriculture'] > 50)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Multi-axis indexing/selecting\n",
    "\n",
    "So far we've been returning either all rows of a single column, or all columns from rows matching a condition. There are two simple ways of selecting along rows and columns simultaneously: \n",
    "\n",
    "- The first based on \"labels\" like column names and index values *(preferred, when possible)*\n",
    "- The second based on integer position along rows and columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `df.loc[row,col]` for label-based, multi-axis indexing\n",
    "\n",
    "The `df.loc[row,col]` method let's you select first along rows, and then along columns, in both directions simultaneoulsy using row and column \"labels\", which are the row index and column names. You can use [(quoting from the documentation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#different-choices-for-indexing)\n",
    "\n",
    "- A single label, e.g. 5 or 'a' (Note that 5 is interpreted as a label of the index. This use is not an integer position along the index.).\n",
    "- A list or array of labels ['a', 'b', 'c'].\n",
    "- A slice object with labels 'a':'f' *(Note that contrary to usual python slices, both the start and the stop are included, when present in the index! See [Slicing with labels](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-slicing-with-labels) and [Endpoints are inclusive](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#advanced-endpoints-are-inclusive).)*\n",
    "- A boolean array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Row and Column specified to return single value\n",
    "\n",
    "If you give one specific row and column label, you'll get back a single value of the data type contained in that cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30.75938956"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[1980, 'Agriculture']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Row or column list to return a Series or DataFrame\n",
    "\n",
    "- A single list will return a 1D result, so a Series\n",
    "- Two lists will return a 2D results, so a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Business    19.686249\n",
       "Health      78.900000\n",
       "Name: 1975, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[1975, ['Business','Health']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Engineering</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1980</th>\n",
       "      <td>30.759390</td>\n",
       "      <td>10.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>32.703444</td>\n",
       "      <td>14.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>45.057766</td>\n",
       "      <td>18.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture  Engineering\n",
       "Year                          \n",
       "1980    30.759390         10.3\n",
       "1990    32.703444         14.1\n",
       "2000    45.057766         18.4"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[[1980,1990,2000],['Agriculture','Engineering']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slice notation is like giving the bounds of a list\n",
    "\n",
    "Remember that *both the rows and columns of a DataFrame are ordered* (which is different than a Python dictionary). Our DataFrame is in the data order from the CSV file, but things like sorting can change that order. So, when you specify \"slice\" notation to say `start:end`, Pandas will take the order into account, and will give you an empty result if there is nothing between your specified bounds!\n",
    "\n",
    "*You can also leave off the start or end of a slice range to include the beginning or end of the series.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Year\n",
       "1980    83.5\n",
       "1981    84.1\n",
       "1982    84.4\n",
       "1983    84.6\n",
       "1984    85.1\n",
       "1985    85.3\n",
       "Name: Health, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[1980:1985, 'Health']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Engineering    18.4\n",
       "Health         83.5\n",
       "Psychology     77.5\n",
       "Name: 2000, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[2000, 'Engineering':'Psychology']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>47.672754</td>\n",
       "      <td>49.791851</td>\n",
       "      <td>17.9</td>\n",
       "      <td>86.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>46.790300</td>\n",
       "      <td>49.210914</td>\n",
       "      <td>16.8</td>\n",
       "      <td>85.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>47.605026</td>\n",
       "      <td>49.000459</td>\n",
       "      <td>16.8</td>\n",
       "      <td>85.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>47.570834</td>\n",
       "      <td>48.888027</td>\n",
       "      <td>16.5</td>\n",
       "      <td>85.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>48.667224</td>\n",
       "      <td>48.840474</td>\n",
       "      <td>16.8</td>\n",
       "      <td>85.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>48.730042</td>\n",
       "      <td>48.757988</td>\n",
       "      <td>17.2</td>\n",
       "      <td>85.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>50.037182</td>\n",
       "      <td>48.180418</td>\n",
       "      <td>17.5</td>\n",
       "      <td>84.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health\n",
       "Year                                             \n",
       "2005    47.672754  49.791851         17.9    86.0\n",
       "2006    46.790300  49.210914         16.8    85.9\n",
       "2007    47.605026  49.000459         16.8    85.4\n",
       "2008    47.570834  48.888027         16.5    85.2\n",
       "2009    48.667224  48.840474         16.8    85.1\n",
       "2010    48.730042  48.757988         17.2    85.0\n",
       "2011    50.037182  48.180418         17.5    84.8"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[2005:, :'Health']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Non-intuitive result when Index isn't sorted\n",
    "\n",
    "*Be careful! If we sort by the Health column values, the Year index is now not in time order.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>14.074623</td>\n",
       "      <td>16.204850</td>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "      <td>52.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4\n",
       "1974    14.074623  16.204850          2.2    77.9        52.6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_health_sort = df.sort_values(by='Health')\n",
    "df_health_sort.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**So if we try to make a slice selection on the years, we get an empty result!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], Name: Health, dtype: float64)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_health_sort.loc[1970:1972, 'Health']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `:` by itself to specify whole axis\n",
    "\n",
    "For either rows or columns, you can specify a `:` by itself to denote all values along that axis, returned as a `Series` or `DataFrame`, again depending on the dimensionality of the result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Agriculture     4.229798\n",
       "Business        9.064439\n",
       "Engineering     0.800000\n",
       "Health         77.100000\n",
       "Psychology     44.400000\n",
       "Name: 1970, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[1970, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Year\n",
       "1970     4.229798\n",
       "1971     5.452797\n",
       "1972     7.420710\n",
       "1973     9.653602\n",
       "1974    14.074623\n",
       "Name: Agriculture, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:, 'Agriculture'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>14.074623</td>\n",
       "      <td>16.204850</td>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "      <td>52.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4\n",
       "1974    14.074623  16.204850          2.2    77.9        52.6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:,:].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean array from condition for either axis\n",
    "\n",
    "**Note: This is the more explicit (and perhaps more clear) version of the earlier syntax:**\n",
    "\n",
    "`df[df['Business'] < 15]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "      <th>Psychology</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "      <td>44.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "      <td>46.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "      <td>47.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "      <td>50.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering  Health  Psychology\n",
       "Year                                                         \n",
       "1970     4.229798   9.064439          0.8    77.1        44.4\n",
       "1971     5.452797   9.503187          1.0    75.5        46.2\n",
       "1972     7.420710  10.558962          1.2    76.9        47.6\n",
       "1973     9.653602  12.804602          1.6    77.4        50.4"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df['Business'] < 15, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### String/Text functions of Index, Series & DataFrame are under the `.str.` methods\n",
    "\n",
    "Pandas documentation: \n",
    "[Working with text data](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True, False, False])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns.str.contains('i')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Agriculture    45.057766\n",
       "Business       49.803616\n",
       "Engineering    18.400000\n",
       "Name: 2000, dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[2000, df.columns.str.contains('i')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## `df.loc[row,col]` for setting values\n",
    "\n",
    "So far we've been accessing values in DataFrames to read/get what's already there, but **the same methods can be used for setting values on DataFrame subsets**!\n",
    "\n",
    "As was mentioned above, explicitly invoking the `.copy()` method is the best way to ensure you're making a copy of a subset of a DataFrame, if your intention is to work on a copy. This avoids the dreaded *SettingWithCopyWarning*.\n",
    "\n",
    "The `df.loc[,]` method is the preferred way to index into a DataFrame for setting values on a slice of the data, and making sure you're not doing what's called \"Chained indexing\", which often leads to a *SettingWithCopyWarning*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Engineering  Health\n",
       "Year                     \n",
       "1970          0.8    77.1\n",
       "1971          1.0    75.5\n",
       "1972          1.2    76.9\n",
       "1973          1.6    77.4\n",
       "1974          2.2    77.9"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_set = df.loc[:,['Engineering','Health']].copy()\n",
    "df_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>50.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>1.6</td>\n",
       "      <td>77.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Engineering  Health\n",
       "Year                     \n",
       "1970         50.0    50.0\n",
       "1971          1.0    75.5\n",
       "1972          1.2    76.9\n",
       "1973          1.6    77.4\n",
       "1974          2.2    77.9"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_set.loc[1970,:] = 50\n",
    "df_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>50.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>0.0</td>\n",
       "      <td>75.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>0.0</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>0.0</td>\n",
       "      <td>77.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>2.2</td>\n",
       "      <td>77.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Engineering  Health\n",
       "Year                     \n",
       "1970         50.0    50.0\n",
       "1971          0.0    75.5\n",
       "1972          0.0    76.9\n",
       "1973          0.0    77.4\n",
       "1974          2.2    77.9"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_set.loc[df_set['Engineering']<2.0, 'Engineering'] = 0\n",
    "df_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## `df.iloc[row,col]` for integer position-based, multi-axis indexing\n",
    "\n",
    "**I don't use this as often, but it's good to know it exists, in case you want to select certain positions without needing to know the labels. Label-based selection tends to be less error-prone, though!**\n",
    "\n",
    "The `df.iloc[row,col]` method let's you select first along rows, and then along columns, in both directions simultaneoulsy using row and column *integer position (from 0 to length-1 of the axis).* You can use [(quoting from the documentation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#different-choices-for-indexing)\n",
    "\n",
    "- An integer e.g. 5.\n",
    "- A list or array of integers [4, 3, 0].\n",
    "- A slice object with ints 1:7.\n",
    "- A boolean array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single value with two specific positions\n",
    "\n",
    "Notice that we get the value, but we have to look back at our DataFrame to make sure we've gotten what we wanted! Here, as long as we haven't re-sorted `df.iloc[0,0]` happens to be the same as `df.loc[1970,'Agriculture']`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.22979798"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Health</th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Business</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>83.9</td>\n",
       "      <td>14.1</td>\n",
       "      <td>47.200851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1980</th>\n",
       "      <td>83.5</td>\n",
       "      <td>10.3</td>\n",
       "      <td>36.765725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>77.1</td>\n",
       "      <td>0.8</td>\n",
       "      <td>9.064439</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Health  Engineering   Business\n",
       "Year                                \n",
       "1990    83.9         14.1  47.200851\n",
       "1980    83.5         10.3  36.765725\n",
       "1970    77.1          0.8   9.064439"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[[20,10,0], [3,2,1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Location handy when Index is out of order\n",
    "\n",
    "Remember when we sorted the rows by increasing Health percentage, the years ended up out of order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Engineering</th>\n",
       "      <th>Health</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>1.0</td>\n",
       "      <td>75.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>1.2</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>0.8</td>\n",
       "      <td>77.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Engineering  Health\n",
       "Year                     \n",
       "1971          1.0    75.5\n",
       "1972          1.2    76.9\n",
       "1970          0.8    77.1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_health_sort.iloc[:3, 2:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean tests\n",
    "\n",
    "Even if the rows are out of order, we can still test their value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Business</th>\n",
       "      <th>Engineering</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1971</th>\n",
       "      <td>5.452797</td>\n",
       "      <td>9.503187</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1972</th>\n",
       "      <td>7.420710</td>\n",
       "      <td>10.558962</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>4.229798</td>\n",
       "      <td>9.064439</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1973</th>\n",
       "      <td>9.653602</td>\n",
       "      <td>12.804602</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>14.074623</td>\n",
       "      <td>16.204850</td>\n",
       "      <td>2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1975</th>\n",
       "      <td>18.333162</td>\n",
       "      <td>19.686249</td>\n",
       "      <td>3.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Agriculture   Business  Engineering\n",
       "Year                                     \n",
       "1971     5.452797   9.503187          1.0\n",
       "1972     7.420710  10.558962          1.2\n",
       "1970     4.229798   9.064439          0.8\n",
       "1973     9.653602  12.804602          1.6\n",
       "1974    14.074623  16.204850          2.2\n",
       "1975    18.333162  19.686249          3.2"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_health_sort.iloc[df.index <= 1975, df.columns.str.contains('i')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Method you'll see, but don't really need\n",
    "\n",
    "A very good article on more advanced Pandas features where there are multiple ways of doing similar things, and [Ted Petrou's](https://medium.com/@petrou.theodore) \n",
    "opinions on which to use, is \n",
    "[Minimally Sufficient Pandas](https://medium.com/dunder-data/minimally-sufficient-pandas-a8e67f2a2428). \n",
    "\n",
    "*These pieces are taken from that article, but there's a lot more great content there that we don't have time to cover.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting a single column with the \"dot\" notation\n",
    "\n",
    "A very common alternative to selecting a single column with the `df['name']` bracket notation you'll see all the time, is what's called the \"dot\" notation, where you follow the dataframe name by a dot and the column name, `df.name`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>favorite food</th>\n",
       "      <th>age</th>\n",
       "      <th>height</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Jane</th>\n",
       "      <td>NY</td>\n",
       "      <td>Steak</td>\n",
       "      <td>30</td>\n",
       "      <td>165</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Niko</th>\n",
       "      <td>TX</td>\n",
       "      <td>Lamb</td>\n",
       "      <td>2</td>\n",
       "      <td>70</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aaron</th>\n",
       "      <td>FL</td>\n",
       "      <td>Mango</td>\n",
       "      <td>12</td>\n",
       "      <td>120</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Penelope</th>\n",
       "      <td>AL</td>\n",
       "      <td>Apple</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dean</th>\n",
       "      <td>AK</td>\n",
       "      <td>Cheese</td>\n",
       "      <td>32</td>\n",
       "      <td>180</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Christina</th>\n",
       "      <td>TX</td>\n",
       "      <td>Melon</td>\n",
       "      <td>33</td>\n",
       "      <td>172</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cornelia</th>\n",
       "      <td>TX</td>\n",
       "      <td>Beans</td>\n",
       "      <td>69</td>\n",
       "      <td>150</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          state favorite food  age  height  count\n",
       "name                                             \n",
       "Jane         NY         Steak   30     165     10\n",
       "Niko         TX          Lamb    2      70      4\n",
       "Aaron        FL         Mango   12     120      3\n",
       "Penelope     AL         Apple    4      80     12\n",
       "Dean         AK        Cheese   32     180      8\n",
       "Christina    TX         Melon   33     172     99\n",
       "Cornelia     TX         Beans   69     150     44"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df = pd.read_csv('data/min_suff_data.csv', index_col='name')\n",
    "min_suff_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Jane         NY\n",
       "Niko         TX\n",
       "Aaron        FL\n",
       "Penelope     AL\n",
       "Dean         AK\n",
       "Christina    TX\n",
       "Cornelia     TX\n",
       "Name: state, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df['state']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Jane         NY\n",
       "Niko         TX\n",
       "Aaron        FL\n",
       "Penelope     AL\n",
       "Dean         AK\n",
       "Christina    TX\n",
       "Cornelia     TX\n",
       "Name: state, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df.state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Issues with the \"dot\" notation\n",
    "\n",
    "There are three issues with using dot notation. It doesn’t work in the following situations:\n",
    "\n",
    "- When there are spaces in the column name\n",
    "- When the column name is the same as a DataFrame method\n",
    "- When the name of a column you want to access is stored in a variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spaces in the columns name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Jane          Steak\n",
       "Niko           Lamb\n",
       "Aaron         Mango\n",
       "Penelope      Apple\n",
       "Dean         Cheese\n",
       "Christina     Melon\n",
       "Cornelia      Beans\n",
       "Name: favorite food, dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df['favorite food']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-41-01679cbbde06>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-41-01679cbbde06>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    min_suff_df.favorite food\u001b[0m\n\u001b[0m                         ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "min_suff_df.favorite food"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column name is the same as a DataFrame method\n",
    "\n",
    "`count` is the name of one of our columns, and it's easy to access those values with the \"quoted name in square brackets\" notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Jane         10\n",
       "Niko          4\n",
       "Aaron         3\n",
       "Penelope     12\n",
       "Dean          8\n",
       "Christina    99\n",
       "Cornelia     44\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df['count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we try to access that column's values with the \"dot\" notation, there is a problem. The output here will seem confusing, but it's basically saying that `.count` is a method (built-in function) of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method DataFrame.count of           state favorite food  age  height  count\n",
       "name                                             \n",
       "Jane         NY         Steak   30     165     10\n",
       "Niko         TX          Lamb    2      70      4\n",
       "Aaron        FL         Mango   12     120      3\n",
       "Penelope     AL         Apple    4      80     12\n",
       "Dean         AK        Cheese   32     180      8\n",
       "Christina    TX         Melon   33     172     99\n",
       "Cornelia     TX         Beans   69     150     44>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_suff_df.count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The column name is stored in a variable\n",
    "\n",
    "It's not uncommon to want to access a column who's name string has been stored in a variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Jane         165\n",
       "Niko          70\n",
       "Aaron        120\n",
       "Penelope      80\n",
       "Dean         180\n",
       "Christina    172\n",
       "Cornelia     150\n",
       "Name: height, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_name = 'height'\n",
    "min_suff_df[col_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, again, that doesn't work with the \"dot\" notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'DataFrame' object has no attribute 'col_name'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-45-0572c0d94bf8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmin_suff_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcol_name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/pandas/core/generic.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m   5272\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_info_axis\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_can_hold_identifiers_and_holds_name\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5273\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5274\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5275\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5276\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'DataFrame' object has no attribute 'col_name'"
     ]
    }
   ],
   "source": [
    "min_suff_df.col_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lots of Pandas is written with the dot notation. Why?\n",
    "\n",
    "Many tutorials make use of the dot notation to select a single column of data. Why is this done when the brackets seem to be clearly superior? \n",
    "\n",
    "- **It might be because the official documentation contains plenty of examples that use it.**\n",
    "- **It also *uses three fewer characters which entices the very laziest amongst us*.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}