data-advanced.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(advanced-data)=\n",
    "# **pandas** alternatives, validation, and orchestration"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this chapter, you'll learn about data validation, data orchestration, and some alternatives to the **pandas** package.\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51a55374",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import matplotlib_inline.backend_inline\n",
    "\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")\n",
    "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n",
    "\n",
    "# Set max rows displayed for readability\n",
    "pd.set_option(\"display.max_rows\", 6)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternatives to **pandas**\n",
    "\n",
    "**pandas** isn't the only game in town, not by a long way—though it's by far the best supported and the most fully featured. **pandas** is a good default package to use for data in the vast majority of cases. But it's always good to have options—or, put another way, options have value! Other dataframe libraries may have a syntax that you prefer (**datatable**'s syntax is popular with some) or provide a speed-up in certain situations, for example when working with large datasets (**polars** promises to breeze through big data far faster than pandas can).\n",
    "\n",
    "If you're specifically interested in how different dataframe options perform on increasingly large datasets, take a look at the benchmarks by the folks who make DuckDB [here](https://duckdblabs.github.io/db-benchmark/).\n",
    "\n",
    "If you only choose one **pandas** alternative, we recommend **polars**. It brings the benefits of speed and a consistent API.\n",
    "\n",
    "Here's a quick run-through of some alternatives to **pandas**:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Polars\n",
    "\n",
    "#### Polars\n",
    "\n",
    "[**Polars**](https://www.pola.rs/) is one of the fastest dataframe libraries in any language. It also uses Apache Arrow as backend. It currently consists of an 'eager' (for datasets smaller than approximately a few GB) interface that's very similar to **pandas** and a 'lazy' interface (don't worry if you don't know what that means, it's a big data thing) that is somewhat similar to spark (a big data tool). **Polars** is built on the Rust language. It's particularly effective at merging datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tidypolars\n",
    "\n",
    "[**tidypolars**](https://tidypolars.readthedocs.io/) combines the syntax of the **dplyr**, from the R statistical language, with the best-in-class speed (for any language, as of the time of writing) of Python package [**polars**](https://www.pola.rs/). Here's an example of a typical command to give you a flavour of what the syntax looks like:\n",
    "\n",
    "```python\n",
    "import tidypolars as tp\n",
    "from tidypolars import col, desc\n",
    "\n",
    "df = tp.Tibble(x = range(3), y = range(3, 6), z = ['a', 'a', 'b'])\n",
    "\n",
    "(\n",
    "    df\n",
    "    .select('x', 'y', 'z')\n",
    "    .filter(col('x') < 4, col('y') > 1)\n",
    "    .arrange(desc('z'), 'x')\n",
    "    .mutate(double_x = col('x') * 2,\n",
    "            x_plus_y = col('x') + col('y'))\n",
    ")\n",
    "```\n",
    "\n",
    "```text\n",
    "┌─────┬─────┬─────┬──────────┬──────────┐\n",
    "│ x   ┆ y   ┆ z   ┆ double_x ┆ x_plus_y │\n",
    "│ --- ┆ --- ┆ --- ┆ ---      ┆ ---      │\n",
    "│ i64 ┆ i64 ┆ str ┆ i64      ┆ i64      │\n",
    "╞═════╪═════╪═════╪══════════╪══════════╡\n",
    "│ 2   ┆ 5   ┆ b   ┆ 4        ┆ 7        │\n",
    "├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤\n",
    "│ 0   ┆ 3   ┆ a   ┆ 0        ┆ 3        │\n",
    "├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤\n",
    "│ 1   ┆ 4   ┆ a   ┆ 2        ┆ 5        │\n",
    "└─────┴─────┴─────┴──────────┴──────────┘\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datar\n",
    "\n",
    "Another library that gets close to **dplylr** syntax is [**datar**](https://github.com/pwwang/datar). It integrates with [**plotnine**](https://plotnine.readthedocs.io/en/stable/) for visualisation."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Validation\n",
    "\n",
    "When working with data, it's useful to have a way to validate that what you're getting is what you *think* you're getting. And so various data validation methods have appeared that will fail loudly and proudly when your data are not as expected. Although most useful in a situation where data are being regularly updated, for example when producing a monthly index, these tools can also be useful if you want to be extra careful with your data in a research paper, if you are scaling up from a sample to a bigger dataset, or if you are updating an existing project.\n",
    "\n",
    "Of course, the most minimal way to do some validation is to ensure that your dataframe has its columns cast as the right data types from the start, and this is good practice in general. That's about *data type*, but these tools go beyond that and deal with values *within* a data type. So rather than just saying it's an integer, a data validation library might say that a column must be an integer between 0 and 100.\n",
    "\n",
    "The normal way that data validation libraries are used is that you specify some expectations you have of the dataset that will be flagged if and when they fail. Let's see some of the tools that are available that do this. As ever, this chapter has benefitted enormously from the documentation of the featured packages."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandera\n",
    "\n",
    "[**Pandera**](https://pandera.readthedocs.io/en/stable/index.html) {cite:ps}`niels_bantilan-proc-scipy-2020` has tight integration with **pandas** (as the name suggests). There are several different ways to specify data validation checks.\n",
    "\n",
    "#### Dataframe style checks\n",
    "\n",
    "It's easiest to show how it works with an example. We'll first create some data that we'd like to validate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandera as pa\n",
    "\n",
    "# data to validate\n",
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"column1\": [1, 4, 0, 10, 9],\n",
    "        \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "        \"column3\": [\"value_1\", \"value_2\", \"value_3\", \"value_2\", \"value_1\"],\n",
    "    }\n",
    ")\n",
    "\n",
    "df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**pandera** (I'm going to spell it as pantera sooner or later) works by defining *data schemas* for each column that say what values should appear in that column. Let's create a schema for this dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define schema\n",
    "schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"column1\": pa.Column(int, checks=pa.Check.le(10)),\n",
    "        \"column2\": pa.Column(float, checks=pa.Check.lt(-1.2)),\n",
    "        \"column3\": pa.Column(\n",
    "            str,\n",
    "            checks=[\n",
    "                pa.Check.str_startswith(\"value_\"),\n",
    "                # define custom checks as functions that take a series as input and\n",
    "                # outputs a boolean or boolean Series\n",
    "                pa.Check(lambda s: s.str.split(\"_\", expand=True).shape[1] == 2),\n",
    "            ],\n",
    "        ),\n",
    "    }\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What did we ask for here? That:\n",
    "\n",
    "- column 1 should be an integer that is less than or equal to 10\n",
    "- column 2 should be a continuous variable that's less than 1.2\n",
    "- column 3 should be a string that begins 'value_' and has a custom check on it that it can be split into chunks based on the single appearance of an underscore.\n",
    "\n",
    "Alright, let's see what happens when we put well-behaved data through the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validated_df = schema(df)\n",
    "print(validated_df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data comes through the schema without incident because the data satisfies all the checks. Now, what about some data that *aren't* well-behaved, but in a single column (here 1)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# data to validate\n",
    "bad_df = pd.DataFrame({\n",
    "    \"column1\": [11, 4],\n",
    "    \"column2\": [-1.3, -1.4],\n",
    "    \"column3\": [\"value_1\", \"value_2\"],\n",
    "})\n",
    "schema(bad_df)\n",
    "```\n",
    "\n",
    "```\n",
    "---------------------------------------------------------------------------\n",
    "SchemaError                               Traceback (most recent call last)\n",
    "Cell In[6], line 6\n",
    "      1 bad_df = pd.DataFrame({\n",
    "      2     \"column1\": [11, 4],\n",
    "      3     \"column2\": [-1.3, -1.4],\n",
    "      4     \"column3\": [\"value_1\", \"value_2\"],\n",
    "      5 })\n",
    "----> 6 schema(bad_df)\n",
    "...\n",
    "...\n",
    "SchemaError:  failed element-wise validator 0:\n",
    "\n",
    "failure cases:\n",
    "   index  failure_case\n",
    "0      0            11\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error message is rather verbose but scroll to the end and we see the cause of it: a 'schema error'. Not only that but we get a report about what failed (column 1), what check it was (0), and even what value triggered the error (11).\n",
    "\n",
    "Use `schema.validate(df, lazy=True)` to get all of the errors rather than just the first one to be triggered.\n",
    "\n",
    "You can also pass hypothesis tests into a schema as a check; there's a nice [example of using a two-sample t-test](https://pandera.readthedocs.io/en/stable/hypothesis.html) in the documentation.\n",
    "\n",
    "#### Data checks as a class\n",
    "\n",
    "The 'dataframe' style of checks is not the only way to specify a schema. If you're au fait with classes, you can also specify checks that way. Here's the same example in that form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandera.typing import Series\n",
    "\n",
    "\n",
    "class Schema(pa.SchemaModel):\n",
    "\n",
    "    column1: Series[int] = pa.Field(le=10)\n",
    "    column2: Series[float] = pa.Field(lt=-1.2)\n",
    "    column3: Series[str] = pa.Field(str_startswith=\"value_\")\n",
    "\n",
    "    @pa.check(\"column3\")\n",
    "    def column_3_check(cls, series: Series[str]) -> Series[bool]:\n",
    "        \"\"\"Check that column3 values have two elements after being split with '_'\"\"\"\n",
    "        return series.str.split(\"_\", expand=True).shape[1] == 2\n",
    "\n",
    "\n",
    "Schema.validate(df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this passes but again it would not have with `bad_df` as defined earlier.\n",
    "\n",
    "It doesn't *have* to be a single column that you're checking at a time, **pandera** can also check data that is the subject of `groupby` operations. Here's an example where data are grouped by a `group` column and then the means of the `height` column are compared:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"height\": [5.6, 6.4, 4.0, 7.1],\n",
    "        \"group\": [\"A\", \"B\", \"A\", \"B\"],\n",
    "    }\n",
    ")\n",
    "\n",
    "schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"height\": pa.Column(\n",
    "            pa.Float, pa.Check(lambda g: g[\"A\"].mean() < g[\"B\"].mean(), groupby=\"group\")\n",
    "        ),\n",
    "        \"group\": pa.Column(pa.String),\n",
    "    }\n",
    ")\n",
    "\n",
    "schema.validate(df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data checks for pipelines\n",
    "\n",
    "You can also specify the validations as function decorators that check the data as it *enters* (`check_input`) or *exits* (`check_output`) a function (or both, with `check_io`). This could be really useful as part of a larger pipeline of data processes. Here's an example (of a dataframe that passes the input checks):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"column1\": [1, 4, 0, 10, 9],\n",
    "        \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "    }\n",
    ")\n",
    "\n",
    "in_schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"column1\": pa.Column(\n",
    "            pa.Int, pa.Check(lambda x: 0 <= x <= 10, element_wise=True)\n",
    "        ),\n",
    "        \"column2\": pa.Column(pa.Float, pa.Check(lambda x: x < -1.2)),\n",
    "    }\n",
    ")\n",
    "\n",
    "\n",
    "@pa.check_input(in_schema)\n",
    "def preprocessor(dataframe):\n",
    "    dataframe[\"column3\"] = dataframe[\"column1\"] + dataframe[\"column2\"]\n",
    "    return dataframe\n",
    "\n",
    "\n",
    "preprocessed_df = preprocessor(df)\n",
    "preprocessed_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Great Expectations\n",
    "\n",
    "[**Great Expectations**](https://greatexpectations.io/) is another data validation library. It's really geared toward to production (and can integrate well with some other production tools). I wouldn't necessarily recommend it if you're doing a research project but it's got some amazing features that just mean it made it into this chapter--namely that it doesn't just validate data, but it documents and profiles it too.\n",
    "\n",
    "Their philosophy is that 'Expectations', the checks or assertions about data, are unit tests for your data (don't worry if you're not familiar with unit tests-they are checks for your code!). Once you have run your expectations, the library can create data documentation and data quality reports from them. The data documentation is quite an amazing feature, providing a navigable HTML report of your dataset.\n",
    "\n",
    "Using **Great Expectations** is a bit different from **pandera** as it replaces your dataframe with a **Great Expectations** `PandasDataset` that looks and feels just like a regular pandas dataframe but has extra methods related to data validation. To convert a regular pandas dataframe, you wrap it with `ge.from_pandas()`. \n",
    "\n",
    "Let's look at some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import great_expectations as ge\n",
    "\n",
    "df = ge.from_pandas(\n",
    "    pd.DataFrame(\n",
    "        {\n",
    "            \"column1\": [1, 4, 0, 10, 9],\n",
    "            \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "            \"column3\": [\"value_1\", \"value_2\", \"value_3\", \"value_2\", \"value_1\"],\n",
    "        }\n",
    "    )\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run some expectations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_values_to_be_of_type(\"column1\", \"int\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_proportion_of_unique_values_to_be_between(\"column3\", 3, 5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first expectation passed, while the second failed but did *not* throw an error... we were just told in JSON.\n",
    "\n",
    "Let's also see a less stringent expectations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_values_to_be_between(\n",
    "    column=\"column2\",\n",
    "    min_value=-15,\n",
    "    max_value=0,\n",
    "    mostly=0.80,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having a bunch of JSON thrown at you is hardly a pleasant experience, and running a few commands solely in a notebook (which is how this was generated) is not really the way that **Great Expectations** is intended to be used. Instead, it is designed to be used via several commands in the terminal that set up folders and files that will track the expectations you create and produce nice HTML reports on your data and whether it passed your tests or not. There is a [full tutorial](https://docs.greatexpectations.io/en/latest/guides/tutorials/getting_started.html) available on the website."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pydantic\n",
    "\n",
    "[**Pydantic**](https://pydantic-docs.helpmanual.io/) has some of the same features as **pandera** but it piggybacks on the ability of Python 3.5+ to have 'typed' variables (if you're not sure what that is, it's a way to declare a variable has a particular data type from inception) and it is really focused around the validation of objects (created from classes) rather than dataframes.\n",
    "\n",
    "If you've used the [SQLModel](https://sqlmodel.tiangolo.com/) package for writing SQL queries, you may be interested to know that every SQLModel call is also a Pydantic model.\n",
    "\n",
    "Here's an example of a Pydantic schema that also implements a class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from typing import List, Optional\n",
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class User(BaseModel):\n",
    "    id: int\n",
    "    name = \"Katherine Johnson\"\n",
    "    signup_ts: Optional[datetime] = None\n",
    "    friends: List[int] = []"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now pass some (acceptable) external data into this schema, creating an object of type `User`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "external_data = {\n",
    "    \"id\": \"123\",\n",
    "    \"signup_ts\": \"2019-06-01 12:22\",\n",
    "    \"friends\": [1, 2, \"3\"],\n",
    "}\n",
    "user = User(**external_data)\n",
    "user"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That all worked well (as expected) but what if we now create an object that has 'bad' data:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "User(signup_ts='broken', friends=[1, 2, 'not number'])\n",
    "```\n",
    "\n",
    "```\n",
    "---------------------------------------------------------------------------\n",
    "ValidationError                           Traceback (most recent call last)\n",
    "Cell In[16], line 1\n",
    "----> 1 User(signup_ts='broken', friends=[1, 2, 'not number'])\n",
    "\n",
    "\n",
    "ValidationError: 3 validation errors for User\n",
    "id\n",
    "  field required (type=value_error.missing)\n",
    "signup_ts\n",
    "  invalid datetime format (type=value_error.datetime)\n",
    "friends -> 2\n",
    "  value is not a valid integer (type=type_error.integer)\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a loud failure, a `ValidationError`, and a short report about what went wrong."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cerberus\n",
    "\n",
    "Our final stop on this tour of data validation is [**Cerberus**](https://docs.python-cerberus.org/). It is designed to be a simple and lightweight data validation functionality. In this case, the schema is specified as a JSON file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = {\"name\": {\"type\": \"string\"}, \"score\": {\"type\": \"integer\", \"max\": 10}}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we pass data in to a **Cerberus** validator object along with the schema. This will return true for good data but for bad data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cerberus import Validator\n",
    "\n",
    "v = Validator()\n",
    "data = {\"name\": \"Sadie Alexander\", \"score\": 12, \"occupation\": \"Economist\"}\n",
    "v.validate(data, schema)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful to know but we might want a bit more detail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.errors"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much more complex schemas can be [constructed](https://docs.python-cerberus.org/)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fake and Synthetic Data\n",
    "\n",
    "Fake data can be very useful for testing pipelines and trying out analyses before you have the *real* data. The closer fake data is to real data, the more likely that you're going to fully prepare yourself for the real data.\n",
    "\n",
    "*Fake* data are data generated according to a schema that bear no statistical relationship to the real data. There are powerful tools for generating fake data. One of the most popular and fast libraries for doing so is [Mimesis](https://mimesis.name/).\n",
    "\n",
    "*Synthetic* data take this one step further: they do capture some of the statistical properties of the underlying data and are generated from the underlying data. Again, they can be useful for trying out data before using the real thing—especially if the real data are highly confidential. A useful package for comparing real and synthetic data is [SynthGuage](https://datasciencecampus.github.io/synthgauge)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Orchestration\n",
    "\n",
    "Data orchestration is the automation of data-driven processes from end-to-end, including preparing data, making decisions based on that data, and taking actions based on those decisions. Think of a data pipeline in which you extract data (perhaps from files), transform it somehow, and load it into where you want to put it (downstream in your research process, or perhaps into an app). Sometimes these processes are called ETL, for extract, transform, and load. Some of their benefits include reproducibility, and being able to run tasks on a schedule—whether you are at your computer or not, and (if you're using them in the cloud), whether your computer is even on or not! They are dreamboats for automation.\n",
    "\n",
    "A lot of them visualise the tasks they have to carry out as a DAG, or Directed Acyclic Graph. This defines how different tasks depend on each other in order, with one task following from the previous one (or perhaps following from multiple previous tasks). When they carry out a DAG of tasks on an event or at a scheduled time, they create a log of the execution of the steps too—so if something goes wrong, you can always look back at it. However, they are a layer of abstraction between you and the data, and this does make them a more advanced tool.\n",
    "\n",
    "Going into further details of data orchestration is outside of the scope of this book, so we won't be seeing a practical example, but we think it's important enough to mention it and to point you to more resources.\n",
    "\n",
    "There are some truly amazing tools out there to help you do this on a production scale. Broadly, there is a 1st generation of tools that are excellent but might have more trouble with tasks such as:\n",
    "\n",
    "- local development, testing, and storage abstractions\n",
    "- one-off and irregularly scheduled tasks\n",
    "- the movement of data between tasks\n",
    "- dynamic and parameterised workflows\n",
    "\n",
    "And there is a 2nd generation of tools that tries to solve some of these issues.\n",
    "\n",
    "Of the 1st generation, by far the best known and most suited for production is Apache's (original AirBnB's) [**Airflow**](http://airflow.incubator.apache.org/). There's a less mature 1st generation tool from Spotify called [**Luigi**](https://github.com/spotify/luigi) too, but Airflow is widely used in the tech industry, and doesn't just schedule data processes in Python: it can run processes in pretty much whatever you like. Both of these tools try to solve the 'plumbing' associated with long-running batch processes on data: chaining tasks, automating them, dealing with failures, and scheduling. Both Luigi and Airflow have fancy interfaces to show you what's going on with your tasks.\n",
    "\n",
    "Tools in the 2nd generation include the easy-to-use [**Prefect**](https://www.prefect.io/), \"modern workflow orchestration for data and ML engineers\", and [**dagster**](https://dagster.io/), which bills itself as \"a data orchestrator for machine learning, analytics, and ETL [extract, transform, and load]\". These have visualisation interfaces too, can run locally or on the cloud, and are more flexible about triggers for events.\n",
    "\n",
    "For a research project where you only need to execute your DAG once, and you're probably much more concerned about the costliness of each step than anything else, these tools are likely to be overkill or not quite what you need. But if you're automating an ETL process, they're perfect (prefect). The best place to get started is by looking at their webpages and googling around for tutorials."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3.10.12 ('codeforecon')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "c4570b151692b3082981c89d172815ada9960dee4eb0bedb37dc10c95601d3bd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}