Lecture_20_2023.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "003d55f5",
   "metadata": {},
   "source": [
    "## Stargazing\n",
    "\n",
    "Fortune telling frameworks:\n",
    "\n",
    "1) From vague facts, vague advice\n",
    "\n",
    "2) Exaggerated importance\n",
    "\n",
    "3) Optimism\n",
    "\n",
    "Applies to astrologers and statisticians\n",
    "\n",
    "Valid vague advice exists, not sufficient\n",
    "\n",
    "Statistical procedures acquire meaning from scientific models\n",
    "\n",
    "Cannot offload subjective responsibility to an objective procedure\n",
    "\n",
    "Many subjective responsibilities "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad2b4a9",
   "metadata": {},
   "source": [
    "## Planning\n",
    "\n",
    "Goal setting - What for? Estimands\n",
    "\n",
    "Theory building - Which assumptions? \n",
    "\n",
    "Justified sampling plan\n",
    "\n",
    "Justified analysis plan\n",
    "\n",
    "Documentation \n",
    "\n",
    "Open software & data formats "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9764718",
   "metadata": {},
   "source": [
    "## Theory Building\n",
    "\n",
    "Levels of theory building:\n",
    "\n",
    "1) Heuristic causal models (DAGs)\n",
    "\n",
    "2) Structural causal models\n",
    "\n",
    "3) Dynamic models\n",
    "\n",
    "4) Agent-based models - Each entities decisions, difficult to analyse analytically, powerful way to see implication of assumptions\n",
    "\n",
    "Heuristic causal models (DAGs):\n",
    "\n",
    "1) Treatment and outcome\n",
    "\n",
    "2) Other causes\n",
    "\n",
    "3) Other effects \n",
    "\n",
    "4) Unobserved causes "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df40c3b8",
   "metadata": {},
   "source": [
    "## Pre-Registration\n",
    "\n",
    "Pre-registration: Prior public documentation of research design and analysis plan\n",
    "\n",
    "Goal: Make transparent which decisions are sample-dependent\n",
    "\n",
    "Does little to improve data analysis \n",
    "\n",
    "Lots of pre-registered causal salad. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b485c3d1",
   "metadata": {},
   "source": [
    "## Working\n",
    "\n",
    "Control\n",
    "\n",
    "Incremental testing\n",
    "\n",
    "Documentation \n",
    "\n",
    "Review "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756e70c9",
   "metadata": {},
   "source": [
    "## Professional Norms\n",
    "\n",
    "Dangerous lack of professional norms in scientific computing\n",
    "\n",
    "Often impossible to figure out what was done\n",
    "\n",
    "Often impossible to know ifcode works as intended\n",
    "\n",
    "Like pipetting by mouth "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc1c8a13",
   "metadata": {},
   "source": [
    "## Research Engineering\n",
    "\n",
    "Control: Versioning, back-up, accountability\n",
    "\n",
    "Incremental testing: Piece by piece\n",
    "\n",
    "Documentation: Comment everything \n",
    "\n",
    "Review: 4 eyes (2 people) on code and materials "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2871d8ce",
   "metadata": {},
   "source": [
    "## Versioning and Testing\n",
    "\n",
    "Version control: Database of changes to project files, managed history\n",
    "\n",
    "Testing: Incremental milestones, test each before moving to next\n",
    "\n",
    "Most researchers don't need all git's features\n",
    "\n",
    "But do:\n",
    "\n",
    "Commit changes after each milestone\n",
    "\n",
    "Maintain test code in project\n",
    "\n",
    "Do not:\n",
    "\n",
    "Replace raw data with processed data "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a31f69e1",
   "metadata": {},
   "source": [
    "## More on Testing\n",
    "\n",
    "Complex analyses must be built in steps\n",
    "\n",
    "Test each step\n",
    "\n",
    "Social networks lecture (#15) as example\n",
    "\n",
    "Milestones:\n",
    "\n",
    "1) Synthetic data simulation\n",
    "\n",
    "2) Dyadic reciprocity model\n",
    "\n",
    "3) Add generalized giving/receiving\n",
    "\n",
    "4) Add wealth, association index\n",
    "\n",
    "Never process data in Excel; use code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8b9237a",
   "metadata": {},
   "source": [
    "## Reporting\n",
    "\n",
    "Sharing materials\n",
    "\n",
    "Describing methods\n",
    "\n",
    "Describing data\n",
    "\n",
    "Describing results\n",
    "\n",
    "Making decisions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690e95fa",
   "metadata": {},
   "source": [
    "## Sharing Materials\n",
    "\n",
    "The paper is an advertisement; the data and its analysis are the product\n",
    "\n",
    "Make code and data available through a link, not \"by request\"\n",
    "\n",
    "Some data not shareable; code always shareable\n",
    "\n",
    "Archived code & data will be required "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e654d70",
   "metadata": {},
   "source": [
    "## Describing Methods\n",
    "\n",
    "Minimal information:\n",
    "\n",
    "1) Math-stats notation of stat model\n",
    "\n",
    "2) Explanation of how (1) provides estimand (Causal model, DAG)\n",
    "\n",
    "3) Algorithm used to produce estimate (MCMC)\n",
    "\n",
    "4) Diagnostic, code tests (Simulations)\n",
    "\n",
    "5) Cite software packages (Pymc, numpy,pandas,scipy,sunode)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c2e8cf",
   "metadata": {},
   "source": [
    "## Justify Priors\n",
    "\n",
    "\"Priors were chosen through prior predictive simulation so that pre-data predictions span the range of scientifically plausible outcomes.\n",
    "\n",
    "In the results, we explicitlycompare the posterior distribution to the prior, so that the impact of the sample is obvious\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0baaff7b",
   "metadata": {},
   "source": [
    "## Justifying Methods\n",
    "\n",
    "Naive reviewers: \"Good science doesn't need complex stats\"\n",
    "\n",
    "Causal model often requires complexity\n",
    "\n",
    "Big data => unit heterogeneity \n",
    "\n",
    "Ethical responsibility to do our best\n",
    "\n",
    "Change discussion from statistics to causal models \n",
    "\n",
    "Write for the editor, not the reviewer\n",
    "\n",
    "Find other papers in discipline/journal that have used Bayesian methods or similar models (Bayesian or not)\n",
    "\n",
    "Explain results in Bayesian terms, show densities, cite disciplinary guides.\n",
    "\n",
    "Bayes is ancient, normative, often the only practical way to estimate complex models. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "155f58c1",
   "metadata": {},
   "source": [
    "## Describing Data\n",
    "\n",
    "1k observations of 1 person\n",
    "\n",
    "-vs-\n",
    "\n",
    "1 observation of each of 1k people\n",
    "\n",
    "\"Effective\" sample size function of estimand and hierarchical structure\n",
    "\n",
    "Variables measured at which levels?\n",
    "\n",
    "Missing values! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d412c6a0",
   "metadata": {},
   "source": [
    "## Describing Results\n",
    "\n",
    "Estimands, marginal causal effects \n",
    "\n",
    "Warn against causal interpretation of contral variables (Table 2 fallacy)\n",
    "\n",
    "Densities better than intervals; Sample realizations often better than densities \n",
    "\n",
    "Figures assist comparisons"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47350f84",
   "metadata": {},
   "source": [
    "## Making Decisions\n",
    "\n",
    "Academic research: Communicate uncertainty, conditional on sample & models\n",
    "\n",
    "Industry research: What should we do, given the uncertainty, conditional on sample & models?\n",
    "\n",
    "Also:\"Does my boss have any idea what 'uncertainty' means, or does he think that's the refuge of cowards?\" \n",
    "\n",
    "Bayesian decision theory:\n",
    "\n",
    "1) State costs & benefits of outcomes\n",
    "\n",
    "2) Compute posterior benefits of hypothetical policy choices \n",
    "\n",
    "Simple example in Chapter 3\n",
    "\n",
    "Can be integrated with dynamic optimzation "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22afeaf9",
   "metadata": {},
   "source": [
    "## Horoscopes for Research\n",
    "\n",
    "No one knows how research works\n",
    "\n",
    "But many easy fixes at hand\n",
    "\n",
    "1) No stats without associated causal model\n",
    "\n",
    "2) Prove that your code works (in principle)\n",
    "\n",
    "3) Share as much as possible\n",
    "\n",
    "4) Beware proxies of research quality\n",
    "\n",
    "Many things you dislike about academia were once well-intentioned reforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea02216b",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "https://datacarpentry.org"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}