mb1_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Attach packages to be used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: RCall.jl: Loading required package: Matrix\n",
      "└ @ RCall /Users/reinholdkliegl/.julia/packages/RCall/g7dhB/src/io.jl:113\n",
      "\u001b[31m(compiled mode) evaluation error starting at /Users/reinholdkliegl/Google Drive/ZiF_CG_WS2/mb1_v2/none:0\u001b[39m┌ Warning: RCall.jl: \n",
      "│ Attaching package: ‘dplyr’\n",
      "│ \n",
      "│ The following objects are masked from ‘package:stats’:\n",
      "│ \n",
      "│     filter, lag\n",
      "│ \n",
      "│ The following objects are masked from ‘package:base’:\n",
      "│ \n",
      "│     intersect, setdiff, setequal, union\n",
      "│ \n",
      "└ @ RCall /Users/reinholdkliegl/.julia/packages/RCall/g7dhB/src/io.jl:113\n",
      "\n",
      "  \u001b[91mcould not open file /Users/reinholdkliegl/Google Drive/ZiF_CG_WS2/mb1_v2/dataframes.jl\u001b[39m\n",
      "Stacktrace:\n",
      " [1] \u001b[1minclude\u001b[22m"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RObject{StrSxp}\n",
       " [1] \"dplyr\"     \"lme4\"      \"Matrix\"    \"stats\"     \"graphics\"  \"grDevices\"\n",
       " [7] \"utils\"     \"datasets\"  \"methods\"   \"base\"     \n"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " at \u001b[1m./boot.jl:328\u001b[22m [inlined]\n",
      " [2] \u001b[1minclude_relative\u001b[22m\u001b[1m(\u001b[22m::Module, ::String\u001b[1m)\u001b[22m at \u001b[1m./loading.jl:1105\u001b[22m\n",
      " [3] \u001b[1minclude\u001b[22m at \u001b[1m./Base.jl:31\u001b[22m [inlined]\n",
      " [4] \u001b[1minclude\u001b[22m at \u001b[1m/Users/reinholdkliegl/.julia/packages/Gadfly/ISHWw/src/Gadfly.jl:1\u001b[22m [inlined]\n",
      " [5] \u001b[1mlink_dataframes\u001b[22m\u001b[1m(\u001b[22m\u001b[1m)\u001b[22m at \u001b[1m/Users/reinholdkliegl/.julia/packages/Gadfly/ISHWw/src/mapping.jl:238\u001b[22m\n",
      " [6] top-level scope at \u001b[1m/Users/reinholdkliegl/.julia/packages/IJulia/yLI42/src/kernel.jl:52\u001b[22m\n",
      "while evaluating\n",
      "begin\n",
      "    link_dataframes()\n",
      "    __pkguuid__ = \"a93c6f00-e57d-5684-b7b6-d8193f3e46c0\"\n",
      "end\n",
      "in module Gadfly\n"
     ]
    }
   ],
   "source": [
    "using CSV            # read and write .csv files\n",
    "using Gadfly         # plotting\n",
    "using DataFrames     \n",
    "using DataFramesMeta # dplyr-like operations\n",
    "using MixedModels\n",
    "using StatsBase      # basic statistics functions\n",
    "using RCall          # Call R from Julia\n",
    "\n",
    "R\"library(lme4)\"\n",
    "R\"library(dplyr)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing\n",
    "\n",
    "We are working with the *ManyBabies 1 - Infant-directed Speech Preference* dataset that has been published at:\n",
    "https://github.com/manybabies/mb1-analysis-public\n",
    "\n",
    "\n",
    "Download and read the data, using CSV. \n",
    "Inspect the contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>variable</th><th>mean</th><th>min</th><th>median</th><th>max</th><th>nunique</th></tr><tr><th></th><th>Symbol</th><th>Union…</th><th>Any</th><th>Union…</th><th>Any</th><th>Union…</th></tr></thead><tbody><p>16 rows × 8 columns (omitted printing of 2 columns)</p><tr><th>1</th><td>lab</td><td></td><td>babylabbrookes</td><td></td><td>wsigoettingen</td><td>67</td></tr><tr><th>2</th><td>subid</td><td></td><td>1</td><td></td><td>zu247</td><td>1955</td></tr><tr><th>3</th><td>subid_unique</td><td></td><td>babylabbrookes:mb01</td><td></td><td>wsigoettingen:m417</td><td>2329</td></tr><tr><th>4</th><td>trial_order</td><td>2.47842</td><td>1</td><td>2.0</td><td>4</td><td></td></tr><tr><th>5</th><td>trial_num</td><td>8.5</td><td>1</td><td>8.5</td><td>16</td><td></td></tr><tr><th>6</th><td>trial_type</td><td></td><td>ADS</td><td></td><td>IDS</td><td>2</td></tr><tr><th>7</th><td>stimulus_num</td><td>4.37316</td><td>1</td><td>4.0</td><td>8</td><td></td></tr><tr><th>8</th><td>method</td><td></td><td>eyetracking</td><td></td><td>singlescreen</td><td>3</td></tr><tr><th>9</th><td>age_days</td><td>291.995</td><td>92</td><td>286.0</td><td>456</td><td></td></tr><tr><th>10</th><td>age_mo</td><td>9.59326</td><td>3.02259</td><td>9.3963</td><td>14.9815</td><td></td></tr><tr><th>11</th><td>age_group</td><td></td><td>12-15 mo</td><td></td><td>9-12 mo</td><td>4</td></tr><tr><th>12</th><td>nae</td><td>0.457707</td><td>0</td><td>0.0</td><td>1</td><td></td></tr><tr><th>13</th><td>gender</td><td></td><td>0</td><td></td><td>MALE</td><td>5</td></tr><tr><th>14</th><td>second_session</td><td>0.0184909</td><td>0</td><td>0.0</td><td>1</td><td></td></tr><tr><th>15</th><td>looking_time</td><td>7.80084</td><td>2.0</td><td>6.23</td><td>18.0</td><td></td></tr><tr><th>16</th><td>missing</td><td>0.278258</td><td>0</td><td>0.0</td><td>1</td><td></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& variable & mean & min & median & max & nunique & \\\\\n",
       "\t\\hline\n",
       "\t& Symbol & Union… & Any & Union… & Any & Union… & \\\\\n",
       "\t\\hline\n",
       "\t1 & lab &  & babylabbrookes &  & wsigoettingen & 67 & $\\dots$ \\\\\n",
       "\t2 & subid &  & 1 &  & zu247 & 1955 & $\\dots$ \\\\\n",
       "\t3 & subid\\_unique &  & babylabbrookes:mb01 &  & wsigoettingen:m417 & 2329 & $\\dots$ \\\\\n",
       "\t4 & trial\\_order & 2.47842 & 1 & 2.0 & 4 &  & $\\dots$ \\\\\n",
       "\t5 & trial\\_num & 8.5 & 1 & 8.5 & 16 &  & $\\dots$ \\\\\n",
       "\t6 & trial\\_type &  & ADS &  & IDS & 2 & $\\dots$ \\\\\n",
       "\t7 & stimulus\\_num & 4.37316 & 1 & 4.0 & 8 &  & $\\dots$ \\\\\n",
       "\t8 & method &  & eyetracking &  & singlescreen & 3 & $\\dots$ \\\\\n",
       "\t9 & age\\_days & 291.995 & 92 & 286.0 & 456 &  & $\\dots$ \\\\\n",
       "\t10 & age\\_mo & 9.59326 & 3.02259 & 9.3963 & 14.9815 &  & $\\dots$ \\\\\n",
       "\t11 & age\\_group &  & 12-15 mo &  & 9-12 mo & 4 & $\\dots$ \\\\\n",
       "\t12 & nae & 0.457707 & 0 & 0.0 & 1 &  & $\\dots$ \\\\\n",
       "\t13 & gender &  & 0 &  & MALE & 5 & $\\dots$ \\\\\n",
       "\t14 & second\\_session & 0.0184909 & 0 & 0.0 & 1 &  & $\\dots$ \\\\\n",
       "\t15 & looking\\_time & 7.80084 & 2.0 & 6.23 & 18.0 &  & $\\dots$ \\\\\n",
       "\t16 & missing & 0.278258 & 0 & 0.0 & 1 &  & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "16×8 DataFrame. Omitted printing of 4 columns\n",
       "│ Row │ variable       │ mean      │ min                 │ median │\n",
       "│     │ \u001b[90mSymbol\u001b[39m         │ \u001b[90mUnion…\u001b[39m    │ \u001b[90mAny\u001b[39m                 │ \u001b[90mUnion…\u001b[39m │\n",
       "├─────┼────────────────┼───────────┼─────────────────────┼────────┤\n",
       "│ 1   │ lab            │           │ babylabbrookes      │        │\n",
       "│ 2   │ subid          │           │ 1                   │        │\n",
       "│ 3   │ subid_unique   │           │ babylabbrookes:mb01 │        │\n",
       "│ 4   │ trial_order    │ 2.47842   │ 1                   │ 2.0    │\n",
       "│ 5   │ trial_num      │ 8.5       │ 1                   │ 8.5    │\n",
       "│ 6   │ trial_type     │           │ ADS                 │        │\n",
       "│ 7   │ stimulus_num   │ 4.37316   │ 1                   │ 4.0    │\n",
       "│ 8   │ method         │           │ eyetracking         │        │\n",
       "│ 9   │ age_days       │ 291.995   │ 92                  │ 286.0  │\n",
       "│ 10  │ age_mo         │ 9.59326   │ 3.02259             │ 9.3963 │\n",
       "│ 11  │ age_group      │           │ 12-15 mo            │        │\n",
       "│ 12  │ nae            │ 0.457707  │ 0                   │ 0.0    │\n",
       "│ 13  │ gender         │           │ 0                   │        │\n",
       "│ 14  │ second_session │ 0.0184909 │ 0                   │ 0.0    │\n",
       "│ 15  │ looking_time   │ 7.80084   │ 2.0                 │ 6.23   │\n",
       "│ 16  │ missing        │ 0.278258  │ 0                   │ 0.0    │"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb1 = CSV.read(\n",
    "    download(\"https://raw.githubusercontent.com/manybabies/mb1-analysis-public/fa7e77c026a4dc0b0bb7e78d3bf3771c9bc2f7cb/processed_data/03_data_trial_main.csv\"),\n",
    "    missingstrings=[\"NA\",\"N/A\"],\n",
    "    truestrings=[\"TRUE\"],\n",
    "    falsestrings=[\"FALSE\"]);\n",
    "first(mb1, 10)\n",
    "describe(mb1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a preprocessed dataset, the key columns are:\n",
    "- looking_time : dependent variable, in milliseconds  \n",
    "- trial_type : condition, either IDS or ADS (infant- or adult-directed speech)  \n",
    "- method : 3 levels (hpp, singlescreen, eyetracking; the latter two are eye-movement-baed, the former measures head-movement)   \n",
    "- trial_num : 1 - 16, trial in the experiment  \n",
    "- age_mo : infant age in months, between 3 and 15  \n",
    "- nae : North-American English, boolean, whether infants learned NAE (the stimulus language)\n",
    "- subid_unique : participant identifier\n",
    "- lab : contributing label\n",
    "\n",
    "Note: labs could use multiple methods, participants were tested in age groups (3-6, 6-9. 9-12, 12-15 months) and labs could contribute multiple age groups.\n",
    "\n",
    "The `gender` variable should be `F`, `M`, or `missing` but some values are miscoded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countmap(mb1.gender)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So first let's get the data in shape. In Julia, there is DataFramesMeta for this purpose. \n",
    "\n",
    "Recode the levels of `gender`, add the `item` variable (join condition and stimulus information), center the `age_mo`, and relevel `method` and `age_group`.\n",
    "Add log-transformed looking time `log_lt` for visualization. \n",
    "Drop observations with a missing response (`looking_time`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>variable</th><th>mean</th><th>min</th><th>median</th><th>max</th><th>nunique</th></tr><tr><th></th><th>Symbol</th><th>Union…</th><th>Any</th><th>Union…</th><th>Any</th><th>Union…</th></tr></thead><tbody><p>18 rows × 8 columns (omitted printing of 2 columns)</p><tr><th>1</th><td>lab</td><td></td><td>babylabbrookes</td><td></td><td>wsigoettingen</td><td>67</td></tr><tr><th>2</th><td>subid</td><td></td><td>1</td><td></td><td>zu247</td><td>1936</td></tr><tr><th>3</th><td>subid_unique</td><td></td><td>babylabbrookes:mb01</td><td></td><td>wsigoettingen:m417</td><td>2302</td></tr><tr><th>4</th><td>trial_order</td><td>2.48113</td><td>1</td><td>2.0</td><td>4</td><td></td></tr><tr><th>5</th><td>trial_num</td><td>7.8174</td><td>1</td><td>7.0</td><td>16</td><td></td></tr><tr><th>6</th><td>trial_type</td><td></td><td>ADS</td><td></td><td>IDS</td><td>2</td></tr><tr><th>7</th><td>stimulus_num</td><td>4.27615</td><td>1</td><td>4.0</td><td>8</td><td></td></tr><tr><th>8</th><td>method</td><td></td><td>singlescreen</td><td></td><td>hpp</td><td>3</td></tr><tr><th>9</th><td>age_days</td><td>286.981</td><td>92</td><td>278.0</td><td>456</td><td></td></tr><tr><th>10</th><td>age_mo</td><td>-0.164721</td><td>-6.57067</td><td>-0.45979</td><td>5.38826</td><td></td></tr><tr><th>11</th><td>age_group</td><td></td><td>3-6 mo</td><td></td><td>12-15 mo</td><td>4</td></tr><tr><th>12</th><td>nae</td><td>0.459714</td><td>0</td><td>0.0</td><td>1</td><td></td></tr><tr><th>13</th><td>gender</td><td></td><td>F</td><td></td><td>M</td><td>2</td></tr><tr><th>14</th><td>second_session</td><td>0.012493</td><td>0</td><td>0.0</td><td>1</td><td></td></tr><tr><th>15</th><td>looking_time</td><td>7.80084</td><td>2.0</td><td>6.23</td><td>18.0</td><td></td></tr><tr><th>16</th><td>missing</td><td>0.0</td><td>0</td><td>0.0</td><td>0</td><td></td></tr><tr><th>17</th><td>item</td><td></td><td>1ADS</td><td></td><td>8IDS</td><td>16</td></tr><tr><th>18</th><td>log_lt</td><td>1.86434</td><td>0.693147</td><td>1.82938</td><td>2.89037</td><td></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& variable & mean & min & median & max & nunique & \\\\\n",
       "\t\\hline\n",
       "\t& Symbol & Union… & Any & Union… & Any & Union… & \\\\\n",
       "\t\\hline\n",
       "\t1 & lab &  & babylabbrookes &  & wsigoettingen & 67 & $\\dots$ \\\\\n",
       "\t2 & subid &  & 1 &  & zu247 & 1936 & $\\dots$ \\\\\n",
       "\t3 & subid\\_unique &  & babylabbrookes:mb01 &  & wsigoettingen:m417 & 2302 & $\\dots$ \\\\\n",
       "\t4 & trial\\_order & 2.48113 & 1 & 2.0 & 4 &  & $\\dots$ \\\\\n",
       "\t5 & trial\\_num & 7.8174 & 1 & 7.0 & 16 &  & $\\dots$ \\\\\n",
       "\t6 & trial\\_type &  & ADS &  & IDS & 2 & $\\dots$ \\\\\n",
       "\t7 & stimulus\\_num & 4.27615 & 1 & 4.0 & 8 &  & $\\dots$ \\\\\n",
       "\t8 & method &  & singlescreen &  & hpp & 3 & $\\dots$ \\\\\n",
       "\t9 & age\\_days & 286.981 & 92 & 278.0 & 456 &  & $\\dots$ \\\\\n",
       "\t10 & age\\_mo & -0.164721 & -6.57067 & -0.45979 & 5.38826 &  & $\\dots$ \\\\\n",
       "\t11 & age\\_group &  & 3-6 mo &  & 12-15 mo & 4 & $\\dots$ \\\\\n",
       "\t12 & nae & 0.459714 & 0 & 0.0 & 1 &  & $\\dots$ \\\\\n",
       "\t13 & gender &  & F &  & M & 2 & $\\dots$ \\\\\n",
       "\t14 & second\\_session & 0.012493 & 0 & 0.0 & 1 &  & $\\dots$ \\\\\n",
       "\t15 & looking\\_time & 7.80084 & 2.0 & 6.23 & 18.0 &  & $\\dots$ \\\\\n",
       "\t16 & missing & 0.0 & 0 & 0.0 & 0 &  & $\\dots$ \\\\\n",
       "\t17 & item &  & 1ADS &  & 8IDS & 16 & $\\dots$ \\\\\n",
       "\t18 & log\\_lt & 1.86434 & 0.693147 & 1.82938 & 2.89037 &  & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "18×8 DataFrame. Omitted printing of 4 columns\n",
       "│ Row │ variable       │ mean      │ min                 │ median   │\n",
       "│     │ \u001b[90mSymbol\u001b[39m         │ \u001b[90mUnion…\u001b[39m    │ \u001b[90mAny\u001b[39m                 │ \u001b[90mUnion…\u001b[39m   │\n",
       "├─────┼────────────────┼───────────┼─────────────────────┼──────────┤\n",
       "│ 1   │ lab            │           │ babylabbrookes      │          │\n",
       "│ 2   │ subid          │           │ 1                   │          │\n",
       "│ 3   │ subid_unique   │           │ babylabbrookes:mb01 │          │\n",
       "│ 4   │ trial_order    │ 2.48113   │ 1                   │ 2.0      │\n",
       "│ 5   │ trial_num      │ 7.8174    │ 1                   │ 7.0      │\n",
       "│ 6   │ trial_type     │           │ ADS                 │          │\n",
       "│ 7   │ stimulus_num   │ 4.27615   │ 1                   │ 4.0      │\n",
       "│ 8   │ method         │           │ singlescreen        │          │\n",
       "│ 9   │ age_days       │ 286.981   │ 92                  │ 278.0    │\n",
       "│ 10  │ age_mo         │ -0.164721 │ -6.57067            │ -0.45979 │\n",
       "│ 11  │ age_group      │           │ 3-6 mo              │          │\n",
       "│ 12  │ nae            │ 0.459714  │ 0                   │ 0.0      │\n",
       "│ 13  │ gender         │           │ F                   │          │\n",
       "│ 14  │ second_session │ 0.012493  │ 0                   │ 0.0      │\n",
       "│ 15  │ looking_time   │ 7.80084   │ 2.0                 │ 6.23     │\n",
       "│ 16  │ missing        │ 0.0       │ 0                   │ 0.0      │\n",
       "│ 17  │ item           │           │ 1ADS                │          │\n",
       "│ 18  │ log_lt         │ 1.86434   │ 0.693147            │ 1.82938  │"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb1a = @linq mb1 |>\n",
    "    transform(gender = recode(:gender, \"0\"=>missing, \"MALE\"=>\"M\", \"FEMALE\"=>\"F\"),\n",
    "              item = string.(:stimulus_num, :trial_type),\n",
    "              age_mo = :age_mo .- mean(:age_mo),\n",
    "              log_lt = log.(:looking_time),\n",
    "              method = levels!(categorical(:method), [\"singlescreen\", \"eyetracking\", \"hpp\"]),\n",
    "              age_group = levels!(categorical(:age_group), [\"3-6 mo\", \"6-9 mo\", \"9-12 mo\", \"12-15 mo\"])) |>\n",
    "    where(.!ismissing.(:looking_time));\n",
    "disallowmissing!(mb1a, error=false);\n",
    "describe(mb1a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A histogrm of the `looking_time` shows the thresholding effect.\n",
    "Trials with looking times shorter than 2s were excluded as uninformative, and maximal trial duration was 18s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1a.looking_time, Geom.histogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also split the plot by our key comparison to inspect how the censoring affects the DV in the two conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1a.looking_time, xgroup=mb1a.trial_type, Geom.subplot_grid(Geom.histogram))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data are being log transformed in the analysis, so we can also visualize that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1a.log_lt, Geom.histogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Fitting \n",
    "\n",
    "Fit the linear mixed-effects model from the paper. We replicate the reported results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear mixed model fit by REML\n",
       " :(log(looking_time)) ~ 1 + trial_type + method + trial_num + age_mo + nae + trial_type & method + trial_type & trial_num + age_mo & trial_num + trial_type & age_mo + trial_type & nae + age_mo & nae + trial_type & age_mo & nae + (1 | subid_unique) + (1 | lab) + (1 | item)\n",
       " REML criterion at convergence: 43346.73059085285\n",
       "\n",
       "Variance components:\n",
       "                Column     Variance    Std.Dev.  \n",
       "subid_unique (Intercept)  0.072911220 0.270020777\n",
       "lab          (Intercept)  0.034756013 0.186429646\n",
       "item         (Intercept)  0.003313774 0.057565389\n",
       "Residual                  0.257245639 0.507193887\n",
       " Number of obs: 26895; levels of grouping factors: 2302, 67, 16\n",
       "\n",
       "  Fixed-effects parameters:\n",
       "─────────────────────────────────────────────────────────────────────────────────\n",
       "                                           Estimate   Std.Error  z value  P(>|z|)\n",
       "─────────────────────────────────────────────────────────────────────────────────\n",
       "(Intercept)                             2.17749      0.0505755     43.05   <1e-99\n",
       "trial_type: IDS                         0.099282     0.036048       2.75   0.0059\n",
       "method: eyetracking                    -0.265154     0.0457677     -5.79   <1e-8 \n",
       "method: hpp                            -0.0519927    0.0508692     -1.02   0.3067\n",
       "trial_num                              -0.0377045    0.001513     -24.92   <1e-99\n",
       "age_mo                                 -0.0353204    0.00444079    -7.95   <1e-14\n",
       "nae                                    -0.0163462    0.0489224     -0.33   0.7383\n",
       "trial_type: IDS & method: eyetracking  -0.00912279   0.0166455     -0.55   0.5836\n",
       "trial_type: IDS & method: hpp           0.0341055    0.0150124      2.27   0.0231\n",
       "trial_type: IDS & trial_num            -0.00289885   0.00212317    -1.37   0.1721\n",
       "age_mo & trial_num                      0.000713798  0.00022736     3.14   0.0017\n",
       "trial_type: IDS & age_mo                0.0123196    0.00286676     4.30   <1e-4 \n",
       "trial_type: IDS & nae                   0.0392234    0.012724       3.08   0.0021\n",
       "age_mo & nae                            0.00115973   0.00586571     0.20   0.8433\n",
       "trial_type: IDS & age_mo & nae          0.00428713   0.00406433     1.05   0.2915\n",
       "─────────────────────────────────────────────────────────────────────────────────"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m1form = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   (1 | subid_unique) +\n",
    "                   (1 | lab) + \n",
    "                   (1 | item);\n",
    "m1 = fit(MixedModel, m1form, mb1a, REML=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The thresholding of the response produces some unusual patterns in the residuals versus fitted values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=fitted(m1), y=residuals(m1), Geom.density2d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preregistered maximal model \n",
    "\n",
    "Fit the authors' intended maximal mixed-effects model. lme4 in R did initally not converge and now throws singularity warnings for this model. \n",
    "We switch from REML to ML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m2form = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   (1 + trial_type * trial_num | subid_unique) +\n",
    "                   (1 + trial_type * age_mo | lab) +\n",
    "                   (1 + method + age_mo * nae | item);\n",
    "m2 = fit(MixedModel, m2form, mb1a, REML=false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`rePCA` is analogous to rePCA() in R's lme4, which runs a Principal Component Analysis on the random effects matrix estimates to be able to detect overfitting. In MixedModels, it is a property of the fitted model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m2.rePCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to `rePCA` there is overparameterization for each of the three random factors. \n",
    "Let's go with zerocorr parameter linear mixed models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3form = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   zerocorr(1 + trial_type * trial_num | subid_unique) +\n",
    "                   zerocorr(1 + trial_type * age_mo | lab) +\n",
    "                   zerocorr(1 + method + age_mo * nae | item);\n",
    "m3 = fit(MixedModel, m3form, mb1a, REML=false);\n",
    "VarCorr(m3)\n",
    "\n",
    "m3.rePCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like quite a few variance components have very small values. Drop interaction terms and check LRT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m4form = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   zerocorr(1 + trial_type | subid_unique) +\n",
    "                   zerocorr(1 + age_mo | lab) +\n",
    "                   zerocorr(1 + method + nae | item);\n",
    "m4 = fit(MixedModel, m4form, mb1a, REML=false);\n",
    "\n",
    "VarCorr(m4)\n",
    "\n",
    "m4.rePCA\n",
    "MixedModels.likelihoodratiotest(m3, m4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks ok. So let's expand with CPs again. \n",
    "Intermediate linear mixed models showed that CP is not supported for `Subject`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m5form = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   zerocorr(1 + trial_type | subid_unique) +\n",
    "                   (1 + method + nae | item) +\n",
    "                   (1 + age_mo | lab);\n",
    "m5 = fit(MixedModel, m5form, mb1a, REML=false)\n",
    "\n",
    "m5.rePCA                                  # overparameterized\n",
    "MixedModels.likelihoodratiotest(m4, m5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not losing goodness of fit. That looks almost good; there are sizeable item-related correlation parameters, \n",
    "but the item-related RE structure is overparameterized. We should possibly drop the large CP. \n",
    "\n",
    "### What happens if the random effects are not proberly nested?\n",
    "\n",
    "We have labs and participants within labs, so if we naively use those variables, what would happen?\n",
    "\n",
    "Note that we are not fitting the model (it takes a very long time, an indicator something is going on), we instead construct and evaluate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m2form_nesting = @formula log(looking_time) ~ trial_type * method +\n",
    "                   trial_type * trial_num +\n",
    "                   age_mo * trial_num +\n",
    "                   trial_type * age_mo * nae +\n",
    "                   (1 + trial_type * trial_num | subid) +\n",
    "                   (1 + trial_type * age_mo | lab) +\n",
    "                   (1 + method + age_mo * nae | item);\n",
    "#m2n = fit(MixedModel, m2form_nesting, mb1a, REML=false) # This would fit the model\n",
    "m2n = LinearMixedModel(m2form_nesting, mb1a) # Construct the model \n",
    "#fit!(m2_n) # Fit the model we constructed \n",
    "describeblocks(m2n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare that to the blocks in the original model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "describeblocks(m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out there are way less unique observations now, which is of course a problem. \n",
    "\n",
    "\n",
    "We can also see the issue when inspecting the dataset itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cellmeans = by(mb1a, [:trial_type, :nae, :method], \n",
    "  meanLT = :looking_time => mean, \n",
    "  sdLT = :looking_time => std, \n",
    "  n_labs = :lab => (x -> length(unique(x))),\n",
    "  n_sub = :subid => (x -> length(unique(x))),\n",
    "  n_sub_unique = :subid_unique => (x -> length(unique(x))),\n",
    "  n_obs = :looking_time => length\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with censored data\n",
    "\n",
    "As we saw in the initial histograms, there are a number of observations that fall on the maximum trial length, so for robustness checks we see what happens if we discard those observations (i.e. all observations where looking time is 18s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#How many observations are affected exactly?\n",
    "sum(mb1a.looking_time.==18)\n",
    "\n",
    "#remove 18s observations\n",
    "mb1b = @linq mb1a |>\n",
    "    where(:looking_time.<18);\n",
    "describe(mb1b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the data again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1b.looking_time, Geom.histogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we fit the original and final models again, do our conclusions change or are they robust to excluding those observations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1b = fit(MixedModel, m1form, mb1b, REML=true)\n",
    "\n",
    "m5b = fit(MixedModel, m5form, mb1b, REML=false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out our conclusions are rather robust to excluding the ceiling data. \n",
    "\n",
    "\n",
    "Next, we can check what happens when we do not exclude too short trials. \n",
    "For that, we need an unfiltered dataset and re-clean it \"by hand\" (this is handled by the exclusions-markdowns in the ManyBabies1 repository)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb1_raw = CSV.read(\n",
    "    download(\"https://raw.githubusercontent.com/manybabies/mb1-analysis-public/fa7e77c026a4dc0b0bb7e78d3bf3771c9bc2f7cb/processed_data/02_validated_output.csv\"),\n",
    "    missingstrings=[\"NA\",\"N/A\"],\n",
    "    truestrings=[\"TRUE\"],\n",
    "    falsestrings=[\"FALSE\"])\n",
    "describe(mb1_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This datastructure is much wider and hasn't been cleaned yet. There are the 81 columns. Let's take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(names(mb1_raw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this datastructure, what has to be removed have already been flagged, and we follow the exclusion procedure now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#first remove all missing data so the next step with logical operations is not thrown off\n",
    "\n",
    "mb1_alltrials = @linq mb1_raw |>\n",
    "    where( \n",
    "        .!ismissing.(:age_mo),\n",
    "        .!ismissing.(:looking_time), \n",
    "        .!ismissing.(:trial_type)\n",
    "    );\n",
    "\n",
    "mb1_alltrials = @linq mb1_alltrials |>\n",
    "    where(\n",
    "        :pilot .== false, #no pilot data\n",
    "        :age_mo .>= 3, #no participants outside the pre-specified age range\n",
    "        :age_mo .<= 15,\n",
    "        :monolingual .== true, #only monolingual participants\n",
    "        :full_term .== true, #no preterm \n",
    "        :td .== true, #no known developmental delays\n",
    "        :session_error .== false, #no participants where something went wrong during the session_error\n",
    "        :trial_error .== false, #no trials where something went wrong\n",
    "        :trial_num .> 0, #no training trials (out of order here)\n",
    "        :trial_type .!= \"TRAIN\"\n",
    "    );\n",
    "\n",
    "# only keep rows we plan to use (this also avoids issues with empty fields / missing data down the line)\n",
    "mb1_alltrials = select!(mb1_alltrials, [:lab, :subid, :subid_unique, :trial_order, :trial_num, :trial_type, :stimulus_num, :method, :age_days, :age_mo, :age_group, :nae, :gender, :second_session, :looking_time]);\n",
    "\n",
    "# apply the same conversions as before for comparability\n",
    "mb1_alltrials = @linq mb1_alltrials |>\n",
    "    transform(gender = recode(:gender, \"0\"=>missing, \"MALE\"=>\"M\", \"FEMALE\"=>\"F\"),\n",
    "              item = string.(:stimulus_num, :trial_type),\n",
    "              age_mo = :age_mo .- mean(:age_mo),\n",
    "              log_lt = log.(:looking_time),\n",
    "              method = levels!(categorical(:method), [\"singlescreen\", \"eyetracking\", \"hpp\"]),\n",
    "              age_group = levels!(categorical(:age_group), [\"3-6 mo\", \"6-9 mo\", \"9-12 mo\", \"12-15 mo\"])) |>\n",
    "    where(.!ismissing.(:looking_time));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One exclusion criterion is whether infants contributed at least one usable trial per condition, for simplicity I am copying over the R code \n",
    "\n",
    "But first we need to drop all missing values, because they are handled differently in the languages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dropmissing(mb1_alltrials);\n",
    "\n",
    "@rput mb1_alltrials;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've passed the data to R and can manipulate it there. While we are at it, we also reconstruct the item variable.\n",
    "\n",
    "```{R;label=usable_pairs}\n",
    "usable_pairs <- mb1_alltrials %>%\n",
    "  group_by(subid_unique, stimulus_num) %>%\n",
    "  summarise(n_usable = sum(!is.na(looking_time))) %>%\n",
    "  summarise(usable_pair = any(n_usable == 2, na.rm=TRUE)) %>%\n",
    "  mutate(usable_pair = ifelse(is.na(usable_pair), FALSE, usable_pair)) %>%\n",
    "  ungroup()\n",
    "\n",
    "mb1_alltrials <- mb1_alltrials %>% \n",
    "  left_join(usable_pairs, by = \"subid_unique\") %>%\n",
    "  filter(usable_pair) \n",
    "\n",
    "```\n",
    "\n",
    "Now return the data to julia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@rget mb1_alltrials;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1_alltrials.looking_time, xgroup=mb1_alltrials.trial_type, Geom.subplot_grid(Geom.histogram))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to remove 0 looking time trials, i.e. trials without any data. This is both conceptually appropriate and saves our log transform of the looking times. We also again remove the right-censored data (i.e. looking times of exact 18s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb1_alltrials = @linq mb1_alltrials |>\n",
    "    where(\n",
    "        :looking_time .> 0, \n",
    "        :looking_time .< 18\n",
    "        );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the data again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=mb1_alltrials.looking_time, xgroup=mb1_alltrials.trial_type, Geom.subplot_grid(Geom.histogram))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can finally fit the model again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1_censored = fit(MixedModel, m1form, mb1_alltrials, REML=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the residuals again, it's not perfect, yet (of course), but better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(x=fitted(m1_censored), y=residuals(m1_censored), Geom.density2d)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.3.1",
   "language": "julia",
   "name": "julia-1.3"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}