notebooks/quilt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Quilt for data versioning\n",
    "We use Quilt to version our data. A data set is assembled on a local machine (ideally an EC2 instance since we'll be interacting a lot with S3, for better up/download speed and reduced bandwidth costs)\n",
    "\n",
    "Currently, our workflow doesn't take advantage of Quilt as much as it can. Our main processing routine is to use AWS Batch to process many brain images through our Neuromorphometry Reporting tool (re:THINQ). This is done using a custom-built wrapper around `awscli`, which provides a command line tool to submit the brain images of subjects stored in our S3 infrastructure, and check their processing status (i.e. SUBMITTED, RUNNING, COMPLETED, FAILED)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial upload of data\n",
    "We use Quilt's CLI to create a Package and push it to S3:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "local_data_dir=\"/home/ubuntu/data/LA5c\"\n",
    "quilt_bucket=\"s3://cmet-quilt\"\n",
    "code_version=\"1.0.0\"\n",
    "quilt_package_name=\"ltirrell/rend\"\n",
    "\n",
    "quilt3 push \\\n",
    "  \"${quilt_package_name}\" \\\n",
    "  --dir \"${local_data_dir}\" \\\n",
    "  --dest \"${quilt_bucket}/${quilt_package_name}/data\" \\\n",
    "  --registry \"${quilt_bucket}\" \\\n",
    "  --message \"Initial commit of data\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run our processing on AWS Batch, which points to the data stored in this s3 bucket:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "submit_subjects \\\n",
    "    --input_data \"${quilt_bucket}/${quilt_package_name}/data\" \\\n",
    "    --container_tag \"${thinq_version}\" \\\n",
    "    --output_bucket \"${quilt_bucket}/${quilt_package_name}/results/${code_version}\" \\\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The status of the indivdiual submissions is checked:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "check_status ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and when all are completed, results are added to an updated version of the Quilt Package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "quilt3 push \\\n",
    "  \"$quilt_package_name\" \\\n",
    "  --dir \"${quilt_bucket}/${quilt_package_name}/results/${code_version}\" \\\n",
    "  --dest \"${quilt_bucket}/${quilt_package_name}/results/${code_version}\" \\\n",
    "  --registry \"${quilt_bucket}\" \\\n",
    "  --message \"Add results of data using version ${code_version} of code\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to update our AWS Batch wrapper to take advantage of data stored and versioned with Quilt. \n",
    "By using the `quilt3` Python library, as opposed to the command line tool usage above, we would be able to more explicitly and succinctly point to exact versions of input data in our workflows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}