notebooks/99_Outlook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outlook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Existing things not covered in the workshop\n",
    "\n",
    "- [What to do if there's no local disk on the compute nodes?](http://jobqueue.dask.org/en/stable/configuration-setup.html#no-local-storage)\n",
    "\n",
    "- [Managing ressources like GPUs and local disk.](https://jobqueue.dask.org/en/latest/examples.html?highlight=ssdGB#slurm-deployment-providing-additional-arguments-to-the-dask-workers)\n",
    "\n",
    "- _(Only touched very briefly)_ [How many processes / threads to use?](https://jobqueue.dask.org/en/latest/configuration-setup.html#processes)\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Possible future developments\n",
    "\n",
    "Things that would be nice to have are\n",
    "\n",
    "- Partially preemptible clusters.  Currently, the Dask supports a relatively simple way of allowing for failing tasks by counting failures and increasing the suspiciouusness of each task until a threshold is exceeded.  It would be nice to allow for marking some workers as preemptible and some as non-preemptible.  Then, we could allow for more failures on the preemptible tasks.  Also, we could try and schedule tasks that depend of others (such as aggregations of partial reductions) preferably on more reliable workers.\n",
    "\n",
    "- If we have very large calculations, we might have to get used to parts of the calculations not finishing at all (be it because of resources not available within a given time frame, because of data stores not being online atm, ...).  It would be interesting to subclass Dask arrays to still come up with a best-guess for a calculation even if not all the data could be processed.\n",
    "\n",
    "- _(This is nearly done but needs somebody with **SLURM admin skills**.)_  There's a semi-functional binder with SLURM running on the same node as Jupyter.  See [this repo](https://github.com/lesteve/test-binder) and [this issue](https://github.com/willirath/dask_jobqueue_workshop_materials/issues/5)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:dask_jobqueue_workshop]",
   "language": "python",
   "name": "conda-env-dask_jobqueue_workshop-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}