PythonFinanceNotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SciDB and Python\n",
    "====\n",
    "\n",
    "> **Jonathan Rivers, Paradigm4, jrivers@paradigm4.com**\n",
    "\n",
    "> **Kriti Sen, Paradigm4, ksen@paradigm4.com**\n",
    "\n",
    "> **Bryan Lewis, Paradigm4, blewis@paradigm4.com**\n",
    "\n",
    "www.paradigm4.com\n",
    "\n",
    "$$\\,$$\n",
    "\n",
    "$$\\,$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAABsCAMAAACy7ZzBAAAAAXNSR0IArs4c6QAAAMBQTFRFpq+9\nI4+u4eTpruPuetDiGlJ1l6GyOHSSFDVZvOfxbMzfe4iclNnoxMrTqqqq7u7ud3d3MzMziIiIX8fd\nERERiZSnyezz3d3du7u7mZmZZmZmTl97zMzMVVVVREREIiIiIjdaP1JwNrnU8vr88PL009febHqR\nMUVlRL3XUcLatbzIh9Xl5Pb51/H2XW6Hod7ryuz0KKvJlNroJqPBHm+Qu9/oXb7UuM7Yns3bNKjE\ng7vNTajBEypPKbTRAAAA////GAJX/gAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB90HGxcz\nByQq7LkAAAm6SURBVHja7ZoJc+I6EoA9QPIGApZtbHxgbG5IIDDZN3uD+f//atW6LINtrLxQs7tP\nqkqVj7ak/tTd6hYxLn/yZmgAGoAGoAFoABqABqABaAAl7X002t88XA0GrT8FgH3nTFrvmENYvcxO\n0Kbe8vaD5aDYlv/bANrzs2i9EZWl2pM2K6i38sbT022bjr03lcl4+JsBu+6fq9t8OOx3F4VP68TP\n22F/rwpgPy90MRxdjE1BuZmk/vhU3TbG1wOgrbNoCgDabqEGoCNUHw4Ji39/B63HY4FBrO3bqb6t\nHwTgPH9XAHB+VQMwl7ntj7+fzx+/GcTsW7OiXktq/GPPu4oBnnegtJ4/C2Dbr2id12sCdeI7Kt1W\nAkD1/wvTcXZ62mIa7B0lMGZ3L3DzUhHxBiC7aQqgJYVO0GhYLbrvFVe1XnwE67n9BADqN6vpaTa4\nHOfn14Vk8xwArLJXrROIrj4Rne8AYE7abSjezrVRAUB7NPACE+jD85baXAEAXNfseBAgjUcAuGxh\nf2oqDuqMPgfAOE15vNsxr5MBDIo7Qqlfew8BcIQo8DAArxwA1n8lWRIhQAB4OYDxLwGwhym+PwrA\njsWYgv6MQEveBu8BgMAmp84rw1uPeTt4zwM5ZHqe12oMgPhA91EAujQIXulPCOxJEJxyhe64wJU9\nbG7TxTytvt4G7wAYYpH+owDQROBf1/qD570eCukN3DSsjlqz0kxJpNWKAHbNAXTleNEMAPT48b0k\ngHc+ICcUEyVZwaGR/kui/8x7k5Kl9VSyIEUAfWkbqBdfgLfs1ABAiHkqS2MXv50Kbv9MMv7nBsUf\n6DdblVExPg1gKF2/jspbdwfm3FEth3fnn6fvfytZx2nBAHhiWFoM4mpQ4jItzYmgv8OXAKgrBDoj\n5fOAxT9Pp2/nHzfPX05XcX81q6+FllK4LCuM1iJZfhyA+XD3rnwiND494QT6On8cEK3+UVjDktgu\nGwJbdaMiJwS1Hw3grB4DIOP/dus6S6Lrz/kVF7y7H8bjClOYtm7Uu1zt/g+NAaM2CQJix2gIYHPa\nzG+rSLIFzio6uz0/NMb5llkFoLocVgRQI76AzHa+UAGADeCZ1FCFMLCmK9or76ysHUTS9BAAnYbi\n+6pMyKip4lq05JSOHQyWBOM65NgQwFIkSl8OQCERonlzXwFAi0am1wIBgx/w/Kg8YbptM67VlwNQ\nSYULwk0APNOtbkEJtHP7Jw6NAZz3zYohciDwGABbyaq/HMCBJaiUAJyNLQ/5USAAOH4ZAFEvqgEg\nbr1/FIAp35wZgeHfN9JR6I88/NwFUHCB0p8JxImBGoC+fM735QDyip8R+Pgp1YCdfLiWVBrXB0FA\n9VIh8QkAi7mc3Hx5ECQVDk1iFz2aSf3+xA+4u2dpuGm9a3sCECg6LSmZDMFaBcBiKHvAXfFR1cF4\nHYDT9BlnsQPj8JNn1J0u3hB+dM4yAAiN00oCz5LdjG9+UmNnzvxAQQHA6LWY3d4Tn1cdC9cCEO3p\n4za1Hko1DlT5xuCmeeuNlArTMmJjFE5PWt40DyKNfxgZbmmNt2j0w0i/s60uBoya0CW1799uAByL\n2VFNM2R3ILWkaMVySfGnseGiGBHvtJ7S7wLXWo3/ur3qL/+99a22GNxI7mFMSyVWl08A2Lavt4T6\ngrivWA4XfvEdw7TaBQSF4d/WFYXgeF3c+FrP14Kbg1GZCFX/3N07vt/uiTW20mkvVA9Eli+iml3z\nFer2BM92aT5TbKvqg3Lefv3/mtQciLSgxF97BTUW3X5vOOwcF5f/l6b/SUoD0AA0AA1AAyhtvmma\nMWrUS2I1HM6x+VVgm1VCAR44aNZfYAcgf1/QN9UB2Bk0u0HvceY0BGALAE4W1w7sNmJqZeiCMvs+\nKNdVBoCyCV4Jx22gnJM1XDCURfxy4tYOHNqNoBKgpn9/glmiDMCkM0CpS9wgcBxYsiCQLIo9u0RU\nGdOi1oJlAocZDuIPL37oBGYW8u7T5GI6QW6cyESFgYMMm0AMXsjfooC6JYdtOiYBSjtA+JagwLdx\nGOLOYidkXTquVYnTuLusDkzaTzKb2EKSktHcifSMrgO2FewwiNwmcEmHTm360CJ2nfHF8rMkxQ8i\neEFUTFIkzDpgAMg3xGjwJCZZFprUK0EydEl/eG4RdgM6ekpv4TM3SLgkfhc7mXoMSFy+IBZWOMIz\nn+Dpg9PBkvvSMx8Wy8kshMwMwGD/9f0IJmO5JqgSwbQchOI05b2HWeqg2MZfU6uQZkgHtvCD0ETI\nn8CIUZZGoYOC0EcI3uDvIx/exRQ/uY1hMjiCTGIsk1kxikinkRtfSCeKANyETzW8JAm9MskfNdL8\nmYn/iE3g0V36iK6gSbXCM2RhkgoRryEzArUJPkxTGhir7fBogQhyW9gHtnXoNZ1Qs4GQYfHR4dYn\nH1KqRDJKYzn4NgYQZxafahxkiYUbAA6gZzsBBcUzmCIL6ja+pEYCAJIMRCysHA150m5BZ0TWHUeD\nyyRXMCBmTkY3I+xj4EuITSa2bHAcn/snYAAt2eguu+X+a2HJCNzAzyxlAMIm8fzwchBNLDoV0rv0\nDJRhDgOXVDdYA9emIiYzp3y3YBrBDGHdQslFnWxiWaFP0CeWSbyKmZKT2VZo2ikfjgCFPmhkYrdI\n7DEJmJ4NLUttXxEA9xrwOEvyIDvxyRadPyOhmGoN9se2ushFSHBnV/nWx7YDMCWIgrlr5ANjdelu\nwJQEBSO+ztyAApm5uGUdg6RPVgBMUdUCmNdYNFDT3QsRqpOUvuDPqFewyBUz3cADeIYCucokn5ZY\nemZlId5MpBCVcndNWJiICw5Didip2H4hepAOEPCl+KnJB8LjqkNAJQC8TeEsKEpJH3Hmhih2EupX\n1BzzZ2SK2CN8n+y2+AryJ5ssZhQjM8LKp25oOmnuiUnm+DzQxVmeHfAgRk0BJw6TzBXBE8sHpuXC\nOuN3ZpgA4ICOaVsTNyOhKbzwOCySTcZfCUBId9yIdhFC6KGzCHln4hl1wAjuQhKKyQYPq4cmPJmO\ncTBLo9zTXYhuLNv1C9MLBY2YiKSJWEoEOfKEuD9c4uzG4lpatj1hKOJLMRDkOZ2aCyBo0qYgEjBU\n8ky+w0xikdUhk+WpCN7mn+IrkwtFbmGPloRMk92KJDHPem/eMd3RVSdltw8th5sXRjdL/vkW4sUy\npVTiV54HNC6M+OK4kz88ZiBS5v8CAL6pKo/++KCmYzmxPhHSADQADUAD0AA0AA1AA9AANAANQAPQ\nADQADUAD0AA0AA1AA9AANAANQAPQADQADUAD0AA0AA1AA9AANAANQAPQADSAq/YfHWIiCtL9d5AA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image at 0x2a96290>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='scidb.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$$\\,$$\n",
    "\n",
    "Introduction to SciDB-py package, and SciDB as a Tick database\n",
    "====\n",
    "\n",
    "* https://github.com/Paradigm4/TAQ/blob/master/TAQ_ipynb.ipynb\n",
    "\n",
    "* https://github.com/paradigm4/SciDB-py\n",
    "* http://www.paradigm4.com/forum\n",
    "* http://www.paradigm4.com\n",
    "\n",
    "  \n",
    "$$\\,$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Loading the scidbpy package and connecting to SciDB\n",
    "====\n",
    "\n",
    "This step requires a connection to a SciDB cluster. This notebook assumes SciDB is available at http://localhost:8080. Replace this address with one that points to a SciDB connection.\n",
    "\n",
    "Ready-to-roll SciDB AMIs for Amazon EC2 are available. Information is available on the SciDB forum at http://www.paradigm4.com."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scidbpy import interface, SciDBQueryError, SciDBArray\n",
    "sdb = interface.SciDBShimInterface('http://localhost:8080')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating SciDB array objects...\n",
    "====\n",
    "...from a numpy array\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = np.random.random((5, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_sci = sdb.from_array(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 4)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SciDBArray objects provide a mix of familiar numpy things and some SciDB-specific things:\n",
    "X_sci.shape      # The usual shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'py1101086155612_00002'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_sci.name       # SciDB array name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...using an array generating function\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a 10x10 array of double-precision zeros:\n",
    "A = sdb.zeros( (10,10,) )\n",
    "\n",
    "# Create a 10x10 array of 64-bit signed integer ones:\n",
    "B = sdb.ones( (10,10), dtype='int64' )\n",
    "\n",
    "# Create a 10x10 array of numbers between -1 and 2 (inclusive) sampled from a uniform random distribution.\n",
    "C = sdb.random( (10,10), lower=-1, upper=2)\n",
    "\n",
    "# Create a vector of 5 equally spaced numbers between 1 and 10, including the endpoints:\n",
    "D = sdb.linspace(1,10,num=5,endpoint=True)\n",
    "\n",
    "# Create a 10x10 sparse, double-precision-valued identity matrix:\n",
    "E = sdb.identity(10, dtype='double', sparse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...from a SciDB query\n",
    "----\n",
    "\n",
    "The scidbpy package query method simplifies writing low-level SciDB AFL queries involving SciDBArray objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tridiag = sdb.new_array((10, 10))\n",
    "sdb.query('store(build({A}, \\\n",
    "                 iif({A.d0}={A.d1}, 2, -1), \\\n",
    "                 {A.d0} <= {A.d1}+1 and {A.d0} >= {A.d1}-1), \\\n",
    "           {A})', A=tridiag)\n",
    "tridiag.nonempty()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...from an existing SciDB array\n",
    "----\n",
    "\n",
    "This example array holds one day of ARCA consolidated quote data. It has about 185 million elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "185857392"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quotes = sdb.wrap_array(scidbname=\"arca_seq\")\n",
    "quotes.count()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scope of SciDB arrays connected to SciDBArray objects\n",
    "----\n",
    "The `persistent` argument of the `new_array` method determines if a SciDB array should be removed when its corresponding Python reference falls out of scope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "py1101086155612_00022\n"
     ]
    }
   ],
   "source": [
    "X = sdb.random((10,10), persistent=False)\n",
    "Y = X\n",
    "\n",
    "xname = X.name\n",
    "print(xname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now let's delete the variable X:\n",
    "del(X)\n",
    "\n",
    "# and check to see if its array is still in SciDB:\n",
    "sdb.list_arrays().has_key(xname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# OK, let's delete the other reference to the SciDB array:\n",
    "del(Y)\n",
    "\n",
    "# and check to see if its array is still in SciDB:\n",
    "sdb.list_arrays().has_key(xname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Materializing data to Python\n",
    "===="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [-1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2.]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Materialize SciDB array to Python as a numpy array:\n",
    "tridiag.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<10x10 sparse matrix of type '<type 'numpy.float64'>'\n",
       "\twith 28 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Materialize SciDB array to Python as a sparse array:\n",
    "from scipy import sparse\n",
    "tridiag.tosparse('csr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operations on `SciDBArray` objects\n",
    "====\n",
    "Subarrays\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a 3x10 subarray (returned as a new SciDBArray object)\n",
    "X = tridiag[2:5,:]\n",
    "X.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  2.,  0.,  0.,  0.],\n",
       "       [ 0., -1., -1.,  0.,  0.],\n",
       "       [ 0.,  0.,  2.,  0.,  0.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = tridiag[2:5,::2]\n",
    "X.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data aggregation\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sum over all elements in 'tridiag'\n",
    "tridiag.sum()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.5,  3. ,  3. ,  3. ,  3. ,  3. ,  3. ,  3. ,  3. ,  4.5])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the variance along dimension 1 (columns)\n",
    "tridiag.var(1).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...also `min` `max` `std` `mean` `approxdc` and `count`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shape and layout functions\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(20, 5)\n",
      "(5, 20)\n"
     ]
    }
   ],
   "source": [
    "tri_reshape = tridiag.reshape((20,5))\n",
    "print tri_reshape.shape\n",
    "print tri_reshape.transpose().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arithmetic\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4.4408920985006262e-16"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = sdb.from_array(np.random.random((10,10)))\n",
    "Y = sdb.from_array(np.random.random((10,10)))\n",
    "\n",
    "S = X + Y\n",
    "D = X - Y\n",
    "M = 2 * X\n",
    "(S + D - M).sum()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SciDBArray('not empty py1101266337305_00044<multiply:double> [i0=0:9,1000,0,i1=0:9,1000,0]')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sdb.dot(X,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.88774937,  1.7899071 ,  2.02491303,  1.72463329,  2.58387152,\n",
       "        1.70449874,  1.0778353 ,  2.45668368,  1.43828023,  1.75958135])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sdb.dot(X.T, Y[:,1]).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadcasting\n",
    "----\n",
    "We center a matrix by subtracting from each column its mean (also illustrating the aggregates along a coordinate axis):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -7.77156117e-17,  -7.21644966e-17,  -1.11022302e-17,\n",
       "        -3.88578059e-17,  -3.88578059e-17])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a small test array with 5 columns:\n",
    "X = sdb.random((10,5))\n",
    "\n",
    "# Create a vector of column means directly using a SciDB aggregation function:\n",
    "xcolmean = X.mean(1)\n",
    "\n",
    "# Subtract the column means from the original matrix using broadcasting:\n",
    "XC = X - xcolmean\n",
    "\n",
    "# Check the mean of one of the columns of the centered matrix:\n",
    "XC.mean(1).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\,$$\n",
    "Putting things together for...\n",
    "====\n",
    "Covariance and correlation matrices\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a small test array with 5 columns:\n",
    "X = sdb.from_array(np.random.random((10,5)))\n",
    "\n",
    "# Create a second small test array with 10 columns:\n",
    "Y = sdb.from_array(np.random.random((10,10)))\n",
    "\n",
    "# Subtract the column means from X using broadcasting:\n",
    "xcolmean = X.mean(1)\n",
    "XC = X - xcolmean\n",
    "\n",
    "# Similarly subtract the column means from Y:\n",
    "ycolmean = Y.mean(1)\n",
    "YC = Y - ycolmean\n",
    "\n",
    "# Compute the covariance matrix between X and Y:\n",
    "COV = sdb.dot(XC.T, YC)/(X.shape[0] - 1)\n",
    "\n",
    "# Column vector with column standard deviations of X matrix:\n",
    "xsd = X.std(1).reshape((5,1))\n",
    "# Row vector with column standard deviations of Y matrix:\n",
    "ysd = Y.std(1).reshape((1,10))\n",
    "# Their outer product (outer is not yet overloaded in the package):\n",
    "outersd = sdb.dot(xsd,ysd)\n",
    "\n",
    "# Element-wise division of the matrices gives us the correlation between X and Y:\n",
    "COR = COV/outersd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\,$$\n",
    "$$\\,$$\n",
    "\n",
    "# SciDB as a Tick Database \n",
    "\n",
    "### Connect to the SciDB DBMS\n",
    "\n",
    "The below imports the required libraries and establishes connectivity to the SciDB DBMS. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "from scidbpy import interface, SciDBQueryError, SciDBArray\n",
    "import matplotlib.pyplot as plt\n",
    "from scidbpy import connect\n",
    "import time\n",
    "\n",
    "#sdb = connect('http://localhost:5959')\n",
    "#afl = sdb.afl\n",
    " \n",
    "sdb = connect(\"http://10.0.20.185:8080\")\n",
    "afl = sdb.afl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing best bid/offer and order books\n",
    "----\n",
    "https://github.com/Paradigm4/orderbook-example\n",
    "\n",
    "SciDB can be extended using any language, as long as it's C++. (That's not quite true, we're working on ways to extend internal SciDB functionality with Python and R.)\n",
    "\n",
    "arca is a time by symbol data matrix,\n",
    "ms is the name of the time dimension,\n",
    "order_records is a specially formatted string input attribute for orderbook.\n",
    "The orderbook function expects the input attribute to have the following special string form:\n",
    "\n",
    "order_type, ref_id, price, size, symbol, ask_bid_type|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46.5135610104\n"
     ]
    }
   ],
   "source": [
    "sdb.query(\"load_library('nbbo')\")\n",
    "import time\n",
    "start_time = time.time()\n",
    "sdb.query(\"variable_window(arca, ms, 1, 0, nbbo(order_records))\")\n",
    "print(time.time() - start_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nbbo = sdb._execute_query(\"variable_window(  \\\n",
    "                        between(arca,'DEXO',34200000,'DEXO',57600000), \\\n",
    "                        ms,1,0,nbbo(order_records))\",response=True,fmt=\"lcsv+\").split(\"\\n\")[1:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(34200057.0, 1.6399999999999999, 480.0, 1.97, 480.0),\n",
       " (34200641.0, 1.6399999999999999, 480.0, 1.97, 480.0),\n",
       " (34202223.0, 1.6399999999999999, 480.0, 1.97, 480.0),\n",
       " (34202411.0, 1.6399999999999999, 480.0, 1.8700000000000001, 1000.0)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nbbo = [tuple(double(y) for y in x.replace(\"'\",\"\").split(\",\")[1:]) for x in nbbo]\n",
    "nbbo[0:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is a 2-d array with the same dimensions as the input array and a single string attribute with the format\n",
    "\n",
    "bp1, bv1, bp2, bv2, bp3, bv3 |  ap1, av1, ap2, av2, ap3, av3\n",
    "where, bp1 means \"bid price 1\", \"bv1 means bid volume 1\", and ap1 and av1 are similarly defined for ask prices and volumes, and bp1 < bp2 < bp3 < ap1 < ap2 < ap3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIsAAAFlCAYAAABr+Wg7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3W+MHOV9B/Df+k9kJaQxUDjHd5bcyrg+EgxW7SISER2C\nszGUE6FV6igVp+SMLSQUIVKBX5SUVkKxXyBE7L6wE5fYSoUcqQJbgpxiix4SUW2/IUpVinCkWhxn\nnxPJWCmJEOTu+oLusLe3f2Zn9vb2dj+fN/buPDPzzOzc7Mx3n+eZwszMzEwAAAAAQEQsWegKAAAA\nANA+hEUAAAAAJIRFAAAAACSERQAAAAAkhEUAAAAAJIRFAAAAACRyh0Wjo6OxYcOGuOGGG2Lfvn1z\npv/rv/5r3HzzzbFx48b48pe/HL/85S9TzwsAAABAaxVmZmZmss48NTUVf/ZnfxanTp2K3t7e2LJl\nS7zwwgvR39+flPmP//iPuPHGG+Nzn/tcjI6OxlNPPRWnT59ONS8AAAAArZWrZdHZs2dj3bp1sXbt\n2li+fHns2LEjjh8/PqvMbbfdFp/73OciIuLWW2+Nd999N/W8AAAAALRWrrBoYmIi1qxZk7zu6+uL\niYmJquUPHz4c99xzT6Z5AQAAAJh/y/LMXCgUUpf993//9/iXf/mX+PnPf97QvGNjYzE2Npa8HhgY\niIGBgUaqCQAAAEBKucKi3t7eGB8fT16Pj49HX1/fnHK//OUv46GHHorR0dG4+uqrG5pXOAQAAADQ\nOrm6oW3evDnOnTsX58+fjw8//DCOHTsWQ0NDs8q888478cADD8SPf/zjWLduXUPzAgAAANBauVoW\nLVu2LA4cOBDbtm2LqampGBkZif7+/jh48GBEROzevTv+6Z/+Kd577714+OGHIyJi+fLlcfbs2arz\nAgAAALBwCjMzMzMLXQkAAAAA2kOubmgAAAAAdBZhEQAAAAAJYREAAAAACWERAAAAAAlhEQAAAAAJ\nYREAAAAACWERAAAAAAlhEQAAAAAJYREAAAAACWERAAAAAAlhEQAAAAAJYREAAAAACWERAAAAAAlh\nEQAAAAAJYREAAAAACWERAAAAAAlhEQAAAAAJYREAAAAACWERAAAAAAlhEQAAAAAJYREAAAAACWFR\ni0xPZ5sGAAAA0EqFmZmZmYWuRLc4dKjy+7t2tbYeAAAAANVoWQQAAABAQlgEAAAAQEJYBAAAAEBC\nWAQAAABAQlgEAAAAQCJ3WDQ6OhobNmyIG264Ifbt2zdn+ltvvRW33XZbrFixIp555plZ09auXRsb\nN26MTZs2xV/8xV/krQoAAAAAOS3LM/PU1FQ88sgjcerUqejt7Y0tW7bE0NBQ9Pf3J2Wuvfba2L9/\nf7z00ktz5i8UCjE2NhbXXHNNnmoAAAAA0CS5WhadPXs21q1bF2vXro3ly5fHjh074vjx47PKXHfd\ndbF58+ZYvnx5xWXMzMzkqQIAAAAATZQrLJqYmIg1a9Ykr/v6+mJiYiL1/IVCIe66667YvHlz/OAH\nP8hTFQAAAACaIFc3tEKhkGvlP//5z+Pzn/98/OY3v4nBwcHYsGFD3H777bPKjI2NxdjYWPJ6YGAg\nBgYGcq0XAAAAgMpyhUW9vb0xPj6evB4fH4++vr7U83/+85+PiI+7qn31q1+Ns2fPzgmLhEMAAAAA\nrZOrG9rmzZvj3Llzcf78+fjwww/j2LFjMTQ0VLFs+dhEv//97+N///d/IyLid7/7XfzsZz+Lm266\nKU91AAAAAMgpV8uiZcuWxYEDB2Lbtm0xNTUVIyMj0d/fHwcPHoyIiN27d8fk5GRs2bIlfvvb38aS\nJUviueeeizfffDN+/etfxwMPPBAREX/4wx/iG9/4RmzdujX/FgEAAACQWWHG48ha5tChyu/v2tXa\negAAAABUk6sbGgAAAACdRVgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABA\nQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBC\nWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJY\nBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAECiMDMzM7PQlegG09MRS6pEc7WmAQAAALSSsKiFDh2q\n/P6uXa2tBwAAAEA1uduzjI6OxoYNG+KGG26Iffv2zZn+1ltvxW233RYrVqyIZ555pqF5AQAAAGit\nXGHR1NRUPPLIIzE6OhpvvvlmvPDCC/Hf//3fs8pce+21sX///vi7v/u7hucFAAAAoLVyhUVnz56N\ndevWxdq1a2P58uWxY8eOOH78+Kwy1113XWzevDmWL1/e8LwAAAAAtNayPDNPTEzEmjVrktd9fX1x\n5syZps47NjYWY2NjyeuBgYEYGBjIXGcAAAAAqssVFhUKhXmfVzgEAAAA0Dq5uqH19vbG+Ph48np8\nfDz6+vrmfV4AAAAA5keusGjz5s1x7ty5OH/+fHz44Ydx7NixGBoaqlh2ZmYm87wAAAAAtEaubmjL\nli2LAwcOxLZt22JqaipGRkaiv78/Dh48GBERu3fvjsnJydiyZUv89re/jSVLlsRzzz0Xb775Zlx1\n1VUV5wUAAABg4RRmypv8MG8OHar8/q5dra0HAAAAQDW5uqEBAAAA0FmERQAAAAAkhEUAAAAAJIRF\nAAAAACSERQAAAAAkhEUAAAAAJIRFAAAAACSERQAAAAAkhEUAAAAAJIRFAAAAACSERQAAAAAkhEUA\nAAAAJIRFAAAAACSERQAAAAAkhEUAAAAAJIRF0Camp/NNBwAAgGZYttAVAD62ZEnEoUPVp+/a1bq6\nAAAA0L20LAIAAAAgoWVRi0xPV28ZMj39casSAAAAgIVWmJmZmVnoSnSLal2MdC+aq1aA1qnhWr3t\n6tTtBgAAoL1oWURbqjV+T6eGa8YsAgAAoB1opwAAAABAQlgEAAAAQEJY1MGmp/NNBwAAALqPMYs6\nWKUxcHbu/GSQ5PLBkg2gDAAAAAiLukw3DhwNAAAApKcdCQAAAAAJLYta6MEHIz71qepdvearG9iD\nD0asWNH85ZKPbn8AAAC0o9xh0ejoaDz66KMxNTUVO3fujCeeeGJOmW9/+9vx05/+ND796U/Hj370\no9i0aVNERKxduzb+6I/+KJYuXRrLly+Ps2fP5q1OWysGNq3uBrZixSfr1NWsfZR3CfTZAAAA0A5y\nhUVTU1PxyCOPxKlTp6K3tze2bNkSQ0ND0d/fn5R55ZVX4le/+lWcO3cuzpw5Ew8//HCcPn06IiIK\nhUKMjY3FNddck28rWNS0sAEAAID2kSssOnv2bKxbty7Wrl0bERE7duyI48ePzwqLTpw4EcPDwxER\nceutt8aVK1fi0qVL0dPTExERMzMzeaqwaCxEIDI9/fHTz+qp1U2tFfWuNOi2VjYAAACwMHKFRRMT\nE7FmzZrkdV9fX5w5c6ZumYmJiejp6YlCoRB33XVXLF26NHbv3h0PPfRQnuq0tYVoOZN2naXd1MoJ\nbZhvtQJJrc4AAABaL1dYVCgUUpWr1nro9ddfj9WrV8dvfvObGBwcjA0bNsTtt98+q8zY2FiMjY0l\nrwcGBmJgYCBrlWlzBuPuPpValhUVw0qBEgAAQOvkCot6e3tjfHw8eT0+Ph59fX01y7z77rvR29sb\nERGrV6+OiIjrrrsuvvrVr8bZs2fnhEXCoe5SbOXULS2aRkYili5d6Fq0v9JAqTxQLA2KBEcAAAD5\n5QqLNm/eHOfOnYvz58/H6tWr49ixY/HCCy/MKjM0NBQHDhyIHTt2xOnTp2PlypXR09MTv//972Nq\naio++9nPxu9+97v42c9+Fv/wD/+Qa2NgsVm61JPqGpW222S94ChrayWBFAAA0OlyhUXLli2LAwcO\nxLZt22JqaipGRkaiv78/Dh48GBERu3fvjnvuuSdeeeWVWLduXXzmM5+J559/PiIiJicn44EHHoiI\niD/84Q/xjW98I7Zu3Zpzc+gU09PVwxM3650lzWddq0y11lm1urdF1A7n0nSNAwAA6FS5wqKIiO3b\nt8f27dtnvbd79+5Zrw8cODBnvj/90z+NX/ziF3lXzyJXLfjpxpv1WoFIcXonhmRpPuvSMuX7qLR1\nVqV5AQAAaEzusAjy6MTwI6s8LWEAAACgWdyqAwAAAJDQsqgL1OvetNAq1c+YRezc+fG/WT7vBx9s\nfn06wfT07NdZBwAHAAA6m7CoC9Qa76UdVKpfN45ZxGzFYyDL571iRfPr0wnKwx/dHgEAgEqERbSV\n6elPWpQU7dzpMeYAAADQKsIiWqpeuDNfT0Yrdr9pRujUzGUx/x58sHpLI58VAADAXMIiEiMjEYXC\nx/9vZhBSb55aYxaVj7HSqOK6i+vPEzqV1qUYYFUKIkq3VRix8Fas0KURAACgEcIiEkuXfvL/Zt5c\nl7YMKu9iVj69fMyivDfzS5ZEfPBBc8awqRT6rFhRe/lLlizewKhY7zRh33xt386di3f/tZtGBrcG\nAAC6m7BoEVjIm/VmK+9S1oyWHfW2f74HO67VciVi8bZeKQ3sFmr7FstxvRjYlwAAQFrCokWg0pg9\npYM+l3d7Kn8vrdKWB5W6hWUNrCp1MytfZ6UWR+XTqy2/3phGpS1kivUorW+9rm5ZutGlma9TtTrc\nrHV80b3HIQAAkJ2waJGqFpAUb5qzdOEqvaEsX3ae1iWVuplVWmejdUqz7mrrKK9TnpYztT6LTmxx\nVE+t8C6i+dtdaX2dum+zaFaXTgAAoHv4vZkFVdqqJ+9g1vUUb5prBRm1TE/Pfx0BAABgoWlZ1GL1\nusx0W5eR0m1t1+3O0rXvwQezrcexMVe3bncWlY7VNN30Kj3Vr3y5WboW1uueWl7XRtbZCvXq34r6\nLYb9BABA5xEWtViru+gspGo3rsXxiRbLDU6aeta72S5XbYwmPjYy8sn/8x4n9cbMWizHYRrFbSkf\nRL5eN7Q0g7RnmV5rvtIus7WmL6R646G1qg4RC18PAAC6i7CoxYo3ruUDLBcHYC6+jpg9vXSA58XS\nFaraWEFFtW5gF9ugxeU327UG5I6oP45SJYtpf+S1dGnzltUON/ztaLH9jQEAAK0jLFoA5aFQJZUG\nha42SHQnWuyD8nZSa5VGLXR3utJWSVRXa+B5SKsduurNl07eNgBYSL5jFwdhUYtVO/CLwchiDkgg\nYuFbSDWzVRJQWye33OvkbQMgP4FHdr5jFwdhEfMmy8DQi7EOpV8G5WMXpRk4uBknxEpfSKXdG8u7\nPNZaRqUytbajfPkRjbUsqrQPsn7BPvhg+nlHRtrjGE2j0j6u9TnW6wZZLFe63xsdd6sbVPv7LG8V\nWvo6y0Wji00AyG8hvmurXT+XX+v6Lp+tm8YUXcyERcybdvgjb0UdStdRHLuoePKrNnBwcXqzugJV\nSudLW6qVrqNeil+pC2CtAZBLy9UbjLe8fDVZP7dGwo7SFkiV9l07qdUttdSuXXM/g2rbUr7MtJ9x\nN6n2q1f531Hp/smyD/261r7KxwvM8lRAAFqj3vdps79rS6+5imPSFt8v/U6o9oPuQmmH7zPXPouD\nsAiaoJmtY+ZT1hS/VmuVdtxOgGYoD1U9hACAaorXwx980N4/whW/26q1Kq/W04DuIyyCJqjUGqfR\nE2wrBobOmuLXajFU3nKpGXVa6C9RAADIYrF062/nQIv2ICyio7VjKl5tXJ2FHhh6sSqGbLWeLtju\nGh07qRWPvS/9tSntOEjQSdKMr1arfJZy7fidBSysStcIlc5PrTyvOI91l3botsbCEBbR0fKcuEZG\n0j9ZqzSwKP5bbd2L5deGiI8Dg3ZXqVVXGu30xVZvrKfybas1lk4jioFQpX1R+mtTrXGSsqyX9Nrp\nOO028xXoG6cBaESla4RKY+e18rxSeh4r78rUjPF5aj10pdayW/FjWrcp7mM/aHcnYRFUUSikL1sa\nWGQJLtpVM4OtdnvalhvwuQOy034cpwtHUAdQX62uTBH5hihodKw4YXzzFb8Hd+6c/bpcrSczpylL\nexIWQRVZW6yUStPSqFzax9S3QjPXV+1iYufO+l9AzD+/xrWvet0AF/Kiq/y46aSm6oulnkDnWEzn\nyKzydGHL88Njte/SesMYlNapWv2K17GdrLSFW3nvi2r7cMmSTwb7rrQs2p+wCOZRpV9G6t2Qt9Ng\nc604mZeuI89A2eRT/mucz6B9NNpNsZUqHTeaqgNk0w0P/cjT+idPK6Zq36WNfG9Vu1aq1nqmPJxq\nRjBVOq1S+UbGs4qYO0ZlcRm1LF1aeT9U2reVPrNOO6Y7mbAIFomRkYWuQX31vtgqfTm0469opXXN\n2m++kXUVl19t/7TaYhirivZQvBiOqHwct1NLIwOyAu1OK9/Wa+ThBY18PtXGmip/r1StYKpSudIy\npUNh1PsRp3Q95WNUFpcxNZV+7NZaKl0XsHgIi6DN1Oq61uqWBY12n6ulWYMyt1ppE9r5aDFR3L/t\ntH/aaWwp2lt5y8BGBl0tTm8VY1kA7a4bWha1QiPd1RoZvLmbWmEXWw/l3cbywdA/9an8daN1codF\no6Oj8eijj8bU1FTs3LkznnjiiTllvv3tb8dPf/rT+PSnPx0/+tGPYtOmTannhW5TbVC/rCfrRn+l\nKrYUSNuipqhdA4Ysv9KV/1LTrttGdyj+Cldv3KKIfC1n8rS8Sft3Vm28iGY8ArrRc1anaqfWXAAL\nQden9uR6evHJFRZNTU3FI488EqdOnYre3t7YsmVLDA0NRX9/f1LmlVdeiV/96ldx7ty5OHPmTDz8\n8MNx+vTpVPNCpyvtzzyfTY/T3hxUKrfYbyraqcUOZFHa/L1WmbxNuxtteVN6vkj7QIBqv5o38gjo\nRprll9q5s3r9ysdpWMznvWoDitbTzJApzbIi8oWCecq3Yt7y6VmWlaVu5essqlaPWsspnS/t+vPU\nPcv8je63vHVr9DOP0P0VWLxyhUVnz56NdevWxdq1ayMiYseOHXH8+PFZgc+JEydieHg4IiJuvfXW\nuHLlSkxOTsb//M//1J0XOl1pf+bSfsLzuY40ZbNotOWBPsvQmDRN5fN0syoff6hamflsvTOf42WV\nN4UvDVNKQ6+I+TkXt1KWByU00hWjnrTLyhMKpl1fnq6QefdJpfFEGllWlq6T5essqlaPavVqZGyV\nevVodN408xfPQWkC6EbW3+zPPEL3V2DxyhUWTUxMxJo1a5LXfX19cebMmbplJiYm4sKFC3XnBZqr\n1mND8978NTpvM1pCAJ+o1ToxTchTPv5QJbWe/NIMrWqiXi1McfMGi0Pa1owAZFeYmZmZyTrzv/3b\nv8Xo6Gj84Ac/iIiIH//4x3HmzJnYv39/Uua+++6LPXv2xJe//OWIiLjrrrti3759cf78+brzRkSM\njY3F2NhY8npgYCAGBgayVnnBzExPR6FWgdKr+PJ2q/Velyqdlqb9cel7lcqXl6liZno6CmXTZ21z\ntTbRjfSryNCOd85+r7Qf6rXlrlbHSvWrti+rtQuvNm+lu7p621nvc63VxyJtapN2P9VrG19tnfX2\nUbXlVPucat0dpz3e0xyDUeUzSKHi30mlejby919ePu1xnOWcEzW2vc4xGxFV91nW/dlMqbcrxTFS\na3tqbmv555bxGCxOS72uNMspn5jmvFWvz0iOfht1tzHteb90W0rrkuF7sNFyzTjuk8+mwXXUW3cj\ndUuzrIhsf/9VrzcyHDNZ1pV5etbrqDyfY5rrwDR/Cw2sP0/dG5k/4v+Pn3rnlCbXreHPPKKhfVj1\neq58OXmvEXPKc2w39F3VwHbOWna14yIi1/Eyp+6VrvdLX1eS9rOqVa9K9wqN3D9Uukeo9n6l65Bq\n10H19nnKa5p2uAbkY7laFvX29sb4+Hjyenx8PPr6+mqWeffdd6Ovry8++uijuvNGLN5wqFzdA75S\nP6Rq7VhrtWstnZam/XHpe5XKl5epolDhp51Z21ytTXStn4Tq1SlF3ebUq9J+qNUGu7QO5Se8SvWr\nti/L11eufN4GHytUSPO5Vjq2SsumkWY/1StXa5319lG15VQ7lirtx0rrr3W8pzkGo/LfQBoV/07K\n61I6Lc3ff3n5Wn9rec85UeX4K113tXnqLXOBFZYsafxYzrA9VddTvuway0/WUWf9qdeVZjnl0py3\nap2rSuuQ43yf6nNL0/em2NelqNa+i/THbK3PoBnHfaH0O6qBddQ8NhqsW+plNXpOqVKPusd1xu3K\nu09mzZ/1OirP5ziP/dAa2vYG5009f7Xv+Hb6zIv1TLmsiuuvtJxGrxGbLM+xneazTco0sJ3J/LWO\ni4hcx8uculfbt7XOwzUe1VazYcF8WLKk+iB3pZ9BqWqfS3FaqWrXk6XLqqAdrgH5WK6waPPmzXHu\n3Lk4f/58rF69Oo4dOxYvvPDCrDJDQ0Nx4MCB2LFjR5w+fTpWrlwZPT09ce2119adF9qKhBugu1S7\n8QaAUpVayVb73qjWaqj0vVr3HcVplcqXz1drPdPT1Qfwm67Rt52ukSssWrZsWRw4cCC2bdsWU1NT\nMTIyEv39/XHw4MGIiNi9e3fcc8898corr8S6deviM5/5TDz//PM154V5UaeJacM++MAJFAAAqN36\ntPyeoV1+gE4TSNHVcoVFERHbt2+P7du3z3pv9+7ds14fOHAg9bwwL5pxwivtjlBM4bOGUM0Or5qp\n3pghC8GvG3S7dj5nAADQcXKHRTSo2iCzxbERiv+6KWg/tcbByLKsdlTst1xvPKlW0xWEblX8PiiO\nK9BOx7/vKgDqfTdl+a4ovSdq5HvPj4vQVMKiVqs0YFulgY1Jr91uoNpZtT7UxS/yo0ftS2gnpRfY\nR49WLlP8m611kVz6hJNaZcqXWWs5AFD+3dSMLlfVxt9JM597qflRbxymSqoNnu3HpkVDWLRYVbvg\nz/KH3O7SbFOax0vSWH/qrLL8qtOs4/aDDyI+9anu/rzT7kt/F/XVOs+Wvt9IyNLIekrnqxWKl7Y+\nqibNhXea80PWC3gAiEj3nZdl3jTrjWjuPVL5EzM72Q9/+Mn/R0YqXwc1qxWYa9S2ISxarEr/YMt1\nUlAU0dgNTLX5+ViaVlh5m/Bm6a5X71GkaeukZVT9fVlajtrShqtp9mXWc1RxWq1WRT5LABaLPNfs\neb7vmjmcRKVldtP15+HDs183uu31rq9c17QNYRHtpR0HV+4klW44KzUXbrcmvKVfKjt3tlfdoJuV\nn7Mrtbry9wqQjnNm601Npe+94PNpXKV9plXRoiEsor20Y1CRpotJ1uUulpNhrealpSf8+dim8mWm\nbTnTjSp9Tq24sEnbXat8WkR7Xyy0y8VMu9SjkvJzdiu6ugJ0qkrnUOfOxjUynunhw/Vbv1f6P+mU\n7jPjHS46wiI6WzNupErnb0bz1cU4rk6tG8CIdN0B86y7PJCqFIYs5qcJVhsAsJpa4Uz5tmfdF6UX\nWmmCnUoWqql5s7RD99Z2D6Uq/f01GhCWz1PrV968g3g3qta5p9r6G1lm1mXUW347/P0AzZHmh6BW\n/6jQzj9iRKRrSd8M3dLSqFlPvGtWGVpGWLRQym/Eyv8Aa92oNeuCtRu0434ofoF1w5dLXqVdXDp5\n4N1Gx1rK0nqjPNCpN2hyvSdvlS+L+dHKUC1LOFb+d5dnGUXlYyGUSvPrb5qBxNM+JKLa+IB5/g7m\nO4RM89S7WsFfNY1cW6RZVkS+ULCR8nkekJFnn+QNG/N+jpVaRqcNGfK2/swbJqQNnhvt4lJv/c3+\nzIv1TLusautvt7E72+HHlEqynguy3jt1y5hF1a4JI4wx1OGERQulXuKdZzDT0n61pa+r8ej5yub7\n1wKhXn0//GG61gr2Y32VLkgbac0ErTCfP4ak6UpQ3lx+sZ53mhHaNTq9mctqtP7z1YqxmduRZVl5\nP8c0y5yv4yFvmJB2/kb3WyPLzju9WZ/5Yj4XtVralvCV5qMyjRS6mrCok5WPJVFNN7R0Ke/mkyYg\nS7v/smq3E2uabiTzsc60F43GQWmubvk1jMWlnbonOu8A7cC5iHYlKOp4wqJ2Ugww5vMXhG7pW1uq\n2IIiyyC8pWUi0o8TU77sVrRSarQZdrmF6MZVa79FfLLviv+vVrZSq5k02zE1FVEotP/fRKPjB1Xb\n9uJ4WUWtHggbFhO/5gPtwLkovVa3gumGa6d27XJISwiL2klx3JL5/KOsNP5CN5zk6jVJrTdwdXG/\nNzrAdbXxLurN16h2+tWpGV+cpfu7uKxGti/t30m1sVHa7W+iXl/xUrW2vXQ5pcd98byjpRF8wgUy\n0A7a6Rqv3bW6darvAjqcsAgWSqf+GlTtoqbVFzTG4squ1r7r1OMWAABICItorUpP5uhWC911rZ7S\nAdJb3ay3tHtY1mXnGYtrofd9KzTyJLRKrewAgPmnGxqwQIRFtFZ5q5NOvyGvp9L4M+3SMqf8iR61\nmkA3+0Ll8OG53aRaqRsuvMq7pAEA7UeXWGCBCItgIZWOadSON+zlT5Fr9rI/9amFu9DR1eoT3ba9\nAABATcKiTtYNXWmYX8VB15uh3Y7HWtvWbcFJt20vAABQk7CokzV6A1hrPKE8Y9akmTei8vzt0OKh\n1eMsZRmYuV2CmFqfV6Pd69plmzpFvTEPIrS0AgDIwthSdCBhEZ8oPYk1+ojOev2pq409Uzx5lj4i\nPe1yW6W0q1ip+Qoyqj0mvdb62mE/RTS3HsX9LjBqjrR/o5X4DAAAqjO2FB1IWMTCcvJMr1N/sWj3\nVi3F9ddr8ZW1nvWeftfqJ9EBAABdT1hE+2v3kKRaHZrdjWqx/GKRpRvdfNY/7/GzZMnH5aoN9F1c\nRtZtKM5Xq/temvnzEEgBAAAlhEV8Is3YPFluGtPciFabPjUVsXRp493iWqna/qjWfS2iPeo9Xxrt\nRlerC9SuXZ+ENY0ck2nDj7TH8mIJ6rJqRSAFAAAsGsIiPpHmhjDLTWO9x8PXGrPo8OH2D1bavdVF\nu3fzSqP5NtCqAAAPBklEQVReoFSpfL3lAQAAUJGwiPaXtxtR1u5gU1PpW6eUjzvTTt16mjF4cZrP\noN42eboZAADAoiAsov1V6s5VDB3StobKEpYcPlx7eum6y8ed6bSWLWla9qTZ5krLECABAAC0FWFR\nN1ioVi71uj9BRPpWS4u5Gx0AAMAiIizqBs3ohpR1vQtNYNX+8rZaaofjDAAAoIMIi+hsaYKEdhpf\nCAAAABZY5jvgy5cvx+DgYKxfvz62bt0aV65cqVhudHQ0NmzYEDfccEPs27cvef+pp56Kvr6+2LRp\nU2zatClGR0ezVgXy6bTxhQAAACCHzC2L9u7dG4ODg/H444/Hvn37Yu/evbF3795ZZaampuKRRx6J\nU6dORW9vb2zZsiWGhoaiv78/CoVCPPbYY/HYY4/l3ohFRSsWqun0Y8PT0AAAABaFzGHRiRMn4rXX\nXouIiOHh4RgYGJgTFp09ezbWrVsXa9eujYiIHTt2xPHjx6O/vz8iImZmZrKufvGqNz7LYg4DmqnT\ng5PFJu0A0/XKAQAA0PYyh0WXLl2Knp6eiIjo6emJS5cuzSkzMTERa9asSV739fXFmTNnktf79++P\no0ePxubNm+OZZ56JlStXZq0OnaYbQ7WFGog8jR/+sPq00noZiBoAAGDRqxkWDQ4OxuTk5Jz3n376\n6VmvC4VCFAqFOeUqvVf08MMPx3e/+92IiHjyySfjO9/5Thw+fHhOubGxsRgbG0teDwwMxMDAQK1q\n084qtTzp1u5J3brdAAAAtLWaYdHJkyerTuvp6YnJyclYtWpVXLx4Ma6//vo5ZXp7e2N8fDx5PT4+\nHn19fRERs8rv3Lkz7rvvvorrEQ51mPLWM8VWQocOCU4AAACgDWTuFzI0NBRHjhyJiIgjR47E/fff\nP6fM5s2b49y5c3H+/Pn48MMP49ixYzE0NBQRERcvXkzKvfjii3HTTTdlrQosTrplAQAA0IYy363u\n2bMnTp48GevXr49XX3019uzZExERFy5ciHvvvTciIpYtWxYHDhyIbdu2xY033hh/8zd/kwxu/cQT\nT8TGjRvj5ptvjtdeey2effbZJmwOAAAAAHlkHuD6mmuuiVOnTs15f/Xq1fHyyy8nr7dv3x7bt2+f\nU+7o0aNZVw00IuuT5YrzeTIdAABAV3GHB52uXpBTbXrx/azzAwAAsCi5y2PhffCBwa0BAACgTWTu\nhgaZlHdpmp6OWLFi4erDwqvVzU0XNwAAgJZzF0Zrld/4CwKodQw4PgAAAFrOnRgAAAAACWERAAAA\nAAlhEQAAAAAJYREAAAAACWERAAAAAAlhEQAAAAAJYREAAAAAiWULXQFaYHo6Yteu2tOXyA0BAAAA\nLYu6Q70gSFAEAAAA/D8pAQAAAAAJYREAAAAACWER7aXe+EoAAADAvBIW0V6WLIk4dGihawEAAABd\nS1gEAAAAQEJYBAAAAEBCWAQAAABAYtlCVwD4f/UG956e/nhMJwAAAJhH7jyhXdQLggRFAAAAtIC7\nTwAAAAASwiIAAAAAEsIiAAAAABLCIgAAAAASwiIAAAAAEsIiAAAAABKZw6LLly/H4OBgrF+/PrZu\n3RpXrlypWO5b3/pW9PT0xE033ZRpfgAAAABaJ3NYtHfv3hgcHIy333477rzzzti7d2/Fct/85jdj\ndHQ08/wAAAAAtE7msOjEiRMxPDwcERHDw8Px0ksvVSx3++23x9VXX515fgAAAABaJ3NYdOnSpejp\n6YmIiJ6enrh06VJL5wcAAACg+ZbVmjg4OBiTk5Nz3n/66adnvS4UClEoFDJXotb8Y2NjMTY2lrwe\nGBiIgYGBzOsCAAAAoLqaYdHJkyerTuvp6YnJyclYtWpVXLx4Ma6//vqGVpx2fuEQAAAAQOtk7oY2\nNDQUR44ciYiII0eOxP3339/S+elA09MLXQMAAADoepnDoj179sTJkydj/fr18eqrr8aePXsiIuLC\nhQtx7733JuW+/vWvx5e+9KV4++23Y82aNfH888/XnJ8utiTz4QgAAAA0Sc1uaLVcc801cerUqTnv\nr169Ol5++eXk9QsvvNDQ/AAAAAAsHE05AAAAAEgIiwAAAABICIsAAAAASGQeswjm1QcfROzaVXna\n9LTBsAEAAGCeCItoT0ePzn2vGB4JigAAAGDeuOsGAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEs\nAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwC\nAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIA\nAAAgISwCAAAAICEsAgAAACAhLAIAAAAgkTksunz5cgwODsb69etj69atceXKlYrlvvWtb0VPT0/c\ndNNNs95/6qmnoq+vLzZt2hSbNm2K0dHRrFUBAAAAoEkyh0V79+6NwcHBePvtt+POO++MvXv3Viz3\nzW9+s2IQVCgU4rHHHos33ngj3njjjbj77ruzVgUAAACAJskcFp04cSKGh4cjImJ4eDheeumliuVu\nv/32uPrqqytOm5mZybp6AAAAAObBsqwzXrp0KXp6eiIioqenJy5dutTwMvbv3x9Hjx6NzZs3xzPP\nPBMrV66cU2ZsbCzGxsaS1wMDAzEwMJC12gAAAADUUDMsGhwcjMnJyTnvP/3007NeFwqFKBQKDa34\n4Ycfju9+97sREfHkk0/Gd77znTh8+PCccsIhAAAAgNapGRadPHmy6rSenp6YnJyMVatWxcWLF+P6\n669vaMWl5Xfu3Bn33XdfQ/PToaanI3btqj5tiQf4AQAAwHzKfOc9NDQUR44ciYiII0eOxP3339/Q\n/BcvXkz+/+KLL855WhpdqlYYJCgCAACAeZf57nvPnj1x8uTJWL9+fbz66quxZ8+eiIi4cOFC3Hvv\nvUm5r3/96/GlL30p3n777VizZk08//zzERHxxBNPxMaNG+Pmm2+O1157LZ599tmcmwIAAABAXoUZ\njyRrvUOHKr9frfsVAAAAQIvo1wMAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJY1Eof\nfLDQNQAAAACoSVjUSkePLnQNAAAAAGoSFgEAAACQEBa1yvR0xK5dC10LAAAAgJqERa2yxK4GAAAA\n2p8EAwAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACA\nhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICEsAgAAACAhLAIAAAAgISwCAAAAICE\nsAgAAACAhLAIAAAAgETmsOjy5csxODgY69evj61bt8aVK1fmlBkfH4877rgjvvCFL8QXv/jF+P73\nv9/Q/AAAAAC0VuawaO/evTE4OBhvv/123HnnnbF37945ZZYvXx7PPvts/Nd//VecPn06/vmf/zne\neuut1PMDAAAA0FqZw6ITJ07E8PBwREQMDw/HSy+9NKfMqlWr4pZbbomIiKuuuir6+/tjYmIi9fwA\nAAAAtFbmsOjSpUvR09MTERE9PT1x6dKlmuXPnz8fb7zxRtx6662Z5gcAAABg/i2rNXFwcDAmJyfn\nvP/000/Pel0oFKJQKFRdzvvvvx9//dd/Hc8991xcddVVc6bXmn9sbCzGxsaS1wMDAzEwMFCr2gAA\nAABkVDMsOnnyZNVpPT09MTk5GatWrYqLFy/G9ddfX7HcRx99FH/1V38Vf/u3fxv3339/w/MLhwAA\nAABaJ3M3tKGhoThy5EhERBw5cmRWEFQ0MzMTIyMjceONN8ajjz7a8PwAAAAAtFZhZmZmJsuMly9f\njq997WvxzjvvxNq1a+MnP/lJrFy5Mi5cuBAPPfRQvPzyy/H666/HV77yldi4cWPSzex73/te3H33\n3VXn7wqHDlV+f9eu1tYDAAAAoEzmsIgchEUAAABAm8rcDQ0AAACAziMsAgAAACChG1qrTU9HLKmS\n0dWaBgAAANACwqKFYMwiAAAAoE0Ji1pNyyIAAACgjQmLAAAAAEhoxgIAAABAQlgEAAAAQEJYBAAA\nAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAA\nQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABAQlgEAAAAQEJYBAAAAEBCWAQAAABA\nQlgEAAAAQEJYBAAAAEBCWAQAAABAInNYdPny5RgcHIz169fH1q1b48qVK3PKjI+Pxx133BFf+MIX\n4otf/GJ8//vfT6Y99dRT0dfXF5s2bYpNmzbF6Oho1qoAAAAA0CSFmZmZmSwzPv744/HHf/zH8fjj\nj8e+ffvivffei717984qMzk5GZOTk3HLLbfE+++/H3/+538ex48fjw0bNsQ//uM/xmc/+9l47LHH\nmrIhAAAAAOSXuWXRiRMnYnh4OCIihoeH46WXXppTZtWqVXHLLbdERMRVV10V/f39MTExkUzPmFMB\nAAAAME8yh0WXLl2Knp6eiIjo6emJS5cu1Sx//vz5eOONN+LWW29N3tu/f3/cfPPNMTIyUrEbGwAA\nAACtVbMb2uDgYExOTs55/+mnn47h4eF47733kveuueaauHz5csXlvP/++zEwMBB///d/H/fff39E\nRPz617+O6667LiIinnzyybh48WIcPnx4zrxjY2MxNjaWvB4YGIiBgYFUGwcAAABAYzKPWbRhw4YY\nGxuLVatWxcWLF+OOO+6It956a065jz76KP7yL/8ytm/fHo8++mjFZZ0/fz7uu++++M///M8sVQEA\nAACgSTJ3QxsaGoojR45ERMSRI0eSFkOlZmZmYmRkJG688cY5QdHFixeT/7/44otx0003Za0KAAAA\nAE2SuWXR5cuX42tf+1q88847sXbt2vjJT34SK1eujAsXLsRDDz0UL7/8crz++uvxla98JTZu3BiF\nQiEiIr73ve/F3XffHQ8++GD84he/iEKhEH/yJ38SBw8eTMZAAgAAAGBhZA6LAAAAAOg8mbuhAQAA\nANB5hEUAAAAAJIRFAAAAACSERQAAAAAkhEUAAAAAJIRFAAAAACT+D/RglJLE7b+WAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x5d39150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Data to be represented\n",
    "markettime = [x[0]/60000 for x in nbbo]\n",
    "\n",
    "avolwtspread = [(x[3]-x[1])*x[2]/(x[2] + x[4]) for x in nbbo]\n",
    "bvolwtspread = [-(x[3]-x[1])*x[4]/(x[2] + x[4]) for x in nbbo]\n",
    "\n",
    "fig = plt.figure(figsize=(20,6),facecolor=\"white\")\n",
    "axes = plt.subplot(111, axisbelow=True)\n",
    "\n",
    "plt.bar(markettime, avolwtspread, width=3, facecolor='#9999ff', edgecolor='white')\n",
    "plt.bar(markettime, bvolwtspread, width=3, facecolor='#ff9999', edgecolor='white')\n",
    "\n",
    "axes.set_xticks([])\n",
    "\n",
    "axes.spines['top'].set_color('none')\n",
    "axes.spines['left'].set_color('none')\n",
    "axes.spines['right'].set_color('none')\n",
    "axes.spines['bottom'].set_color('none')\n",
    "axes.yaxis.set_ticks_position('left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parallel loading in SciDB using the accelerated load tools. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SciDB Enterprise Edition includes the aio_input operator which allows the ingest of files from around the file system of a distributed cluster. You have the ability to pick a number of paths and the SciDB instance number. Each selected SciDB instance will load from the corresponding data path. This can provide the benefit of loading from multiple IO devices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sdb.query('\n",
    "insert(\n",
    "        redimension(\n",
    "                project(\n",
    "                        apply(                 \n",
    "                            cast(\n",
    "           \n",
    "aio_input('paths=/scidbData/importFifo1;/scidbData/importFifo2;'\n",
    "          ,'instances=3;21;','num_attributes=5','attribute_delimiter=,')\n",
    "\n",
    "                            ,<_security_id:string null,_price:string null,_volume:string null, error:string null>\n",
    "                            [source_instance_id=0:*,1,0,chunk_no=0:*,1,0,line_no=0:*,100000,0])\n",
    "                        ,security_id,int64(_security_id),price,int64(_price),volume,int64(_volume))\n",
    "                ,security_id,price,volume)\n",
    "        ,option_quote,false)\n",
    ",option_quote)\n",
    "')\n",
    "          \n",
    "          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tick Data Operations\n",
    "\n",
    "Link Python objects with SciDB arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trades = sdb.wrap_array('trades')\n",
    "quotes = sdb.wrap_array('quotes')\n",
    "tkr = sdb.wrap_array('tkr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Materialize first 5 values into interpreter from SciDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>        A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>       AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> AA    PR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>     AADR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>     AAIT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                symbol\n",
       "symbol_index          \n",
       "0                    A\n",
       "1                   AA\n",
       "2             AA    PR\n",
       "3                 AADR\n",
       "4                 AAIT"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tkr.todataframe().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looking up trades by symbol string\n",
    "Join with the auxiliary tkr array to look up data by ticker symbol name. Here are examples that count the number of trades and quotes for 'BAM'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6337\n",
      "49667\n"
     ]
    }
   ],
   "source": [
    "symbol = \"BAM\"\n",
    "print(sdb.merge(trades,\n",
    "            tkfilter(\"symbol='%s'\" % symbol)\n",
    "          ).nonempty())\n",
    "print(sdb.merge(quotes,\n",
    "            tkr.filter(\"symbol='%s'\" % symbol)\n",
    "          ).nonempty())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected we see more quotes than trades for this instrument. Note that you can also just filter directly by symbol index using between if you know it. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td> BAM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             symbol\n",
       "symbol_index       \n",
       "615             BAM"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tkr.filter(\"symbol='%s'\" % symbol).todataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'dummy', u'symbol_index', u'ms']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "6337"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(trades.dim_names)\n",
    "trades.between('null', 615, 'null', 'null', 615, 'null').nonempty()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing minute bars\n",
    "\n",
    "Financial data arrive over irregular moments in time. We can use SciDB regrid functions to compute statistics like open price, min/max price, close price, etc., over regular time intervals (aka 'bars'). This example illustrates issuing a complicated low-level query to SciDB and parsing a DataFrame-like list of returned values.\n",
    "\n",
    "The trade data are now organized by symbol, time, and a dummy coordinate that separates collisions (due to, say exchanges) in a sparse array.\n",
    "The following query computes and store one-minute open/high/low/close bars from these data. We need some extra aggregates from the axial_aggregate plugin: load that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "minute_bars = trades.apply('timeprice', 'tuple(ms,price)')\n",
    "minute_bars = sdb.afl.regrid(minute_bars, 1000, 1, 60000,\n",
    "            'axial_first(timeprice) as open',\n",
    "            'max(price) as high',\n",
    "            'min(price) as low',\n",
    "            'axial_last(timeprice) as close'\n",
    "          )\n",
    "minute_bars = minute_bars.slice('dummy', 0)\\\n",
    "                            .eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th>ms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>570</th>\n",
       "      <td> 61.0</td>\n",
       "      <td> 63.500</td>\n",
       "      <td> 60.025</td>\n",
       "      <td> 62.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>571</th>\n",
       "      <td> 62.7</td>\n",
       "      <td> 63.500</td>\n",
       "      <td> 62.200</td>\n",
       "      <td> 63.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>572</th>\n",
       "      <td> 63.6</td>\n",
       "      <td> 64.899</td>\n",
       "      <td> 63.000</td>\n",
       "      <td> 63.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>573</th>\n",
       "      <td> 63.5</td>\n",
       "      <td> 64.500</td>\n",
       "      <td> 63.500</td>\n",
       "      <td> 64.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>574</th>\n",
       "      <td> 64.1</td>\n",
       "      <td> 64.400</td>\n",
       "      <td> 64.000</td>\n",
       "      <td> 64.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  open    high     low  close\n",
       "symbol_index ms                              \n",
       "0            570  61.0  63.500  60.025   62.8\n",
       "             571  62.7  63.500  62.200   63.5\n",
       "             572  63.6  64.899  63.000   63.6\n",
       "             573  63.5  64.500  63.500   64.3\n",
       "             574  64.1  64.400  64.000   64.4"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minute_bars.todataframe().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pull out one of these minute bar time series for a particuar stock, CVS. We can consult the symbols array to find it's index directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1612</th>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             symbol\n",
       "symbol_index       \n",
       "1612            CVS"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "symbol = 'CVS'\n",
    "tkr[tkr == symbol].todataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this says that symbol index 1612 corresponds to CVS.\n",
    "\n",
    "We can use SciDB's cross_join to avoid an explicit index lookup. We do need to use a repart to bring the symbols array schema into a conformable chunking scheme with the minute_bars array. Only the first 10 minutes of bars are shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th>ms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">1612</th>\n",
       "      <th>513</th>\n",
       "      <td> 72.0</td>\n",
       "      <td> 72.0</td>\n",
       "      <td> 67.1</td>\n",
       "      <td> 67.1</td>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>560</th>\n",
       "      <td> 70.1</td>\n",
       "      <td> 70.1</td>\n",
       "      <td> 70.1</td>\n",
       "      <td> 70.1</td>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>561</th>\n",
       "      <td> 70.1</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.1</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>565</th>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> 70.2</td>\n",
       "      <td> CVS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  open  high   low  close symbol\n",
       "symbol_index ms                                 \n",
       "1612         513  72.0  72.0  67.1   67.1    CVS\n",
       "             560  70.1  70.1  70.1   70.1    CVS\n",
       "             561  70.1  70.2  70.1   70.2    CVS\n",
       "             563  70.2  70.2  70.2   70.2    CVS\n",
       "             565  70.2  70.2  70.2   70.2    CVS"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb1 = sdb.merge(minute_bars, \n",
    "                tkr[tkr == symbol]).todataframe()\n",
    "mb1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note! That 570 minutes = 9:30 AM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8ZFWd9/HPt6HZZWsgAVtoQZpVEAREFons6MAoLgzD\nPNAygvOooOK4IGp3fFxwAUVcZhShER9QVBYVRBCIM4oomw47Ct2splmEZt86v/njnKJvKpWkks6t\nquR+369XvVJ16tx7f3Xr5HdPnbspIjAzs6lvWrsDMDOz1nDCNzOrCCd8M7OKcMI3M6sIJ3wzs4pw\nwjczqwgn/GUgaSA/Pt3uWJolaY9C3G9odzzDkfTTHOOR7Y5lrCStKekJSY9LWq/d8SwLSUfk72GJ\npA0nYH59eX5XTER8NjZO+AxqhAOSbq17bxVJjxbeP73w9tXA74H7SoytrI1Kx56AIWlH4K1AP3BW\n3XuzJZ0uaYGkZyU9Jum3ko6WpLq6jb4zJM0tvLdhLjtCTW4IJb1V0sWSFkl6TtI9ki6UtB9ARDwG\nnAasCoz7e5PUJenbku6S9IykRyTdKOkHktYY73zbLGii7UlaWL9RUHKkpCvzung2r5uzJb2urjPz\n7gbz/Gp+7zlJ6+ayPSVdnr/LZyQ9IOm/JB1XmG6jydaxG44TfhKFx2xJ+xbeOwJYgwYNNSJ2iYhd\nI2JQQikpvklD0vRlnMWHSJ/53Ih4oTDfvYHrgTnATOAu4AXg9cB/ABdKqm/TI627Ru+NuK7zxuOn\nwH6kdvEX0v/RPwDvKVT9ASDgCEkvG2mewyxHwMV5ni8HbgMeBDYBDs3Lnqw0epXB34OkFYBfkjak\nbwBWAm4nbVQPAQ6JiN8AC/K0h9dNvxzwT/m9iyPiodyx+CXQQ/oObwaWALuQvs8pxwl/KZGSB8Cx\nhfL3kxrJi0MmqOt91/UwjpD0C0lP5V7IkYXpGg6rFOdXq8PShj8vv3dXof6+kq6QtDj3Tq6WNJaG\nOlPSz3OM90h6KWHlXzYX5NifzL2pOyT1FhO6lv46ulLSRyU9QOqZI2n/3Pv+u6Sn87zOk7TRcAFJ\nWpnUuwf4RaF8JeD7wMrAI8D2EbElsB7w3VztzaTvqxSSjiJtbAK4DNggIraOiJnALODcWt2IuA74\nG7AK8LbCPJodUpsNbJeXtV9EbBcRW5AS/YHAo3l+V+V5nVMX6925/PP59cL8+kxJJ+bv5AFJ75E0\nQ9JP8vd82whtaKvc+30mt4W31C1zq/z9PpR70XdJ+rKkVUdZtc2YC+yb18cPgHUjYtuI6AK2Bmq/\nBuaT/pd3lTSrMP1+QFd+fkb+ezAwHfgrsH5E7BARryB1Jk6cgJg7T0RU/gFcSdqyXwPcSkrumwB7\nAwPAJcBDuc7phekGctmn8+s9CmXPAXeS/jEHSBuT2Q3qvaHR/Ej/7FcVyu7Jr3+S6749ly8BFpJ6\nO0ty7AeP8FmLy36C1EN9sFC2f643I5fdD1wH3J3fHwC+2GDdPQs8D9yY5zkjr4NafNcX1uEuI8S3\nZyGWNQrlBxXK59ZNs1r+LEuAaxusz9Pr6s8tvLdhLjui0XdSN90fc52nSQlntHZ1QZ7fmcOs/4bL\nyfW2KNQ7A9gdWK1BvcMKMa2Zy3YsTLtpLluQXz8DLMrfa63Ozbl9PZLLHivMq7hengBuye8P5O97\nq1xvc+DxXP44cBOpzQ8A/13XXgaAK0ZZdwuAywuv+3MM9wMrjjDdhiz9v/h0ofyHebmLgOVy2RcL\n8f7fvM6nNZjnRnl+n2p3rlrWR9sD6IRHoRH+EXhvfv414ML8Rb+JsSf8H+ayVxfKjm5Qr2HCH6ks\nl9+Vy88qlH0n179thM9aXPb8XLY6KSkvAa7MZdOBzeum/X6e9u4G624JsE8uE7A9S5PHSoX6WwMz\nRoivtv4frSv/SGE5BzaY7ob8/hMN1t3AMI+xJvwn8/t/brJdnZLneXWhbEdS0rwZ2GGU6S+vi/8F\n0kb/iEKdFUhJbAnw/lxWS2RXFeotyGX9wMtIHZra5/0fYHkGb2z3bbBeenPZy1m6gT0jl53J0uQ5\nM5e9pzDtm+vay4gJv249rFOYz4VN1L8s1/9LoX0/nac/qVBvi8LnqK3jR4EfAVu2Ive0+uEhnaHm\nA4uBfyUNEfw1Ii4ex3zOzn9vKZR1Nao4VpLWIQ0hABxWGyIAajuqNpW0VhOz+jFARDxO+hUjUkKG\n9E9wuKTb83DOAPAv+b0NGszr9oi4LM8vSAntTlJyeUjS9ZLOIvUIHxkhpjXz3ydGqDPWsfeHSDvY\na4/x7mSvjT03u0/l8fy39pmIiGsiYsuI2Coirh1l+v2ADwL/Tfq1NA3YGThD0nvz/J4njWsLqA0b\nvi3HOL9ufgH8NiKeIG3gay6NiBdJnYiaRm313LzM+4HfMbi97JDnf1VE1Nbv2YVpdxjls46kOObf\nzLqfn/9uLOn1wDtIY/7F94iIW4FtgG+Q2mqQNg7vAH4nqXsZYu5ITvh1IuJp0k/oVUkN7ZvjnNVj\neX5LCmWNEsZyAJJWH+dy7mJwMrsa+AOph74sjgc+DryK9DO6mCgbtZtFxRcR8RzwWtL+kPNJvad/\nBs6R9IERlrs4/63f0Xl74fn2xTckrQZsSlqvxXo1F0Xawb5LROwCfG+E5Y/kZtJ3OFv5KI9R1L7T\nx8azsIh4MSJOjYg98rzeTFo/ARTHz/+TtH63zfuKNiZtIH7UYLaP53kvqS9jcLtsZsdqS0TEQ6Rh\nRwE75B24IzmPpZ/pCJbuwL0hIm6sm/eCiPhARGwKrA98Ob+1OukXz5TihN/YqeThAZbu4JlIDxae\nb5L/HjxM3Wfy35d2fEXEw6QemkhjpbsXktkhwBci4sH6GTXwdoB8FMl+uaz2D/G6/PeOiNgE2I30\n07+RIUcw5XluGRHfjIjDI2IH4NJcb6R/pDvy39U1+NDDS0nDEQKOkbRNXs404GSWrp8zR5h3s1aQ\ntGLhUUsw38l/VwS+L2lGbQJJr5T0L3Xz2Yj0ee8o1Nsx7xi9VdKwvV5JMyV9QtLLASIdrXQlaZwd\nlm4YiYh7SDu4RRpGCtLQx2Im1jtybBuQjmQJlraXa/Lyd5H0ilx2WGHaawrPm/2FVHRa/rs+8B1J\nq9TekLRNcQdyRDxD2tiJ1MnYLS+z/vDcf5J0sKQV83QPAr8uVJno9dd+7R5T6oQHhTH8QtkawMsK\nr8c6hj/s2DypV7+QpTvCriCNMTYaw78ulz9L2sfwuVz+TpaOPT5E2il6fy4bdny0EGNtvPWvefr6\nnbafLdS7C7gXeLhWr8G6u6JuObUx4keAP5N2hteW8ZkR4lstf9YlwN517+3F0nH0F0jDZQ8X5vsz\nCjvdCuVj2Wnb6LGgMO33Cuv9WdIG955cdl7dcmrfx7sarP/RdtpuVlw+KWEWP+tb6+rvU3jvpe+x\n8H5tDH+k9rtRoezwBuvlcQbvtH0B2DrX25yUIGtt+ibSAQRLgP8arb008T9aOyyztu6fyu3qb/n1\nyXX1d677Dp8B1qqr84XC93gj8Kf8mWoHGqze7tw00Q/38Jca1EuNiMWRxjob1Wt6Pg0rpJ/T7yTt\naJxGGuM9eJjpjiX1rIM0RPKqPI9zgQNIO/amk/7haj2brzQR4wBpp9otpEMd7wPeFxGX5DqfJ413\nPkoaXjmbNLzV6PM1KnuE1KN6gJRINiRtXE4EPjNsYBFPko5ugXRkTvG9y0nDOfNJyXRj0s7Gq4B/\nA/4xIgaG+byNyob7HPWPl+YZEf9KGiO/hJQAX0X6Di8iDa0AkHvv65O+k582sex69wOfIiXI5Uhj\n5SuSEv+ciDh/0AzT/pO/kHq1/cCvmvzMzdSprYN3kn6drpCX9c6IuCkv/zbS+RDnkxLopqQN4VeA\n/ccRx+AJIp6PiANI+6n68jI2Ix0pdC51w1cRcTXp3IXasn4WEY/WzfbHwLdzvW5gS1K7PY+0wXyc\nKUZ5S2fWMSS9jnQG8yJSD/yFUSbpOJJOAY4BvhURpZ0bULfMC0knDH0pIo5vxTJtcnHCt44k6aek\nHZNHR8R4d7K2haQ1Sb3bIB0H38z+lGVZ3idIZ5/uS/pFsWlEPFDmMm1ycsI3m+QkXUk6MetO4LiI\nuKjNIVmHcsI3M6sI77Q1M6sIJ3wzs4pwwjczq4hSE77SzSpuyNdRuSFfxvdYSdtK+n0u++NIZxya\nmdnEaNlO23wK/H2kU/ZPI1217lJJBwAfjYg3tiQQM7OKauWQzt7AnRFxL+msvdp1UtYknVVoZmYl\namUP/3vAdRHxLUmbk079Vn7skjcEZmZWkpYk/HxLvAdIV098KJ92fmVEXCDp7cB7ImKf0gMxM6uw\nViX8g4D3RsT++fVjEbFm4f3FETHkpsySfFaYmdk4RMSQexq0agz/UKB4k+X7Je0BIGkvCtcLr9fu\ny4nWP+bOndv2GCZDTJ0al2NyTFWIazjLT1RGH06+UcHewNGF4qOAr0tajnSZ06MbTWtmZhOn9IQf\n6ZaB69aVXcWy3ePSzMzGyGfajlFPT0+7QxiiE2OCzozLMTXHMTWvU+NqpKOvlikpOjk+M7NOJIlo\n405bMzNrMyd8M7OKcMI3m0Lmz5/f7hCsgznhm00hCxcubHcI1sGc8M1sKP9SmJKc8M1sKP9SmJKc\n8M2sXP610DGc8M2sXP610DGc8M3azEfWWKs44Zu12XiOrOnu7kbSkEdvb++Qsu7u7tFmBtLgR2/v\n0DIp1bVJywnfbBJatGjRxNUdw7xGrNtowzHcxsMbjrZwwjeziTFRGw4rjRO+2STU1dVVSt2O5n0d\ny8wJ32wS6u/vb/ruS/39/SPPbCwbhHZuPHy0zzJzwh8jH1Fh4zWWHa1N7WydKP39EDH4MXfu0LKI\nVHc4E7nh8I7kUjjhj5GvVWLjNZYdreOp33aNNhzDbTxG+9Xh/QGlcMI3G4dO/aU3a9asdodgHcwJ\n32wcOvWX3pw5c9odgnUwJ3wzs4pwwjczqwgnfDOzinDCNzOrCCd8sxYZ6xmvU+YMWesYTvhmLTKW\ns2ObOkPWbIyc8M3MKsIJ38ysIpzwzUbR6Bo4bb/+jXWODj3ruhEnfLNRTOjNRmzq6dCzrhtxwjcz\nqwgnfDOziig14UuaLekGSdfnv4slHZvfO0bSrZJulHRimXF0rEk09mc2bh14Bc9Ovdpp2ZYvc+YR\ncQewHYCkacB9wPmS3ggcCLw6Il6UtE6ZcXSsSTT2ZzZuHXgFz0692mnZWjmkszdwZ0TcC/wbcGJE\nvAgQEQ+3MA4zG00H9spt2bUy4R8CnJ2fzwbeIOlqSVdK2qGFcZjZaDqwVz4eE3pbyUa3XRzu1osd\nenhuqUM6NZKmAwcBHyssd62I2FnSjsC5wMaNpp03b95Lz3t6eujp6Sk1VjObOib0tpIdfNvFvr4+\n+vr6Rq3XkoQPHABcVxi6uRc4DyAirpE0IGlGRDxSP2Ex4ZuZ2VD1neHe3t6G9Vo1pHMocE7h9QXA\nnpCO5AGmN0r2ZmY2cUpP+JJWIe2wPa9QfAawsaQbSeP6h5cdh5lZ1ZWe8CPi6YhYNyKeKJS9EBH/\nJyJeHRE7RMRvyo7DDPC5D1ZpPtPWqqWix1+bgRO+dZjhDqOr4iF0lTaWu335zmBNc8K3jjJhV6bs\n4EPorAn9/RAx+DF37tCyiFTXmuKEb2ZWEU74ZmYV4YRvZlYRTvhmNmV1jXGH7ljrTzaturSCmVnL\n9XuH7iDu4ZuZVYQTvplNDr5G/zJzwjezyWGKXKO/nZzwzcwqwgnfzKwinPDNzCrCCd/MrCKc8M3M\nKsIJ38ysIpzwzcwqwgnfzKwinPDNzCrCCd/MrCKc8M3MKsIJ38ysIpzwbWoay40spvhNL8xqnPBt\naurvh4ihj7lzh5b5JhlWEU74ZqMYy23vpvot8mxyc8I3G0V/fz8RMegxd+7cIWUR4VvqWUdzwjcb\nh1m++5JNQk74ZuMwx3dfsknICd+qxT1zqzAnfKsW98ytwkpN+JJmS7pB0vX572JJxxbe/7CkAUlr\nlxmHWSfz/gBrleXLnHlE3AFsByBpGnAfcH5+PRPYB7i7zBjMOp33B1irtHJIZ2/gzoi4N7/+KvCR\nFi7fzKzSWpnwDwHOAZB0EHBvRNzYwuWbmVVaqUM6NZKmAwcBH5O0MvAJ0nDOS1WGm3bevHkvPe/p\n6aGnp6ecIG3CzZ8/38MVZi3Q19dHX1/fqPVakvCBA4DrIuJhSVsDs4A/SxIwE7hO0k4R8WD9hMWE\nb5PLwoUL2x2CWSXUd4Z7e3sb1mtVwj+UPJwTETcB3bU3JC0Ato+IR1sUi5nZ2HV1waJFzdftQKUn\nfEmrkHbYHj1MlWCEIR0zs44wBa6TVPpO24h4OiLWjYgnhnl/44j4e9lxWHm6u7uRNOTR29s7pKy7\nu3v0GZpZKXymrS2zRc3+zB1jXTObWE741lF87Xmz8jjhW0fxtefNyuOEb2ZWEU74ZmYV4YRvZlYR\nTvhmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhmZhXhhG9mVhFNJXxJ60laNT9fWdIJ\nkk6UtH654ZmZ2URptof/Q2BGft4LvAp4FDi7jKDMimbNmtXuEMymhFETvqQjgE2Anvz8EOBaoB/Y\nSNLhkrYpN0yrMt8I3WxiNHOLwz7gKeB/SL38RcDPSbclfF9+f3E54ZmZ2UQZNeFHxN2STgV+BQwA\nR0XEPZI2BB6JiHvKDtLMzJZdUzcxj4hvSzoLGIiIp3PxI8ChpUVmZmYTqqmEDxART9a9fmriwzEz\ns7L4OHwzs4pwwjczqwgnfBtk/vz57Q7BzErihG+DLFy4sN0hmFlJnPDNzCrCCd/MrCKc8M3MKsIJ\n38ysIpzwzcwqwgnfzKwimr60wnhImg38CAjS1TU3Bj4FzAQOBJ4D7gTeFRGPlxmLmVnVldrDj4g7\nImK7iNgeeC3pMsvnA5cCW0XEa4C/AMeXGYcN1d3djaQhj97e3obl3d3d7Q7ZzJZRK4d09gbujIh7\nI+LXETGQy68m9fithRYtWlRqfTPrPK1M+IcA5zQoPxL4ZQvjMDOrpJYkfEnTgYOAH9eVnwC8EBG+\nN66ZWclK3WlbcABwXUQ8VCuQNAd4E7DnSBPOmzfvpec9PT309PSUEqCNX1dXV9NDPl1dXSVHY1Y9\nfX199PX1jVqvVQn/UArDOZL2Bz4CvCEinhtpwmLCt87U39/fsHzevHn+/sxaoL4z3Nvb27Be6UM6\nklYh7bA9r1B8KrAacJmk6yV9q+w4zMyqrvQefr4H7rp1ZZuWvVwzMxvMZ9qamVWEE76ZWUU44ZuZ\nVYQTvplZRTjhm5lVhBO+mVlFOOGbmVWEE76ZWUU44ZuZVYQTvplZRTjhm5lVhBO+mVlFOOGbmVWE\nE76ZWUU44ZuZVYQTvplZRTjhW2lmzZrV7hDMrMAJ30ozZ86cdodgZgVO+FPA/Pnz2x2CmU0CTvhT\nwMKFC9sdgplNAk74ZmYV4YRvZlYRTvhmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhm\nZhXhhG9mVhFO+GZmFeGEb2ZWEaUmfEmzJd0g6fr8d7GkYyWtJelSSbdL+pWkNcqMw8zMSk74EXFH\nRGwXEdsDrwWeAs4HPg78OiI2A64Aji8zDjMza+2Qzt7AnRFxL/CPwJm5/EzgLS2Mw8ysklqZ8A8B\nzs7PuyJiEUBE9APrtTAOM7NKWr4VC5E0HTgI+Fguiroq9a9fMm/evJee9/T00NPTM8HRmZlNbn19\nffT19Y1aryUJHzgAuC4iHs6vF0nqiohFkrqBB4ebsJjwzcxsqPrOcG9vb8N6rRrSORQ4p/D6Z8Cc\n/PwI4MIWxWFmVlmlJ3xJq5B22J5XKP4isI+k24G9gBPLjsMG6+rqKrW+mXWe0hN+RDwdEetGxBOF\nsr9HxN4RsVlE7BsRj5Udhw3W399PRAx5zJ07t2F5f39/u0M2s2XkM23NzCrCCb8VurtBGvro7R1a\n1t09yqy6kTTo0dvbO6RMEt2jzMvMqsUJvxUWLZqwuovGMK+x1DWzqc8J38ysIpzwzcwqwgnfzKwi\nnPDNzCrCCd/MrCKc8M3MKsIJ38ysIpzwzcwqwgl/BD6r1cymEif8EfisVjObSpzwW2Eslxb2ZYjN\nrCRO+K3Q3w8RQx9z5w4t82WIzawkTvhmZhXhhG9mVhFO+DbIrFmz2h2CmZXECX+SGcu9ZcdzH9o5\nc+aMeRozmxyc8CeZRvei9X1ozawZTvhmZhXhhG9mVhFO+GZmFeGEPwX4yBoza4YT/hTgI2vMrBlO\n+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhmZhXhhG9mVhGlJ3xJa0j6saRbJd0s6XWStpX0e0k3SPqj\npB3KjmMq6+vra3cIDXViXI6pOY6peZ0aVyOt6OGfAlwcEVsA2wK3AV8C5kbEdsBc4MstiGPK6tQG\n14lxOabmOKbmdWpcjSxf5swlrQ7sHhFzACLiRWCxpAFgjVxtTeD+MuMwM7OSEz7wSuBhSWeQevfX\nAh8APgT8StJJgIBdSo7DzKzyFBHlzVx6LXA18PqIuFbSV4EnSL37KyPiAklvB94TEfs0mL684MzM\nprCIUH1Z2Qm/C/h9RGycX+8GfBzYNSLWKtRbHBFrDDMbMzObAKXutI2IRcC9kmbnor2Am4EHJO0B\nIGkv4I4y4zAzs5J7+ACStgVOA6YDdwHvArYmHb2zHPAs8N6IuKHUQMzMKq70hG9mZp3BZ9qamVVE\n2xJ+YVy/Y0hasd0xTDaShhwJ0C6SVml3DPXczqeGqdLO25LwJR0AXJEvsdARvzIk7QecIukLkso+\nP6EpkvaR9D5Jx+TXbW90knokHVWLKSKiQ+I6APhPSTu1O5Yat/PmuJ2PKa5lauctb4SStgROAo6I\niD+3evmNSNoT+DpwKbA/cEJ7I3rpENazSTu1D5F0KrBrO/9JJb0J+BZpB/xxkr4N6Z+hXTEVbAZs\nCeyb111buZ03x+18zJapnbej1xHApRFxuaQNgU9KOl7S7q3+qSlpmqTlgEOAb0bEecCxwMskHS5p\ng/x+O+wEfCMivgfsDSwG3p7LW672XQHHRMS3gNcAr5a0WSf0fIC7gYdJbfofJG2eL9w3vU3xDNA5\n7Vz5F4bb+SimejtvWcKXtFJ++iKwh6R9gO8BzwEbAW8jHaffSitGxBLgOmBPSYcBF5IuOfHPwPFA\nu8ZgbwJeL2l2RDwL/D/g6RxXOzwFfC4nsBVyLM8Aa3dIz+fXwG+AHwKPkS7Kdy6wdpviWUKHtPNI\nBujMdn4jHdDOC8n8OeCzndLOa3EV4lumdt6ShJ/HDS+WtEFE/AU4g7QVvy0ivgi8n5RQWnZNnRzT\nRZJmAH3AFaQexo8i4oPAQcC6Oc5WxbSFpK0krU/6Ym8FdpO0fkQ8B3wG2EnSnBbGtKWkLQAi4qL8\n9/mIeIF0XsVArve6Vo5T53W1paTuXDSdNExxF6kXdADwJLBeG2J6eW7nZ9L+dl5rU+sA55Paervb\n+SskLS9ptRzP7cDu7WznwMz8K+fRiLgYOqOd57iWB2o7aldkGdp5K66Hvx/pJKsBlvYi/kDq6e8l\nadd8Fc2FwDqSppf906kQE8BWEXFHRHwd+AawJG+Yngd+C6zUip+7OaYLSBeX+wmwEmlLvivwJkmb\n5x7Qz8mNr0UxnQ8cB/xc0rq5vLY+1gRWlnQo8P9pUXItrKsPAudLWi8i/g78APgYqZd4Aqn3+FZJ\nK7c4pp9IWoPUfgZobzuvxXQh8EJEfI00Pt2udr4/cB7wOeA/gC5S29kNeHOb2nktps8DZ0jaLJev\nkKu0q53X4vos8N28bh4m7fMYXzuPiNIepC3R9cDuwEeAXxTe2wb4KPA70s6tBcCWZcbTREybAt8B\nTswr87YWxbQZ6ZITb8yvvw6slZ/vBJxMSv6nAIuALdoQ01eBdYBVC3VOAi7LsW1VdkzDxPU1Ug+1\n1sO/GziwUHe9NsR0Cukn9nJ5nZ3QhnbeKKZ1gdWAVYHT29DOZ5N+te6e4+jN62P93M5PakM7r49p\nLnAvMLvN7Xy4uF4J7EfqOIy5nZcd9MnAHvn59LzC3l1XZ/P8AWa1aEU2iunIwvsHAO8j9YI2b1FM\nrwC+nZ/PIu2UmU8ac904l78BeDfwqjbGdDrwJ2DrXP7RnGBbsp5GiOsM0qW3ZwLT8nvT2hzTfFLH\nYtNcvlWL2/lw39/1pH0J25J23Laync8ETiu83hq4hjSks37+f9ytxe28UUx/AP4KbJLL/r0N7bxR\nXH8kbZxnAGvkco1pviUFexAwp/B6hfz3vaQdIpCug9/Kf8pRY6qrP6YVOc6YDgSOAVbPjey7wN9y\nIp0GfJg0Vld6L3UMMX2I1LtYG9geeEWHxPXvxXXVQd/fAqCrg76/D+dktmaL19OxpKGRu0m/eFYn\n/cI4Kr/+RIvzwWgxHQ98MtfdscXtfKS4PpnX1XLjWV9lBLwvqRe4b4P3tgUeAPZv1Rc73pjK/kco\nxLR/fj2D1BM7GVi5UO9MYGaL19NoMX0fWL8N399occ0HXt5hMXXq99eumDYBriJtiC4i9er3AU5q\nY3saLqaTWxVTq9bVRAe8C2nsbaf8eg3Sz8fVgOm57CjgLGBGi1biZIhpBukn+NrAT0nHAAMcBvyZ\nFvQQOzGmTo3LMY07pnVIByN0MXgD9G7SsNyKlN/RGmtMK5UdUyvX1USfzfYI8AKwfj7c8SekY1if\nBC6RdBbp2vc7A89P8LIne0zPkX7+3wHMk7Qzacf2IZHuK1DFmDo1Lsc0vph+nGN6gnSY9rnAW4BP\nAW+OdEhmp8X0bAtiGk9c41tXJWyptiWNpT5A6jlPA44knShQO/Jk7bK3mJM0pqOBr5BOnd4Q6K56\nTJ0al2NappiOJB1aOBN4J3nHdpVjalVcZQW+JfC+urJLgNe2eiVOwpguA7ZxTJ0fl2Napph+RQuP\nepkMMbUirlIuUBQRtwC31F5LehvpGOD7yljeFItpbdJYnmMq6MS4HNMyxbQO6dIAjqmg9LhK3lqJ\n9JPkFlq53PD4AAACNUlEQVR0woJjmroxdWpcjskxTZa4Sr3FYT51fA+gPyJuK21BY+CYmtOJMUFn\nxuWYmuOYmldWXL6nrZlZRXTEXXjMzKx8TvhmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZgWSDpT00WWY\n/gNaev9ms47iwzLNJpCkBaTLdfy93bGY1XMP3ypD0kaSbpV0hqTbJf1A0l6Sfptf7yjpCEmn5vpn\nSDpF0u8k/VXSwbl8D0k/L8z3VEmHSzoG2AC4UtLl+b19JV0l6VpJP5K0Si4/UdJNkv4k6UutXxtW\nRU74VjWbAF+OiM1It9c8NCJ2I93f+BNA5EdNd0TsSroT0RcL5UN+GkfEqaQrHfZExF75MrcnAHtF\nxA6kW1YeJ2lt4C0RsXVEvIZ0k2qz0pVy8TSzDrYg0gWqIN3k+/L8/EbS3aHqXQAQEbdKWq/JZSj/\n3Zl09cPf5VPlp5PuYLQYeEbSaaQ7Gf1irB/CbDyc8K1qijeOGCi8HqDx/0Oxfi2Rv8jgX8fD7aQV\ncGlEHDbkDWknYC/gHcD783OzUnlIx6pGo1cZddq7gS0lTZe0JoOT9eOkm04DXA3sKmkTAEmrSNpU\n0qqkm4hfAhxHuguVWencw7eqiWGeN/06Iu7Lt5y7iXQLwesLdb5LunXm/Xkc/13AOZJWzNN/knTb\nugsLh29+aNyfxmwMfFimmVlFeEjHzKwinPDNzCrCCd/MrCKc8M3MKsIJ38ysIpzwzcwqwgnfzKwi\nnPDNzCrifwEbyTz+QshhSAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3620983bd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.finance import candlestick_ohlc\n",
    "\n",
    "# Split the dimensions to get the minute co-ordinates\n",
    "l1, l2 = zip(*mb1.index)\n",
    "# unify into a nparray of tuples for use in candlestick graph\n",
    "q2 = zip(l2, mb1.open, mb1.high, mb1.low, mb1.close)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "fig.subplots_adjust(bottom=0.2)\n",
    "fig.suptitle('Minute bars (OHLC): Symbol \\'CVS\\' ', fontsize=14, fontweight='bold')\n",
    "ax.set_xlabel('minutes')\n",
    "ax.set_ylabel('$')\n",
    "\n",
    "candlestick_ohlc(ax, q2[15:30], width=0.6)\n",
    "\n",
    "ax.autoscale_view()\n",
    "plt.setp(plt.gca().get_xticklabels(), rotation=45, horizontalalignment='right')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Microsecond tick data example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u\"<seq:int64 NULL DEFAULT null COMPRESSION 'zlib',sub:uint8 NULL DEFAULT null COMPRESSION 'zlib',exch:uint8 NULL DEFAULT null COMPRESSION 'zlib',nsec:int64 NULL DEFAULT null COMPRESSION 'zlib',line:string NULL DEFAULT null COMPRESSION 'zlib',flag:int64 NULL DEFAULT null COMPRESSION 'zlib',bid:float NULL DEFAULT null COMPRESSION 'zlib',ask:float NULL DEFAULT null COMPRESSION 'zlib',bidsz:int32 NULL DEFAULT null COMPRESSION 'zlib',asksz:int32 NULL DEFAULT null COMPRESSION 'zlib',condition:string NULL DEFAULT null COMPRESSION 'zlib'> [secid=0:*,100000,0,dt=19800101:21991231,1,0,tm=0:8640000000,1000000,0,synth_seq=0:9999,100000,0]\""
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "option_quote = sdb.wrap_array('option_quote')\n",
    "option_quote.schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use SciDB-Py SciDB Array Objects to store lazily evaluated AFL query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'between(option_quote,null,20151204,3060000000,0,null,20151204,5400000000,null)'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " starttm = np.int64(8.5*3600*100000)\n",
    " endtm   = np.int64(15*3600*100000)\n",
    " startdt = enddt = datetime.date(2015, 12, 4)\n",
    "    \n",
    "option_quote_between = afl.between(option_quote, 'null' ,int(startdt.strftime('%Y%m%d')),starttm ,0, 'null', int(enddt.strftime('%Y%m%d')), endtm, 'null')\n",
    "option_quote_between.query"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return the count of the \"option_quote\" array to a NumPy ndarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11245389864\n"
     ]
    }
   ],
   "source": [
    "print(option_quote_between.nonempty())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple aggregation example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " option_quote_between.aggregate('bidsz_max':'max(bidsz)').toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross_join example\n",
    "\n",
    "The below example demonstrates the use of cross_join and other afl operators to join a time of interest with an array that contains the national best bid and offer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    minsecid = 100000\n",
    "    maxsecid = 1500000\n",
    "    \n",
    "    starttm = np.int64(8.5*3600*100000)\n",
    "    endtm   = np.int64(15*3600*100000)\n",
    "    \n",
    "    startdt = enddt = datetime.date(2015, 12, 4)\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    fill = sdb.wrap_array('fill')\n",
    "    redim = afl.redimension(fill, '<pre1m:int64, value1:int64, value2:int64> [secid=0:*,100000,0, interval_id=0:*,10000,0]').eval()\n",
    "    \n",
    "    option_nbbo  = sdb.wrap_array('nbbo_option_data') \n",
    "    between_nbbo = afl.between(option_nbbo,minsecid,int(startdt.strftime('%Y%m%d')),starttm,0,maxsecid,int(enddt.strftime('%Y%m%d')),endtm,10000)\n",
    "    \n",
    "    pre1mprj     = afl.project(redim,'pre1m')\n",
    "    joinarr      = afl.cross_join(between_nbbo, pre1mprj,'secid','secid').filter('tm < pre1m').apply('tm_attribute','tm')\n",
    "    min_times    = afl.redimension(joinarr, '<lower_bound:int64 null> [secid=0:*,10000,0, interval_id=0:*,10000,0],max(tm_attribute) as lower_bound').eval()\n",
    "     \n",
    "    dataframe = min_times.todataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### As Of Join example\n",
    "\n",
    "The asof operator joins trade data with quote data. At time points where quote data is not available, the last known value is looked up and filled in. This is sometimes also called a 'last value carry forward' join or 'piecewise constant interpolation join'.\n",
    "The syntax is:\n",
    "asof(A, B [, aggrDim [, missingCode]])\n",
    "where the inputs are:\n",
    "A & B: two arrays that have the same dimensionality;\n",
    "aggrDim: the dimension in A along which to find last_value. Default is the last dimension.\n",
    "missingCode: the missing code to use, when an A record cannot be found. Default is 0.\n",
    "Here is an example that joins trades and quotes for 'BAM'. We use the fact that we know the symbol index for BAM is 615 from the last example. In this example, we only choose the values with dummy dimension of 0 (see comments below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol_index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td> BAM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             symbol\n",
       "symbol_index       \n",
       "615             BAM"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "symbol = 'BAM'\n",
    "tkr[tkr == symbol].todataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next run some counts for the data for this symbol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions of trades and quotes arrays:\n",
      "[u'dummy', u'symbol_index', u'ms']\n",
      "\n",
      "# of quotes for symbol = BAM\n",
      "33447\n",
      "\n",
      "# of trades for symbol = BAM\n",
      "4362\n"
     ]
    }
   ],
   "source": [
    "print('Dimensions of trades and quotes arrays:')\n",
    "print(trades.dim_names)\n",
    "print('\\n# of quotes for symbol = %s' % symbol)\n",
    "q1 = quotes.between('null', 615, 'null', 0, 615, 'null')\n",
    "print(q1.nonempty())\n",
    "print('\\n# of trades for symbol = %s' % symbol)\n",
    "t1 = trades.between('null', 615, 'null', 0, 615, 'null')\n",
    "print(t1.nonempty())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now note that the dimensions of the quotes and trades arrays did not have identical chunking in the first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from re import search\n",
    "print(search('\\[.*\\]', q1.schema).group(0))\n",
    "print(search('\\[.*\\]', t1.schema).group(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[dummy=0:999,1000,0,symbol_index=0:*,10,0,ms=0:86399999,86400000,0]\n",
    "\n",
    "[dummy=0:999,1000,0,symbol_index=0:*,100,0,ms=0:86399999,86400000,0]\n",
    "\n",
    "We resolve this by redimension-ing the (smaller) trades subarray for selected symbol to the same dimension schema as the quotes subarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "t2 = t1.redimension(\"<price:double NULL DEFAULT null,volume:int64 NULL DEFAULT null,\\\n",
    "                    sequence_number:int64 NULL DEFAULT null,condition:string NULL DEFAULT null,\\\n",
    "                    exchange:string NULL DEFAULT null> \\\n",
    "                    [dummy=0:999,1000,0,symbol_index=0:*,10,0,ms=0:86399999,86400000,0]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we run the asof join and make some simple checks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The head() of the trades array\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>volume</th>\n",
       "      <th>sequence_number</th>\n",
       "      <th>condition</th>\n",
       "      <th>exchange</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <th>symbol_index</th>\n",
       "      <th>ms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">615</th>\n",
       "      <th>34185171</th>\n",
       "      <td> 38.9</td>\n",
       "      <td>   91</td>\n",
       "      <td> 3309</td>\n",
       "      <td>   TI</td>\n",
       "      <td> P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185172</th>\n",
       "      <td> 39.0</td>\n",
       "      <td>  100</td>\n",
       "      <td> 3310</td>\n",
       "      <td>  FT </td>\n",
       "      <td> T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185173</th>\n",
       "      <td> 38.8</td>\n",
       "      <td>   91</td>\n",
       "      <td> 3312</td>\n",
       "      <td>   TI</td>\n",
       "      <td> T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185950</th>\n",
       "      <td> 38.8</td>\n",
       "      <td>    9</td>\n",
       "      <td> 3313</td>\n",
       "      <td>   TI</td>\n",
       "      <td> T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34200381</th>\n",
       "      <td> 39.8</td>\n",
       "      <td> 9761</td>\n",
       "      <td> 3695</td>\n",
       "      <td> O   </td>\n",
       "      <td> N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             price  volume  sequence_number condition exchange\n",
       "dummy symbol_index ms                                                         \n",
       "0     615          34185171   38.9      91             3309        TI        P\n",
       "                   34185172   39.0     100             3310       FT         T\n",
       "                   34185173   38.8      91             3312        TI        T\n",
       "                   34185950   38.8       9             3313        TI        T\n",
       "                   34200381   39.8    9761             3695      O           N"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asof1 = q1.asof(t2)\n",
    "print('The head() of the trades array')\n",
    "t1.todataframe().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The head() of the quotes array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The head() of the quotes array\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>ask_price</th>\n",
       "      <th>bid_price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <th>symbol_index</th>\n",
       "      <th>ms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"10\" valign=\"top\">615</th>\n",
       "      <th>25269846</th>\n",
       "      <td>  0.0</td>\n",
       "      <td> 36.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25269850</th>\n",
       "      <td>  0.0</td>\n",
       "      <td>  0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28800370</th>\n",
       "      <td> 51.5</td>\n",
       "      <td> 42.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34050049</th>\n",
       "      <td>  0.0</td>\n",
       "      <td> 37.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34050060</th>\n",
       "      <td> 39.4</td>\n",
       "      <td>  0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34050068</th>\n",
       "      <td> 46.0</td>\n",
       "      <td> 38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34050074</th>\n",
       "      <td> 45.2</td>\n",
       "      <td> 40.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34080013</th>\n",
       "      <td> 46.6</td>\n",
       "      <td> 28.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34183007</th>\n",
       "      <td> 41.6</td>\n",
       "      <td> 37.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185948</th>\n",
       "      <td> 42.0</td>\n",
       "      <td> 37.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             ask_price  bid_price\n",
       "dummy symbol_index ms                            \n",
       "0     615          25269846        0.0       36.6\n",
       "                   25269850        0.0        0.0\n",
       "                   28800370       51.5       42.6\n",
       "                   34050049        0.0       37.1\n",
       "                   34050060       39.4        0.0\n",
       "                   34050068       46.0       38.0\n",
       "                   34050074       45.2       40.6\n",
       "                   34080013       46.6       28.1\n",
       "                   34183007       41.6       37.8\n",
       "                   34185948       42.0       37.8"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The head() of the quotes array')\n",
    "qhead = q1.todataframe().head(10)\n",
    "qhead[['ask_price','bid_price']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# of elements in the joined result:\n",
      "4362\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>ask_price</th>\n",
       "      <th>ask_size</th>\n",
       "      <th>price</th>\n",
       "      <th>volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <th>symbol_index</th>\n",
       "      <th>ms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">615</th>\n",
       "      <th>34185171</th>\n",
       "      <td> 41.6</td>\n",
       "      <td> 6</td>\n",
       "      <td> 38.9</td>\n",
       "      <td>   91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185172</th>\n",
       "      <td> 41.6</td>\n",
       "      <td> 6</td>\n",
       "      <td> 39.0</td>\n",
       "      <td>  100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185173</th>\n",
       "      <td> 41.6</td>\n",
       "      <td> 6</td>\n",
       "      <td> 38.8</td>\n",
       "      <td>   91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34185950</th>\n",
       "      <td> 42.0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38.8</td>\n",
       "      <td>    9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34200381</th>\n",
       "      <td> 40.1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 39.8</td>\n",
       "      <td> 9761</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             ask_price  ask_size  price  volume\n",
       "dummy symbol_index ms                                          \n",
       "0     615          34185171       41.6         6   38.9      91\n",
       "                   34185172       41.6         6   39.0     100\n",
       "                   34185173       41.6         6   38.8      91\n",
       "                   34185950       42.0         1   38.8       9\n",
       "                   34200381       40.1         1   39.8    9761"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('\\n# of elements in the joined result:')\n",
    "print(asof1.nonempty())\n",
    "foo = asof1.todataframe().head()\n",
    "foo[['ask_price','ask_size','price','volume']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$$\\,$$\n",
    "\n",
    "$$\\,$$\n",
    "\n",
    "\n",
    "References again\n",
    "----\n",
    "* https://github.com/Paradigm4/TAQ/blob/master/TAQ_ipynb.ipynb\n",
    "* https://www.github.com/paradigm4/SciDB-py\n",
    "* https://www.github.com/paradigm4/\n",
    "* http://www.paradigm4.com\n",
    "\n",
    "Thanks!\n",
    "===="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}