loosing_your_loops_using_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World\n"
     ]
    }
   ],
   "source": [
    "#Python is fast and it is not just fast, it is increadibly fast.\n",
    "# for example: \n",
    "print \"Hello World\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# so hello world just print \"Hello World\". :D\n",
    "#other important thing in python is wide range of available libraries, \n",
    "\n",
    "# So what's wrong about Python? There should be some otherwise we can't say it is perfect or anything like that, \n",
    "# python is slow while our code do refrence-counting and typechecking in repeated-interval of time, For exampple:\n",
    "\n",
    "def silly_func(N):\n",
    "    d = 0.0\n",
    "    for i in xrange(N):\n",
    "        d+=(i%3-1)*i\n",
    "    return d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 1.63 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit silly_func(10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python is hogh level intepreted Dynamic Language, It does lot of type checking and refrence Counting where major amount of time is consumed. \n",
    "\n",
    "So here Numpy comes in the scene where you can use power of C in terms of SPEED and reliability of Python interms of fast development, So now we are talking about four strategies we can use to improve the speed of Python code using Numpy. \n",
    "\n",
    "1. Using Numpy's Ufuncs\n",
    "2. Use Numpy's aggregations\n",
    "3. Using Numpy's broadcasting\n",
    "4. Using Numppy's sliccing,masking and fancyIndexing\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ufuncs does element wise operations on arrays/lists\n",
    "\n",
    "a = xrange(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b = [i+5 for i in a]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 67.78 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "10000000 loops, best of 3: 31.7 ns per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 9.58 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit sum(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ok now lets's fire-up Numpy. \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b = np.array(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 17.50 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "100000 loops, best of 3: 2.63 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit b+5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 19.19 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "100000 loops, best of 3: 7.55 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit np.sum(b+5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# so here in both the cases we can observe that our code is much faster, cleaner and useable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can also do many other operations using Ufunc in Numpy as follows:\n",
    "\n",
    "1. Airthmetic Operations: + , - , * ,// ,% , **\n",
    "2. Bitwise operations: & , | , ^ , >> ,<< \n",
    "3. Comparison Operator: >, < , => , <=, != , ==\n",
    "4. Trignometry Operations: np.sin,np.cos,np.tan , ....\n",
    "5. Exponential Family : np.exp, np.log, np.log10, ....\n",
    "6. Special Functions: scipy.special.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now Let's talk about using Numpy's aggregation, Which summerise the values of arrays. \n",
    "# for example, min, max, mean, mod etc\n",
    "\n",
    "from random import random\n",
    "\n",
    "b = [random() for i in xrange(10000)]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 329 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit min(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np_array = np.array(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 10.82 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "100000 loops, best of 3: 7.58 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit np_array.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# there are also nanmin() nanmax() nansum() functions which are same but also able to do operations if Null,None, NaN \n",
    "#values exist in the python array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now Let's Talk about Broadcasting, Broadcasting is set of rules by which ufuncs operate on arrays of different sizes \n",
    "or dimensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rules of BroadCasting:\n",
    "1. If array shape is defferent, leftpad the smaller shape with 1s\n",
    "2. if dimesnsion does not match broacast the dimension with size = 1\n",
    "3. if neighter non matching dimension is one, raise an error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l = [2,3,4,5,6]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 6]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l[3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's talk about 4th strategy, Slicing,masking and fancy-indexing in numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = xrange(200)\n",
    "b = np.array(a)\n",
    "b[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,\n",
       "       22, 23, 24])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[5:25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,\n",
       "        13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,\n",
       "        26,  27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,\n",
       "        39,  40,  41,  42,  43,  44,  45,  46,  47,  48,  49,  50,  51,\n",
       "        52,  53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,  64,\n",
       "        65,  66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,  77,\n",
       "        78,  79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,  90,\n",
       "        91,  92,  93,  94,  95,  96,  97,  98,  99, 100, 101, 102, 103,\n",
       "       104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,\n",
       "       117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,\n",
       "       130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,\n",
       "       143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,\n",
       "       156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,\n",
       "       169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,\n",
       "       182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,\n",
       "       195, 196, 197, 198, 199])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([], dtype=int64)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[200:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# now talk about MASKING which is quite more important to abolish the for loop condition:\n",
    "l = np.array([2,4,5,6,7,8,9,9])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mask = np.array([True,True,True])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 4, 5])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l[mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#so we have passed the condition in which looks for mask as condition: we can use masking in various cases as\n",
    "\n",
    "# present python code\n",
    "\n",
    "a = xrange(100000)\n",
    "new_list = []\n",
    "for i in a:\n",
    "    if i%2==0:\n",
    "        new_list.append(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 67.83 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "10000000 loops, best of 3: 31.6 ns per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit new_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# now let's do similer thing using Numpy's masking approach:\n",
    "a = np.array(xrange(100000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([    0,     1,     2, ..., 99997, 99998, 99999])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mask = a%2==0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 794 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit a[mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([23, 45, 89])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's talk about Fancy-indexing\n",
    "fancy = [23,45,89] \n",
    "a[fancy] # i want 23rd,45th and 89th element on array a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# all the above operations and fnctions are used to loose loops in Python and as Python Programmer one should think if one\n",
    "# can loose loops as much as one can loose.  :)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}