01-TensorFlow-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introducción a TensorFlow y Keras\n",
    "===\n",
    "\n",
    "**Juan David Velásquez Henao**  \n",
    "jdvelasq@unal.edu.co   \n",
    "Universidad Nacional de Colombia, Sede Medellín  \n",
    "Facultad de Minas  \n",
    "Medellín, Colombia\n",
    "\n",
    "---\n",
    "\n",
    "Haga click [aquí](https://github.com/jdvelasq/neural-networks/tree/master/) para acceder al repositorio online.\n",
    "\n",
    "Haga click [aquí](http://nbviewer.jupyter.org/github/jdvelasq/neural-networks/tree/master/) para explorar el repositorio usando `nbviewer`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sitios web:\n",
    "\n",
    "* https://keras.io\n",
    "\n",
    "\n",
    "* https://www.tensorflow.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operaciones matemáticas básicas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En el siguiente ejemplo se evaluará la expresión 5 * 2 - (3 + 2). Esta expresión puede ser representada a través del siguiente árbol sintáctico:\n",
    "\n",
    "     [-]\n",
    "      +--- [*]\n",
    "      |     +---- [5]\n",
    "      |     +---- [2]\n",
    "      |\n",
    "      +--- [+]\n",
    "            +---- [3]\n",
    "            +---- [2]\n",
    "\n",
    "Note que en esta expresión el nodo con la constante 2 es común a ambas ramas y podría representarse de forma optimizada como:\n",
    "\n",
    "     [-]\n",
    "      +--- [*]\n",
    "      |     +---- [5]\n",
    "      |     +-----------+\n",
    "      |                 |\n",
    "      +--- [+]         [2] \n",
    "            +---- [3]   |\n",
    "            +-----------+ \n",
    "\n",
    "Para realizar la evaluación usando `TensorFlow`, se debe construir explícitamente el grafo, para lo cual se enumeran los nodos de la siguiente forma:\n",
    "\n",
    "     [-] (node6)\n",
    "      +--- [*] (node4)\n",
    "      |     +---- [5] (node1)\n",
    "      |     +---------------------+\n",
    "      |                           |\n",
    "      +--- [+] (node5)           [2] (node2)\n",
    "            +---- [3] (node3)     |\n",
    "            +---------------------+ \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "node1 = tf.constant(5.0)  \n",
    "node2 = tf.constant(2.0)  \n",
    "node3 = tf.constant(3.0)  \n",
    "node4 = tf.multiply(node1, node2)\n",
    "node5 = tf.add(node3, node2)\n",
    "node6 = tf.subtract(node4, node5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.ops.Graph at 0x7fba559d22e8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## el grafo puede accederse asi:\n",
    "tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para poder ejecutar los cálculos es necesario abrir una sesión."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outs = 5.0\n"
     ]
    }
   ],
   "source": [
    "## abre la sesión y luego la cierra\n",
    "with tf.Session() as sess:\n",
    "    outs = sess.run(node6)          ## realiza la evaluación\n",
    "    print(\"outs = {}\".format(outs)) ## imprime el resultado de la evaluación"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node6 = 5.0\n",
      "node4 = 10.0\n"
     ]
    }
   ],
   "source": [
    "## Para computar el resultado en un nodo se puede usar eval()\n",
    "with tf.Session() as sess:\n",
    "    print('node6 = {}'.format(node6.eval()))\n",
    "    print('node4 = {}'.format(node4.eval()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outs = [5.0, 10.0, 5.0]\n"
     ]
    }
   ],
   "source": [
    "## run() admite una lista de nodos a evaluar\n",
    "with tf.Session() as sess:\n",
    "    outs = sess.run([node6, node4, node5])\n",
    "    print(\"outs = {}\".format(outs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outs = 5.0\n"
     ]
    }
   ],
   "source": [
    "## la sesión puede ser almacenada en una variable\n",
    "## y usada en cálculos posteriores\n",
    "sess = tf.Session()             ## abre la sesion (en IPython use tf.InteractiveSession())\n",
    "outs = sess.run(node6)          ## evalua el grafo y almacena el resultado\n",
    "sess.close()                    ## cierra la sesión\n",
    "print(\"outs = {}\".format(outs)) ## imprime el resultado de la evaluación"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación se resumen las operaciones mas comunes (la lista completa de operadores puede ser consultada en https://www.tensorflow.org/api_guides/python/math_ops)\n",
    "\n",
    "    Operacion               equivalente\n",
    "    --------------------------------------\n",
    "    tf.add(a, b)            a + b\n",
    "    tf.multiply(a, b)       a * b\n",
    "    tf.subtract(a, b)       a - b\n",
    "    tf.divide(a, b)         a / b\n",
    "    tf.pow(a, b)            a ** b\n",
    "    tf.mod(a, b)            a % b\n",
    "    \n",
    "    tf.abs(a)               abs(a)\n",
    "    tf.square(a)            a ** 2\n",
    "    tf.sqrt(a)              sqrt(a)\n",
    "    tf.exp(a)               exp(a)\n",
    "    tf.negative(a)          -a\n",
    "    tf.sigmoid(a)           1 / (1 + exp(-a))\n",
    "    tf.sign                 sign(a)\n",
    "    tf.maximum\n",
    "    tf.minimum\n",
    "    \n",
    "    tf.logical_and(a, b)    a & b\n",
    "    tf.logical_or(a, b)     a | b   \n",
    "    tf.greater(a, b)        a > b\n",
    "    tf.greater_equal(a, b)  a >= b\n",
    "    tf.less_equal(a, b)     a <= b\n",
    "    tf.less(a, b)           a < b\n",
    "    tf.logical_not(a)       ~a \n",
    "    tf.equal(a, b)          a == b\n",
    "    tf.not_equal            a != b\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Compute $5*2^2 + 3 * 5 - 18$ usando TensorFlow.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outs = 17.0\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "node1 = tf.constant(5.0)\n",
    "node2 = tf.constant(2.0)\n",
    "node3 = tf.constant(3.0)\n",
    "node4 = tf.constant(18.0)\n",
    "node5 = tf.square(node2)\n",
    "node6 = tf.multiply(node1, node5)\n",
    "node7 = tf.multiply(node3, node1)\n",
    "node8 = tf.add(node6, node7)\n",
    "node9 = tf.subtract(node8, node4)\n",
    "outs = sess.run(node9)\n",
    "sess.close()\n",
    "print(\"outs = {}\".format(outs))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.client.session.InteractiveSession at 0xb1e91eeb8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Esta opción abre la sesión interactiva y la \n",
    "## mantiene abierta\n",
    "tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.],\n",
       "       [1., 1., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.ones((3,3))\n",
    "b = tf.convert_to_tensor(a)\n",
    "b.eval()\n",
    "#sess.run(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0.],\n",
       "       [0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.zeros((2,2)).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1.],\n",
       "       [1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.ones((2,2)).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.26979545,  0.5981732 ],\n",
       "       [ 0.13635306,  0.75549424],\n",
       "       [ 1.4228015 , -0.66110176]], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random_normal((3,2),     ## shape\n",
    "                 0,         ## mean\n",
    "                 1).eval()  ## sd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.9580271e-04,  6.2689185e-01],\n",
       "       [-7.0928586e-01,  2.4103402e-01],\n",
       "       [ 2.4478497e-01, -1.0959780e-01]], dtype=float32)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.truncated_normal((3,2),  ## shape\n",
    "                    0,      ## mean\n",
    "                    1).eval()     ## sd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.2325803 , 0.6902174 ],\n",
       "       [0.75136256, 0.12303627],\n",
       "       [0.10225999, 0.6465963 ]], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random_uniform((3,2),     ## shape\n",
    "                  0,         ## minval\n",
    "                  1).eval()  ## maxval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4, 4],\n",
       "       [4, 4],\n",
       "       [4, 4]], dtype=int32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.fill((3,2),      ## shape\n",
    "        4).eval()   ## value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.1       , 0.2       , 0.3       , 0.4       ,\n",
       "       0.5       , 0.6       , 0.7       , 0.8       , 0.90000004,\n",
       "       1.        ], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.linspace(0.0, 1.0, 11).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6]], dtype=int32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant([[1,2,3], \n",
    "             [4,5,6]]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 5, 2, 1, 3, 4], dtype=int32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random_shuffle(tf.constant([1,2,3,4,5,6])).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = tf.constant([[1,2,3], \n",
    "                 [4,5,6]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 7, 9], dtype=int32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## suma de las columnas\n",
    "tf.reduce_sum(m, reduction_indices = 0).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 6, 15], dtype=int32)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## suma de las filas\n",
    "tf.reduce_sum(m, reduction_indices = 1).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## suma de todos los elementos\n",
    "tf.reduce_sum(m).eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TensorFlow` implementa las siguiente funciones para reducir un tensor (https://www.tensorflow.org/api_guides/python/math_ops#Reduction)\n",
    "\n",
    "    tf.reduce_sum\n",
    "    tf.reduce_prod\n",
    "    tf.reduce_min\n",
    "    tf.reduce_max\n",
    "    tf.reduce_mean\n",
    "    tf.reduce_all\n",
    "    tf.reduce_any\n",
    "    tf.reduce_logsumexp\n",
    "    tf.count_nonzero\n",
    "    tf.accumulate_n\n",
    "    tf.einsum\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([Dimension(2), Dimension(3)])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## dimensiones\n",
    "m = tf.constant([[1,2,3], \n",
    "                 [4,5,6]])\n",
    "m.get_shape()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  4,  6],\n",
       "       [ 8, 10, 12]], dtype=int32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = m + m\n",
    "a.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 4, 5, 6]], dtype=int32)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reshape(m,(1,6)).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4],\n",
       "       [5, 6]], dtype=int32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reshape(m,(3,2)).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7,  4],\n",
       "       [ 6, 14]], dtype=int32)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tf.constant([[1, 2], [3, 4]])\n",
    "b = tf.constant([[5, 0], [0, 6]])\n",
    "tf.accumulate_n([a, b, a]).eval()  # [[7, 4], [6, 14]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7,  4],\n",
       "       [ 6, 14]], dtype=int32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.add_n([a, b, a]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.argmax(tf.constant([1,3,2,4,0])).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.argmin(tf.constant([1,3,2,4,0])).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 0.]\n"
     ]
    }
   ],
   "source": [
    "x = tf.constant([10.0, -1.0])\n",
    "y = tf.map_fn(lambda m: tf.cond(tf.greater_equal(m, 0.0), \n",
    "                                true_fn = lambda: tf.constant(1.), \n",
    "                                false_fn = lambda : tf.constant(0.0)),\n",
    "              x)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.]\n",
      " [0.]]\n"
     ]
    }
   ],
   "source": [
    "x = tf.constant([[10.0], [-1.0]])\n",
    "y = tf.map_fn(lambda m: tf.cond(tf.greater_equal(m[0], 0.0), \n",
    "                                true_fn = lambda: tf.constant([1.]), \n",
    "                                false_fn = lambda : tf.constant([0.0])),\n",
    "              x)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(y))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables y Placeholders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Las variables pueden entenderse como las variables locales de una función, las cuales no pueden ser accesadas desde el exterior de la función. Los placeholders son equivalentes a los parámetros, los cuales pueden tomar cualquier valor cuando se llama la función. En el siguiente ejemplo se evalua la expresión a * b - (b + c) para diferentes valores de a, b y c.\n",
    "\n",
    "     [-] (node6)\n",
    "      +--- [*] (node4)\n",
    "      |     +---- [a] (node1)\n",
    "      |     +---------------------+\n",
    "      |                           |\n",
    "      +--- [+] (node5)           [b] (node2)\n",
    "            +---- [c] (node3)     |\n",
    "            +---------------------+ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.0\n",
      "3.0\n",
      "8.0\n"
     ]
    }
   ],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "c = tf.placeholder(tf.float32)\n",
    "\n",
    "node4 = tf.multiply(a, b)\n",
    "node5 = tf.add(c, b)\n",
    "node6 = tf.subtract(node4, node5)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(node6, feed_dict={a: 5, b:3, c:2}))\n",
    "    print(sess.run(node6, feed_dict={a: 4, b:2, c:3}))\n",
    "    print(sess.run(node6, feed_dict={a: 4, b:3, c:1}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.46384197 -1.2342931   0.5352072 ]\n",
      " [-2.9913986  -2.8914263   0.07552639]\n",
      " [ 1.3343722   1.0699565  -0.2756308 ]]\n"
     ]
    }
   ],
   "source": [
    "##\n",
    "## la misma operación pero matricial\n",
    "##\n",
    "a = tf.placeholder(tf.float32, shape=(3,3))\n",
    "b = tf.placeholder(tf.float32, shape=(3,3))\n",
    "c = tf.placeholder(tf.float32, shape=(3,3))\n",
    "\n",
    "node4 = tf.multiply(a, b)\n",
    "node5 = tf.add(c, b)\n",
    "node6 = tf.subtract(node4, node5)\n",
    "\n",
    "with tf.Session() as sess:    \n",
    "    print(sess.run(node6, feed_dict={a: [[1, 0, 0,],\n",
    "                                         [0, 1, 0,],\n",
    "                                         [0, 0, 1,]],\n",
    "                                     b: np.random.normal(size=(3,3)), \n",
    "                                     c: np.random.normal(size=(3,3))}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El siguiente ejemplo presente un acumulador para el calculo de $n = n + 1$.\n",
    "\n",
    "    [=] (node1)  \n",
    "     +---[n]\n",
    "     +---[+] (node0)\n",
    "          +----[n]\n",
    "          +----[1]\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "## crea la variable y la inicializa a cero\n",
    "n = tf.Variable(0)\n",
    "\n",
    "## arbol sintactico que representa las operaciones\n",
    "node0 = tf.add(n, tf.constant(1))\n",
    "node1 = tf.assign(n, node0)\n",
    "\n",
    "## evalua\n",
    "with tf.Session() as sess:\n",
    "    \n",
    "    ## las variables deben ser inicializadas antes\n",
    "    ## de usarse\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for i in range(5):\n",
    "        ## el valor de las variables se retiene entre\n",
    "        ## las llamadas a run()\n",
    "        print(sess.run(node1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "## crea la variable y la inicializa a cero\n",
    "n = tf.Variable(0)\n",
    "\n",
    "## arbol sintactico que representa las operaciones\n",
    "## se usa tf.assign_add para simplificar el arbol\n",
    "## tambien existe tf.assign_sub\n",
    "node0 = tf.assign_add(n, tf.constant(1))\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for i in range(5):\n",
    "        print(sess.run(node0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Indique para que sirven las siguientes funciones del modulo `tf` y realice un ejemplo:\n",
    "\n",
    "    tf.clip_by_value\n",
    "    tf.clip_by_norm\n",
    "    tf.clip_by_average_norm\n",
    "    tf.clip_by_global_norm\n",
    "    tf.clip_by_norm\n",
    "    tf.concat\n",
    "    tf.case\n",
    "    tf.cond\n",
    "    tf.cumprod\n",
    "    tf.cumsum\n",
    "    tf.diag\n",
    "    tf.inverse_permutation\n",
    "    tf.log_sigmoid \n",
    "    tf.map_fn\n",
    "    \n",
    "    tf.matrix_inverse\n",
    "    tf.matrix_diag\n",
    "    tf.matrix_set_diag\n",
    "    tf.matrix_solve\n",
    "    tf.matrix_transpose\n",
    "    tf.parallel_stack\n",
    "    tf.stack\n",
    "    tf.random_shuffle\n",
    "    tf.reverse\n",
    "    \n",
    "    tf.maximum\n",
    "    tf.minimum\n",
    "    \n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n"
     ]
    }
   ],
   "source": [
    "x = tf.constant(2)\n",
    "y = tf.constant(5)\n",
    "def f1(): return tf.multiply(x, 17)\n",
    "def f2(): return tf.add(y, 23)\n",
    "r = tf.cond(tf.less(x, y), f1, f2)\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(r))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[20. 20. 30.]\n",
      " [40. 50. 60.]\n",
      " [70. 70. 70.]]\n"
     ]
    }
   ],
   "source": [
    "t = tf.placeholder(tf.float32, shape=(3,3))\n",
    "mi = tf.placeholder(tf.float32)\n",
    "ma = tf.placeholder(tf.float32)\n",
    "\n",
    "with tf.Session() as sess:    \n",
    "    print(sess.run(tf.clip_by_value(t, mi, ma), feed_dict={\n",
    "                                    t: [[10., 20., 30.,],\n",
    "                                         [40., 50., 60.,],\n",
    "                                         [70., 80., 90.,]],\n",
    "                                     mi: 20.0, \n",
    "                                     ma: 70.0\n",
    "                            }))\n",
    "sess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Funciones de activación"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  3., 10.], dtype=float32)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.relu([-3., 3., 10.]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 3., 6.], dtype=float32)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.relu6([-3., 3., 10.]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.26894143, 0.5       , 0.7310586 ], dtype=float32)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.sigmoid([-1., 0., 1.]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.5,  0. , -0.5], dtype=float32)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.softsign([-1., 0., -1.]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.31326166, 0.6931472 , 0.31326166], dtype=float32)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.softplus([-1., 0., -1.]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.63212055,  0.        , -0.63212055], dtype=float32)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.elu([-1., 0., -1.]).eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimización de modelos (regresión lineal)\n",
    "\n",
    "El siguiente ejemplo ilustra la construcción y estimación de un modelo de regresión lineal usando `TensorFlow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "211.96011\n",
      "13.4133625\n",
      "9.287522\n",
      "8.65929\n",
      "8.426337\n",
      "8.27987\n",
      "8.155496\n",
      "8.0380745\n",
      "7.9239626\n",
      "7.8122544\n",
      "7.7027025\n",
      "7.5952086\n",
      "7.4897175\n",
      "7.386186\n",
      "7.284568\n",
      "7.184822\n",
      "7.086908\n",
      "6.9907846\n",
      "6.8964167\n",
      "6.8037624\n",
      "6.712787\n",
      "6.6234508\n",
      "6.5357246\n",
      "6.4495687\n",
      "6.364952\n",
      "6.2818418\n",
      "6.200206\n",
      "6.120012\n",
      "6.041228\n",
      "5.9638257\n",
      "5.887778\n",
      "5.813052\n",
      "5.739623\n",
      "5.6674614\n",
      "5.5965433\n",
      "5.5268407\n",
      "5.45833\n",
      "5.3909836\n",
      "5.3247786\n",
      "5.259691\n",
      "5.195699\n",
      "5.132779\n",
      "5.0709085\n",
      "5.010068\n",
      "4.9502306\n",
      "4.8913817\n",
      "4.8335013\n",
      "4.776564\n",
      "4.7205544\n",
      "4.665455\n",
      "4.611245\n",
      "4.557907\n",
      "4.5054235\n",
      "4.4537783\n",
      "4.4029527\n",
      "4.352932\n",
      "4.3036976\n",
      "4.2552357\n",
      "4.2075315\n",
      "4.160569\n",
      "4.114334\n",
      "4.068813\n",
      "4.0239897\n",
      "3.9798524\n",
      "3.9363856\n",
      "3.893581\n",
      "3.8514204\n",
      "3.8098931\n",
      "3.768985\n",
      "3.7286904\n",
      "3.688991\n",
      "3.6498775\n",
      "3.6113381\n",
      "3.573366\n",
      "3.535946\n",
      "3.4990659\n",
      "3.4627204\n",
      "3.426898\n",
      "3.391588\n",
      "3.3567805\n",
      "3.3224683\n",
      "3.2886393\n",
      "3.2552853\n",
      "3.2223988\n",
      "3.189972\n",
      "3.1579943\n",
      "3.1264572\n",
      "3.0953546\n",
      "3.0646765\n",
      "3.0344188\n",
      "3.0045686\n",
      "2.9751234\n",
      "2.9460733\n",
      "2.9174113\n",
      "2.889132\n",
      "2.8612285\n",
      "2.8336906\n",
      "2.806519\n",
      "2.7797003\n",
      "2.75323\n",
      "2.7271056\n",
      "2.7013147\n",
      "2.67586\n",
      "2.6507266\n",
      "2.6259165\n",
      "2.6014194\n",
      "2.5772316\n",
      "2.5533485\n",
      "2.5297642\n",
      "2.5064745\n",
      "2.4834733\n",
      "2.4607553\n",
      "2.4383163\n",
      "2.4161537\n",
      "2.3942604\n",
      "2.3726332\n",
      "2.3512669\n",
      "2.3301573\n",
      "2.309302\n",
      "2.288694\n",
      "2.2683313\n",
      "2.2482088\n",
      "2.228324\n",
      "2.2086742\n",
      "2.189252\n",
      "2.1700544\n",
      "2.15108\n",
      "2.1323242\n",
      "2.113784\n",
      "2.095457\n",
      "2.0773382\n",
      "2.059424\n",
      "2.041712\n",
      "2.0242007\n",
      "2.0068831\n",
      "1.9897602\n",
      "1.9728266\n",
      "1.9560822\n",
      "1.9395192\n",
      "1.9231384\n",
      "1.9069424\n",
      "1.8909171\n",
      "1.8750697\n",
      "1.8593898\n",
      "1.8438792\n",
      "1.8285341\n",
      "1.8133569\n",
      "1.7983383\n",
      "1.7834806\n",
      "1.7687774\n",
      "1.7542306\n",
      "1.7398375\n",
      "1.7255951\n",
      "1.7114992\n",
      "1.6975465\n",
      "1.6837399\n",
      "1.6700771\n",
      "1.6565522\n",
      "1.643166\n",
      "1.6299149\n",
      "1.6168019\n",
      "1.6038206\n",
      "1.5909675\n",
      "1.5782444\n",
      "1.5656478\n",
      "1.5531796\n",
      "1.5408337\n",
      "1.5286088\n",
      "1.5165064\n",
      "1.5045228\n",
      "1.4926537\n",
      "1.4809012\n",
      "1.4692671\n",
      "1.4577434\n",
      "1.4463305\n",
      "1.4350245\n",
      "1.423835\n",
      "1.4127481\n",
      "1.4017668\n",
      "1.3908879\n",
      "1.380117\n",
      "1.3694456\n",
      "1.3588775\n",
      "1.3484055\n",
      "1.3380342\n",
      "1.3277534\n",
      "1.3175817\n",
      "1.3074902\n",
      "1.2974999\n",
      "1.2875998\n",
      "1.277797\n",
      "1.2680737\n",
      "1.258447\n",
      "1.2489064\n",
      "1.239451\n",
      "1.230082\n",
      "1.2207997\n",
      "1.2115998\n",
      "1.2024807\n",
      "1.1934491\n",
      "1.1844909\n",
      "1.1756235\n",
      "1.1668234\n",
      "1.1581076\n",
      "1.1494696\n",
      "1.1409066\n",
      "1.1324189\n",
      "1.1240077\n",
      "1.1156685\n",
      "1.1074023\n",
      "1.0992136\n",
      "1.0910833\n",
      "1.0830402\n",
      "1.0750542\n",
      "1.0671409\n",
      "1.0593016\n",
      "1.0515201\n",
      "1.0438148\n",
      "1.0361687\n",
      "1.0285904\n",
      "1.0210795\n",
      "1.0136302\n",
      "1.006243\n",
      "0.9989213\n",
      "0.9916583\n",
      "0.9844582\n",
      "0.9773209\n",
      "0.97023827\n",
      "0.963223\n",
      "0.95625895\n",
      "0.94935787\n",
      "0.94251096\n",
      "0.9357217\n",
      "0.92899245\n",
      "0.92231435\n",
      "0.91569585\n",
      "0.9091294\n",
      "0.9026158\n",
      "0.89615846\n",
      "0.8897513\n",
      "0.8833989\n",
      "0.8770989\n",
      "0.87084746\n",
      "0.8646513\n",
      "0.8585009\n",
      "0.85240465\n",
      "0.84635645\n",
      "0.8403536\n",
      "0.8344071\n",
      "0.8284968\n",
      "[array([0.02485408, 0.0371238 , 0.22055584, 0.33567455, 0.48434687],\n",
      "      dtype=float32), array([1.9212362, 2.8591158, 2.6779587, 2.8796468, 2.6940854],\n",
      "      dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "## \n",
    "## El modelo es el siguiente:\n",
    "##\n",
    "##    y = w' x + b\n",
    "##\n",
    "## Muestra de datos:\n",
    "##\n",
    "##    x      y\n",
    "## -------------\n",
    "##  1.1   1.98\n",
    "##  2.0   3.01\n",
    "##  2.6   3.51\n",
    "##  3.2   4.34\n",
    "##  4.3   5.32\n",
    "##\n",
    "\n",
    "## x y y son constantes\n",
    "x = tf.constant([[ 1.1,  2.0,  2.6,  3.2 ,  4.3],\n",
    "               [ 2.1,  4.0,  3.6,  4.2 ,  5.3],\n",
    "               [ 3.1,  5.0,  4.6,  5.2 ,  6.3]])\n",
    "#x = tf.constant([ 1.1,  2.0,  2.6,  3.2 ,  4.3])\n",
    "y = tf.constant([ 1.98, 3.01, 3.51, 4.34, 5.32])\n",
    "\n",
    "## w y b son las variables a estimar\n",
    "w = tf.Variable([0.0, 0.0, 0.0, 0.0, 0.0])\n",
    "\n",
    "b = tf.Variable([0.0, 0.0, 0.0, 0.0, 0.0])\n",
    "\n",
    "## Define el modelo\n",
    "\n",
    "## elems = np.array([1, 2, 3, 4, 5, 6])\n",
    "## map_fn(lambda x: x * x, elems)\n",
    "\n",
    "\n",
    "m = tf.add(tf.multiply(x, w), b)\n",
    "\n",
    "#elems = np.array([0, 1, 2, 3, 4, 5])\n",
    "## map_fn(lambda x: x * x, elems)\n",
    "\n",
    "\n",
    "#m = tf.add(tf.map_fn(lambda i: (tf.multiply(x[i], w), b[i]), elems))\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Define la función de error\n",
    "sse = tf.reduce_sum(tf.square(m - y)) # sum of the squares\n",
    "\n",
    "## Inicializa el optimizador\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.0001)\n",
    "\n",
    "## Minimiza la función de error\n",
    "opt = optimizer.minimize(sse)\n",
    "\n",
    "## estima el modelo\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for i in range(50000):\n",
    "        sess.run(opt)\n",
    "        if (i % 200 == 0):\n",
    "            print(sess.run(sse))\n",
    "        \n",
    "    print(sess.run([w, b]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Modifique el código anterior para que pueda operar sobre datos matriciales.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introducción a TensorFlow y Keras\n",
    "===\n",
    "\n",
    "**Juan David Velásquez Henao**  \n",
    "jdvelasq@unal.edu.co   \n",
    "Universidad Nacional de Colombia, Sede Medellín  \n",
    "Facultad de Minas  \n",
    "Medellín, Colombia\n",
    "\n",
    "---\n",
    "\n",
    "Haga click [aquí](https://github.com/jdvelasq/neural-networks/tree/master/) para acceder al repositorio online.\n",
    "\n",
    "Haga click [aquí](http://nbviewer.jupyter.org/github/jdvelasq/neural-networks/tree/master/) para explorar el repositorio usando `nbviewer`. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}