RNN.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Embedding, Dense, SimpleRNN,LSTM\n",
    "from tensorflow.keras.models import Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### مدل ساده RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 32, 16)            8000      \n",
      "_________________________________________________________________\n",
      "simple_rnn_1 (SimpleRNN)     (None, 64)                5184      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 5)                 325       \n",
      "=================================================================\n",
      "Total params: 13,509\n",
      "Trainable params: 13,509\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential([\n",
    "    Embedding(input_dim = 500, output_dim = 16,input_length = 32),\n",
    "    SimpleRNN(64,activation = 'tanh'),\n",
    "    Dense(5,activation = 'softmax')\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### مدل LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 32, 16)            8000      \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 64)                20736     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 5)                 325       \n",
      "=================================================================\n",
      "Total params: 29,061\n",
      "Trainable params: 29,061\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential([\n",
    "    Embedding(input_dim = 500, output_dim = 16,input_length = 32),\n",
    "    LSTM(64,activation = 'tanh'),\n",
    "    Dense(5,activation = 'softmax')\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "استفاده ازbatch norm در شبکه های RNN کمی پیچ و خم فنی دارد. برای همین به جای اسفتاده از آن ها برای مواجه با مشکل محو گرادیان ها یا انفجار گرادیان ها می توانیم از روش دیگری به نام graidan clipping بهره ببریم.\n",
    "\n",
    "گرادیان ها را در زمان پس انتشار () clipp می کنیم."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(clipvalue = 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "در این حالت تمام گرادیان هابه محدوده بین -1 تا 1 در آورده می شوند. این مقدار را هم می توانیم به صورت یک ابرپارامتر تنظیم سازی کنیم.\n",
    "\n",
    "به طور مثال اگر گرادیان های اصلی به صورت )-2.3، 0.8، 14) باشند بعد از clip   در می آیند کردن  به صورت (-1،0.8،1 )\n",
    "\n",
    "به صورت جایگزین می توانیم از clipnorm استفاده کنیم که جهت گرادیان را حفظ  می کند و به گونه ای گرادیان ها را تبدیل می کند که بین 0 و 1 قرار بگیرند ولی جهت اصلی گرادیان ها حفظ شود.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}