sound_recognition_using_tcn_urbansound8k.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "sound_recognition_using_tcn_urbansound8k.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "language_info": {
      "name": "python",
      "version": "3.6.4",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8CQnLTiKZ83f",
        "colab_type": "text"
      },
      "source": [
        "# **Code for the Paper \"End-to-End Sound Recognition using Temporal Convolutional Networks\"** \n",
        "*Author: Eric Schölzel, TU Dresden*\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "A quick-and-dirty implementation of the proposed pipeline, using mel spectrograms, a CNN-TCN architecture and live training data augmentation with SpecAugment.\n",
        "\n",
        "The UrbanSound8K Dataset (non-Kaggle Version!) is used here as an example.\n",
        "It contains 8732 samples in 10 (mutually exclusive) classes. All Samples have a length of <=4s (most are 4s) - so keep in mind that this is the type of dataset where a variety of approaches like fully-fledged State-of-the-Art Image Recognition networks with millions of parameters or non-neural-network approaches like Decision Trees usually have their domain.\n",
        "It's just one example of the pipeline that can easily adapted to tasks with other requirements such as Speech Recognition or Segmentation of audio files.\n",
        "\n",
        "*Disclaimer: This is run on a custom train test split. For actuall scientific comparisons to other approaches 10-fold cross validation is strongly recommended by the authers of the dataset: https://urbansounddataset.weebly.com/urbansound8k.html*\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwbgXfRHg-El",
        "colab_type": "text"
      },
      "source": [
        "**Install Additional Dependencies**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nJfQw4qGHqge",
        "colab_type": "code",
        "outputId": "bd27db6b-882c-4a11-e1d2-f06f3f9ffc06",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 629
        }
      },
      "source": [
        "!pip install librosa specaugment wget keras-tcn\n",
        "\n",
        "!apt-get -y install ffmpeg"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: librosa in /usr/local/lib/python3.6/dist-packages (0.6.3)\n",
            "Requirement already satisfied: specaugment in /usr/local/lib/python3.6/dist-packages (1.2.5)\n",
            "Requirement already satisfied: wget in /usr/local/lib/python3.6/dist-packages (3.2)\n",
            "Collecting keras-tcn\n",
            "  Downloading https://files.pythonhosted.org/packages/ea/71/a23ddfcee18342a4c3ce464f99c44e5dad1c637be13c73638d8551d57906/keras_tcn-2.8.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (0.21.2)\n",
            "Requirement already satisfied: audioread>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (2.1.8)\n",
            "Requirement already satisfied: six>=1.3 in /usr/local/lib/python3.6/dist-packages (from librosa) (1.12.0)\n",
            "Requirement already satisfied: numpy>=1.8.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (1.16.4)\n",
            "Requirement already satisfied: resampy>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (0.2.1)\n",
            "Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (1.3.0)\n",
            "Requirement already satisfied: joblib>=0.12 in /usr/local/lib/python3.6/dist-packages (from librosa) (0.13.2)\n",
            "Requirement already satisfied: numba>=0.38.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (0.40.1)\n",
            "Requirement already satisfied: decorator>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa) (4.4.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from specaugment) (3.0.3)\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.6/dist-packages (from keras-tcn) (2.2.4)\n",
            "Requirement already satisfied: llvmlite>=0.25.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.38.0->librosa) (0.29.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->specaugment) (2.4.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->specaugment) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->specaugment) (1.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->specaugment) (2.5.3)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from keras->keras-tcn) (1.1.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from keras->keras-tcn) (1.0.8)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras->keras-tcn) (2.8.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras->keras-tcn) (3.13)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->specaugment) (41.0.1)\n",
            "Installing collected packages: keras-tcn\n",
            "Successfully installed keras-tcn-2.8.2\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "ffmpeg is already the newest version (7:3.4.6-0ubuntu0.18.04.1).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-410\n",
            "Use 'apt autoremove' to remove it.\n",
            "0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hH1UnROnaPkI",
        "colab_type": "text"
      },
      "source": [
        "**Download and extract the dataset**\n",
        "(non-Kaggle version)\n",
        "\n",
        "*This may take a while. It's around ~6GB. If you already have the spectrograms calculated and saved on Google Drive, you can skip that.*\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iKiOb46Dun0Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import wget\n",
        "import tarfile\n",
        "\n",
        "local_file = \"urbansound8k.tar.gz\"\n",
        "print(\"Downloading dataset...\")\n",
        "url = \"https://zenodo.org/record/1203745/files/UrbanSound8K.tar.gz\"\n",
        "wget.download(url, local_file)\n",
        "\n",
        "print(\"Extracting dataset...\")\n",
        "with tarfile.open(local_file, \"r:gz\") as tar:\n",
        "    tar.extractall()\n",
        "    tar.close()\n",
        "\n",
        "print(\"Done.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ity78yWua0Zo",
        "colab_type": "text"
      },
      "source": [
        "**Import Python libraries and set paths**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "id": "MEfVJg9Es6k1",
        "colab_type": "code",
        "outputId": "ffeb8c1d-f4d1-4270-85c3-9e1cc5f59aa4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import numpy as np # linear algebra\n",
        "import librosa\n",
        "# see https://github.com/librosa/librosa/issues/477\n",
        "# import soundfile\n",
        "import matplotlib\n",
        "matplotlib.use('Agg')\n",
        "import matplotlib.pyplot as plt \n",
        "import tcn\n",
        "import os\n",
        "import zipfile\n",
        "import pickle\n",
        "import keras\n",
        "import random\n",
        "import tensorflow as tf\n",
        "import time\n",
        "import sys\n",
        "import gc\n",
        "from sklearn.model_selection import train_test_split\n",
        "from math import ceil\n",
        "from tensorflow.keras import backend as K\n",
        "from tcn import TCN\n",
        "\n",
        "script_path = \"./\"\n",
        "dataset_path = \"./UrbanSound8K/\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3pujEWLa-Co",
        "colab_type": "text"
      },
      "source": [
        "**Calculate features/Spectrograms.**\n",
        "\n",
        "You only need to run this once (unless the data get deleted - which probably will if you're on Google Colab and close the session).\n",
        "\n",
        "This may take a while... In the meantime, you can drink a coffee. \n",
        "(It will take some time. It's slow. Really. That's one of the reasons why real time data augmentation on the spectrograms is so great! And yeah, it's long enough that I've spent the time to include a timer...)\n",
        "\n",
        "This could probably be faster with some parameter tuning... our outside of Google Colab with a better CPU.\n",
        "\n",
        "(I tried to save them to Google Drive once calculated and use the API to download them here, but the API was bugged and just gave errors... Maybe that'll get fixed at some point. Uploading them myself was extremely slow so that didn't work out either. Idk why.)\n",
        "\n",
        "*Note: we load the entire dataset into RAM here since RAM in Colab is sufficient for that. When there's not as much RAM, you'll have to change that.*\n",
        "\n",
        "***Do NOT close Google Colab while calculating. This may reset your session and you'll have to start over again.***\n",
        "\n",
        "**If you saved the spectrograms to Google Drive earlier, you can skip this block.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ddCeOaGQ6PRO",
        "colab_type": "code",
        "outputId": "64473ea0-b4d5-4612-c807-5ce7b6b07455",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "\n",
        "def load_sample_and_calc_features(filename):\n",
        "    audio, sample_rate = librosa.load(filename, res_type=\"kaiser_fast\")\n",
        "    features = librosa.feature.melspectrogram(y=audio, sr=sample_rate, n_fft=2048, n_mels=256, hop_length=512, power=2.0)\n",
        "    return features\n",
        "\n",
        "def load_dataset(label_file, sample_folder, dataset_labels=set()):\n",
        "    dataset_x = []\n",
        "    dataset_y = []\n",
        "    processed = 0\n",
        "    with open(label_file, \"r\") as train_labels_file:\n",
        "        all_content = train_labels_file.readlines()\n",
        "        \n",
        "        for line in all_content[1:]:\n",
        "            line = line.replace(\"\\n\", \"\")\n",
        "            slice_file_name, fsID, start, end, salience, fold, classID, classname = line.split(\",\")\n",
        "            \n",
        "            dataset_labels.add(classname)\n",
        "            \n",
        "            filename = sample_folder + \"fold\" + fold + \"/\" + slice_file_name\n",
        "            if processed % 100 == 0:\n",
        "                print(processed, end= \" -> \")\n",
        "            x = load_sample_and_calc_features(filename)\n",
        "            dataset_x.append(x)\n",
        "            dataset_y.append(int(classID))\n",
        "            processed += 1\n",
        "\n",
        "    dataset_labels = list(dataset_labels)\n",
        "\n",
        "    dataset_y = keras.utils.to_categorical(dataset_y)\n",
        "\n",
        "    dataset = list(zip(dataset_x, dataset_y))\n",
        "    return dataset, dataset_labels\n",
        "\n",
        "time_load_start = time.time()\n",
        "dataset_and_classes = load_dataset(dataset_path + \"/metadata/UrbanSound8K.csv\", dataset_path + \"/audio/\")\n",
        "time_load = time.time() - time_load_start\n",
        "\n",
        "dataset_full, classes = dataset_and_classes\n",
        "pickle_path = script_path + \"/dataset_and_classes.pickle\"\n",
        "\n",
        "print(\"\\nDone. Loading and calculating features took \" + str(time_load) + \" seconds\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0 -> 100 -> 200 -> 300 -> 400 -> 500 -> 600 -> 700 -> 800 -> 900 -> 1000 -> 1100 -> 1200 -> 1300 -> 1400 -> 1500 -> 1600 -> 1700 -> 1800 -> 1900 -> 2000 -> 2100 -> 2200 -> 2300 -> 2400 -> 2500 -> 2600 -> 2700 -> 2800 -> 2900 -> 3000 -> 3100 -> 3200 -> 3300 -> 3400 -> 3500 -> 3600 -> 3700 -> 3800 -> 3900 -> 4000 -> 4100 -> 4200 -> 4300 -> 4400 -> 4500 -> 4600 -> 4700 -> 4800 -> 4900 -> 5000 -> 5100 -> 5200 -> 5300 -> 5400 -> 5500 -> 5600 -> 5700 -> 5800 -> 5900 -> 6000 -> 6100 -> 6200 -> 6300 -> 6400 -> 6500 -> 6600 -> 6700 -> 6800 -> 6900 -> 7000 -> 7100 -> 7200 -> 7300 -> 7400 -> 7500 -> 7600 -> 7700 -> 7800 -> 7900 -> 8000 -> 8100 -> 8200 -> 8300 -> 8400 -> 8500 -> 8600 -> 8700 -> \n",
            "Done. Loading and calculating features took 4805.445519685745 seconds\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w0rdBgWhIo2e",
        "colab_type": "text"
      },
      "source": [
        "**Optional**:\n",
        "Connect to Google Drive (You have to enter the Authorization Code and press enter).\n",
        "\n",
        "Pickle your calculated spectrograms to Google Drive instead for not having to calculate it again later.\n",
        "\n",
        "*OR*\n",
        "load it if you did that already and skipped the previous section**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cUQBFjTyHWAi",
        "colab_type": "code",
        "outputId": "23a03258-5752-4b7f-f3ac-73c91c72132a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "pickle_path = script_path + \"drive/My Drive/dataset_and_classes.pickle\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qjhh7SqPIxKy",
        "colab_type": "text"
      },
      "source": [
        "***Pickle it... (If not done already)***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wTQas5hMHQea",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if dataset_and_classes is not None:\n",
        "  with open(pickle_path, \"wb+\") as pfile:\n",
        "      pickle.dump(dataset_and_classes, pfile)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZrK2jWQjmBk",
        "colab_type": "text"
      },
      "source": [
        "**Loading pickled Spectrograms**\n",
        "\n",
        "You can load previously the previously saved spectrograms here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qsUE9dXpJhX-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with open(pickle_path, \"rb\") as pfile:\n",
        "    dataset_full, classes = pickle.load(pfile)\n",
        "pfile.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JD2fq2eIs6k-",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Since the \"full\" SpecAugment implementation (from https://github.com/shelling203/SpecAugment) still seems to be buggy (Tensorflow implementation throws errors and is extremely brutaly slow and PyTorch implementation isn't completed yet and still seems to be slow - could maybe be Google Colab related), instead skip the next block and use the small one. This will be without Time Warping, tho. All of those implementations are \"inofficial\" btw.\n",
        " \n",
        "The PyTorch version of that works. Since Time Warping isn't completely implemented yet, this code doesn't depend on PyTorch yet. It's not throwing errors and being unusable slow like the tensorflow one, but it's still slow. This could be Google Colab related. *TODO for people with Nvidia GPU: Test the Tensorflow Implementation outside of Google Colab ;)*\n",
        "\n",
        "*This has been copyied here because we need to use matplotlib.use('Agg') (see import section) to avoid errors. That doesn't seem to work when just using the imported module.*\n",
        "\n",
        "**TL;DR:  since it doesn't work properly in Google Colab atm, you can just skip the next 2 code blocks**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F2Ji9SQrtvmd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# SpecAugment PyTorch Implementation\n",
        "\n",
        "# Copyright 2019 RnD at Spoon Radio\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# ==============================================================================\n",
        "\"\"\"SpecAugment Implementation for Tensorflow.\n",
        "Related paper : https://arxiv.org/pdf/1904.08779.pdf\n",
        "\n",
        "In this paper, show summarized parameters by each open datasets in Tabel 1.\n",
        "-----------------------------------------\n",
        "Policy | W  | F  | m_F |  T  |  p  | m_T\n",
        "-----------------------------------------\n",
        "None   |  0 |  0 |  -  |  0  |  -  |  -\n",
        "-----------------------------------------\n",
        "LB     | 80 | 27 |  1  | 100 | 1.0 | 1\n",
        "-----------------------------------------\n",
        "LD     | 80 | 27 |  2  | 100 | 1.0 | 2\n",
        "-----------------------------------------\n",
        "SM     | 40 | 15 |  2  |  70 | 0.2 | 2\n",
        "-----------------------------------------\n",
        "SS     | 40 | 27 |  2  |  70 | 0.2 | 2\n",
        "-----------------------------------------\n",
        "LB : LibriSpeech basic\n",
        "LD : LibriSpeech double\n",
        "SM : Switchboard mild\n",
        "SS : Switchboard strong\n",
        "\"\"\"\n",
        "\n",
        "import librosa\n",
        "import librosa.display\n",
        "import math\n",
        "import numpy as np\n",
        "import random\n",
        "import matplotlib\n",
        "# matplotlib.use('TkAgg')\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "def spec_augment_pytorch(mel_spectrogram, time_warping_para=80, frequency_masking_para=27,\n",
        "                 time_masking_para=100, frequency_mask_num=1, time_mask_num=1):\n",
        "    \"\"\"Spec augmentation Calculation Function.\n",
        "\n",
        "    'SpecAugment' have 3 steps for audio data augmentation.\n",
        "    first step is time warping using Tensorflow's image_sparse_warp function.\n",
        "    Second step is frequency masking, last step is time masking.\n",
        "\n",
        "    # Arguments:\n",
        "      mel_spectrogram(numpy array): audio file path of you want to warping and masking.\n",
        "      time_warping_para(float): Augmentation parameter, \"time warp parameter W\".\n",
        "        If none, default = 80 for LibriSpeech.\n",
        "      frequency_masking_para(float): Augmentation parameter, \"frequency mask parameter F\"\n",
        "        If none, default = 100 for LibriSpeech.\n",
        "      time_masking_para(float): Augmentation parameter, \"time mask parameter T\"\n",
        "        If none, default = 27 for LibriSpeech.\n",
        "      frequency_mask_num(float): number of frequency masking lines, \"m_F\".\n",
        "        If none, default = 1 for LibriSpeech.\n",
        "      time_mask_num(float): number of time masking lines, \"m_T\".\n",
        "        If none, default = 1 for LibriSpeech.\n",
        "\n",
        "    # Returns\n",
        "      mel_spectrogram(numpy array): warped and masked mel spectrogram.\n",
        "    \"\"\"\n",
        "    v = mel_spectrogram.shape[0]\n",
        "    tau = mel_spectrogram.shape[1]\n",
        "\n",
        "    # Step 1 : Time warping (TO DO...)\n",
        "    warped_mel_spectrogram = np.zeros(mel_spectrogram.shape,\n",
        "                                      dtype=mel_spectrogram.dtype)\n",
        "\n",
        "    for i in range(v):\n",
        "        for j in range(tau):\n",
        "            offset_x = 0\n",
        "            offset_y = 0\n",
        "            if i + offset_y < v:\n",
        "                warped_mel_spectrogram[i, j] = mel_spectrogram[(i + offset_y) % v, j]\n",
        "            else:\n",
        "                warped_mel_spectrogram[i, j] = mel_spectrogram[i, j]\n",
        "\n",
        "    # Step 2 : Frequency masking\n",
        "    for i in range(frequency_mask_num):\n",
        "        f = np.random.uniform(low=0.0, high=frequency_masking_para)\n",
        "        f = int(f)\n",
        "        f0 = random.randint(0, v - f)\n",
        "        warped_mel_spectrogram[f0:f0 + f, :] = 0\n",
        "\n",
        "    # Step 3 : Time masking\n",
        "    for i in range(time_mask_num):\n",
        "        t = np.random.uniform(low=0.0, high=time_masking_para)\n",
        "        t = int(t)\n",
        "        t0 = random.randint(0, tau - t)\n",
        "        warped_mel_spectrogram[:, t0:t0 + t] = 0\n",
        "\n",
        "    return warped_mel_spectrogram\n",
        "\n",
        "\n",
        "def visualization_spectrogram(mel_spectrogram, title):\n",
        "    \"\"\"visualizing result of SpecAugment\n",
        "\n",
        "    # Arguments:\n",
        "      mel_spectrogram(ndarray): mel_spectrogram to visualize.\n",
        "      title(String): plot figure's title\n",
        "    \"\"\"\n",
        "    # Show mel-spectrogram using librosa's specshow.\n",
        "    plt.figure(figsize=(10, 4))\n",
        "    librosa.display.specshow(librosa.power_to_db(mel_spectrogram, ref=np.max), y_axis='mel', fmax=8000, x_axis='time')\n",
        "    # plt.colorbar(format='%+2.0f dB')\n",
        "    plt.title(title)\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b3stTgSq_AJl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# SpecAugment Tensorflow Implementation\n",
        "\n",
        "\n",
        "# Copyright 2019 RnD at Spoon Radio\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# ==============================================================================\n",
        "\"\"\"SpecAugment Implementation for Tensorflow.\n",
        "Related paper : https://arxiv.org/pdf/1904.08779.pdf\n",
        "\n",
        "In this paper, show summarized parameters by each open datasets in Tabel 1.\n",
        "-----------------------------------------\n",
        "Policy | W  | F  | m_F |  T  |  p  | m_T\n",
        "-----------------------------------------\n",
        "None   |  0 |  0 |  -  |  0  |  -  |  -\n",
        "-----------------------------------------\n",
        "LB     | 80 | 27 |  1  | 100 | 1.0 | 1\n",
        "-----------------------------------------\n",
        "LD     | 80 | 27 |  2  | 100 | 1.0 | 2\n",
        "-----------------------------------------\n",
        "SM     | 40 | 15 |  2  |  70 | 0.2 | 2\n",
        "-----------------------------------------\n",
        "SS     | 40 | 27 |  2  |  70 | 0.2 | 2\n",
        "-----------------------------------------\n",
        "LB : LibriSpeech basic\n",
        "LD : LibriSpeech double\n",
        "SM : Switchboard mild\n",
        "SS : Switchboard strong\n",
        "\"\"\"\n",
        "\n",
        "import librosa\n",
        "import librosa.display\n",
        "import tensorflow as tf\n",
        "from tensorflow.contrib.image import sparse_image_warp\n",
        "import numpy as np\n",
        "import random\n",
        "import matplotlib\n",
        "# matplotlib.use('TkAgg')\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "def spec_augment_tensorflow(mel_spectrogram, time_warping_para=80, frequency_masking_para=27,\n",
        "                 time_masking_para=100, frequency_mask_num=1, time_mask_num=1):\n",
        "    \"\"\"Spec augmentation Calculation Function.\n",
        "\n",
        "    'SpecAugment' have 3 steps for audio data augmentation.\n",
        "    first step is time warping using Tensorflow's image_sparse_warp function.\n",
        "    Second step is frequency masking, last step is time masking.\n",
        "\n",
        "    # Arguments:\n",
        "      mel_spectrogram(numpy array): audio file path of you want to warping and masking.\n",
        "      time_warping_para(float): Augmentation parameter, \"time warp parameter W\".\n",
        "        If none, default = 80 for LibriSpeech.\n",
        "      frequency_masking_para(float): Augmentation parameter, \"frequency mask parameter F\"\n",
        "        If none, default = 100 for LibriSpeech.\n",
        "      time_masking_para(float): Augmentation parameter, \"time mask parameter T\"\n",
        "        If none, default = 27 for LibriSpeech.\n",
        "      frequency_mask_num(float): number of frequency masking lines, \"m_F\".\n",
        "        If none, default = 1 for LibriSpeech.\n",
        "      time_mask_num(float): number of time masking lines, \"m_T\".\n",
        "        If none, default = 1 for LibriSpeech.\n",
        "\n",
        "    # Returns\n",
        "      mel_spectrogram(numpy array): warped and masked mel spectrogram.\n",
        "    \"\"\"\n",
        "    v = mel_spectrogram.shape[0]\n",
        "    tau = mel_spectrogram.shape[1]\n",
        "\n",
        "    # Step 1 : Time warping\n",
        "    # Image warping control point setting.\n",
        "    mel_spectrogram_holder = tf.placeholder(tf.float32, shape=[1, v, tau, 1])\n",
        "    location_holder = tf.placeholder(tf.float32, shape=[1, 1, 2])\n",
        "    destination_holder = tf.placeholder(tf.float32, shape=[1, 1, 2])\n",
        "\n",
        "    center_position = v/2\n",
        "    random_point = np.random.randint(low=time_warping_para, high=tau - time_warping_para)\n",
        "    # warping distance chose.\n",
        "    w = np.random.uniform(low=0, high=time_warping_para)\n",
        "\n",
        "    control_point_locations = [[center_position, random_point]]\n",
        "    control_point_locations = np.float32(np.expand_dims(control_point_locations, 0))\n",
        "\n",
        "    control_point_destination = [[center_position, random_point + w]]\n",
        "    control_point_destination = np.float32(np.expand_dims(control_point_destination, 0))\n",
        "\n",
        "    # mel spectrogram data type convert to tensor constant for sparse_image_warp.\n",
        "    mel_spectrogram = mel_spectrogram.reshape([1, mel_spectrogram.shape[0], mel_spectrogram.shape[1], 1])\n",
        "    mel_spectrogram = np.float32(mel_spectrogram)\n",
        "\n",
        "    warped_mel_spectrogram_op, _ = sparse_image_warp(mel_spectrogram_holder,\n",
        "                                                     source_control_point_locations=location_holder,\n",
        "                                                     dest_control_point_locations=destination_holder,\n",
        "                                                     interpolation_order=2,\n",
        "                                                     regularization_weight=0,\n",
        "                                                     num_boundary_points=1\n",
        "                                                     )\n",
        "\n",
        "    # Change warp result's data type to numpy array for masking step.\n",
        "    feed_dict = {mel_spectrogram_holder:mel_spectrogram,\n",
        "                 location_holder:control_point_locations,\n",
        "                 destination_holder:control_point_destination}\n",
        "\n",
        "    with tf.Session() as sess:\n",
        "        warped_mel_spectrogram = sess.run(warped_mel_spectrogram_op, feed_dict=feed_dict)\n",
        "\n",
        "    warped_mel_spectrogram = warped_mel_spectrogram.reshape([warped_mel_spectrogram.shape[1],\n",
        "                                                             warped_mel_spectrogram.shape[2]])\n",
        "\n",
        "    # Step 2 : Frequency masking\n",
        "    for i in range(frequency_mask_num):\n",
        "        f = np.random.uniform(low=0.0, high=frequency_masking_para)\n",
        "        f = int(f)\n",
        "        f0 = random.randint(0, v - f)\n",
        "        warped_mel_spectrogram[f0:f0 + f, :] = 0\n",
        "\n",
        "    # Step 3 : Time masking\n",
        "    for i in range(time_mask_num):\n",
        "        t = np.random.uniform(low=0.0, high=time_masking_para)\n",
        "        t = int(t)\n",
        "        t0 = random.randint(0, tau - t)\n",
        "        warped_mel_spectrogram[:, t0:t0 + t] = 0\n",
        "\n",
        "    return warped_mel_spectrogram\n",
        "\n",
        "\n",
        "def visualization_spectrogram(mel_spectrogram, title):\n",
        "    \"\"\"visualizing result of SpecAugment\n",
        "\n",
        "    # Arguments:\n",
        "      mel_spectrogram(ndarray): mel_spectrogram to visualize.\n",
        "      title(String): plot figure's title\n",
        "    \"\"\"\n",
        "    # Show mel-spectrogram using librosa's specshow.\n",
        "    plt.figure(figsize=(10, 4))\n",
        "    librosa.display.specshow(librosa.power_to_db(mel_spectrogram, ref=np.max), y_axis='mel', fmax=8000, x_axis='time')\n",
        "    # plt.colorbar(format='%+2.0f dB')\n",
        "    plt.title(title)\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4s0GxO9b5xx",
        "colab_type": "text"
      },
      "source": [
        "**Simple SpecAugment (currently used!)**\n",
        "\n",
        "\n",
        "Due to the problems described above (could be a Google Colab problem, I couldn't test this properly at home since I don't have any Nvidia GPU),\n",
        "we use this simple implementation without Time Warping for now. It's taken from https://www.kaggle.com/davids1992/specaugment-quick-implementation\n",
        "It doesn't contain Time Warping, but it's fast."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FpQuXkamHD-j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# from https://www.kaggle.com/davids1992/specaugment-quick-implementation\n",
        "# without time warping\n",
        "def spec_augment_simple(spec: np.ndarray, num_mask=2, \n",
        "                 freq_masking_max_percentage=0.15, time_masking_max_percentage=0.3):\n",
        "\n",
        "    spec = spec.copy()\n",
        "    for i in range(num_mask):\n",
        "        all_frames_num, all_freqs_num = spec.shape\n",
        "        freq_percentage = random.uniform(0.0, freq_masking_max_percentage)\n",
        "        \n",
        "        num_freqs_to_mask = int(freq_percentage * all_freqs_num)\n",
        "        f0 = np.random.uniform(low=0.0, high=all_freqs_num - num_freqs_to_mask)\n",
        "        f0 = int(f0)\n",
        "        spec[:, f0:f0 + num_freqs_to_mask] = 0\n",
        "\n",
        "        time_percentage = random.uniform(0.0, time_masking_max_percentage)\n",
        "        \n",
        "        num_frames_to_mask = int(time_percentage * all_frames_num)\n",
        "        t0 = np.random.uniform(low=0.0, high=all_frames_num - num_frames_to_mask)\n",
        "        t0 = int(t0)\n",
        "        spec[t0:t0 + num_frames_to_mask, :] = 0\n",
        "    \n",
        "    return spec"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7vcwybiGdaDF",
        "colab_type": "text"
      },
      "source": [
        "**Model Building**\n",
        "\n",
        "Build the network model. It contains an Input Layer, two Convolutional Layers for additional feature extraction (see Paper) and a TCN unit.\n",
        "\n",
        "*This architecture has only ~560k Weights (for comparison: State-of-the-Art image recognition networks often have tens of millions of weights!). Weight file size is ~7.28mb which is definitely suitable for mobile apps, for example. (Who wants 500mb apps just for sound detection? :P)*\n",
        "\n",
        "Adam (with default parameters) is used for optimization. SGD+Momentum can lead to better results, when hyperparameters are set good enough.\n",
        "\n",
        "\n",
        "Due to make the generator simpler, currently a fixed input size is used. The length of the samples varies (128, n) and if n < 174, it gets zero-padded to that size (174 is max length of the spectrogram which is 4s in the raw audio).\n",
        "However, if this is adapted to datasets where samples can have arbitrary length it makes sense to change that to a varying size (which would save memory/computation time for smaller samples).\n",
        "\n",
        "With TCN it's possible to set a variable time length - but it takes some adjustments, e.g. the generator would have to look for the longest sample in the batch first, because in a batch every sample must have the same length.\n",
        "\n",
        "To make things simpler, here a fixed maximum size is used. Samples are NOT simply scaled to match that size here (would result in stretching/squashing smaller/longer samples) how it is done in many other approaches. Zero-Padding is used instead."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y03h9Y0T_fFe",
        "colab_type": "text"
      },
      "source": [
        "First, determine maximum length and class count."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "meFY0v5a_Dju",
        "colab_type": "code",
        "outputId": "7568180f-d14a-4bf6-909e-72af9c9290be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "maxlen = 0\n",
        "for item in dataset_full:\n",
        "  arr = np.asarray(item[0])\n",
        "  itemlen = arr.shape[1]\n",
        "  if itemlen > maxlen:\n",
        "    maxlen = itemlen\n",
        "\n",
        "item0 = np.asarray(dataset_full[0][0])\n",
        "input_shape = (item0.shape[0], maxlen)\n",
        "print(\"Shape is \" + str(input_shape))\n",
        "\n",
        "n_classes = len(classes)\n",
        "print(str(n_classes) + \" target classes\")\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shape is (256, 174)\n",
            "10 target classes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_vEFtm-alsTb",
        "colab_type": "text"
      },
      "source": [
        "Now build the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true,
        "id": "EFvht0cRs6k_",
        "colab_type": "code",
        "outputId": "70b6f49d-ac04-4739-bb04-483657453662",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "dataset_full = list(dataset_full)\n",
        "\n",
        "dataset_train, dataset_test = train_test_split(dataset_full, train_size=0.9, test_size=0.1)\n",
        "\n",
        "def build_model():\n",
        "    # Use he_normal as initializer for CNNs whenever possible. Here's why -> https://towardsdatascience.com/why-default-cnn-are-broken-in-keras-and-how-to-fix-them-ce295e5e5f2\n",
        "    # Unfortunately, keras-tcn doesn't use that yet. However, that shouldn't be a big deal here.\n",
        "  \n",
        "    # See https://towardsdatascience.com/get-started-with-using-cnn-lstm-for-forecasting-6f0f4dde5826 how CNN-LSTM works.\n",
        "    # That can be adopted for CNN-TCN. Except here we're not using 1D-Conv Layers, but 2D feature extractors instead.\n",
        "    # (So 3x3 Kernels instead of slicing 1x3 Kernels)\n",
        "  \n",
        "    # Input_shape is (Features, TimeSteps)\n",
        "    # For our network, it has to be (TimeSteps, Features), so we change it.\n",
        "    input_layer = keras.layers.Input(shape=(input_shape[1], input_shape[0], 1))\n",
        "    \n",
        "    conv1 = keras.layers.Conv2D(filters=32, kernel_size=3, activation=\"relu\", kernel_initializer=\"he_normal\", name=\"conv1\", padding=\"same\")(input_layer)\n",
        "    # Stride of (1, 2) -> stride of 2 in feature dimension, reducing feature dimensionality per timestep\n",
        "    conv2 = keras.layers.Conv2D(filters=32, kernel_size=3, strides=(1,2), activation=\"relu\", kernel_initializer=\"he_normal\", padding=\"same\")(conv1)\n",
        "    \n",
        "    # Slicing 1D Flattening!\n",
        "    feature_distributor = keras.layers.TimeDistributed(keras.layers.Flatten())(conv2)\n",
        "    \n",
        "    # TCN Unit\n",
        "    tcn1 = tcn.TCN(return_sequences=False, kernel_size=(2), nb_filters=64, dilations=[1, 2, 4, 8, 16, 32, 64], nb_stacks=2,\n",
        "                   dropout_rate=0.00, name=\"tcn1\", padding=\"same\")(feature_distributor)\n",
        "    tcn1 = keras.layers.BatchNormalization()(tcn1)\n",
        " \n",
        "    output_layer = keras.layers.Dense(n_classes, activation=\"softmax\")(tcn1)\n",
        "\n",
        "    model = keras.Model(inputs=input_layer, outputs=output_layer)\n",
        "\n",
        "    optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)\n",
        "    model.compile(optimizer, loss=keras.losses.categorical_crossentropy, metrics=[\"categorical_accuracy\"])\n",
        "    return model\n",
        "\n",
        "model = build_model()\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_2 (InputLayer)            (None, 174, 256, 1)  0                                            \n",
            "__________________________________________________________________________________________________\n",
            "conv1 (Conv2D)                  (None, 174, 256, 32) 320         input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_2 (Conv2D)               (None, 174, 128, 32) 9248        conv1[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "time_distributed_2 (TimeDistrib (None, 174, 4096)    0           conv2d_2[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_44 (Conv1D)              (None, 174, 64)      262208      time_distributed_2[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_45 (Conv1D)              (None, 174, 64)      8256        conv1d_44[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "activation_43 (Activation)      (None, 174, 64)      0           conv1d_45[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_29 (SpatialDr (None, 174, 64)      0           activation_43[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_46 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_29[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_44 (Activation)      (None, 174, 64)      0           conv1d_46[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_30 (SpatialDr (None, 174, 64)      0           activation_44[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_47 (Conv1D)              (None, 174, 64)      4160        conv1d_44[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "add_16 (Add)                    (None, 174, 64)      0           conv1d_47[0][0]                  \n",
            "                                                                 spatial_dropout1d_30[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_45 (Activation)      (None, 174, 64)      0           add_16[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_48 (Conv1D)              (None, 174, 64)      8256        activation_45[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_46 (Activation)      (None, 174, 64)      0           conv1d_48[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_31 (SpatialDr (None, 174, 64)      0           activation_46[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_49 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_31[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_47 (Activation)      (None, 174, 64)      0           conv1d_49[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_32 (SpatialDr (None, 174, 64)      0           activation_47[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_50 (Conv1D)              (None, 174, 64)      4160        activation_45[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_17 (Add)                    (None, 174, 64)      0           conv1d_50[0][0]                  \n",
            "                                                                 spatial_dropout1d_32[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_48 (Activation)      (None, 174, 64)      0           add_17[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_51 (Conv1D)              (None, 174, 64)      8256        activation_48[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_49 (Activation)      (None, 174, 64)      0           conv1d_51[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_33 (SpatialDr (None, 174, 64)      0           activation_49[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_52 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_33[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_50 (Activation)      (None, 174, 64)      0           conv1d_52[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_34 (SpatialDr (None, 174, 64)      0           activation_50[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_53 (Conv1D)              (None, 174, 64)      4160        activation_48[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_18 (Add)                    (None, 174, 64)      0           conv1d_53[0][0]                  \n",
            "                                                                 spatial_dropout1d_34[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_51 (Activation)      (None, 174, 64)      0           add_18[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_54 (Conv1D)              (None, 174, 64)      8256        activation_51[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_52 (Activation)      (None, 174, 64)      0           conv1d_54[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_35 (SpatialDr (None, 174, 64)      0           activation_52[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_55 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_35[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_53 (Activation)      (None, 174, 64)      0           conv1d_55[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_36 (SpatialDr (None, 174, 64)      0           activation_53[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_56 (Conv1D)              (None, 174, 64)      4160        activation_51[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_19 (Add)                    (None, 174, 64)      0           conv1d_56[0][0]                  \n",
            "                                                                 spatial_dropout1d_36[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_54 (Activation)      (None, 174, 64)      0           add_19[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_57 (Conv1D)              (None, 174, 64)      8256        activation_54[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_55 (Activation)      (None, 174, 64)      0           conv1d_57[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_37 (SpatialDr (None, 174, 64)      0           activation_55[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_58 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_37[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_56 (Activation)      (None, 174, 64)      0           conv1d_58[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_38 (SpatialDr (None, 174, 64)      0           activation_56[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_59 (Conv1D)              (None, 174, 64)      4160        activation_54[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_20 (Add)                    (None, 174, 64)      0           conv1d_59[0][0]                  \n",
            "                                                                 spatial_dropout1d_38[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_57 (Activation)      (None, 174, 64)      0           add_20[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_60 (Conv1D)              (None, 174, 64)      8256        activation_57[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_58 (Activation)      (None, 174, 64)      0           conv1d_60[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_39 (SpatialDr (None, 174, 64)      0           activation_58[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_61 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_39[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_59 (Activation)      (None, 174, 64)      0           conv1d_61[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_40 (SpatialDr (None, 174, 64)      0           activation_59[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_62 (Conv1D)              (None, 174, 64)      4160        activation_57[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_21 (Add)                    (None, 174, 64)      0           conv1d_62[0][0]                  \n",
            "                                                                 spatial_dropout1d_40[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_60 (Activation)      (None, 174, 64)      0           add_21[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_63 (Conv1D)              (None, 174, 64)      8256        activation_60[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_61 (Activation)      (None, 174, 64)      0           conv1d_63[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_41 (SpatialDr (None, 174, 64)      0           activation_61[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_64 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_41[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_62 (Activation)      (None, 174, 64)      0           conv1d_64[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_42 (SpatialDr (None, 174, 64)      0           activation_62[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_65 (Conv1D)              (None, 174, 64)      4160        activation_60[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_22 (Add)                    (None, 174, 64)      0           conv1d_65[0][0]                  \n",
            "                                                                 spatial_dropout1d_42[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_63 (Activation)      (None, 174, 64)      0           add_22[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_66 (Conv1D)              (None, 174, 64)      8256        activation_63[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_64 (Activation)      (None, 174, 64)      0           conv1d_66[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_43 (SpatialDr (None, 174, 64)      0           activation_64[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_67 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_43[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_65 (Activation)      (None, 174, 64)      0           conv1d_67[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_44 (SpatialDr (None, 174, 64)      0           activation_65[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_68 (Conv1D)              (None, 174, 64)      4160        activation_63[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_23 (Add)                    (None, 174, 64)      0           conv1d_68[0][0]                  \n",
            "                                                                 spatial_dropout1d_44[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_66 (Activation)      (None, 174, 64)      0           add_23[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_69 (Conv1D)              (None, 174, 64)      8256        activation_66[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_67 (Activation)      (None, 174, 64)      0           conv1d_69[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_45 (SpatialDr (None, 174, 64)      0           activation_67[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_70 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_45[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_68 (Activation)      (None, 174, 64)      0           conv1d_70[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_46 (SpatialDr (None, 174, 64)      0           activation_68[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_71 (Conv1D)              (None, 174, 64)      4160        activation_66[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_24 (Add)                    (None, 174, 64)      0           conv1d_71[0][0]                  \n",
            "                                                                 spatial_dropout1d_46[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_69 (Activation)      (None, 174, 64)      0           add_24[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_72 (Conv1D)              (None, 174, 64)      8256        activation_69[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_70 (Activation)      (None, 174, 64)      0           conv1d_72[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_47 (SpatialDr (None, 174, 64)      0           activation_70[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_73 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_47[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_71 (Activation)      (None, 174, 64)      0           conv1d_73[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_48 (SpatialDr (None, 174, 64)      0           activation_71[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_74 (Conv1D)              (None, 174, 64)      4160        activation_69[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_25 (Add)                    (None, 174, 64)      0           conv1d_74[0][0]                  \n",
            "                                                                 spatial_dropout1d_48[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_72 (Activation)      (None, 174, 64)      0           add_25[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_75 (Conv1D)              (None, 174, 64)      8256        activation_72[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_73 (Activation)      (None, 174, 64)      0           conv1d_75[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_49 (SpatialDr (None, 174, 64)      0           activation_73[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_76 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_49[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_74 (Activation)      (None, 174, 64)      0           conv1d_76[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_50 (SpatialDr (None, 174, 64)      0           activation_74[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_77 (Conv1D)              (None, 174, 64)      4160        activation_72[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_26 (Add)                    (None, 174, 64)      0           conv1d_77[0][0]                  \n",
            "                                                                 spatial_dropout1d_50[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_75 (Activation)      (None, 174, 64)      0           add_26[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_78 (Conv1D)              (None, 174, 64)      8256        activation_75[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_76 (Activation)      (None, 174, 64)      0           conv1d_78[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_51 (SpatialDr (None, 174, 64)      0           activation_76[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_79 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_51[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_77 (Activation)      (None, 174, 64)      0           conv1d_79[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_52 (SpatialDr (None, 174, 64)      0           activation_77[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_80 (Conv1D)              (None, 174, 64)      4160        activation_75[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_27 (Add)                    (None, 174, 64)      0           conv1d_80[0][0]                  \n",
            "                                                                 spatial_dropout1d_52[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_78 (Activation)      (None, 174, 64)      0           add_27[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_81 (Conv1D)              (None, 174, 64)      8256        activation_78[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_79 (Activation)      (None, 174, 64)      0           conv1d_81[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_53 (SpatialDr (None, 174, 64)      0           activation_79[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_82 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_53[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_80 (Activation)      (None, 174, 64)      0           conv1d_82[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_54 (SpatialDr (None, 174, 64)      0           activation_80[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_83 (Conv1D)              (None, 174, 64)      4160        activation_78[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_28 (Add)                    (None, 174, 64)      0           conv1d_83[0][0]                  \n",
            "                                                                 spatial_dropout1d_54[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_81 (Activation)      (None, 174, 64)      0           add_28[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_84 (Conv1D)              (None, 174, 64)      8256        activation_81[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_82 (Activation)      (None, 174, 64)      0           conv1d_84[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_55 (SpatialDr (None, 174, 64)      0           activation_82[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_85 (Conv1D)              (None, 174, 64)      8256        spatial_dropout1d_55[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "activation_83 (Activation)      (None, 174, 64)      0           conv1d_85[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "spatial_dropout1d_56 (SpatialDr (None, 174, 64)      0           activation_83[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "add_30 (Add)                    (None, 174, 64)      0           spatial_dropout1d_30[0][0]       \n",
            "                                                                 spatial_dropout1d_32[0][0]       \n",
            "                                                                 spatial_dropout1d_34[0][0]       \n",
            "                                                                 spatial_dropout1d_36[0][0]       \n",
            "                                                                 spatial_dropout1d_38[0][0]       \n",
            "                                                                 spatial_dropout1d_40[0][0]       \n",
            "                                                                 spatial_dropout1d_42[0][0]       \n",
            "                                                                 spatial_dropout1d_44[0][0]       \n",
            "                                                                 spatial_dropout1d_46[0][0]       \n",
            "                                                                 spatial_dropout1d_48[0][0]       \n",
            "                                                                 spatial_dropout1d_50[0][0]       \n",
            "                                                                 spatial_dropout1d_52[0][0]       \n",
            "                                                                 spatial_dropout1d_54[0][0]       \n",
            "                                                                 spatial_dropout1d_56[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "lambda_2 (Lambda)               (None, 64)           0           add_30[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "batch_normalization_2 (BatchNor (None, 64)           256         lambda_2[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "dense_2 (Dense)                 (None, 10)           650         batch_normalization_2[0][0]      \n",
            "==================================================================================================\n",
            "Total params: 557,930\n",
            "Trainable params: 557,802\n",
            "Non-trainable params: 128\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kxvM-BRWePK7",
        "colab_type": "text"
      },
      "source": [
        "**Generator Functions and Training**\n",
        "\n",
        "An own generator function is needed since augmented batches should be created. Since all the samples have to be the same length inside a batch, zero padding is used there.\n",
        "Data Augmentation is performed live when creating the batch.\n",
        "\n",
        "Batches of *Test* data won't be augmented (wouldn't make sense). A *Model Checkpoint* is used to save model at the best epoch (and only that).\n",
        "\n",
        "To switch the SpecAugment implementation, just switch the lines \"x = spec_augment(_simple)(x)\"\n",
        "\n",
        "(Results can vary by some percentage, depending on train test split, random weight initialization and therefore maybe star constellation.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZejSVGjxHIO9",
        "colab_type": "code",
        "outputId": "2b86e787-ee9e-4058-cd2a-e9ca62493a28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "gc.collect()\n",
        "\n",
        "def generator(dataset, augment=False, debug=False):\n",
        "    n_batches = 0\n",
        "    current_batch_size = min(batch_size, len(dataset))\n",
        "    batch_x = np.zeros((current_batch_size, input_shape[0], input_shape[1], 1))\n",
        "    batch_y = np.zeros((current_batch_size, n_classes))\n",
        "    samples_in_batch = 0\n",
        "    while True:\n",
        "        samples_in_epoch = 0\n",
        "        epoch_order = list(np.random.permutation(len(dataset)))\n",
        "        current_batch_size = min(batch_size, len(epoch_order))\n",
        "          \n",
        "        for sample_id in epoch_order:\n",
        "            sample = dataset[sample_id]\n",
        "            x = sample[0]\n",
        "            x_zeropad = np.zeros((input_shape[0], input_shape[1]))\n",
        "            x_zeropad[:, :sample[0].shape[1]] = x\n",
        "            x = x_zeropad\n",
        "            if augment:\n",
        "                x = spec_augment_simple(x)  # simple, no time warping, but faster\n",
        "                # x = spec_augment_pytorch(mel_spectrogram=x)\n",
        "                # x = spec_augment_tensorflow(mel_spectrogram=x)  # seems broken + extreeemely slow (no gpu usage?), but includes Time Warping... :/\n",
        "            sample = (x, sample[1])\n",
        "            if debug:\n",
        "              print(current_batch_size)\n",
        "              print(samples_in_batch)\n",
        "              print(samples_in_epoch)\n",
        "            batch_x[samples_in_batch, :, :sample[0].shape[1], :] = np.expand_dims(sample[0], 2)\n",
        "            batch_y[samples_in_batch, :] = sample[1]\n",
        "            samples_in_batch += 1\n",
        "            if samples_in_batch == current_batch_size:\n",
        "                batch = (np.swapaxes(batch_x[0:current_batch_size, :, :], 1, 2), batch_y[0:current_batch_size, :])\n",
        "                yield batch\n",
        "                samples_in_epoch += samples_in_batch\n",
        "                samples_in_batch = 0\n",
        "                current_batch_size=min(batch_size, len(dataset) - epoch_order.index(sample_id))\n",
        "                batch_x = np.zeros((batch_size, input_shape[0], input_shape[1], 1))\n",
        "                batch_y = np.zeros((batch_size, n_classes))\n",
        "                \n",
        "                n_batches += 1\n",
        "                \n",
        "n_epochs = 300\n",
        "batch_size = 64\n",
        "train_gen = generator(dataset_train, augment=True, debug=False)\n",
        "test_gen = generator(dataset_test, debug=False)\n",
        "\n",
        "steps_per_epoch = int(len(dataset_train) / batch_size)\n",
        "validation_steps = ceil(len(dataset_test) / batch_size)\n",
        "\n",
        "max_lr = 0.15\n",
        "num_samples = len(dataset_train)\n",
        "\n",
        "callbacks = []\n",
        "model_checkpoint = keras.callbacks.ModelCheckpoint(\"model_best.h5\", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1)\n",
        "callbacks.append(model_checkpoint)\n",
        "\n",
        "\n",
        "model.fit_generator(train_gen, steps_per_epoch=steps_per_epoch, epochs=n_epochs, \\\n",
        "                   validation_data=test_gen, validation_steps=validation_steps, \\\n",
        "                    callbacks=callbacks, max_queue_size=64)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/300\n",
            "122/122 [==============================] - 48s 394ms/step - loss: 2.2203 - categorical_accuracy: 0.2145 - val_loss: 2.1918 - val_categorical_accuracy: 0.2254\n",
            "\n",
            "Epoch 00001: val_categorical_accuracy improved from -inf to 0.22545, saving model to model_best.h5\n",
            "Epoch 2/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 2.0586 - categorical_accuracy: 0.2561 - val_loss: 2.1166 - val_categorical_accuracy: 0.2913\n",
            "\n",
            "Epoch 00002: val_categorical_accuracy improved from 0.22545 to 0.29129, saving model to model_best.h5\n",
            "Epoch 3/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 1.9847 - categorical_accuracy: 0.2897 - val_loss: 1.9485 - val_categorical_accuracy: 0.3471\n",
            "\n",
            "Epoch 00003: val_categorical_accuracy improved from 0.29129 to 0.34710, saving model to model_best.h5\n",
            "Epoch 4/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 1.8428 - categorical_accuracy: 0.3477 - val_loss: 2.0035 - val_categorical_accuracy: 0.2723\n",
            "\n",
            "Epoch 00004: val_categorical_accuracy did not improve from 0.34710\n",
            "Epoch 5/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 1.6537 - categorical_accuracy: 0.4051 - val_loss: 1.5002 - val_categorical_accuracy: 0.4989\n",
            "\n",
            "Epoch 00005: val_categorical_accuracy improved from 0.34710 to 0.49888, saving model to model_best.h5\n",
            "Epoch 6/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 1.4974 - categorical_accuracy: 0.4647 - val_loss: 1.4828 - val_categorical_accuracy: 0.5301\n",
            "\n",
            "Epoch 00006: val_categorical_accuracy improved from 0.49888 to 0.53013, saving model to model_best.h5\n",
            "Epoch 7/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 1.3884 - categorical_accuracy: 0.5067 - val_loss: 1.2511 - val_categorical_accuracy: 0.5893\n",
            "\n",
            "Epoch 00007: val_categorical_accuracy improved from 0.53013 to 0.58929, saving model to model_best.h5\n",
            "Epoch 8/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 1.3180 - categorical_accuracy: 0.5339 - val_loss: 1.2084 - val_categorical_accuracy: 0.6228\n",
            "\n",
            "Epoch 00008: val_categorical_accuracy improved from 0.58929 to 0.62277, saving model to model_best.h5\n",
            "Epoch 9/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 1.1835 - categorical_accuracy: 0.5863 - val_loss: 1.1514 - val_categorical_accuracy: 0.6228\n",
            "\n",
            "Epoch 00009: val_categorical_accuracy did not improve from 0.62277\n",
            "Epoch 10/300\n",
            "122/122 [==============================] - 40s 330ms/step - loss: 1.0708 - categorical_accuracy: 0.6301 - val_loss: 1.2592 - val_categorical_accuracy: 0.5703\n",
            "\n",
            "Epoch 00010: val_categorical_accuracy did not improve from 0.62277\n",
            "Epoch 11/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.9582 - categorical_accuracy: 0.6698 - val_loss: 1.0833 - val_categorical_accuracy: 0.6864\n",
            "\n",
            "Epoch 00011: val_categorical_accuracy improved from 0.62277 to 0.68638, saving model to model_best.h5\n",
            "Epoch 12/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.8755 - categorical_accuracy: 0.7082 - val_loss: 0.9247 - val_categorical_accuracy: 0.7109\n",
            "\n",
            "Epoch 00012: val_categorical_accuracy improved from 0.68638 to 0.71094, saving model to model_best.h5\n",
            "Epoch 13/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.8219 - categorical_accuracy: 0.7309 - val_loss: 0.6571 - val_categorical_accuracy: 0.8025\n",
            "\n",
            "Epoch 00013: val_categorical_accuracy improved from 0.71094 to 0.80246, saving model to model_best.h5\n",
            "Epoch 14/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.7680 - categorical_accuracy: 0.7460 - val_loss: 0.8230 - val_categorical_accuracy: 0.7500\n",
            "\n",
            "Epoch 00014: val_categorical_accuracy did not improve from 0.80246\n",
            "Epoch 15/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.7373 - categorical_accuracy: 0.7522 - val_loss: 0.8525 - val_categorical_accuracy: 0.7522\n",
            "\n",
            "Epoch 00015: val_categorical_accuracy did not improve from 0.80246\n",
            "Epoch 16/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.6713 - categorical_accuracy: 0.7798 - val_loss: 0.5491 - val_categorical_accuracy: 0.8326\n",
            "\n",
            "Epoch 00016: val_categorical_accuracy improved from 0.80246 to 0.83259, saving model to model_best.h5\n",
            "Epoch 17/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.6296 - categorical_accuracy: 0.7966 - val_loss: 0.7987 - val_categorical_accuracy: 0.7857\n",
            "\n",
            "Epoch 00017: val_categorical_accuracy did not improve from 0.83259\n",
            "Epoch 18/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.6009 - categorical_accuracy: 0.8035 - val_loss: 0.5218 - val_categorical_accuracy: 0.8181\n",
            "\n",
            "Epoch 00018: val_categorical_accuracy did not improve from 0.83259\n",
            "Epoch 19/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.5831 - categorical_accuracy: 0.8105 - val_loss: 0.5951 - val_categorical_accuracy: 0.8237\n",
            "\n",
            "Epoch 00019: val_categorical_accuracy did not improve from 0.83259\n",
            "Epoch 20/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.5588 - categorical_accuracy: 0.8128 - val_loss: 0.6024 - val_categorical_accuracy: 0.8170\n",
            "\n",
            "Epoch 00020: val_categorical_accuracy did not improve from 0.83259\n",
            "Epoch 21/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.5096 - categorical_accuracy: 0.8356 - val_loss: 0.4941 - val_categorical_accuracy: 0.8471\n",
            "\n",
            "Epoch 00021: val_categorical_accuracy improved from 0.83259 to 0.84710, saving model to model_best.h5\n",
            "Epoch 22/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.4937 - categorical_accuracy: 0.8376 - val_loss: 0.4432 - val_categorical_accuracy: 0.8728\n",
            "\n",
            "Epoch 00022: val_categorical_accuracy improved from 0.84710 to 0.87277, saving model to model_best.h5\n",
            "Epoch 23/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.4850 - categorical_accuracy: 0.8436 - val_loss: 0.5546 - val_categorical_accuracy: 0.8449\n",
            "\n",
            "Epoch 00023: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 24/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.4599 - categorical_accuracy: 0.8550 - val_loss: 0.4313 - val_categorical_accuracy: 0.8717\n",
            "\n",
            "Epoch 00024: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 25/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.4394 - categorical_accuracy: 0.8623 - val_loss: 0.5007 - val_categorical_accuracy: 0.8616\n",
            "\n",
            "Epoch 00025: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 26/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.4047 - categorical_accuracy: 0.8736 - val_loss: 0.4705 - val_categorical_accuracy: 0.8717\n",
            "\n",
            "Epoch 00026: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 27/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.4113 - categorical_accuracy: 0.8672 - val_loss: 0.5092 - val_categorical_accuracy: 0.8426\n",
            "\n",
            "Epoch 00027: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 28/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.4152 - categorical_accuracy: 0.8665 - val_loss: 0.5652 - val_categorical_accuracy: 0.8549\n",
            "\n",
            "Epoch 00028: val_categorical_accuracy did not improve from 0.87277\n",
            "Epoch 29/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.3604 - categorical_accuracy: 0.8841 - val_loss: 0.3895 - val_categorical_accuracy: 0.8761\n",
            "\n",
            "Epoch 00029: val_categorical_accuracy improved from 0.87277 to 0.87612, saving model to model_best.h5\n",
            "Epoch 30/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.3738 - categorical_accuracy: 0.8806 - val_loss: 0.3496 - val_categorical_accuracy: 0.9040\n",
            "\n",
            "Epoch 00030: val_categorical_accuracy improved from 0.87612 to 0.90402, saving model to model_best.h5\n",
            "Epoch 31/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.3652 - categorical_accuracy: 0.8879 - val_loss: 0.3984 - val_categorical_accuracy: 0.8884\n",
            "\n",
            "Epoch 00031: val_categorical_accuracy did not improve from 0.90402\n",
            "Epoch 32/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.3901 - categorical_accuracy: 0.8776 - val_loss: 0.3616 - val_categorical_accuracy: 0.9074\n",
            "\n",
            "Epoch 00032: val_categorical_accuracy improved from 0.90402 to 0.90737, saving model to model_best.h5\n",
            "Epoch 33/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.3299 - categorical_accuracy: 0.8942 - val_loss: 0.3213 - val_categorical_accuracy: 0.9074\n",
            "\n",
            "Epoch 00033: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 34/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.3504 - categorical_accuracy: 0.8922 - val_loss: 0.4834 - val_categorical_accuracy: 0.8627\n",
            "\n",
            "Epoch 00034: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 35/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.3311 - categorical_accuracy: 0.8997 - val_loss: 0.2690 - val_categorical_accuracy: 0.9029\n",
            "\n",
            "Epoch 00035: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 36/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.3252 - categorical_accuracy: 0.8978 - val_loss: 0.3589 - val_categorical_accuracy: 0.9062\n",
            "\n",
            "Epoch 00036: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 37/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.3167 - categorical_accuracy: 0.9014 - val_loss: 0.5197 - val_categorical_accuracy: 0.8661\n",
            "\n",
            "Epoch 00037: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 38/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.3023 - categorical_accuracy: 0.9036 - val_loss: 0.3215 - val_categorical_accuracy: 0.9051\n",
            "\n",
            "Epoch 00038: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 39/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.2852 - categorical_accuracy: 0.9086 - val_loss: 0.3628 - val_categorical_accuracy: 0.8839\n",
            "\n",
            "Epoch 00039: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 40/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.2842 - categorical_accuracy: 0.9116 - val_loss: 0.3440 - val_categorical_accuracy: 0.9051\n",
            "\n",
            "Epoch 00040: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 41/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.2595 - categorical_accuracy: 0.9180 - val_loss: 0.3757 - val_categorical_accuracy: 0.8962\n",
            "\n",
            "Epoch 00041: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 42/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.2933 - categorical_accuracy: 0.9048 - val_loss: 0.3570 - val_categorical_accuracy: 0.9007\n",
            "\n",
            "Epoch 00042: val_categorical_accuracy did not improve from 0.90737\n",
            "Epoch 43/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.2660 - categorical_accuracy: 0.9179 - val_loss: 0.3232 - val_categorical_accuracy: 0.9196\n",
            "\n",
            "Epoch 00043: val_categorical_accuracy improved from 0.90737 to 0.91964, saving model to model_best.h5\n",
            "Epoch 44/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.2508 - categorical_accuracy: 0.9228 - val_loss: 0.3385 - val_categorical_accuracy: 0.8973\n",
            "\n",
            "Epoch 00044: val_categorical_accuracy did not improve from 0.91964\n",
            "Epoch 45/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.2986 - categorical_accuracy: 0.9096 - val_loss: 0.3570 - val_categorical_accuracy: 0.9196\n",
            "\n",
            "Epoch 00045: val_categorical_accuracy did not improve from 0.91964\n",
            "Epoch 46/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.2551 - categorical_accuracy: 0.9202 - val_loss: 0.3175 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00046: val_categorical_accuracy improved from 0.91964 to 0.92522, saving model to model_best.h5\n",
            "Epoch 47/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.2451 - categorical_accuracy: 0.9243 - val_loss: 0.3323 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00047: val_categorical_accuracy did not improve from 0.92522\n",
            "Epoch 48/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.2511 - categorical_accuracy: 0.9168 - val_loss: 0.3065 - val_categorical_accuracy: 0.9185\n",
            "\n",
            "Epoch 00048: val_categorical_accuracy did not improve from 0.92522\n",
            "Epoch 49/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.2596 - categorical_accuracy: 0.9182 - val_loss: 0.4043 - val_categorical_accuracy: 0.9040\n",
            "\n",
            "Epoch 00049: val_categorical_accuracy did not improve from 0.92522\n",
            "Epoch 50/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.2505 - categorical_accuracy: 0.9246 - val_loss: 0.2892 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00050: val_categorical_accuracy improved from 0.92522 to 0.94085, saving model to model_best.h5\n",
            "Epoch 51/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.2363 - categorical_accuracy: 0.9255 - val_loss: 0.2689 - val_categorical_accuracy: 0.9286\n",
            "\n",
            "Epoch 00051: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 52/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.2166 - categorical_accuracy: 0.9337 - val_loss: 0.3087 - val_categorical_accuracy: 0.9141\n",
            "\n",
            "Epoch 00052: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 53/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.2197 - categorical_accuracy: 0.9328 - val_loss: 0.4198 - val_categorical_accuracy: 0.9096\n",
            "\n",
            "Epoch 00053: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 54/300\n",
            "122/122 [==============================] - 40s 331ms/step - loss: 0.2378 - categorical_accuracy: 0.9273 - val_loss: 0.3753 - val_categorical_accuracy: 0.9208\n",
            "\n",
            "Epoch 00054: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 55/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.2132 - categorical_accuracy: 0.9351 - val_loss: 0.3608 - val_categorical_accuracy: 0.9040\n",
            "\n",
            "Epoch 00055: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 56/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1907 - categorical_accuracy: 0.9413 - val_loss: 0.3581 - val_categorical_accuracy: 0.9174\n",
            "\n",
            "Epoch 00056: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 57/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.2063 - categorical_accuracy: 0.9330 - val_loss: 0.3235 - val_categorical_accuracy: 0.9196\n",
            "\n",
            "Epoch 00057: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 58/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.2075 - categorical_accuracy: 0.9367 - val_loss: 0.2835 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00058: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 59/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.2201 - categorical_accuracy: 0.9328 - val_loss: 0.2710 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00059: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 60/300\n",
            "122/122 [==============================] - 40s 330ms/step - loss: 0.1892 - categorical_accuracy: 0.9387 - val_loss: 0.3240 - val_categorical_accuracy: 0.9152\n",
            "\n",
            "Epoch 00060: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 61/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1868 - categorical_accuracy: 0.9399 - val_loss: 0.3073 - val_categorical_accuracy: 0.9118\n",
            "\n",
            "Epoch 00061: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 62/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.2040 - categorical_accuracy: 0.9352 - val_loss: 0.3139 - val_categorical_accuracy: 0.9074\n",
            "\n",
            "Epoch 00062: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 63/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1872 - categorical_accuracy: 0.9410 - val_loss: 0.2767 - val_categorical_accuracy: 0.9241\n",
            "\n",
            "Epoch 00063: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 64/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1898 - categorical_accuracy: 0.9424 - val_loss: 0.4820 - val_categorical_accuracy: 0.8750\n",
            "\n",
            "Epoch 00064: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 65/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1931 - categorical_accuracy: 0.9403 - val_loss: 0.3218 - val_categorical_accuracy: 0.9263\n",
            "\n",
            "Epoch 00065: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 66/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.2100 - categorical_accuracy: 0.9355 - val_loss: 0.3110 - val_categorical_accuracy: 0.9152\n",
            "\n",
            "Epoch 00066: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 67/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1930 - categorical_accuracy: 0.9389 - val_loss: 0.7179 - val_categorical_accuracy: 0.8359\n",
            "\n",
            "Epoch 00067: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 68/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1994 - categorical_accuracy: 0.9395 - val_loss: 0.3038 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00068: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 69/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1913 - categorical_accuracy: 0.9415 - val_loss: 0.2452 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00069: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 70/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1679 - categorical_accuracy: 0.9462 - val_loss: 0.4054 - val_categorical_accuracy: 0.9096\n",
            "\n",
            "Epoch 00070: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 71/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.2034 - categorical_accuracy: 0.9404 - val_loss: 0.4085 - val_categorical_accuracy: 0.8895\n",
            "\n",
            "Epoch 00071: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 72/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.2033 - categorical_accuracy: 0.9388 - val_loss: 0.2777 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00072: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 73/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1781 - categorical_accuracy: 0.9451 - val_loss: 0.2214 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00073: val_categorical_accuracy did not improve from 0.94085\n",
            "Epoch 74/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1725 - categorical_accuracy: 0.9447 - val_loss: 0.2166 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00074: val_categorical_accuracy improved from 0.94085 to 0.94308, saving model to model_best.h5\n",
            "Epoch 75/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1618 - categorical_accuracy: 0.9503 - val_loss: 0.2306 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00075: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 76/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1594 - categorical_accuracy: 0.9494 - val_loss: 0.2647 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00076: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 77/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1532 - categorical_accuracy: 0.9540 - val_loss: 0.2675 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00077: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 78/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1537 - categorical_accuracy: 0.9534 - val_loss: 0.2460 - val_categorical_accuracy: 0.9241\n",
            "\n",
            "Epoch 00078: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 79/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1595 - categorical_accuracy: 0.9503 - val_loss: 0.2236 - val_categorical_accuracy: 0.9319\n",
            "\n",
            "Epoch 00079: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 80/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1544 - categorical_accuracy: 0.9534 - val_loss: 0.3072 - val_categorical_accuracy: 0.9275\n",
            "\n",
            "Epoch 00080: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 81/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1574 - categorical_accuracy: 0.9501 - val_loss: 0.2831 - val_categorical_accuracy: 0.9275\n",
            "\n",
            "Epoch 00081: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 82/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1606 - categorical_accuracy: 0.9516 - val_loss: 0.3474 - val_categorical_accuracy: 0.9241\n",
            "\n",
            "Epoch 00082: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 83/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1658 - categorical_accuracy: 0.9490 - val_loss: 0.2375 - val_categorical_accuracy: 0.9319\n",
            "\n",
            "Epoch 00083: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 84/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1745 - categorical_accuracy: 0.9462 - val_loss: 0.3180 - val_categorical_accuracy: 0.9241\n",
            "\n",
            "Epoch 00084: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 85/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1594 - categorical_accuracy: 0.9545 - val_loss: 0.3763 - val_categorical_accuracy: 0.9129\n",
            "\n",
            "Epoch 00085: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 86/300\n",
            "122/122 [==============================] - 40s 330ms/step - loss: 0.1756 - categorical_accuracy: 0.9470 - val_loss: 0.3673 - val_categorical_accuracy: 0.9085\n",
            "\n",
            "Epoch 00086: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 87/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1695 - categorical_accuracy: 0.9477 - val_loss: 0.2050 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00087: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 88/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1643 - categorical_accuracy: 0.9522 - val_loss: 0.2383 - val_categorical_accuracy: 0.9308\n",
            "\n",
            "Epoch 00088: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 89/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1504 - categorical_accuracy: 0.9531 - val_loss: 0.2794 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00089: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 90/300\n",
            "122/122 [==============================] - 40s 330ms/step - loss: 0.1292 - categorical_accuracy: 0.9603 - val_loss: 0.3064 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00090: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 91/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1501 - categorical_accuracy: 0.9563 - val_loss: 0.2557 - val_categorical_accuracy: 0.9286\n",
            "\n",
            "Epoch 00091: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 92/300\n",
            "122/122 [==============================] - 40s 330ms/step - loss: 0.1373 - categorical_accuracy: 0.9591 - val_loss: 0.2241 - val_categorical_accuracy: 0.9275\n",
            "\n",
            "Epoch 00092: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 93/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 0.1408 - categorical_accuracy: 0.9572 - val_loss: 0.2422 - val_categorical_accuracy: 0.9386\n",
            "\n",
            "Epoch 00093: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 94/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1717 - categorical_accuracy: 0.9468 - val_loss: 0.2503 - val_categorical_accuracy: 0.9375\n",
            "\n",
            "Epoch 00094: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 95/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1368 - categorical_accuracy: 0.9585 - val_loss: 0.2300 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00095: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 96/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1445 - categorical_accuracy: 0.9566 - val_loss: 0.3873 - val_categorical_accuracy: 0.9062\n",
            "\n",
            "Epoch 00096: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 97/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1547 - categorical_accuracy: 0.9507 - val_loss: 0.2579 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00097: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 98/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1273 - categorical_accuracy: 0.9612 - val_loss: 0.3989 - val_categorical_accuracy: 0.9152\n",
            "\n",
            "Epoch 00098: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 99/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1499 - categorical_accuracy: 0.9541 - val_loss: 0.1995 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00099: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 100/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1244 - categorical_accuracy: 0.9611 - val_loss: 0.2135 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00100: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 101/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1436 - categorical_accuracy: 0.9584 - val_loss: 0.2309 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00101: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 102/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1314 - categorical_accuracy: 0.9612 - val_loss: 0.2408 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00102: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 103/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1495 - categorical_accuracy: 0.9548 - val_loss: 0.2297 - val_categorical_accuracy: 0.9319\n",
            "\n",
            "Epoch 00103: val_categorical_accuracy did not improve from 0.94308\n",
            "Epoch 104/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1489 - categorical_accuracy: 0.9550 - val_loss: 0.2120 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00104: val_categorical_accuracy improved from 0.94308 to 0.94866, saving model to model_best.h5\n",
            "Epoch 105/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1300 - categorical_accuracy: 0.9602 - val_loss: 0.3046 - val_categorical_accuracy: 0.9275\n",
            "\n",
            "Epoch 00105: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 106/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1346 - categorical_accuracy: 0.9593 - val_loss: 0.2569 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00106: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 107/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1164 - categorical_accuracy: 0.9661 - val_loss: 0.2557 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00107: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 108/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1107 - categorical_accuracy: 0.9673 - val_loss: 0.2524 - val_categorical_accuracy: 0.9263\n",
            "\n",
            "Epoch 00108: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 109/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1113 - categorical_accuracy: 0.9662 - val_loss: 0.2439 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00109: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 110/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1309 - categorical_accuracy: 0.9593 - val_loss: 0.2138 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00110: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 111/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1388 - categorical_accuracy: 0.9576 - val_loss: 0.1995 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00111: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 112/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1250 - categorical_accuracy: 0.9622 - val_loss: 0.2958 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00112: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 113/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1066 - categorical_accuracy: 0.9649 - val_loss: 0.2137 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00113: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 114/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1211 - categorical_accuracy: 0.9616 - val_loss: 0.3210 - val_categorical_accuracy: 0.9174\n",
            "\n",
            "Epoch 00114: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 115/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1299 - categorical_accuracy: 0.9629 - val_loss: 0.2262 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00115: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 116/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1188 - categorical_accuracy: 0.9643 - val_loss: 0.2633 - val_categorical_accuracy: 0.9386\n",
            "\n",
            "Epoch 00116: val_categorical_accuracy did not improve from 0.94866\n",
            "Epoch 117/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1289 - categorical_accuracy: 0.9622 - val_loss: 0.2373 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00117: val_categorical_accuracy improved from 0.94866 to 0.95089, saving model to model_best.h5\n",
            "Epoch 118/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1153 - categorical_accuracy: 0.9632 - val_loss: 0.2350 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00118: val_categorical_accuracy did not improve from 0.95089\n",
            "Epoch 119/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1198 - categorical_accuracy: 0.9626 - val_loss: 0.2217 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00119: val_categorical_accuracy did not improve from 0.95089\n",
            "Epoch 120/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1098 - categorical_accuracy: 0.9671 - val_loss: 0.1870 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00120: val_categorical_accuracy did not improve from 0.95089\n",
            "Epoch 121/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1049 - categorical_accuracy: 0.9688 - val_loss: 0.2261 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00121: val_categorical_accuracy did not improve from 0.95089\n",
            "Epoch 122/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1324 - categorical_accuracy: 0.9621 - val_loss: 0.2008 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00122: val_categorical_accuracy did not improve from 0.95089\n",
            "Epoch 123/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1157 - categorical_accuracy: 0.9644 - val_loss: 0.1783 - val_categorical_accuracy: 0.9531\n",
            "\n",
            "Epoch 00123: val_categorical_accuracy improved from 0.95089 to 0.95312, saving model to model_best.h5\n",
            "Epoch 124/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0981 - categorical_accuracy: 0.9690 - val_loss: 0.2802 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00124: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 125/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1169 - categorical_accuracy: 0.9655 - val_loss: 0.2318 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00125: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 126/300\n",
            "122/122 [==============================] - 39s 318ms/step - loss: 0.1277 - categorical_accuracy: 0.9629 - val_loss: 0.4242 - val_categorical_accuracy: 0.8906\n",
            "\n",
            "Epoch 00126: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 127/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1176 - categorical_accuracy: 0.9657 - val_loss: 0.2242 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00127: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 128/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.1066 - categorical_accuracy: 0.9681 - val_loss: 0.2607 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00128: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 129/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.1255 - categorical_accuracy: 0.9606 - val_loss: 0.2512 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00129: val_categorical_accuracy did not improve from 0.95312\n",
            "Epoch 130/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1129 - categorical_accuracy: 0.9672 - val_loss: 0.1786 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00130: val_categorical_accuracy improved from 0.95312 to 0.95424, saving model to model_best.h5\n",
            "Epoch 131/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1038 - categorical_accuracy: 0.9695 - val_loss: 0.3445 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00131: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 132/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.1144 - categorical_accuracy: 0.9635 - val_loss: 0.1864 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00132: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 133/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0821 - categorical_accuracy: 0.9746 - val_loss: 0.2193 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00133: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 134/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0989 - categorical_accuracy: 0.9722 - val_loss: 0.3214 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00134: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 135/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0966 - categorical_accuracy: 0.9694 - val_loss: 0.2080 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00135: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 136/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1035 - categorical_accuracy: 0.9681 - val_loss: 0.1965 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00136: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 137/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0838 - categorical_accuracy: 0.9741 - val_loss: 0.2726 - val_categorical_accuracy: 0.9230\n",
            "\n",
            "Epoch 00137: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 138/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1003 - categorical_accuracy: 0.9708 - val_loss: 0.2276 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00138: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 139/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1114 - categorical_accuracy: 0.9658 - val_loss: 0.3032 - val_categorical_accuracy: 0.9275\n",
            "\n",
            "Epoch 00139: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 140/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.1024 - categorical_accuracy: 0.9694 - val_loss: 0.2156 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00140: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 141/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0881 - categorical_accuracy: 0.9718 - val_loss: 0.2544 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00141: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 142/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1054 - categorical_accuracy: 0.9680 - val_loss: 0.2302 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00142: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 143/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.0983 - categorical_accuracy: 0.9698 - val_loss: 0.2309 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00143: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 144/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.1144 - categorical_accuracy: 0.9662 - val_loss: 0.2619 - val_categorical_accuracy: 0.9375\n",
            "\n",
            "Epoch 00144: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 145/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0966 - categorical_accuracy: 0.9711 - val_loss: 0.2485 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00145: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 146/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1049 - categorical_accuracy: 0.9673 - val_loss: 0.2932 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00146: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 147/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1059 - categorical_accuracy: 0.9700 - val_loss: 0.2547 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00147: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 148/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0891 - categorical_accuracy: 0.9731 - val_loss: 0.2649 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00148: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 149/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1003 - categorical_accuracy: 0.9708 - val_loss: 0.2071 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00149: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 150/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.0918 - categorical_accuracy: 0.9728 - val_loss: 0.1764 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00150: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 151/300\n",
            "122/122 [==============================] - 39s 318ms/step - loss: 0.1001 - categorical_accuracy: 0.9700 - val_loss: 0.2134 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00151: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 152/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0949 - categorical_accuracy: 0.9734 - val_loss: 0.2528 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00152: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 153/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0915 - categorical_accuracy: 0.9714 - val_loss: 0.2707 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00153: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 154/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.1033 - categorical_accuracy: 0.9693 - val_loss: 0.2231 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00154: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 155/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0865 - categorical_accuracy: 0.9753 - val_loss: 0.2011 - val_categorical_accuracy: 0.9252\n",
            "\n",
            "Epoch 00155: val_categorical_accuracy did not improve from 0.95424\n",
            "Epoch 156/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.1049 - categorical_accuracy: 0.9673 - val_loss: 0.1817 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00156: val_categorical_accuracy improved from 0.95424 to 0.95871, saving model to model_best.h5\n",
            "Epoch 157/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0866 - categorical_accuracy: 0.9757 - val_loss: 0.2068 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00157: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 158/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1073 - categorical_accuracy: 0.9700 - val_loss: 0.2284 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00158: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 159/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0890 - categorical_accuracy: 0.9725 - val_loss: 0.2132 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00159: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 160/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0917 - categorical_accuracy: 0.9705 - val_loss: 0.2090 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00160: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 161/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0892 - categorical_accuracy: 0.9703 - val_loss: 0.1985 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00161: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 162/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0925 - categorical_accuracy: 0.9691 - val_loss: 0.2900 - val_categorical_accuracy: 0.9308\n",
            "\n",
            "Epoch 00162: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 163/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1150 - categorical_accuracy: 0.9672 - val_loss: 0.2409 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00163: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 164/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0854 - categorical_accuracy: 0.9736 - val_loss: 0.2071 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00164: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 165/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0866 - categorical_accuracy: 0.9758 - val_loss: 0.2010 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00165: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 166/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0952 - categorical_accuracy: 0.9698 - val_loss: 0.2101 - val_categorical_accuracy: 0.9565\n",
            "\n",
            "Epoch 00166: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 167/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0974 - categorical_accuracy: 0.9699 - val_loss: 0.1864 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00167: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 168/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0913 - categorical_accuracy: 0.9728 - val_loss: 0.1796 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00168: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 169/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0966 - categorical_accuracy: 0.9702 - val_loss: 0.2500 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00169: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 170/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0936 - categorical_accuracy: 0.9718 - val_loss: 0.1962 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00170: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 171/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0999 - categorical_accuracy: 0.9718 - val_loss: 0.3147 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00171: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 172/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.1014 - categorical_accuracy: 0.9699 - val_loss: 0.2557 - val_categorical_accuracy: 0.9386\n",
            "\n",
            "Epoch 00172: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 173/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0870 - categorical_accuracy: 0.9718 - val_loss: 0.2194 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00173: val_categorical_accuracy did not improve from 0.95871\n",
            "Epoch 174/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0811 - categorical_accuracy: 0.9766 - val_loss: 0.1314 - val_categorical_accuracy: 0.9643\n",
            "\n",
            "Epoch 00174: val_categorical_accuracy improved from 0.95871 to 0.96429, saving model to model_best.h5\n",
            "Epoch 175/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0944 - categorical_accuracy: 0.9717 - val_loss: 0.2286 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00175: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 176/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0925 - categorical_accuracy: 0.9727 - val_loss: 0.2548 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00176: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 177/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.1088 - categorical_accuracy: 0.9673 - val_loss: 0.2455 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00177: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 178/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0849 - categorical_accuracy: 0.9763 - val_loss: 0.1959 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00178: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 179/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0842 - categorical_accuracy: 0.9744 - val_loss: 0.1833 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00179: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 180/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0886 - categorical_accuracy: 0.9736 - val_loss: 0.1994 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00180: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 181/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0827 - categorical_accuracy: 0.9761 - val_loss: 0.1809 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00181: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 182/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0987 - categorical_accuracy: 0.9718 - val_loss: 0.2701 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00182: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 183/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0895 - categorical_accuracy: 0.9744 - val_loss: 0.2265 - val_categorical_accuracy: 0.9565\n",
            "\n",
            "Epoch 00183: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 184/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.1019 - categorical_accuracy: 0.9699 - val_loss: 0.2131 - val_categorical_accuracy: 0.9364\n",
            "\n",
            "Epoch 00184: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 185/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0797 - categorical_accuracy: 0.9768 - val_loss: 0.2235 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00185: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 186/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0893 - categorical_accuracy: 0.9764 - val_loss: 0.2007 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00186: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 187/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0911 - categorical_accuracy: 0.9709 - val_loss: 0.2420 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00187: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 188/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0775 - categorical_accuracy: 0.9750 - val_loss: 0.1681 - val_categorical_accuracy: 0.9565\n",
            "\n",
            "Epoch 00188: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 189/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0794 - categorical_accuracy: 0.9776 - val_loss: 0.2161 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00189: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 190/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0930 - categorical_accuracy: 0.9726 - val_loss: 0.1885 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00190: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 191/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0946 - categorical_accuracy: 0.9704 - val_loss: 0.1556 - val_categorical_accuracy: 0.9598\n",
            "\n",
            "Epoch 00191: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 192/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 0.0800 - categorical_accuracy: 0.9746 - val_loss: 0.1824 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00192: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 193/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0666 - categorical_accuracy: 0.9796 - val_loss: 0.2760 - val_categorical_accuracy: 0.9241\n",
            "\n",
            "Epoch 00193: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 194/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0916 - categorical_accuracy: 0.9721 - val_loss: 0.2311 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00194: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 195/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0750 - categorical_accuracy: 0.9773 - val_loss: 0.2048 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00195: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 196/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0834 - categorical_accuracy: 0.9748 - val_loss: 0.1601 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00196: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 197/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0777 - categorical_accuracy: 0.9764 - val_loss: 0.3432 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00197: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 198/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0865 - categorical_accuracy: 0.9766 - val_loss: 0.1862 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00198: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 199/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0956 - categorical_accuracy: 0.9734 - val_loss: 0.2535 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00199: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 200/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0821 - categorical_accuracy: 0.9754 - val_loss: 0.2029 - val_categorical_accuracy: 0.9520\n",
            "\n",
            "Epoch 00200: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 201/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0854 - categorical_accuracy: 0.9755 - val_loss: 0.1925 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00201: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 202/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0715 - categorical_accuracy: 0.9784 - val_loss: 0.2669 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00202: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 203/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0834 - categorical_accuracy: 0.9768 - val_loss: 0.1763 - val_categorical_accuracy: 0.9520\n",
            "\n",
            "Epoch 00203: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 204/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0857 - categorical_accuracy: 0.9737 - val_loss: 0.3893 - val_categorical_accuracy: 0.9174\n",
            "\n",
            "Epoch 00204: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 205/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0727 - categorical_accuracy: 0.9775 - val_loss: 0.2432 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00205: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 206/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0669 - categorical_accuracy: 0.9807 - val_loss: 0.2297 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00206: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 207/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0691 - categorical_accuracy: 0.9789 - val_loss: 0.2056 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00207: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 208/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0889 - categorical_accuracy: 0.9745 - val_loss: 0.1929 - val_categorical_accuracy: 0.9531\n",
            "\n",
            "Epoch 00208: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 209/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0680 - categorical_accuracy: 0.9794 - val_loss: 0.2519 - val_categorical_accuracy: 0.9386\n",
            "\n",
            "Epoch 00209: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 210/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0766 - categorical_accuracy: 0.9763 - val_loss: 0.2242 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00210: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 211/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0932 - categorical_accuracy: 0.9721 - val_loss: 0.1557 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00211: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 212/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0840 - categorical_accuracy: 0.9775 - val_loss: 0.2001 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00212: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 213/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0697 - categorical_accuracy: 0.9805 - val_loss: 0.2003 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00213: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 214/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0771 - categorical_accuracy: 0.9768 - val_loss: 0.2033 - val_categorical_accuracy: 0.9353\n",
            "\n",
            "Epoch 00214: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 215/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0755 - categorical_accuracy: 0.9791 - val_loss: 0.1740 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00215: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 216/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0787 - categorical_accuracy: 0.9772 - val_loss: 0.2520 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00216: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 217/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0760 - categorical_accuracy: 0.9782 - val_loss: 0.1573 - val_categorical_accuracy: 0.9643\n",
            "\n",
            "Epoch 00217: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 218/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0727 - categorical_accuracy: 0.9787 - val_loss: 0.1990 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00218: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 219/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0927 - categorical_accuracy: 0.9743 - val_loss: 0.2720 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00219: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 220/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0744 - categorical_accuracy: 0.9768 - val_loss: 0.2622 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00220: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 221/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0691 - categorical_accuracy: 0.9794 - val_loss: 0.2062 - val_categorical_accuracy: 0.9609\n",
            "\n",
            "Epoch 00221: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 222/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0747 - categorical_accuracy: 0.9773 - val_loss: 0.3075 - val_categorical_accuracy: 0.9330\n",
            "\n",
            "Epoch 00222: val_categorical_accuracy did not improve from 0.96429\n",
            "Epoch 223/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0631 - categorical_accuracy: 0.9821 - val_loss: 0.1547 - val_categorical_accuracy: 0.9654\n",
            "\n",
            "Epoch 00223: val_categorical_accuracy improved from 0.96429 to 0.96540, saving model to model_best.h5\n",
            "Epoch 224/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0577 - categorical_accuracy: 0.9844 - val_loss: 0.1627 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00224: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 225/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0638 - categorical_accuracy: 0.9813 - val_loss: 0.1739 - val_categorical_accuracy: 0.9621\n",
            "\n",
            "Epoch 00225: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 226/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0624 - categorical_accuracy: 0.9804 - val_loss: 0.2055 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00226: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 227/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.1132 - categorical_accuracy: 0.9691 - val_loss: 0.3209 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00227: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 228/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0880 - categorical_accuracy: 0.9746 - val_loss: 0.2181 - val_categorical_accuracy: 0.9576\n",
            "\n",
            "Epoch 00228: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 229/300\n",
            "122/122 [==============================] - 40s 328ms/step - loss: 0.0781 - categorical_accuracy: 0.9767 - val_loss: 0.1864 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00229: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 230/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0870 - categorical_accuracy: 0.9730 - val_loss: 0.2296 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00230: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 231/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0846 - categorical_accuracy: 0.9730 - val_loss: 0.2157 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00231: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 232/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0629 - categorical_accuracy: 0.9798 - val_loss: 0.2150 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00232: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 233/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0605 - categorical_accuracy: 0.9818 - val_loss: 0.2143 - val_categorical_accuracy: 0.9531\n",
            "\n",
            "Epoch 00233: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 234/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0698 - categorical_accuracy: 0.9785 - val_loss: 0.2604 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00234: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 235/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0654 - categorical_accuracy: 0.9816 - val_loss: 0.1862 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00235: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 236/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0611 - categorical_accuracy: 0.9836 - val_loss: 0.2440 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00236: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 237/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0578 - categorical_accuracy: 0.9822 - val_loss: 0.2451 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00237: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 238/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0641 - categorical_accuracy: 0.9816 - val_loss: 0.2264 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00238: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 239/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0579 - categorical_accuracy: 0.9822 - val_loss: 0.3148 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00239: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 240/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0786 - categorical_accuracy: 0.9768 - val_loss: 0.2705 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00240: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 241/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0810 - categorical_accuracy: 0.9776 - val_loss: 0.2729 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00241: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 242/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0676 - categorical_accuracy: 0.9796 - val_loss: 0.2040 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00242: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 243/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0774 - categorical_accuracy: 0.9795 - val_loss: 0.2503 - val_categorical_accuracy: 0.9576\n",
            "\n",
            "Epoch 00243: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 244/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0732 - categorical_accuracy: 0.9803 - val_loss: 0.2013 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00244: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 245/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0642 - categorical_accuracy: 0.9816 - val_loss: 0.3127 - val_categorical_accuracy: 0.9297\n",
            "\n",
            "Epoch 00245: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 246/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0669 - categorical_accuracy: 0.9810 - val_loss: 0.2091 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00246: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 247/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0666 - categorical_accuracy: 0.9807 - val_loss: 0.1836 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00247: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 248/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0701 - categorical_accuracy: 0.9799 - val_loss: 0.2517 - val_categorical_accuracy: 0.9453\n",
            "\n",
            "Epoch 00248: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 249/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0674 - categorical_accuracy: 0.9800 - val_loss: 0.3484 - val_categorical_accuracy: 0.9230\n",
            "\n",
            "Epoch 00249: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 250/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0920 - categorical_accuracy: 0.9739 - val_loss: 0.2909 - val_categorical_accuracy: 0.9442\n",
            "\n",
            "Epoch 00250: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 251/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 0.0790 - categorical_accuracy: 0.9777 - val_loss: 0.2135 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00251: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 252/300\n",
            "122/122 [==============================] - 40s 331ms/step - loss: 0.0599 - categorical_accuracy: 0.9810 - val_loss: 0.1994 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00252: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 253/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0665 - categorical_accuracy: 0.9795 - val_loss: 0.1638 - val_categorical_accuracy: 0.9609\n",
            "\n",
            "Epoch 00253: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 254/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0717 - categorical_accuracy: 0.9784 - val_loss: 0.2474 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00254: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 255/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0666 - categorical_accuracy: 0.9798 - val_loss: 0.2927 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00255: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 256/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0683 - categorical_accuracy: 0.9804 - val_loss: 0.2497 - val_categorical_accuracy: 0.9609\n",
            "\n",
            "Epoch 00256: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 257/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0460 - categorical_accuracy: 0.9864 - val_loss: 0.2170 - val_categorical_accuracy: 0.9520\n",
            "\n",
            "Epoch 00257: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 258/300\n",
            "122/122 [==============================] - 40s 329ms/step - loss: 0.0578 - categorical_accuracy: 0.9828 - val_loss: 0.1951 - val_categorical_accuracy: 0.9621\n",
            "\n",
            "Epoch 00258: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 259/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0544 - categorical_accuracy: 0.9841 - val_loss: 0.2083 - val_categorical_accuracy: 0.9632\n",
            "\n",
            "Epoch 00259: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 260/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0594 - categorical_accuracy: 0.9819 - val_loss: 0.1940 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00260: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 261/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0785 - categorical_accuracy: 0.9778 - val_loss: 0.1941 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00261: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 262/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0674 - categorical_accuracy: 0.9804 - val_loss: 0.2491 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00262: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 263/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0715 - categorical_accuracy: 0.9798 - val_loss: 0.1827 - val_categorical_accuracy: 0.9654\n",
            "\n",
            "Epoch 00263: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 264/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0656 - categorical_accuracy: 0.9818 - val_loss: 0.3761 - val_categorical_accuracy: 0.9319\n",
            "\n",
            "Epoch 00264: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 265/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0715 - categorical_accuracy: 0.9784 - val_loss: 0.2696 - val_categorical_accuracy: 0.9576\n",
            "\n",
            "Epoch 00265: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 266/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 0.0692 - categorical_accuracy: 0.9791 - val_loss: 0.2235 - val_categorical_accuracy: 0.9464\n",
            "\n",
            "Epoch 00266: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 267/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0565 - categorical_accuracy: 0.9828 - val_loss: 0.2147 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00267: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 268/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0546 - categorical_accuracy: 0.9831 - val_loss: 0.1900 - val_categorical_accuracy: 0.9576\n",
            "\n",
            "Epoch 00268: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 269/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0673 - categorical_accuracy: 0.9798 - val_loss: 0.2529 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00269: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 270/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0610 - categorical_accuracy: 0.9827 - val_loss: 0.1691 - val_categorical_accuracy: 0.9609\n",
            "\n",
            "Epoch 00270: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 271/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0562 - categorical_accuracy: 0.9836 - val_loss: 0.2318 - val_categorical_accuracy: 0.9520\n",
            "\n",
            "Epoch 00271: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 272/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0523 - categorical_accuracy: 0.9858 - val_loss: 0.1943 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00272: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 273/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0573 - categorical_accuracy: 0.9832 - val_loss: 0.1692 - val_categorical_accuracy: 0.9598\n",
            "\n",
            "Epoch 00273: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 274/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0797 - categorical_accuracy: 0.9762 - val_loss: 0.1817 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00274: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 275/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0782 - categorical_accuracy: 0.9764 - val_loss: 0.2201 - val_categorical_accuracy: 0.9475\n",
            "\n",
            "Epoch 00275: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 276/300\n",
            "122/122 [==============================] - 39s 319ms/step - loss: 0.0592 - categorical_accuracy: 0.9835 - val_loss: 0.1844 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00276: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 277/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0570 - categorical_accuracy: 0.9828 - val_loss: 0.3205 - val_categorical_accuracy: 0.9364\n",
            "\n",
            "Epoch 00277: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 278/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0673 - categorical_accuracy: 0.9807 - val_loss: 0.2839 - val_categorical_accuracy: 0.9342\n",
            "\n",
            "Epoch 00278: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 279/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0635 - categorical_accuracy: 0.9812 - val_loss: 0.2184 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00279: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 280/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0617 - categorical_accuracy: 0.9814 - val_loss: 0.2315 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00280: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 281/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0572 - categorical_accuracy: 0.9834 - val_loss: 0.2441 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00281: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 282/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0508 - categorical_accuracy: 0.9842 - val_loss: 0.1624 - val_categorical_accuracy: 0.9643\n",
            "\n",
            "Epoch 00282: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 283/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0542 - categorical_accuracy: 0.9840 - val_loss: 0.2405 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00283: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 284/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.0470 - categorical_accuracy: 0.9844 - val_loss: 0.2060 - val_categorical_accuracy: 0.9598\n",
            "\n",
            "Epoch 00284: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 285/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0495 - categorical_accuracy: 0.9848 - val_loss: 0.1846 - val_categorical_accuracy: 0.9621\n",
            "\n",
            "Epoch 00285: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 286/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0541 - categorical_accuracy: 0.9846 - val_loss: 0.3018 - val_categorical_accuracy: 0.9509\n",
            "\n",
            "Epoch 00286: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 287/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0621 - categorical_accuracy: 0.9825 - val_loss: 0.1840 - val_categorical_accuracy: 0.9542\n",
            "\n",
            "Epoch 00287: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 288/300\n",
            "122/122 [==============================] - 39s 324ms/step - loss: 0.0694 - categorical_accuracy: 0.9805 - val_loss: 0.2514 - val_categorical_accuracy: 0.9431\n",
            "\n",
            "Epoch 00288: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 289/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0673 - categorical_accuracy: 0.9819 - val_loss: 0.2903 - val_categorical_accuracy: 0.9654\n",
            "\n",
            "Epoch 00289: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 290/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0530 - categorical_accuracy: 0.9859 - val_loss: 0.2496 - val_categorical_accuracy: 0.9408\n",
            "\n",
            "Epoch 00290: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 291/300\n",
            "122/122 [==============================] - 40s 325ms/step - loss: 0.0520 - categorical_accuracy: 0.9839 - val_loss: 0.1825 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00291: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 292/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.0557 - categorical_accuracy: 0.9842 - val_loss: 0.2382 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00292: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 293/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0657 - categorical_accuracy: 0.9808 - val_loss: 0.1793 - val_categorical_accuracy: 0.9587\n",
            "\n",
            "Epoch 00293: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 294/300\n",
            "122/122 [==============================] - 39s 320ms/step - loss: 0.0684 - categorical_accuracy: 0.9800 - val_loss: 0.2209 - val_categorical_accuracy: 0.9554\n",
            "\n",
            "Epoch 00294: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 295/300\n",
            "122/122 [==============================] - 40s 324ms/step - loss: 0.0662 - categorical_accuracy: 0.9787 - val_loss: 0.2098 - val_categorical_accuracy: 0.9498\n",
            "\n",
            "Epoch 00295: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 296/300\n",
            "122/122 [==============================] - 39s 322ms/step - loss: 0.0569 - categorical_accuracy: 0.9831 - val_loss: 0.2258 - val_categorical_accuracy: 0.9643\n",
            "\n",
            "Epoch 00296: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 297/300\n",
            "122/122 [==============================] - 39s 323ms/step - loss: 0.0599 - categorical_accuracy: 0.9826 - val_loss: 0.2640 - val_categorical_accuracy: 0.9420\n",
            "\n",
            "Epoch 00297: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 298/300\n",
            "122/122 [==============================] - 40s 326ms/step - loss: 0.0507 - categorical_accuracy: 0.9846 - val_loss: 0.2696 - val_categorical_accuracy: 0.9397\n",
            "\n",
            "Epoch 00298: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 299/300\n",
            "122/122 [==============================] - 39s 321ms/step - loss: 0.0587 - categorical_accuracy: 0.9827 - val_loss: 0.1601 - val_categorical_accuracy: 0.9576\n",
            "\n",
            "Epoch 00299: val_categorical_accuracy did not improve from 0.96540\n",
            "Epoch 300/300\n",
            "122/122 [==============================] - 40s 327ms/step - loss: 0.0609 - categorical_accuracy: 0.9812 - val_loss: 0.2206 - val_categorical_accuracy: 0.9487\n",
            "\n",
            "Epoch 00300: val_categorical_accuracy did not improve from 0.96540\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f4c5f0f5cf8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U_WL14U_Mo2g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}