3-us8k-cnn-extract-train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Workbook 3 - Extracting UrbanSound8K audio features for a Convolutional Neural Network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, import the libraries. As with the earlier notebooks, the audio processing is handled by a library called librosa, if you haven't already installed it on your local system, do that with: pip install librosa\n",
    "\n",
    "You'll also need Keras and Tensorflow installed. This was updated on 23 July 2017 to use **Python3**, **Keras 2** and **Tensorflow 1.2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "import librosa\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebook 1 explained how to extract and aggregate several audio features using processing methods provided by the librosa library which could be fed into a Feed-Forward Network.  \n",
    "\n",
    "This notebook is going to describe how to learn audio data using a different architecture, a Convolutional Neural Network (CNN). \n",
    "\n",
    "A CNN organises hidden units to take advantage of the local structure present in two-dimensional input data (the classic example being edges in images). By concentrating on identifying local features each hidden unit only needs to process a tiny part of the whole input space, instead of being connected to all the inputs coming from the previous\n",
    "layer. Processing proceeds by successively considering small windows of the data set, (e.g. 3x3 pixels), called the receptive field.\n",
    "\n",
    "The weights of hidden units create a convolutional kernel (or filter) which is applied to the whole input space, like a succession of tiles, resulting in a feature map. As a result, one set of weights can be reused for the whole input space. As a feature like an edge can occur anywhere in the input space, the CNN approach greatly reduces the number of parameters required, as well as improving the model's robustness.  \n",
    "\n",
    "A typical convolutional layer will consist of numerous filters (feature maps). Further dimensionality reduction can be achieved through pooling layers, which merge adjacent cells of a feature map, using pooling operations such as \n",
    "max (winner takes all) or the mean of the input cells. This downsampling further improves the tolerance of the network to variation and noise.\n",
    "\n",
    "CNNs have proved especially successful at classification tasks, particularly of images, but to use them to classify audio files, we'll need some way of extracting audio features in such a way that they reflect the kind of input data a CNN expects. The method is explained by Karol J. Piczak in his paper Environmental sound classification with convolutional neural networks (http://karol.piczak.com/papers/Piczak2015-ESC-ConvNet.pdf). This describes how to get equal size segments from varying length audio clips, and which audio features can be fed into the network as separate channels (akin to RGB channels of colour images). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Extraction for CNNs (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea is to calculate log-scaled mel-spectrograms and their corresponding deltas from a sound clip. Because we need fixed size input, we split each sound clip into 41 overlapping frames, one for each of the 60 mel-bands, giving an array of 60 rows and 41 columns. The mel-spectrogram for each band/segment and its time-series deltas will become two channels, which becomes our input to feed into the CNN. Other features could be calculated in the same way and supplied as a separate input channel, but we'll stick with just the mel-spectragram data for this example. \n",
    "\n",
    "The feature extraction process is based on example code posted by Aaqib Saeed:\n",
    "http://aqibsaeed.github.io/2016-09-24-urban-sound-classification-part-2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def windows(data, window_size):\n",
    "    start = 0\n",
    "    while start < len(data):\n",
    "        yield start, start + window_size\n",
    "        start += (window_size / 2)\n",
    "\n",
    "def extract_features(parent_dir,sub_dirs,file_ext=\"*.wav\",bands = 60, frames = 41):\n",
    "    window_size = 512 * (frames - 1)\n",
    "    log_specgrams = []\n",
    "    labels = []\n",
    "    for l, sub_dir in enumerate(sub_dirs):\n",
    "        for fn in glob.glob(os.path.join(parent_dir, sub_dir, file_ext)):\n",
    "            sound_clip,s = librosa.load(fn)\n",
    "            label = fn.split('fold')[1].split('-')[1]        \n",
    "            for (start,end) in windows(sound_clip,window_size):\n",
    "                if(len(sound_clip[start:end]) == int(window_size)):\n",
    "                    signal = sound_clip[start:end]\n",
    "                    melspec = librosa.feature.melspectrogram(signal, n_mels = bands)\n",
    "                    logspec = librosa.logamplitude(melspec)\n",
    "                    logspec = logspec.T.flatten()[:, np.newaxis].T\n",
    "                    log_specgrams.append(logspec)\n",
    "                    labels.append(label)\n",
    "            \n",
    "    log_specgrams = np.asarray(log_specgrams).reshape(len(log_specgrams),bands,frames,1)\n",
    "    features = np.concatenate((log_specgrams, np.zeros(np.shape(log_specgrams))), axis = 3)\n",
    "    for i in range(len(features)):\n",
    "        features[i, :, :, 1] = librosa.feature.delta(features[i, :, :, 0])\n",
    "    \n",
    "    return np.array(features), np.array(labels,dtype = np.int)\n",
    "\n",
    "def one_hot_encode(labels):\n",
    "    n_labels = len(labels)\n",
    "    n_unique_labels = len(np.unique(labels))\n",
    "    one_hot_encode = np.zeros((n_labels,n_unique_labels))\n",
    "    one_hot_encode[np.arange(n_labels), labels] = 1\n",
    "    return one_hot_encode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the sound clip is longer than the expected window size, the extraction process will create several features rather than just one. As the sound recordings are often repetitive, this is an example of Data Augmentation, creating several training examples from a single source recording. In the example below, we can see how a sample audio file results in the creation of 7 feature rows, each consisting of 3-dimensional array (60x41x2) of data values.\n",
    "\n",
    "This code also shows how the feature representation used for a CNN is not as compact as the inputs fed into the Feed-Forward Network, which had only 193 features. In this case, 88200 data points have been reduced to 34440. The hope is by retaining more of the source data, we are able to learn from it more subtly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IN: Initial Data Points = 88200\n",
      "OUT: Total features = (7, 60, 41, 2)\n"
     ]
    }
   ],
   "source": [
    "def extract_feature_array(filename, bands = 60, frames = 41):\n",
    "    window_size = 512 * (frames - 1)\n",
    "    log_specgrams = []\n",
    "    sound_clip,s = librosa.load(filename)        \n",
    "    for (start,end) in windows(sound_clip,window_size):\n",
    "        start = int(start)\n",
    "        end = int(end)\n",
    "        if(len(sound_clip[start:end]) == window_size):\n",
    "            signal = sound_clip[start:end]\n",
    "            melspec = librosa.feature.melspectrogram(signal, n_mels = bands)\n",
    "            logspec = librosa.logamplitude(melspec)\n",
    "            logspec = logspec.T.flatten()[:, np.newaxis].T\n",
    "            log_specgrams.append(logspec)\n",
    "            \n",
    "    log_specgrams = np.asarray(log_specgrams).reshape(len(log_specgrams),bands,frames,1)\n",
    "    features = np.concatenate((log_specgrams, np.zeros(np.shape(log_specgrams))), axis = 3)\n",
    "    for i in range(len(features)):\n",
    "        features[i, :, :, 1] = librosa.feature.delta(features[i, :, :, 0])\n",
    "    \n",
    "    return np.array(features)\n",
    "\n",
    "sample_filename = \"samples/us8k/music.wav\"\n",
    "features = extract_feature_array(sample_filename)\n",
    "data_points, _ = librosa.load(sample_filename)\n",
    "print (\"IN: Initial Data Points =\", len(data_points))\n",
    "print (\"OUT: Total features =\", np.shape(features))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving Extracted Features (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in the cell below can be run (once) to convert the raw audio files into features, which are stored as numpy arrays. As this process is quite time consuming, we'd prefer to just do it once, and then load the numpy data when we want to do some training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Saving fold1\n",
      "Features of fold1  =  (2400, 60, 41, 2)\n",
      "Labels of fold1  =  (2400, 10)\n",
      "Saved data/us8k-np-cnn-mini/fold1_x.npy\n",
      "Saved data/us8k-np-cnn-mini/fold1_y.npy\n",
      "\n",
      "Saving fold2\n",
      "Features of fold2  =  (2076, 60, 41, 2)\n",
      "Labels of fold2  =  (2076, 10)\n",
      "Saved data/us8k-np-cnn-mini/fold2_x.npy\n",
      "Saved data/us8k-np-cnn-mini/fold2_y.npy\n",
      "\n",
      "Saving fold3\n",
      "Features of fold3  =  (2371, 60, 41, 2)\n",
      "Labels of fold3  =  (2371, 10)\n",
      "Saved data/us8k-np-cnn-mini/fold3_x.npy\n",
      "Saved data/us8k-np-cnn-mini/fold3_y.npy\n"
     ]
    }
   ],
   "source": [
    "# use this to process the audio files into numpy arrays\n",
    "def save_folds(data_dir):\n",
    "    for k in range(1,11):\n",
    "        fold_name = 'fold' + str(k)\n",
    "        print (\"\\nSaving \" + fold_name)\n",
    "        features, labels = extract_features(parent_dir, [fold_name])\n",
    "        labels = one_hot_encode(labels)\n",
    "        \n",
    "        print (\"Features of\", fold_name , \" = \", features.shape)\n",
    "        print (\"Labels of\", fold_name , \" = \", labels.shape)\n",
    "        \n",
    "        feature_file = os.path.join(data_dir, fold_name + '_x.npy')\n",
    "        labels_file = os.path.join(data_dir, fold_name + '_y.npy')\n",
    "        np.save(feature_file, features)\n",
    "        print (\"Saved \" + feature_file)\n",
    "        np.save(labels_file, labels)\n",
    "        print (\"Saved \" + labels_file)\n",
    "\n",
    "def assure_path_exists(path):\n",
    "    mydir = os.path.join(os.getcwd(), path)\n",
    "    if not os.path.exists(mydir):\n",
    "        os.makedirs(mydir)\n",
    "        \n",
    "# uncomment this to recreate and save the feature vectors\n",
    "parent_dir = \"raw/us8k\" # Where you have saved the UrbanSound8K data set\"       \n",
    "save_dir = \"data/us8k-np-cnn\"\n",
    "assure_path_exists(save_dir)\n",
    "save_folds(save_dir)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a powerful computer, and don't mind the training process taking longer, you can set the all_folds flag to true - which will create a single large training set of 43722 examples, using the first 8 folds. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will aggregate all the training data \n",
    "def load_all_folds(): \n",
    "    subsequent_fold = False\n",
    "    for k in range(1,9):\n",
    "        fold_name = 'fold' + str(k)\n",
    "        print (\"\\nAdding \" + fold_name)\n",
    "        feature_file = os.path.join(data_dir, fold_name + '_x.npy')\n",
    "        labels_file = os.path.join(data_dir, fold_name + '_y.npy')\n",
    "        loaded_features = np.load(feature_file)\n",
    "        loaded_labels = np.load(labels_file)\n",
    "        print (\"New Features: \", loaded_features.shape)\n",
    "\n",
    "        if subsequent_fold:\n",
    "            train_x = np.concatenate((features, loaded_features))\n",
    "            train_y = np.concatenate((labels, loaded_labels))\n",
    "        else:\n",
    "            train_x = loaded_features\n",
    "            train_y = loaded_labels\n",
    "            subsequent_fold = True\n",
    "        \n",
    "    # use the penultimate fold for validation\n",
    "    valid_fold_name = 'fold9'\n",
    "    feature_file = os.path.join(data_dir, valid_fold_name + '_x.npy')\n",
    "    labels_file = os.path.join(data_dir, valid_fold_name + '_y.npy')\n",
    "    valid_x = np.load(feature_file)\n",
    "    valid_y = np.load(labels_file) \n",
    "\n",
    "    # and use the last fold for testing\n",
    "    test_fold_name = 'fold10'\n",
    "    feature_file = os.path.join(data_dir, test_fold_name + '_x.npy')\n",
    "    labels_file = os.path.join(data_dir, test_fold_name + '_y.npy')\n",
    "    test_x = np.load(feature_file)\n",
    "    test_y = np.load(labels_file)\n",
    "    \n",
    "\n",
    "# this is used to load the folds incrementally\n",
    "def load_folds(folds):\n",
    "    subsequent_fold = False\n",
    "    for k in range(len(folds)):\n",
    "        fold_name = 'fold' + str(folds[k])\n",
    "        feature_file = os.path.join(data_dir, fold_name + '_x.npy')\n",
    "        labels_file = os.path.join(data_dir, fold_name + '_y.npy')\n",
    "        loaded_features = np.load(feature_file)\n",
    "        loaded_labels = np.load(labels_file)\n",
    "        print (fold_name, \"features: \", loaded_features.shape)\n",
    "\n",
    "        if subsequent_fold:\n",
    "            features = np.concatenate((features, loaded_features))\n",
    "            labels = np.concatenate((labels, loaded_labels))\n",
    "        else:\n",
    "            features = loaded_features\n",
    "            labels = loaded_labels\n",
    "            subsequent_fold = True\n",
    "        \n",
    "    return features, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Convolutional Neural Network with Keras and TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the imports we need and a few configuration variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "tf.set_random_seed(0)\n",
    "np.random.seed(0)\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers import Convolution2D, MaxPooling2D\n",
    "from keras.optimizers import SGD, Adam\n",
    "from keras.callbacks import EarlyStopping\n",
    "from sklearn.metrics import precision_recall_fscore_support, roc_auc_score\n",
    "from keras.utils import np_utils\n",
    "\n",
    "frames = 41\n",
    "bands = 60\n",
    "feature_size = bands * frames #60x41\n",
    "num_labels = 10\n",
    "num_channels = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method defines some evaluation metrics that will be used to evaluate the performance of a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model):\n",
    "    y_prob = model.predict_proba(test_x, verbose=0)\n",
    "    y_pred = y_prob.argmax(axis=-1)\n",
    "    y_true = np.argmax(test_y, 1)\n",
    "\n",
    "    roc = roc_auc_score(test_y, y_prob)\n",
    "    print (\"ROC:\",  round(roc,3))\n",
    "\n",
    "    # evaluate the model\n",
    "    score, accuracy = model.evaluate(test_x, test_y, batch_size=32)\n",
    "    print(\"\\nAccuracy = {:.2f}\".format(accuracy))\n",
    "\n",
    "    # the F-score gives a similiar value to the accuracy score, but useful for cross-checking\n",
    "    p,r,f,s = precision_recall_fscore_support(y_true, y_pred, average='micro')\n",
    "    print (\"F-Score:\", round(f,2))\n",
    "    \n",
    "    return roc, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following section defines the successive layers of our convolutional neural network (CNN). \n",
    "\n",
    "CNNs differ from the FFN described in notebook 2 by having dedicated convolutional layers between the input layer and the hidden layers. It begins with two Convolution2D layers, the first with 96 output filters, the second with 64 - the idea being to apply convolutions that will progressively squeeze the spatial dimensions while increasing the depth. In effect, space is transformed into depth, spatial information is removed until only semantic complexity remains. \n",
    "\n",
    "Once a deep and narrow representation has been created, the output is fed into a regular deep neural network of densely connected layers, and used to train a classifier just as we did with the Feed-Forward Network.\n",
    "\n",
    "As well as familiar techniques like Dropout and Activation Functions, you'll notice something new in this model - a \n",
    "process called Max Pooling. This works by looking at a small neighbourhood around every point in the future map, and computing the maximum of all the responses around it. Combining many weaker signals into one stronger signal helps reduce the risk of over-fitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_model():\n",
    "    \n",
    "    model = Sequential()\n",
    "    # input: 60x41 data frames with 2 channels => (60,41,2) tensors\n",
    "\n",
    "    # filters of size 1x1 \n",
    "    f_size = 1\n",
    "\n",
    "    # first layer has 48 convolution filters \n",
    "    model.add(Convolution2D(48, f_size, strides=f_size, kernel_initializer='normal', padding='same', input_shape=(bands, frames, num_channels)))\n",
    "    model.add(Convolution2D(48, f_size, strides=f_size, kernel_initializer='normal', padding='same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # next layer has 96 convolution filters\n",
    "    model.add(Convolution2D(96, f_size, strides=f_size, kernel_initializer='normal', padding='same'))\n",
    "    model.add(Convolution2D(96, f_size, strides=f_size, kernel_initializer='normal', padding='same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # flatten output into a single dimension \n",
    "    # Keras will do shape inference automatically\n",
    "    model.add(Flatten())\n",
    "\n",
    "    # then a fully connected NN layer\n",
    "    model.add(Dense(256))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # finally, an output layer with one node per class\n",
    "    model.add(Dense(num_labels))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    # use the Adam optimiser\n",
    "    adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0)\n",
    "    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=adam)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training on a minimised data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set is a large quantity of data, which is quite computationally intensive to process. But if you've checked out the git repository for this project, I've included a selection from the original data set, which should allow you to run the CNN and see how it works. Obviously as this training set size is much smaller and contains much less diversity than the full set, lower accuracies will be achievable when models are trained on it, but it will allow you to run the code and explore how it works. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold1 features:  (2400, 60, 41, 2)\n",
      "fold2 features:  (2076, 60, 41, 2)\n",
      "fold3 features:  (2371, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n",
      "Train on 2400 samples, validate on 2076 samples\n",
      "Epoch 1/3\n",
      "2400/2400 [==============================] - 47s - loss: 1.7494 - acc: 0.3550 - val_loss: 1.8051 - val_acc: 0.3921\n",
      "Epoch 2/3\n",
      "2400/2400 [==============================] - 48s - loss: 1.3732 - acc: 0.5050 - val_loss: 1.8185 - val_acc: 0.4027\n",
      "Epoch 00001: early stopping\n",
      "Evaluating model...\n",
      "ROC: 0.737\n",
      "2368/2371 [============================>.] - ETA: 0s\n",
      "Accuracy = 0.39\n",
      "F-Score: 0.39\n"
     ]
    }
   ],
   "source": [
    "data_dir = \"data/us8k-np-cnn-mini\"\n",
    "\n",
    "# load fold1 for testing\n",
    "train_x, train_y = load_folds([1])\n",
    "\n",
    "# load fold2 for validation\n",
    "valid_x, valid_y = load_folds([2])\n",
    "    \n",
    "# load fold3 for testing\n",
    "test_x, test_y = load_folds([3])\n",
    "\n",
    "print(\"Building model...\")\n",
    "model = build_model()\n",
    "\n",
    "# a stopping function to stop training before we excessively overfit to the training set\n",
    "earlystop = EarlyStopping(monitor='val_loss', patience=0, verbose=1, mode='auto')\n",
    "\n",
    "# now fit the model to the training data, evaluating loss against the validation data\n",
    "print(\"Training model...\")\n",
    "model.fit(train_x, train_y, validation_data=(valid_x, valid_y), callbacks=[earlystop], batch_size=20, epochs=3)\n",
    "\n",
    "# now evaluate the trained model against the unseen test data\n",
    "print(\"Evaluating model...\")\n",
    "roc, acc = evaluate(model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training on the full data set (intensive) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this code you'll need to have downloaded and run the feature extraction process on the entire UrbanSound8k data set (which requires some quite intensive processing).\n",
    "\n",
    "The training process below is an implementation of multi-fold cross-validation, it will create a fresh model for each trial, load the next fold of input data, fit it to the model and evaluate the accuracy achieved. When all the input data has been used, we can calculate the average accuracy for all the folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "*** Train on 1 & 2 Validate on 3 Test on 4 ***\n",
      "fold1 features:  (5446, 60, 41, 2)\n",
      "fold2 features:  (5388, 60, 41, 2)\n",
      "fold3 features:  (5852, 60, 41, 2)\n",
      "fold4 features:  (6048, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jaron/anaconda/envs/tflearn/lib/python3.5/site-packages/keras/models.py:844: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  warnings.warn('The `nb_epoch` argument in `fit` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 10834 samples, validate on 5852 samples\n",
      "Epoch 1/1\n",
      "10834/10834 [==============================] - 163s - loss: 1.7122 - acc: 0.3662 - val_loss: 1.9999 - val_acc: 0.3337\n",
      "Evaluating model...\n",
      "ROC: 0.726\n",
      "6048/6048 [==============================] - 18s    \n",
      "\n",
      "Accuracy = 0.33\n",
      "F-Score: 0.33\n",
      "\n",
      "*** Train on 2 & 3 Validate on 4 Test on 5 ***\n",
      "fold2 features:  (5388, 60, 41, 2)\n",
      "fold3 features:  (5852, 60, 41, 2)\n",
      "fold4 features:  (6048, 60, 41, 2)\n",
      "fold5 features:  (5689, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n",
      "Train on 11240 samples, validate on 6048 samples\n",
      "Epoch 1/1\n",
      "11240/11240 [==============================] - 181s - loss: 1.7266 - acc: 0.3722 - val_loss: 1.8495 - val_acc: 0.3833\n",
      "Evaluating model...\n",
      "ROC: 0.722\n",
      "5689/5689 [==============================] - 22s    \n",
      "\n",
      "Accuracy = 0.31\n",
      "F-Score: 0.31\n",
      "\n",
      "*** Train on 3 & 4 Validate on 5 Test on 6 ***\n",
      "fold3 features:  (5852, 60, 41, 2)\n",
      "fold4 features:  (6048, 60, 41, 2)\n",
      "fold5 features:  (5689, 60, 41, 2)\n",
      "fold6 features:  (5080, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n",
      "Train on 11900 samples, validate on 5689 samples\n",
      "Epoch 1/1\n",
      "11900/11900 [==============================] - 191s - loss: 1.5940 - acc: 0.4369 - val_loss: 1.9867 - val_acc: 0.4045\n",
      "Evaluating model...\n",
      "ROC: 0.833\n",
      "5080/5080 [==============================] - 16s    \n",
      "\n",
      "Accuracy = 0.42\n",
      "F-Score: 0.42\n",
      "\n",
      "*** Train on 4 & 5 Validate on 6 Test on 7 ***\n",
      "fold4 features:  (6048, 60, 41, 2)\n",
      "fold5 features:  (5689, 60, 41, 2)\n",
      "fold6 features:  (5080, 60, 41, 2)\n",
      "fold7 features:  (5277, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n",
      "Train on 11737 samples, validate on 5080 samples\n",
      "Epoch 1/1\n",
      "11737/11737 [==============================] - 209s - loss: 1.6998 - acc: 0.3842 - val_loss: 1.8827 - val_acc: 0.3825\n",
      "Evaluating model...\n",
      "ROC: 0.807\n",
      "5277/5277 [==============================] - 17s    \n",
      "\n",
      "Accuracy = 0.33\n",
      "F-Score: 0.33\n",
      "\n",
      "*** Train on 5 & 6 Validate on 7 Test on 8 ***\n",
      "fold5 features:  (5689, 60, 41, 2)\n",
      "fold6 features:  (5080, 60, 41, 2)\n",
      "fold7 features:  (5277, 60, 41, 2)\n",
      "fold8 features:  (4942, 60, 41, 2)\n",
      "Building model...\n",
      "Training model...\n",
      "Train on 10769 samples, validate on 5277 samples\n",
      "Epoch 1/1\n",
      "10769/10769 [==============================] - 177s - loss: 1.8312 - acc: 0.3290 - val_loss: 2.0738 - val_acc: 0.2746\n",
      "Evaluating model...\n",
      "ROC: 0.8\n",
      "4942/4942 [==============================] - 15s    \n",
      "\n",
      "Accuracy = 0.31\n",
      "F-Score: 0.31\n",
      "\n",
      "Average R.O.C: 0.777\n",
      "Average Accuracy: 0.339\n"
     ]
    }
   ],
   "source": [
    "data_dir = \"data/us8k-np-cnn\"\n",
    "all_folds = False\n",
    "av_acc = 0.\n",
    "av_roc = 0.\n",
    "num_folds = 0\n",
    "\n",
    "# as we use two folds for training, there are 9 possible trails rather than 10\n",
    "max_trials = 5\n",
    "\n",
    "# earlystopping ends training when the validation loss stops improving\n",
    "earlystop = EarlyStopping(monitor='val_loss', patience=0, verbose=0, mode='auto')\n",
    "\n",
    "if all_folds:\n",
    "    load_all_folds()\n",
    "    model.fit(train_x, train_y, validation_data=(valid_x, valid_y), callbacks=[earlystop], batch_size=32, epochs=1)\n",
    "else:\n",
    "    # use folds incrementally\n",
    "    for f in range(1,max_trials+1):\n",
    "        num_folds += 1\n",
    "        v = f + 2\n",
    "        if v > 10: v = 1\n",
    "        t = v + 1\n",
    "        if t > 10: t = 1\n",
    "        \n",
    "        print (\"\\n*** Train on\", f, \"&\",(f+1), \"Validate on\", v, \"Test on\", t, \"***\")\n",
    "    \n",
    "        # load two folds for training data\n",
    "        train_x, train_y = load_folds([f,f+1])\n",
    "    \n",
    "        # load one fold for validation\n",
    "        valid_x, valid_y = load_folds([v])\n",
    "    \n",
    "        # load one fold for testing\n",
    "        test_x, test_y = load_folds([t])\n",
    "        \n",
    "        print(\"Building model...\")\n",
    "        model = build_model()\n",
    "\n",
    "        # now fit the model to the training data, evaluating loss against the validation data\n",
    "        print(\"Training model...\")\n",
    "        model.fit(train_x, train_y, validation_data=(valid_x, valid_y), callbacks=[earlystop], batch_size=64, epochs=1)\n",
    "\n",
    "        # now evaluate the trained model against the unseen test data\n",
    "        print(\"Evaluating model...\")\n",
    "        roc, acc = evaluate(model)\n",
    "        av_roc += roc\n",
    "        av_acc += acc\n",
    "    \n",
    "print ('\\nAverage R.O.C:', round(av_roc/max_trials, 3))\n",
    "print ('Average Accuracy:', round(av_acc/max_trials, 3))\n",
    "\n",
    "# using all folds: best ROC = 0.889, f-score = 0.591\n",
    "# using 2 folds: average ROC = 0.792, average f-score = 0.335"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating Predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, test the prediction performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-----  air conditioner -----\n",
      "Top guess:  air conditioner  ( 0.304 )\n",
      "2nd guess:  children playing  ( 0.202 )\n",
      "\n",
      "-----  car horn -----\n",
      "Top guess:  car horn  ( 0.403 )\n",
      "2nd guess:  drilling  ( 0.155 )\n",
      "\n",
      "-----  children playing -----\n",
      "Top guess:  jackhammer  ( 0.33 )\n",
      "2nd guess:  children playing  ( 0.19 )\n",
      "\n",
      "-----  dog bark -----\n",
      "Top guess:  children playing  ( 0.349 )\n",
      "2nd guess:  street music  ( 0.228 )\n",
      "\n",
      "-----  drilling -----\n",
      "Top guess:  jackhammer  ( 0.768 )\n",
      "2nd guess:  drilling  ( 0.174 )\n",
      "\n",
      "-----  engine idling -----\n",
      "Top guess:  street music  ( 0.505 )\n",
      "2nd guess:  engine idling  ( 0.214 )\n",
      "\n",
      "-----  gun shot -----\n",
      "Top guess:  drilling  ( 0.166 )\n",
      "2nd guess:  air conditioner  ( 0.156 )\n",
      "\n",
      "-----  jackhammer -----\n",
      "Top guess:  jackhammer  ( 0.398 )\n",
      "2nd guess:  drilling  ( 0.186 )\n",
      "\n",
      "-----  siren -----\n",
      "Top guess:  siren  ( 0.727 )\n",
      "2nd guess:  engine idling  ( 0.192 )\n",
      "\n",
      "-----  street music -----\n",
      "Top guess:  drilling  ( 0.253 )\n",
      "2nd guess:  engine idling  ( 0.241 )\n"
     ]
    }
   ],
   "source": [
    "sound_file_paths = [\"aircon.wav\", \"carhorn.wav\", \"play.wav\", \"dogbark.wav\", \"drill.wav\",\n",
    "                    \"engine.wav\",\"gunshots.wav\",\"jackhammer.wav\",\"siren.wav\",\"music.wav\"]\n",
    "sound_names = [\"air conditioner\",\"car horn\",\"children playing\",\"dog bark\",\"drilling\",\"engine idling\",\n",
    "               \"gun shot\",\"jackhammer\",\"siren\",\"street music\"]\n",
    "parent_dir = 'samples/us8k/'\n",
    "\n",
    "\n",
    "\n",
    "# create predictions for each of the sound classes\n",
    "for s in range(len(sound_names)):\n",
    "\n",
    "    print (\"\\n----- \", sound_names[s], \"-----\")\n",
    "    # load audio file and extract features\n",
    "    predict_file = parent_dir + sound_file_paths[s]\n",
    "    predict_x = extract_feature_array(predict_file)\n",
    "    \n",
    "    # generate prediction, passing in just a single row of features\n",
    "    predictions = model.predict(predict_x)\n",
    "    \n",
    "    if len(predictions) == 0: \n",
    "        print (\"No prediction\")\n",
    "        continue\n",
    "    \n",
    "    #for i in range(len(predictions[0])):\n",
    "    #    print sound_names[i], \"=\", round(predictions[0,i] * 100, 1)\n",
    "    \n",
    "    # get the indices of the top 2 predictions, invert into descending order\n",
    "    ind = np.argpartition(predictions[0], -2)[-2:]\n",
    "    ind[np.argsort(predictions[0][ind])]\n",
    "    ind = ind[::-1]\n",
    "    \n",
    "    print (\"Top guess: \", sound_names[ind[0]], \" (\",round(predictions[0,ind[0]],3),\")\")\n",
    "    print (\"2nd guess: \", sound_names[ind[1]], \" (\",round(predictions[0,ind[1]],3),\")\")\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The accuracy score gives a measure of the model's predictive power over the entire test set, but it can also be useful to look at how the model behaves at the smallest scale, when asked to predict the class of individual files. \n",
    "\n",
    "If you run the code above, you'll see the predicted classes for each of the sample files, 6 of the 10 predictions are correct, but the other 4 are incorrect. If you listen to the sample files, you might be able to identify most of them, but some are ambiguous, such as air conditioning that sounds like distant traffic. \n",
    "\n",
    "The best accuracy I have achieved with the code in this notebook was ~59% - and that was using the full UrbanSound8K data set. So clearly audio classification, like image classification, is not a trivial challenge. The next notebook (#4) will describe the implementation of a more sophisticated CNN, with the aim of achieving even better classification performance."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}