digit_recognizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Project\n",
    "## Build a Digit Recognition Program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, we will design and implement a deep learning model that learns to recognize sequences of digits. We will train the model using synthetic data generated by concatenating character images from [MNIST](http://yann.lecun.com/exdb/mnist/). \n",
    "\n",
    "To produce a synthetic sequence of digits for testing, we will limit the to sequences to up to five digits, and use five classifiers on top of your deep network. We will incorporate an additional ‘blank’ character to account for shorter number sequences.\n",
    "\n",
    "We will use ** Keras ** to implement the model. You can read more about Keras at [keras.io](https://keras.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation\n",
    "\n",
    "Let's start by importing the modules we'll require fot this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#Module Imports\n",
    "from __future__ import print_function\n",
    "import random\n",
    "from os import listdir\n",
    "import glob\n",
    "\n",
    "import numpy as np\n",
    "from scipy import misc\n",
    "import tensorflow as tf\n",
    "import h5py\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.utils import np_utils\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Setting the random seed so that the results are reproducible. \n",
    "random.seed(101)\n",
    "\n",
    "#Setting variables for MNIST image dimensions\n",
    "mnist_image_height = 28\n",
    "mnist_image_width = 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 11s 1us/step\n"
     ]
    }
   ],
   "source": [
    "#Import MNIST data from keras\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of training dataset: (60000, 28, 28)\n",
      "Shape of test dataset: (10000, 28, 28)\n",
      "Label for image: 5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADgpJREFUeJzt3X+MVfWZx/HPs1j+kKI4aQRCYSnEYJW4082IjSWrxkzVDQZHrekkJjQapn8wiU02ZA3/VNNgyCrslmiamaZYSFpKE3VB0iw0otLGZuKIWC0srTFsO3IDNTjywx9kmGf/mEMzxbnfe+fec++5zPN+JeT+eM6558kNnznn3O+592vuLgDx/EPRDQAoBuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxDUZc3cmJlxOSHQYO5u1SxX157fzO40syNm9q6ZPVrPawFoLqv12n4zmybpj5I6JQ1Jel1St7sfSqzDnh9osGbs+ZdJetfd33P3c5J+IWllHa8HoInqCf88SX8Z93goe+7vmFmPmQ2a2WAd2wKQs3o+8Jvo0OJzh/Xu3i+pX+KwH2gl9ez5hyTNH/f4y5KO1dcOgGapJ/yvS7rGzL5iZtMlfVvSrnzaAtBoNR/2u/uImfVK2iNpmqQt7v6H3DoD0FA1D/XVtDHO+YGGa8pFPgAuXYQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfMU3ZJkZkclnZZ0XtKIu3fk0RTyM23atGT9yiuvbOj2e3t7y9Yuv/zy5LpLlixJ1tesWZOsP/XUU2Vr3d3dyXU//fTTZH3Dhg3J+uOPP56st4K6wp+5zd0/yOF1ADQRh/1AUPWG3yXtNbM3zKwnj4YANEe9h/3fcPdjZna1pF+b2f+6+/7xC2R/FPjDALSYuvb87n4suz0h6QVJyyZYpt/dO/gwEGgtNYffzGaY2cwL9yV9U9I7eTUGoLHqOeyfLekFM7vwOj939//JpSsADVdz+N39PUn/lGMvU9aCBQuS9enTpyfrN998c7K+fPnysrVZs2Yl173vvvuS9SINDQ0l65s3b07Wu7q6ytZOnz6dXPett95K1l999dVk/VLAUB8QFOEHgiL8QFCEHwiK8ANBEX4gKHP35m3MrHkba6L29vZkfd++fcl6o79W26pGR0eT9YceeihZP3PmTM3bLpVKyfqHH36YrB85cqTmbTeau1s1y7HnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOfPQVtbW7I+MDCQrC9atCjPdnJVqffh4eFk/bbbbitbO3fuXHLdqNc/1ItxfgBJhB8IivADQRF+ICjCDwRF+IGgCD8QVB6z9IZ38uTJZH3t2rXJ+ooVK5L1N998M1mv9BPWKQcPHkzWOzs7k/WzZ88m69dff33Z2iOPPJJcF43Fnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgqr4fX4z2yJphaQT7r40e65N0g5JCyUdlfSAu6d/6FxT9/v89briiiuS9UrTSff19ZWtPfzww8l1H3zwwWR9+/btyTpaT57f5/+ppDsveu5RSS+5+zWSXsoeA7iEVAy/u++XdPElbCslbc3ub5V0T859AWiwWs/5Z7t7SZKy26vzawlAMzT82n4z65HU0+jtAJicWvf8x81sriRltyfKLeju/e7e4e4dNW4LQAPUGv5dklZl91dJ2plPOwCapWL4zWy7pN9JWmJmQ2b2sKQNkjrN7E+SOrPHAC4hFc/53b27TOn2nHsJ69SpU3Wt/9FHH9W87urVq5P1HTt2JOujo6M1bxvF4go/ICjCDwRF+IGgCD8QFOEHgiL8QFBM0T0FzJgxo2ztxRdfTK57yy23JOt33XVXsr53795kHc3HFN0Akgg/EBThB4Ii/EBQhB8IivADQRF+ICjG+ae4xYsXJ+sHDhxI1oeHh5P1l19+OVkfHBwsW3vmmWeS6zbz/+ZUwjg/gCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiKcf7gurq6kvVnn302WZ85c2bN2163bl2yvm3btmS9VCrVvO2pjHF+AEmEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9siaYWkE+6+NHvuMUmrJf01W2ydu/+q4sYY57/kLF26NFnftGlTsn777bXP5N7X15esr1+/Pll///33a972pSzPcf6fSrpzguf/093bs38Vgw+gtVQMv7vvl3SyCb0AaKJ6zvl7zez3ZrbFzK7KrSMATVFr+H8kabGkdkklSRvLLWhmPWY2aGblf8wNQNPVFH53P+7u5919VNKPJS1LLNvv7h3u3lFrkwDyV1P4zWzuuIddkt7Jpx0AzXJZpQXMbLukWyV9ycyGJH1f0q1m1i7JJR2V9N0G9gigAfg+P+oya9asZP3uu+8uW6v0WwFm6eHqffv2JeudnZ3J+lTF9/kBJBF+ICjCDwRF+IGgCD8QFOEHgmKoD4X57LPPkvXLLktfhjIyMpKs33HHHWVrr7zySnLdSxlDfQCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIrf50dsN9xwQ7J+//33J+s33nhj2VqlcfxKDh06lKzv37+/rtef6tjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNPcUuWLEnWe3t7k/V77703WZ8zZ86ke6rW+fPnk/VSqZSsj46O5tnOlMOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjjOb2bzJW2TNEfSqKR+d/+hmbVJ2iFpoaSjkh5w9w8b12pclcbSu7u7y9YqjeMvXLiwlpZyMTg4mKyvX78+Wd+1a1ee7YRTzZ5/RNK/uftXJX1d0hozu07So5JecvdrJL2UPQZwiagYfncvufuB7P5pSYclzZO0UtLWbLGtku5pVJMA8jepc34zWyjpa5IGJM1295I09gdC0tV5Nwegcaq+tt/MvijpOUnfc/dTZlVNByYz65HUU1t7ABqlqj2/mX1BY8H/mbs/nz193MzmZvW5kk5MtK6797t7h7t35NEwgHxUDL+N7eJ/Iumwu28aV9olaVV2f5Wknfm3B6BRKk7RbWbLJf1G0tsaG+qTpHUaO+//paQFkv4s6VvufrLCa4Wconv27NnJ+nXXXZesP/3008n6tddeO+me8jIwMJCsP/nkk2VrO3em9xd8Jbc21U7RXfGc391/K6nci90+maYAtA6u8AOCIvxAUIQfCIrwA0ERfiAowg8ExU93V6mtra1sra+vL7lue3t7sr5o0aKaesrDa6+9lqxv3LgxWd+zZ0+y/sknn0y6JzQHe34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCCrMOP9NN92UrK9duzZZX7ZsWdnavHnzauopLx9//HHZ2ubNm5PrPvHEE8n62bNna+oJrY89PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EFWacv6urq656PQ4dOpSs7969O1kfGRlJ1lPfuR8eHk6ui7jY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUObu6QXM5kvaJmmOpFFJ/e7+QzN7TNJqSX/NFl3n7r+q8FrpjQGom7tbNctVE/65kua6+wEzmynpDUn3SHpA0hl3f6rapgg/0HjVhr/iFX7uXpJUyu6fNrPDkor96RoAdZvUOb+ZLZT0NUkD2VO9ZvZ7M9tiZleVWafHzAbNbLCuTgHkquJh/98WNPuipFclrXf3581stqQPJLmkH2js1OChCq/BYT/QYLmd80uSmX1B0m5Je9x90wT1hZJ2u/vSCq9D+IEGqzb8FQ/7zcwk/UTS4fHBzz4IvKBL0juTbRJAcar5tH+5pN9IeltjQ32StE5St6R2jR32H5X03ezDwdRrsecHGizXw/68EH6g8XI77AcwNRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCavYU3R9I+r9xj7+UPdeKWrW3Vu1Lorda5dnbP1a7YFO/z/+5jZsNuntHYQ0ktGpvrdqXRG+1Kqo3DvuBoAg/EFTR4e8vePsprdpbq/Yl0VutCumt0HN+AMUpes8PoCCFhN/M7jSzI2b2rpk9WkQP5ZjZUTN728wOFj3FWDYN2gkze2fcc21m9msz+1N2O+E0aQX19piZvZ+9dwfN7F8L6m2+mb1sZofN7A9m9kj2fKHvXaKvQt63ph/2m9k0SX+U1ClpSNLrkrrd/VBTGynDzI5K6nD3wseEzexfJJ2RtO3CbEhm9h+STrr7huwP51Xu/u8t0ttjmuTMzQ3qrdzM0t9Rge9dnjNe56GIPf8ySe+6+3vufk7SLyStLKCPlufu+yWdvOjplZK2Zve3auw/T9OV6a0luHvJ3Q9k909LujCzdKHvXaKvQhQR/nmS/jLu8ZBaa8pvl7TXzN4ws56im5nA7AszI2W3Vxfcz8UqztzcTBfNLN0y710tM17nrYjwTzSbSCsNOXzD3f9Z0l2S1mSHt6jOjyQt1tg0biVJG4tsJptZ+jlJ33P3U0X2Mt4EfRXyvhUR/iFJ88c9/rKkYwX0MSF3P5bdnpD0gsZOU1rJ8QuTpGa3Jwru52/c/bi7n3f3UUk/VoHvXTaz9HOSfubuz2dPF/7eTdRXUe9bEeF/XdI1ZvYVM5su6duSdhXQx+eY2YzsgxiZ2QxJ31TrzT68S9Kq7P4qSTsL7OXvtMrMzeVmllbB712rzXhdyEU+2VDGf0maJmmLu69vehMTMLNFGtvbS2PfePx5kb2Z2XZJt2rsW1/HJX1f0n9L+qWkBZL+LOlb7t70D97K9HarJjlzc4N6Kzez9IAKfO/ynPE6l364wg+IiSv8gKAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8E9f/Ex0YKZYOZcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1176b0588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Checking the downloaded data\n",
    "print(\"Shape of training dataset: {}\".format(np.shape(X_train)))\n",
    "print(\"Shape of test dataset: {}\".format(np.shape(X_test)))\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(X_train[0], cmap='gray')\n",
    "\n",
    "print(\"Label for image: {}\".format(y_train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building synthetic data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is very popular for beginner Deep Learning projects. So, to add a twist to the tale, we're going to predict images that can contain 1 to 5 digits. We'll have to change the architecture of our deep learning model for this, but before that, we'll need to generate this dataset first.\n",
    "\n",
    "To generate the synthetic training data, we will first start by randomly picking out up to 5 individual digits out from the MNIST training set. The individual images will be then stacked together, and blanks will be used to make up the number of digits if there were less than 5. By this approach, we could increase the size of our training data. We'll build around 60,000 such examples.\n",
    "\n",
    "While concatenating images together, we'll also build the labels for each image. First, labels for single digits will be arranged in tuples of 5. Labels 0-9 will be used for digits 0-9, and a 10 will be used to indicate a blank.\n",
    "\n",
    "The same approach will be used to build the test data, but using the MNIST test set for individual digits, for 10,000 synthetic test images.\n",
    "\n",
    "\n",
    "\n",
    "Let's write a function that does this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_synth_data(data,labels,dataset_size):\n",
    "    \n",
    "    #Define synthetic image dimensions\n",
    "    synth_img_height = 64\n",
    "    synth_img_width = 64\n",
    "    \n",
    "    #Define synthetic data\n",
    "    synth_data = np.ndarray(shape=(dataset_size,synth_img_height,synth_img_width),\n",
    "                           dtype=np.float32)\n",
    "    \n",
    "    #Define synthetic labels\n",
    "    synth_labels = [] \n",
    "    \n",
    "    #For a loop till the size of the synthetic dataset\n",
    "    for i in range(0,dataset_size):\n",
    "        \n",
    "        #Pick a random number of digits to be in the dataset\n",
    "        num_digits = random.randint(1,5)\n",
    "        \n",
    "        #Randomly sampling indices to extract digits + labels afterwards\n",
    "        s_indices = [random.randint(0,len(data)-1) for p in range(0,num_digits)]\n",
    "        \n",
    "        #stitch images together\n",
    "        new_image = np.hstack([X_train[index] for index in s_indices])\n",
    "        #stitch the labels together\n",
    "        new_label =  [y_train[index] for index in s_indices]\n",
    "        \n",
    "        \n",
    "        #Loop till number of digits - 5, to concatenate blanks images, and blank labels together\n",
    "        for j in range(0,5-num_digits):\n",
    "            new_image = np.hstack([new_image,np.zeros(shape=(mnist_image_height,\n",
    "                                                                   mnist_image_width))])\n",
    "            new_label.append(10) #Might need to remove this step\n",
    "        \n",
    "        #Resize image\n",
    "        new_image = misc.imresize(new_image,(64,64))\n",
    "        \n",
    "        #Assign the image to synth_data\n",
    "        synth_data[i,:,:] = new_image\n",
    "        \n",
    "        #Assign the label to synth_data\n",
    "        synth_labels.append(tuple(new_label))\n",
    "        \n",
    "    \n",
    "    #Return the synthetic dataset\n",
    "    return synth_data,synth_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:36: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    }
   ],
   "source": [
    "#Building the training dataset\n",
    "X_synth_train,y_synth_train = build_synth_data(X_train,y_train,60000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:36: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    }
   ],
   "source": [
    "#Building the test dataset\n",
    "X_synth_test,y_synth_test = build_synth_data(X_test,y_test,10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 4, 1, 9, 7)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAG4NJREFUeJztnW2sVeWVx/9LUEEUEXnpFShvUoGqBaSIYWIRC3U6bfFDa2ybCTMh4UtnYjOdVJ1JJu1kJrFp0nY+mCZ0dMqHtmptOxjaVA1gEWJFGAHl5V7e4crLBQRFtCp2zYez73Y9y3vuPfees599Tp//L7k5a59nn3vW3eesu9d61vOsJaoKQkhaXFK2AoSQ+NDwCUkQGj4hCULDJyRBaPiEJAgNn5AEoeETkiB1Gb6I3CUi7SKyT0QeaJRShJBikYEu4BGRQQA6ACwG0AngJQBfVdVdjVOPEFIEg+t47TwA+1T1AACIyGMAlgKoavgiwmWChBSMqkpf59Tj6o8DcNQcd2bPEUKanHru+D39V/nIHV1EVgBYUcf7EEIaTD2G3wlggjkeD+CYP0lVVwJYCdDVJ6RZqMfVfwnANBGZLCKXAbgXwFONUYsQUiQDvuOr6kUR+QcATwMYBOBRVd3ZMM0IIYUx4HTegN6Mrj4hhVP0rD4hpEWh4ROSIDR8QhKEhk9IgtDwCUkQGj4hCULDJyRBaPiEJAgNn5AEoeETkiA0fEIShIZPSILQ8AlJEBo+IQlCwyckQWj4hCQIDZ+QBKHhE5IgNHxCEoSGT0iC1FNXv3Ta2tqC4/Pnz+fyW2+9FVudmrj00ktz+brrrgvGTpw4kcvvvvtuNJ08Ih/WavzYxz6Wy2PHjg3Oa29vz+V33nmneMVIw+Adn5AEoeETkiA0fEISpKVj/Dlz5gTHNubct29fbHVqYtiwYbm8cOHCYOx3v/tdLp86dSqWSh/Bxvj2Gi9ZsiQ47/vf/34ud3Z2Fq8YaRh93vFF5FER6RKRV81zI0XkWRHZmz1eU6yahJBGUour/1MAd7nnHgCwVlWnAVibHRNCWoQ+XX1V3SAik9zTSwEszORVAJ4DcH8D9aqJT37yk8HxG2+8kcvN6upfeeWVuTx//vxgbNOmTblcpqt/ySUf3g9uuummXL777ruD837yk5/kMl391mKgk3tjVfU4AGSPYxqnEiGkaAqf3BORFQBWFP0+hJDaGajhnxSRNlU9LiJtALqqnaiqKwGsBBrTJtu6odZtBoBrr702l+3MdKZHvW/dEEaOHJnLU6dODcas/mWGKvYajxo1Kpevuuqq4Lyrr746mk6ksQzU1X8KwLJMXgZgdWPUIYTEoJZ03i8AvADgBhHpFJHlAB4CsFhE9gJYnB0TQlqEWmb1v1pl6M4G60IIiUTLrdyzsbuP8W082qzYOH7KlCnBmI3/y2TQoEG5bK/p8OHDg/P8MWkduFafkASh4ROSIC3n6lusSwoAgwc3359jU2NA6N4PHTq013PLwqbpbLETnyJtFmya8ZZbbgnGRowYkctPP/10LqdeOKQ5vmmEkKjQ8AlJEBo+IQnSfEFxH9g408fEzRjj+7jYxviXX355bHV6xOto42Jf0LQZsWnFxYsXB2N2WfTzzz+fy4zxCSHJQcMnJEGazzeuA1uzvlm47LLLguMJEybksk9HloUPmT7xiU/ksl1pePHixeC8999/v1jFquCv6Y033pjLc+fODcZsXwCfPo1Jb/0Urrnmw8p1o0ePzmUfjmzdurXqWH/hHZ+QBKHhE5IgLefq24IaH3zwQTBmXcBmKcQxZMiQ4Ni7ec2Az4bMnj07l+2MuW/r9ac//alYxargr+lnPvOZXJ4xY0YwZr8j/nUxsRkce32BUGdbh/HMmTPBeR0dHblMV58Q0m9o+IQkCA2fkARpuRjfxu4+HVZmuqYavliIbTXt5x38nEUsfDpv4sSJuWxbfp0+fTo47+233y5WsSpcccUVwfGsWbNyedy4ccHY66+/nssxi4P6eZOPf/zjuTxv3rxgzLYps/MVXV1hDVtbqMWP9Rfe8QlJEBo+IQnScq6+dUutGwqEK8mapWiEXfkGhCuzvGtfVnqsVnwK6cKFC6Xo4T/3MWM+bOTkP3ebRhs/fnwu21VwReDDkc997nO5bN15IAz/bIjgaxracGHPnj116cc7PiEJQsMnJEFo+IQkSEvH+D6OKisd1ht255U/Pnv2bDBWVozv4+Jq8yNvvvlmcFxWjO97+Nl5kz//+c/BWG+74orEp3GXLFmSy7fddltNv8P/nT5VWQ+1tNCaICLrRWS3iOwUkfuy50eKyLMisjd7vKav30UIaQ5qcfUvAviWqs4AMB/AN0RkJoAHAKxV1WkA1mbHhJAWoJbeeccBHM/k8yKyG8A4AEsBLMxOWwXgOQD3F6JlC2Pr1wHhDrdm2e3ma//ZtlnWdf7jH/8YnHf+/PliFavC5MmTg2Ob3vNpOpsCi4ktAAKEqcSB0sjdhf2a3BORSQBmA3gRwNjsn0L3P4cx1V9JCGkmap7cE5ErAfwKwDdV9c1aF8iIyAoAKwamHiGkCGq644vIpagY/c9U9dfZ0ydFpC0bbwPQ464BVV2pqnNVdW5P44SQ+PR5x5fKrf0RALtV9Qdm6CkAywA8lD2uLkTDFse3vv5LivHfeuutYhWrgm8vbmP8zZs3B2NltfL2/Qjsjsda8V61LzJaD7W4+gsA/C2AV0RkW/bcv6Bi8E+IyHIARwB8pWFaEUIKpZZZ/Y0AqgX0dzZWHUJIDFpu5V4rYHdYTZo0KRizBRT8KrOyikH2tnLPFgvxrn3MlZJ2FZut+w8AnZ2duexdfbvaLWYo5dN59jth9QWAvXv35vIbb7yRy7feemtwXiN3nHKtPiEJQsMnJEHo6heA3Rgyffr0YOz48eO57It0lDUD7UMOW9DEuvpl9SYAwhWQ/ppaV/mFF14IxmyoFXNTkS0OAoRh0f79+4OxJ598sscxHy40Et7xCUkQGj4hCULDJyRBGOMXgF1h5WO9I0eO5LKP6X2sHQufJmqW9t0We638qrh169bl8tGjR4Ox9vb2XLZ9F3yRi0bvNPRFYo4dO5bLfgWknZewqT6fPvUrPeuBd3xCEoSGT0iC0NUvAOvq+40VJ0+ezGW/2cTXtIuFTT8CH61b3wzYGna+VdqpU6dy2a8mPHz4cC4vWrQol339PRsSFMGWLVtyec2aNcHYa6+9lsu2pqTfPNXIcIR3fEIShIZPSILQ8AlJkJaL8W3qybcibhZsjOzj5xMnTuTyxYsXg7Gy6tT7WNKmouwy3dh9C+xnbfsReH3tNfUpUdva2y7f9cUvOzo6crkRS5N9SnT79u25vGnTpqqvs7sJ/fyQ3blXL7zjE5IgNHxCEqQ5feVesO6PTzv5llRlccMNN+Sydxtt6qZMbNrI7wKzrvR7772Xy2+//Xbxihmsjtdff30u+xDp0KFDueyvtw0RbrzxxlzesWNHcN4f/vCHqr9/IPr67+bOnTtr+h1WX/v7gDBtWS+84xOSIDR8QhKk5Vx964b6jqTNQiu4+nbGvFldfTszPnXq1Fz2rrhdneevty12MnPmzFy2n5F/r4G6+vaaelffbx6qhi3H7jMDNkNRL7zjE5IgNHxCEoSGT0iCtFyMb2vP+xbUA43N6sWvJLPtks6dOxeM2TiwrDr6Hh+P2r/Hrtbz6aWisSszbezrV7DZuQdfVKRazOx3ujW6CIqfa6h1fsS2APc6vfPOO/UrltHnJykiQ0Rks4hsF5GdIvLd7PnJIvKiiOwVkcdFpHGNvQghhVLLv/B3ASxS1U8BmAXgLhGZD+B7AH6oqtMAnAWwvDg1CSGNpJbeeQqgu/jXpdmPAlgE4GvZ86sAfAfAjxuvYoh1j6+++upgrBld/ddffz0Ys+5rma6+dUW9Hvbvse5l7E1RdoOTddl9wRKrow9HfLfibvzn0ghXv1rrMaD39l32ddbVH2i4UAs1BW0iMijrlNsF4FkA+wGcU9VuS+sEMK7a6wkhzUVNhq+qH6jqLADjAcwDMKOn03p6rYisEJEtIrKlp3FCSHz6NU2rqucAPAdgPoARItLt+40HcKzKa1aq6lxVnVuPooSQxtFn0CYiowG8r6rnRGQogM+iMrG3HsCXATwGYBmA1UUq2o2N4XyhAluHPGafN1+jfezYsbnsiy7Y9Fgj2x73F3t9fHxrr7GN6/31LppqhSd9Os/Wm/c62qW+dnfbgQMHgvMaEePba+pr4Pf2+23xUDs/5Jd3N7Kufi2zNW0AVonIIFQ8hCdUdY2I7ALwmIj8B4CXATzSMK0IIYVSy6z+DgCze3j+ACrxPiGkxWi5lXu90ciaZP3BryC0O8J27doVjNm0TlnpR09vK8JsSi22q2+xYZEvuGJ3EPrWVTfddFMuHzx4MJdrLYwxUHz6rjdX36albV1A3/Lbti+vF67VJyRBaPiEJEjLufp2dtfPituNFzFn9X1HXLvpxdaDA0J3zZerLmuWv7f3tRtbYrv6NqNgZ/htGzIgvKa2Zh0Qus42w2K70gKN+b7Y3+E73fb2+21oaL9LNjQBGlvenHd8QhKEhk9IgtDwCUmQlovxberM78SKWQzSvrdvd23TdD7FaOM0n87zxRWLxMb1fgdbtV1mMedNgDA1Z9OK+/fvD86z19GvorRzQnZHXhHtwOz18YUx7TyEn1Ox7bzsTkn/Oxp5/XnHJyRBaPiEJMhflKvfyJpkfWHdNZsyAkK3zrv6NiXmXf2YhS7stfMpMPu3NboWXX+wm1esq+832FgX2P8t9noX7epbfMrRri70TJgwIZer6QvQ1SeE1AkNn5AEoeETkiAtEePbmHPUqFFVz/O10ovEpona2tqCMTvX4Isn2BjOL4H1qagisdfUv2+ZBUIsds7DFzS12N5/S5cuDcbsNT5z5kwuFz130dHRERzb76ZP29rCLVbHRrbF9vCOT0iC0PAJSZCWc/XtKjOf3rhw4ULVsUZjXc/rrrsuGLMrCL2rb91qn77zBT2KxF5T3268WVx966b35urbUOvuu+8Oxmwa0LYzK8LVt9857+rbFK//3OnqE0KiQMMnJEFawtW3q8yGDx+ey371VcyVe1YPn2mwhTjuuOOOYMyuLPP14WJu0mmG9+0LGybZVY4333xzcJ5tqWXdZj927FiPrR8KobeWWTYLAYQrPw8fPlzT76gX3vEJSRAaPiEJQsMnJEFaLsa3Nch9jN/IFkN9YWN1WywRCHW85557gjG748yn0UiITc/a3W0LFiwIzrNzOz7tZ4ud+pZUZeFXetoYf/Pmzbnc246+eqn5jp+1yn5ZRNZkx5NF5EUR2Ssij4tIed0WCCH9oj+u/n0Adpvj7wH4oapOA3AWwPJGKkYIKY6aXH0RGQ/gbwD8J4B/ksrSrkUAvpadsgrAdwD8uAAdA1ffusre1S/SNfLYVJxN3wHh6rxPf/rTwVi1DrDNil1xFjNdCoTpLLsacvr06cF5va2Ks8Us7Mq92NjVkOPGjQvGbGrYhiZFtlir9Y7/IwDfBtC9zvFaAOdUtVuzTgDjenohIaT56NPwReQLALpUdat9uodTe1wcLyIrRGSLiGwZoI6EkAZTi6u/AMCXROTzAIYAGI6KBzBCRAZnd/3xAHpcFqWqKwGsBAARiVufmRDSI30avqo+COBBABCRhQD+WVW/LiK/BPBlAI8BWAZgdVFK2rjY7rbyMX7MttNWJ1/00y6B7a1mfbO0ye4NG9f79tRFY9OztmipX7Jrd9r5nYX2dTHngDz2OzJx4sRgzM5R2H55RRYLqWcBz/2oTPTtQyXmf6QxKhFCiqZfC3hU9TkAz2XyAQDzGq8SIaRoWm7lni3OUGY6z7phviWyTef51XlWZ9/yy4cMzYBNqdmdbjGwrr5tJ+ULlthr6guw2PCkTFffhqi+D4MNp+wOwiKLyTTfN40QUjg0fEISpCVcfTtLbt1o77rF3KTT1dWVyxs3bgzGbrnllly2nVCBcCWZ3cwDfHQFYCz8TLg9tm50zOsLhLPde/furaqHDbX8akh7vcvMotjv7fXXXx+MHTlyJJdjlYjnHZ+QBKHhE5IgNHxCEqQlYny748rGwT7WK7I4ocfWPN+0aVMwZtNNvm3z8ePHc9nH+HbnYUxaIca3rbG9HjZl5693M8b4U6dODcY2bNiQy4zxCSGFQcMnJEFawtW3K9qs229rsgEfXclXJNbdfOWVV4Ix6yrv2rUrGLMpyAkTJgRjdpWf/R1FrOCy4YivRW/fz25ysa53bNrb23P50UcfDcbs9+OLX/xiMGZDhDKxdfZ8d2Ib/sUKR3jHJyRBaPiEJAgNn5AEaYkY38a7Nsb3qY8iCxd4bKy+c+fOYMz2PxsyZEgwZvum3XvvvcGYTVUWHePbtJfv5VYtxi9zd9v+/ftz+eGHHw7G7FzJ7bffHozZpb4x8SlSG+P7pdknTpzI5VjzVLzjE5IgNHxCEqQlXH3r3lvX2dYgB+Km82xYYd3hno4t1l32hThsmsemqIoIYeyOx952BVqX1buvMbFhnQ/x7LXyY3blXkxswRgAmDlzZi77NLQNDYssvmHhHZ+QBKHhE5IgLeHq23pltnWV3SgDxHX1B4qtr+ZdfdtKybriRazmstfKvpfHrib07amaEe/qe7c6FvZ7CgBz5szJ5ZMnTwZjPmSNAe/4hCQIDZ+QBKHhE5IgzR+0IYxBbTFFnzaLuXJvoPTWdtrOZRSdOrPzBr3V87dxfTPW/QfCFK+fDylrtaEvCDJt2rRc3rNnTzBmV+7FoibDF5FDAM4D+ADARVWdKyIjATwOYBKAQwDuUdW4zdUIIQOiP//C71DVWao6Nzt+AMBaVZ0GYG12TAhpAepx9ZcCWJjJq1DpqXd/nfr0C5++i7XqqR5sOOJrBNrQpejUpP39vYVIVsdmTZfaTUY+jVYWviOuTdX64iyxaxkCtd/xFcAzIrJVRFZkz41V1eMAkD2OKUJBQkjjqfWOv0BVj4nIGADPisiePl+Rkf2jWNHniYSQaNR0x1fVY9ljF4DfoNIe+6SItAFA9thV5bUrVXWumRsghJRMn3d8ERkG4BJVPZ/JSwD8O4CnACwD8FD2uLooJW0MauUy66QPFDsP4Zfs2jbUMWP83t7Lphyb9Xo3Y4w/ZcqU4NgufbaFQ4FyYvxaXP2xAH6T5ZUHA/i5qv5eRF4C8ISILAdwBMBXilOTENJI+jR8VT0A4FM9PH8GwJ1FKEUIKZaWWLlnV1+99tpruexXPLVCOs+6y/ZvAcLdhkX/Lda9PHPmTDBm3dR9+/blcswWZf1h9OjRuVxmsRC7qnTGjBnBmG3l3dnZGU2najTnGkxCSKHQ8AlJEBo+IQnSEjG+jUdtXHzs2LHgvFaI8W3qzPZMA5onxrcpU1vPnjF+79gY3xbXBMJKQM3wveUdn5AEoeETkiAt4epbF3PDhg25fPTo0TLUqQvrRu/YsSMY6+rqcdVzIdgU6fr164Oxc+fO5fL27dtzuVl25/nioHbn28GDB4OxmIU47Oo835bMhnFnz5ZftoJ3fEIShIZPSIK0hKtvZ6CfeeaZXD59+nRwXivM6ltXf9u2bcFYzFlz6wL/9re/DcY2bdqUy0eOHMnlZrm+tjYhELrYu3fvDsZiboCxLdBGjRoVjB04cCCX7WassuAdn5AEoeETkiA0fEISpCVifLujza4ka0VsnOxX7sXEpuY6OjpK02Mg+BbU9pr6GD9m8RAb4w8dOjQYs62wyyi84eEdn5AEoeETkiAt4eoTYvFutE2P+fZUMbGuvk992oImzQDv+IQkCA2fkASh4ROSIIzxScthW3cD4a7GQ4cORdbmQ4YNG5bLfvk1Y3xCSOnQ8AlJELr6pOU4f/58cPz888/ncszCGx67GtKHHL7OXtnUdMcXkREi8qSI7BGR3SJym4iMFJFnRWRv9nhN0coSQhpDra7+fwH4vapOR6Wd1m4ADwBYq6rTAKzNjgkhLUAt3XKHA7gdwN8BgKq+B+A9EVkKYGF22ioAzwG4vwglCbF4V3/jxo25XGZHX+vq28IbwEfbpZVNLXf8KQBOAfgfEXlZRP47a5c9VlWPA0D2OKZAPQkhDaQWwx8MYA6AH6vqbAAX0A+3XkRWiMgWEdkyQB0JIQ2mFsPvBNCpqi9mx0+i8o/gpIi0AUD22GNtaFVdqapzVXVuIxQmhNRPnzG+qp4QkaMicoOqtgO4E8Cu7GcZgIeyx9WFakpIht/5VmZcb7Htr9etWxeMNVv7sVrz+P8I4GcichmAAwD+HhVv4QkRWQ7gCICvFKMiIaTR1GT4qroNQE+u+p2NVYcQEgOJWStdRJqjMDshBWA36di2XgBw8uTJXLa9FYpAVftsGcy1+oQkCA2fkASh4ROSIIzxCfkLgzE+IaRHaPiEJEjsQhynARwGMCqTy6QZdACoh4d6hPRXj4m1nBQ1xs/fVGRL2Wv3m0EH6kE9ytKDrj4hCULDJyRByjL8lSW9r6UZdACoh4d6hBSiRykxPiGkXOjqE5IgUQ1fRO4SkXYR2Sci0aryisijItIlIq+a56KXBxeRCSKyPitRvlNE7itDFxEZIiKbRWR7psd3s+cni8iLmR6PZ/UXCkdEBmX1HNeUpYeIHBKRV0RkW3eZuJK+I1FK2UczfBEZBOBhAH8NYCaAr4rIzEhv/1MAd7nnyigPfhHAt1R1BoD5AL6RXYPYurwLYJGqfgrALAB3ich8AN8D8MNMj7MAlhesRzf3oVKyvZuy9LhDVWeZ9FkZ35E4pexVNcoPgNsAPG2OHwTwYMT3nwTgVXPcDqAtk9sAtMfSxeiwGsDiMnUBcAWA/wNwKyoLRQb39HkV+P7jsy/zIgBrAEhJehwCMMo9F/VzATAcwEFkc29F6hHT1R8H4Kg57syeK4tSy4OLyCQAswG8WIYumXu9DZUiqc8C2A/gnKp2F7CL9fn8CMC3AXRXp7i2JD0UwDMislVEVmTPxf5copWyj2n4Pe0YSjKlICJXAvgVgG+q6ptl6KCqH6jqLFTuuPMAzOjptCJ1EJEvAOhS1a326dh6ZCxQ1TmohKLfEJHbI7ynp65S9v0hpuF3AphgjscDKLOTYE3lwRuNiFyKitH/TFV/XaYuAKCq51DpgjQfwAgR6d6/EePzWQDgSyJyCMBjqLj7PypBD6jqseyxC8BvUPlnGPtzqauUfX+IafgvAZiWzdheBuBeAE9FfH/PU6iUBQcilQcXEQHwCIDdqvqDsnQRkdEiMiKThwL4LCqTSOsBfDmWHqr6oKqOV9VJqHwf1qnq12PrISLDROSqbhnAEgCvIvLnoqonABwVkRuyp7pL2Tdej6InTdwkxecBdKAST/5rxPf9BYDjAN5H5b/qclRiybUA9maPIyPo8VeouK07AGzLfj4fWxcANwN4OdPjVQD/lj0/BcBmAPsA/BLA5RE/o4UA1pShR/Z+27Ofnd3fzZK+I7MAbMk+m/8FcE0RenDlHiEJwpV7hCQIDZ+QBKHhE5IgNHxCEoSGT0iC0PAJSRAaPiEJQsMnJEH+H7w2jgGBltfZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1176c3d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#checking a sample\n",
    "plt.figure()\n",
    "plt.imshow(X_synth_train[232], cmap='gray')\n",
    "\n",
    "y_synth_train[232]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like things work as we expect them to. Let's prepare the datset and labels so that keras can handle them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparatory Preprocessing\n",
    "\n",
    "#### Preprocessing Labels for model\n",
    "\n",
    "The labels are going to be encoded to \"One Hot\" arrays, to make them compatible with Keras. Note that, as the our Deep Learning model will have 5 classifiers, we'll need 5 such One Hot arrays, one for each digit position in the image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Converting labels to One-hot representations of shape (set_size,digits,classes)\n",
    "possible_classes = 11\n",
    "\n",
    "def convert_labels(labels):\n",
    "    \n",
    "    #As per Keras conventions, the multiple labels need to be of the form [array_digit1,...5]\n",
    "    #Each digit array will be of shape (60000,11)\n",
    "    \n",
    "    #Code below could be better, but cba for now. \n",
    "    \n",
    "    #Declare output ndarrays\n",
    "    dig0_arr = np.ndarray(shape=(len(labels),possible_classes))\n",
    "    dig1_arr = np.ndarray(shape=(len(labels),possible_classes))\n",
    "    dig2_arr = np.ndarray(shape=(len(labels),possible_classes))\n",
    "    dig3_arr = np.ndarray(shape=(len(labels),possible_classes)) #5 for digits, 11 for possible classes  \n",
    "    dig4_arr = np.ndarray(shape=(len(labels),possible_classes))\n",
    "    \n",
    "    for index,label in enumerate(labels):\n",
    "        \n",
    "        #Using np_utils from keras to OHE the labels in the image\n",
    "        dig0_arr[index,:] = np_utils.to_categorical(label[0],possible_classes)\n",
    "        dig1_arr[index,:] = np_utils.to_categorical(label[1],possible_classes)\n",
    "        dig2_arr[index,:] = np_utils.to_categorical(label[2],possible_classes)\n",
    "        dig3_arr[index,:] = np_utils.to_categorical(label[3],possible_classes)\n",
    "        dig4_arr[index,:] = np_utils.to_categorical(label[4],possible_classes)\n",
    "        \n",
    "    return [dig0_arr,dig1_arr,dig2_arr,dig3_arr,dig4_arr]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels = convert_labels(y_synth_train)\n",
    "test_labels = convert_labels(y_synth_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 11)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Checking the shape of the OHE array for the first digit position\n",
    "np.shape(train_labels[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np_utils.to_categorical(y_synth_train[234][0],11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing Images for model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function below will pre-process the images so that they can be handled by keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prep_data_keras(img_data):\n",
    "    \n",
    "    #Reshaping data for keras, with tensorflow as backend\n",
    "    img_data = img_data.reshape(len(img_data),64,64,1)\n",
    "    \n",
    "    #Converting everything to floats\n",
    "    img_data = img_data.astype('float32')\n",
    "    \n",
    "    #Normalizing values between 0 and 1\n",
    "    img_data /= 255\n",
    "    \n",
    "    return img_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_images = prep_data_keras(X_synth_train)\n",
    "test_images = prep_data_keras(X_synth_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 64, 64, 1)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(train_images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 64, 64, 1)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(test_images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing relevant keras modules\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Dropout, Activation, Flatten, Input\n",
    "from keras.layers import Convolution2D, MaxPooling2D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use a Convolutional Neural Network for our network. \n",
    "\n",
    "Starting with a 2D Convolutional layer, we'll use ReLU activations after every Convolutional Layer. \n",
    "\n",
    "After the second CovLayer + ReLU, we'll add 2DMaxPooling, and a dropout to make the model robust to overfitting. A flattening layer will be added to make the data ready for classification layers, which were in the form of Dense Layers, of the same size as the no. of classes (11 for us), activated using softmax to give us the probability of each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:23: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), padding=\"same\")`\n",
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:25: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3))`\n",
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:46: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor(\"in..., outputs=[<tf.Tenso...)`\n",
      "/Users/melvynnfernandez/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:53: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      " 1920/60000 [..............................] - ETA: 23:04 - loss: 8.9949 - dense_2_loss: 2.3609 - dense_3_loss: 2.2989 - dense_4_loss: 1.9205 - dense_5_loss: 1.4320 - dense_6_loss: 0.9826 - dense_2_acc: 0.1562 - dense_3_acc: 0.2328 - dense_4_acc: 0.3958 - dense_5_acc: 0.5938 - dense_6_acc: 0.7714"
     ]
    }
   ],
   "source": [
    "#Building the model\n",
    "\n",
    "batch_size = 128\n",
    "nb_classes = 11\n",
    "nb_epoch = 12\n",
    "\n",
    "#image input dimensions\n",
    "img_rows = 64\n",
    "img_cols = 64\n",
    "img_channels = 1\n",
    "\n",
    "#number of convulation filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "pool_size = (2, 2)\n",
    "# convolution kernel size\n",
    "kernel_size = (3, 3)\n",
    "\n",
    "#defining the input\n",
    "inputs = Input(shape=(img_rows,img_cols,img_channels))\n",
    "\n",
    "#Model taken from keras example. Worked well for a digit, dunno for multiple\n",
    "cov = Convolution2D(nb_filters,kernel_size[0],kernel_size[1],border_mode='same')(inputs)\n",
    "cov = Activation('relu')(cov)\n",
    "cov = Convolution2D(nb_filters,kernel_size[0],kernel_size[1])(cov)\n",
    "cov = Activation('relu')(cov)\n",
    "cov = MaxPooling2D(pool_size=pool_size)(cov)\n",
    "cov = Dropout(0.25)(cov)\n",
    "cov_out = Flatten()(cov)\n",
    "\n",
    "\n",
    "#Dense Layers\n",
    "cov2 = Dense(128, activation='relu')(cov_out)\n",
    "cov2 = Dropout(0.5)(cov2)\n",
    "\n",
    "\n",
    "\n",
    "#Prediction layers\n",
    "c0 = Dense(nb_classes, activation='softmax')(cov2)\n",
    "c1 = Dense(nb_classes, activation='softmax')(cov2)\n",
    "c2 = Dense(nb_classes, activation='softmax')(cov2)\n",
    "c3 = Dense(nb_classes, activation='softmax')(cov2)\n",
    "c4 = Dense(nb_classes, activation='softmax')(cov2)\n",
    "\n",
    "#Defining the model\n",
    "model = Model(input=inputs,output=[c0,c1,c2,c3,c4])\n",
    "\n",
    "#Compiling the model\n",
    "model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
    "\n",
    "#Fitting the model\n",
    "model.fit(train_images,train_labels,batch_size=batch_size,nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(test_images, test_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(test_images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.shape(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(predictions[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.shape(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define a custom to calculate accuracy for predicting individual digits, as well as for predicting complete sequence of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_acc(predictions,real_labels):\n",
    "    \n",
    "    individual_counter = 0\n",
    "    global_sequence_counter = 0\n",
    "    for i in range(0,len(predictions[0])):\n",
    "        #Reset sequence counter at the start of each image\n",
    "        sequence_counter = 0 \n",
    "        \n",
    "        for j in range(0,5):\n",
    "            if np.argmax(predictions[j][i]) == np.argmax(real_labels[j][i]):\n",
    "                individual_counter += 1\n",
    "                sequence_counter +=1\n",
    "        \n",
    "        if sequence_counter == 5:\n",
    "            global_sequence_counter += 1\n",
    "         \n",
    "    ind_accuracy = individual_counter/50000.0\n",
    "    global_accuracy = global_sequence_counter/10000.0\n",
    "    \n",
    "    return ind_accuracy,global_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind_acc,glob_acc = calculate_acc(predictions,test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The individual accuracy is {} %\".format(ind_acc*100))\n",
    "print(\"The sequence prediction accuracy is {} %\".format(glob_acc*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Printing some examples of real and predicted labels\n",
    "for i in random.sample(range(0,10000),5):\n",
    "    \n",
    "    actual_labels = []\n",
    "    predicted_labels = []\n",
    "    \n",
    "    for j in range(0,5):\n",
    "        actual_labels.append(np.argmax(test_labels[j][i]))\n",
    "        predicted_labels.append(np.argmax(predictions[j][i]))\n",
    "        \n",
    "    print(\"Actual labels: {}\".format(actual_labels))\n",
    "    print(\"Predicted labels: {}\\n\".format(predicted_labels))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that model achieved good accuracy, with around 98.5% accurate for identifying individual digits or blanks, and around 92.8% for identifying whole sequences."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}