1 Multi-layer Fully Connected Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/keras-logo-small.jpg\" width=\"20%\" />\n",
    "\n",
    "## Keras: Deep Learning library for Theano and TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. \n",
    "\n",
    ">It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\n",
    "ref: https://keras.io/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"kaggle\"></a>\n",
    "### Kaggle Challenge Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The Otto Group is one of the world’s biggest e-commerce companies, A consistent analysis of the performance of products is crucial. However, due to diverse global infrastructure, many identical products get classified differently.\n",
    "For this competition, we have provided a dataset with 93 features for more than 200,000 products. The objective is to build a predictive model which is able to distinguish between our main product categories. \n",
    "Each row corresponds to a single product. There are a total of 93 numerical features, which represent counts of different events. All features have been obfuscated and will not be defined any further.\n",
    "\n",
    "https://www.kaggle.com/c/otto-group-product-classification-challenge/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### For this section we will use the Kaggle Otto Group Challenge Data. You will find these data in \n",
    "`data/kaggle_ottogroup/` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "This algorithm has nothing to do with the canonical _linear regression_, but it is an algorithm that allows us to solve problems of classification (supervised learning). \n",
    "\n",
    "In fact, to estimate the dependent variable, now we make use of the so-called **logistic function** or **sigmoid**. \n",
    "\n",
    "It is precisely because of this feature we call this algorithm logistic regression.\n",
    "\n",
    "![](imgs/sigmoid.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from kaggle_data import load_data, preprocess_data, preprocess_labels\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 classes\n",
      "93 dims\n"
     ]
    }
   ],
   "source": [
    "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
    "X_train, scaler = preprocess_data(X_train)\n",
    "Y_train, encoder = preprocess_labels(labels)\n",
    "\n",
    "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
    "X_test, _ = preprocess_data(X_test, scaler)\n",
    "\n",
    "nb_classes = Y_train.shape[1]\n",
    "print(nb_classes, 'classes')\n",
    "\n",
    "dims = X_train.shape[1]\n",
    "print(dims, 'dims')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Class_1', 'Class_2', 'Class_3', 'Class_4', 'Class_5', 'Class_6',\n",
       "       'Class_7', 'Class_8', 'Class_9'], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  1.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  1., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  1.,  0.],\n",
       "       ..., \n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  1.,  0.,  0.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train  # one-hot encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Theano"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import theano as th\n",
    "import theano.tensor as T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1\n",
      "Epoch 2\n",
      "Epoch 3\n",
      "Epoch 4\n",
      "Epoch 5\n",
      "Epoch 6\n",
      "Epoch 7\n",
      "Epoch 8\n",
      "Epoch 9\n",
      "Epoch 10\n",
      "target values for Data:\n",
      "[ 0.  0.  0. ...,  0.  0.  0.]\n",
      "prediction on training set:\n",
      "[False False False ...,  True False  True]\n"
     ]
    }
   ],
   "source": [
    "#Based on example from DeepLearning.net\n",
    "rng = np.random\n",
    "N = 400\n",
    "feats = 93\n",
    "training_steps = 10\n",
    "\n",
    "# Declare Theano symbolic variables\n",
    "x = T.matrix(\"x\")\n",
    "y = T.vector(\"y\")\n",
    "w = th.shared(rng.randn(feats), name=\"w\")\n",
    "b = th.shared(0., name=\"b\")\n",
    "\n",
    "# Construct Theano expression graph\n",
    "p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b))         # Probability that target = 1\n",
    "prediction = p_1 > 0.5                          # The prediction thresholded\n",
    "xent = -y * T.log(p_1) - (1-y) * T.log(1-p_1)   # Cross-entropy loss function\n",
    "cost = xent.mean() + 0.01 * (w ** 2).sum()      # The cost to minimize\n",
    "gw, gb = T.grad(cost, [w, b])                   # Compute the gradient of the cost\n",
    "                                                \n",
    "\n",
    "# Compile\n",
    "train = th.function(\n",
    "          inputs=[x,y],\n",
    "          outputs=[prediction, xent],\n",
    "          updates=((w, w - 0.1 * gw), (b, b - 0.1 * gb)),\n",
    "          allow_input_downcast=True)\n",
    "predict = th.function(inputs=[x], outputs=prediction, allow_input_downcast=True)\n",
    "\n",
    "#Transform for class1\n",
    "y_class1 = []\n",
    "for i in Y_train:\n",
    "    y_class1.append(i[0])\n",
    "y_class1 = np.array(y_class1)\n",
    "\n",
    "# Train\n",
    "for i in range(training_steps):\n",
    "    print('Epoch %s' % (i+1,))\n",
    "    pred, err = train(X_train, y_class1)\n",
    "\n",
    "print(\"target values for Data:\")\n",
    "print(y_class1)\n",
    "print(\"prediction on training set:\")\n",
    "print(predict(X_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "learning_rate = 0.01\n",
    "training_epochs = 25\n",
    "display_step = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# tf Graph Input\n",
    "x = tf.placeholder(\"float\", [None, dims]) \n",
    "y = tf.placeholder(\"float\", [None, nb_classes])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Placeholder:0' shape=(?, 93) dtype=float32>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model (Introducing Tensorboard)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Construct (linear) model\n",
    "with tf.name_scope(\"model\") as scope:\n",
    "    # Set model weights\n",
    "    W = tf.Variable(tf.zeros([dims, nb_classes]))\n",
    "    b = tf.Variable(tf.zeros([nb_classes]))\n",
    "    activation = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax\n",
    "\n",
    "    # Add summary ops to collect data\n",
    "    w_h = tf.summary.histogram(\"weights_histogram\", W)\n",
    "    b_h = tf.summary.histogram(\"biases_histograms\", b)\n",
    "    tf.summary.scalar('mean_weights', tf.reduce_mean(W))\n",
    "    tf.summary.scalar('mean_bias', tf.reduce_mean(b))\n",
    "\n",
    "# Minimize error using cross entropy\n",
    "# Note: More name scopes will clean up graph representation\n",
    "with tf.name_scope(\"cost_function\") as scope:\n",
    "    cross_entropy = y*tf.log(activation)\n",
    "    cost = tf.reduce_mean(-tf.reduce_sum(cross_entropy,reduction_indices=1))\n",
    "    # Create a summary to monitor the cost function\n",
    "    tf.summary.scalar(\"cost_function\", cost)\n",
    "    tf.summary.histogram(\"cost_histogram\", cost)\n",
    "\n",
    "with tf.name_scope(\"train\") as scope:\n",
    "    # Set the Optimizer\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('Accuracy') as scope:\n",
    "    correct_prediction = tf.equal(tf.argmax(activation, 1), tf.argmax(y, 1))\n",
    "    # Calculate accuracy\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "    # Create a summary to monitor the cost function\n",
    "    tf.summary.scalar(\"accuracy\", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning in a TF Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LOGDIR = \"/tmp/logistic_logs\"\n",
    "import os, shutil\n",
    "if os.path.isdir(LOGDIR):\n",
    "    shutil.rmtree(LOGDIR)\n",
    "os.mkdir(LOGDIR)\n",
    "\n",
    "# Plug TensorBoard Visualisation \n",
    "writer = tf.summary.FileWriter(LOGDIR, graph=tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model/weights_histogram:0\n",
      "model/biases_histograms:0\n",
      "model/mean_weights:0\n",
      "model/mean_bias:0\n",
      "cost_function/cost_function:0\n",
      "cost_function/cost_histogram:0\n",
      "Accuracy/accuracy:0\n",
      "Tensor(\"add:0\", shape=(), dtype=string)\n"
     ]
    }
   ],
   "source": [
    "for var in tf.get_collection(tf.GraphKeys.SUMMARIES):\n",
    "    print(var.name)\n",
    "    \n",
    "summary_op = tf.summary.merge_all()\n",
    "print('Summary Op: ' + summary_op)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy epoch 0:0.6649530529975891\n",
      "accuracy epoch 1:0.6652600765228271\n",
      "accuracy epoch 2:0.6657125353813171\n",
      "accuracy epoch 3:0.6659550070762634\n",
      "accuracy epoch 4:0.6662943959236145\n",
      "accuracy epoch 5:0.6666014194488525\n",
      "accuracy epoch 6:0.666811466217041\n",
      "accuracy epoch 7:0.6671023964881897\n",
      "accuracy epoch 8:0.6674579381942749\n",
      "accuracy epoch 9:0.6678457856178284\n",
      "accuracy epoch 10:0.6680881977081299\n",
      "accuracy epoch 11:0.6682498455047607\n",
      "accuracy epoch 12:0.668459951877594\n",
      "accuracy epoch 13:0.6687508225440979\n",
      "accuracy epoch 14:0.6690417528152466\n",
      "accuracy epoch 15:0.6692517995834351\n",
      "accuracy epoch 16:0.6695103645324707\n",
      "accuracy epoch 17:0.6697689294815063\n",
      "accuracy epoch 18:0.6699790358543396\n",
      "accuracy epoch 19:0.670205295085907\n",
      "accuracy epoch 20:0.6705445647239685\n",
      "accuracy epoch 21:0.670835554599762\n",
      "accuracy epoch 22:0.6710779070854187\n",
      "accuracy epoch 23:0.6713850498199463\n",
      "accuracy epoch 24:0.6716921329498291\n",
      "Training phase finished\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FfX59/H3LUmFAhWpqBhQ9KlSCQQoEVmLlSIgtCil\npVJcqAo+lZ/yqIhbH62/Uqi0Wn9edeEpuKLUDaVqi2JdLzcCRBGoshTbRCoIBlmVwP38cSYhxCxz\nksxZP6/rysU5M9+Zc0+G5M58v997xtwdERGR+hyS7ABERCQ9KGGIiEgoShgiIhKKEoaIiISihCEi\nIqEoYYiISChKGCIiEooShoiIhKKEISIioeQkO4CmdMQRR3inTp2SHYaISNpYunTpp+7eLkzbjEoY\nnTp1oqioKNlhiIikDTP7KGxbdUmJiEgoShgiIhKKEoaIiIQS2RiGmXUEHgCOAhyY7e63V2vzM2Aa\nYMB24H+7+7vBumHA7UAz4E/uPjOqWCX17d27l5KSEvbs2ZPsUETSUvPmzenQoQO5ubkN3keUg97l\nwJXuvszMWgNLzewFd19Vpc0/gUHu/pmZDQdmA6eaWTPgj8AQoARYYmYLq20rWaSkpITWrVvTqVMn\nzCzZ4YikFXdny5YtlJSUcPzxxzd4P5ElDHffCGwMXm83s9VAHrCqSps3qmzyFtAheN0bWOvu6wHM\nbD4wquq2TeWp5aXMWvQBH5ft5pg2LZg6tDNn9cxr6o+RRtqzZ4+ShUgDmRnf/OY32bx5c6P2k5Ax\nDDPrBPQE3q6j2YXAX4PXecC/q6wrCZbVtO+JZlZkZkXxfjOeWl7KtU+uoLRsNw6Ulu3m2idX8NTy\n0rj2I4mhZCHScE3x8xN5wjCzVsATwBR3/7yWNt8jljCmxbt/d5/t7oXuXtiuXajak0qzFn3A7r37\nDlq2e+8+Zi36IN4wREQyXqQJw8xyiSWLee7+ZC1tCoA/AaPcfUuwuBToWKVZh2BZk/q4bHdcyyW7\ntWrVqtH7+PjjjxkzZkyt68vKyrjzzjtDt6/uggsu4Pjjj6dHjx50796dF198sVHxNrW7776bBx54\noFH7WLFiBT169KBHjx60bdu28ni///3vx7WfoUOHsn379jrbXH/99bz00kuNCbdeAwYMoLi4ONLP\naCpRzpIyYA6w2t1vraXNscCTwLnu/mGVVUuAE83seGKJ4qfAuKaO8Zg2LSitITkc06ZFU3+UJFiq\njk0dc8wxPP7447Wur0gYv/jFL0K1r8msWbMYM2YML730EhMnTmTNmjWNihmgvLycnJzG/7q45JJL\nGr2Pbt26Vf6CveCCCxg5cmSNSbW+mBctWlTvZ02fPr3hgWagKK8w+gPnAqebWXHwdaaZXWJmFf9r\n/i/wTeDOYH0RgLuXA5OBRcBq4FF3X9nUAU4d2pkWuc0OWtYitxlTh3Zu6o+SBErk2NSGDRs4/fTT\nKSgoYPDgwfzrX/8CYN26dfTp04du3bpxww03VF6dbNiwga5duwKwcuVKevfuTY8ePSgoKGDNmjVc\nc801rFu3jh49ejB16tSD2u/bt4+rrrqKrl27UlBQwB133FFnbH379qW09MAxL126lEGDBtGrVy+G\nDh3Kxo0bAViyZAkFBQWVn1nxeffddx8//OEPOf300xk8eDAQS0annHIKBQUF3HjjjQDs3LmTESNG\n0L17d7p27cqf//xnAK655hq6dOlCQUEBV111FQA33XQTv/vd7wAoLi6mT58+FBQUcPbZZ/PZZ58B\ncNpppzFt2jR69+7NSSedxGuvvRb6fCxevJjTTjuNkSNH0q1bNwB+8IMf0KtXL/Lz8/nTn/5U2bZD\nhw6UlZWxdu1aunbtyoUXXkh+fj7Dhw+vnL49fvx4nnrqqcr2N910Ez179qSgoIAPP4z9jbtp0yYG\nDx5Mfn4+kyZNIi8vj7KysoPiKi8vp02bNlx22WXk5+czZMgQtmzZUrl+/vz59O7dm86dO/PGG7G5\nQOvWrWPgwIH07NmTXr168fbbsSHg0tJSBgwYQI8ePejatWtl+7/+9a/07duX73znO4wdO5adO3eG\n/r6F5u4Z89WrVy+P14JlJd5vxoveadoz3m/Gi75gWUnc+5DorVq1KnTbfjNe9OOmPfOVr34zXmxU\nDC1btvzKspEjR/p9993n7u5z5szxUaNGubv7iBEj/OGHH3Z397vuuqty23/+85+en5/v7u6TJ0/2\nhx56yN3dv/jiC9+1a9dB66u3v/POO/1HP/qR7927193dt2zZ8pV4zj//fH/sscfc3X3BggV+zjnn\nuLv7l19+6X379vVNmza5u/v8+fN9woQJ7u6en5/vb7zxhru7T5s2rfLz7r33Xs/Ly6v8nEWLFvnF\nF1/s+/fv93379vmIESP8lVde8ccff9wvuuiiyhjKysr8008/9ZNOOsn379/v7u6fffaZu7vfeOON\nPmvWLHd379atm7/88svu7v7LX/7SL7/8cnd3HzRokF9xxRXu7v7ss8/64MGDazkjBx+vu/sLL7zg\nLVu29I8++qhyWUX8O3fu9JNPPtm3bt3q7u55eXn+2Wef+Zo1azwnJ8ffe+89d3c/++yz/ZFHHnF3\n95/97Ge+YMGCyvZ33nmnu7vffvvtPmnSJHd3nzRpkt9yyy3u7v6Xv/zFgcrjrbB3714HfP78+V85\n3v79+/vVV1/t7u5PP/20Dx06tDLe3bt3u7v76tWrvXfv3u7uPnPmTJ85c6a7u5eXl/v27dv9k08+\n8e9+97u+c+dOd3f/9a9/7dOnT//K96umnyOgyEP+js2omw82xFk981Kiq0KaTiLHpt58802efDI2\nPHfuuedy9dVXVy6v+Mt03LhxlX9hV9W3b1+mT59OSUkJo0eP5sQTT6zzsxYvXswll1xS2c3Stm3b\nGttNnTqV6667jpKSEt58800APvjgA95//32GDBkCxK5W2rdvT1lZGdu3b6dv376VsT7zzDOV+xoy\nZEjl5zz//PM8//zz9OzZE4AdO3awZs0aBg4cyJVXXsm0adMYOXIkAwcOpLy8nObNm3PhhRcycuRI\nRo4ceVCM27Zto6ysjEGDBgFw/vnn8+Mf/7hy/ejRowHo1asXGzZsqPP7Ul3fvn059thjK9/fdttt\nLFy4EIjV86xbt47CwsKDtvnWt75VeUVS12dWjeu5554D4PXXX+f6668HYOTIkbRu3brGbXNyciqP\ncfz48Ywbd6CXvabj/eKLL5g8eTLvvvsuOTk5rFu3DoBTTjmFSZMmsWfPHs466yy6d+/O4sWLWbVq\nFf369QPgyy+/ZMCAASG+W/HRrUEk49Q2BpVqY1Pjxo1j4cKFtGjRgjPPPJO///3vTbLfWbNm8eGH\nH/Lb3/6Wn//850CsJyE/P5/i4mKKi4tZsWIFzz//fL37atmyZeVrd+faa6+t3MfatWu58MILOemk\nk1i2bFll99vNN99MTk4O77zzDmPGjOGZZ55h2LBhcR3DoYceCkCzZs0oLy+Pa9uqMS9evJhXX32V\nt956i3fffZeCgoIa7xZQ8Xn1fWZj4qqu6jTXmvb7+9//no4dO7JixQreeecdvvjiCwBOP/10Xn75\nZdq3b895553HvHnzcHeGDRtWeW5WrVrF7NmzGxVfTZQwJOMkcmyqX79+zJ8/H4B58+YxcOBAAPr0\n6cMTTzwBULm+uvXr13PCCSdw2WWXMWrUKN577z1at25d68ydIUOGcM8991T+Qtm6dWudsU2ePJn9\n+/ezaNEiOnfuzObNmyuvOPbu3cvKlStp06YNrVu3ruwfry1WiM0qmjt3Ljt27ABifembNm3i448/\n5utf/zrjx49n6tSpLFu2jB07drBt2zbOPPNMbrvtNt59992D9nXYYYdx+OGHV45PPPjgg5VXG01p\n27ZttG3blhYtWrBy5UqWLFnS5J/Rv39/Hn30UQCee+65Ws9feXl55dXoww8/XO8VwLZt22jfvj1m\nxv3330+s9wg++ugjjj76aCZOnMiECRNYvnw5/fr145VXXmH9+vVAbFypKSY7VJf1XVLxStXZN3JA\nxflo6vO0a9cuOnToUPn+iiuu4I477mDChAnMmjWLdu3ace+99wLwhz/8gfHjxzN9+nSGDRvGYYcd\n9pX9Pfroozz44IPk5uZy9NFHc91119G2bVv69+9P165dGT58OJdeemll+4suuogPP/yQgoICcnNz\nufjii5k8eXKt8ZoZN9xwA7fccgtDhw7l8ccf57LLLmPbtm2Ul5czZcoU8vPzmTNnDhdffDGHHHII\ngwYNqjFWgDPOOIPVq1dXdl+1atWKhx56iLVr1zJ16lQOOeQQcnNzueuuu9i+fTujRo1iz549uDu3\n3vrViZL3338/l1xyCbt27eKEE06o/N41pREjRjB79my6dOlC586dOfXUU5v8M371q18xbtw47r33\nXgYMGMCRRx550FVOhcMOO4zXXnuNG2+8kfbt21dODqjN5MmTGTNmDHPnzmXEiBGVVyEvvvgit956\nK7m5ubRu3ZoHH3yQo446ijlz5jB27Fi+/PJLAH7zm9/U280ZL6vIWpmgsLDQo3yAUsXsm6rFfi1y\nmzFjdDcljYitXr2ak08+OdlhhLZr1y5atGiBmTF//nweeeQRnn766WSHVaMdO3ZUzuKaOXMmGzdu\n5Pbbb69nK6mwZ88ecnJyyMnJ4fXXX2fKlClfeZBbeXk5RxxxxFdmTyVaTT9HZrbU3Qtr2eQgusKI\nQ12V4UoYUtXSpUuZPHky7k6bNm2YO3duskOq1bPPPsuMGTMoLy/nuOOO47777kt2SGllw4YNnHPO\nOezbt49DDz2Ue+65J9khRUYJIw6qDJewBg4c+JV++1Q1duxYxo4dm+ww0ta3v/1tli9fXmebnJyc\npF9dNAUNeschXWbfZKpM6j4VSbSm+PlRwoiDKsOTp3nz5mzZskVJQ6QBPHgeRvPmzRu1H3VJxSGq\n2TdSvw4dOlBSUtLo+/mLZKuKJ+41hmZJiYhksXhmSalLSkREQlGXVAKo2E9EMoESRsSqF/tV3Gob\nUNIQkbSiLqmI6TGwIpIplDAipmI/EckUShgRU7GfiGQKJYyIqdhPRDKFBr0jpmI/EckUShgJoMfA\nikgmUJeUiIiEooQhIiKhqEsqRak6XERSjRJGClJ1uIikInVJpSBVh4tIKlLCSEGqDheRVKSEkYJU\nHS4iqSiyhGFmHc3sJTNbZWYrzezyGtp828zeNLMvzOyqaus2mNkKMys2s6x6KpKqw0UkFUU56F0O\nXOnuy8ysNbDUzF5w91VV2mwFLgPOqmUf33P3TyOMMSWpOlxEUlFkCcPdNwIbg9fbzWw1kAesqtJm\nE7DJzEZEFUe6UnW4iKSahIxhmFknoCfwdhybObDYzJaa2cQ69j3RzIrMrGjz5s2NC1RERGoVeR2G\nmbUCngCmuPvncWw6wN1LzexI4AUz+4e7v1q9kbvPBmYDFBYWepMEnaZU7CciUYr0CsPMcokli3nu\n/mQ827p7afDvJmAB0LvpI8wcFcV+pWW7cQ4U+z21vDTZoYlIhohylpQBc4DV7n5rnNu2DAbKMbOW\nwBnA+00fZeZQsZ+IRC3KLqn+wLnACjMrDpZdBxwL4O53m9nRQBHwDWC/mU0BugBHAAtiOYcc4GF3\n/1uEsaY9FfuJSNSinCX1OmD1tPkP0KGGVZ8D3aOIK1Md06YFpTUkBxX7iUhTUaV3hlCxn4hETXer\nzRAq9hORqClhZBAV+4lIlNQlJSIioegKI4up0E9E4qGEkaX0VD8RiZe6pLKUCv1EJF5KGFlKhX4i\nEi8ljCylp/qJSLyUMLKUCv1EJF4a9M5SKvQTkXgpYWQxFfqJSDzUJSUiIqHoCkPiomI/keylhCGh\nqdhPJLupS0pCU7GfSHZTwpDQVOwnkt2UMCQ0FfuJZDclDAlNxX4i2U2D3hKaiv1EspsShsRFxX4i\n2UtdUiIiEoquMCRyKvYTyQxKGBIpFfuJZA51SUmkVOwnkjmUMCRSKvYTyRxKGBIpFfuJZI7IEoaZ\ndTSzl8xslZmtNLPLa2jzbTN708y+MLOrqq0bZmYfmNlaM7smqjglWir2E8kcUQ56lwNXuvsyM2sN\nLDWzF9x9VZU2W4HLgLOqbmhmzYA/AkOAEmCJmS2stq2kARX7iWSOyBKGu28ENgavt5vZaiAPWFWl\nzSZgk5mNqLZ5b2Ctu68HMLP5wKiq20r6ULGfSGZIyBiGmXUCegJvh9wkD/h3lfclwbKa9j3RzIrM\nrGjz5s2NCVNEROoQeR2GmbUCngCmuPvnTb1/d58NzAYoLCz0pt6/JJ4K/URSU6QJw8xyiSWLee7+\nZByblgIdq7zvECyTDKdCP5HUFeUsKQPmAKvd/dY4N18CnGhmx5vZ14CfAgubOkZJPSr0E0ldUV5h\n9AfOBVaYWXGw7DrgWAB3v9vMjgaKgG8A+81sCtDF3T83s8nAIqAZMNfdV0YYq6QIFfqJpK4oZ0m9\nDlg9bf5DrLuppnXPAc9FEJqksGPatKC0huSgQj+R5FOlt6QUFfqJpC7drVZSigr9RFKXEoakHBX6\niaQmJQzJCKrdEImeEoakPdVuiCSGBr0l7al2QyQxlDAk7al2QyQxlDAk7ekhTSKJoYQhaU+1GyKJ\noUFvSXuq3RBJDCUMyQiq3RCJnrqkREQkFF1hSNZSsZ9IfJQwJCup2E8kfuqSkqykYj+R+IVKGGb2\n4zDLRNKFiv1E4hf2CuPakMtE0oKK/UTiV+cYhpkNB84E8szsf6qs+gZQHmVgIlGaOrTzQWMYoGI/\nkfrUN+j9MbFnbv8QWFpl+Xbg/0QVlEjUVOwnEj9z9/obmeW6+97g9eFAR3d/L+rg4lVYWOhFRUXJ\nDkNEJG2Y2VJ3LwzTNuy02hfM7IdB+6XAJjN7w911lSFZQ3Ubku3CDnof5u6fA6OBB9z9VGBwdGGJ\npJaKuo3Sst04B+o2nlpemuzQRBImbMLIMbP2wE+AZyKMRyQlqW5DJHzCuBlYBKxz9yVmdgKwJrqw\nRFKL6jZEQo5huPtjwGNV3q8HfhRVUCKp5pg2LSitITmobkOySdhK7w5mtsDMNgVfT5hZh6iDE0kV\nekiTSPguqXuBhcAxwddfgmUiWeGsnnnMGN2NvDYtMCCvTQtmjO6mWVKSVcLWYRS7e4/6llVb3xF4\nADgKcGC2u99erY0BtxOrJt8FXODuy4J1G4gVCO4DysPME1YdhohIfKKow9hiZuOBR4L35wBb6tmm\nHLjS3ZeZWWtgqZm94O6rqrQZDpwYfJ0K3BX8W+F77v5pyBhFRCRCYRPGz4E7gNuIXS28AVxQ1wbu\nvhHYGLzebmargTygasIYRayuw4G3zKyNmbUPthVJeyr2k0wSz7Ta8929nbsfSSyB/Crsh5hZJ6An\n8Ha1VXnAv6u8LwmWQSwxLTazpWY2sY59TzSzIjMr2rx5c9iQRCKnYj/JNGETRoG7f1bxxt23EksA\n9TKzVsATwJSgWjysAcEYyXDgUjP7bk2N3H22uxe6e2G7du3i2L1ItFTsJ5kmbMI4JLjpIABm1pYQ\n3VlmlkssWcxz9ydraFIKdKzyvkOwDHev+HcTsADoHTJWkZSgYj/JNGETxu+BN83sv83sv4mNYdxS\n1wbBDKg5wGp3v7WWZguB8yymD7DN3TeaWctgoBwzawmcAbwfMlaRlKCHNEmmCVvp/YCZFQGnB4tG\nV5vtVJP+wLnACjMrDpZdBxwb7PNu4DliU2rXEptWOyFodxSwIJZzyAEedve/hToikRShhzRJpgk7\nS4ogQdSXJKq2fx2weto4cGkNy9cD3cN+lkgq0kOaJNOEThgiEr+zeuYpQUjGUMIQSTGq3ZBUpYQh\nkkIqajcqxj0qajcAJQ1JurCzpEQkAVS7IalMCUMkhah2Q1KZEoZIClHthqQyJQyRFKIHNUkq06C3\nSApR7YakMiUMkRSj2g1JVUoYImlOdRuSKEoYImlMdRuSSBr0FkljqtuQRFLCEEljqtuQRFLCEElj\nqtuQRFLCEEljqtuQRNKgt0gaU92GJJIShkiaU92GJIoShkgWUu2GNIQShkiWUe2GNJQGvUWyjGo3\npKGUMESyjGo3pKGUMESyjGo3pKGUMESyjGo3pKE06C2SZVS7IQ2lhCGShVS7IQ2hhCEioah2Q5Qw\nRKReqt0QiHDQ28w6mtlLZrbKzFaa2eU1tDEz+x8zW2tm75nZd6qsG2ZmHwTrrokqThGpn2o3BKKd\nJVUOXOnuXYA+wKVm1qVam+HAicHXROAuADNrBvwxWN8FOKeGbUUkQVS7IRBhwnD3je6+LHi9HVgN\nVL92HQU84DFvAW3MrD3QG1jr7uvd/UtgftBWRJJAtRsCCarDMLNOQE/g7Wqr8oB/V3lfEiyrbXlN\n+55oZkVmVrR58+amCllEqlDthkACEoaZtQKeAKa4++dNvX93n+3uhe5e2K5du6bevYgQG9ieMbob\neW1aYEBemxbMGN1NA95ZJtJZUmaWSyxZzHP3J2toUgp0rPK+Q7Ast5blIpIk8dZuaBpu5olylpQB\nc4DV7n5rLc0WAucFs6X6ANvcfSOwBDjRzI43s68BPw3aikgaqJiGW1q2G+fANNynluvvvnQW5RVG\nf+BcYIWZFQfLrgOOBXD3u4HngDOBtcAuYEKwrtzMJgOLgGbAXHdfGWGsItKE6pqGq6uM9BVZwnD3\n1wGrp40Dl9ay7jliCUVE0oym4WYm3a1WRJqcpuFmJiUMEWlymoabmXQvKRFpcrqFemZSwhCRSOgW\n6plHCUNEUoZqN1KbEoaIpATdQj31adBbRFKCbqGe+pQwRCQlqHYj9SlhiEhKUO1G6lPCEJGUoNqN\n1KdBbxFJCardSH1KGCKSMhpSu6GpuImjhCEiaUtTcRNLYxgikrY0FTexlDBEJG1pKm5iKWGISNrS\nVNzEUsIQkbSlqbiJpUFvEUlbmoqbWEoYIpLWdBv1xFHCEJGsorqNhlPCEJGsobqNxtGgt4hkDdVt\nNI4ShohkDdVtNI4ShohkDdVtNI4ShohkDdVtNI4GvUUka6huo3GUMEQkq+gW6g0XWZeUmc01s01m\n9n4t6w83swVm9p6ZvWNmXaus22BmK8ys2MyKoopRRKQ+FVNxS8t24xyYivvU8tJkh5ZwUY5h3AcM\nq2P9dUCxuxcA5wG3V1v/PXfv4e6FEcUnIlIvTcU9ILKE4e6vAlvraNIF+HvQ9h9AJzM7Kqp4REQa\nQlNxD0jmLKl3gdEAZtYbOA7oEKxzYLGZLTWziXXtxMwmmlmRmRVt3rw50oBFJPtoKu4ByUwYM4E2\nZlYM/BewHKi47hvg7j2A4cClZvbd2nbi7rPdvdDdC9u1axd50CKSXTQV94CkzZJy98+BCQBmZsA/\ngfXButLg301mtgDoDbyapFBFJItpKu4BSUsYZtYG2OXuXwIXAa+6++dm1hI4xN23B6/PAG5OVpwi\nIpqKGxNZwjCzR4DTgCPMrAS4EcgFcPe7gZOB+83MgZXAhcGmRwELYhcd5AAPu/vfoopTRKSpZepd\ncSNLGO5+Tj3r3wROqmH5eqB7VHGJiEStrqm46ZwwdC8pEZEmlqlTcZUwRESaWKZOxVXCEBFpYpk6\nFVc3HxQRaWKZOhVXCUNEJALxTsVNh2m4ShgiIkmWLtNwNYYhIpJk6XJHXCUMEZEkS5dpuEoYIiJJ\nli7TcJUwRESSLF2m4WrQW0QkydJlGq4ShohICkiHO+IqYYiIpKFkTMXVGIaISBpKxlRcJQwRkTSU\njKm4ShgiImkoGVNxlTBERNJQMqbiatBbRCQNJWMqrhKGiEiaashU3MZQl5SIiISihCEiIqEoYYiI\nSChKGCIiEooShoiIhGLunuwYmoyZbQY+auDmRwCfNmE46SSbjx2y+/h17Nmr4viPc/d2YTbIqITR\nGGZW5O6FyY4jGbL52CG7j1/Hnp3HDg07fnVJiYhIKEoYIiISihLGAbOTHUASZfOxQ3Yfv449e8V9\n/BrDEBGRUHSFISIioWR9wjCzYWb2gZmtNbNrkh1PopnZBjNbYWbFZlaU7HiiZGZzzWyTmb1fZVlb\nM3vBzNYE/x6ezBijVMvx32RmpcH5LzazM5MZY1TMrKOZvWRmq8xspZldHizP+PNfx7HHfe6zukvK\nzJoBHwJDgBJgCXCOu69KamAJZGYbgEJ3z/j56Gb2XWAH8IC7dw2W3QJsdfeZwR8Mh7v7tGTGGZVa\njv8mYIe7/y6ZsUXNzNoD7d19mZm1BpYCZwEXkOHnv45j/wlxnvtsv8LoDax19/Xu/iUwHxiV5Jgk\nIu7+KrC12uJRwP3B6/uJ/SBlpFqOPyu4+0Z3Xxa83g6sBvLIgvNfx7HHLdsTRh7w7yrvS2jgNzKN\nObDYzJaa2cRkB5MER7n7xuD1f4CjkhlMkvyXmb0XdFllXJdMdWbWCegJvE2Wnf9qxw5xnvtsTxgC\nA9y9BzAcuDTotshKHuufzbY+2ruAE4AewEbg98kNJ1pm1gp4Apji7p9XXZfp57+GY4/73Gd7wigF\nOlZ53yFYljXcvTT4dxOwgFg3XTb5JOjjrejr3ZTkeBLK3T9x933uvh/4f2Tw+TezXGK/MOe5+5PB\n4qw4/zUde0POfbYnjCXAiWZ2vJl9DfgpsDDJMSWMmbUMBsEws5bAGcD7dW+VcRYC5wevzweeTmIs\nCVfxyzJwNhl6/s3MgDnAane/tcqqjD//tR17Q859Vs+SAgimkv0BaAbMdffpSQ4pYczsBGJXFRB7\nvvvDmXz8ZvYIcBqxu3R+AtwIPAU8ChxL7E7HP3H3jBwYruX4TyPWJeHABmBSlT79jGFmA4DXgBXA\n/mDxdcT68jP6/Ndx7OcQ57nP+oQhIiLhZHuXlIiIhKSEISIioShhiIhIKEoYIiISihKGiIiEooQh\nkgLM7DQzeybZcYjURQlDRERCUcIQiYOZjTezd4LnB9xjZs3MbIeZ3RY8a+BFM2sXtO1hZm8FN3db\nUHFzNzP7lpktNrN3zWyZmf2vYPetzOxxM/uHmc0LKnRFUoYShkhIZnYyMBboH9ywcR/wM6AlUOTu\n+cArxCo6sy+TAAABP0lEQVSoAR4Aprl7AbEq24rl84A/unt3oB+xG79B7C6iU4AuxG4K1z/ygxKJ\nQ06yAxBJI4OBXsCS4I//FsRuVrcf+HPQ5iHgSTM7DGjj7q8Ey+8HHgvu3ZXn7gsA3H0PQLC/d9y9\nJHhfDHQCXo/+sETCUcIQCc+A+9392oMWmv2yWruG3m/niyqv96GfT0kx6pISCe9FYIyZHQmVz4M+\njtjP0ZigzTjgdXffBnxmZgOD5ecCrwRPPCsxs7OCfRxqZl9P6FGINJD+ghEJyd1XmdkNwPNmdgiw\nF7gU2An0DtZtIjbOAbHbZd8dJIT1wIRg+bnAPWZ2c7CPHyfwMEQaTHerFWkkM9vh7q2SHYdI1NQl\nJSIioegKQ0REQtEVhoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKh/H9VReuLPoMZWwAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7facf4dc8fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 5 5 ..., 2 1 1]\n"
     ]
    }
   ],
   "source": [
    "# Launch the graph\n",
    "with tf.Session() as session:\n",
    "    # Initializing the variables\n",
    "    session.run(tf.global_variables_initializer())\n",
    "    \n",
    "    cost_epochs = []\n",
    "    # Training cycle\n",
    "    for epoch in range(training_epochs):\n",
    "        _, summary, c = session.run(fetches=[optimizer, summary_op, cost], \n",
    "                                    feed_dict={x: X_train, y: Y_train})\n",
    "        cost_epochs.append(c)\n",
    "        writer.add_summary(summary=summary, global_step=epoch)\n",
    "        print(\"accuracy epoch {}:{}\".format(epoch, accuracy.eval({x: X_train, y: Y_train})))\n",
    "        \n",
    "    print(\"Training phase finished\")\n",
    "    \n",
    "    #plotting\n",
    "    plt.plot(range(len(cost_epochs)), cost_epochs, 'o', label='Logistic Regression Training phase')\n",
    "    plt.ylabel('cost')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "    prediction = tf.argmax(activation, 1)\n",
    "    print(prediction.eval({x: X_test}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m tensorflow.tensorboard --logdir=/tmp/logistic_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "93 dims\n",
      "Building model...\n",
      "9 classes\n",
      "Epoch 1/10\n",
      "61878/61878 [==============================] - 3s - loss: 1.9833     \n",
      "Epoch 2/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.8358     \n",
      "Epoch 3/10\n",
      "61878/61878 [==============================] - 3s - loss: 1.7817     \n",
      "Epoch 4/10\n",
      "61878/61878 [==============================] - 5s - loss: 1.7472     \n",
      "Epoch 5/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.7225     \n",
      "Epoch 6/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.7037     \n",
      "Epoch 7/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.6890     \n",
      "Epoch 8/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.6771     \n",
      "Epoch 9/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.6673     \n",
      "Epoch 10/10\n",
      "61878/61878 [==============================] - 2s - loss: 1.6589     \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x12027acf8>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dims = X_train.shape[1]\n",
    "print(dims, 'dims')\n",
    "print(\"Building model...\")\n",
    "\n",
    "nb_classes = Y_train.shape[1]\n",
    "print(nb_classes, 'classes')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(nb_classes, input_shape=(dims,), activation='sigmoid'))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
    "model.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simplicity is pretty impressive right? :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theano**:\n",
    "\n",
    "`shape = (channels, rows, cols)`\n",
    "\n",
    "**Tensorflow**:\n",
    "\n",
    "`shape = (rows, cols, channels)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`image_data_format` : `channels_last | channels_first`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \"image_dim_ordering\": \"tf\",\r\n",
      "    \"epsilon\": 1e-07,\r\n",
      "    \"floatx\": \"float32\",\r\n",
      "    \"backend\": \"tensorflow\"\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!cat ~/.keras/keras.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets understand:\n",
    "<pre>The core data structure of Keras is a <b>model</b>, a way to organize layers. The main type of model is the <b>Sequential</b> model, a linear stack of layers.</pre>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we did here is stacking a Fully Connected (<b>Dense</b>) layer of trainable weights from the input to the output and an <b>Activation</b> layer on top of the weights layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "from keras.layers.core import Dense\n",
    "\n",
    "Dense(units, activation=None, use_bias=True, \n",
    "      kernel_initializer='glorot_uniform', bias_initializer='zeros', \n",
    "      kernel_regularizer=None, bias_regularizer=None, \n",
    "      activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `units`: int > 0.\n",
    "\n",
    "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n",
    "\n",
    "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n",
    "\n",
    "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n",
    "\n",
    "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n",
    "\n",
    "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n",
    "\n",
    "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n",
    "\n",
    "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n",
    "\n",
    "* `bias_constraint`: instance of the constraints module, applied to the bias.\n",
    "\n",
    "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (some) others `keras.core.layers`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `keras.layers.core.Flatten()`\n",
    "* `keras.layers.core.Reshape(target_shape)`\n",
    "* `keras.layers.core.Permute(dims)`\n",
    "\n",
    "```python\n",
    "model = Sequential()\n",
    "model.add(Permute((2, 1), input_shape=(10, 64)))\n",
    "# now: model.output_shape == (None, 64, 10)\n",
    "# note: `None` is the batch dimension\n",
    "```\n",
    "\n",
    "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n",
    "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/dl_overview.png\" >\n",
    "\n",
    "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "from keras.layers.core import Activation\n",
    "\n",
    "Activation(activation)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Supported Activations** : [https://keras.io/activations/]\n",
    "\n",
    "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n",
    "Here we used <b>SGD</b> (stochastic gradient descent) as an optimization algorithm for our trainable weights.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif\" width=\"40%\">\n",
    "\n",
    "Source & Reference: http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"Data Sciencing\" this example a little bit more\n",
    "====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we did here is nice, however in the real world it is not useable because of overfitting.\n",
    "Lets try and solve it with cross validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n",
    "\n",
    "A model that has been overfit has poor predictive performance, as it overreacts to minor fluctuations in the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"imgs/overfitting.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n",
    "Next: we will use two of keras's callbacks <b>EarlyStopping</b> and <b>ModelCheckpoint</b></pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see first the model we implemented"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 9)                 846       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 9)                 0         \n",
      "=================================================================\n",
      "Total params: 846\n",
      "Trainable params: 846\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 52596 samples, validate on 9282 samples\n",
      "Epoch 1/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6547 - val_loss: 1.6514\n",
      "Epoch 2/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6532 - val_loss: 1.6499\n",
      "Epoch 3/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6517 - val_loss: 1.6485\n",
      "Epoch 4/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6503 - val_loss: 1.6471\n",
      "Epoch 5/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6489 - val_loss: 1.6457\n",
      "Epoch 6/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6476 - val_loss: 1.6444\n",
      "Epoch 7/50\n",
      "52596/52596 [==============================] - 1s - loss: 1.6462 - val_loss: 1.6431\n",
      "Epoch 8/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6450 - val_loss: 1.6418\n",
      "Epoch 9/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6437 - val_loss: 1.6406\n",
      "Epoch 10/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6425 - val_loss: 1.6394\n",
      "Epoch 11/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6413 - val_loss: 1.6383\n",
      "Epoch 12/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6402 - val_loss: 1.6371\n",
      "Epoch 13/50\n",
      "52596/52596 [==============================] - 1s - loss: 1.6391 - val_loss: 1.6361\n",
      "Epoch 14/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6380 - val_loss: 1.6350\n",
      "Epoch 15/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6370 - val_loss: 1.6340\n",
      "Epoch 16/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6360 - val_loss: 1.6330\n",
      "Epoch 17/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6350 - val_loss: 1.6321\n",
      "Epoch 18/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6340 - val_loss: 1.6311\n",
      "Epoch 19/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6331 - val_loss: 1.6302\n",
      "Epoch 20/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6322 - val_loss: 1.6294\n",
      "Epoch 21/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6314 - val_loss: 1.6285\n",
      "Epoch 22/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6305 - val_loss: 1.6277\n",
      "Epoch 23/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6297 - val_loss: 1.6269\n",
      "Epoch 24/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6289 - val_loss: 1.6261\n",
      "Epoch 25/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6281 - val_loss: 1.6253\n",
      "Epoch 26/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6274 - val_loss: 1.6246\n",
      "Epoch 27/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6266 - val_loss: 1.6239\n",
      "Epoch 28/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6259 - val_loss: 1.6232\n",
      "Epoch 29/50\n",
      "52596/52596 [==============================] - 1s - loss: 1.6252 - val_loss: 1.6225\n",
      "Epoch 30/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6245 - val_loss: 1.6218\n",
      "Epoch 31/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6239 - val_loss: 1.6212\n",
      "Epoch 32/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6232 - val_loss: 1.6205\n",
      "Epoch 33/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6226 - val_loss: 1.6199\n",
      "Epoch 34/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6219 - val_loss: 1.6193\n",
      "Epoch 35/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6213 - val_loss: 1.6187\n",
      "Epoch 36/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6207 - val_loss: 1.6181\n",
      "Epoch 37/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6202 - val_loss: 1.6176\n",
      "Epoch 38/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6196 - val_loss: 1.6170\n",
      "Epoch 39/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6190 - val_loss: 1.6165\n",
      "Epoch 40/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6185 - val_loss: 1.6159\n",
      "Epoch 41/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6179 - val_loss: 1.6154\n",
      "Epoch 42/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6174 - val_loss: 1.6149\n",
      "Epoch 43/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6169 - val_loss: 1.6144\n",
      "Epoch 44/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6164 - val_loss: 1.6139\n",
      "Epoch 45/50\n",
      "52596/52596 [==============================] - 1s - loss: 1.6159 - val_loss: 1.6134\n",
      "Epoch 46/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6154 - val_loss: 1.6130\n",
      "Epoch 47/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6149 - val_loss: 1.6125\n",
      "Epoch 48/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6145 - val_loss: 1.6120\n",
      "Epoch 49/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6140 - val_loss: 1.6116\n",
      "Epoch 50/50\n",
      "52596/52596 [==============================] - 0s - loss: 1.6136 - val_loss: 1.6112\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x120c81cc0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.15, random_state=42)\n",
    "\n",
    "fBestModel = 'best_model.h5' \n",
    "early_stop = EarlyStopping(monitor='val_loss', patience=2, verbose=1) \n",
    "best_model = ModelCheckpoint(fBestModel, verbose=0, save_best_only=True)\n",
    "\n",
    "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=50, \n",
    "          batch_size=128, verbose=True, callbacks=[best_model, early_stop]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Layer Fully Connected Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/MLP.png\" width=\"45%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Forward and Backward Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/backprop.png\" width=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** _How hard can it be to build a Multi-Layer Fully-Connected Network with keras?_\n",
    "\n",
    "**A:** _It is basically the same, just add more layers!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_2 (Dense)              (None, 100)               9400      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 9)                 909       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 9)                 0         \n",
      "=================================================================\n",
      "Total params: 10,309\n",
      "Trainable params: 10,309\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(100, input_shape=(dims,)))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))\n",
    "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 52596 samples, validate on 9282 samples\n",
      "Epoch 1/20\n",
      "52596/52596 [==============================] - 1s - loss: 1.1979 - val_loss: 0.8706\n",
      "Epoch 2/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.8248 - val_loss: 0.7727\n",
      "Epoch 3/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.7633 - val_loss: 0.7333\n",
      "Epoch 4/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.7334 - val_loss: 0.7105\n",
      "Epoch 5/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.7150 - val_loss: 0.6960\n",
      "Epoch 6/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.7021 - val_loss: 0.6850\n",
      "Epoch 7/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6927 - val_loss: 0.6775\n",
      "Epoch 8/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6854 - val_loss: 0.6713\n",
      "Epoch 9/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6794 - val_loss: 0.6659\n",
      "Epoch 10/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6747 - val_loss: 0.6623\n",
      "Epoch 11/20\n",
      "52596/52596 [==============================] - 1s - loss: 0.6706 - val_loss: 0.6591\n",
      "Epoch 12/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6672 - val_loss: 0.6556\n",
      "Epoch 13/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6644 - val_loss: 0.6535\n",
      "Epoch 14/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6619 - val_loss: 0.6512\n",
      "Epoch 15/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6597 - val_loss: 0.6499\n",
      "Epoch 16/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6578 - val_loss: 0.6479\n",
      "Epoch 17/20\n",
      "52596/52596 [==============================] - 1s - loss: 0.6560 - val_loss: 0.6462\n",
      "Epoch 18/20\n",
      "52596/52596 [==============================] - 1s - loss: 0.6545 - val_loss: 0.6456\n",
      "Epoch 19/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6531 - val_loss: 0.6442\n",
      "Epoch 20/20\n",
      "52596/52596 [==============================] - 0s - loss: 0.6518 - val_loss: 0.6432\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x121ef9f28>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
    "          batch_size=128, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Your Turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hands On - Keras Fully Connected\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take couple of minutes and try to play with the number of layers and the number of parameters in the layers to get the best results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(100, input_shape=(dims,)))\n",
    "\n",
    "# ...\n",
    "# ...\n",
    "# Play with it! add as much layers as you want! try and get better results.\n",
    "\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))\n",
    "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
    "          batch_size=128, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Theoretical Motivations for depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Much has been studied about the depth of neural nets. Is has been proven mathematically[1] and empirically that convolutional neural network benifit from depth! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] - On the Expressive Power of Deep Learning: A Tensor Analysis - Cohen, et al 2015"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Theoretical Motivations for depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One much quoted theorem about neural network states that:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of $\\mathbb{R}^n$, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] - Approximation Capabilities of Multilayer Feedforward Networks - Kurt Hornik 1991"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Addendum\n",
    "\n",
    "[1.1 Keras Backend](1.1  Keras Backend.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}