"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dims = X_train.shape[1]\n",
+ "print(dims, 'dims')\n",
+ "print(\"Building model...\")\n",
+ "\n",
+ "nb_classes = Y_train.shape[1]\n",
+ "print(nb_classes, 'classes')\n",
+ "\n",
+ "model = Sequential()\n",
+ "model.add(Dense(nb_classes, input_shape=(dims,), activation='sigmoid'))\n",
+ "model.add(Activation('softmax'))\n",
+ "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
+ "model.fit(X_train, Y_train)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Simplicity is pretty impressive right? :)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now lets understand:\n",
+ "The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Dense"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": true
+ },
+ "source": [
+ "```python\n",
+ "from keras.layers.core import Dense\n",
+ "\n",
+ "Dense(units, activation=None, use_bias=True, \n",
+ " kernel_initializer='glorot_uniform', bias_initializer='zeros', \n",
+ " kernel_regularizer=None, bias_regularizer=None, \n",
+ " activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* `units`: int > 0.\n",
+ "\n",
+ "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n",
+ "\n",
+ "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n",
+ "\n",
+ "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n",
+ "\n",
+ "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n",
+ "\n",
+ "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n",
+ "\n",
+ "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n",
+ "\n",
+ "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n",
+ "\n",
+ "* `bias_constraint`: instance of the constraints module, applied to the bias.\n",
+ "\n",
+ "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## (some) others `keras.core.layers`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* `keras.layers.core.Flatten()`\n",
+ "* `keras.layers.core.Reshape(target_shape)`\n",
+ "* `keras.layers.core.Permute(dims)`\n",
+ "\n",
+ "```python\n",
+ "model = Sequential()\n",
+ "model.add(Permute((2, 1), input_shape=(10, 64)))\n",
+ "# now: model.output_shape == (None, 64, 10)\n",
+ "# note: `None` is the batch dimension\n",
+ "```\n",
+ "\n",
+ "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n",
+ "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![](\"imgs/dl_overview.png\")
\n",
+ "\n",
+ "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Activation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": true
+ },
+ "source": [
+ "```python\n",
+ "from keras.layers.core import Activation\n",
+ "\n",
+ "Activation(activation)\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Supported Activations** : [https://keras.io/activations/]\n",
+ "\n",
+ "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Optimizer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n",
+ "Here we used SGD (stochastic gradient descent) as an optimization algorithm for our trainable weights. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![](\"http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif\")
\n",
+ "\n",
+ "Source & Reference: http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\"Data Sciencing\" this example a little bit more\n",
+ "====="
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What we did here is nice, however in the real world it is not useable because of overfitting.\n",
+ "Lets try and solve it with cross validation."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Overfitting"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n",
+ "\n",
+ "A model that has been overfit has poor predictive performance, as it overreacts to minor fluctuations in the training data."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "![](\"imgs/overfitting.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n",
+ "Next: we will use two of keras's callbacks EarlyStopping and ModelCheckpoint
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's see first the model we implemented"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "_________________________________________________________________\n",
+ "Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ "dense_1 (Dense) (None, 9) 846 \n",
+ "_________________________________________________________________\n",
+ "activation_1 (Activation) (None, 9) 0 \n",
+ "=================================================================\n",
+ "Total params: 846\n",
+ "Trainable params: 846\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model.summary()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.model_selection import train_test_split\n",
+ "from keras.callbacks import EarlyStopping, ModelCheckpoint"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Train on 52596 samples, validate on 9282 samples\n",
+ "Epoch 1/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6547 - val_loss: 1.6514\n",
+ "Epoch 2/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6532 - val_loss: 1.6499\n",
+ "Epoch 3/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6517 - val_loss: 1.6485\n",
+ "Epoch 4/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6503 - val_loss: 1.6471\n",
+ "Epoch 5/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6489 - val_loss: 1.6457\n",
+ "Epoch 6/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6476 - val_loss: 1.6444\n",
+ "Epoch 7/50\n",
+ "52596/52596 [==============================] - 1s - loss: 1.6462 - val_loss: 1.6431\n",
+ "Epoch 8/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6450 - val_loss: 1.6418\n",
+ "Epoch 9/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6437 - val_loss: 1.6406\n",
+ "Epoch 10/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6425 - val_loss: 1.6394\n",
+ "Epoch 11/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6413 - val_loss: 1.6383\n",
+ "Epoch 12/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6402 - val_loss: 1.6371\n",
+ "Epoch 13/50\n",
+ "52596/52596 [==============================] - 1s - loss: 1.6391 - val_loss: 1.6361\n",
+ "Epoch 14/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6380 - val_loss: 1.6350\n",
+ "Epoch 15/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6370 - val_loss: 1.6340\n",
+ "Epoch 16/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6360 - val_loss: 1.6330\n",
+ "Epoch 17/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6350 - val_loss: 1.6321\n",
+ "Epoch 18/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6340 - val_loss: 1.6311\n",
+ "Epoch 19/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6331 - val_loss: 1.6302\n",
+ "Epoch 20/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6322 - val_loss: 1.6294\n",
+ "Epoch 21/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6314 - val_loss: 1.6285\n",
+ "Epoch 22/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6305 - val_loss: 1.6277\n",
+ "Epoch 23/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6297 - val_loss: 1.6269\n",
+ "Epoch 24/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6289 - val_loss: 1.6261\n",
+ "Epoch 25/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6281 - val_loss: 1.6253\n",
+ "Epoch 26/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6274 - val_loss: 1.6246\n",
+ "Epoch 27/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6266 - val_loss: 1.6239\n",
+ "Epoch 28/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6259 - val_loss: 1.6232\n",
+ "Epoch 29/50\n",
+ "52596/52596 [==============================] - 1s - loss: 1.6252 - val_loss: 1.6225\n",
+ "Epoch 30/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6245 - val_loss: 1.6218\n",
+ "Epoch 31/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6239 - val_loss: 1.6212\n",
+ "Epoch 32/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6232 - val_loss: 1.6205\n",
+ "Epoch 33/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6226 - val_loss: 1.6199\n",
+ "Epoch 34/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6219 - val_loss: 1.6193\n",
+ "Epoch 35/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6213 - val_loss: 1.6187\n",
+ "Epoch 36/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6207 - val_loss: 1.6181\n",
+ "Epoch 37/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6202 - val_loss: 1.6176\n",
+ "Epoch 38/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6196 - val_loss: 1.6170\n",
+ "Epoch 39/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6190 - val_loss: 1.6165\n",
+ "Epoch 40/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6185 - val_loss: 1.6159\n",
+ "Epoch 41/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6179 - val_loss: 1.6154\n",
+ "Epoch 42/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6174 - val_loss: 1.6149\n",
+ "Epoch 43/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6169 - val_loss: 1.6144\n",
+ "Epoch 44/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6164 - val_loss: 1.6139\n",
+ "Epoch 45/50\n",
+ "52596/52596 [==============================] - 1s - loss: 1.6159 - val_loss: 1.6134\n",
+ "Epoch 46/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6154 - val_loss: 1.6130\n",
+ "Epoch 47/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6149 - val_loss: 1.6125\n",
+ "Epoch 48/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6145 - val_loss: 1.6120\n",
+ "Epoch 49/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6140 - val_loss: 1.6116\n",
+ "Epoch 50/50\n",
+ "52596/52596 [==============================] - 0s - loss: 1.6136 - val_loss: 1.6112\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.15, random_state=42)\n",
+ "\n",
+ "fBestModel = 'best_model.h5' \n",
+ "early_stop = EarlyStopping(monitor='val_loss', patience=2, verbose=1) \n",
+ "best_model = ModelCheckpoint(fBestModel, verbose=0, save_best_only=True)\n",
+ "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=50, \n",
+ " batch_size=128, verbose=True, callbacks=[best_model, early_stop]) "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Multi-Layer Fully Connected Networks"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![](\"imgs/MLP.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Forward and Backward Propagation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![](\"imgs/backprop.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Q:** _How hard can it be to build a Multi-Layer Fully-Connected Network with keras?_\n",
+ "\n",
+ "**A:** _It is basically the same, just add more layers!_"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "_________________________________________________________________\n",
+ "Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ "dense_2 (Dense) (None, 100) 9400 \n",
+ "_________________________________________________________________\n",
+ "dense_3 (Dense) (None, 9) 909 \n",
+ "_________________________________________________________________\n",
+ "activation_2 (Activation) (None, 9) 0 \n",
+ "=================================================================\n",
+ "Total params: 10,309\n",
+ "Trainable params: 10,309\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = Sequential()\n",
+ "model.add(Dense(100, input_shape=(dims,)))\n",
+ "model.add(Dense(nb_classes))\n",
+ "model.add(Activation('softmax'))\n",
+ "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Train on 52596 samples, validate on 9282 samples\n",
+ "Epoch 1/20\n",
+ "52596/52596 [==============================] - 1s - loss: 1.1979 - val_loss: 0.8706\n",
+ "Epoch 2/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.8248 - val_loss: 0.7727\n",
+ "Epoch 3/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7633 - val_loss: 0.7333\n",
+ "Epoch 4/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7334 - val_loss: 0.7105\n",
+ "Epoch 5/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7150 - val_loss: 0.6960\n",
+ "Epoch 6/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7021 - val_loss: 0.6850\n",
+ "Epoch 7/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6927 - val_loss: 0.6775\n",
+ "Epoch 8/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6854 - val_loss: 0.6713\n",
+ "Epoch 9/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6794 - val_loss: 0.6659\n",
+ "Epoch 10/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6747 - val_loss: 0.6623\n",
+ "Epoch 11/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6706 - val_loss: 0.6591\n",
+ "Epoch 12/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6672 - val_loss: 0.6556\n",
+ "Epoch 13/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6644 - val_loss: 0.6535\n",
+ "Epoch 14/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6619 - val_loss: 0.6512\n",
+ "Epoch 15/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6597 - val_loss: 0.6499\n",
+ "Epoch 16/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6578 - val_loss: 0.6479\n",
+ "Epoch 17/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6560 - val_loss: 0.6462\n",
+ "Epoch 18/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6545 - val_loss: 0.6456\n",
+ "Epoch 19/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6531 - val_loss: 0.6442\n",
+ "Epoch 20/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6518 - val_loss: 0.6432\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
+ " batch_size=128, verbose=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Your Turn!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Hands On - Keras Fully Connected\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Take couple of minutes and try to play with the number of layers and the number of parameters in the layers to get the best results. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "model = Sequential()\n",
+ "model.add(Dense(100, input_shape=(dims,)))\n",
+ "\n",
+ "# ...\n",
+ "# ...\n",
+ "# Play with it! add as much layers as you want! try and get better results.\n",
+ "\n",
+ "model.add(Dense(nb_classes))\n",
+ "model.add(Activation('softmax'))\n",
+ "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
+ "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
+ " batch_size=128, verbose=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Theoretical Motivations for depth"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ ">Much has been studied about the depth of neural nets. Is has been proven mathematically[1] and empirically that convolutional neural network benifit from depth! "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "[1] - On the Expressive Power of Deep Learning: A Tensor Analysis - Cohen, et al 2015"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Theoretical Motivations for depth"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "One much quoted theorem about neural network states that:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ ">Universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of $\\mathbb{R}^n$, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "[1] - Approximation Capabilities of Multilayer Feedforward Networks - Kurt Hornik 1991"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Addendum\n",
+ "\n",
+ "[1.1 Keras Backend](1.1 Keras Backend.ipynb)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/1.1 Keras Backend.ipynb b/1.1 Keras Backend.ipynb
new file mode 100644
index 0000000..493533c
--- /dev/null
+++ b/1.1 Keras Backend.ipynb
@@ -0,0 +1,496 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Keras Backend\n",
+ "\n",
+ "In this notebook we will be using the [Keras backend module](http://keras.io/backend/), which provides an abstraction over both Theano and Tensorflow."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's try to re-implement the Logistic Regression Model using the `keras.backend` APIs.\n",
+ "\n",
+ "The following code will look like very similar to what we would write in Theano or Tensorflow (with the *only difference* that it may run on both the two backends)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Using TensorFlow backend.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import keras.backend as K\n",
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "%matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from kaggle_data import load_data, preprocess_data, preprocess_labels"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "9 classes\n",
+ "93 dims\n"
+ ]
+ }
+ ],
+ "source": [
+ "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
+ "X_train, scaler = preprocess_data(X_train)\n",
+ "Y_train, encoder = preprocess_labels(labels)\n",
+ "\n",
+ "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
+ "\n",
+ "X_test, _ = preprocess_data(X_test, scaler)\n",
+ "\n",
+ "nb_classes = Y_train.shape[1]\n",
+ "print(nb_classes, 'classes')\n",
+ "\n",
+ "dims = X_train.shape[1]\n",
+ "print(dims, 'dims')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "feats = dims\n",
+ "training_steps = 25"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "x = K.placeholder(dtype=\"float\", shape=X_train.shape) \n",
+ "target = K.placeholder(dtype=\"float\", shape=Y_train.shape)\n",
+ "\n",
+ "# Set model weights\n",
+ "W = K.variable(np.random.rand(dims, nb_classes))\n",
+ "b = K.variable(np.random.rand(nb_classes))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Define model and loss\n",
+ "y = K.dot(x, W) + b\n",
+ "loss = K.categorical_crossentropy(y, target)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "activation = K.softmax(y) # Softmax"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "lr = K.constant(0.01)\n",
+ "grads = K.gradients(loss, [W,b])\n",
+ "updates = [(W, W-lr*grads[0]), (b, b-lr*grads[1])]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Loss: [ 2.13178873 1.99579716 3.72429109 ..., 2.75165343 2.29350972\n",
+ " 1.77051127]\n",
+ "Loss: [ 2.95424724 0.10998608 1.07148504 ..., 0.23925911 2.9478302\n",
+ " 2.90452051]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Training\n",
+ "loss_history = []\n",
+ "for epoch in range(training_steps):\n",
+ " current_loss = train([X_train, Y_train])[0]\n",
+ " loss_history.append(current_loss)\n",
+ " if epoch % 20 == 0:\n",
+ " print(\"Loss: {}\".format(current_loss))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "loss_history = [np.mean(lh) for lh in loss_history]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VNW9//H3V8gPUqUGNLVARNS2HAuEIJGC4NGDCCiI\nlOpRkQpVSz2Vgu0jKj/9ecFaqanK6TmnRU69YEHRUqUcq3IRrbWCkEAAgSIX0QI+hYrhgFwk+P39\nMTtxiLnsmWTPJJnP63nyMLP22nu+e4bkO2utvdY2d0dERKQux6U7ABERaRqUMEREJBQlDBERCUUJ\nQ0REQlHCEBGRUJQwREQkFCUMEREJJdKEYWbbzGytmZWaWXE1283Mfmlmm81sjZmdHbdtjJltCn7G\nRBmniIjUrWUKXuNf3P0fNWy7GPh68PMt4NfAt8ysHXA3UAg4UGJm89394xTEKyIi1UhFwqjNZcBT\nHptuvszMcsysPXABsMjd9wCY2SJgCPBMTQc6+eSTvXPnztFHLCLSjJSUlPzD3XPD1I06YTiw0Mwc\neNTdZ1TZ3hH4W9zz7UFZTeXHMLNxwDiATp06UVz8hV4vERGphZm9H7Zu1IPe/dz9bGJdTzeZ2T9X\n2W7V7OO1lB9b4D7D3QvdvTA3N1SCFBGRJEWaMNx9Z/DvLuAFoHeVKtuBU+Oe5wE7aykXEZE0iSxh\nmNnxZtam4jEwCHinSrX5wLXB1VJ9gL3u/iGwABhkZm3NrG2w74KoYhURkbpFOYZxCvCCmVW8ztPu\n/oqZ3Qjg7tOBl4BLgM3AAeB7wbY9ZnYfsCI41pSKAXDJTEeOHGH79u0cOnQo3aGINEmtW7cmLy+P\nrKyspI9hzeV+GIWFha5B7+brvffeo02bNpx00kkEX0JEJCR356OPPmLfvn2cfvrpx2wzsxJ3Lwxz\nnHRfVtvkzFu1g6IFG9lZdpAOOdlMGtyFET2/cAGXNLBDhw7RuXNnJQuRJJgZJ510Ert3767XcZQw\nEjBv1Q4mP7+Wg0eOArCj7CCTn18LoKSRAkoWIslriN8frSWVgKIFGyuTRYWDR45StGBjmiISEUkd\nJYwE7Cw7mFC5NC8nnHBCvY+xc+dOLr/88hq3l5WV8atf/Sp0/arGjh3L6aefTkFBAT169ODVV1+t\nV7wNbfr06Tz11FP1OsbatWspKCigoKCAdu3aVZ7vwIEDEzrO4MGD2bdvX6117rjjDl577bX6hFun\n/v37U1paGulrNBR1SSWgQ042O6pJDh1ystMQjdSmsY41dejQgblz59a4vSJh/PCHPwxVvzpFRUVc\nfvnlvPbaa4wbN45NmzbVK2aA8vJyWras/5+LG2+8sd7H6N69e+Uf2LFjxzJs2LBqk2pdMS9YUPeV\n+vfff3/ygTZDamEkYNLgLmRntTimLDurBZMGd0lTRFKdirGmHWUHcT4fa5q3akeDv9b777/PhRde\nSH5+PhdeeCEffPABAFu2bKFPnz6cc8453HXXXZWtk23bttGtWzcA1q1bR+/evSkoKCA/P59NmzZx\n++23s2XLFgoKCpg0adIx9Y8ePcott9xC9+7dyc/P5z/+4z9qja1v377s2PH5OZeUlHD++efTq1cv\nBg8ezIcffgjAihUryM/Pp2/fvkyaNKny9Z588kmuuOIKLr30UgYNGgTEktE555xDfn4+d999NwCf\nfPIJQ4cOpUePHnTr1o1nn30WgNtvv51vfvOb5Ofnc8sttwBwzz338Itf/AKA0tJS+vTpQ35+Pt/+\n9rf5+OPY2qIXXHABt912G7179+Yb3/gGf/7zn0N/HosXL2bgwIFcddVV9OzZE4BLL72UXr160bVr\nV37zm99U1s3Ly6OsrIzNmzfTrVs3rr/+erp27crFF19cefn26NGjmTdvXmX9e+65h549e5Kfn8+7\n774LwK5du7jwwgs5++yz+eEPf0jHjh0pKys7Jq7y8nJycnL48Y9/zNlnn81FF13ERx99VLl9zpw5\n9O7dmy5duvDWW28Bsf9D5513Hj179qRXr168/fbbAOzYsYP+/ftTUFBAt27dKuu//PLL9O3bl7PP\nPpsrr7ySTz75JPT7FpYSRgJG9OzIAyO70zEnGwM65mTzwMjujeKbq3wulWNN48eP59prr2XNmjVc\nc801TJgwAYCJEycyceJEVqxYQYcOHardd/r06UycOJHS0lKKi4vJy8tj6tSpnHnmmZSWllJUVHRM\n/RkzZvDee++xatWqyterzSuvvMKIESOA2DyWH/3oR8ydO5eSkhKuu+467rjjDgC+973vMX36dJYu\nXUqLFsd+IVq6dCkzZ85kyZIlLFy4kE2bNrF8+XJKS0spKSnhjTfe4JVXXqFDhw6sXr2ad955hyFD\nhrBnzx5eeOEF1q1bx5o1a7jzzju/EN+1117Lz3/+c9asWUP37t259957K7eVl5ezfPlypk2bdkx5\nGMuWLePBBx9k7drYBSkzZ86kpKSEFStW8PDDD1cmpngbN27k5ptvZt26dWRnZ1cmiapOOeUUVq1a\nxQ033MDDDz8MwF133cWQIUNYuXIll1xyCTt3Vr8oxd69e+nTpw8rV66kb9++3HfffZXb3J3ly5dT\nVFTElClTAGjfvj2LFi1i1apVzJ49u/L/1qxZs7j00kspLS1l9erV5Ofns2vXLqZOncqrr77KypUr\nyc/P59///d8Tet/CUJdUgkb07KgE0cilcqxp6dKlPP/88wB897vf5dZbb60sr/ijM2rUqMpv2PH6\n9u3L/fffz/bt2xk5ciRf//rXa32txYsXc+ONN1Z2s7Rr167aepMmTeLWW29l165dLFu2DIj9QXzn\nnXe46KKLgFhrpX379pSVlbFv3z7OPffcylhffPHFymNddNFFla+zcOFCFi5cWPnNff/+/WzatInz\nzjuPW265hdtuu41hw4Zx3nnnUV5eTuvWrbnhhhsYOnQow4YNOybGvXv3UlZWxvnnnw/AmDFjuOKK\nKyq3jxw5EoBevXqxbdu2Wt+Xqvr27UunTp0qnz/yyCPMnz8fgO3bt7NlyxYKC4+ddvC1r32N7t27\n1/ma8XG99NJLALz55puVyXfYsGG0adOm2n1btmxZeY6jR49m1KhR1R634rUPHz7M+PHjWb16NS1b\ntmTLli0AnHPOOfzgBz/g0KFDjBgxgh49erB48WLWr19f+Tl++umn9O/fP8S7lRi1MKTZqWlMKRVj\nTYlcujhq1Cjmz59PdnY2gwcPZsmSJbXWd/dQxy8qKmLz5s389Kc/ZcyYMZX7du3aldLSUkpLS1m7\ndi0LFy6krom7xx9//DGvP3ny5MpjbN68meuvv55vfOMblJSU0L17dyZPnsyUKVNo2bIly5cv5zvf\n+Q7z5s1jyJAhId6Rz7Vq1QqAFi1aUF5entC+8TEvXryYN954g2XLllV+G69utYCK16vrNauLK+zk\n56qfXfzz6o770EMPceqpp7J27VqWL1/O4cOHARgwYACvv/467du355prrmH27Nm4O0OGDKn8bNav\nX8+MGVUXB68/JQxpdlI51nTuuecyZ84cAGbPnl35ra5Pnz78/ve/B6jcXtXWrVs544wzmDBhAsOH\nD2fNmjW0adOmxit3Bg0axPTp0yv/oOzZU/NqOccddxwTJ07ks88+Y8GCBXTp0oXdu3ezdOlSINZF\ntW7dOtq2bUubNm0qWyI1xQqxq4oef/xx9u/fD8T60nft2sXOnTv50pe+xOjRo7nllltYuXIl+/fv\nZ+/evVxyySVMmzbtC1cBnXjiibRt27ZyfOK3v/1tZWujIe3du5d27dqRnZ3NunXrWLFiRd07Jah/\n//4899xzALz00ks1fn5HjhypbI0+/fTTdbYA9u7dS/v27TEzZs6cWZmY3n//fb761a8ybtw4xo4d\ny6pVqzj33HP505/+xNatW4HYuFJDXOxQlbqkpNmp6DJs6KukDhw4QF5eXuXzn/zkJ/zyl7/kuuuu\no6ioiNzcXJ544gkApk2bxujRo3nooYcYOnQoJ5544heO9+yzzzJr1iyysrL46le/yl133UW7du3o\n168f3bp14+KLL+amm26qrH/DDTfw7rvvkp+fT1ZWFt///vcZP358jfGaGXfeeScPPvgggwcPZu7c\nuUyYMIG9e/dSXl7OzTffTNeuXXnsscf4/ve/z/HHH88FF1xQbawQS1gbNmygb9++QOwy41mzZrF5\n82YmTZrEcccdR1ZWFr/+9a/Zt28fl112GYcOHcLdeeSRR75wvJkzZ3LjjTdy4MABzjjjjMr3riEN\nHTqUGTNm0KNHD/7pn/6Jb33rWw3+Gvfeey+jRo1i9uzZDBgwgFNOOeWYVk6FE088kZUrV/Kzn/2M\ndu3aVV4cUJPx48dz+eWX88wzzzBw4MDKVsirr77Kww8/TFZWVuVncMopp/DYY49x5ZVX8umnnwLw\ns5/9rM5uzkRpLSlpEjZs2MBZZ52V7jBCO3DgANnZ2ZgZc+bM4ZlnnuEPf/hDusOq1v79+yuv4po6\ndSoffvhhJAOmzdWhQ4do2bIlLVu25M033+Tmm2/+ws3cysvLOfnkk79w9VSqVfd7pLWkRNKspKSE\n8ePH4+7k5OTw+OOPpzukGv3xj3/kgQceoLy8nNNOO40nn3wy3SE1Kdu2bePqq6/m6NGjtGrVikcf\nfTTdIUVGLQxpEppaC0OkMapvC0OD3tJkNJcvNyLp0BC/P0oY0iS0bt2ajz76SElDJAkV98No3bp1\nvY6jMQxpEvLy8ti+fXu91/MXyVQVd9yrDyUMaRKysrK+cKcwEUmtyBOGmbUAioEd7j6syrZHgH8J\nnn4J+Iq75wTbjgJrg20fuPvwqGMVEZGapaKFMRHYAHy56gZ3/3HFYzP7EdAzbvNBdy+IPjwREQkj\n0kFvM8sDhgK/qasucDXwTJTxiIhI8qK+SmoacCvwWW2VzOw04HQgfvW11mZWbGbLzGxEhDGKiEgI\nkSUMMxsG7HL3khDVrwLmunv8TQw6BZNJRgHTzOzMal5jXJBUinX1jIhItKJsYfQDhpvZNmAOMMDM\nZtVQ9yqqdEe5+87g363A6xw7vlFRZ4a7F7p7YW5ubgOGLiIiVUWWMNx9srvnuXtnYglhibuPrlrP\nzLoAbYGlcWVtzaxV8PhkYslnfVSxiohI3VI+D8PMpgDF7j4/KLoamOPHTuE9C3jUzD4jltSmursS\nhohIGmnxQRGRDKbFB0VEpMEpYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiE\nooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhI\nKJEnDDNrYWarzOzFaraNNbPdZlYa/NwQt22MmW0KfsZEHaeIiNSuZQpeYyKwAfhyDdufdffx8QVm\n1g64GygEHCgxs/nu/nGkkTYi81btoGjBRnaWHaRDTjaTBndhRM+O6Q5LRDJYpC0MM8sDhgK/SXDX\nwcAid98TJIlFwJCGjq+xmrdqB5OfX8uOsoM4sKPsIJOfX8u8VTvSHZqIZLCou6SmAbcCn9VS5ztm\ntsbM5prZqUFZR+BvcXW2B2UZoWjBRg4eOXpM2cEjRylasDFNEYmIRJgwzGwYsMvdS2qp9j9AZ3fP\nBxYDMyt2r6auV/Ma48ys2MyKd+/eXe+YozJv1Q76TV3C6bf/kX5Tl9TZUthZdjChchGRVIiyhdEP\nGG5m24A5wAAzmxVfwd0/cvfDwdP/BnoFj7cDp8ZVzQN2Vn0Bd5/h7oXuXpibm9vQ8TeIZLqXOuRk\nJ1Re8TqJJCURkURFljDcfbK757l7Z+AqYIm7j46vY2bt454OJzY4DrAAGGRmbc2sLTAoKGtykule\nmjS4C9lZLY4py85qwaTBXaqtrzEPEUmFlM/DMLMpZjY8eDrBzNaZ2WpgAjAWwN33APcBK4KfKUFZ\nk5NM99KInh15YGR3OuZkY0DHnGweGNm9xqukNOYhIqmQistqcffXgdeDx3fFlU8GJtewz+PA4ykI\nL1IdcrLZUU1yqK17CWJJI+xltBrzEJFU0EzviCXavZSMZMY8REQSpYQRsUS7l5KRiqQkIpKSLqlM\nl0j3UrLHBzQzXEQipYTRTESdlERElDAymNarEpFEKGFkqIq5GxWX41bM3QCUNESkWhr0zlCauyEi\niVLCyFCauyEiiVLCyFCauyEiiVLCyFDJzt3QIocimUuD3hkqmbkbGigXyWxKGBks0bkbtQ2UK2GI\nNH9KGBJaMgPlmuuROL1n0lgpYUhoia68qy6sxOk9k8ZMg94SWqID5ZrrkTi9Z9KYqYUhoSU6UK65\nHonTeyaNmRKGJCSRgfJkbx6VyfSeSWOmLimJjO7TkTi9Z9KYqYUhkdF9OhKn90waM3P3aF/ArAVQ\nDOxw92FVtv0EuAEoB3YD17n7+8G2o8DaoOoH7j68ttcpLCz04uLihg5fRKRZM7MSdy8MUzcVLYyJ\nwAbgy9VsWwUUuvsBM/s34EHgymDbQXcvSEF8IiISQqRjGGaWBwwFflPddnd/zd0PBE+XAXlRxiPS\nHGl9L0mVqAe9pwG3Ap+FqHs98HLc89ZmVmxmy8xsRHU7mNm4oE7x7t27GyBckaalYqLfjrKDOJ9P\n9FPSkChEljDMbBiwy91LQtQdDRQCRXHFnYJ+tVHANDM7s+p+7j7D3QvdvTA3N7ehQhdpMjTRT1Ip\nyhZGP2C4mW0D5gADzGxW1UpmNhC4Axju7ocryt19Z/DvVuB1oGeEsYo0SZroJ6kUWcJw98nunufu\nnYGrgCXuPjq+jpn1BB4llix2xZW3NbNWweOTiSWf9VHFKtJU6UZYkkopn7hnZlPMrOIS2SLgBOB3\nZlZqZvOD8rOAYjNbDbwGTHV3JQyRKjTRT1Ip8nkYqaJ5GJKptBy61Edjm4chIhFK9EZYIslSwpBG\nR9+YRRonJQxpVHQDIZHGS6vVSqOieQUijZcShjQqmlcg0nipS0oaFd1AqPHS2JKohSGNiuYVNE5a\ns0pALQxpZHQDodRItLVQ29iSPpvMoYQhjU4y8wrUXRJeMleiaWxJQF1S0gyouyQxyVyJlsyaVbpP\nR/OjhCFNni7FTUwyrYVEx5aUxJsnJQxp8tRdkphkWgsjenbkgZHd6ZiTjQEdc7J5YGT3Gruwkk3i\napU0bqHGMMzsCnf/XV1lIumgS3ETM2lwl2PGMCDclWiJjC0lk8Q1y7/xC9vCmByyTCTldCluYhJt\nLSQjmVaMWiWNX60tDDO7GLgE6Ghmv4zb9GWgPMrARMJK5lLcTL+qKuoVbpNpxahV0vjV1SW1EygG\nhgPx9+beB/w4qqBEEpXIH0D9kYleMkk8ma5FzQ9JrVoThruvBlab2dPufgRit08FTnX3j1MRoEhD\n0x+Z1Ei0FZPKVkkmty7rI+wYxiIz+7KZtQNWA0+Y2cMRxiUSGV1V1TglM7aS6FiJLvetn7AJ40R3\n/19gJPCEu/cCBobZ0cxamNkqM3uxmm2tzOxZM9tsZm+bWee4bZOD8o1mNjhknCJ1SmZAVlJjRM+O\n/OX2Abw3dSh/uX1And/8E73gQXN26idswmhpZu2BfwW+8Ie/DhOBDTVsux742N2/BjwC/BzAzL4J\nXAV0BYYAvzKzFjUcQyQhuqqq+Ui0VZJs61JXYsWEXUtqCrAA+Iu7rzCzM4BNde1kZnnAUOB+4CfV\nVLkMuCd4PBf4TzOzoHyOux8G3jOzzUBvYGnIeEVqpAUOm5dExkqSGVjXRRKfC5Uwggl6v4t7vhX4\nTohdpwG3Am1q2N4R+FtwzHIz2wucFJQvi6u3PSgTaRBRX1YqjVMyA+vJXiTRHAfXQ3VJmVmemb1g\nZrvM7O9m9vug9VDbPsOAXe5eUlu1asq8lvKqrzHOzIrNrHj37t21noOISDID6/WZH9LcBtfDdkk9\nATwNXBE8Hx2UXVTLPv2A4WZ2CdAa+LKZzXL30XF1tgOnAtvNrCVwIrAnrrxCHrE5Icdw9xnADIDC\nwsIvJBQRkaoSbV1qfsjnwg5657r7E+5eHvw8CeTWtoO7T3b3PHfvTGwAe0mVZAEwHxgTPL48qONB\n+VXBVVSnA18HloeMVUSkwSRzkUSyrZLGPrAetoXxDzMbDTwTPL8a+CiZFzSzKUCxu88HHgN+Gwxq\n7yGWWHD3dWb2HLCe2BIkN7n70ZqOKZIKzbFPWuqWilnrTWVg3WJf6OuoZNYJ+E+gL7GxhLeACe7+\nQbThhVdYWOjFxcXpDkOaqaq/0BD7ltnQi/ZJ85Do/5d+U5dUm2A65mTzl9sHRBqrmZW4e2GYumFb\nGPcBYyqWAwlmfP8CuC65EEWalubaJy3RSLRVUp/5Ials9YZNGPnxa0e5+x4z6xlRTCKNjpYTkUQ1\nx/khYQe9jwsWHQQqWxhhk41Ik6flRCRKyQysp2OZk7AJ4yHgLTO7Lxi0fgt4MLKoRBoZLSciUUrV\n/JD6CjvT+ykzKwYGEJtUN9Ld10cWlUgjo+VEJGqpmB9SX6G7lYIEoSQhGUvLiUhjkuy92etD4xAi\nIk1QOlq9ShgiIk1Uqlu9YQe9RUQkwylhiIhIKEoYIiISisYwRCKkBQulOVHCEIlIU1mBVCQsdUmJ\nRCQdSzeIREkJQyQiWrBQmhslDJGIaMFCaW6UMEQiogULpbnRoLdIRLRgoTQ3kSUMM2sNvAG0Cl5n\nrrvfXaXOI8C/BE+/BHzF3XOCbUeBtcG2D9x9eFSxikRFCxZKcxJlC+MwMMDd95tZFvCmmb3s7ssq\nKrj7jysem9mPgPi7+B1094II4xMRkQRENobhMfuDp1nBj9eyy9XAM1HFIyIi9RPpoLeZtTCzUmAX\nsMjd366h3mnA6cCSuOLWZlZsZsvMbESUcYqISN0iTRjufjToVsoDeptZtxqqXkVsjCN+llMndy8E\nRgHTzOzMqjuZ2bggqRTv3r27weMXEZHPpeSyWncvA14HhtRQ5SqqdEe5+87g363Bvj2r7uTuM9y9\n0N0Lc3NzGzJkERGpIrKEYWa5ZlZxxVM2MBD4azX1ugBtgaVxZW3NrFXw+GSgH7o9rIhIWkV5lVR7\nYKaZtSCWmJ5z9xfNbApQ7O7zg3pXA3PcPX5A/CzgUTP7LNh3anBPcRERSRM79u9001VYWOjFxcXp\nDkNEpEkxs5JgvLhOWhpERERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBER\nCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhAR\nkVCUMEREJJTIEoaZtTaz5Wa22szWmdm91dQZa2a7zaw0+LkhbtsYM9sU/IyJKk4REQmnZYTHPgwM\ncPf9ZpYFvGlmL7v7sir1nnX38fEFZtYOuBsoBBwoMbP57v5xhPGKiEgtImtheMz+4GlW8OMhdx8M\nLHL3PUGSWAQMiSBMEREJKdIxDDNrYWalwC5iCeDtaqp9x8zWmNlcMzs1KOsI/C2uzvagrOrxx5lZ\nsZkV7969u8HjFxGRz0WaMNz9qLsXAHlAbzPrVqXK/wCd3T0fWAzMDMqtusNVc/wZ7l7o7oW5ubkN\nGbqIiFSRkquk3L0MeJ0q3Uru/pG7Hw6e/jfQK3i8HTg1rmoesDPiMEVEpBZRXiWVa2Y5weNsYCDw\n1yp12sc9HQ5sCB4vAAaZWVszawsMCspERCRNorxKqj0w08xaEEtMz7n7i2Y2BSh29/nABDMbDpQD\ne4CxAO6+x8zuA1YEx5ri7nsijFVEROpg7mEvXGrcCgsLvbi4ON1hiIg0KWZW4u6FYepqpreIiISi\nhCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgo\nShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioUSWMMystZktN7PVZrbOzO6t\nps5PzGy9ma0xs1fN7LS4bUfNrDT4mR9VnCIiEk7LCI99GBjg7vvNLAt408xedvdlcXVWAYXufsDM\n/g14ELgy2HbQ3QsijE9ERBIQWQvDY/YHT7OCH69S5zV3PxA8XQbkRRWPiIjUT6RjGGbWwsxKgV3A\nInd/u5bq1wMvxz1vbWbFZrbMzEZEGaeIiNQtyi4p3P0oUGBmOcALZtbN3d+pWs/MRgOFwPlxxZ3c\nfaeZnQEsMbO17r6lyn7jgHEAnTp1iuw8REQkRVdJuXsZ8DowpOo2MxsI3AEMd/fDcfvsDP7dGuzb\ns5rjznD3QncvzM3NjSZ4EREBor1KKjdoWWBm2cBA4K9V6vQEHiWWLHbFlbc1s1bB45OBfsD6qGIV\nEZG6Rdkl1R6YaWYtiCWm59z9RTObAhS7+3ygCDgB+J2ZAXzg7sOBs4BHzeyzYN+p7q6EISKSRpEl\nDHdfQ/XdSHfFPR5Yw75vAd2jik1ERBKnmd4iIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiI\nhKKEISIioUS6llRTMG/VDooWbGRn2UE65GQzaXAXRvTsmO6wREQanYxOGPNW7WDy82s5eOQoADvK\nDjL5+bUAShoiIlVkdJdU0YKNlcmiwsEjRylasDFNEYmINF4ZnTB2lh1MqFxEJJNldMLokJOdULmI\nSCbL6IQxaXAXsrNaHFOWndWCSYO7pCkiEZHGK6MHvSsGtnWVlIhI3TI6YUAsaShBiIjULaO7pERE\nJDwlDBERCUUJQ0REQlHCEBGRUJQwREQkFHP3dMfQIMxsN/B+PQ5xMvCPBgqnqdG5Z65MPv9MPnf4\n/PxPc/fcMDs0m4RRX2ZW7O6F6Y4jHXTumXnukNnnn8nnDsmdv7qkREQkFCUMEREJRQnjczPSHUAa\n6dwzVyaffyafOyRx/hrDEBGRUNTCEBGRUDI+YZjZEDPbaGabzez2dMeTama2zczWmlmpmRWnO54o\nmdnjZrbLzN6JK2tnZovMbFPwb9t0xhilGs7/HjPbEXz+pWZ2STpjjIqZnWpmr5nZBjNbZ2YTg/Jm\n//nXcu67ugq8AAAD/UlEQVQJf/YZ3SVlZi2Ad4GLgO3ACuBqd1+f1sBSyMy2AYXu3uyvRzezfwb2\nA0+5e7eg7EFgj7tPDb4wtHX329IZZ1RqOP97gP3u/ot0xhY1M2sPtHf3lWbWBigBRgBjaeaffy3n\n/q8k+NlnegujN7DZ3be6+6fAHOCyNMckEXH3N4A9VYovA2YGj2cS+0Vqlmo4/4zg7h+6+8rg8T5g\nA9CRDPj8azn3hGV6wugI/C3u+XaSfCObMAcWmlmJmY1LdzBpcIq7fwixXyzgK2mOJx3Gm9maoMuq\n2XXJVGVmnYGewNtk2Odf5dwhwc8+0xOGVVOWaX10/dz9bOBi4Kag20Iyx6+BM4EC4EPgofSGEy0z\nOwH4PXCzu/9vuuNJpWrOPeHPPtMTxnbg1LjnecDONMWSFu6+M/h3F/ACsW66TPL3oI+3oq93V5rj\nSSl3/7u7H3X3z4D/phl//maWRewP5mx3fz4ozojPv7pzT+azz/SEsQL4upmdbmb/B7gKmJ/mmFLG\nzI4PBsEws+OBQcA7te/V7MwHxgSPxwB/SGMsKVfxxzLwbZrp529mBjwGbHD3h+M2NfvPv6ZzT+az\nz+irpACCS8mmAS2Ax939/jSHlDJmdgaxVgXE7u/+dHM+fzN7BriA2CqdfwfuBuYBzwGdgA+AK9y9\nWQ4M13D+FxDrknBgG/CDij795sTM+gN/BtYCnwXF/5dYX36z/vxrOferSfCzz/iEISIi4WR6l5SI\niISkhCEiIqEoYYiISChKGCIiEooShoiIhKKEIdIImNkFZvZiuuMQqY0ShoiIhKKEIZIAMxttZsuD\n+wc8amYtzGy/mT1kZivN7FUzyw3qFpjZsmBxtxcqFnczs6+Z2WIzWx3sc2Zw+BPMbK6Z/dXMZgcz\ndEUaDSUMkZDM7CzgSmILNhYAR4FrgOOBlcEijn8iNoMa4CngNnfPJzbLtqJ8NvBf7t4DOJfYwm8Q\nW0X0ZuCbwBlAv8hPSiQBLdMdgEgTciHQC1gRfPnPJrZY3WfAs0GdWcDzZnYikOPufwrKZwK/C9bu\n6ujuLwC4+yGA4HjL3X178LwU6Ay8Gf1piYSjhCESngEz3X3yMYVm/69KvdrW26mtm+lw3OOj6PdT\nGhl1SYmE9ypwuZl9BSrvB30asd+jy4M6o4A33X0v8LGZnReUfxf4U3Afgu1mNiI4Risz+1JKz0Ik\nSfoGIxKSu683szuJ3aHwOOAIcBPwCdDVzEqAvcTGOSC2XPb0ICFsBb4XlH8XeNTMpgTHuCKFpyGS\nNK1WK1JPZrbf3U9IdxwiUVOXlIiIhKIWhoiIhKIWhoiIhKKEISIioShhiIhIKEoYIiISihKGiIiE\nooQhIiKh/H/9eM3h1dqi1QAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# plotting\n",
+ "plt.plot(range(len(loss_history)), loss_history, 'o', label='Logistic Regression Training phase')\n",
+ "plt.ylabel('cost')\n",
+ "plt.xlabel('epoch')\n",
+ "plt.legend()\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your Turn\n",
+ "\n",
+ "Please switch to the **Theano** backend and **restart** the notebook.\n",
+ "\n",
+ "You _should_ see no difference in the execution!\n",
+ "\n",
+ "**Reminder**: please keep in mind that you *can* execute shell commands from a notebook (pre-pending a `!` sign).\n",
+ "Thus:\n",
+ "\n",
+ "```shell\n",
+ " !cat ~/.keras/keras.json\n",
+ "```\n",
+ "should show you the content of your keras configuration file."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": true
+ },
+ "source": [
+ "### Moreover\n",
+ "\n",
+ "Try to play a bit with the **learning reate** parameter to see how the loss history floats... "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Exercise: Linear Regression\n",
+ "To get familiar with automatic differentiation, we start by learning a simple linear regression model using Stochastic Gradient Descent (SGD).\n",
+ "\n",
+ "Recall that given a dataset $\\{(x_i, y_i)\\}_{i=0}^N$, with $x_i, y_i \\in \\mathbb{R}$, the objective of linear regression is to find two scalars $w$ and $b$ such that $y = w\\cdot x + b$ fits the dataset. In this tutorial we will learn $w$ and $b$ using SGD and a Mean Square Error (MSE) loss:\n",
+ "\n",
+ "$$\\mathcal{l} = \\frac{1}{N} \\sum_{i=0}^N (w\\cdot x_i + b - y_i)^2$$\n",
+ "\n",
+ "Starting from random values, parameters $w$ and $b$ will be updated at each iteration via the following rule:\n",
+ "\n",
+ "$$w_t = w_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial w}$$\n",
+ "
\n",
+ "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n",
+ "\n",
+ "where $\\eta$ is the learning rate.\n",
+ "\n",
+ "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Definition: Placeholders and Variables"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n",
+ "\n",
+ "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. \n",
+ "\n",
+ "Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Placeholders and variables\n",
+ "x = K.placeholder()\n",
+ "target = K.placeholder()\n",
+ "w = K.variable(np.random.rand())\n",
+ "b = K.variable(np.random.rand())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Notes:\n",
+ "\n",
+ "In case you're wondering what's the difference between a **placeholder** and a **variable**, in short:\n",
+ "\n",
+ "* Use `K.variable()` for trainable variables such as weights (`W`) and biases (`b`) for your model.\n",
+ "* Use `K.placeholder()` to feed actual data (e.g. training examples)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": true
+ },
+ "source": [
+ "## Model definition\n",
+ "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Define model and loss"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol1.py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol12.py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Training\n",
+ "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n",
+ "\n",
+ "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Generate data\n",
+ "np_x = np.random.rand(1000)\n",
+ "np_target = 0.96*np_x + 0.24"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Training\n",
+ "# Training\n",
+ "loss_history = []\n",
+ "for epoch in range(200):\n",
+ " current_loss = train([np_x, np_target])[0]\n",
+ " loss_history.append(current_loss)\n",
+ " if epoch % 20 == 0:\n",
+ " print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also plot the loss history:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Plot loss history"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol13.py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Final Note:\n",
+ "\n",
+ "Please switch back your backend to `tensorflow` before moving on. It may be useful for next notebooks !-)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/1.3 Introduction - Keras.ipynb b/1.3 Introduction - Keras.ipynb
deleted file mode 100644
index 973ab2f..0000000
--- a/1.3 Introduction - Keras.ipynb
+++ /dev/null
@@ -1,878 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"imgs/keras-logo-small.jpg\")
\n",
- "\n",
- "## Keras: Deep Learning library for Theano and TensorFlow"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\n",
- "ref: https://keras.io/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Why this name, Keras?\n",
- "\n",
- "Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).\n",
- "\n",
- "Keras was initially developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).\n",
- "\n",
- ">_\"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them.\"_ Homer, Odyssey 19. 562 ff (Shewring translation)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "### Kaggle Challenge Data (again)\n",
- "\n",
- "See: [Data Description](1.2 Introduction - Tensorflow.ipynb#kaggle)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "9 classes\n",
- "93 dims\n"
- ]
- }
- ],
- "source": [
- "from kaggle_data import load_data, preprocess_data, preprocess_labels\n",
- "\n",
- "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
- "X_train, scaler = preprocess_data(X_train)\n",
- "Y_train, encoder = preprocess_labels(labels)\n",
- "\n",
- "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
- "\n",
- "X_test, _ = preprocess_data(X_test, scaler)\n",
- "\n",
- "nb_classes = Y_train.shape[1]\n",
- "print(nb_classes, 'classes')\n",
- "\n",
- "dims = X_train.shape[1]\n",
- "print(dims, 'dims')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Hands On - Keras Logistic Regression\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "from keras.models import Sequential\n",
- "from keras.layers import Dense, Activation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "93 dims\n",
- "Building model...\n",
- "9 classes\n",
- "Epoch 1/10\n",
- "61878/61878 [==============================] - 2s - loss: 1.0577 \n",
- "Epoch 2/10\n",
- "61878/61878 [==============================] - 3s - loss: 0.7702 \n",
- "Epoch 3/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.7283 \n",
- "Epoch 4/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.7075 \n",
- "Epoch 5/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6945 \n",
- "Epoch 6/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6854 \n",
- "Epoch 7/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6787 \n",
- "Epoch 8/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6735 \n",
- "Epoch 9/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6694 \n",
- "Epoch 10/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6659 \n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dims = X_train.shape[1]\n",
- "print(dims, 'dims')\n",
- "print(\"Building model...\")\n",
- "\n",
- "nb_classes = Y_train.shape[1]\n",
- "print(nb_classes, 'classes')\n",
- "\n",
- "model = Sequential()\n",
- "model.add(Dense(nb_classes, input_shape=(dims,)))\n",
- "model.add(Activation('softmax'))\n",
- "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
- "model.fit(X_train, Y_train)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Simplicity is pretty impressive right? :)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Now lets understand:\n",
- "The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.
\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Dense"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "source": [
- "```python\n",
- "from keras.layers.core import Dense\n",
- "\n",
- "Dense(units, activation=None, use_bias=True, \n",
- " kernel_initializer='glorot_uniform', bias_initializer='zeros', \n",
- " kernel_regularizer=None, bias_regularizer=None, \n",
- " activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "* `units`: int > 0.\n",
- "\n",
- "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n",
- "\n",
- "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n",
- "\n",
- "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n",
- "\n",
- "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n",
- "\n",
- "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n",
- "\n",
- "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n",
- "\n",
- "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n",
- "\n",
- "* `bias_constraint`: instance of the constraints module, applied to the bias.\n",
- "\n",
- "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## (some) others `keras.core.layers`"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "* `keras.layers.core.Flatten()`\n",
- "* `keras.layers.core.Reshape(target_shape)`\n",
- "* `keras.layers.core.Permute(dims)`\n",
- "\n",
- "```python\n",
- "model = Sequential()\n",
- "model.add(Permute((2, 1), input_shape=(10, 64)))\n",
- "# now: model.output_shape == (None, 64, 10)\n",
- "# note: `None` is the batch dimension\n",
- "```\n",
- "\n",
- "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n",
- "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "\n",
- "\n",
- "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Activation"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "source": [
- "```python\n",
- "from keras.layers.core import Activation\n",
- "\n",
- "Activation(activation)\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "**Supported Activations** : [https://keras.io/activations/]\n",
- "\n",
- "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Optimizer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n",
- "Here we used SGD (stochastic gradient descent) as an optimization algorithm for our trainable weights. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "\n",
- "\n",
- "Source & Reference: http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "\"Data Sciencing\" this example a little bit more\n",
- "====="
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "What we did here is nice, however in the real world it is not useable because of overfitting.\n",
- "Lets try and solve it with cross validation."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Overfitting"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n",
- "\n",
- "A model that has been overfit has poor predictive performance, as it overreacts to minor fluctuations in the training data."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "source": [
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n",
- "Next: we will use two of keras's callbacks EarlyStopping and ModelCheckpoint
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's see first the model we implemented"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "_________________________________________________________________\n",
- "Layer (type) Output Shape Param # \n",
- "=================================================================\n",
- "dense_2 (Dense) (None, 9) 846 \n",
- "_________________________________________________________________\n",
- "activation_2 (Activation) (None, 9) 0 \n",
- "=================================================================\n",
- "Total params: 846.0\n",
- "Trainable params: 846.0\n",
- "Non-trainable params: 0.0\n",
- "_________________________________________________________________\n"
- ]
- }
- ],
- "source": [
- "model.summary()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "from sklearn.model_selection import train_test_split\n",
- "from keras.callbacks import EarlyStopping, ModelCheckpoint"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train on 52596 samples, validate on 9282 samples\n",
- "Epoch 1/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6634 - val_loss: 0.6561\n",
- "Epoch 2/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6626 - val_loss: 0.6562\n",
- "Epoch 3/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6619 - val_loss: 0.6562\n",
- "Epoch 4/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6613 - val_loss: 0.6562\n",
- "Epoch 5/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6607 - val_loss: 0.6561\n",
- "Epoch 6/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6601 - val_loss: 0.6557\n",
- "Epoch 7/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6596 - val_loss: 0.6554\n",
- "Epoch 8/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6591 - val_loss: 0.6551\n",
- "Epoch 9/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6586 - val_loss: 0.6550\n",
- "Epoch 10/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6582 - val_loss: 0.6548\n",
- "Epoch 11/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6577 - val_loss: 0.6545\n",
- "Epoch 12/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6572 - val_loss: 0.6544\n",
- "Epoch 13/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6568 - val_loss: 0.6541\n",
- "Epoch 14/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6563 - val_loss: 0.6538\n",
- "Epoch 15/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6559 - val_loss: 0.6534\n",
- "Epoch 16/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6555 - val_loss: 0.6533\n",
- "Epoch 17/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6551 - val_loss: 0.6534\n",
- "Epoch 18/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6548 - val_loss: 0.6529\n",
- "Epoch 19/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6544 - val_loss: 0.6525\n",
- "Epoch 20/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6540 - val_loss: 0.6523\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.15, random_state=42)\n",
- "\n",
- "fBestModel = 'best_model.h5' \n",
- "early_stop = EarlyStopping(monitor='val_loss', patience=4, verbose=1) \n",
- "best_model = ModelCheckpoint(fBestModel, verbose=0, save_best_only=True)\n",
- "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
- " batch_size=128, verbose=True, callbacks=[best_model, early_stop]) "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Multi-Layer Perceptron and Fully Connected"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "So, how hard can it be to build a Multi-Layer percepton with keras?\n",
- "It is baiscly the same, just add more layers!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "_________________________________________________________________\n",
- "Layer (type) Output Shape Param # \n",
- "=================================================================\n",
- "dense_3 (Dense) (None, 100) 9400 \n",
- "_________________________________________________________________\n",
- "dense_4 (Dense) (None, 9) 909 \n",
- "_________________________________________________________________\n",
- "activation_3 (Activation) (None, 9) 0 \n",
- "=================================================================\n",
- "Total params: 10,309.0\n",
- "Trainable params: 10,309.0\n",
- "Non-trainable params: 0.0\n",
- "_________________________________________________________________\n"
- ]
- }
- ],
- "source": [
- "model = Sequential()\n",
- "model.add(Dense(100, input_shape=(dims,)))\n",
- "model.add(Dense(nb_classes))\n",
- "model.add(Activation('softmax'))\n",
- "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
- "model.summary()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train on 52596 samples, validate on 9282 samples\n",
- "Epoch 1/20\n",
- "52596/52596 [==============================] - 1s - loss: 1.2076 - val_loss: 0.8897\n",
- "Epoch 2/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.8247 - val_loss: 0.7779\n",
- "Epoch 3/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.7595 - val_loss: 0.7378\n",
- "Epoch 4/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.7289 - val_loss: 0.7153\n",
- "Epoch 5/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.7101 - val_loss: 0.7008\n",
- "Epoch 6/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6973 - val_loss: 0.6903\n",
- "Epoch 7/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6880 - val_loss: 0.6814\n",
- "Epoch 8/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6809 - val_loss: 0.6760\n",
- "Epoch 9/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6753 - val_loss: 0.6711\n",
- "Epoch 10/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6705 - val_loss: 0.6678\n",
- "Epoch 11/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6669 - val_loss: 0.6642\n",
- "Epoch 12/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6636 - val_loss: 0.6607\n",
- "Epoch 13/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6608 - val_loss: 0.6588\n",
- "Epoch 14/20\n",
- "52596/52596 [==============================] - 1s - loss: 0.6584 - val_loss: 0.6565\n",
- "Epoch 15/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6563 - val_loss: 0.6559\n",
- "Epoch 16/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6545 - val_loss: 0.6547\n",
- "Epoch 17/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6529 - val_loss: 0.6524\n",
- "Epoch 18/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6513 - val_loss: 0.6503\n",
- "Epoch 19/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6500 - val_loss: 0.6489\n",
- "Epoch 20/20\n",
- "52596/52596 [==============================] - 0s - loss: 0.6487 - val_loss: 0.6481\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
- " batch_size=128, verbose=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Your Turn!"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Hands On - Keras Fully Connected\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Take couple of minutes and Try and optimize the number of layers and the number of parameters in the layers to get the best results. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "model = Sequential()\n",
- "model.add(Dense(100, input_shape=(dims,)))\n",
- "\n",
- "# ...\n",
- "# ...\n",
- "# Play with it! add as much layers as you want! try and get better results.\n",
- "\n",
- "model.add(Dense(nb_classes))\n",
- "model.add(Activation('softmax'))\n",
- "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
- "model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20, \n",
- " batch_size=128, verbose=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "#### Theoretical Motivations for depth"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- ">Much has been studied about the depth of neural nets. Is has been proven mathematically[1] and empirically that convolutional neural network benifit from depth! "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "[1] - On the Expressive Power of Deep Learning: A Tensor Analysis - Cohen, et al 2015"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "#### Theoretical Motivations for depth"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "One much quoted theorem about neural network states that:"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- ">Universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of $\\mathbb{R}^n$, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "[1] - Approximation Capabilities of Multilayer Feedforward Networks - Kurt Hornik 1991"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.5.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/1.4 Keras Backend.ipynb b/1.4 Keras Backend.ipynb
deleted file mode 100644
index b70d98c..0000000
--- a/1.4 Keras Backend.ipynb
+++ /dev/null
@@ -1,497 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "# Linear Regression\n",
- "To get familiar with automatic differentiation, we start by learning a simple linear regression model using Stochastic Gradient Descent (SGD).\n",
- "\n",
- "Recall that given a dataset $\\{(x_i, y_i)\\}_{i=0}^N$, with $x_i, y_i \\in \\mathbb{R}$, the objective of linear regression is to find two scalars $w$ and $b$ such that $y = w\\cdot x + b$ fits the dataset. In this tutorial we will learn $w$ and $b$ using SGD and a Mean Square Error (MSE) loss:\n",
- "\n",
- "$$\\mathcal{l} = \\frac{1}{N} \\sum_{i=0}^N (w\\cdot x_i + b - y_i)^2$$\n",
- "\n",
- "Starting from random values, parameters $w$ and $b$ will be updated at each iteration via the following rule:\n",
- "\n",
- "$$w_t = w_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial w}$$\n",
- "
\n",
- "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n",
- "\n",
- "where $\\eta$ is the learning rate.\n",
- "\n",
- "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Placeholders and variables\n",
- "To implement and run this simple model, we will use the [Keras backend module](http://keras.io/backend/), which provides an abstraction over Theano and Tensorflow, two popular tensor manipulation libraries that provide automatic differentiation.\n",
- "\n",
- "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n",
- "\n",
- "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- }
- ],
- "source": [
- "import keras.backend as K\n",
- "import numpy as np\n",
- "\n",
- "# Placeholders and variables\n",
- "x = K.placeholder()\n",
- "target = K.placeholder()\n",
- "lr = K.variable(0.1)\n",
- "w = K.variable(np.random.rand())\n",
- "b = K.variable(np.random.rand())"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Model definition\n",
- "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "# Define model and loss\n",
- "y = w * x + b\n",
- "loss = K.mean(K.square(y-target))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "grads = K.gradients(loss, [w,b])\n",
- "updates = [(w, w-lr*grads[0]), (b, b-lr*grads[1])]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Training\n",
- "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n",
- "\n",
- "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Loss: 0.045, w, b: [0.48, 0.38]\n",
- "Loss: 0.010, w, b: [0.64, 0.41]\n",
- "Loss: 0.006, w, b: [0.72, 0.37]\n",
- "Loss: 0.003, w, b: [0.78, 0.34]\n",
- "Loss: 0.002, w, b: [0.82, 0.31]\n",
- "Loss: 0.001, w, b: [0.86, 0.30]\n",
- "Loss: 0.001, w, b: [0.88, 0.28]\n",
- "Loss: 0.000, w, b: [0.90, 0.27]\n",
- "Loss: 0.000, w, b: [0.92, 0.26]\n",
- "Loss: 0.000, w, b: [0.93, 0.26]\n"
- ]
- }
- ],
- "source": [
- "# Generate data\n",
- "np_x = np.random.rand(1000)\n",
- "np_target = 0.96*np_x + 0.24\n",
- "\n",
- "# Training\n",
- "loss_history = []\n",
- "for epoch in range(200):\n",
- " current_loss = train([np_x, np_target])[0]\n",
- " loss_history.append(current_loss)\n",
- " if epoch % 20 == 0:\n",
- " print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "We can also plot the loss history:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[]"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEACAYAAABYq7oeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+4lHWd//HnC/CgIKaEZKGgCWJqpVJo1tYYrmC1e7RS\noa2lr1Rb/mi/a+2Cu7Yedq/d0uuqtK9LfSv6hmQXV7H5i/xBhrPlZnJKgjSQ0w9YEFASTQQEPLy/\nf3zuA+PMHGbO4ZyZM+e8Htd1X3PPfX/mM58Zhnmdz+ee+/4oIjAzMys0qN4NMDOzvsfhYGZmJRwO\nZmZWwuFgZmYlHA5mZlbC4WBmZiWqCgdJ0yStkbRW0uwy+5skLZLUJukRSWOL9o+VtF3StQXb1kla\nKWmFpOWH/lLMzKynVAwHSYOAW4GpwOnADEmnFhWbBWyLiAnAzcBNRfu/BNxbtG0fkIuIsyJicnca\nb2ZmvaOansNkoC0i1kfEXmAR0FxUphlYkK0vBqZ07JDUDPwOeKLoMary+c3MrMaq+XIeA2wouL8x\n21a2TES0A89LGilpGPAPwFxSGBQK4AFJrZI+3p3Gm5lZ7xhSRZniL3VIX+wHK6OszFzgyxGxU1Jx\nufMiYoukY4EfSVodEQ9X2W4zM+tF1YTDRqDwAPPxwKaiMhuAE4BNkgYDR0XEc5LOAT4g6SbgGKBd\n0q6ImBcRWwAiYqukO0jDVyXhIMkXfzIz64aIKPfHfVWqGVZqBcZLGiepCZgO3F1U5h5gZrZ+KbAs\na9g7I+L1EfF60oHqf4+IeZKGSToSQNJw4ELg8c4aEBFeemC54YYb6t6G/rT4/fT72ZeXQ1Wx5xAR\n7ZKuBpaSwmR+RKyWNBdojYglwHxgoaQ24FlSgBzMa4A7sl7BEOD2iFh6KC/EzMx6TjXDSkTE/cDE\nom03FKzvBi6rUMfcgvU/AGd2qaVmZlYz/inpAJLL5erdhH7F72fP8vvZt6gnxqZ6k6To6200M+tr\nJBG9fEDazMwGGIeDmZmVcDiYmVkJh4OZmZVwOJiZWQmHg5mZlXA4mJlZCYeDmZmVcDiYmVkJh4OZ\nmZVwOJiZWQmHg5mZlXA4mJlZCYeDmZmVqCocJE2TtEbSWkmzy+xvkrRIUpukRySNLdo/VtJ2SddW\nW6eZmdVPxXCQNAi4FZgKnA7MkHRqUbFZwLaImECaK/qmov1fAu7tYp1mZlYn1fQcJgNtEbE+IvYC\ni4DmojLNwIJsfTEwpWOHpGbgd8ATXazTzMzqpJpwGANsKLi/MdtWtkxEtAPPSxopaRjwD8BcQOXK\nH6ROMzOrkyFVlCk3zVzxvJ3FZZSVmQt8OSJ2SjpY+XJ17tfS0rJ/PZfLea5ZM7Mi+XyefD7fY/VV\nnENa0rlAS0RMy+7PASIibiwoc19W5lFJg4HNETFa0k+A47NixwDtwD8Dj1Wqs6BuzyFtZtZFhzqH\ndDU9h1ZgvKRxwGZgOjCjqMw9wEzgUeBSYBlARLyzoKE3ANsjYl4WIJXqNDOzOqkYDhHRLulqYCnp\nGMX8iFgtaS7QGhFLgPnAQkltwLOkL/su13mIr8XMzHpIxWGlevOwkplZ1x3qsJLPkDYzsxIOBzMz\nK9EQ4dDeXu8WmJkNLA0RDrt317sFZmYDi8PBzMxKOBzMzKyEw8HMzEo4HMzMrITDwczMSjgczMys\nhMPBzMxKOBzMzKxEQ4TDnj31boGZ2cDSEOHgnoOZWW05HMzMrITDwczMSlQVDpKmSVojaa2k2WX2\nN0laJKlN0iOSxmbb3yppRcFyccFj1klamW1ffrDndziYmdVWxWlCJQ0CbgWmAJuAVkl3RcSagmKz\ngG0RMUHS5cBNpKlCfw1Mioh9ko4DVkq6OyL2AfuAXEQ8V6kNDgczs9qqpucwGWiLiPURsRdYBDQX\nlWkGFmTri0lBQkS8lAUBwBGkQOigKp/f4WBmVmPVfDmPATYU3N+YbStbJiLagecljQSQNFnS48BK\n4JMFYRHAA5JaJX38YA1wOJiZ1VbFYSXSX/jFokIZdZSJiOXAGZImArdJui8i9gDnRcQWSccCP5K0\nOiIeLteABx5o2R8QuVyOXC5XRbPNzAaOfD5PPp/vsfoUUfw9X1RAOhdoiYhp2f05QETEjQVl7svK\nPCppMLA5IkaXqWsZ8NmIeKxo+w3A9oj4UpnHxJw5wec/341XZ2Y2QEkiIsr9cV+VaoaVWoHxksZJ\naiIdaL67qMw9wMxs/VJgWda4E7OwQNI44BRgnaRhko7Mtg8HLgQe76wBHlYyM6utisNKEdEu6Wpg\nKSlM5kfEaklzgdaIWALMBxZKagOeJQUIwDuAOZL2kA5Gfyoitkk6CbhDUmRtuD0ilnbWBoeDmVlt\nVRxWqjdJMWtW8M1v1rslZmaNoxbDSnXnnoOZWW05HMzMrITDwczMSjREOHg+BzOz2mqIcHDPwcys\nthwOZmZWwuFgZmYlHA5mZlbC4WBmZiUaIhx27ap3C8zMBpaGCIcdO+rdAjOzgaUhwuHFF+vdAjOz\ngaUhwiHCJ8KZmdVSQ4TDkUe692BmVksNEQ4jRjgczMxqqSHCwT0HM7PaqiocJE2TtEbSWkmzy+xv\nkrRIUpukRySNzba/VdKKguXiauss5HAwM6utiuEgaRBwKzAVOB2YIenUomKzgG0RMQG4Gbgp2/5r\nYFJEnAVcBPxfSYOqrHM/h4OZWW1V03OYDLRFxPqI2AssApqLyjQDC7L1xcAUgIh4KSL2ZduPIM0j\nXW2d+zkczMxqq5pwGANsKLi/MdtWtkxEtAPPSxoJIGmypMeBlcAns7Cops79HA5mZrU1pIoy5Sao\njgpl1FEmIpYDZ0iaCNwm6b4q69zvySdb2L4d1q6FXC5HLperotlmZgNHPp8nn8/3WH3VhMNGYGzB\n/eOBTUVlNgAnAJskDQaOiojnCgtExJOSdgBnVFnnfu96VwtjxsBnPlNFa83MBqDiP5znzp17SPVV\nM6zUCoyXNE5SEzAduLuozD3AzGz9UmAZgKQTs7BA0jjgFGBdlXXu52ElM7PaqthziIh2SVcDS0lh\nMj8iVkuaC7RGxBJgPrBQUhvwLOnLHuAdwBxJe0gHoz8VEdsAytXZWRtGjIBnnun2azQzsy5SRKdD\n/X2CpJg3L1i1Cr761Xq3xsysMUgiIsod362Kz5A2M7MSDgczMyvhcDAzsxIOBzMzK+FwMDOzEg4H\nMzMr4XAwM7MSDRMO27fXuxVmZgNHQ4TD4YfDyy/D3r31bomZ2cDQEOEgpd7Djh31bomZ2cDQEOEA\nPu5gZlZLDgczMyvhcDAzsxIOBzMzK+FwMDOzElWFg6RpktZIWitpdpn9TZIWSWqT9Iiksdn2CyT9\nQtJKSa2Szi94zENZnSskPSZp1MHa4HAwM6udijPBSRoE3ApMIc3z3CrprohYU1BsFrAtIiZIuhy4\niTQb3FbgfRGxRdLpwAOk+aI7zIiIFdU01OFgZlY71fQcJgNtEbE+IvYCi4DmojLNwIJsfTEpSIiI\nlRGxJVt/Ahgq6bAuPj/gcDAzq6VqvpzHABsK7m/MtpUtExHtwPOSRhYWkPRBYEUWMB2+lQ0pXV+p\nESNGOBzMzGqlmnAoNwdp8cTTxWVUWCYbUvo88ImCMh+KiDcDfwb8maQPH6wRvr6SmVntVDzmQOop\njC24fzzp2EOhDcAJwCZJg4GjIuI5AEnHAz8APhIR6zoeEBGbs9sdkr5LGr76TrkGtLS08MtfwlNP\nwUUX5cjlctW8NjOzASOfz5PP53usPkUUdwKKCqQv+ydJxxE2A8tJB5JXF5S5EjgjIq6UNB24OCKm\nSzoayANzI+KOojqPjohns2MQ3wV+FBFfL/P8ERHccQcsWAB33nmoL9nMrP+TRESUG/mpSsWeQ0S0\nS7oaWEoahpofEaslzQVaI2IJMB9YKKkNeJb0SyWAq4CTgc9J+mfSUNOFwE7gAUlDgMHAg8A3DtaO\nY4+FrVu78xLNzKyrKvYc6q2j57BmDTQ3w5NP1rtFZmZ936H2HBrmDGn3HMzMaqdheg779sHQobBz\nJxx2WOXHmZkNZAOm5zBoEBxzDGzbVu+WmJn1fw0TDuChJTOzWnE4mJlZiYYKh1Gj4I9/rHcrzMz6\nv4YKB/cczMxqw+FgZmYlGiocPKxkZlYbDRUO7jmYmdWGw8HMzEo0VDh4WMnMrDYaKhzcczAzq42G\nubYSwO7dabrQ3btB3b5iiJlZ/zdgrq0E6cJ7hx8OL7xQ75aYmfVvDRUO4KElM7NaqCocJE2TtEbS\nWkmzy+xvkrRIUpukRySNzbZfIOkXklZKapV0fsFjzpa0Kqvz5mobPHo0PP10taXNzKw7KoaDpEHA\nrcBU4HRghqRTi4rNArZFxATgZuCmbPtW4H0R8Wbgo8DCgsd8FfhYRJwCnCJpajUNPvFEWLeumpJm\nZtZd1fQcJgNtEbE+IvYCi4DmojLNwIJsfTEwBSAiVkbElmz9CWCopMMkHQeMiIjl2WNuAy6upsEn\nnwy/+101Jc3MrLuqCYcxwIaC+xuzbWXLREQ78LykkYUFJH0QWJEFzJisnoPVWdbrXw+//301Jc3M\nrLuqCYdyP4Uq/v1rcRkVlpF0OvB54BNdqLMsh4OZWe8bUkWZjcDYgvvHA5uKymwATgA2SRoMHBUR\nzwFIOh74AfCRiFhXUOcJFercr6WlZf/6G96Q4/e/z1XRbDOzgSOfz5PP53usvoonwWVf9k+SjiNs\nBpYDMyJidUGZK4EzIuJKSdOBiyNiuqSjgTwwNyLuKKr3UeAaoBX4IfCViLi/zPNHYRvb22H4cHju\nOTjiiO68ZDOz/q/XT4LLjiFcDSwFngAWRcRqSXMlvS8rNh8YJakN+N/AnGz7VcDJwOckrZD0mKRR\n2b4rs8etJR3wLgmGcgYPhrFj/YslM7Pe1FCXz+hw0UVw9dXw3vfWqVFmZn3cgLp8RgcflDYz610N\nGw4+18HMrPc0bDi452Bm1nsaNhzcczAz6z0NeUB65840K9y2bekS3mZm9koD8oD0sGEwcSKsXFnv\nlpiZ9U8NGQ4A55wDjz5a71aYmfVPDRsOkyfD8uWVy5mZWdc5HMzMrERDHpCGdI2lo4+G9eth5Mgy\nDzQzG8AG5AFpSNdYmjQJfvGLerfEzKz/adhwAB+UNjPrLQ0dDuefD/dXdS1XMzPrioY95gCweze8\n5jWwZg0cd1yNG2Zm1ocN2GMOAEOHwrRpcPfd9W6JmVn/0tDhAHDJJXDnnfVuhZlZ/1JVOEiaJmmN\npLWSZpfZ3yRpkaQ2SY9IGpttHylpmaTtkr5S9JiHsjqLZ4jrkosugocfhhde6M6jzcysnIrhIGkQ\ncCswFTgdmCHp1KJis4BtETEBuBm4Kdv+EnA98JlOqp8REWdFxNkR8cfuvICjjoJcDr73ve482szM\nyqmm5zCZNMfz+ojYCywCmovKNAMLsvXFwBSAiNgZET8Ddh/C81d0zTVwyy3Qx4+tm5k1jGq+nMcA\nGwrub8y2lS0TEe3A85KqOW/5W9mQ0vXVNLYzF1wA+/bBQw8dSi1mZtZhSBVlyv0Uqvhv9OIyKlOm\n2IciYrOk4cAPJH04Ir5TrmBLS8v+9VwuRy6Xe+WTCT79abj5Znj3uys8q5lZP5TP58nn8z1WX8Xz\nHCSdC7RExLTs/hwgIuLGgjL3ZWUelTQY2BwRowv2zwQmRcSnO3mOTvcf7DyHQjt3wvjxcNdd8Na3\nVixuZtav1eI8h1ZgvKRxkpqA6UDxmQX3ADOz9UuBZeXaun9FGizp1dn6YcD7gMe72PZXGDYM/uVf\n4LOf9bEHM7NDVdUZ0pKmAbeQwmR+RHxB0lygNSKWSBoKLATOAp4FpkfEuuyxfwBGAE3A88CFwP8A\nPyENaw0GHgSuLddFqLbnAOlKrWeeCf/6r3DxxVU9xMysXzrUnkNDXz6jnB//GK64Alatgle9qhcb\nZmbWhzkcyvjEJ9Lt17/eCw0yM2sADocyXngB3vhGmDcP3vveXmqYmVkfNqAvvNeZo46C22+HWbNg\nw4bK5c3M7JX6ZTgAvOMd8Hd/B5dfni7tbWZm1euXw0od9u2Dyy6DI46A225LJ8uZmQ0EHlY6iEGD\nUiisXZt+3mpmZtWp5vIZDW3YsHTW9LnnpjOoP/SherfIzKzv6/fhAGkK0SVL0nWXxoyBd72r3i0y\nM+vb+vWwUqEzzoBFi+DSS6G1td6tMTPr2wZMOEDqOcyfD3/xF/D4IV3JycysfxtQ4QApGL78ZZg6\nFdra6t0aM7O+aUAccyg2Ywa8+CJMmZKuxTRhQr1bZGbWtwzIcAD4+MfT7fnnp4CYOLG+7TEz60sG\nbDhACojDDkvHIn70IzjttHq3yMysbxjQ4QDw0Y/CkCFpiGnJEpg0qd4tMjOrvwEfDgAf/jAMHw4X\nXQTf/S5ccEG9W2RmVl9V/VpJ0jRJayStlTS7zP4mSYsktUl6RNLYbPtIScskbZf0laLHnC1pVVbn\nzT3zcrrvkktg8eJ0BvX3vlfv1piZ1VfFcJA0CLgVmAqcDsyQdGpRsVnAtoiYANwM3JRtfwm4HvhM\nmaq/CnwsIk4BTpE0tXsvoee8853w4INw7bXwH/9R79aYmdVPNT2HyUBbRKyPiL3AIqC5qEwzsCBb\nXwxMAYiInRHxM+AVF82WdBwwIiKWZ5tuA/rErM9vehP89Kdwyy3wuc9BH79orZlZr6gmHMYAhVPm\nbMy2lS0TEe3A85JGVqhzY4U66+akk+Dhh2HpUvirv4Jdu+rdIjOz2qrmgHS564EX/z1dXEZlynS1\nzv1aWlr2r+dyOXK53EGq7hmjR0M+D1dckS7Ud9dd8NrX9vrTmpl1Sz6fJ5/P91h9FSf7kXQu0BIR\n07L7c4CIiBsLytyXlXlU0mBgc0SMLtg/E5gUEZ/O7h8HPBQRb8juTwfeFRGfKvP83Z7spydEwL/9\nG3z963DnnXD22XVriplZ1Wox2U8rMF7SOElNwHTg7qIy9wAzs/VLgWXl2tqxEhFbgBckTZYk4K+B\nu7ra+FqQ4PrrD1yPafHierfIzKz3VTVNqKRpwC2kMJkfEV+QNBdojYglkoYCC4GzgGeB6RGxLnvs\nH4ARQBPwPHBhRKyRNAn4NnA4cG9E/G0nz13XnkOhFSvg4oth+vTUmxjis0TMrI861J5Dv55Dujf8\n8Y/ppLmdO9P8EK97Xb1bZGZWynNI19ioUXDvvfDnfw5veQssKzeAZmbW4NxzOAQPPggf+QhcdRVc\ndx0MHlzvFpmZJR5WqrOnnkqX3AC47TYYN66+7TEzAw8r1d2YMWlo6b3vTcNMCxf6rGoza3zuOfSg\nFSvSwerTT4evfQ1GHuwccTOzXuSeQx9y1lnwi1+kXzC9+c1wzz31bpGZWfe459BLHnoozTT3lrfA\nV76SLsdhZlYr7jn0UeefD6tWpQPUb3wjLFjgYxFm1jjcc6iBxx6DWbPg2GPTPBETJtS7RWbW37nn\n0ADOPhuWL4cLL4S3vQ3mzIEXX6x3q8zMOudwqJHDDoPPfhZ+/WvYtAlOPTXNV93gnSIz66c8rFQn\n//3fcM01MHw4fPGLMHlyvVtkZv2Jh5Ua1NvfDq2tMHMmXHIJXHYZtLXVu1VmZonDoY4GD4aPfSyF\nwplnpuMRV10FTz9d75aZ2UDncOgDhg2Df/xHWLMGmprgtNPSQeutW+vdMjMbqBwOfcioUWnGuRUr\n4E9/SgetZ892SJhZ7VUVDpKmSVojaa2k2WX2N0laJKlN0iOSxhbsuy7bvlrShQXb10laKWmFpOU9\n83L6h7Fj4atfhV/9Kv3kdeJE+Pu/T79yMjOrhYrhIGkQcCswFTgdmCHp1KJis4BtETEBuBm4KXvs\nacBlwBuAi4B52ZzRAPuAXEScFRH+rU4ZJ5yQTppbtQp274YzzoArroAnnqh3y8ysv6um5zAZaIuI\n9RGxF1gENBeVaQYWZOuLgXdn638JLIqIl7M5pduy+gBU5fMPeMcfn67P1NYGJ58MF1yQLhH+0EM+\nT8LMekc1X85jgA0F9zdm28qWiYh24E+SRpZ57FMFjw3gAUmtkj7ejbYPOK9+NfzTP8Ef/pB+/nrl\nlfCmN8G8efDCC/VunZn1J0OqKFPuJIriv1c7K3Owx54XEVskHQv8SNLqiHi4XANaWlr2r+dyOXK5\nXKU292uHH55+AjtrVuo9zJsH118Pl18On/pUCgwzG1jy+Tz5fL7H6qt4hrSkc4GWiJiW3Z8DRETc\nWFDmvqzMo5IGA5sjYnRxWUn3AzdExKNFz3EDsD0ivlTm+fvlGdI97amn4JvfhG98A048ET75SXj/\n+9PPZM1s4KnFGdKtwHhJ4yQ1AdOBu4vK3APMzNYvBZZl63cD07NfM50EjAeWSxom6cjsBQwHLgQe\n7+6LsDRd6Q03pCGna6+F73wnbbviCsjnYd++erfQzBpJVddWkjQNuIUUJvMj4guS5gKtEbFE0lBg\nIXAW8CwwPTsAjaTrSL9m2gv8bUQszYLiDtIQ0xDg9oj4QifP7Z5DN23enC7ut2BBOm/iIx9Jy8SJ\n9W6ZmfW2Q+05+MJ7A8TKlXDbbXD77eknsh/8IHzgAzB+fL1bZma9weFgXfLyy/Bf/wWLF8MPfpDm\nu/7gB9PiHoVZ/+FwsG5rb0+XDl+8GP7zP+GYY6C5Gd7zHjj33HRhQDNrTA4H6xH79sHPfw5LlsAP\nf5h+/TR1ajrZburUdI6FmTUOh4P1io0b4d57U1A89FC6dMcFF8CUKalXMXRovVtoZgfjcLBe99JL\n8NOfwo9/DMuWwerVae6JKVPg3e9Oc2R7CMqsb3E4WM0991w6qL1sWQqMjRtTb+K889IMd+ecAyNG\n1LuVZgObw8HqbutWeOSRdHD7Zz9L81FMmHAgLM47D8aNA3X7Y2pmXeVwsD5nzx547LEUFB2BsXcv\nTJqUhqAmTUrLiSc6MMx6i8PBGsKmTSkwfvnLA8uuXQfC4swz00HviRPTVKlmdmgcDtawtmw5EBir\nVsHjj8O6dXDSSSkoCpeTT/ZBb7OucDhYv/LSS/Dkk2m2u8cfP7Bs3pxC45RT0vGMwtvXvtbDU2bF\nHA42IOzcCb/9bZoNb+3aV97u2JGCYsKEdK2ok05KB8BPPDHNx3344fVuvVntORxswPvTnw6ExW9/\nC+vXp+Gp9ethwwYYOTIFRUdgjBuXljFj0rWlRo1yz8P6H4eD2UG0t6chqY7A6AiN9evTQfJNm+DF\nF9PQ1Otel5aO0ChcP+44eNWrHCLWOBwOZofopZdSgDz11IHA6Fg6tm3ZksqNGgWjR8Oxx6alY734\n9tWvTmEyqJrptMx6QU3CIZvs52YOTPZzY9H+JuA2YBLwR+DyiPifbN91wBXAy2ST/VRTZ0HdDgfr\nE3bvTif8bd0KzzzT+e0zz8C2balHMmJEutrtyJHptmMpd//oo+Goo9JjjjoqHStxT8W6q9fDQdIg\nYC0wBdhEmjZ0ekSsKSjzKeCNEXGlpMuBSyJiuqTTgNuBtwLHAw8CEwBVqrOgbodDD8nn8+RyuXo3\no9+o9H62t6fjIdu2pUuOPPdc5+sdy/btaXnhhfT4jqDouO1s/cgjYfjwNGd44VK87Ygj+m5vxp/P\nnnWo4TCkijKTgbaIWJ894SKgGSj8Im8GbsjWFwP/J1v/S2BRRLwMrJPUltWnKuq0Hub/fD2r0vs5\neHDqEYwc2b369+w5EBQHu920Kd3u3Fm67Njxyvu7dqUr6pYLko7l8MPTMnRoWsqtV9pfuN7UBIcd\n9sqlXED589m3VBMOY4ANBfc3kr7gy5aJiHZJf5I0Mtv+SEG5p7JtqqJOswGtqSkdu+jJuTQiUkB0\nFiI7dqThs9270zGWwttdu1Lvpnh/ubKF63v2pMunFC6DBh0IiiFD0u2uXfDtb5cGSTXLkCEpjMst\nB9vXE+U7lkGD0jDgoEGl6z19v9y+jqWnVBMO5Z6ueJynszKdbS/XsfXYkVkvkw70EOolIg2ZFQfG\njTfCNdekqWyL91VaXn451dnZ0rF/z56DlysuX+2yb9+BJaL8elfvd/WxHf++3/9+mh++B/6h4qAL\ncC5wf8H9OcDsojL3Aedk64OBZ8qVBe4HzqmmzoJ94cWLFy9eur5U+n4/2FJNz6EVGC9pHLAZmA7M\nKCpzDzATeBS4FFiWbb8buF3Sl0nDSeOB5aSeQ6U6Ib06/17DzKzGKoZDdgzhamApB352ulrSXKA1\nIpYA84GF2QHnZ0lf9kTEbyR9D/gNsBe4MvvpUdk6e+H1mZlZN/T5k+DMzKz2+ugvntNJcpLWSFor\naXa929OIJK2TtFLSCknLs23HSFoq6UlJD0h6Vb3b2VdJmi/paUmrCrZ1+v5J+oqkNkm/knRmfVrd\nN3XyXt4gaaOkx7JlWsG+67L3crWkC+vT6r5L0vGSlkn6jaRfS/p0tr3HPp99MhyyE+9uBaYCpwMz\nJJ1a31Y1pH1ALiLOioiOnwrPAR6MiImkY0PX1a11fd//I30GC5V9/yRdBJwcEROAvwG+VsuGNoBy\n7yXAlyLi7Gy5H0DSG4DLgDcAFwHzJJ8rXuRl4NqIOA14G3BV9h3ZY5/PPhkOFJx4FxF7gY6T5Kxr\nROm/cTOwIFtfAFxc0xY1kIh4GHiuaHPx+9dcsP227HGPAq+S9JpatLMRdPJeQvmfuzeTnTwbEeuA\njpNnLRMRWyLiV9n6i8Bq0lUoeuzz2VfDodyJd2Pq1JZGFsADklolfSzb9pqIeBrSBww4tm6ta0yj\ni96/0dn24s9sxwmfdnBXZcMc3ywYAvF72QWSTgTOBH5O6f/vbn8++2o4VHPinVV2XkS8BXgP6T/h\nn+H3sbf4M9t180hDHWcCW4AvZtv9XlZJ0pGkSxb9bdaD6Ox96vJ72lfDYSMwtuD+8aQL9FkXZH85\nEBFbgTtJXfOnO7qTko4DnqlfCxtSZ+/fRuCEgnL+zFYQEVsLrqr5DQ4MHfm9rIKkIaRgWBgRd2Wb\ne+zz2VfDYf+Jd9nlwKeTTqizKkkalv1VgaThwIXAr0nv40ezYjOBu8pWYB3EK//qKnz/PsqB9+9u\n4K8BJJ3F4w4NAAAA60lEQVQLPN/Rvbf9XvFeZl9eHd4PPJ6t3w1Ml9Qk6SQOnDxrr/Qt4DcRcUvB\nth77fPbZ8xyyn7XdwoGT5L5Q5yY1lOw/1R2kruMQ4PaI+EJ2QcTvkf6K+B/g0oh4vn4t7bskfRfI\nAa8GniZdefhO4PuUef8k3QpMA3YA/ysiHqtDs/ukTt7L80lj5fuAdcDfdHxhZfPAzCKdPLt/HhhL\nJL0d+AnpD76Oy2X8IylEy/7/7urns8+Gg5mZ1U9fHVYyM7M6cjiYmVkJh4OZmZVwOJiZWQmHg5mZ\nlXA4mJlZCYeDmZmVcDiYmVmJ/w856gDqst6XTAAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# Plot loss history\n",
- "import matplotlib.pyplot as plt\n",
- "%matplotlib inline\n",
- "plt.plot(loss_history)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Your Turn\n",
- "\n",
- "Please switch to the **Theano** backend and re-run the notebook.\n",
- "\n",
- "You _should_ see no difference in the execution!\n",
- "\n",
- "**Reminder**: please keep in mind that you *can* execute shell commands from a notebook (pre-pending a `!` sign).\n",
- "Thus:\n",
- "\n",
- "```shell\n",
- " !cat ~/.keras/keras.json\n",
- "```\n",
- "should show you the content of your keras configuration file."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Logistic Regression"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's try to re-implement the Logistic Regression Model using the `keras.backend` APIs.\n",
- "\n",
- "The following code will look like very similar to what we would write in Theano or Tensorflow - with the *only difference* that it may run on both the two backends."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "from kaggle_data import load_data, preprocess_data, preprocess_labels"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "9 classes\n",
- "93 dims\n"
- ]
- }
- ],
- "source": [
- "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
- "X_train, scaler = preprocess_data(X_train)\n",
- "Y_train, encoder = preprocess_labels(labels)\n",
- "\n",
- "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
- "\n",
- "X_test, _ = preprocess_data(X_test, scaler)\n",
- "\n",
- "nb_classes = Y_train.shape[1]\n",
- "print(nb_classes, 'classes')\n",
- "\n",
- "dims = X_train.shape[1]\n",
- "print(dims, 'dims')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "feats = dims\n",
- "training_steps = 25"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "x = K.placeholder(dtype=\"float\", shape=X_train.shape) \n",
- "target = K.placeholder(dtype=\"float\", shape=Y_train.shape)\n",
- "\n",
- "# Set model weights\n",
- "W = K.variable(np.random.rand(dims, nb_classes))\n",
- "b = K.variable(np.random.rand(nb_classes))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "# Define model and loss\n",
- "y = K.dot(x, W) + b\n",
- "loss = K.categorical_crossentropy(y, target)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "activation = K.softmax(y) # Softmax"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "# Minimize error using cross entropy\n",
- "cross_entropy = K.categorical_crossentropy(activation, target)\n",
- "loss = K.mean(-K.sum(cross_entropy))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "grads = K.gradients(loss, [W,b])\n",
- "updates = [(W, W-lr*grads[0]), (b, b-lr*grads[1])]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {
- "collapsed": false,
- "scrolled": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Loss: -218494.890625\n",
- "Loss: -989694.875\n"
- ]
- }
- ],
- "source": [
- "# Training\n",
- "loss_history = []\n",
- "for epoch in range(training_steps):\n",
- " current_loss = train([X_train, Y_train])[0]\n",
- " loss_history.append(current_loss)\n",
- " if epoch % 20 == 0:\n",
- " print(\"Loss: {}\".format(current_loss))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEPCAYAAAATXoCrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW9///Xe1g0YBg2QQSEEYgajLtoFnXQqIiPiNeI\nwSBBXGLikvw0RjDEMIQkXkmMRr3e3K9ixBU1XhWjUVCcLNcFjMENQYyCMIgim4Bhm/n8/uiasWl6\nNpbp0nk/H496UP2pU+ecrmn60+dUdbUiAjMzs7QqKnQHzMzM6uJEZWZmqeZEZWZmqeZEZWZmqeZE\nZWZmqeZEZWZmqZbKRCVpoqQ3JM2W9KCkdlnbrpQ0P9l+QlZ8kKS5kt6UNDor3lvS85LmSbpXUssk\n3lrSlKSu5yTtVV8bZmbW9FKZqIBpQP+IOAiYD1wJIOmLwBnAfsBJwM3KKAJuAk4E+gNnSto3qesa\n4NqI2AdYBZybxM8FVkREP+B6YGJdbezk52tmZrVIZaKKiKcioip5+DzQI1k/BZgSEZsjYgGZJDYg\nWeZHxMKI2ARMAYYk+xwLPJisTwZOTdaHJI8B/piUq6sNMzMrgFQmqhznAI8n692BRVnbKpJYbnwx\n0F1SJ2BlVtJbnJTdoq6IqARWS+pYRxtmZlYALQvVsKTpQNfsEBDA2Ih4NCkzFtgUEfdmlckV5E+4\nkZTP3af6nlG11VVb3MzMCqBgiSoijq9ru6SRwGA+mZKDzIioZ9bjHsASMsllr9x4RHwoqb2komRU\nVV0+u64lkloAxRGxUlJtbeT2z8nLzGwbRESjzvuncupP0iDgCuCUiNiQtWkqMCy5Yq8E6AvMBGYB\nfSX1ktQaGAY8kuwzAxiarI/Mik9NHpNsn1FPG1uJCC8RjBs3ruB9SMviY+Fj4WNR97ItUpmogBuB\n3YDpkl6SdDNARMwB7gfmkDlvdWFkVAIXk7la8HUyF0PMTeoaA1wm6U2gIzApiU8COkuaD/x/Sbla\n28jXybPOGs877yzcsc/czMy2ULCpv7pE5pLx2rZdDVydJ/4EsE+e+DvAEXniG8hcht7gNnLdfffl\nPP/8OKZPv4SSkl71FTczs22Q1hHVp0Rb/vWv8Vx11e2F7khBlZaWFroLqeFj8Qkfi0/4WGwfbeuc\nYXOXuZgic+wGDhzHjBnjC9wjM7P0k0Q08mKKVE79fbqsY889PTCtT+/evVm40OfzzJqLXr16sWDB\ngh1Sl0dU2ygzolpLnz4+R9UQyaeoQnfDzJpIbf/nt2VE5aHAdhg+/DdOUmZmO5lHVNtIUm1XrVse\nHlGZNS8eUZmZWbPhRGW2E9xzzz0MGjRom/bdf//9+etf/7qDe5R+gwcP5s477yx0N+p14okncu+9\n99ZfsJFld6QRI0bw85//vMnb3Vk89beNPPXXOGme+ispKWHSpEkce+yx9RfewUaNGkXPnj23+01l\n4cKFlJSUsNtuuwHQuXNnLrjgAkaPHl3Pnp9tgwcP5m9/+xuSWL9+PZLYZZddADjrrLO4+eabC9zD\nnWPEiBH069ePn/3sZwXrw46c+vPl6VZQ77yzkKuuup2Kiiq6dy9iwoSzG31xyo6o47NAEqtXr0YS\n//jHPzjmmGM47LDDOO6443ZoO5WVlbRo0WKH1rmzPP744zXrDflQ8Gl6bs2Jp/6sYN55ZyHHH38j\nd999OeXl47n77ss5/vgbG3X/xB1RR11uueUW+vXrR+fOnTn11FN57733arZNmzaNfffdlw4dOnDR\nRRdRWlrKbbfdBsDkyZM56qijaspeeumldO3alfbt23PQQQcxZ84cbrnlFu6++24mTpxIu3btGDIk\n81ufJSUlzJiRuUdyVVUVv/rVr+jbty/FxcUcfvjhVFRU1Nrf6k+whx56KP3792f27Nk129577z1O\nP/10unTpQp8+fbjxxhtrtq1fv56RI0fSsWNH+vfvz69//Wt69vzkRwRKSkqYOHEiBx54ILvtthtV\nVVV11jdr1iwOP/xwiouL6datG5dffjkAGzZsYMSIEXTu3JkOHTpwxBFHsGzZMgAGDhxYc/wigl/8\n4hf07t2bPfbYg7PPPpuPPvoIyIwei4qKuOOOO+jVqxddunThV7/6VYP/ptmefvppSkpKuPrqq+nW\nrRvf/e53WbFiBSeffDJdunShU6dOnHLKKSxZ8skPKBx11FHccccdAEyaNInS0lIuu+wyOnToQN++\nfZk+ffo2lX377bc56qijKC4uZtCgQVx44YWMGjWqzn5PmDCBzp0706dPH+67774tyixfvpzBgwfT\nrl07vvrVr27xPcZLLrmEnj170r59e4444giee+65mm0vvPAChx56aM3fLntU/n//9398+ctfpkOH\nDhxyyCH87W9/25bD3niFvpPup3XJHDprqHzHa/jwsoC1AZG1rI3hw8saXO+OqKN3797x9NNPbxV/\n+umno3PnzjF79uzYuHFjXHLJJXH00UdHRMSyZcuiXbt28fDDD0dlZWX87ne/i9atW8ekSZMiIuL2\n22+Po446KiIinnzyyTjssMPio48+ioiIuXPnxtKlSyMi4uyzz46rrrqq1v5MnDgxDjjggJg/f35E\nRLzyyiuxYsWKrfq6YMGCKCoqis2bN0dExHPPPRdt27aNhx9+OCIiqqqq4tBDD41f/OIXsXnz5njn\nnXeiT58+MW3atIiIGD16dJSWlsbq1aujoqIiDjjggOjZs+cWfTr44IOjoqIi1q9fX299X/7yl+Ou\nu+6KiIh169bFCy+8EBER//M//xOnnHJKTR0vvfRSrFmzJiIiSktLa47fpEmTol+/frFgwYJYt25d\nnHbaaTFixIia5yopvvvd78aGDRvi5Zdfjl122SXmzp1b598537F+6qmnomXLlvHTn/40Nm3aFOvX\nr49ly5bFww8/HBs2bIg1a9bEN7/5zRg6dGjNPl/72tdi8uTJERFx6623RuvWreP222+PqqqquPHG\nG7c4bo0pO2DAgLjyyitj06ZN8de//jU+//nPx6hRo/I+l+p+jx49OjZu3BgzZsyINm3axL/+9a+I\niDjrrLNi9913j5deeik2b94c3/rWt2qOX0TEXXfdFatWrYrKysqYOHFidO/ePTZu3BgREYcffnhM\nmTIlIiLWrl0bM2fOjIiIRYsWRadOnWL69OkRkXldd+7cOe/rMSL///mseKPebz2isoKpqKgC2uZE\n27JkSVW+4jutjtrcc889nHvuuRx44IG0atWKq6++mueff553332XP//5z+y///4MGTKEoqIifvCD\nH9C1a9e89bRq1Yo1a9YwZ84cIoJ99tmn1rK5Jk2axC9/+Uv69u0LwJe+9CU6dOiQt2xEsPvuu9Om\nTRu++tWvcuGFF9aM0mbNmsWHH37I2LFjadGiBb179+a8885jypQpADzwwAOMHTuWdu3aseeee/KD\nH/xgq/p/+MMfsueee7LLLrvUW1+rVq146623WL58OW3atGHAgAE18eXLl/Pmm28iiYMPPrjmvFru\nsb/sssvo1asXbdq04eqrr2bKlClUVWX+rpIoKyujdevWHHDAARx44IG8/PLLDTqmuVq1akVZWRkt\nW7Zkl112oXPnzgwZMoTWrVuz2267MWbMGP7yl7/Uun+fPn0YOXIkkhg5ciSLFy9mxYoVDSpbUVHB\nihUrePvtt3nllVcYN24cLVu25KijjuLkk0+us98tWrSgrKyMVq1aMXDgQAYNGsQDDzxQs/3000/n\n4IMPpkWLFgwfPnyL0fXw4cMpLi6mqKiIyy+/nI8++oi33noLgNatWzN//nxWrFhB27ZtOfzwwwG4\n4447GDJkCF//+tcBOOGEEzjwwAN54oknGnagt4MTlRVM9+5FwLqcaONuSbUj6qjNkiVL6NXrk3Nd\nbdu2pWPHjlRUVLBkyZItpsYAevTokbeegQMHcvHFF3PRRRexxx578L3vfY+1a9c2qA+LFi1i7733\nblBZSSxfvpx169bxm9/8hvLycjZv3gxkpssqKiro2LEjHTt2pEOHDlx99dV88MEHNc81u/+5zy33\n+dVX32233ca8efPYd999OeKII3jssceAzEn+E088kWHDhtGjRw9Gjx5NZWXlVm3lHvtevXqxefNm\n3n///ZpYdrJv06ZNg49prq5du25xXurjjz/mvPPOo1evXrRv357jjjuODz/8sNb999hjjy36AdTa\nl9yyEcHatWt577336NSpU82FHpD/b5CtU6dO7LrrrjWPe/XqtcUUZW5b2X2aOHEi++23Hx06dKBj\nx458/PHHNc/xD3/4A6+//jr77LMPRx55JH/+85+BzN/8nnvu2eJv/sILL2zR5s7iRGUFM2HC2fTp\nM45PEs06+vQZx4QJZzdpHbXZc889t5jXX7duHcuXL6d79+5069aNRYsWbVF+8eLFtdZ18cUX8+KL\nL/L6668zb948fv3rXwOZ5FKXnj178q9//avBfY4IJHHppZeyyy671FzV1rNnT/bee29WrFjBihUr\nWLlyJatXr+bRRx+tea7Z/X/33Xe3qju7r/XV16dPH+655x6WLVvGFVdcwemnn86///1vWrZsyVVX\nXcXrr7/Os88+y5/+9KeaczjZco/9woULadWqVYNHoo2R+zeYOHEiCxcu5MUXX2TVqlU15wt3pm7d\nurF8+XI2btxYE8t9feVavnw5GzZ88ruy7777LnvuuWe9bT3zzDNcd911PPTQQ6xcuZKVK1fStm3b\nmvOb/fr1495772XZsmVcdtllfPOb32Tjxo307NmTc845Z4u/+Zo1a/jRj360jc+64ZyorGBKSnox\nffolDB/+GwYOHLdNt6TaEXUAbNy4kQ0bNtQslZWVfPvb3+YPf/gDr7zyChs2bOAnP/kJRx55JHvt\ntRcnn3wyr732GlOnTqWyspKbbrppi0/72V588UVmzpzJ5s2b+dznPseuu+5a8wm+a9euvP3227X2\n67zzzuOqq66qmZZ59dVXWblyZd6y1W801caMGcM111zDxo0bGTBgAO3atWPixImsX7+eyspKXn/9\ndV588UUAhg4dytVXX82qVauoqKjgv/7rv+o8XvXVd/fdd9d8Qi8uLkYSLVq0oLy8nNdee42qqip2\n2203WrVqRcuWW198fOaZZ3LdddexYMEC1q5dy9ixYxk2bBhFRUV5n+uOtGbNGtq0aUNxcTHLly9n\n/Pid/8sIe++9N1/60pcYP348mzZt4u9//3vNKLQ2lZWVlJWVsWnTJsrLy3niiScYOnRonftAZrTX\nqlUrOnbsyMaNGxk3bhwff/xxzfa77rqL5cuXA9CuXTuKioooKipixIgRPPTQQzz11FNUVVWxfv16\nysvLWbp06fY9+QZworKCKinpxV13ZX4m5a67xm3TZeU7oo6TTz6ZNm3a8LnPfY42bdowfvx4jj32\nWCZMmMBpp51G9+7deeedd2rOwXTq1IkHHniAH//4x3Tu3Jm5c+dy2GGHbTF1U+2jjz7i/PPPp2PH\njpSUlNC5c+eaq+DOPfdcXn/9dTp27Mhpp50GbPkJ/7LLLuOMM87ghBNOoLi4mPPOO49///vfeZ9D\n7sjg5JNPpmPHjtxyyy0UFRXx6KOPMnv2bEpKSujSpQvnn39+zZV0P/vZz+jevTslJSWccMIJDB06\ndIvnklt3ffU98cQT9O/fn3bt2nHppZdy33330bp1a5YuXcrpp59OcXEx/fv3Z+DAgQwfPnyrNs45\n5xxGjBjB0UcfTZ8+fWjTpg033HBDrf2pb2Ta0DKQOearVq2iU6dOfO1rX9vqXFF99WRvb0zZe++9\nl7/85S907tyZCRMmMGzYsLyvp2o9e/akbdu2dOvWjVGjRjFp0qSaaeK62h08eDDHHXcc/fr1Y++9\n96Z9+/Z069atZvvjjz/OfvvtR3FxMVdccQX3338/LVu2pFevXjz00ENMmDCB3Xffnd69e/Pb3/62\n5rzhzuQv/G4jf+G3cdL8hd8dISLo0aMH99xzD8ccc0yhu7Pdfv/733PffffxzDPPFLorzVb1xRBj\nx47datvTTz/N+eefX+dovNB8rz+zFJg2bRqrV69mw4YN/PKXvwTgyCOPLHCvts3SpUt59tlniQjm\nzZvHtddeWzPCs6Yxa9YsFixYQETw+OOP89hjj9VctdncpTJRSfq5pJcl/VPSE5L2yNp2g6T5kmZL\nOigrPlLSm5LmSfpOVvwQSa8k267PineQNC0p/6Sk4vraMMv23HPP0adPH7p06cJjjz3GI488UudU\nTZpt3LiRCy64gHbt2vH1r3+d//iP/+D73/9+obvVrCxZsoSjjz6adu3a8aMf/Yhbb72V/fffv9Dd\nSoVUTv1J2i0i1ibrlwBfjIjvSxoMXBQRJ0s6AvhdRBwpqQPwInAIIOAfwCERsVrSC8AlETFT0uPJ\nPk9KugZYHhETJY0GOkTEGEknARfntpGnj576a4TP+tSfmW3pMz/1V52kEm2B6rN1pwB3JGVeAIol\ndQVOBKZFxOqIWAVMAwYlI7HPR8TMZP87gFOT9SHA5GR9cvK4Op6vDTMzK4DU3pRW0i+A7wCrgIFJ\nuDuQ/eWCxUksN16RFV+cpzxA14h4HyAilkrqUksb1XXlv/bYzMx2qoKNqCRNT84dVS+vJv9+AyAi\nfhoRewF3A5dU75ZbDRB54tQTr7Nr27CPmZntJAUbUUXE8Q0sei/wJ6CMzIgo+74iPYAlSbw0J/5M\nHeUBlkrqGhHvJ1OEHyTxuvbZQllZWc16aWkppaWl+YoZmdu7NPR7LGb26Vd9C6zy8nLKy8u3q660\nXkzRNyLeStYvAY6KiDNyLqY4Erg+z8UURcn6oRGxqvpiCmAW8BhwQ0Q8kVxMsSIirpE0BmifXEyR\nt408ffTFFGZmjfRZ+uHE/5T0BTIXUSwEvgcQEY9LGizpLTI3dxuVxFdKmkAmQQUwPrmoAuBC4HZg\nV+DxiKi+1e81wP2SzgHeBYbW1YaZmRVGKkdUnwYeUZmZNd5n5vJ0MzOzak5UZmaWak5UZmaWak5U\nZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaW\nak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWaqlOVJIul1QlqWNW7AZJ8yXNlnRQ\nVnykpDclzZP0naz4IZJeSbZdnxXvIGlaUv5JScX1tWFmZk0vtYlKUg/g68DCrNhJQJ+I6AdcAPw+\niXcAfgYcDhwBjMtKPP8NnBcRXwC+IOnEJD4GeCoi9gFmAFfW1YaZmRVGahMVcB3w45zYEOAOgIh4\nASiW1BU4EZgWEasjYhUwDRgkaQ/g8xExM9n/DuDUrLomJ+uTk8d1tWFmZgWQykQl6RvAooh4NWdT\nd2BR1uPFSSw3XpEVX5ynPEDXiHgfICKWAl1qaaMiax8zM2tiLQvVsKTpQPZIRUAAPwV+Ahyfb7c8\njyNPnHridXatofuUlZXVrJeWllJaWlpP1WZmzUt5eTnl5eXbVUfBElVE5EtESNof6A28LElAD+Al\nSQPIjIh6ZhXvASxJ4qU58WfqKA+wVFLXiHg/mSL8IInXtc8WshOVmZltLfdD/Pjx4xtdR+qm/iLi\ntYjYIyL2jogSMonj4Ij4AJgKfAdA0pHAqmT67kngeEnFyYUVxwNPJlN6H0kakCS97wCPJE1NBc5O\n1s/Oiedrw8zMCqBgI6pGqJnCi4jHJQ2W9BawDhiVxFdKmgC8mJQfn1xUAXAhcDuwK/B4RDyRxK8B\n7pd0DvAuMLSuNszMrDAUUd8pG8tHUvjYmZk1jiQiIt+1ALVK3dSfmZlZNicqMzNLNScqMzNLNScq\nMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNL\nNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLtVQmKknjJC2W9FKyDMradqWk+ZLe\nkHRCVnyQpLmS3pQ0OiveW9LzkuZJuldSyyTeWtKUpK7nJO1VXxtmZtb0UpmoEr+NiEOS5QkASfsB\nZwD7AScBNyujCLgJOBHoD5wpad+knmuAayNiH2AVcG4SPxdYERH9gOuBiUkbX8zXxs5/umZmlk+a\nE1W+5DAEmBIRmyNiATAfGJAs8yNiYURsAqYkZQGOBR5M1icDp2bVNTlZ/2NSDuCUWtowM7MCSHOi\nukjSbEm3SipOYt2BRVllKpJYbnwx0F1SJ2BlRFRlx3PriohKYLWkjnW0YWZmBdCyUA1Lmg50zQ4B\nAYwFbgZ+HhEh6RfAtcB55B9lBfkTbiTlc/eJrPZq2ydffCtlZWU166WlpZSWluYrZmbWbJWXl1Ne\nXr5ddRQsUUXE8Q0segvwaLK+GOiZta0HsIRMctkrNx4RH0pqL6koGVVVl8+ua4mkFkBxRKyUVFsb\nW8lOVGZmtrXcD/Hjx49vdB2pnPqTtEfWw9OA15L1qcCw5Iq9EqAvMBOYBfSV1EtSa2AY8Eiyzwxg\naLI+Mis+NXlMsn1GPW2YmVkBFGxEVY+Jkg4CqoAFwAUAETFH0v3AHGATcGFEBFAp6WJgGpnkOyki\n5iZ1jQGmSJoA/BOYlMQnAXdKmg8sJ5Pc6mrDzMwKQH4P3jaSnL/MzBpJEhHRqK/8pHLqz8zMrJoT\nlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZ\npZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpVqDEpWkoQ2JmZmZ7WgN+uFESS9FxCH1xZoT/3CimVnj\nbcsPJ9b5U/SSTgIGA90l3ZC1qR2wufFdNDMza5w6ExWwBHgROAX4R1Z8DXDpzuqUmZlZtTrPUUXE\nyxExGegbEZOT9anAWxGxcmd2TNIlkuZKelXSf2bFr5Q0X9Ibkk7Iig9Kyr8paXRWvLek5yXNk3Sv\npJZJvLWkKUldz0naq742zMys6TX0qr/pktpJ6gi8BNwi6bqd1SlJpcA3gP0j4kvAb5L4fsAZwH7A\nScDNyigCbgJOBPoDZ0raN6nuGuDaiNgHWAWcm8TPBVZERD/gemBi0sYX87Wxs56rmZnVraGJqjgi\nPgJOA+6IiCOA43Zet/g+8J8RsRkgIj5M4kOAKRGxOSIWAPOBAckyPyIWRsQmYEpSFuBY4MFkfTJw\nalZdk5P1PyblIDPNma8NMzMrgIYmqpaSupEZafxpJ/an2heAo5Mpu2ckHZrEuwOLsspVJLHc+GIy\nF4B0AlZGRFV2PLeuiKgEVicjxtraMDOzAqjvYopqPweeBP4vImZJ2pvMSGObSZoOdM0OAQH8NOlX\n+4g4UtLhwAPA3kmZXEH+hBtJ+dx9qq8pr62u2uJbKSsrq1kvLS2ltLQ0XzEzs2arvLyc8vLy7aqj\nQYkqIh4gkyyqH78NfHN7Go6I42vbJul7wP8m5WZJqkxGR4uBvbKK9iBzZaLyxSPiQ0ntJRUlo6rq\n8iR19QSWSGpBZnpzpaTqeG4bW8lOVGZmtrXcD/Hjx49vdB0NvTNFD0kPSfpA0vuSHpTUo9GtNdzD\nJOfAJH0BaB0Ry8lccfit5Iq9EqAvMBOYBfSV1EtSa2AY8EhS1wyg+i4aI7PiU5PHJNtnZMWH5WnD\nzMwKoKFTf38A7uGTN/yzklito6Lt9AfgNkmvAhuA7wBExBxJ9wNzgE3AhcntISolXQxMI5N8J0XE\n3KSuMcAUSROAfwKTkvgk4E5J84HlZJJbXW2YmVkBNPQWSrMj4qD6Ys2Jb6FkZtZ423ILpYZe9feh\npLMktUiWs8iMQszMzHaqhiaqc8hcmr4UeA84HRi1szplZmZWraHnqCYAI6tvm5R83+g3ZBKYmZnZ\nTtPQEdUB2ff2i4gVwME7p0tmZmafaGiiKpLUofpBMqJq6GjMzMxsmzU02VwLPCvpj2Tu0nAG8Mud\n1iszM7NEgy5Ph5q7ih9L5i4QT0fEnJ3ZsbTz5elmZo23LZenNzhR2ZacqMzMGm9nfo/KzMysIJyo\nzMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws\n1VKZqCRNkfRSsrwj6aWsbVdKmi/pDUknZMUHSZor6U1Jo7PivSU9L2mepHsltUzirZN25kt6TtJe\n9bVhZmZNL5WJKiKGRcQhEXEI8CDwvwCS9iPzEyP7AScBNyujCLgJOBHoD5wpad+kumuAayNiH2AV\ncG4SPxdYERH9gOuBiUkbX8zXxs5+zmZmll8qE1WOM4B7kvUhwJSI2BwRC4D5wIBkmR8RCyNiEzAl\nKQuZnyZ5MFmfDJyaVdfkZP2PSTmAU2ppw8zMCiDViUrSUcDSiHg7CXUHFmUVqUhiufHFQHdJnYCV\nEVGVHc+tKyIqgdXJLxfX1oaZmRVAwX5OXtJ0oGt2iMyvB4+NiEeT2JnAvTllcgX5E24k5XP3qf4R\nqdrqqi2+lbKyspr10tJSSktL8xUzM2u2ysvLKS8v3646CpaoIuL4urZLagGcBhySFV4M9Mx63ANY\nQia57JUbj4gPJbWXVJSMqqrLZ9e1JGmrOCJWSqqtja1kJyozM9ta7of48ePHN7qONE/9HQ+8ERHZ\nSWIqMCy5Yq8E6AvMBGYBfSX1ktQaGAY8kuwzAxiarI/Mik9NHpNsn1FPG2ZmVgAFG1E1wLfYctqP\niJgj6X5gDrAJuDD5PfhKSRcD08gk30kRMTfZbQwwRdIE4J/ApCQ+CbhT0nxgOZnkVlcbZmZWAPJ7\n8LaR5PxlZtZIkoiIRn3lJ81Tf2ZmZk5UZmaWbk5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaW\nak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5U\nZmaWak5UZmaWak5UZmaWaqlMVJIOlPScpH9Kminp8KxtN0iaL2m2pIOy4iMlvSlpnqTvZMUPkfRK\nsu36rHisIpIdAAAMQElEQVQHSdOS8k9KKq6vDTMza3qpTFTARGBcRBwMjEseI2kw0Cci+gEXAL9P\n4h2AnwGHA0cA47ISz38D50XEF4AvSDoxiY8BnoqIfYAZwJVJXSfla8PMzAojrYmqCqhONO2BimT9\nFOAOgIh4ASiW1BU4EZgWEasjYhUwDRgkaQ/g8xExM9n/DuDUZH0IMDlZn5w8ro7na8PMzAqgZaE7\nUItLgSclXQsI+EoS7w4syiq3OInlxiuy4ovzlAfoGhHvA0TEUkldammjuq73t/M5mZnZNihYopI0\nHcgeqQgIYCzwdeCHEfGwpNOB24DjkzLk2Sc3Tj3xOrvW0H3Kyspq1ktLSyktLa2najOz5qW8vJzy\n8vLtqkMR9b1vNz1JqyKife5jSb8HnomI+5L4XOAYYCBQGhHfS+K/B54B/pKU3y+JDwOOiYjvS3oj\n2ef9ZIrwmYjYr7Y2qkdfWX2KNB47M7M0k0RE5BsQ1Cqt56gqJB0DIOk4YH4Snwp8J4kfCaxKEsiT\nwPGSipMLK44HnoyIpcBHkgZIUrLvI1l1nZ2sn50Tz9eGmZkVQFrPUZ0P3CCpBbAe+C5ARDwuabCk\nt4B1wKgkvlLSBOBFMtN045OLKgAuBG4HdgUej4gnkvg1wP2SzgHeBYbW1YaZmRVGKqf+Pg089Wdm\n1nifpak/MzMzwInKzMxSzonKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxS\nzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnK\nzMxSLZWJStIBkp6V9LKkRyTtlrXtSknzJb0h6YSs+CBJcyW9KWl0Vry3pOclzZN0r6SWSby1pClJ\nXc9J2qu+NszMrOmlMlEBtwJXRMSBwEPAFQCSvgicAewHnATcrIwi4CbgRKA/cKakfZO6rgGujYh9\ngFXAuUn8XGBFRPQDrgcm1tXGTn6+ZmZWi7Qmqi9ExN+T9aeAbybrpwBTImJzRCwA5gMDkmV+RCyM\niE3AFGBIss+xwIPJ+mTg1GR9SPIY4I9JubraMDOzAkhronpN0jeS9TOAHsl6d2BRVrmKJJYbXwx0\nl9QJWBkRVdnx3LoiohJYLaljHW2YmVkBtCxUw5KmA12zQ0AAY4FzgBsl/QyYCmzMKpMryJ9wIymf\nu0/UU1dt8a2UlZXVrJeWllJaWpqvmJlZs1VeXk55efl21VGwRBURx9dT5EQASf2Ak5PYYqBnVpke\nwBIyyWWv3HhEfCipvaSiZFRVXT67riWSWgDFEbFSUm1tbCU7UZmZ2dZyP8SPHz++0XWkcupP0u7J\nv0XAT4HfJ5umAsOSK/ZKgL7ATGAW0FdSL0mtgWHAI8k+M4ChyfrIrPjU5DHJ9hn1tGFmZgVQsBFV\nPc6UdBGZKbf/jYjbASJijqT7gTnAJuDCiAigUtLFwDQyyXdSRMxN6hoDTJE0AfgnMCmJTwLulDQf\nWE4mudXVhpmZFYD8HrxtJDl/mZk1kiQiolFf+Unl1J+ZmVk1JyozM0s1JyozM0s1JyozM0s1Jyoz\nM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1\nJyozM0s1JyozM0s1JyozM0u1tP4U/WfOO+8s5Kqrbqeiooru3YuYMOFsSkp67fB9mrKttPevKdtK\ne/+asq20968p23L/dpCIKMgCnA68BlQCh+RsuxKYD7wBnJAVHwTMBd4ERmfFewPPA/OAe4GWSbw1\nMCWp6zlgr21tI0//o6HefntB9Onzo4C1ARGwNvr0+VG8/faCHbpPU7aV9v75WPhYFLot9y+/5L2z\ncfmisTvsqAXYB+gHzMhOVMB+wD/JjPZ6A28BIjNN+RbQC2gFzAb2Tfa5DxiarP83cEGy/n3g5mT9\nW8CUZP2LjW0jT/8b9EeJiBg+vCzrjxo1f9zhw8t26D5N2daW+zyTuv75WPhYFLqttB+Lpjx+2bYl\nURXsHFVEzIuI+UmCyDaETELZHBELyIx6BiTL/IhYGBGbyIyUhiT7HAs8mKxPBk7Nqmtysv7HpBzA\nKdvQxjarqKgC2uZE27JkSdUO3acp29pyn/LU9a8p2/KxqG2f8tT1rynbSvuxaMrjt73SeDFFd2BR\n1uOKJJYbXwx0l9QJWBkRVdnx3LoiohJYLaljY9vY7ifUvQhYlxNdx5571n74t2Wfpmwr7f1ryrbS\n3r+mbCvt/WvKtty/HaixQ7DGLMB04JWs5dXk329klXmGLaf+bgK+nfX4VuA/yJzT+n9Z8bOA3wGd\nyYyCquM9gJeT9deAPbO2zQc6NLaNWp5bg4a5EemfP97+fcalrn8+Fj4WhW4r7cfC56gal8xyE9UY\ntrxQ4gngCOBI4Il85YBlQFGyfiTw5+x9k/UWwAfb2kaefocXL168eGn80tg8kZbL07PPU00F7pZ0\nHZlpt77ATDLTlH0l9QLeA4YlC2QuyBhK5qKKkcAjWXWNBF5Its/YhjbOzNfhiMg9t2ZmZjtBwRKV\npFOBG8lM3f1J0uyIOCki5ki6H5gDbAIuTIaLlZIuBqaRSSiTImJuUt0YYIqkCWSu5puUxCcBd0qa\nDywnSWyNbOONnXwozMysDkqmsczMzFIpjVf9pZ6kQZLmSnpT0uhC96eQJC2Q9LKkf0qaWej+NCVJ\nkyS9L+mVrFgHSdMkzZP0pKTiQvaxqdRyLMZJWizppWQZVMg+NhVJPSTNkDRH0quSfpDEm91rI8+x\nuCSJN+q14RFVI0kqInPXiuOAJcAsYFjWNGSzIult4NCIWFnovjQ1SV8D1gJ3RMQBSewaYHlETEw+\nxHSIiDGF7GdTqOVYjAPWRMRvC9q5JiZpD2CPiJgtaTfgH2S+jzmKZvbaqONYfItGvDY8omq8nfKl\n4E+x6jt6NDsR8XcgN0Fnf8k8+8vnn2m1HAvY+gv9n3kRsTQiZifra8ncpq0HzfC1UcuxqP5uaoNf\nG83yDWY77ZQvBX+KBfCkpFmSzi90Z1KgS0S8D5n/pMDuBe5PoV0kabakW5vDVFcuSb2Bg8jci7Rr\nc35tZB2LF5JQg18bTlSNl+9TQHOeP/1KRBwGDCbzwvtaoTtkqXEz0CciDgKWAs1tCnA3Mrdu+2Ey\nmmi27xN5jkWjXhtOVI23GNgr63EPMueqmqXkkyERsQx4iMzUaHP2vqSuUDM//0GB+1MwEbEsPjkJ\nfgtweCH705QktSTzxnxnRFR/r7NZvjbyHYvGvjacqBpvFsmXgiW1JvPdrKkF7lNBSGqTfFJCUlvg\nBDK3rWpOxNZfWD87Wc/+8nlzsMWxSN6Mq51G83pt3AbMiYjfZcWa62tjq2PR2NeGr/rbBsmllL/j\nky8F/2eBu1QQkkrIjKKCzJfH725Ox0LSPUAp0Al4HxgHPAw8APQE3iXz8zOrCtXHplLLsRhI5pxE\nFbCAzM/vvF+gLjYZSV8F/krm3qbVtw36CZm739xPM3pt1HEsvk0jXhtOVGZmlmqe+jMzs1RzojIz\ns1RzojIzs1RzojIzs1RzojIzs1RzojIzs1RzojJrhiQdI+nRQvfDrCGcqMyaL3+J0j4VnKjMUkzS\ncEkvJD8u99+SiiStkfRbSa9Jmi6pU1L2IEnPJXekfrD6jtSS+iTlZkt6MbmjCMDnJT0g6Q1Jdxbs\nSZrVw4nKLKUk7UvmB+a+EhGHkLndzHCgDTAzIvYnc3uacckuk4EfJ3ekfi0rfjdwYxL/CvBeEj8I\n+AHwRaCPpK/s/Gdl1ngtC90BM6vVccAhwCxJAnYlcx+9KjL3jAO4C3hQUjugOPkBQ8gkrfuTmwZ3\nj4ipABGxESBTHTMj4r3k8WygN/BsEzwvs0ZxojJLLwGTI2LsFkHpqpxykVU+Xx212ZC1XonfDyyl\nPPVnll5PA6dL2h1AUgdJewEtgNOTMsOBv0fER8CK5G7VACOAv0TEGmCRpCFJHa0lfa5Jn4XZdvIn\nKLOUiog3JP0UmCapCNgIXAysAwYkI6v3yZzHgsxvHP1PkojeBkYl8RHA/5P086SOofma23nPxGz7\n+Gc+zD5lJK2JiM8Xuh9mTcVTf2afPv50ac2KR1RmZpZqHlGZmVmqOVGZmVmqOVGZmVmqOVGZmVmq\nOVGZmVmqOVGZmVmq/f8TBfwcp3ZQPwAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "#plotting\n",
- "plt.plot(range(len(loss_history)), loss_history, 'o', label='Logistic Regression Training phase')\n",
- "plt.ylabel('cost')\n",
- "plt.xlabel('epoch')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.5.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/2.2 CNN HandsOn - MNIST Dataset.ipynb b/2.0 - MNIST Dataset.ipynb
similarity index 81%
rename from 2.2 CNN HandsOn - MNIST Dataset.ipynb
rename to 2.0 - MNIST Dataset.ipynb
index 4c4fbc6..6c43362 100644
--- a/2.2 CNN HandsOn - MNIST Dataset.ipynb
+++ b/2.0 - MNIST Dataset.ipynb
@@ -3,21 +3,30 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
},
"source": [
- "# CNN HandsOn with Keras"
+ "# MNIST Dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Also known as `digits` if you're familiar with `sklearn`:\n",
+ "\n",
+ "```python\n",
+ "\n",
+ "from sklearn.datasets import digits\n",
+ "\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -32,8 +41,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -54,8 +61,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -72,9 +77,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -91,9 +94,7 @@
"execution_count": null,
"metadata": {
"code_folding": [],
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -107,8 +108,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -121,9 +120,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -137,9 +134,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -153,9 +148,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -169,9 +162,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -185,9 +176,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -201,9 +190,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -217,9 +204,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -233,9 +218,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -249,9 +232,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -265,8 +246,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -284,8 +263,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -302,8 +279,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -317,9 +292,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -337,8 +310,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -352,9 +323,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": []
@@ -376,9 +345,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/2.1 Hidden Layer Representation and Embeddings.ipynb b/2.1 Hidden Layer Representation and Embeddings.ipynb
new file mode 100644
index 0000000..64f17ab
--- /dev/null
+++ b/2.1 Hidden Layer Representation and Embeddings.ipynb
@@ -0,0 +1,871 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Fully Connected Feed-Forward Network\n",
+ "\n",
+ "In this notebook we will play with Feed-Forward FC-NN (Fully Connected Neural Network) for a *classification task*: \n",
+ "\n",
+ "Image Classification on MNIST Dataset\n",
+ "\n",
+ "**RECALL**\n",
+ "\n",
+ "In the FC-NN, the output of each layer is computed using the activations from the previous one, as follows:\n",
+ "\n",
+ "$$h_{i} = \\sigma(W_i h_{i-1} + b_i)$$\n",
+ "\n",
+ "where ${h}_i$ is the activation vector from the $i$-th layer (or the input data for $i=0$), ${W}_i$ and ${b}_i$ are the weight matrix and the bias vector for the $i$-th layer, respectively. \n",
+ "
\n",
+ "$\\sigma(\\cdot)$ is the activation function. In our example, we will use the *ReLU* activation function for the hidden layers and *softmax* for the last layer.\n",
+ "\n",
+ "To regularize the model, we will also insert a Dropout layer between consecutive hidden layers. \n",
+ "\n",
+ "Dropout works by “dropping out” some unit activations in a given layer, that is setting them to zero with a given probability.\n",
+ "\n",
+ "Our loss function will be the **categorical crossentropy**."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Model definition\n",
+ "Keras supports two different kind of models: the [Sequential](http://keras.io/models/#sequential) model and the [Graph](http://keras.io/models/#graph) model. The former is used to build linear stacks of layer (so each layer has one input and one output), and the latter supports any kind of connection graph.\n",
+ "\n",
+ "In our case we build a Sequential model with three [Dense](http://keras.io/layers/core/#dense) (aka fully connected) layers, with some [Dropout](http://keras.io/layers/core/#dropout). Notice that the output layer has the softmax activation function. \n",
+ "\n",
+ "The resulting model is actually a `function` of its own inputs implemented using the Keras backend. \n",
+ "\n",
+ "We apply the binary crossentropy loss and choose SGD as the optimizer. \n",
+ "\n",
+ "Please remind that Keras supports a variety of different [optimizers](http://keras.io/optimizers/) and [loss functions](http://keras.io/objectives/), which you may want to check out. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "%matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Introducing ReLU"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The **ReLu** function is defined as $f(x) = \\max(0, x),$ [1]\n",
+ "\n",
+ "A smooth approximation to the rectifier is the *analytic function*: $f(x) = \\ln(1 + e^x)$\n",
+ "\n",
+ "which is called the **softplus** function.\n",
+ "\n",
+ "The derivative of softplus is $f'(x) = e^x / (e^x + 1) = 1 / (1 + e^{-x})$, i.e. the **logistic function**.\n",
+ "\n",
+ "[1] [http://www.cs.toronto.edu/~fritz/absps/reluICML.pdf]() by G. E. Hinton "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Note: Keep in mind this function as it is heavily used in CNN"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Using TensorFlow backend.\n"
+ ]
+ }
+ ],
+ "source": [
+ "from keras.models import Sequential\n",
+ "from keras.layers.core import Dense\n",
+ "from keras.optimizers import SGD\n",
+ "\n",
+ "nb_classes = 10\n",
+ "\n",
+ "# FC@512+relu -> FC@512+relu -> FC@nb_classes+softmax\n",
+ "# ... your Code Here"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol21.py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Data preparation (`keras.dataset`)\n",
+ "\n",
+ "We will train our model on the MNIST dataset, which consists of 60,000 28x28 grayscale images of the 10 digits, along with a test set of 10,000 images. \n",
+ "\n",
+ "\n",
+ "\n",
+ "Since this dataset is **provided** with Keras, we just ask the `keras.dataset` model for training and test data.\n",
+ "\n",
+ "We will:\n",
+ "\n",
+ "* download the data\n",
+ "* reshape data to be in vectorial form (original data are images)\n",
+ "* normalize between 0 and 1.\n",
+ "\n",
+ "The `binary_crossentropy` loss expects a **one-hot-vector** as input, therefore we apply the `to_categorical` function from `keras.utilis` to convert integer labels to **one-hot-vectors**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from keras.datasets import mnist\n",
+ "from keras.utils import np_utils\n",
+ "\n",
+ "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
+ "X_train = X_train.reshape(60000, 784)\n",
+ "X_test = X_test.reshape(10000, 784)\n",
+ "X_train = X_train.astype(\"float32\")\n",
+ "X_test = X_test.astype(\"float32\")\n",
+ "\n",
+ "# Put everything on grayscale\n",
+ "X_train /= 255\n",
+ "X_test /= 255\n",
+ "\n",
+ "# convert class vectors to binary class matrices\n",
+ "Y_train = np_utils.to_categorical(y_train, 10)\n",
+ "Y_test = np_utils.to_categorical(y_test, 10)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Split Training and Validation Data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.model_selection import train_test_split\n",
+ "\n",
+ "X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(784,)"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X_train[0].shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADStJREFUeJzt3X+MXOV1xvHn8Xqxw5IAFrVxjFsoNQiEWpNuTRvaQkqJ\nMKIyqA0KqhJXRdn8EdJGStsgpAr+aCUaNRC3jVItZWWjBghSQnElNwmyUEjUxGIhKIY4BQQuce3Y\npCa13cS/T//Y62pjdt7Znbkzd5bz/UhoZu65d+7xiGfuzL537uuIEIB8FjTdAIBmEH4gKcIPJEX4\ngaQIP5AU4QeSIvxAUoQfSIrwA0kt7OfOzvCiWKyRfu4SSOWw/ldH44hns25X4bd9g6QNkoYk/VNE\n3Ftaf7FGdJWv62aXAAq2xdZZr9vxx37bQ5I+J2mtpMsl3Wb78k6fD0B/dfOdf42kVyLi1Yg4KulR\nSevqaQtAr3UT/hWSfjDt8a5q2c+wPWZ70vbkMR3pYncA6tRN+Gf6o8Jbfh8cEeMRMRoRo8Na1MXu\nANSpm/DvkrRy2uMLJO3urh0A/dJN+J+RtMr2RbbPkPRBSZvraQtAr3U81BcRx23fIemrmhrqm4iI\nF2vrDEBPdTXOHxFbJG2pqRcAfcTpvUBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK\n8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9I\nivADSRF+ICnCDyTV1Sy9tndKOijphKTjETFaR1MAeq+r8FfeFxE/quF5APQRH/uBpLoNf0j6mu1n\nbY/V0RCA/uj2Y//VEbHb9lJJT9r+fkQ8PX2F6k1hTJIW68wudwegLl0d+SNid3W7T9LjktbMsM54\nRIxGxOiwFnWzOwA16jj8tkdsv/PUfUnvl/RCXY0B6K1uPvYvk/S47VPP83BEfKWWrgD0XMfhj4hX\nJf1Kjb10xQvL/5Qjv3tlsb7zD6JYv/+aR1vWbh45VNz2sUNnF+t/+Z11xfrZ/zZSrI/88HjL2ju+\n/VJx2xM//p9iHW9fDPUBSRF+ICnCDyRF+IGkCD+QFOEHknJEeYirTu/ykrjK1/XkuYfOKQ+n/euL\nT5W3d/l98EScnHNPg2DjgXcX65/bcEuxvmzjd4r1k4cPz7kn9M622KoDsd+zWZcjP5AU4QeSIvxA\nUoQfSIrwA0kRfiApwg8kxTh/Dd48+dNi/U9ev6lYX764/LPaT58/Oeee6vI3/31Zsf7Ux99brC/4\nevk8AdSLcX4AbRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJvm3F+LRgqlg/f+KvF+us3lp/e0Xro9JKN\nPyluG89sL9YXLF5c3vmlFxXLR5a1vrT3a7eU39+3rP1ssX7JcLm3fSfK//ZrHv7zlrWL7vxWcVvM\nHeP8ANoi/EBShB9IivADSRF+ICnCDyRF+IGk2o7z256QdJOkfRFxRbVsiaQvSrpQ0k5Jt0bEm+12\n1tNxfnRk4Yrydf1f27CkWN/+Gw8V66VrHay9+8+K2y6Z4DyAuap7nH+jpBtOW3anpK0RsUrS1uox\ngHmkbfgj4mlJ+09bvE7Spur+Jkk319wXgB7r9Dv/sojYI0nV7dL6WgLQDwt7vQPbY5LGJGmxzuz1\n7gDMUqdH/r22l0tSdbuv1YoRMR4RoxExOqxFHe4OQN06Df9mSeur++slPVFPOwD6pW34bT8i6VuS\nLrW9y/btku6VdL3tlyVdXz0GMI+8fX7Pj55oNx/CK5+6vFh//kMbWtZ+EseK23IewNzxe34AbRF+\nICnCDyRF+IGkCD+QFOEHkmKoDz316sOrW9a+f81Ecdt2lwX/8B9+vFhf8I1804Mz1AegLcIPJEX4\ngaQIP5AU4QeSIvxAUoQfSKrnl/FCbqv+qjBWf0152+ULzyrWD593RrHORePKOPIDSRF+ICnCDyRF\n+IGkCD+QFOEHkiL8QFKM86O3jpYvz43mcOQHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaTajvPbnpB0\nk6R9EXFFteweSR+R9Ea12l0RsaVXTQIz2fPe8rHr4sf71Mg8NZsj/0ZJN8yw/P6IWF39R/CBeaZt\n+CPiaUn7+9ALgD7q5jv/Hba/a3vC9rm1dQSgLzoN/+clXSxptaQ9kj7TakXbY7YnbU8e05EOdweg\nbh2FPyL2RsSJiDgp6QFJawrrjkfEaESMDmtRp30CqFlH4be9fNrDWyS9UE87APplNkN9j0i6VtJ5\ntndJulvStbZXSwpJOyV9tIc9AuiBtuGPiNtmWPxgD3oB5mRk1Y+bbmFe4ww/ICnCDyRF+IGkCD+Q\nFOEHkiL8QFJcuhvz1gJH0y3Maxz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApxvkxsE7EyWL94Mvn\nFOtL62zmbYgjP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kxTg/emr32uXtV+rQmXs4dnWDVw9IivAD\nSRF+ICnCDyRF+IGkCD+QFOEHkmo7zm97paSHJJ0v6aSk8YjYYHuJpC9KulDSTkm3RsSbvWsV89E5\nv7e7422HXD42nfnD8u/9UTabI/9xSZ+MiMsk/bqkj9m+XNKdkrZGxCpJW6vHAOaJtuGPiD0R8Vx1\n/6CkHZJWSFonaVO12iZJN/eqSQD1m9N3ftsXSrpS0jZJyyJijzT1BiGumgTMK7MOv+2zJH1J0ici\n4sActhuzPWl78piOdNIjgB6YVfhtD2sq+F+IiC9Xi/faXl7Vl0vaN9O2ETEeEaMRMTqsRXX0DKAG\nbcNv25IelLQjIu6bVtosaX11f72kJ+pvD0CvzOYnvVdL+pCk7bafr5bdJeleSY/Zvl3S65I+0JsW\nMchOvO89xfpjl/19ofqO8nO3uXT3T88rH7vOLlbRNvwR8U1JblG+rt52APQLZ/gBSRF+ICnCDyRF\n+IGkCD+QFOEHkuLS3ejKzhvLZ20uHRppWWs3jt/uJ70HfvlosX5+sQqO/EBShB9IivADSRF+ICnC\nDyRF+IGkCD+QFOP8KDr5W1cW6//8+//Q5hmGOt73a8cOFeuXPFAe50cZR34gKcIPJEX4gaQIP5AU\n4QeSIvxAUoQfSIpxfhTtXVO+tv6vLWp1Vffu3fSPf1GsX/Dtf+/ZvjPgyA8kRfiBpAg/kBThB5Ii\n/EBShB9IivADSbUd57e9UtJDmroM+klJ4xGxwfY9kj4i6Y1q1bsiYkuvGkUz3v13k8X6JatvL9Zf\n+p0HW9Yu/fofF7f9pfueLdajWEU7sznJ57ikT0bEc7bfKelZ209Wtfsj4m971x6AXmkb/ojYI2lP\ndf+g7R2SVvS6MQC9Nafv/LYvlHSlpG3Vojtsf9f2hO1zW2wzZnvS9uQxHemqWQD1mXX4bZ8l6UuS\nPhERByR9XtLFklZr6pPBZ2baLiLGI2I0IkaHVZ7XDUD/zCr8toc1FfwvRMSXJSki9kbEiYg4KekB\nSWt61yaAurUNv21LelDSjoi4b9ry5dNWu0XSC/W3B6BXHFEeMLH9m5K+IWm7pob6JOkuSbdp6iN/\nSNop6aPVHwdbepeXxFW+rsuWAbSyLbbqQOyf1e+sZ/PX/m9KmunJGNMH5jHO8AOSIvxAUoQfSIrw\nA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyTV9vf8te7MfkPSf05bdJ6kH/Wt\ngbkZ1N4GtS+J3jpVZ2+/EBE/N5sV+xr+t+zcnoyI0cYaKBjU3ga1L4neOtVUb3zsB5Ii/EBSTYd/\nvOH9lwxqb4Pal0RvnWqkt0a/8wNoTtNHfgANaST8tm+w/R+2X7F9ZxM9tGJ7p+3ttp+3XZ6itve9\nTNjeZ/uFacuW2H7S9svV7YzTpDXU2z22/6t67Z63fWNDva20/ZTtHbZftP2n1fJGX7tCX428bn3/\n2G97SNJLkq6XtEvSM5Jui4jv9bWRFmzvlDQaEY2PCdv+bUmHJD0UEVdUyz4taX9E3Fu9cZ4bEZ8a\nkN7ukXSo6Zmbqwlllk+fWVrSzZL+SA2+doW+blUDr1sTR/41kl6JiFcj4qikRyWta6CPgRcRT0va\nf9ridZI2Vfc3aep/nr5r0dtAiIg9EfFcdf+gpFMzSzf62hX6akQT4V8h6QfTHu/SYE35HZK+ZvtZ\n22NNNzODZadmRqpulzbcz+naztzcT6fNLD0wr10nM17XrYnwzzT7zyANOVwdEe+RtFbSx6qPt5id\nWc3c3C8zzCw9EDqd8bpuTYR/l6SV0x5fIGl3A33MKCJ2V7f7JD2uwZt9eO+pSVKr230N9/P/Bmnm\n5plmltYAvHaDNON1E+F/RtIq2xfZPkPSByVtbqCPt7A9Uv0hRrZHJL1fgzf78GZJ66v76yU90WAv\nP2NQZm5uNbO0Gn7tBm3G60ZO8qmGMj4raUjSRET8dd+bmIHtX9TU0V6amsT04SZ7s/2IpGs19auv\nvZLulvQvkh6T9POSXpf0gYjo+x/eWvR2reY4c3OPems1s/Q2Nfja1TnjdS39cIYfkBNn+AFJEX4g\nKcIPJEX4gaQIP5AU4QeSIvxAUoQfSOr/AJdb49R+7hOAAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plt.imshow(X_train[0].reshape(28, 28))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[0 1 2 3 4 5 6 7 8 9]\n",
+ "[0 0 0 0 0 0 0 1 0 0]\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(np.asarray(range(10)))\n",
+ "print(Y_train[0].astype('int'))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADohJREFUeJzt3X+QVfV5x/HP47osBPAHEAxFDMQwqDWT1a4QozVmjA4Y\nJ5CksSEzKf3lph1pk6nTaG06Oh3rOJ38kDYZRow74kz8NTVUkjqJlppqWoa4EAsYUmTMqht2WBET\n0FZgl6d/7CHdwJ7v3b333Hvu7vN+zTD33vOcc88zd/jsufd+zz1fc3cBiOeUshsAUA7CDwRF+IGg\nCD8QFOEHgiL8QFCEHwiK8ANBEX4gqFMbubNJ1uaTNbWRuwRCeVtv6YgfttGsW1P4zWyppDWSWiR9\n093vSq0/WVO1xK6qZZcAErb4plGvW/XbfjNrkfQNScskXSBppZldUO3zAWisWj7zL5a0x91fcvcj\nkh6WtLyYtgDUWy3hnyvp1WGPe7Nlv8bMOs2s28y6j+pwDbsDUKRawj/Slwon/T7Y3de5e4e7d7Sq\nrYbdAShSLeHvlTRv2OOzJe2trR0AjVJL+J+TtNDMFpjZJEmflrSxmLYA1FvVQ33uPmBmqyV9X0ND\nfV3u/kJhnQGoq5rG+d39CUlPFNQLgAbi9F4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAo\nwg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmroFN3NrGXmjGTdJk2q\n2753/dW7k/XLL9mVrLdP782t/cWMl5Lb3n9wdrL+8N5LkvV/WfSdZH3hhj/NrS26eWdy22NvvZWs\nozYc+YGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKHP36jc265F0SNKgpAF370itf5rN8CV2VdX7S2k5\n4/Rkfc/a+cn63168MVn/5LT9Y20JFbR/48+S9bPv/M8GdTJxbPFNOugHbDTrFnGSz4fdnWQA4wxv\n+4Ggag2/S3rSzLaaWWcRDQFojFrf9l/m7nvNbLakp8zsp+7+zPAVsj8KnZI0We+ocXcAilLTkd/d\n92a3/ZI2SFo8wjrr3L3D3Tta1VbL7gAUqOrwm9lUM5t+/L6kaySlf6YFoGnU8rb/LEkbzOz48zzo\n7t8rpCsAdVd1+N39JUnvL7CXmtiUKcn6C1d0NaiTk925/33J+sGBycn6hs3p39TP+HH9Bm3e+M30\neSArPvSjZP3Od23JrS1a+mJy27fuTJZRI4b6gKAIPxAU4QeCIvxAUIQfCIrwA0FNmEt3+5Ejyfp9\nvzwnWf/6A8uT9TN3D465p+Omfb+2S1QvVP5wWb3NrFCvdFbX270DubXfm7M5ue1avbfCs6MWHPmB\noAg/EBThB4Ii/EBQhB8IivADQRF+IKgJM84/+PqBZP07yy5O1ue+XL/LRB+r2zOX78WvL0nWJ9tz\nDeoEY8WRHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCmjDj/JUMvPxq2S2MS/s7L03Wd65Yk6yfopbc\n2hf/6bPJbRco/Xt/1IYjPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVXGc38y6JF0nqd/dL8yWzZD0\niKT5knokXe/ub9SvTdTN4vT04ZM/sS9Zb7X8cXxJWtXzkdza2U8fTW6L+hrNkf9+SUtPWHaLpE3u\nvlDSpuwxgHGkYvjd/RlJJ14mZ7mk9dn99ZJWFNwXgDqr9jP/We7eJ0nZ7eziWgLQCHU/t9/MOiV1\nStJkvaPeuwMwStUe+feZ2RxJym7781Z093Xu3uHuHa1qq3J3AIpWbfg3SlqV3V8l6fFi2gHQKBXD\nb2YPSdosaZGZ9ZrZH0m6S9LVZvaipKuzxwDGkYqf+d19ZU7pqoJ7QY6WWTOTdZs+Lbf2s8/MTW57\n9x/cm6x/eMrbyXrf4P8m63vveG9ure1JrulfJs7wA4Ii/EBQhB8IivADQRF+ICjCDwQV5tLd49mu\nuxYk67uX3dOgTk629Ed/kqzP+153gzrBWHHkB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOdHTf7r\n0vXJ+nX/ujy39uoPzklu+xvPpn9O3PKDbck60jjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPOj\nrr57XmI+l/PS2/b+cfqy4Nfe98VkfcE9e3Jrg/tyJ5kKgyM/EBThB4Ii/EBQhB8IivADQRF+ICjC\nDwRl7p5ewaxL0nWS+t39wmzZ7ZJukPRattqt7v5EpZ2dZjN8iTGz91idOj/9u/eB2ac3qJOx67nJ\ncmt//r6nk9t2nt5T077Pe/TG3NqiO3Yntx18/UBN+y7LFt+kg34g/0UfZjRH/vslLR1h+dfcvT37\nVzH4AJpLxfC7+zOSxuefQQC5avnMv9rMtptZl5mdWVhHABqi2vCvlXSupHZJfZK+kreimXWaWbeZ\ndR/V4Sp3B6BoVYXf3fe5+6C7H5N0r6TFiXXXuXuHu3e0qq3aPgEUrKrwm9mcYQ8/LmlnMe0AaJSK\nP+k1s4ckXSlplpn1SrpN0pVm1i7JJfVI+lwdewRQBxXH+YvEOD+Ga5k1M1kfWDQvWT/4128m68++\n/5Hc2pI7Vie3fefazcl6syp6nB/ABET4gaAIPxAU4QeCIvxAUIQfCIqhPoxfp7Qky+1bB3NrrZZf\nk6Stv31Gsn7s0KFkvSwM9QGoiPADQRF+ICjCDwRF+IGgCD8QFOEHgmKK7iaw+55LkvXz7/5F+gn6\n8qebHvzFL6tpaVw4ZcrkZH16y+u5tZtnvpDcdsXUa5P1Zh3nHwuO/EBQhB8IivADQRF+ICjCDwRF\n+IGgCD8QFOP8DXDodz+QrO/46N3Jett1rcn6Fduvz62dtmzijvP/dM35yfrGmffk1n585FhyWx8Y\nqKqn8YQjPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVXGc38zmSXpA0rskHZO0zt3XmNkMSY9Imi+p\nR9L17v5G/Vodv87Ynv+7ckn6H09fQ77N0uP8D16wPrf2+X//neS2rzz2nmR92s/TvVUydUN3bs0u\nTo/T918yPVl/5Kp/qLD3/GPbZzbfkNzy3P3PV3ju8W80R/4BSTe5+/mSPiDpRjO7QNItkja5+0JJ\nm7LHAMaJiuF39z5335bdPyRpl6S5kpZLOn7IWS9pRb2aBFC8MX3mN7P5ki6StEXSWe7eJw39gZA0\nu+jmANTPqMNvZtMkPSbpC+5+cAzbdZpZt5l1H9XhanoEUAejCr+ZtWoo+N9y929ni/eZ2ZysPkfS\niFeRdPd17t7h7h2taiuiZwAFqBh+MzNJ90na5e5fHVbaKGlVdn+VpMeLbw9AvVScotvMLpf0rKQd\nGhrqk6RbNfS5/1FJ50h6RdKn3P1A6rmYontkr9z2wWR9e+c/NqiT4n2p/7dyax87fVty28Vt9Zs+\n/sKu1cn6/L/ZXLd919NYpuiuOM7v7j+UlPdkJBkYpzjDDwiK8ANBEX4gKMIPBEX4gaAIPxBUxXH+\nIjHOn8PSw7J7//LSZH3Kh17Lrf1H+8NVtTQRfPBL+WP5Mx9Mn2Pgh8fnqehjGefnyA8ERfiBoAg/\nEBThB4Ii/EBQhB8IivADQTHOPwHYqfm/zG6ZNTO57c9uODdZv/8P1yTrF02q3/Hjmp98Ilnv/7e5\nyfq8L+dfNtyPHqmqp2bHOD+Aigg/EBThB4Ii/EBQhB8IivADQRF+ICjG+YEJhHF+ABURfiAowg8E\nRfiBoAg/EBThB4Ii/EBQFcNvZvPM7Gkz22VmL5jZ57Plt5vZz83s+ezftfVvF0BR8q8C8f8GJN3k\n7tvMbLqkrWb2VFb7mrt/uX7tAaiXiuF39z5Jfdn9Q2a2S1L6EioAmt6YPvOb2XxJF0naki1abWbb\nzazLzM7M2abTzLrNrPuoxucUSMBENOrwm9k0SY9J+oK7H5S0VtK5kto19M7gKyNt5+7r3L3D3Tta\n1VZAywCKMKrwm1mrhoL/LXf/tiS5+z53H3T3Y5LulbS4fm0CKNpovu03SfdJ2uXuXx22fM6w1T4u\naWfx7QGol9F823+ZpM9K2mFmz2fLbpW00szaJbmkHkmfq0uHAOpiNN/2/1DSSL8PfqL4dgA0Cmf4\nAUERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmroFN1m9pqk\nl4ctmiVpf8MaGJtm7a1Z+5LorVpF9vZud3/naFZsaPhP2rlZt7t3lNZAQrP21qx9SfRWrbJ6420/\nEBThB4IqO/zrSt5/SrP21qx9SfRWrVJ6K/UzP4DylH3kB1CSUsJvZkvN7L/NbI+Z3VJGD3nMrMfM\ndmQzD3eX3EuXmfWb2c5hy2aY2VNm9mJ2O+I0aSX11hQzNydmli71tWu2Ga8b/rbfzFok7ZZ0taRe\nSc9JWunuP2loIznMrEdSh7uXPiZsZldIelPSA+5+Ybbs7yUdcPe7sj+cZ7r7zU3S2+2S3ix75uZs\nQpk5w2eWlrRC0u+rxNcu0df1KuF1K+PIv1jSHnd/yd2PSHpY0vIS+mh67v6MpAMnLF4uaX12f72G\n/vM0XE5vTcHd+9x9W3b/kKTjM0uX+tol+ipFGeGfK+nVYY971VxTfrukJ81sq5l1lt3MCM7Kpk0/\nPn367JL7OVHFmZsb6YSZpZvmtatmxuuilRH+kWb/aaYhh8vc/WJJyyTdmL29xeiMaubmRhlhZumm\nUO2M10UrI/y9kuYNe3y2pL0l9DEid9+b3fZL2qDmm3143/FJUrPb/pL7+ZVmmrl5pJml1QSvXTPN\neF1G+J+TtNDMFpjZJEmflrSxhD5OYmZTsy9iZGZTJV2j5pt9eKOkVdn9VZIeL7GXX9MsMzfnzSyt\nkl+7ZpvxupSTfLKhjLsltUjqcve/a3gTIzCz92joaC8NTWL6YJm9mdlDkq7U0K++9km6TdI/S3pU\n0jmSXpH0KXdv+BdvOb1dqaG3rr+aufn4Z+wG93a5pGcl7ZB0LFt8q4Y+X5f22iX6WqkSXjfO8AOC\n4gw/ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANB/R+rqTZuA0QxYwAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plt.imshow(X_val[0].reshape(28, 28))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[0 1 2 3 4 5 6 7 8 9]\n",
+ "[0 0 0 1 0 0 0 0 0 0]\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(np.asarray(range(10)))\n",
+ "print(Y_val[0].astype('int'))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Training\n",
+ "Having defined and compiled the model, it can be trained using the `fit` function. We also specify a validation dataset to monitor validation loss and accuracy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Train on 45000 samples, validate on 15000 samples\n",
+ "Epoch 1/2\n",
+ "45000/45000 [==============================] - 6s - loss: 1.2717 - acc: 0.7238 - val_loss: 0.6574 - val_acc: 0.8535\n",
+ "Epoch 2/2\n",
+ "45000/45000 [==============================] - 6s - loss: 0.5226 - acc: 0.8696 - val_loss: 0.4406 - val_acc: 0.8867\n"
+ ]
+ }
+ ],
+ "source": [
+ "network_history = model.fit(X_train, Y_train, batch_size=128, \n",
+ " epochs=2, verbose=1, validation_data=(X_val, Y_val))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Plotting Network Performance Trend\n",
+ "The return value of the `fit` function is a `keras.callbacks.History` object which contains the entire history of training/validation loss and accuracy, for each epoch. We can therefore plot the behaviour of loss and accuracy during the training phase."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOX5//H3nT2BLIQQFsMS2ZMYIISlbAmgFBBcKUrF\nFkWpuKAiP6v92opba1sUiqKIC1apKK0W2QStJLIIQpBFCEECAVkNCRCWELI9vz9mSAYIIZBMzkzm\nfl3XXJ0588zkPkj55DnnWcQYg1JKKQXgZXUBSimlXIeGglJKqTIaCkoppcpoKCillCqjoaCUUqqM\nhoJSSqkyGgpKKaXKaCgopZQqo6GglFKqjI/VBVypiIgI06pVK6vLUEopt7Jhw4YcY0yjy7Vzu1Bo\n1aoVaWlpVpehlFJuRUT2VqWdXj5SSilVRkNBKaVUGQ0FpZRSZdzunoJSqu4oKipi//79FBQUWF1K\nnREQEEBUVBS+vr5X9XkNBaWUZfbv309wcDCtWrVCRKwux+0ZY8jNzWX//v1ER0df1Xfo5SOllGUK\nCgpo2LChBkINEREaNmxYrZ6XhoJSylIaCDWrun+eHhMKOafO8sKidI7nF1pdilJKuSyPCYXVmTnM\nXp1F/ympzFm7l5JS3ZtaKU+Xm5tL586d6dy5M02aNOGaa64pe11YWLVfIO+55x527NhRaZsZM2bw\nr3/9qyZKdjoxxr3+cUxMTDRXO6N5+6ETPLdwG2t3HyWmaQiTb4qle3R4DVeolKqq7du307FjR6vL\nAGDy5MnUr1+fSZMmnXfcGIMxBi8v9/kduqI/VxHZYIxJvNxn3ecsa0DHpiHMvb8nM36dwPH8Qka+\ntYYJczdyKO+M1aUppVxIZmYmcXFxPPDAAyQkJHDo0CHGjRtHYmIisbGxPP/882Vt+/Tpw6ZNmygu\nLiYsLIynnnqKTp068Ytf/ILs7GwAnnnmGaZNm1bW/qmnnqJ79+60b9+eb7/9FoDTp09z++2306lT\nJ0aNGkViYiKbNm2q9XP3uCGpIsKN8U0Z0CGSmd/sYuY3u/gq/WceHtCGsX2iCfD1trpEpTzScwu3\nkX7wRI1+Z0yzEJ4dHntVn01PT2f27NnMnDkTgJdffpnw8HCKi4vp378/I0aMICYm5rzP5OXlkZSU\nxMsvv8zEiRN57733eOqppy76bmMM69atY8GCBTz//PMsXbqU1157jSZNmvDpp5+yefNmEhISrqru\n6vKonoKjQD9vHr+hHf+bmERy+0b8fdkOBk1dwZfbDuNul9SUUjWvdevWdOvWrez13LlzSUhIICEh\nge3bt5Oenn7RZwIDAxkyZAgAXbt2Zc+ePRV+92233XZRm1WrVnHnnXcC0KlTJ2Jjry7MqsvjegoX\nah4exJuju7I6M4fnFm5j3Icb6Ns2gmeHx9AmMtjq8pTyGFf7G72z1KtXr+z5zp07+cc//sG6desI\nCwtj9OjRFc4F8PPzK3vu7e1NcXFxhd/t7+9/URtX+WXUY3sKF+rdJoLFE/ry7PAYNu07zuBpK3lx\nUTonCoqsLk0pZbETJ04QHBxMSEgIhw4dYtmyZTX+M/r06cO8efMA+OGHHyrsidQGj+8pOPL19uKe\n3tHc1KkZU77cwburs5i/6QBP/rIDI7pG4eWlk2yU8kQJCQnExMQQFxfHtddeS+/evWv8ZzzyyCP8\n5je/IT4+noSEBOLi4ggNDa3xn3M5HjUk9Ur9sD+PyQu3sWHvMTpFhfLsTbEktGhQKz9bKU/gSkNS\nrVZcXExxcTEBAQHs3LmTQYMGsXPnTnx8rvx39+oMSdWeQiWuiwrlPw/8gs83HeTPS7Zz2xvfcntC\nFL8f0p7I4ACry1NK1SGnTp1i4MCBFBcXY4zhrbfeuqpAqC4NhcsQEW7pcg3XxzRmRkom767MYtm2\nw0wY2IYxvaLx89HbMkqp6gsLC2PDhg1Wl6E3mquqvr8Pvx/cgWWP96NHdDh/XpLB4GkrSNmRbXVp\nSilVYzQUrlB0RD3eHdON2ffYxi/fM3s9Y99fz56c0xZXppRS1aehcJX6t49k6WP9+MPQDqzdncug\nqSv469IMTp+teFyyUkq5A6eFgoi8JyLZIrL1Eu/fJSJb7I9vRaSTs2pxFj8fL8b1a03KpGSGd2rG\nm6m7GPBKKvM3HnCZiShKKXUlnNlTeB8YXMn7WUCSMSYeeAGY5cRanCoyJIBXRnbiswd70TgkgMc+\n2cSImWvYeiDP6tKUUpVITk6+aCLatGnTePDBBy/5mfr16wNw8OBBRowYccnvvdzQ+WnTppGfn1/2\neujQoRw/fryqpTuN00LBGLMCOFrJ+98aY47ZX64FopxVS21JaNGA+Q/25m8j4tmbe5rhr6/i6c+2\nkHvqrNWlKaUqMGrUKD7++OPzjn388ceMGjXqsp9t1qwZ//nPf676Z18YCkuWLCEsLOyqv6+muMo9\nhbHAF5d6U0TGiUiaiKQdOXKkFsu6cl5ewsjE5iyflMzY3tH8O20/yVNSmb06i6KSUqvLU0o5GDFi\nBIsWLeLsWdsvbnv27OHgwYN07tyZgQMHkpCQwHXXXcfnn39+0Wf37NlDXFwcAGfOnOHOO+8kPj6e\nO+64gzNnypfjHz9+fNmS288++ywA06dP5+DBg/Tv35/+/fsD0KpVK3JycgB49dVXiYuLIy4urmzJ\n7T179tCxY0fuv/9+YmNjGTRo0Hk/p6ZYPk9BRPpjC4U+l2pjjJmF/fJSYmKiW1ysDwnw5ZlhMdzZ\nvTnPLUznuYXpzF33E88Oj6V3mwiry1PK9XzxFBz+oWa/s8l1MOTlS77dsGFDunfvztKlS7n55pv5\n+OOPueOOOwgMDOS///0vISEh5OTk0LNnT2666aZL7n/85ptvEhQUxJYtW9iyZct5y16/9NJLhIeH\nU1JSwsCBA9myZQsTJkzg1VdfJSUlhYiI8/892LBhA7Nnz+a7777DGEOPHj1ISkqiQYMG7Ny5k7lz\n5/L2228zcuRIPv30U0aPHl0zf1Z2lvYURCQeeAe42RiTa2UtztImMpgP7u3OrLu7cqaohLve+Y4H\nPtzAvqP5l/+wUsrpHC8hnbt0ZIzhD3/4A/Hx8Vx//fUcOHCAn3/++ZLfsWLFirJ/nOPj44mPjy97\nb968eSQkJNClSxe2bdt22YXuVq1axa233kq9evWoX78+t912GytXrgQgOjqazp07A5UvzV0dlvUU\nRKQF8BlwtzHmR6vqqA0iwqDYJvRr14h3Vu5mRsouUnZk87uk1oxPak2gn27so1Rlv9E70y233MLE\niRP5/vvvOXPmDAkJCbz//vscOXKEDRs24OvrS6tWrSpcKttRRb2IrKwspkyZwvr162nQoAFjxoy5\n7PdUNnLx3JLbYFt22xmXj5w5JHUusAZoLyL7RWSsiDwgIg/Ym/wJaAi8ISKbRKR2VrmzUICvNw8P\naMvySUn8MrYJ07/eycBXUlm85ZAOYVXKIvXr1yc5OZl777237AZzXl4ekZGR+Pr6kpKSwt69eyv9\njn79+vGvf/0LgK1bt7JlyxbAtuR2vXr1CA0N5eeff+aLL8pvnQYHB3Py5MkKv2v+/Pnk5+dz+vRp\n/vvf/9K3b9+aOt3LclpPwRhT6e17Y8x9wH3O+vmurGloINNHdWF0z5Y8u2AbD330PT2vDWfyTbF0\naBJidXlKeZxRo0Zx2223lV1Guuuuuxg+fDiJiYl07tyZDh06VPr58ePHc8899xAfH0/nzp3p3r07\nYNtBrUuXLsTGxl605Pa4ceMYMmQITZs2JSUlpex4QkICY8aMKfuO++67jy5dujjlUlFFdOlsi5WU\nGuau+4kpX+7gxJkiRvdsycQb2hEW5Hf5Dyvl5nTpbOeoztLZrjIk1WN5ewmje7YkdVIyo3u2ZM7a\nvfSfksqctXspKXWvwFZKuT8NBRcRFuTH8zfHsXhCX9o3CeaZ+VsZ/toq1mVdcv6fUkrVOA0FF9Ox\naQhz7+/JjF8ncDy/kJFvrWHC3I0cyqv5UQZKuQJ3u4Tt6qr756mh4IJEhBvjm/L1E8lMGNiWZdsO\nM2DKN8xIyaSgqMTq8pSqMQEBAeTm5mow1BBjDLm5uQQEXP3OkHqj2Q3sO5rPn5ds54uth2kRHsQz\nN3bkhpjGl5xdqZS7KCoqYv/+/Zcdu6+qLiAggKioKHx9fc87XtUbzRoKbmR1Zg6TF2xjZ/Yp+raN\n4NnhMbSJDLa6LKWUG9DRR3VQ7zYRLHm0L88Oj2HTvuMMnraSFxelc6KgyOrSlFJ1hIaCm/H19uKe\n3tGkTkrmV4lRvLs6iwFTUpm3fh+lOoRVKVVNGgpuqmF9f/5yWzwLHupDi/Agnvx0C7e+sZrvfzp2\n+Q8rpdQlaCi4ueuiQvl0fC+m3dGZQ3kF3PbGtzwxbzPZJ/XGnVLqymko1AEiwi1drmH5pGTGJ7dm\n4eaDDJjyDbNW7KKwWDf2UUpVnYZCHVLf34ffD+7Assf70SM6nD8vyWDwtBWk7Mi2ujSllJvQUKiD\noiPq8e6Ybswe0w0D3DN7PWPfX8+enNNWl6aUcnEaCnVY/w6RLHusH08P6cDa3bkMmrqCvy7N4PTZ\nYqtLU0q5KA2FOs7Px4vfJbUmZVIywzs1483UXQx4JZX5Gw/o0gJKqYtoKHiIyJAAXhnZic8e7EXj\nkAAe+2QTI2auYeuBPKtLU0q5EA0FD5PQogHzH+zN326PZ2/uaYa/voqnP9tC7qmzVpemlHIBGgoe\nyMtLGNmtOcsnJTO2dzT/TttP8pRUZq/OoqhEh7Aq5ck0FDxYSIAvzwyLYeljfencPIznFqZz4/SV\nrM7Msbo0pZRFNBQUbSKD+eDe7sy6uytnikq4653veODDDew7mm91aUqpWuZjdQHKNYgIg2Kb0K9d\nI95ZuZsZKbtI2ZHN75JaMz6pNYF+3laXqJSqBdpTUOcJ8PXm4QFtWT4piV/GNmH61zsZ+Eoqi7cc\n0iGsSnkADQVVoaahgUwf1YVPxvUkNMiPhz76nlFvryXj8AmrS1NKOZGGgqpUj2sbsuiRPrx4SxwZ\nh08y9B8r+dPnWzmeX2h1aUopJ9BQUJfl7SWM7tmS1EnJjO7Zkjlr99J/Sipz1u6lRDf2UapO0VBQ\nVRYW5MfzN8exeEJf2jcJ5pn5Wxn+2irWZR21ujSlVA3RUFBXrGPTEObe35MZv07geH4hI99aw4S5\nGzmUd8bq0pRS1aShoK6KiHBjfFO+fiKZCQPbsnTbYQZM+YYZKZkUFJVYXZ5S6ippKKhqCfTzZuIN\n7fh6YhJJ7Rrx92U7GDR1BV+l/6xDWJVyQxoKqkY0Dw9i5t1d+dd9PfD38eL+D9L4zXvryMw+aXVp\nSqkroKGgalTvNhEsebQvzw6PYdO+4wyetpIXF6VzoqDI6tKUUlWgoaBqnK+3F/f0jiZ1UjK/Sozi\n3dVZDJiSyry0fZTqEFalXJqGgnKahvX9+ctt8Sx4qA8twoN48j9buPWN1Wz86ZjVpSmlLkFDQTnd\ndVGhfDq+F9Pu6MyhvAJufeNbnpi3meyTBVaXppS6gIaCqhUiwi1drmH5pGTGJ7dmweYDDJjyDbNW\n7KKwWDf2UcpVaCioWlXf34ffD+7Al48n0SM6nD8vyWDwtBWk7Mi2ujSlFBoKyiLREfV4d0w3Zo/p\nhgHumb2ese+vZ0/OaatLU8qjOS0UROQ9EckWka2XeF9EZLqIZIrIFhFJcFYtynX17xDJssf68fSQ\nDqzdncugqSv469IMTp8ttro0pTySM3sK7wODK3l/CNDW/hgHvOnEWpQL8/Px4ndJrUmZlMzwTs14\nM3UXA15JZf7GAzorWqla5rRQMMasACpbPvNm4ANjsxYIE5GmzqpHub7IkABeGdmJzx7sReOQAB77\nZBMjZq5h64E8q0tTymNYeU/hGmCfw+v99mPKwyW0aMD8B3vzt9vj2ZNzmuGvr+Lpz34g99RZq0tT\nqs6zMhSkgmMVXisQkXEikiYiaUeOHHFyWcoVeHkJI7s1Z/mkZO7tHc2/0/bRf0oqs1dnUVSiQ1iV\nchYrQ2E/0NzhdRRwsKKGxphZxphEY0xio0aNaqU45RpCA33547AYlj7Wl07Nw3huYTo3Tl/J6swc\nq0tTqk6yMhQWAL+xj0LqCeQZYw5ZWI9yYW0ig/ng3u7MursrZ4pKuOud7xg/ZwP7juZbXZpSdYqP\ns75YROYCyUCEiOwHngV8AYwxM4ElwFAgE8gH7nFWLapuEBEGxTahX7tGvLNyNzNSdrE8I5vfJbVm\nfFJrAv28rS5RKbcn7jbkLzEx0aSlpVldhnIBh/LO8JclGSzYfJBmoQH8340xDL2uCSIV3a5SyrOJ\nyAZjTOLl2umMZuW2moYGMn1UFz4Z15PQID8e+uh7Rr29lozDJ6wuTSm3paGg3F6Paxuy6JE+vHhL\nHBmHTzL0Hyt59vOtHM8vtLo0pdyOhoKqE7y9hNE9W5I6KZnRPVvy4dq99J+Sypy1eynRjX2UqjIN\nBVWnhAX58fzNcSye0Jd2jYN5Zv5Whr+2inVZlU2uV0qdo6Gg6qSOTUP4eFxPXv91F47nFzLyrTVM\nmLuRQ3lnrC5NKZemoaDqLBFhWHwzvn4imQkD27J022EGTPmGGSmZFBSVWF2eUi5JQ0HVeYF+3ky8\noR1fT0wiqV0j/r5sB4OmruCr9J91FValLqChoDxG8/AgZt7dlTlje+Dv48X9H6Txm/fWkZl90urS\nlHIZGgrK4/RpG8GSR/vyp2ExbNp3nMHTVvLionROFBRZXZpSltNQUB7J19uLe/tEkzopmV8lRvHu\n6iwGTEllXto+SnUIq/JgGgrKozWs789fbotnwUN9aBEexJP/2cKtb6xm40/HrC5NKUtoKCgFXBcV\nyqfjezH1jk4cyivg1je+5Yl5m8k+WWB1aUrVKg0FpexEhFu7RLF8UjLjk1uzYPMBBkz5hlkrdlFY\nrBv7KM+goaDUBer7+/D7wR348vEkekSH8+clGQyetoKUHdlWl6aU02koKHUJ0RH1eHdMN2aP6YYB\n7pm9nrHvr2dPzmmrS1PKaTQUlLqM/h0iWfZYP54e0oG1u3MZNHUFf12awemzxVaXplSN01BQqgr8\nfLz4XVJrUiYlM6xTU95M3cWAV1KZv/GAzopWdYqGglJXIDIkgFdHduazB3vROCSAxz7ZxIiZa9h6\nIM/q0pSqERoKSl2FhBYNmP9gb/52ezx7ck4z/PVVPP3ZD+SeOmt1aUpVi4aCUlfJy0sY2a05yycl\nc2/vaP6dto/+U1KZvTqLohIdwqrcU5VCQURai4i//XmyiEwQkTDnlqaUewgN9OWPw2L44tG+dGoe\nxnML07lx+kpWZ+ZYXZpSV6yqPYVPgRIRaQO8C0QDHzmtKqXcUNvGwXxwb3dm3d2VM0Ul3PXOd4yf\ns4F9R/OtLk2pKqtqKJQaY4qBW4FpxpjHgabOK0sp9yQiDIptwlePJzFpUDtSdxzh+le/4dWvfuRM\noW7so1xfVUOhSERGAb8FFtmP+TqnJKXcX4CvNw8PaMvXTyQxKLYJ07/eycBXUlm85ZAOYVUuraqh\ncA/wC+AlY0yWiEQDc5xXllJ1Q7OwQF4b1YVPxvUkNMiPhz76nlFvryXj8AmrS1OqQnKlv7WISAOg\nuTFmi3NKqlxiYqJJS0uz4kcrVS0lpYa5635iypc7OHGmiLt7tuTxG9oRFuRndWnKA4jIBmNM4uXa\nVXX0UaqIhIhIOLAZmC0ir1a3SKU8ibeXMLpnS1InJTO6Z0s+XLuX/lNSmbN2LyW6sY9yEVW9fBRq\njDkB3AbMNsZ0Ba53XllK1V1hQX48f3Mciyf0pV3jYJ6Zv5Xhr61iXdZRq0tTqsqh4CMiTYGRlN9o\nVkpVQ8emIXw8riev/7oLx/MLGfnWGibM3cihvDNWl6Y8WFVD4XlgGbDLGLNeRK4FdjqvLKU8g4gw\nLL4Z/3siiQkD2rB022EGTPmGGSmZFBTpEFZV+674RrPV9Eazqsv2Hc3npcXbWbrtMC3Cg/jjsBiu\n7xiJiFhdmnJzNX2jOUpE/isi2SLys4h8KiJR1S9TKeWoeXgQM+/uypyxPfD38eL+D9L47ez1ZGaf\nsro05SGqevloNrAAaAZcAyy0H1NKOUGfthEsebQvfxoWw8afjjF42gpeXJTOiYIiq0tTdVxVQ6GR\nMWa2MabY/ngfaOTEupTyeL7eXtzbJ5qUScmM6BrFu6uzGDAllXlp+yjVIazKSaoaCjkiMlpEvO2P\n0UCuMwtTStlE1Pfn5dvjWfBQH1qEB/Hkf7Zw6xur2fjTMatLU3VQVUPhXmzDUQ8Dh4AR2Ja+UErV\nkuuiQvl0fC+m3tGJQ3kF3PrGtzwxbzPZJwusLk3VIVc9+khEHjPGTKvhei5LRx8pBafOFvP68kze\nXbUbfx9vJgxsw5he0fj56L5ZqmI1OvroEiZW47NKqWqo7+/DU0M68OXjSfSIDufPSzIYPG0FqTuy\nrS5NubnqhIIOnFbKYtER9Xh3TDdmj+mGAcbMXs99/1zPnpzTVpem3FR1QuGy151EZLCI7BCRTBF5\nqoL3W4hIiohsFJEtIjK0GvUo5bH6d4hk2WP9eHpIB9bsymXQ1BX8dWkGp88WW12acjOV3lMQkZNU\n/I+/AIHGGJ9KPusN/AjcAOwH1gOjjDHpDm1mARuNMW+KSAywxBjTqrKC9Z6CUpXLPlHAy0sz+Oz7\nAzQO8efpIR25uXMznRXt4WrknoIxJtgYE1LBI7iyQLDrDmQaY3YbYwqBj4GbL/wRQIj9eShw8HIF\nK6UqFxkSwKsjO/Pp+F5EBgfw2Ceb+NXMNWw9kGd1acoNOHOowjXAPofX++3HHE0GRovIfmAJ8EhF\nXyQi40QkTUTSjhw54oxalapzurZswOcP9eZvt8eTlXOa4a+v4unPfiD31FmrS1MuzJmhUFFf9cJL\nUaOA940xUcBQ4EMRuagmY8wsY0yiMSaxUSOdSK1UVXl5CSO7NWf5pGTu7R3Nv9P20X9KKu+vzqK4\npNTq8pQLcmYo7AeaO7yO4uLLQ2OBeQDGmDVAABDhxJqU8kihgb78cVgMXzzal/ioMCYvTGfo9JWs\nzsyxujTlYpwZCuuBtiISLSJ+wJ3YFtVz9BMwEEBEOmILBb0+pJSTtG0czIdju/PW3V3JLyzhrne+\nY/ycDew7mm91acpFOC0UjDHFwMPYNufZDswzxmwTkedF5CZ7syeA+0VkMzAXGGPcbYMHpdyMiPDL\n2Cb8b2IST9zQjtQdR7j+1W+Y+tWPnCnUjX08nW6yo5SHO3j8DH/5IoOFmw9yTVggfxjakaHXNdEh\nrHVMbSxzoZSqA5qFBfLaqC58Mq4nwQE+PPTR94x6ey0Zh09YXZqygIaCUgqAHtc2ZNEjfXjhljgy\nDp9k6D9W8uznWzmeX2h1aaoWaSgopcr4eHtxd8+WpE5KZnTPlny4di/9p6QyZ+1eSnRjH4+goaCU\nukhYkB/P3xzH4gl9adc4mGfmb2X4a6tYl3XU6tKUk2koKKUuqWPTED4e15PXf92F4/mFjHxrDRPm\nbuRQ3hmrS1NOoqGglKqUiDAsvhn/eyKJCQPasHTbYQZM+YYZKZkUFOkQ1rpGQ0EpVSVBfj5MHNSe\nrycm0a9dBH9ftoNBU1fwVfrPuNvQdnVpGgpKqSvSPDyIt+5OZM7YHvj7eHH/B2n8dvZ6MrNPWV2a\nqgEaCkqpq9KnbQRLHu3Ln4bFsPGnYwyetoIXF6VzoqDI6tJUNWgoKKWumq+3F/f2iSZlUjIjukbx\n7uosBkxJZV7aPkp1CKtb0lBQSlVbRH1/Xr49ngUP9aFFeBBP/mcLt76xmo0/HbO6NHWFNBSUUjXm\nuqhQPh3fi6l3dOJQXgG3vvEtT8zbTPbJAqtLU1WkoaCUqlEiwq1dolg+KZkHklqzYPMBBkz5hlkr\ndlFYrBv7uDoNBaWUU9T39+GpIR348vEkukeH8+clGQyetoLUHdlWl6YqoaGglHKq6Ih6vDemG7PH\ndMMAY2av575/rmdPzmmrS1MV0FBQStWK/h0iWfZYP54e0oE1u3IZNHUFf12awemzxVaXphxoKCil\nao2fjxe/S2pNyqRkhnVqypupuxjwSirzNx7QWdEuQkNBKVXrIkMCeHVkZz4d34vI4AAe+2QTv5q5\nhq0H8qwuzeNpKCilLNO1ZQM+f6g3f739OrJyTjP89VU8/dkP5J46a3VpHktDQSllKS8v4Y5uLVg+\nKZl7e0czL20f/aek8v7qLIpLdAhrbdNQUEq5hNBAX/44LIalj/YlPiqMyQvTGTp9Jaszc6wuzaNo\nKCilXErbxsF8OLY7b93dlfzCEu565zvGz9nAvqP5VpfmETQUlFIuR0T4ZWwT/jcxiSduaEfKjmyu\nf/Ubpn71I2cKdWMfZ9JQUEq5rABfbx4Z2JblTyQzKLYJ//h6J9e/+g2LtxzSIaxOoqGglHJ5zcIC\neW1UFz4Z15PgAB8e+uh7Rr29lozDJ6wurc7RUFBKuY0e1zZk0SN9eOGWODIOn2ToP1by7OdbOZ5f\naHVpdYaGglLKrfh4e3F3z5akPJHMXT1a8uHavfSfksqctXsp0Y19qs1zQuH4T7Dhfdi3Dgp01qRS\n7q5BPT9euCWOxRP60q5xMM/M38rw11axLuuo1aW5NXG3mzWJiYkmLS3tyj+4aS7Mf6D8dUgURHaA\nRh0gMqb8uV+9mitWKVUrjDEs/uEQLy3ezqG8Am7q1Iynh3agaWig1aW5DBHZYIxJvGw7jwmF0lI4\nvheOZEB2OmRnwJHtcORHKHGYUh/WEiI7nh8WEe3AV/9yKeXq8guLmZm6i5krduMtwsMD2jC2TzQB\nvt5Wl2Y5DYWqKimGY3tsAZHt8MjdCaX2JX3FCxpE28KiLDA6QsO24ONXc7UopWrEvqP5vLg4nWXb\nfqZFeBB/HBbD9R0jERGrS7OMhkJ1lRRB7i5br+JIRnlYHN0Nxj55xssHwlvbehORMeW9i/BrwdvH\n+TUqpSq1amcOkxduIzP7FP3aNeJPw2JoE1nf6rIsoaHgLMVnIWenLSAcexfH9gD2P0tvP1svIrKj\n/V6FvYenuICGAAAR8klEQVTRoBV4aTdWqdpUVFLKh2v2MvV/ttnQY3q1YsL1bQkJ8LW6tFqloVDb\nCvMhZ0f5vYrs7bbneT+Vt/EJgEbt7SHh0LsIbQ5enjMQTCkr5Jw6y5RlO/gkbR8N6/nx5OAOjEiI\nwsvLMy4paSi4irMn4ciO8h7FEXtYnDxY3sa3nkOPwn6/olFHCGkGHnwNVCln2LL/OJMXbOP7n47T\nKSqUyTfF0qVFA6vLcjoNBVd35lgFYbEdTh8pb+MfWsGw2Y5QP1LDQqlqKC01zN90gJe/yCD75FlG\ndI3iycHtiQwOsLo0p9FQcFenc8+/V3FuCO2ZY+VtAsPPHwV1rmdRr6F1dSvlhk6dLeb15Zm8u2o3\n/j7eTBjYhjG9ovHzqXuXczUU6hJj4FT2xcNmj2TAWYcFwepFXjASyh4cgWHW1a6UG8jKOc0Li9JZ\nnpHNtRH1+NPwGJLbR1pdVo1yiVAQkcHAPwBv4B1jzMsVtBkJTMY2dGezMebXlX2nR4bCpRgDJw6e\nf68iO912WarodHm74GYXD5tt1B78PXNonlKXsjzjZ15YtJ2snNNc3zGSZ26MoVVE3VjlwPJQEBFv\n4EfgBmA/sB4YZYxJd2jTFpgHDDDGHBORSGNMdmXfq6FQBaWltlFP542E2g45P0JxQXm70BYXD5tt\n1F5nbyuPdra4hNmr9/Da1zspKjGM7RvNw/3bUM/fveceuUIo/AKYbIz5pf310wDGmL84tPkb8KMx\n5p2qfq+GQjWUltjmU5w3xyLDFhalRfZGYptPce7G9rneRURb8PG3sHilalf2iQJeXprBZ98foHGI\nP08P6cjNnZu57axoVwiFEcBgY8x99td3Az2MMQ87tJmPrTfRG9slpsnGmKWVfa+GghOUFNlmal84\nbDY3s3z2tnhDw9YX39xu2Bq8PWsSkPIsG/YeY/KCbfxwII/Elg2YfFMscdeEWl3WFXOFUPgV8MsL\nQqG7MeYRhzaLgCJgJBAFrATijDHHL/iuccA4gBYtWnTdu3evU2pWFyg+awuGC0dCHc2ibPa2l6+t\nF3HhsNnwaJ29reqM0lLDvzfs429Ld3A0v5A7u7Vg0qB2NKzvPr3nqoaCMy+S7QeaO7yOAg5W0Gat\nMaYIyBKRHUBbbPcfyhhjZgGzwNZTcFrF6nw+/tA41vZwVHTGdsnJMSwOpMG2zxw+G2APi47nLyQY\n1lJnbyu34+Ul3NGtBYPjmjL96528/+0eFm85yMQb2jG6Z0t8vOvO32ln9hR8sF0aGggcwPYP/a+N\nMdsc2gzGdvP5tyISAWwEOhtjci/1vXr5yIWdPWUb+XThPIsTB8rb+AbZbmY7DpuN7Agh1+iEPOU2\ndv58kucWprMqM4d2jeszeXgsvdpEWF1WpSy/fGQvYigwDdv9gveMMS+JyPNAmjFmgdju2LwCDAZK\ngJeMMR9X9p0aCm6oIM9hJNS5YbMZcOrn8jb+Ifaw6Hj+2lD1G2tYKJdkjOHL9J95YVE6+4+dYUhc\nE/4wtCPNw4OsLq1CLhEKzqChUIfkH714JFR2Opxx2E4xIOz8exXnehb1XPu3MuU5CopKeHvFbmak\nZmIMPJDUmgeSWhPo51r31DQUlHsyxrb+k+ON7Wz7fhZnHfbWDoooDwjH3kVg3V/YTLmmg8fP8Ocl\n21m05RDXhAXyfzd2ZEhcE5cZwqqhoOoWY+DkoYuHzR7JgMJT5e2Cm148bLZRewgIsa525VHW7s5l\n8oJtZBw+Sc9rw5l8Uywdmlj/909DQXkGYyBv3/n3KrK32254F58pbxfa/IKw6GALC7+6sYSBci3F\nJaXMXb+PV77cwYkzRdzdsyWP39COsCDrtu/VUFCerbQEju+9YI6FfamPkkJ7I4EGLS8eNhvRDnzr\n7hLKqvYcO13Iq1/9yL++20tooC+TftmeO7u1wNuCjX00FJSqSEkxHMtyuFdh713kZkJpsa2NeNn2\n2b5wJFR4a/Cx7jc95b7SD55g8sJtrMs6SkzTEJ67OZZurcJrtQYNBaWuRHGhLRguHDZ7dDeYUlsb\nLx9o2MYhLM7tvR0N3u69WJpyPmMMi384xEuLt3Mor4CbOjXj6aEdaBpaOwtQaigoVROKCmyXnBxH\nQh3ZbltY8Bxvf9slpwt3yQtrpbO31UXyC4uZmbqLmSt24y3CwwPaMLZPNAG+zh3CqqGglDMVnrbP\n3nYMiwzbTe9zfAIdJuQ5hEVoc52Qp9h3NJ8XF6ezbNvPtAgP4o/DYri+Y6TThrBqKChlhYIT9r23\nHUZCZW+HU4fL2/gF28Pigl3ygptqWHigVTtzmLxwG5nZp+jXrhF/GhZDm8ia3wBLQ0EpV5J/tOKw\nyM8pbxMQevFIqMgYqN/IurpVrSgqKeXDNXuZ+r8fOVNYwpherZhwfVtCAmpuWXoNBaXcwakjDhPx\nHBYSLHBYPT6ooUNYOCz3EVS7o1eU8+WcOsuUZTv4JG0fDev58eTgDoxIiMKrBoawaigo5a6MsS0W\neOGw2ewMKDxZ3q5+4/PvVZzbezvA/TaAUefbsv84kxds4/ufjtMpKpTJN8XSpUX1lnDRUFCqrjEG\n8vaff/npiH32dlF+ebuQaxwuPzlcitLZ226ltNQwf9MB/vJFBkdOnmVE1yieHNyeyOCrm1jpCpvs\nKKVqkgiENbc92t5Qfry01DZ7+8Jhs1kroeRsebuwlhePhIpoB761M05eXRkvL+G2hCgGxTbhteU7\neW9VFkF+3jx/c5xTf672FJSqq0qKbfMpLtz0KGcnlBbZ2oiXbfLdhT2Lhm119raL2X3kFKGBvle9\nBaj2FJTydN4+ENHG9ug4vPx4SRHk7rp4JNSOL8CU2Np4+diW9bhw2Gx4a529bZFrG9X8MNWK6H9d\npTyNt6/9H/sO5x8vPmvrRThufHRoC6QvAOxXFLz9bL2IyA7nL/fRoBV4udamMurqaCgopWx8/KFJ\nnO3hqDAfcnacP2x233rY+qnDZwPsS31csEteaHNd6sPNaCgopSrnFwTNutgejs6etE/IcxgJlbUC\ntjhss+5bzyEkHHoXIc109raL0lBQSl0d/2CISrQ9HJ05fvGw2Z1fwqY5Dp8NdVhA0GGXvPqRGhYW\n01BQStWswDBo0dP2cHQ69+KRUNsXwPf/dPhseAVzLDpCvYa1ew4eTENBKVU76jWEen2gVZ/yY8bA\nqezzwyJ7O/zwbzh7wuGzkeffqzgXHIFhtX8edZyGglLKOiIQ3Nj2uDa5/LgxcOKgw0go+8S8jXOg\n6HR5u+BmFw+bbdTedmlLXRUNBaWU6xGB0Gtsj7bXlx8vLYW8ny5eQHD9O1BcUN4utMXFw2Yj2tlu\nmqtKaSgopdyHl5dtTkSDVtB+cPnx0hLb7G3HORbZGbArpXz2NmL73IXDZiPa2objKkBDQSlVF3h5\nQ8PWtkfHYeXHS4ps+2w7joTKzoAfl5bP3hb7Zy+8ud2wtW2in4fRUFBK1V3evrZ7DI3aQ+wt5ceL\nz0Ju5vkjoX7eCtsXUjZ728vX1ou4MCzCo+v07G0NBaWU5/Hxh8axtoejojOQ8+P5YXEgDbZ9Vt7G\n2x8atbt4JFRYyzoxe1tDQSmlzvENhKadbA9HZ0/Zl/pwCIu9q+GHeQ6fDbL3ShzCIrKjbX8LN5qQ\np6GglFKX418frulqezgqyHMYCWUfNrvra9j8kcNnQ2xhUTYS6tze241dMiw0FJRS6moFhEKLHraH\no/yjF4+E2r4Ivv/A4bNh59+rOPe8XkTtnsMFNBSUUqqmBYVDq962xznGwOkj5Zefzu2St/VTW4+j\n7LMR59+rODeENrB6ezRXlYaCUkrVBhHbgn/1I+HapPLjxsDJQxeHxaaPoPBUebv6TaDXw9DrEaeW\nqaGglFJWErEtJR7SDNoMLD9uDOTtK79XcSQDgps6vRwNBaWUckUiENbC9mg3qNZ+rPsPqlVKKVVj\nNBSUUkqV0VBQSilVRkNBKaVUGaeGgogMFpEdIpIpIk9V0m6EiBgRSbxUG6WUUs7ntFAQEW9gBjAE\niAFGiUhMBe2CgQnAd86qRSmlVNU4s6fQHcg0xuw2xhQCHwM3V9DuBeBvQEEF7ymllKpFzgyFa4B9\nDq/324+VEZEuQHNjzCIn1qGUUqqKnDl5raLl/0zZmyJewFRgzGW/SGQcMM7+8pSI7LjKmiKAnKv8\nrLvSc/YMes6eoTrn3LIqjZwZCvuB5g6vo4CDDq+DgTggVWzLxzYBFojITcaYNMcvMsbMAmZVtyAR\nSTPGeNTNbD1nz6Dn7Blq45ydefloPdBWRKJFxA+4E1hw7k1jTJ4xJsIY08oY0wpYC1wUCEoppWqP\n00LBGFMMPAwsA7YD84wx20TkeRG5yVk/Vyml1NVz6oJ4xpglwJILjv3pEm2TnVmLXbUvQbkhPWfP\noOfsGZx+zmKMuXwrpZRSHkGXuVBKKVWmTobC5ZbXEBF/EfnE/v53ItKq9qusWVU454kiki4iW0Tk\naxGp0vA0V+aJy6hU5ZxFZKT9v/U2EfmoojbupAp/t1uISIqIbLT//R5qRZ01RUTeE5FsEdl6ifdF\nRKbb/zy2iEhCjRZgjKlTD8Ab2AVcC/gBm4GYC9o8CMy0P78T+MTqumvhnPsDQfbn4z3hnO3tgoEV\n2Ea3JVpddy38d24LbAQa2F9HWl13LZzzLGC8/XkMsMfquqt5zv2ABGDrJd4fCnyBbS5YT+C7mvz5\ndbGnUJXlNW4G/ml//h9goNgnS7ipy56zMSbFGJNvf7kW27wRd+aJy6hU5ZzvB2YYY44BGGOya7nG\nmlaVczZAiP15KOfPh3I7xpgVwNFKmtwMfGBs1gJhIlJj+3TWxVC47PIajm2MbehsHtCwVqpzjqqc\ns6Ox2H7TcGeeuIxKVf47twPaichqEVkrIoNrrTrnqMo5TwZGi8h+bKMdnbuzvfWu9P/vV6Qu7tFc\n6fIaV9DGnVT5fERkNJAIJDm1IuersWVU3EhV/jv7YLuElIytN7hSROKMMcedXJuzVOWcRwHvG2Ne\nEZFfAB/az7nU+eVZwqn/ftXFnsLlltc4r42I+GDrclbWXXN1VTlnROR64P+wzRw/W0u1OcuVLKOy\nB9u11wVufrO5qn+3PzfGFBljsoAd2ELCXVXlnMcC8wCMMWuAAGxrBNVVVfr/+9Wqi6FQ6fIadguA\n39qfjwCWG/sdHDd12XO2X0p5C1sguPt1ZvDMZVSq8nd7PrZBBYhIBLbLSbtrtcqaVZVz/gkYCCAi\nHbGFwpFarbJ2LQB+Yx+F1BPIM8Ycqqkvr3OXj4wxxSJybnkNb+A9Y19eA0gzxiwA3sXWxczE1kO4\n07qKq6+K5/x3oD7wb/s99Z+MMW673EgVz7lOqeI5LwMGiUg6UAL8P2NMrnVVV08Vz/kJ4G0ReRzb\nZZQx7vxLnojMxXb5L8J+n+RZwBfAGDMT232ToUAmkA/cU6M/343/7JRSStWwunj5SCml1FXSUFBK\nKVVGQ0EppVQZDQWllFJlNBSUUkqV0VBQyk5ESkRkk8PjkiuvXsV3t7rUqpdKuZI6N09BqWo4Y4zp\nbHURSllJewpKXYaI7BGRv4rIOvujjf14S/veFOf2qGhhP95YRP4rIpvtj172r/IWkbft+xx8KSKB\n9vYTHPa6+Nii01QK0FBQylHgBZeP7nB474QxpjvwOjDNfux1bEsYxwP/Aqbbj08HvjHGdMK2Lv42\n+/G22Ja1jgWOA7fbjz8FdLF/zwPOOjmlqkJnNCtlJyKnjDH1Kzi+BxhgjNktIr7AYWNMQxHJAZoa\nY4rsxw8ZYyJE5AgQ5bjooNh29/vKGNPW/vr3gK8x5kURWQqcwrZu0XxjzCknn6pSl6Q9BaWqxlzi\n+aXaVMRxZdoSyu/p3QjMALoCG+wr9yplCQ0FparmDof/XWN//i3liyneBayyP/8a25aniIi3iJzb\nFewi9n0fmhtjUoAngTBsCxcqZQn9jUSpcoEissnh9VJjzLlhqf4i8h22X6RG2Y9NAN4Tkf+Hbanm\nc6tVPgrMEpGx2HoE44FLLW3sDcwRkVBsm6dMdeMNcVQdoPcUlLoM+z2FRGNMjtW1KOVsevlIKaVU\nGe0pKKWUKqM9BaWUUmU0FJRSSpXRUFBKKVVGQ0EppVQZDQWllFJlNBSUUkqV+f8nbM4pALBz1AAA\nAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VGXa+PHvTUhISKeXEBJ6QqSEIYgNEXABRewSC4vr\nymt/X8uu6I9VQF1Z166sigrWlXV1dQNSbNhRCBY0oQUIEELvpJf798cMMAkJGSCTyST357pyMefM\nc87ch8C553nOU0RVMcYYY46nia8DMMYYU/9ZsjDGGFMjSxbGGGNqZMnCGGNMjSxZGGOMqZElC2OM\nMTWyZGGMMaZGliyMMcbUyJKFMcaYGjX1dQC1pVWrVhoXF+frMIwxxq8sX758l6q2rqlcg0kWcXFx\npKen+zoMY4zxKyKy0ZNy1gxljDGmRpYsjDHG1MiShTHGmBpZsjDGGFMjSxbGGGNqZMnCGGNMjSxZ\nGGOMqVGDGWdhjDGNRtFB2LsR9mY7f4JCwXG9Vz/SkoUxxtQ35WVwYMvRZLA3u2JyyN9VsXzMQEsW\nxhjTIBXsc974922slBSyYd9mKC85WlYCIKoTRHWGXhdAdFzFn5Bor4drycIYY7yhrAT25xybCA7/\nFO6rWD6kBUR3hvZ9IXFsxWQQEQMBvr1dW7IwxpiToQoFe2HvhqqTwf4toGVHyzcJhKhY582/44BK\ntYPOEBxZ55dwIixZGGNMdUqLnE1Ce7MrJYWNzuajogMVy4e2dt78Ow2CPnEVE0J4e2gSULfx1yKv\nJgsRGQk8AwQAr6jq9ErvxwKvA1GuMpNUdb6IBAKvAMmuGN9Q1Ue9GasxphFShbyd1T9IPrAF0KPl\nmwY7nxtEx0HnM47WCqLjnPubhdX5JdQVryULEQkAZgAjgBxgmYikqWqmW7HJwLuq+oKIJALzgTjg\nCqCZqp4mIs2BTBF5R1WzvRWvMaaBKik4mgCqephckl+xfHh7580//uyjieHwT1hbaNI4h6d5s2aR\nAmSp6noAEZkDjAXck4UCEa7XkUCu2/5QEWkKhADFQKX6njHGAOXlcGhbxRqB+8+hbRXLB4Yevfl3\nObdiMoiKhcCQuovdj3gzWXQENrtt5wCDKpWZAnwsIrcDocBw1/73cCaWrUBz4E5V3VP5A0RkIjAR\nIDY2tjZjN8bUJ0WHqq4VHG42KityKywQ0dF58+82/NhupqGtQKSOL8D/eTNZVPXb0ErbqcBrqvqE\niAwG3hSRJJy1kjKgAxANfC0inx6upRw5mepMYCaAw+GofG5jjL8oL4MDudV3M608CK1ZhPNZQeue\n0ON3bskgHiJjoGmzuo2/EfBmssgBOrltx3C0memwG4CRAKq6RESCgVbA1cBCVS0BdojIt4ADWI8x\nxj8V7q+6VrA3G/ZtOnYQWmSMMwFUNwjNagd1ypvJYhnQXUTigS3AOJxJwN0mYBjwmogkAMHATtf+\n80TkLZzNUKcDT3sxVmPMqao8CK1ys1HB3orlQ6KdN/72fSDxoqM9iqLjnIkiILBu4zfH5bVkoaql\nInIbsAhnt9hZqpohItOAdFVNA+4GXhaRO3E2UU1QVRWRGcBs4DeczVmzVXWFt2I1xnjgyCC07GoG\noeVUPwitQ/LRLqaHk0JIVJ1fgjl5otowmvodDoemp6f7Ogxj/FtpMezfXM2o5OMMQqvcxTQ6DiI6\n+PUgtMZCRJarqqOmcjaC25jGRBXydlVfO6g8CC2g2dEaQezgSt1MG/YgNFORJQtjGpqSAucD4+oe\nJpfkVSwf1s55848769jaQSMehGYqsmRhjL8pL4dD26t/kHxwa8Xygc3dBqENsUFo5qRYsjCmPqow\nCK1SMti3EUoL3Qq7DULrOuzY2UxDW1s3U3PKLFkY4wvlZc4aQHXPDvJ2ViwfFA4t4qB1D+hxvuuB\ncryrdtDJBqEZr7NkYYy3FO6vfr6i4w1C6zmqUu0g3gahmeMqKSsnMMC7z5YsWRhzsspK4UB1K6Ft\nhIJK05mFRDtrBO1Og4QxFROCDUIzJ2HLvgL+On8lKMy4Jtmrn2XJwpjqVDUIzf1h8r7NlQahNXUb\nhNb/2G6mNgjN1JLCkjJe/HIdL365DlW4aUhXVBXxYu3TkoVp3CoMQqvcZLQRivZXLN+8lWtZTAck\nXW6D0EydUlXm/7qNv85fyZZ9BVxwWnvuG92LmOjmXv9sSxamYVOF/N1uCWADFXoYHdgCWn60fIVB\naKdXrBlEd4Zm4T64CGNg5dYDTJ2bwffr95DQPoInruzL6V1a1tnnW7Iw/q+ksJpBaNnHH4TW+cyK\ny2JGxznfs0Foph7Zk1fMk5+s5p8/bCIyJJCHL04iNSWWgCZ12+HBkoWp/1QrDkKr3FR0sNLM9+6D\n0OLPOXYQWpD3q+zGnKrSsnLe+n4jT326lkNFpYwfHMf/De9OVPMgn8RjycLUD8V5VQ8+O5wQSgvc\nCh8ehNYZug6tYiU0G4Rm/Nu3WbuYOjeDNdsPcWa3ljxwYW96tvNtE6glC1M3ysudNYDK8xQdGYS2\no2L5oHDnjb9lN7elMW0QmmnYNu/J5+GPMlmUsZ1OLUJ48doB/K53W6/2cvKUJQtTewoPVL9O8r5N\nUFZ8tKw0cRuENtIGoZlGLb+4lH8sXsfMr9cTIMKffteTG86KJziw/vSu82qyEJGRwDM4Fz96RVWn\nV3o/FngdiHKVmaSq813v9QFeAiKAcmCgqrpPiGPqWlmps/dQdQ+SKw9CC45y3vzbJkGvCyvOVxTZ\nyQahmUZPVUn7JZdH569i24FCLu7XgUmjEmgXGezr0I7htWQhIgHADGAEzvW4l4lImqpmuhWbDLyr\nqi+ISCIwH4gTkabAW8B1qvqLiLQESjDeV9NKaOWlR8tWGIR2caVFcDo7awfGmCr9mrOfKXMzWL5x\nL6d1jGTGNf0Z0LmFr8OqljdrFilAlqquBxCROcBYwD1ZKM6aA0AkcLhby/nAClX9BUBVd3sxzsbl\nyCC07KofJhcebxDaZZUGoXW0QWjGnKBdh4r4+8LVvLt8My1Dg/jbZadxxYBONKnjrrAnypvJoiOw\n2W07BxhUqcwU4GMRuR0IBYa79vcAVEQWAa2BOar6mBdjbTiOGYSWXfGh8oGcqgehRXWGmJRjp7e2\nQWjG1Iri0nLeWJLNM5+upaCkjBvOjOeO4d2JCPaP5lhvJouq0mTlBb9TgddU9QkRGQy8KSJJrrjO\nAgYC+cBnrnViP6vwASITgYkAsbGxtR1//XV4EFp1D5OLD1UsH9bWNQht8LHdTG0QmjFe98XqHUyb\nl8n6nXkM6dGav1yYSLc2/rUkrTeTRQ7QyW07hqPNTIfdAIwEUNUlIhIMtHId+6Wq7gIQkflAMlAh\nWajqTGAmgMPhqJyI/JcqHNpR/bODyoPQmoYcvfnHnW2D0IypJzbsyuPheZl8tmoH8a1CmTXBwdCe\nbepFV9gT5c1ksQzoLiLxwBZgHHB1pTKbgGHAayKSAAQDO4FFwJ9FpDlQDAwBnvJirHWvOL/6mkGV\ng9A6uFZCqzQILaozhLWxbqbG1CMHC0t4/vMsZn27gWZNA7hvVC+uPzOeoKb+W4v3WrJQ1VIRuQ3n\njT8AmKWqGSIyDUhX1TTgbuBlEbkTZxPVBFVVYK+IPIkz4SgwX1U/8lasXlFeXv1KaPs2OqevcBcU\n5hxfUGEQmusnshME1r+udMaYisrLlfd/zOGxRavZebCIywfE8OeRPWkT7v//f8V5b/Z/DodD09PT\n6/ZDiw4eWyNwTwjVDUKr0MXUNSq5eQurHRjjx37atJcpczP5ZfM++nWKYspFvenXqf6vYeJ6Huyo\nqZyN4D4e90FoVTUZ5Vfq0XtkEFoi9Bp9bO3ABqEZ0+DsOFDI9IWr+M+PW2gd3ownrujLJf071vuu\nsCfKkkVZCWzPqGYQ2uZjB6FFdnLe/BMuOrabqQ1CM6bRKCotY9Y32Tz/+VpKypSbz+3KrUO7Edas\nYd5WG+ZVnYj8PTBzyNHt5i1dg9CSIenSigkhvAME2F+ZMY2ZqvLZyh08/FEm2bvzGZ7QlskXJBDX\nKtTXoXmV3fnC2sBVb9sgNGNMjbJ2HGTq3Ey+XruLbm3CeOMPKZzTo7Wvw6oTlixEIOFCX0dhjKnH\n9heU8Myna3ljSTYhQQH85cJExg/uTGCA/3aFPVGWLIwxphpl5cq76Zt5fNFq9uQXM25gJ+45vyct\nwxrfeiqWLIwxpgrLsvcwJS2DjNwDDIyL5vUxKSR1jPR1WD5jycIYY9zk7itg+oJVpP2SS/vIYJ5N\n7c+YPu39coqO2mTJwhhjgMKSMmZ+tZ4XvlhHuSp3nNeNm87tSvMgu02CJQtjTCOnqiz8bRuPzF9J\nzt4CRiW14/7RCXRqYRNwurNkYYxptFZtO8DUtEyWrN9Nz7bh/PPGQZzRtZWvw6qXLFkYYxqdffnF\nPPnJGt76fiMRIYE8NLY3qSmxNG1EXWFPlCULY0yjUVpWzjtLN/HEJ2s4UFDCtad35s7hPYgODfJ1\naPWeJQtjTKPw3bpdTJubyaptBxncpSUPXpRIr3YRvg7Lb1iyMMY0aJv35PPX+StZ8Ns2OkaF8MI1\nyYxMatfou8KeKK8mCxEZCTyDc/GjV1R1eqX3Y4HXgShXmUmqOr/S+5nAFFV93JuxGmMalvziUl78\nYh0vfbWeJiLcPaIHN57TheDAAF+H5pe8lixEJACYAYzAuab2MhFJU9VMt2KTgXdV9QURSQTmA3Fu\n7z8FLPBWjMaYhkdVmbtiK4/OX8nW/YVc1LcDk0b1okNUiK9D82verFmkAFmquh5AROYAY3HWFA5T\n4HCjYSSQe/gNEbkYWA/keTFGY0wD8tuW/Uybm8nS7D307hDBs6n9GRjXwtdhNQjeTBYdgc1u2znA\noEplpgAfi8jtQCgwHEBEQoF7cdZK7vFijMaYBmD3oSIe/3gNc5ZtIrp5EI9eehpXOjoR0MBWq/Ml\nbyaLqn5LlRf8TgVeU9UnRGQw8KaIJAFTgadU9dDxHkKJyERgIkBsbGztRG2M8RslZeW8sWQjT3+6\nhoLiMq4/I57/Hd6dyBBbwri2eTNZ5ACd3LZjcGtmcrkBGAmgqktEJBhohbMGcrmIPIbz4Xe5iBSq\n6vPuB6vqTGAmgMPhqJyIjDEN2FdrdjJtXiZZOw5xTo/WPHBhAt3a2OJl3uLNZLEM6C4i8cAWYBxw\ndaUym4BhwGsikgAEAztV9ezDBURkCnCocqIwxjROG3fn8dC8lXy6cjudWzbnlfEOhiW0sa6wXua1\nZKGqpSJyG7AIZ7fYWaqaISLTgHRVTQPuBl4WkTtxNlFNUFWrIRhjjnGoqJQZi7N49esNBAYI947s\nxR/OiqNZU+sKWxekodybHQ6Hpqen+zoMY0wtKy9XPvx5C9MXrGLHwSIuTe7IvSN70TYi2NehNQgi\nslxVHTWVsxHcxph66+fN+5g6N4OfNu2jb6coXrpuAP1jo30dVqNkycIYU+/sOFjIYwtX897yHFqH\nN+PxK/pyaf+ONLGusD5jycIYU28Ul5Yz+9sNPPd5FkWlZfzPkC7cNrQb4cHWFdbXLFkYY+qFz1dt\n56F5K9mwK49hvdow+cJE4luF+jos42LJwhjjU+t2HuKheZl8sXonXVqH8tr1Azm3Zxtfh2UqsWRh\njPGJA4UlPPvpWl77LpuQwAAmX5DA+MFxBDW11erqI0sWxpg6VV6u/Hv5Zv6+aDW784q5ytGJe37X\nk1ZhzXwdmjkOSxbGmDqzfOMepqRl8uuW/QzoHM3sCSmcFhPp67CMByxZGGO8btv+QqYvWMmHP+fS\nLiKYZ8b146K+HWyKDj9iycIY4zWFJWW88vV6ZixeR5kqtw3txs3ndiW0md16/I39xowxtU5VWZSx\nnUfmZ7J5TwEje7fj/tEJxLZs7uvQzEmyZGGMqVVrth9k6twMvs3aTY+2Ybz9x0Gc2a2Vr8Myp8iS\nhTGmVuzPL+GpT9fw5vcbCWvWlKkX9eaaQbE0DbCusA2BJQtjzCkpK1f+uXQTT368mv0FJVw9KJa7\nRvSkRWiQr0MztciShTHmpH2/fjdT0jJYte0gg+Jb8OCY3iR2iPB1WMYLvFo/FJGRIrJaRLJEZFIV\n78eKyGIR+UlEVojIaNf+ESKyXER+df15njfjNMacmJy9+dz69o+Mm/k9BwtLmXF1MnMmnm6JogGr\nsWbhWu3ubVXdeyInFpEAYAYwAud63MtEJE1VM92KTQbeVdUXRCQRmA/EAbuAMaqaKyJJOFfb63gi\nn2+MqX0FxWW8+OU6XvxyHSJw5/AeTDynCyFBtlpdQ+dJM1Q7nDf6H4FZwCIPlz5NAbJUdT2AiMwB\nxgLuyUKBw19FIoFcAFX9ya1MBhAsIs1UtciDzzXG1DJV5aNft/Lo/FVs2VfAhX3ac9/oBDpGhfg6\nNFNHakwWqjpZRP4CnA9cDzwvIu8Cr6rquuMc2hHY7LadAwyqVGYK8LGI3A6EAsOrOM9lwE+WKIzx\njYzc/Uydm8nSDXtIaB/Bk1f2ZVCXlr4Oy9Qxjx5wq6qKyDZgG1AKRAPvicgnqvrnag6rahx/5RpJ\nKvCaqj4hIoOBN0UkSVXLAUSkN/A3nInq2A8QmQhMBIiNjfXkUowxHtqTV8zjH69mztJNRIYE8sgl\nSYwbGEuArVbXKHnyzOIO4Pc4nyO8AvxJVUtEpAmwFqguWeQAndy2Y3A1M7m5ARgJoKpLRCQYaAXs\nEJEY4ANgfHU1GFWdCcwEcDgcnjSNGWNqUFJWzlvfb+SpT9aQV1zG+MFx3Dm8B5HNbbW6xsyTmkUr\n4FJV3ei+U1XLReTC4xy3DOguIvHAFmAccHWlMpuAYcBrIpIABAM7RSQK+Ai4T1W/9exSjDGn6pu1\nu5g6N4O1Ow5xVrdWPDAmkR5tw30dlqkHPEkW84E9hzdEJBxIVNUfVHVldQepaqmrJ9UiIACYpaoZ\nIjINSFfVNOBu4GURuRNnE9UEV5PXbUA34C+u5yUA56vqjpO5SGPM8W3anc/DH2XyceZ2Yls0Z+Z1\nAxiR2NZmhTVHSE0dm0TkJyD5cA8oV/NTuqom10F8HnM4HJqenu7rMIzxK3lFpcxYnMUrX2+gaYBw\n69Bu3HBWPMGB1hW2sRCR5arqqKmcJzULce8q62p+spHfxvgxVeXDn7cwfcEqth8o4pL+Hbl3ZC/a\nRQb7OjRTT3ly01/vesj9gmv7FmC990IyxnjTipx9TEnL4MdN++gTE8k/rhnAgM7Rvg7L1HOeJIub\ngGdxjrZW4DNc3VWNMf5j58Ei/r5oFf9enkPL0CAeu6wPlw+IoYl1hTUe8GRQ3g6cPZmMMX6ouLSc\n17/L5tnP1lJYWsaNZ3fhtvO6ERFsXWGN5zwZZxGMczxEb5xdWwFQ1T94MS5jTC1YvHoHD83LZP3O\nPIb2bM1fLkykS+swX4dl/JAnzVBvAquA3wHTgGuAarvMGmN8b/3OQzw0L5PFq3fSpVUosycMZGiv\nNr4Oy/gxT5JFN1W9QkTGqurrIvJPnGMnjDH1zMHCEp77PIvZ326gWdMA7h/diwlnxBPU1FarM6fG\nk2RR4vpzn2u68G04pxE3xtQT5eXKez/m8NjC1ew6VMQVA2L408ietAm3rrCmdniSLGaKSDTO3lBp\nQBjwl+MfYoypKz9u2svUtAx+ydlP/9goXv29g76donwdlmlgjpssXKO1D7gWPvoK6FInURljarT9\nQCF/W7CK//y0hbYRzXjqqr6M7dvRusIarzhusnCN1r4NeLeO4jHG1KCwpIxXv9nAjMVZlJYpt5zb\nlVuHdiO0mU2sYLzHk39dn4jIPcC/gLzDO1V1T/WHGGNqm6rySeZ2Hv5oJZv25DMisS2TL0igc8tQ\nX4dmGgFPksXh8RS3uu1TrEnKmDqzdvtBps3L5Ou1u+jWJow3b0jh7O6tfR2WaUQ8GcEdXxeBGGOO\ntb+ghKc/XcMbSzbSPCiABy5M5LrBnQkMsK6wpm55MoJ7fFX7VfWN2g/HGANQVq78a9lmHv94NXvz\ni0lNieXuET1oGdbM16GZRsqTZqiBbq+Dca5s9yNgycIYL1i6YQ9T0jLI3HqAlLgWPHhRIr07RPo6\nLNPIedIMdbv7tohE4pwCpEYiMhJ4BudKea+o6vRK78cCrwNRrjKTVHW+6737cM5JVQbcoao2atw0\naLn7Cvjr/JXMW7GVDpHBPJfanwv7tLfV6ky9cDJ97fKB7jUVEpEAYAYwAsgBlolImqpmuhWbDLyr\nqi+ISCLOJVzjXK/H4Zy8sAPwqYj0UNWyk4jXmHqtsKSMl75czwtfZqEKdwzrzs1DuhISZKvVmfrD\nk2cWc3H2fgJoAiTi2biLFCBLVde7zjMHGAu4JwsFIlyvI4Fc1+uxwBxVLQI2iEiW63xLPPhcY/yC\nqrLgt2088tFKtuwr4ILT2nPf6F7ERDf3dWjGHMOTmsXjbq9LgY2qmuPBcR2BzW7bOcCgSmWmAB+L\nyO1AKDDc7djvKx3bsfIHiMhEXAsxxcbGehCSMfXDyq0HmDo3g+/X76FXu3DeufF0Bndt6euwjKmW\nJ8liE7BVVQsBRCREROJUNbuG46pqaNVK26nAa6r6hIgMBt50TVboybGo6kxgJoDD4TjmfWPqm715\nxTz5yRre/mEjESGBPHRxEqkDO9HUusKaes6TZPFv4Ay37TLXvoFVFz8iB+jkth3D0Wamw24ARgKo\n6hLXQkutPDzWGL9RWlbO2z9s4slP1nCoqJTrTu/MnSN6ENU8yNehGeMRT5JFU1UtPryhqsUi4sm/\n8GVAdxGJB7bgfGB9daUym3B2xX1NRBJwds3diXN223+KyJM4H3B3B5Z68JnG1DvfZe1i6txMVm8/\nyBldW/LgmN70bBfu67CMOSGeJIudInKRqqYBiMhYYFdNB6lqqWsSwkU4u8XOUtUMEZkGpLvOdzfw\nsojcibOZaYKqKpAhIu/ifBheCtxqPaGMv9m8J59HPlrJwoxtxESH8OK1A/hd77bWFdb4JXHem49T\nQKQr8DbOb/jgbCIar6pZXo7thDgcDk1PT/d1GMaQX1zKC1+s46Wv1hMgwq1Du/LHs7sQHGhdYU39\nIyLLVdVRUzlPBuWtA04XkTCcyeVgbQRoTEOjqqT9ksv0BavYur+Qsf06MGlUL9pHhvg6NGNOmSfj\nLP4KPKaq+1zb0cDdqjrZ28EZ4y9+27KfKWkZpG/cS1LHCJ5L7Y8jroWvwzKm1njyzGKUqt5/eENV\n94rIaJyjr41p1HYdKuLxRav5V/pmWjQP4m+XncblAzoRYKvVmQbGk2QRICLNXKOpEZEQwKa+NI1a\nSVk5r3+XzTOfraWguIwbzoznjuHdiQgO9HVoxniFJ8niLeAzEZnt2r4e5+R/xjRKX67ZybS5Gazb\nmceQHq35y4WJdGsT5uuwjPEqTx5wPyYiK3BOxSHAQqCztwMzpr7J3pXHwx9l8unKHcS1bM6rv3dw\nXq821hXWNAqezjq7DSgHrgQ2AO97LSJj6plDRaU89/laZn2zgaCAJkwa1Yvrz4yjWVPrCmsaj2qT\nhYj0wDnqOhXYDfwLZ9fZoXUUmzE+VV6u/OenLfxt4Sp2Hizi8gEx/HlkT9qEB/s6NGPq3PFqFquA\nr4ExhwfguUZaG9Pg/bRpL1PmZvLL5n306xTFy+Md9OsU5euwjPGZ4yWLy3DWLBaLyEJgDlXPBmtM\ng7HjQCF/W7ia93/MoXV4M564oi+X9O9IE+sKaxq5apOFqn4AfCAiocDFwJ1AWxF5AfhAVT+uoxiN\n8bqi0jJmf5vNc5+tpaRMuWlIV247rxthzU5mMUljGh5PekPl4Zwb6m0RaQFcAUwCLFkYv6eqfLZy\nBw9/lEn27nyGJ7Rl8gUJxLUK9XVoxtQrJ/S1SVX3AC+5fozxa1k7DjFtXiZfrdlJ19ahvP6HFIb0\naO3rsIypl6yObRqd/QUlPPvZWl7/LpuQoAD+cmEi4wd3JtBWqzOmWpYsTKNRVq78O30zf1+0mj35\nxYwb2Im7z+9JqzCbvcaYmng1WYjISOAZnIsfvaKq0yu9/xRweNxGc6CNqka53nsMuABoAnwC/K/W\ntPiGMdVIz97DlLkZ/LblAI7O0bx+UQpJHSN9HZYxfsNryUJEAoAZwAicCyYtE5E0Vc08XEZV73Qr\nfzvQ3/X6DOBMoI/r7W+AIcAX3orXNExb9xfw6PxVpP2SS7uIYJ4Z14+L+nawKTqMOUHerFmkAFmq\nuh5AROYAY3EulVqVVOBB12vFuR53EM6xHYHAdi/GahqYwpIyXv5qPf/4Yh1lqtxxXjduOrcrzYOs\n5dWYk+HN/zkdgc1u2znAoKoKikhnIB74HEBVl4jIYmArzmTxvKqu9GKspoFQVRZlbOPhj1aSs7eA\nUUntuH90Ap1aNPd1aMb4NW8mi6rq+dU9cxgHvKeqZQAi0g1IAGJc738iIueo6lcVPkBkIjARIDY2\ntlaCNv5r1bYDTJubyXfrdtOzbTj//OMgzujWytdhGdMgeDNZ5ACd3LZjgNxqyo4DbnXbvgT4XlUP\nAYjIAuB0oEKyUNWZwEwAh8NhD78bqX35xTz5yRre+n4j4cGBTBvbm6tTYmlqXWGNqTXeTBbLgO4i\nEg9swZkQrq5cSER6AtHAErfdm4AbReRRnDWUIcDTXozV+KHSsnLeWbqJJz5Zw4GCEq4Z1Jm7RvQg\nOjTI16EZ0+B4LVmoaqmI3AYswtl1dpaqZojINCBdVdNcRVOBOZW6xb4HnAf8irPpaqGqzvVWrMb/\nLFm3m6lzM1i17SCnd2nBg2N6k9A+wtdhGdNgSUMZuuBwODQ9Pd3XYRgvy9mbz1/nr2T+r9voGBXC\n5AsSGJnUzrrCGnOSRGS5qjpqKmf9CI1fKCgu44Uvsnjpq/WIwF0jejDxnC4EB9pqdcbUBUsWpl5T\nVeau2Mqj81eydX8hY/p24L5RvegQFeLr0IxpVCxZmHrrty37mTY3k6XZe0hsH8Ez4/qTEt/C12EZ\n0yhZsjD1zu5DRTz+8RrmLNtEdPMg/nrJaVw1sBMBtlqdMT5jycLUGyVl5by5ZCNPf7qG/OIyrj8j\nnv8d1p0f/HahAAAXOklEQVTI5oG+Ds2YRs+ShakXvl67k2lzM1m74xBnd2/Fg2MS6dYm3NdhGWNc\nLFkYn9q4O4+H5q3k05Xb6dyyOS+PdzA8oY11hTWmnrFkYXwir6iU5xdn8erXG2gaIPx5ZE9uOCue\nZk2tK6wx9ZElC1OnysuVD3/ewvQFq9hxsIhL+3fk3lG9aBsR7OvQjDHHYcnC1JlfNu9jytwMftq0\nj74xkbx43QCSY6N9HZYxxgOWLIzX7ThYyN8Xrubfy3NoFdaMv1/eh8uSY2hiXWGN8RuWLIzXFJeW\nM/vbDTz3eRZFpWX8zzlduO28boQHW1dYY/yNJQvjFZ+v2s5D81ayYVce5/Vqw+QLEujSOszXYRlj\nTpIlC1Or1u08xEPzMvli9U66tApl9vUDGdqzja/DMsacIksWplYcKCzhuc/WMvvbbEICA/h/oxP4\n/RlxBDW11eqMaQi8mixEZCTwDM7Fj15R1emV3n8KGOrabA60UdUo13uxwCs4l2ZVYLSqZnszXnPi\nysuV95bn8NiiVezOK+bKAZ2453c9aR3ezNehGWNqkdeShYgEADOAETjX414mImmqmnm4jKre6Vb+\ndqC/2yneAB5R1U9EJAwo91as5uQs37iHKWmZ/LplPwM6RzN7QgqnxUT6OixjjBd4s2aRAmSp6noA\nEZkDjAUyqymfCjzoKpsINFXVTwBU9ZAX4zQnaNv+QqYvWMmHP+fSNqIZT1/Vj7H9OtgUHcY0YN5M\nFh2BzW7bOcCgqgqKSGcgHvjctasHsE9E/uPa/ykwSVXLvBeuqUlhSRmvfrOBGYuzKC1Xbh3alVvO\n7UZoM3v0ZUxD583/5VV9zaxuwe9xwHtuyaApcDbOZqlNwL+ACcCrFT5AZCIwESA2NvbUIzZVUlU+\nztzOIx+tZNOefM5PbMvkCxKJbdnc16EZY+qIN5NFDs6H04fFALnVlB0H3Frp2J/cmrA+BE6nUrJQ\n1ZnATACHw1FdIjKnYO32g0ydm8k3Wbvo0TaMt24YxFndW/k6LGNMHfNmslgGdBeReGALzoRwdeVC\nItITiAaWVDo2WkRaq+pO4Dwg3Yuxmkr255fw1KdrePP7jYQGBTBlTCLXnt6ZpgHWFdaYxshryUJV\nS0XkNmARzq6zs1Q1Q0SmAemqmuYqmgrMUVV1O7ZMRO4BPhPnU9PlwMveitUcVVauvLN0E098vJr9\nBSWkpsRy9/k9aREa5OvQjDE+JG73aL/mcDg0Pd0qH6fih/W7mTI3k5VbD5AS34IpY3qT2CHC12EZ\nY7xIRJarqqOmctaNxbBlXwF/nb+Sj1ZspUNkMM9f3Z8LTmtvXWGNMUdYsmjECorLeOmrdbz45TpU\n4f+Gd+d/zulKSJCtVmeMqciSRSOkqsz/dRt/nb+SLfsKuKBPe+4fnUDHqBBfh2aMqacsWTQymbkH\nmDo3gx827CGhfQRPXNmX07u09HVYxph6zpJFI7Enr5gnPl7NO0s3ERkSyMMXJ5GaEkuArVZnjPGA\nJYsGrrSsnLe+38iTn6whr7iM8YPjuHN4DyKb22p1xhjPWbJowL7N2sXUuRms2X6Is7q14oExifRo\nG+7rsIwxfsiSRQO0eU8+D3+UyaKM7XRqEcJL1w3g/MS21hXWGHPSLFk0IHlFpfzjiyxe/noDTZsI\nf/pdT244K57gQOsKa4w5NZYsGgBV5b8/5/LogpVsP1DEJf07cu/IXrSLDPZ1aMaYBsKShZ/7NWc/\nU+ZmsHzjXk7rGMk/rklmQOcWvg7LGNPAWLLwU7sOFfH3hat5d/lmWoYG8dhlfbh8QAxNrCusaQBK\nSkrIycmhsLDQ16E0GMHBwcTExBAYeHI9IS1Z+Jni0nLeWJLNM5+upaCkjD+eFc/tw7oTEWxdYU3D\nkZOTQ3h4OHFxcdYxoxaoKrt37yYnJ4f4+PiTOoclCz+yePUOHpqXyfqdeZzbszV/uTCRrq3DfB2W\nMbWusLDQEkUtEhFatmzJzp07T/ocliz8wIZdeTw0L5PPV+0gvlUosycMZGivNr4OyxivskRRu071\n79OSRT12sLCE5z/PYta3G2jWNID7R/diwhnxBDW11eqM8abdu3czbNgwALZt20ZAQACtW7cGYOnS\npQQF1bwY2PXXX8+kSZPo2bNntWVmzJhBVFQU11xzTe0E7kVeTRYiMhJ4BudKea+o6vRK7z8FDHVt\nNgfaqGqU2/sRwErgA1W9zZux1ifl5cr7P+bwt4Wr2XWoiCsGxPCnkT1pE25dYY2pCy1btuTnn38G\nYMqUKYSFhXHPPfdUKKOqqCpNmlT95W327Nk1fs6tt9566sHWEa99RRWRAGAGMApIBFJFJNG9jKre\nqar9VLUf8Bzwn0qneQj40lsx1kc/bdrLJS98x5/eW0GnFiH899Yz+fsVfS1RGFMPZGVlkZSUxE03\n3URycjJbt25l4sSJOBwOevfuzbRp046UPeuss/j5558pLS0lKiqKSZMm0bdvXwYPHsyOHTsAmDx5\nMk8//fSR8pMmTSIlJYWePXvy3XffAZCXl8dll11G3759SU1NxeFwHElkdcmbNYsUIEtV1wOIyBxg\nLJBZTflU4MHDGyIyAGgLLARqXPLP320/UMjfFqziPz9toU14M568si8X9+toXWFNozd1bgaZuQdq\n9ZyJHSJ4cEzvkzo2MzOT2bNn8+KLLwIwffp0WrRoQWlpKUOHDuXyyy8nMbHC92L279/PkCFDmD59\nOnfddRezZs1i0qRJx5xbVVm6dClpaWlMmzaNhQsX8txzz9GuXTvef/99fvnlF5KTk08q7lPlzWTR\nEdjstp0DDKqqoIh0BuKBz13bTYAngOuAYdV9gIhMBCYCxMbG1krQda2otIxXv9nAjM+zKClTbjm3\nK7cM7UZYM3ucZEx91LVrVwYOHHhk+5133uHVV1+ltLSU3NxcMjMzj0kWISEhjBo1CoABAwbw9ddf\nV3nuSy+99EiZ7OxsAL755hvuvfdeAPr27Uvv3ieX5E6VN+9IVX0l1mrKjgPeU9Uy1/YtwHxV3Xy8\nJ/iqOhOYCeBwOKo7d72kqny6cgcPf5TJxt35jEhsy+QLEujcMtTXoRlTr5xsDcBbQkOP/h9du3Yt\nzzzzDEuXLiUqKoprr722yoGE7g/EAwICKC0trfLczZo1O6aMav24tXmzW00O0MltOwbIrabsOOAd\nt+3BwG0ikg08DowXkelVHeiPsnYcZPyspdz4RjqBAU1484YUXh7vsERhjJ85cOAA4eHhREREsHXr\nVhYtWlTrn3HWWWfx7rvvAvDrr7+SmVldS753ebNmsQzoLiLxwBacCeHqyoVEpCcQDSw5vE9Vr3F7\nfwLgUNVjG/j8zP6CEp7+dA1vLNlI86AAHrgwkesGdyYwwLrCGuOPkpOTSUxMJCkpiS5dunDmmWfW\n+mfcfvvtjB8/nj59+pCcnExSUhKRkZG1/jk1EW9WcURkNPA0zq6zs1T1ERGZBqSrapqrzBQguLpk\n4JYsjtt11uFwaHp6em2GX2vKypV/LdvM4x+vZm9+MeMGxnLP+T1oGdbM16EZUy+tXLmShIQEX4dR\nL5SWllJaWkpwcDBr167l/PPPZ+3atTRteuLf9av6exWR5apaYycirz5FVdX5wPxK+x6otD2lhnO8\nBrxWy6HVmWXZe5iSlkFG7gFS4lrwwJhEkjrW/bcCY4x/OnToEMOGDaO0tBRV5aWXXjqpRHGqrMuN\nl+TuK+DRBauY+0suHSKDeS61Pxf2aW9TGBhjTkhUVBTLly/3dRiWLGpbYUkZM79azz++yEIV7hjW\nnZuHdCUkyFarM8b4L0sWtURVWfDbNh75aCVb9hUw+rR23DcqgU4tmvs6NGOMOWWWLGrBqm0HmJqW\nyZL1u+nVLpx/3jiIM7q28nVYxhhTayxZnIK9ecU8+cka3v5hIxEhgTw0tjepKbE0ta6wxpgGxu5q\nJ6G0zLla3dAnvuCfSzdx3emd+eKec7lucJwlCmMagHPPPfeYAXZPP/00t9xyS7XHhIU5FyLLzc3l\n8ssvr/a8NXXxf/rpp8nPzz+yPXr0aPbt2+dp6F5jd7YT9F3WLi549hse+G8Gie0jmH/H2Uwdm0RU\n85rntzfG+IfU1FTmzJlTYd+cOXNITU2t8dgOHTrw3nvvnfRnV04W8+fPJyoq6jhH1A1LFh7avCef\nm95cztWv/EBecSkvXpvM238cRM924b4OzRhTyy6//HLmzZtHUVERANnZ2eTm5tKvXz+GDRtGcnIy\np512Gv/973+POTY7O5ukpCQACgoKGDduHH369OGqq66ioKDgSLmbb775yNTmDz7onHD72WefJTc3\nl6FDhzJ0qHOpn7i4OHbt2gXAk08+SVJSEklJSUemNs/OziYhIYEbb7yR3r17c/7551f4nNpizyxq\nkF9cygtfrOOlr9YTIMLdI3pw4zldCA60rrDG1IkFk2Dbr7V7znanwajqp5tr2bIlKSkpLFy4kLFj\nxzJnzhyuuuoqQkJC+OCDD4iIiGDXrl2cfvrpXHTRRdWOn3rhhRdo3rw5K1asYMWKFRWmF3/kkUdo\n0aIFZWVlDBs2jBUrVnDHHXfw5JNPsnjxYlq1qthJZvny5cyePZsffvgBVWXQoEEMGTKE6Oho1q5d\nyzvvvMPLL7/MlVdeyfvvv8+1115bO39XLlazqIaq8t+ftzDsiS957vMsRvZux+f3DOH2Yd0tURjT\nCLg3RR1uglJV7r//fvr06cPw4cPZsmUL27dvr/YcX3311ZGbdp8+fejTp8+R9959912Sk5Pp378/\nGRkZNU4Q+M0333DJJZcQGhpKWFgYl1566ZGpzuPj4+nXrx9QcXrz2mQ1iyr8tmU/U+dmsCx7L0kd\nI3g2tT8D41r4OixjGqfj1AC86eKLL+auu+7ixx9/pKCggOTkZF577TV27tzJ8uXLCQwMJC4ursop\nyd1VVevYsGEDjz/+OMuWLSM6OpoJEybUeJ7jzeN3eGpzcE5v7o1mKKtZuNl9qIj7/rOCMc9/w/qd\neUy/9DT+e+tZliiMaYTCwsI499xz+cMf/nDkwfb+/ftp06YNgYGBLF68mI0bNx73HOeccw5vv/02\nAL/99hsrVqwAnFObh4aGEhkZyfbt21mwYMGRY8LDwzl48GCV5/rwww/Jz88nLy+PDz74gLPPPru2\nLrdGVrMASsrKef27bJ75bC0FxWX84cx47hjWnciQQF+HZozxodTUVC699NIjzVHXXHMNY8aMweFw\n0K9fP3r16nXc42+++Wauv/56+vTpQ79+/UhJSQGcK97179+f3r17HzO1+cSJExk1ahTt27dn8eLF\nR/YnJyczYcKEI+f44x//SP/+/b3S5FQVr05RXpdOdoryzXvymTB7Ket25nFOj9Y8cGEC3dpYDydj\nfMmmKPeOejtFuT9oGxFM55ah3DcqgWEJbWxWWGOMqYJXn1mIyEgRWS0iWSJyzOJGIvKUiPzs+lkj\nIvtc+/uJyBIRyRCRFSJylbdiDGrahFkTBjI8sa0lCmOMqYbXahYiEgDMAEbgXI97mYikqeqR/mGq\neqdb+duB/q7NfGC8qq4VkQ7AchFZpKq+H/NujDGNkDdrFilAlqquV9ViYA4w9jjlU4F3AFR1jaqu\ndb3OBXYArb0YqzGmnmkoz1Pri1P9+/RmsugIbHbbznHtO4aIdAbigc+reC8FCALWeSFGY0w9FBwc\nzO7duy1h1BJVZffu3QQHB5/0Obz5gLuqBwDV/ebHAe+palmFE4i0B94Efq+q5cd8gMhEYCJAbGzs\nqUVrjKk3YmJiyMnJYefOnb4OpcEIDg4mJibmpI/3ZrLIATq5bccAudWUHQfc6r5DRCKAj4DJqvp9\nVQep6kxgJji7zp5qwMaY+iEwMJD4+Hhfh2HceLMZahnQXUTiRSQIZ0JIq1xIRHoC0cASt31BwAfA\nG6r6by/GaIwxxgNeSxaqWgrcBiwCVgLvqmqGiEwTkYvciqYCc7Ri4+SVwDnABLeutf28Fasxxpjj\na/QjuI0xpjHzdAR3g0kWIrITOP6sXsfXCthVS+H4i8Z2zY3tesGuubE4lWvurKo1Dk1oMMniVIlI\nuifZtSFpbNfc2K4X7Jobi7q4Zpui3BhjTI0sWRhjjKmRJYujZvo6AB9obNfc2K4X7JobC69fsz2z\nMMYYUyOrWRhjjKlRo0oWHqyv0UxE/uV6/wcRiav7KGuXB9d8l4hkutYN+cw1qaNfq+ma3cpdLiIq\nIn7fc8aTaxaRK12/6wwR+Wddx1jbPPi3HSsii0XkJ9e/79G+iLO2iMgsEdkhIr9V876IyLOuv48V\nIpJcqwGoaqP4AQJwzlzbBecstr8AiZXK3AK86Ho9DviXr+Oug2seCjR3vb65MVyzq1w48BXwPeDw\nddx18HvuDvwERLu22/g67jq45pnAza7XiUC2r+M+xWs+B0gGfqvm/dHAApyTuJ4O/FCbn9+Yahae\nrK8xFnjd9fo9YJj49/J5NV6zqi5W1XzX5vc4J3z0Z56uo/IQ8BhQWJfBeYkn13wjMENV9wKo6o46\njrG2eXLNCkS4XkdS/USmfkFVvwL2HKfIWJzz6ak6J1+Ncs3cXSsaU7LwZH2NI2XUObfVfqBlnUTn\nHR6vKeJyA85vJv6sxmsWkf5AJ1WdV5eBeZEnv+ceQA8R+VZEvheRkXUWnXd4cs1TgGtFJAeYD9xe\nN6H5zIn+fz8h3pyivL7xZH2NE1mDwx94fD0ici3gAIZ4NSLvO+41i0gT4ClgQl0FVAc8+T03xdkU\ndS7O2uPXIpKk/rtUsSfXnAq8pqpPiMhg4E3XNR+zNk4D4dX7V2OqWXiyvsaRMiLSFGfV9XjVvvrO\nozVFRGQ48P+Ai1S1qI5i85aarjkcSAK+EJFsnG27aX7+kNvTf9v/VdUSVd0ArMaZPPyVJ9d8A/Au\ngKouAYJxzqHUUJ3IGkInrDElC0/W10gDfu96fTnwubqeHPmpGq/Z1STzEs5E4e/t2FDDNavqflVt\npapxqhqH8znNRarqz1MWe/Jv+0OcnRkQkVY4m6XW12mUtcuTa94EDAMQkQScyaIhL72XBox39Yo6\nHdivqltr6+SNphlKVUtF5PD6GgHALHWtrwGkq2oa8CrOqmoWzhrFON9FfOo8vOa/A2HAv13P8jep\n6kXVnrSe8/CaGxQPr3kRcL6IZAJlwJ9Udbfvoj41Hl7z3cDLInInzuaYCf785U9E3sHZjNjK9Rzm\nQSAQQFVfxPlcZjSQBeQD19fq5/vx350xxpg60piaoYwxxpwkSxbGGGNqZMnCGGNMjSxZGGOMqZEl\nC2OMMTWyZGFMDUSkTER+dvupdibbkzh3XHWziBpTnzSacRbGnIICVe3n6yCM8SWrWRhzkkQkW0T+\nJiJLXT/dXPs7u9YGObxGSKxrf1sR+UBEfnH9nOE6VYCIvOxaZ+JjEQlxlb/Dba2ROT66TGMASxbG\neCKkUjPUVW7vHVDVFOB54GnXvudxThXdB3gbeNa1/1ngS1Xti3NdggzX/u44pw/vDewDLnPtnwT0\nd53nJm9dnDGesBHcxtRARA6palgV+7OB81R1vYgEAttUtaWI7ALaq2qJa/9WVW0lIjuBGPfJGsW5\nGuMnqtrdtX0vEKiqD4vIQuAQznmdPlTVQ16+VGOqZTULY06NVvO6ujJVcZ/pt4yjzxIvAGYAA4Dl\nrpmQjfEJSxbGnJqr3P5c4nr9HUcnobwG+Mb1+jOcS9ciIgEicngVt2O41t3opKqLgT8DUTgnfDTG\nJ+ybijE1CxGRn922F6rq4e6zzUTkB5xfvFJd++4AZonIn3BOiX149s//BWaKyA04axA3A9VNIR0A\nvCUikTgXtXnKjxcqMg2APbMw5iS5nlk4VHWXr2MxxtusGcoYY0yNrGZhjDGmRlazMMYYUyNLFsYY\nY2pkycIYY0yNLFkYY4ypkSULY4wxNbJkYYwxpkb/H/4VJ1MBiwVHAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "%matplotlib inline\n",
+ "\n",
+ "def plot_history(network_history):\n",
+ " plt.figure()\n",
+ " plt.xlabel('Epochs')\n",
+ " plt.ylabel('Loss')\n",
+ " plt.plot(network_history.history['loss'])\n",
+ " plt.plot(network_history.history['val_loss'])\n",
+ " plt.legend(['Training', 'Validation'])\n",
+ "\n",
+ " plt.figure()\n",
+ " plt.xlabel('Epochs')\n",
+ " plt.ylabel('Accuracy')\n",
+ " plt.plot(network_history.history['acc'])\n",
+ " plt.plot(network_history.history['val_acc'])\n",
+ " plt.legend(['Training', 'Validation'], loc='lower right')\n",
+ " plt.show()\n",
+ "\n",
+ "plot_history(network_history)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "After `2` epochs, we get a `~88%` validation accuracy. \n",
+ "\n",
+ "* If you increase the number of epochs, you will get definitely better results."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Quick Exercise: \n",
+ "\n",
+ "Try increasing the number of epochs (if you're hardware allows to)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Your code here\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Introducing the Dropout Layer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The **dropout layers** have the very specific function to *drop out* a random set of activations in that layers by setting them to zero in the forward pass. Simple as that. \n",
+ "\n",
+ "It allows to avoid *overfitting* but has to be used **only** at training time and **not** at test time. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
+ "\n",
+ "keras.layers.core.Dropout(rate, noise_shape=None, seed=None)\n",
+ "```\n",
+ "\n",
+ "Applies Dropout to the input.\n",
+ "\n",
+ "Dropout consists in randomly setting a fraction rate of input units to 0 at each update during training time, which helps prevent overfitting.\n",
+ "\n",
+ "Arguments\n",
+ "\n",
+ "* rate: float between 0 and 1. Fraction of the input units to drop.\n",
+ "* noise_shape: 1D integer tensor representing the shape of the binary dropout mask that will be multiplied with the input. For instance, if your inputs have shape (batch_size, timesteps, features) and you want the dropout mask to be the same for all timesteps, you can use noise_shape=(batch_size, 1, features).\n",
+ "* seed: A Python integer to use as random seed."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Note** Keras guarantess automatically that this layer is **not** used in **Inference** (i.e. Prediction) phase\n",
+ "(thus only used in **training** as it should be!)\n",
+ "\n",
+ "See `keras.backend.in_train_phase` function"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from keras.layers.core import Dropout\n",
+ "\n",
+ "## Pls note **where** the `K.in_train_phase` is actually called!!\n",
+ "Dropout??"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from keras import backend as K\n",
+ "\n",
+ "K.in_train_phase?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Exercise:\n",
+ "\n",
+ "Try modifying the previous example network adding a Dropout layer:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from keras.layers.core import Dropout\n",
+ "\n",
+ "# FC@512+relu -> DropOut(0.2) -> FC@512+relu -> DropOut(0.2) -> FC@nb_classes+softmax\n",
+ "# ... your Code Here"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol22.py"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Train on 45000 samples, validate on 15000 samples\n",
+ "Epoch 1/4\n",
+ "45000/45000 [==============================] - 7s - loss: 1.3793 - acc: 0.6377 - val_loss: 0.6832 - val_acc: 0.8490\n",
+ "Epoch 2/4\n",
+ "45000/45000 [==============================] - 6s - loss: 0.6157 - acc: 0.8335 - val_loss: 0.4507 - val_acc: 0.8828\n",
+ "Epoch 3/4\n",
+ "45000/45000 [==============================] - 7s - loss: 0.4775 - acc: 0.8635 - val_loss: 0.3786 - val_acc: 0.8983\n",
+ "Epoch 4/4\n",
+ "45000/45000 [==============================] - 7s - loss: 0.4165 - acc: 0.8798 - val_loss: 0.3410 - val_acc: 0.9056\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJ3sgGwlhB4OAyBZCjIDiWr0WbSsuCNLa\n1rpw0WutWttS669aW1u7KKi1WrVqe+tlUdTiXqu2LqAsmrCK7BB2AglLEkKS7++PMwkhZCNkcmaS\n9/PxmAeznDn5HAbmnfPdjjnnEBERAYjwuwAREQkdCgUREammUBARkWoKBRERqaZQEBGRagoFERGp\nplAQEZFqCgUREammUBARkWpRfhdwvDp37uwyMjL8LkNEJKwsXrx4t3MuvbHtwi4UMjIyWLRokd9l\niIiEFTPb2JTt1HwkIiLVghYKZvaMme00s2WNbHe6mVWY2fhg1SIiIk0TzDOF54CxDW1gZpHAb4G3\ng1iHiIg0UdD6FJxzH5hZRiObfR+YA5werDpEJHQdPnyY/Px8SktL/S6lzYiLi6NXr15ER0c36/2+\ndTSbWU/gcuArNBIKZjYZmAzQp0+f4BcnIq0iPz+fxMREMjIyMDO/ywl7zjkKCgrIz8+nb9++zdqH\nnx3N04GfOOcqGtvQOfekcy7HOZeTnt7oiCoRCROlpaWkpaUpEFqImZGWlnZCZ15+DknNAWYG/jF0\nBi4xs3Ln3Cs+1iQirUyB0LJO9O/TtzMF51xf51yGcy4DeBG4OZiBsL2olF+8upyy8spg/QgRkbAX\nzCGpM4D5wEAzyzez681siplNCdbPbEju5kKe/XgDj7632o8fLyIhqKCggKysLLKysujWrRs9e/as\nflxWVtakfXzve99j1apVDW7z2GOP8fzzz7dEyUFnzjm/azguOTk5rrkzmu98IY+XPsvnhSlncNpJ\nqS1cmYgcr5UrVzJo0CC/ywDg3nvvJSEhgTvvvPOo551zOOeIiAifub51/b2a2WLnXE5j7w2fo2wB\n93xjMD1S4rl9Vh4HD5X7XY6IhKg1a9YwdOhQpkyZQnZ2Ntu2bWPy5Mnk5OQwZMgQ7rvvvuptzzrr\nLHJzcykvLyclJYWpU6cyfPhwzjjjDHbu3AnA3XffzfTp06u3nzp1KiNHjmTgwIHMmzcPgIMHD3Ll\nlVcyfPhwJk2aRE5ODrm5ua1+7GG39tGJSIyL5qEJWUx8cj6/fG0FD1yZ6XdJIhLwi1eXs2Lrvhbd\n5+AeSdzzjSHNeu+KFSt49tlneeKJJwB44IEHSE1Npby8nPPPP5/x48czePDgo95TVFTEueeeywMP\nPMAdd9zBM888w9SpU4/Zt3OOBQsWMHfuXO677z7eeustHn30Ubp168acOXPIy8sjOzu7WXWfqHZ1\npgAwsm8qU87tx8yFm3lnxQ6/yxGRENWvXz9OP/3IFKoZM2aQnZ1NdnY2K1euZMWKFce8Jz4+nosv\nvhiA0047jQ0bNtS57yuuuOKYbT766COuvvpqAIYPH86QIc0LsxPVrs4Uqtx+4Sn8Z9Uups5ZQlbv\nc0hPjPW7JJF2r7m/0QdLx44dq++vXr2ahx9+mAULFpCSksI111xT51yAmJiY6vuRkZGUl9fdTB0b\nG3vMNqHSv9vuzhQAYqIimH51FvsPlTN1zpKQ+TBEJDTt27ePxMREkpKS2LZtG2+/3fLLtZ111lnM\nnj0bgKVLl9Z5JtIa2mUoAJzSNZGpY0/l3S92MmPBZr/LEZEQlp2dzeDBgxk6dCg33ngjY8aMafGf\n8f3vf58tW7aQmZnJgw8+yNChQ0lOTm7xn9OYdjUktbbKSse3n/mUzzYW8uYPziajc8fG3yQiLSaU\nhqT6rby8nPLycuLi4li9ejUXXXQRq1evJirq+Fv5NSS1mSIijD9cNZzoSOO2WbmUV2i2s4j448CB\nA4wZM4bhw4dz5ZVX8uc//7lZgXCi2mVHc03dk+P51eXDuHXG5zz2/lp+cOEAv0sSkXYoJSWFxYsX\n+11G+z5TqHLp8B6My+rBI++tJm9zod/liIj4RqEQcN+4oXRJjOX2WbkUl2m2s4i0TwqFgOT4aB68\najjrdh/k12+s9LscERFfKBRqOLN/Z244qy9//2QT76/a6Xc5IiKtTqFQy51fHcjAron8+MUl7DnY\ntKVzRSQ8nXfeecdMRJs+fTo333xzve9JSEgAYOvWrYwfP77e/TY2dH769OkUFxdXP77kkksoLPS/\nT1OhUEtcdCTTJmZRVHyYn76k2c4ibdmkSZOYOXPmUc/NnDmTSZMmNfreHj168OKLLzb7Z9cOhTfe\neIOUlJRm76+lKBTqMLhHEj+86BTeXr6DFxfn+12OiATJ+PHjee211zh06BAAGzZsYOvWrWRlZXHB\nBReQnZ3NsGHD+Mc//nHMezds2MDQoUMBKCkp4eqrryYzM5OJEydSUlJSvd1NN91UveT2PffcA8Aj\njzzC1q1bOf/88zn//PMByMjIYPfu3QA89NBDDB06lKFDh1Yvub1hwwYGDRrEjTfeyJAhQ7jooouO\n+jktpd3PU6jPDWefzHtf7OQXr65g9Mlp9E7t4HdJIm3bm1Nh+9KW3We3YXDxA/W+nJaWxsiRI3nr\nrbcYN24cM2fOZOLEicTHx/Pyyy+TlJTE7t27GT16NJdeemm91z9+/PHH6dChA0uWLGHJkiVHLXt9\n//33k5qaSkVFBRdccAFLlizh1ltv5aGHHuL999+nc+fOR+1r8eLFPPvss3z66ac45xg1ahTnnnsu\nnTp1YvXq1cyYMYOnnnqKCRMmMGfOHK655pqW+bsK0JlCPSIjjAcnDMeAO2bnUlGpZiSRtqhmE1JV\n05FzjrvuuovMzEwuvPBCtmzZwo4d9S+1/8EHH1R/OWdmZpKZeeRaLbNnzyY7O5sRI0awfPnyRhe6\n++ijj7j88svp2LEjCQkJXHHFFXz44YcA9O3bl6ysLKDhpblPhM4UGtCrUwd+MW4Id8zO488frOXm\n8/r7XZJI29XAb/TBdNlll3HHHXfw2WefUVJSQnZ2Ns899xy7du1i8eLFREdHk5GRUedS2TXVdRax\nfv16/vCHP7Bw4UI6derEtdde2+h+GurHrFpyG7xlt4PRfKQzhUZcPqInXxvWnWnvfMmyLUV+lyMi\nLSwhIYHzzjuP6667rrqDuaioiC5duhAdHc3777/Pxo0bG9zHOeecw/PPPw/AsmXLWLJkCeAtud2x\nY0eSk5PZsWMHb775ZvV7EhMT2b9/f537euWVVyguLubgwYO8/PLLnH322S11uI1SKDTCzPjVZUPp\n1CGG22flUnq4wu+SRKSFTZo0iby8vOorn33rW99i0aJF5OTk8Pzzz3Pqqac2+P6bbrqJAwcOkJmZ\nye9+9ztGjhwJeFdQGzFiBEOGDOG66647asntyZMnc/HFF1d3NFfJzs7m2muvZeTIkYwaNYobbriB\nESNGtPAR169dL519PP7z5S6++8wCvjcmI+SuECUSrrR0dnBo6exWcO4p6Xz3jJN49uMNfLh6l9/l\niIgEhULhOEy9eBD90jty5wt5FBZrtrOItD0KheMQHxPJ9IkjKDhQxt2vLNNsZ5EWoP9HLetE/z4V\nCsdpWK9kbrtwAK8t2cbcvK1+lyMS1uLi4igoKFAwtBDnHAUFBcTFxTV7H5qn0AxTzu3H+6t2cfcr\ny8jJSKVnSrzfJYmEpV69epGfn8+uXeqnaylxcXH06tWr2e/X6KNm2lRQzMUPf0BmrxSev2EUERF1\nT38XEQkFGn0UZH3SOvDzbwxm/roCnvl4vd/liIi0iKCFgpk9Y2Y7zWxZPa9/y8yWBG7zzGx4sGoJ\nlgk5vblocFd+99Yqvti+z+9yREROWDDPFJ4Dxjbw+nrgXOdcJvBL4Mkg1hIUZsZvrhhGUnwUt83M\n5VC5ZjuLSHgLWig45z4A9jTw+jzn3N7Aw0+A5veM+CgtIZbfjc/ki+37eeifX/pdjojICQmVPoXr\ngTfre9HMJpvZIjNbFIqjFL5yale+OaoPT364jk/WFfhdjohIs/keCmZ2Pl4o/KS+bZxzTzrncpxz\nOenp6a1X3HG4+2uDOCm1Az+cnce+0sN+lyMi0iy+hoKZZQJPA+Occ2H9K3aHmCimTcxi+75S7v3H\ncr/LERFpFt9Cwcz6AC8B33bOtYnG+BF9OnHL+f156fMtvL5km9/liIgct2AOSZ0BzAcGmlm+mV1v\nZlPMbEpgk58DacCfzCzXzPyfkdYCbvlKf4b3Suaul5eyvajhKyyJiIQazWgOgnW7DvC1Rz4iJ6MT\nf/3eSM12FhHfaUazj05OT+BnXxvEh6t387f5G/wuR0SkyRQKQfKtUX04f2A6v3nzC9bsPPY6rCIi\noUihECRmxm/HZ9IxNorbZuVSVl7pd0kiIo1SKARRl8Q4fn35MJZt2cfD77aJAVYi0sYpFIJs7NBu\nXHVaLx7/91oWbah31Q8RkZCgUGgF91w6hJ6d4rljdh4HDpX7XY6ISL0UCq0gITaKaROyyN9bzH2v\naraziIQuhUIryclIZcq5/Zi9KJ+3l2/3uxwRkTopFFrRbReewtCeSfz0paXs3K/ZziISehQKrSgm\nKoJpE7I4eKicn7y4hHCbTS4ibZ9CoZUN6JrI1ItP5f1Vu/i/BZv8LkdE5CgKBR9894wMzh7QmV+9\ntpJ1uw74XY6ISDWFgg8iIozfjx9OTFQEt8/Oo7xCs51FJDQoFHzSLTmO+y8fSt7mQv74/hq/yxER\nARQKvvp6Zg8uH9GTR99bw+eb9vpdjoiIQsFv9146hK6JsdwxO4/iMs12FhF/KRR8lhwfzYMTsthQ\ncJD7X1/pdzki0s4pFELAGf3SuPHsk3n+002898UOv8sRkXZMoRAifnjRKZzaLZEfv7iUggOH/C5H\nRNophUKIiI2KZPrVWewrOcxPX1qq2c4i4guFQgg5tVsSP/rqQP65YgcvLMr3uxwRaYcUCiHm+rP6\nMvrkVH7x6nI2FRT7XY6ItDMKhRATEWE8OCGLiAjjjtm5VFSqGUlEWo9CIQT1TInnl+OGsmjjXp74\nz1q/yxGRdkShEKLGZfXga5ndmfbOlyzbUuR3OSLSTigUQpSZcf9lQ+mcEMtts3IpPVzhd0ki0g4o\nFEJYSocYfn9VJmt2HuCBN7/wuxwRaQcUCiHu7AHpXHtmBs/N28CHq3f5XY6ItHEKhTAw9eJT6d8l\ngTtfyKOwuMzvckSkDQtaKJjZM2a208yW1fO6mdkjZrbGzJaYWXawagl3cdGRTJ+YRcGBMn728jLN\ndhaRoAnmmcJzwNgGXr8YGBC4TQYeD2ItYW9oz2Ru/69TeH3pNl7J3eJ3OSLSRgUtFJxzHwB7Gthk\nHPA35/kESDGz7sGqpy2Ycm4/ck7qxM9fWU7+Xs12FpGW52efQk9gc43H+YHnjmFmk81skZkt2rWr\n/Xa2RkYY0yZmUekcP5ydR6VmO4tIC/MzFKyO5+r8lnPOPemcy3HO5aSnpwe5rNDWO7UD91w6hE/X\n7+Hpj9b5XY6ItDF+hkI+0LvG417AVp9qCStXndaLrw7pyh/e/pKV2/b5XY6ItCF+hsJc4DuBUUij\ngSLn3DYf6wkbZsavLx9GUnw0t2u2s4i0oGAOSZ0BzAcGmlm+mV1vZlPMbEpgkzeAdcAa4Cng5mDV\n0halJcTy+/GZfLF9Pw/+c5Xf5YhIGxEVrB075yY18roD/idYP789OP/ULlwzug9Pf7Se80/twpn9\nOvtdkoiEOc1oDnN3XTKIjLSO3Dk7j6KSw36XIyJhTqEQ5jrERDFtYhY79h/inn/UOXlcRKTJFApt\nQFbvFG79ygBeyd3Kq3kawCUizadQaCP+5/x+ZPVO4e5XlrG9qNTvckQkTCkU2oioyAimTcyirLyS\nO1/QbGcRaR6FQhvSt3NH7v76ID5as5u/zt/gdzkiEoYUCm3MN0f24YJTu/DAm1+wesd+v8sRkTCj\nUGhjzIwHrsykY2wUP5iZS1l5pd8liUgYUSi0QemJsTxwxTBWbNvH9H996Xc5IhJGFApt1EVDujEx\npzdP/GctCzc0dFkLEZEjFApt2P/7xmB6derA7bNy2V+q2c4i0jiFQhuWEBvFtInD2VpYwn2vrvC7\nHBEJA00KBTPrZ2axgfvnmdmtZpYS3NKkJZx2Uio3n9efFxbn89YyrUwuIg1r6pnCHKDCzPoDfwH6\nAv8XtKqkRf3gwgEM65nMT19ays79mu0sIvVraihUOufKgcuB6c6524HuwStLWlJ0ZATTJg6nuKyC\nH7+4BG/VchGRYzU1FA6b2STgu8Brgeeig1OSBEP/Loncdckg/r1qF3//dJPf5YhIiGpqKHwPOAO4\n3zm33sz6An8PXlkSDN854yTOOSWd+19fwbpdB/wuR0RCUJNCwTm3wjl3q3Nuhpl1AhKdcw8EuTZp\nYWbG78dnEhcdye2zcjlcodnOInK0po4++reZJZlZKpAHPGtmDwW3NAmGrklx/PryYeTlF/Hoe2v8\nLkdEQkxTm4+SnXP7gCuAZ51zpwEXBq8sCaZLhnXnihE9eez9NXy2aa/f5YhICGlqKESZWXdgAkc6\nmiWM3TtuCN2S4rhjVi4HD5X7XY6IhIimhsJ9wNvAWufcQjM7GVgdvLIk2JLionlwwnA27inmV6+v\n9LscEQkRTe1ofsE5l+mcuynweJ1z7srglibBNvrkNCaffTIzFmzi3ZU7/C5HREJAUzuae5nZy2a2\n08x2mNkcM+sV7OIk+O646BQGdU/iJ3OWsPvAIb/LERGfNbX56FlgLtAD6Am8GnhOwlxsVCTTJ2ax\nr6Scn760VLOdRdq5poZCunPuWedceeD2HJAexLqkFQ3slsiPxw7knRU7mL1os9/liIiPmhoKu83s\nGjOLDNyuAQqCWZi0ruvG9OXMfmn84tUVbCw46Hc5IuKTpobCdXjDUbcD24DxeEtfSBsREWH84arh\nREYYt8/KpVyznUXapaaOPtrknLvUOZfunOvinLsMbyKbtCE9UuL51WVD+WxTIU/8Z63f5YiID07k\nymt3NLaBmY01s1VmtsbMptbxeh8ze9/MPjezJWZ2yQnUIy1gXFZPvjG8B9P/tZol+YV+lyMirexE\nQsEafNEsEngMuBgYDEwys8G1NrsbmO2cGwFcDfzpBOqRFvKrcUPpnBDL7bNyKSmr8LscEWlFJxIK\njY1dHAmsCUx0KwNmAuPq2EdS4H4ysPUE6pEWktzBm+28dtdBHnhTs51F2pMGQ8HM9pvZvjpu+/Hm\nLDSkJ1BzfGN+4Lma7gWuMbN84A3g+8dXvgTLmP6duW5MX/46fyP/+XKX3+WISCtpMBScc4nOuaQ6\nbonOuahG9l1X81Lts4tJwHPOuV7AJcD/mtkxNZnZZDNbZGaLdu3SF1Rr+fHYgQzoksCPXshj78Ey\nv8sRkVZwIs1HjckHetd43Itjm4euB2YDOOfmA3FA59o7cs496ZzLcc7lpKdrzlxriYuOZPrVWewt\nLuNnr2i2s0h7EMxQWAgMMLO+ZhaD15E8t9Y2m4ALAMxsEF4o6FQghAzpkcwd/zWQN5Zu56XPtvhd\njogEWdBCwTlXDtyCt+T2SrxRRsvN7D4zuzSw2Q+BG80sD5gBXOv062jImXzOyYzMSOWeucvZvKfY\n73JEJIgs3L6Dc3Jy3KJFi/wuo93ZvKeYix/+kME9kphx42giIxockSwiIcbMFjvnchrbLpjNR9KG\n9E7twL2XDmHB+j08/eE6v8sRkSBRKEiTXZndk7FDuvGHf65ixdZ9fpcjIkGgUJAmMzN+fcUwUjrE\ncNuszyk9rNnOIm2NQkGOS2rHGH4/PpMvdxzgD2+v8rscEWlhCgU5bucN7MK3R5/E0x+tZ96a3X6X\nIyItSKEgzXLXJYM4Ob0jP3whj6Liw36XIyItRKEgzRIf413bedf+Q/x87jK/yxGRFqJQkGbL7JXC\nrRcM4B+5W5mbpwVuRdoChYKckJvP68eIPinc/fJSthaW+F2OiJwghYKckKjICKZNyKK80vGjF/Oo\nrAyvGfIicjSFgpywjM4d+X9fH8zHawp4dt4Gv8sRkROgUJAWcfXpvblwUBd++9YXrNq+3+9yRKSZ\nFArSIsyM31yRSWJsFLfNyuVQuWY7i4QjhYK0mPTEWH57ZSYrt+1j2jur/S5HRJpBoSAt6sLBXZk0\nsjd//mAtC9bv8bscETlO7ScUKiuhpNDvKtqFu782mD6pHbh9Vi77SzXbWSSctJ9QWPsuPDQYXv8h\n7FbTRjB1jI3ioQlZbCsq4d65K/wuR0SOQ/sJhZSTYMhl8Nnf4I858PxVsPY9CLMrz4WL007qxC3n\n92fOZ/m8uXSb3+WISBO1n1BIPwUu+xPcvhzO+yls/Rz+93L402hY9CyU6drDLe37Fwwgs1cyd728\nlJ37Sv0uR0SaoP2EQpWELnDeVC8cLnsCImPgtdtg2mD4171QtMXvCtuM6MgIpk3MouRwBT96cQnh\ndj1wkfao/YVClahYyJoE//0BXPsGZJwFHz8M04fBi9fB5oV+V9gm9EtP4GeXDOI/X+7i759s9Lsc\nEWlElN8F+M4MMsZ4t70bYMFTXr/DsjnQMwdG3wSDx0FktN+Vhq1rRp/Ev1bu5P43VnJm/870S0/w\nuyQRqUf7PVOoS6cM+Or9cMcKuPj3ULIX5lwP0zPhwwehWOPum8PM+P34TOKjI7l9Vi6HKyr9LklE\n6qFQqEtsIoyaDLcsgm/O9jqp373PG9L66g9g50q/Kww7XZLi+M0Vw1iSX8Sj72pIsEioUig0JCIC\nTvkqfOcfcNN8yJwAeTO9EUt/uwy+/Kc3KU6aZOzQ7lyZ3Ys/vr+GxRv3+l2OiNTBwm1ESE5Ojlu0\naJF/BRwsgMXPwsKnYf82SOsPo6bA8EkQq7byxuwvPczY6R8SFWm8cevZdIxVt5ZIazCzxc65nMa2\n05nC8eqYBufcCbcthSv/AnHJ8MadXtPS2z+DvRph05DEuGimTcxi055ifvW6ZjuLhBqFQnNFRsOw\n8XDje3D9v6D/BfDJ4/BIFsz6Nmycr9nS9RjZN5X/PqcfMxZs5p0VO/wuR0RqUPNRSyrK95qVFj0L\npYXQfTiMvhmGXO7Ni5BqZeWVjHvsY3buK+Wt284hPVF/PyLB1NTmo6CGgpmNBR4GIoGnnXMP1LHN\nBOBewAF5zrlvNrTPkA6FKmUHYcks+OQJ2L0KErpCzvWQcx0kpPtdXcj4csd+vv7oR0QYnJ6Rypn9\nOnNmvzSG9kwmMsL8Lk+kTfE9FMwsEvgS+C8gH1gITHLOraixzQBgNvAV59xeM+vinNvZ0H7DIhSq\nOOctuvfJ47DmHW9JjWFXeR3T3TP9ri4kfLZpL3NztzJ/bQGrdniX8UyMi2L0yWmc2S+NM/t15pSu\nCZgpJERORFNDIZhDP0YCa5xz6wIFzQTGATV7F28EHnPO7QVoLBDCjpnX19D/Atj1JSz4M+T+H+Q+\nDxlne+Ew8GKIiPS7Ut9k9+lEdp9OAOzaf4j56wqYv3Y3H68pqO5v6JwQwxmBs4gx/TrTOzVeISES\nJME8UxgPjHXO3RB4/G1glHPulhrbvIJ3NjEGr4npXufcWw3tN6zOFOpSstdbRmPBU1C02VvSe9QU\nGHENxCX5XV1I2bynmPnrCpi3Zjfz1hawc/8hAHqmxDOmf1p1c1OXpDifKxUJfaHQfHQV8NVaoTDS\nOff9Gtu8BhwGJgC9gA+Boc65wlr7mgxMBujTp89pGze2gWGfFeXwxWte09LmTyAmEUZ8C0ZOhrR+\nflcXcpxzrN11gHlrC5i3poD56wooKvGu6ta/S0KgqSmN0SenkdIhxudqRUJPKITCGXi/+X818Pin\nAM6539TY5gngE+fcc4HH7wJTnXP1LlEa9mcKddnyGXz6BCx7CSrLvSalUVOg7zleE5Qco6LSsXLb\nPuYFmpoWrN9DyeEKzGBIjyTG9OvMGf3SOD0jVRPkRAiNUIjCaxq6ANiC19H8Tefc8hrbjMXrfP6u\nmXUGPgeynHMF9e23TYZClX3bYNFfYNEzUFwAXYZ4q7QOuwqi1UTSkLLySvLyC5m3poB5a3fz+aZC\nyioqiYowRvRJqe6TGNEnhdio9tuHI+2X76EQKOISYDpef8Ezzrn7zew+YJFzbq55vYUPAmOBCuB+\n59zMhvbZpkOhyuFSWPqCd/awYxl0SPOGs+ZcD0nd/a4uLJSUVbBo455Ac9Nulm4potJBXHSEhr9K\nuxQSoRAM7SIUqjgHGz70+h1WvQkRUd5EuNE3Qc9sv6sLK0Ulh1mwfg8fr9mt4a/SLikU2pqCtd6I\npc//F8oOQO/RXjic+nWIVJv58ao9/HXTHu8a3Rr+Km2VQqGtKi2Cz5/35jzs3QDJvWHkjZD9HYjv\n5Hd1Yaspw1/P6JdGVw1/lTClUGjrKivgy7e8pqUNH0J0B2/57lFTvIsCSbN5w18PMm/tbg1/lTZD\nodCebF/qrbO0dDZUlEH/C72mpX4XaEhrC6g9/HXhhj0Ulx0Z/lrVaa3hrxLKFArt0YFd3gWAFjwF\nB3dC54EwegpkXg0xHfyurs0oK69kSX4hH2v4q4QRhUJ7Vn4Ilr8Mn/wJtuVBXAqcdq3X95Dcy+/q\n2pzGhr+eEei01vBX8ZNCQbwhrZs+8cLhi9cAg8GXetd46HW6mpaCpKHhr6P6plV3XGv4q7QmhYIc\nbe9GWPgULP4bHCqCnqfBqJtg8DiIUmdpMNUc/jpvbQEbCzT8VVqfQkHqdugA5M3wZksXrIHE7nD6\nDXDa97zrT0vQ5e8tZt7aAuavLeDjNbuPGv56Zr80xvTX8FdpeQoFaVhlJaz5l9e0tO59iIqDzAne\n2UPXwX5X125o+Ku0FoWCNN3Old6ZQ95MKC+Fk8/zwmHARRAR4Xd17UrN4a/z1nqrv2r4q7QEhYIc\nv+I9sPg5b0jr/q2Q2s+bDJc1CWIT/a6uXaoa/jov0NRUc/hrVu8Uzuyv4a/SNAoFab6Kw7DiH97Z\nQ/5CiE3yltEYeSN0yvC7unbtqOGvawtYml+o4a/SJAoFaRmbF8Knj3sh4Sph4CXekNaTztSQ1hBQ\nNfy1qk/TFPl6AAAN+klEQVRCw1+lPgoFaVlFW2Dh096M6ZK90C3TW0pj6JUQFet3dRKg4a9SH4WC\nBEdZMSyZ5TUt7foCOnaB06/3LgKU0MXv6qQWDX+VKgoFCS7nvKGsnzwOq/8JkTEwdLy31lL34X5X\nJ3WoGv5adQ2JmsNf+6V3ZEyg01rDX9smhYK0nt2r4dM/Q+7/weGDcNIYr2lp4CUQoRExoaqy0rGi\nnuGvp3RJpHdqB3qkxNE9OZ4eKXH0SImne3IcXZPiiI7UUOVwo1CQ1ldS6F0Z7tMnoWgTpPSBkf8N\n2d+GuGS/q5NG1Bz+mru5kK2FJWwtLGFfaflR20UYdEmMo3tKHD0CgXF0cMST1jGGCI1+CikKBfFP\nRTmset27xsOmeRCTAFnfglH/DWn9/K5OjtOBQ+VsKyxha1Gp92fV/aISthaWsrWwhEPllUe9JyYy\ngu4pcXRP9oKjeyAwat5Piov26YjaJ4WChIatn3vhsGwOVJbDKV/1mpb6nqshrW2Ec469xYerzyy2\nFZUeCY7A4+37SqmoPPq7JiE2yguNlJpnG/H0SI6je6CpKi5azY8tRaEgoWX/dlj0DCz8CxTvhi6D\nvdnSmRMgOt7v6iTIyisq2XXgUPWZRc2zjG2Bs47dB8qOeV9ax5jAGUc8PQNB0T0lnp6B57okxhKl\n/o0mUShIaDpc6p01fPI47FgK8anQ7yuQ2hc69T3yZ0JXrbvUzpQermB7USlbA4Gx7ahmqhK2FZay\n/9DR/RuREUbXxNjqM4uawVHV35HaMUZzMlAoSKhzDjZ85F3jYevnUJTvzZiuEhXnLanRqa/3Z1VY\ndMqATidpwlw7ta/0MNsKveDYVljVTFVy5LmiUspq9W/ERkVUN1PVHklV9WdiO+jfUChIeCkvg6LN\nsHc97FkPezd4t6r7hw/W2NggqWcgLDKODY4OqT4cgIQC5xwFB8sC/RtHzjJq9m/s2FdKre4NEuOi\nanWI1+jjSImjW3Jc2C842NRQ0Nq7EhqiYryRSXWNTnIODu4KBMT6GmGxHla/Awd2HL19XPKRs4zU\nvkffT+qpuRNtmJnROSGWzgmxZNZzOfLyikp27D/EtsIStlT1aRSWsCUQIkvyi9hz8Nj+jc4JsYEO\n8WNHUvVIjic9MbZNLEKoMwUJf2UHa51Z1AiOwk1QefjIthHR3vyJ2mHRKcO7xXT04wgkxJSUVbAt\n0By1JdCfsa3o6BA5WFZx1HuiIoyuSXHVI6m6p1T1cRzp70jpEO1b/4aaj0QAKitg35a6zzL2bPCu\nV11TQtf6zzI6pmsYrQBeM9W+0vI6R1JV9XNsLyrlcMXR369x0RGBDvCaHeJHD8sN1sWTFAoiTVG8\n59iw2LvRu79vC1Dj/0d0xxp9FxlH92Mk9/aawEQCKisduw8eOmokVe0Q2XXgELW/gpPjo+uYv+FN\nAuyb3pEuic1bvDAkQsHMxgIPA5HA0865B+rZbjzwAnC6c67Bb3yFgrSaw6Ve89PeDbU6wAN/lpce\n2dYiILlX3aOlUvtqmQ+pU1l5JTv2lR51hnFkVJXXZFVYfKT5c/I5J3PXJYOa9bN872g2s0jgMeC/\ngHxgoZnNdc6tqLVdInAr8GmwahFplug4SD/Fu9VWWel1cNcOiz3r4YvXvQl6NcWn1h0WnfpCYnfN\nyWinYqIi6J3agd6pHerdprisvHokVWsscR7M0UcjgTXOuXUAZjYTGAesqLXdL4HfAXcGsRaRlhUR\nAUndvdtJZx77eum+I53fNYNjy2JY/gq4Gp2UkbHe3Iu6+jFS+mjGdzvXISaK/l0S6N8loVV+XjBD\noSewucbjfGBUzQ3MbATQ2zn3mpkpFKTtiEuC7pnerbaKw4E5GRtqdYBvgI0fQ9mBo7dP7FHPaKnA\nnAx1fksLCmYo1PUvtboDw8wigGnAtY3uyGwyMBmgT58+LVSeiE8ioyH1ZO9We1qGc1BcUPdoqbXv\nwf5tR28fm1T/WUZSL4jUVCQ5PsH8F5MP9K7xuBewtcbjRGAo8O/AuN1uwFwzu7R2Z7Nz7kngSfA6\nmoNYs4i/zKBjZ+/W+/RjXy8rhsKNx55l7FwJX74FFTUmXUVEeaOijjnLCDyObZ3mCAkvwQyFhcAA\nM+sLbAGuBr5Z9aJzrgjoXPXYzP4N3NnY6CORdi2mA3QZ5N1qq6zwziTqOsvY+jmU7D16+47p9Y+W\nSuiqZql2Kmih4JwrN7NbgLfxhqQ+45xbbmb3AYucc3OD9bNF2qWISG9YbHIv6Hv2sa+XFNYxJ2MD\nbPoElr149IKEkTHeiKn4TrVuKUc/7lBrm5gEhUmY0+Q1ETmyIGFVWBRt9s4sSvZ6YVJ9fy8cLq5/\nPxFRdQRJXbda4RKbrGG5Qeb7PAURCSMNLUhY2+FSKK0VFLVvxXu8P/dthR0rvPtl+xvYqR0bFE25\nxaWoM72F6W9TRI5PdBxEd4PEbsf3vorDx551NBQqBWu9+6VFHLXcSG2xSYFAqau5q4EzFV2To04K\nBRFpHZHRkJDu3Y5HZYUXDHU1ZdV1K8o/ct9V1L/f6I71N2c1dIuOb9P9JgoFEQltEZFeh/bxXjzJ\nOTi0r4EAqRUwu788cpZSc7n12iJjm9ZPUrszPkw64RUKItI2mXkLEVZddKmpnPM605vSzFVS6M0b\n2ZbbZjrhFQoiIjWZeRdbiunoDe89HsHuhM+5Hs685YQOrzEKBRGRlhLsTviErsGpuwaFgoiI35rb\nCR8Emi0iIiLVFAoiIlJNoSAiItUUCiIiUk2hICIi1RQKIiJSTaEgIiLVFAoiIlIt7C6yY2a7gI3N\nfHtnYHcLluMnHUtoaivH0laOA3QsVU5yzjU6Oy7sQuFEmNmiplx5KBzoWEJTWzmWtnIcoGM5Xmo+\nEhGRagoFERGp1t5C4Um/C2hBOpbQ1FaOpa0cB+hYjku76lMQEZGGtbczBRERaUCbDAUzG2tmq8xs\njZlNreP1WDObFXj9UzPLaP0qm6YJx3Ktme0ys9zA7QY/6myMmT1jZjvNbFk9r5uZPRI4ziVmlt3a\nNTZVE47lPDMrqvGZ/Ly1a2wKM+ttZu+b2UozW25mP6hjm7D4XJp4LOHyucSZ2QIzywscyy/q2CZ4\n32HOuTZ1AyKBtcDJQAyQBwyutc3NwBOB+1cDs/yu+wSO5Vrgj37X2oRjOQfIBpbV8/olwJuAAaOB\nT/2u+QSO5TzgNb/rbMJxdAeyA/cTgS/r+PcVFp9LE48lXD4XAxIC96OBT4HRtbYJ2ndYWzxTGAms\ncc6tc86VATOBcbW2GQf8NXD/ReACM7NWrLGpmnIsYcE59wGwp4FNxgF/c55PgBQz69461R2fJhxL\nWHDObXPOfRa4vx9YCfSstVlYfC5NPJawEPi7PhB4GB241e78Ddp3WFsMhZ7A5hqP8zn2H0f1Ns65\ncqAISGuV6o5PU44F4MrAqf2LZta7dUprcU091nBxRuD0/00zG+J3MY0JND+MwPuttKaw+1waOBYI\nk8/FzCLNLBfYCbzjnKv3c2np77C2GAp1pWXtlG3KNqGgKXW+CmQ45zKBf3Hkt4dwEy6fSVN8hrek\nwHDgUeAVn+tpkJklAHOA25xz+2q/XMdbQvZzaeRYwuZzcc5VOOeygF7ASDMbWmuToH0ubTEU8oGa\nvy33ArbWt42ZRQHJhGZzQKPH4pwrcM4dCjx8CjitlWpraU353MKCc25f1em/c+4NINrMOvtcVp3M\nLBrvS/R559xLdWwSNp9LY8cSTp9LFedcIfBvYGytl4L2HdYWQ2EhMMDM+ppZDF4nzNxa28wFvhu4\nPx54zwV6bEJMo8dSq333Ury21HA0F/hOYLTLaKDIObfN76Kaw8y6VbXvmtlIvP9nBf5WdaxAjX8B\nVjrnHqpns7D4XJpyLGH0uaSbWUrgfjxwIfBFrc2C9h0W1RI7CSXOuXIzuwV4G2/0zjPOueVmdh+w\nyDk3F+8fz/+a2Rq8dL3av4rr18RjudXMLgXK8Y7lWt8KboCZzcAb/dHZzPKBe/A60HDOPQG8gTfS\nZQ1QDHzPn0ob14RjGQ/cZGblQAlwdYj+0jEG+DawNNB+DXAX0AfC7nNpyrGEy+fSHfirmUXiBdds\n59xrrfUdphnNIiJSrS02H4mISDMpFEREpJpCQUREqikURESkmkJBRESqKRREAsysosYKmrlWx6q0\nJ7DvjPpWVRUJJW1unoLICSgJLC0g0m7pTEGkEWa2wcx+G1jjfoGZ9Q88f5KZvRtYjPBdM+sTeL6r\nmb0cWHgtz8zODOwq0syeCqyR/8/AbFXM7FYzWxHYz0yfDlMEUCiI1BRfq/loYo3X9jnnRgJ/BKYH\nnvsj3rLSmcDzwCOB5x8B/hNYeC0bWB54fgDwmHNuCFAIXBl4fiowIrCfKcE6OJGm0IxmkQAzO+Cc\nS6jj+Q3AV5xz6wKLrm13zqWZ2W6gu3PucOD5bc65zma2C+hVY6HCquWc33HODQg8/gkQ7Zz7lZm9\nBRzAW7XzlRpr6Yu0Op0piDSNq+d+fdvU5VCN+xUc6dP7GvAY3gq3iwOrXor4QqEg0jQTa/w5P3B/\nHkcWIvsW8FHg/rvATVB9sZSk+nZqZhFAb+fc+8CPgRTgmLMVkdai30hEjoivscImwFvOuaphqbFm\n9ineL1KTAs/dCjxjZj8CdnFkBdEfAE+a2fV4ZwQ3AfUtNx0J/N3MkvEunDItsIa+iC/UpyDSiECf\nQo5zbrfftYgEm5qPRESkms4URESkms4URESkmkJBRESqKRRERKSaQkFERKopFEREpJpCQUREqv1/\nbP7w4+XSn3MAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FeW5wPHfQ8hGthPCnhCCCLKTxIiKK3VD24o7UG3V\nLlytS93aS1tbLd1s64JtvbZqXW7rFXCnXsV6La2KG0ECCIggawg7JCGQPc/9YybJSUg4h5CTOefk\n+X4++eTMzDtznsmBec77vvO+I6qKMcYYcyQ9vA7AGGNM+LNkYYwxJiBLFsYYYwKyZGGMMSYgSxbG\nGGMCsmRhjDEmIEsWxhhjArJkYYwxJiBLFsYYYwLq6XUAnaVPnz6ak5PjdRjGGBNRli5dukdV+wYq\nFzXJIicnh8LCQq/DMMaYiCIim4MpZ81QxhhjArJkYYwxJiBLFsYYYwKyZGGMMSYgSxbGGGMCsmRh\njDEmIEsWxhhjAoqacRbGGBN1GhqgpgKqDzg/NRVQXe4u+61PyoCCb4Y0FEsWxhjTmVSh9pDfBb3c\n70J/4PCfpvXlLRNATYXzE4yskyI7WYjIFOBhIAZ4QlXva7V9CPAk0BfYB1yjqsXutmuBu92iv1DV\nZ0IZqzGmG1OFuuq2v7m3922+xv+i32q9NgR+T4mB+BSIT3V/J0Ov3pA+BOKS3fXJ7rYUv3UpfutT\nnfU940L+JwpZshCRGOAR4DygGFgiIgtUdbVfsfuB/1bVZ0TkS8Cvga+LSG/gHqAAUGCpu+/+UMVr\njIlA9bVtfGvv4Lf5htog3lD8LtwpzRfulAHNF27/9Y0X9xbr3Z+eCSAS8j9RZwllzWIisF5VNwCI\nyFxgKuCfLEYDt7uvFwGvuK8vAN5S1X3uvm8BU4DnQhivMaYrBNsOX9POt3b/pFBXGdx7xvZq9Q09\nBXxDgvjmngJxfhf42F7Qo3veFxTKZJEJbPVbLgZOblVmOXA5TlPVpUCKiGS0s29m6zcQkZnATIDs\n7OxOC9wY04aGBji0B6rKnYt7wG/t7awPth0+Jv7wJpfkAZAxPLhv7o3r4pIhxrpnj1Uo/4Jt1a+0\n1fJdwB9F5DrgHWAbUBfkvqjqY8BjAAUFBYdtN8YESRUO7YWyYijfBmXboLwYykv8Xm8P3FQTVDu8\nXwLwuB3eBC+UyaIYGOy3nAWU+BdQ1RLgMgARSQYuV9UyESkGzm61779CGKsx0UsVKve3kwS2uQmi\nBOqrW+7XIxZSB0FaFgw+xXmdmgmJvqhphzfBC2WyWAIMF5GhODWG6cDX/AuISB9gn6o2AD/EuTMK\n4E3gVyKS7i6f7243xrRWVdZ84W9KCNuaawnlJc6tnP4kpvniPygPRn0FUrMgLdNZl5oJSX27bfu8\nOVzIkoWq1onIzTgX/hjgSVVdJSKzgUJVXYBTe/i1iChOM9RN7r77ROTnOAkHYHZjZ7cx3Up1Rctv\n/62TQNk2p9PXn/Rw2vbTMqH/GBh+QcskkJYJyf2hR4w352QikqhGR1N/QUGB2pPyTESprWyjRlDs\n/i5xXleVHb5fcv/mWkFaVnMCaEwGKQMgJrbrz8dEJBFZqqoFgcrZLQLGhEJdtV9NoI0kULYNKtuo\nLPfKcC746UNgyKTDawQpg6zj13jCkoUxR6u+Fg5sP7yD2L+J6ODuw/dL8DXXBLJOapkEGl/HJnT9\n+RgTBEsWxvhrqIcDO9pPAmXboGInh93JHZ/afOEfOKFlEkjLcpqN4pI8OSVjOoMlC9N9NDTAwV2t\nbh8tbnkH0YEdoPUt94tNar7wDx/VqkbgJoKEVG/OyZguYsnCRAdVOLjn8IFk/h3IbQ0q65nQfOEf\nembLJJCW6SYCn40dMN2eJQsT/toaVNaUBEraH1QWE+feNeQOKmvdLJSa5YwutkRgTECWLEx4ObQP\nNr8PmxfDzlXtDyrr0dO5Myh1EGTmw6ivNnceN4467tXHBpUZ00ksWRhvVex2EsPmxbBpMexa5azv\nmQD9xzo/I6a0TAKpmZDczwaVGdOFLFmYrlW+3U0M7zm/93zurI/tBYNPhrGXwpDTndpCz3hvYzUm\nDDU0KHsOVrNtfyUlpVVsKz1EcnwsXzs5tDNvW7IwoVW6tWVy2LfBWR+XAtmnQO7XnOQwKNdGHRsD\nVNXWs72sipLSSrbtr2RbaaXz2v1dUlpFTX3LJ/HlZfssWZgIogr7NzU3KW1+D0q3ONsS0iB7kvOc\n4CGnwYDx9owB0+2oKqWHag9LANtKK9lWWsW2/ZXsqWh5o4YI9EuJZ5AvkbGZaVwwZgCZ6YkMSktk\nkC+RzPREUhNC/3/J/reajlOFveubaw2b33c6pMGZtmLIJDjlJsg5DfqNsc5mE/Xq6hvYUV7V1DxU\nUlpF8f6WieFQTctxPPE9e5DpXvRHjuznJAJfIoN8CWT5ejEgLYG4nt7/37FkYYKnCrs/a5kcKnY6\n25L6OUlhyGmQczr0OcGSg4k6FdV1LZqHmmoGbkLYUV5FQ6vB/b2T4sj0JTKsbxJnDu/LIF9CU3IY\n5EskIykOiYDbty1ZmPY1NDh3JzU2KW1+33maGji3rQ49szk5ZBxv4xVMRGtoUPZUVFPcKgE0NxEd\noryqrsU+PXsIA30JDEpL5JRhGWT63KYhv9+JcdFx154lC9Osvg52rGge57D5fagqdbb5sp3nIjTW\nHtJzLDmYiFJVW9/UQbyt9FBTH0FJaSUlZZVsb6PjOCWhp1ML8CVSMCS9qY8g05dApq8XfVPiienR\nPf4fWLLozuproaTIqTVsWgxbPmx+kE7v45yBbjmnO8nBN/jIxzLGQ6rK/kO1lJRWHtZH0Ph6T0VN\ni31EoH9KApnpiYzP8jFlbAJZvuZO40G+RFIT7A69RpYsupO6atj2SXNy2Pox1B50tvUZAeOuaE4O\nqQO9jdUYP7X1DewoqzosATQ2D5WUVlFZ27LjOCG2R1Nz0KiBqc1NQ+lOTWFAWgKxMdavFixLFtGs\nthKKC5vHORQvgboqZ1u/MZB3tZMYhkxyRkQb45EDVf63k1a16C8oKa1kZxsdx32S4xjkS2RE/xTO\nPqFfUx9BY+dxeq/YiOg4jhSWLKJJzUHY+pHbIb0Yti2F+hpAYMC45jEOQyY5E+gZ0wUaGpTdFdWH\nNQ/531F0oFXHcWyMMDDNufBPGtbH6SNIb9l5nBAbHR3HkcKSRSSrKneTg3sra8kyaKgDiXEewHPy\nfzijo7NPgUSf19GaKFVVW3/YLaTFfk1FO8qqqK1vWS1ITehJZnovstITmTi092FNRH2T4+nRTTqO\nI4Uli0hSud/phG5MDtuXgzZAj1hnLqVJtzo1h+yTIT7F62hNFKmqrWdVSTkri0vZsq/5DqJt+yvZ\ne7Blx3EPgQGpCQzyJZI3OJ3M8U4iaOw8HuRLIMU6jiOOJYtwdnBvyxlZd34KKMTEQ1YBnHGXcytr\n1kSI6+V1tCZKqCpb91WybOt+lm0pZdmW/azeXt5UO0iMjWlqEhozKK25iSjNqRn0T7WO42hkySKc\nVOxqrjVsWgy71zjreybC4JPg7B86ySGzAGITvI3VRI0DVbWsKC5j2RYnORRtLW2qLSTGxjBhcBrf\nOv048rJ95A720S8l3jqOuyFLFl4qL2keHb1pMexd56yPS3am6268lXVQPvSM8zZWExXqG5T1uyoo\naqo1lPL5rgOo26UwrG8Sk0f2Iy/bR97gdEb0T6an1RIMliy61v7NLZuV9m901senQvapkP91p0N6\n4ASbkdV0ir0V1RRtdZLCsq37Wb61jIpq586jtMRY8rJ9XDRuIHnZPiZk+UjrZX0Jpm12RQoVVefZ\nDU3TdS+Gsq3OtgSf0xE98TvudN3j7Klv5pjV1DWwZnu505zkJogt+5zH0cb0EEYNTOHSvExyB/vI\ny/YxtE+SNSeZoFmy6CyqsGddc5PS5sVwYLuzrVcfZ2zDpFuc5NBvtM3Iao6JqlJSVsWyLfsp2lLK\nsq2lrNxWRk2dM7dRv5R48rPTufrkbPKy0xmXmRY1E9oZb1iy6KiGBqcDevP7bqf0+3Bwl7MteUCr\n6bpH2KR75pgcqqljZXGZW2Nw+ht2HXAekhPfswfjMtO49tQh5GWnkzvYx8C0BKs1mE4V0mQhIlOA\nh4EY4AlVva/V9mzgGcDnlpmlqq+LSA6wBljrFv1QVW8IZawBNdQ7t6421ho2vw+V+5xtqVkwbHJz\ncuh9nCUH02ENDcrGvQebbltdtqWUtTsPUO/Od5GT0YvTju/T1Jw0ckBqWDwcx0S3kCULEYkBHgHO\nA4qBJSKyQFVX+xW7G5ivqo+KyGjgdSDH3faFquaGKr6A6utgx3K/5PABVJc523xD4IQL3eRwmrNs\nycF0UOmhmqZO6KKtzk9ZZS0AyfE9yR3s47tnD2vqhM5Ijvc4YtMdhbJmMRFYr6obAERkLjAV8E8W\nCqS6r9OAkhDGc2R1NbC9qHmcw5aPmqfrzjgexlzizsg6CdKyPAvTRLa6+gY+23GgxR1KG3Y7M/+K\nwAn9U7ho3ADyBqeTm+1jWN/kbvO8BBPeQpksMoGtfsvFwMmtytwL/ENEbgGSgHP9tg0VkWVAOXC3\nqr7b+g1EZCYwEyA7O7tjUZZvh5f/w5mRtda5c4S+I2H8Vc39DikDOnZs0+3tKq/iEzcpLNtSysri\nsqaptDOS4sjL9nF5fhZ5g32My0qzaTBM2Aplsmjr61CrSYaZATytqg+IyKnAX0VkLLAdyFbVvSJy\nIvCKiIxR1fIWB1N9DHgMoKCgoPWxg9MrA6oPQN7Xm5NDUp8OHcp0b878SWVNg92KtpayrbQScGZR\nHT0ojWknDSYv20d+djpZ6YnWCW0iRiiTRTHg/3i1LA5vZvoWMAVAVT8QkQSgj6ruAqrd9UtF5Atg\nBFDY6VH2jIOZizr9sCa6qSpb9h1qbk5qNX9Spi+RvGwf3zx9KLmDfYwZlGpTapuIFspksQQYLiJD\ngW3AdOBrrcpsAc4BnhaRUUACsFtE+gL7VLVeRI4DhgMbQhirMUfUev6kZVtL2dfO/El5g330S7W5\nu0x0CVmyUNU6EbkZeBPnttgnVXWViMwGClV1AXAn8LiI3I7TRHWdqqqInAnMFpE6oB64QVX3hSpW\nY/w1zp/UnBj2s25XRYv5k75k8yeZbkZUO9bUH24KCgq0sLDzW6lM9NtTUe2Ogt5P0dbSNudPyhuc\nbvMnmagkIktVtSBQORvBbbqVQPMnjRyQwiV5g5qSg82fZIzDkoWJWv7zJzV2Qn9aUm7zJxnTAZYs\nTNQ4VFPndkI7iaFoq82fZExnsWRhIlJDg7Jhz0H31tXD508aktGLScMyyMtOt/mTjOkElixMRPCf\nP2nZ1lKKtuynvMrphLb5k4wJPUsWJiztP1jDayu3NzUntZ4/6cvjB7qzrqbb/EnGdAFLFibsVNfV\nc/UTH7F6e7nNn2RMmLBkYcLOg299zurt5fzX1flcOHaAdUIbEwYsWZiw8sEXe3nsnQ3MmJjNReMG\neh2OMcZlt4eYsFFWWcud84vIyUjiJ18Z5XU4xhg/VrMwYeOnr37KzgPVvHjjJHrF2T9NY8KJ1SxM\nWHi1aBuvFpVw65eGkzvY53U4xphWLFkYz20rreTuVz4lL9vHTZOHeR2OMaYNliyMpxoalDvnF1Hf\noMyZlmtTfRsTpqxh2Hjqifc28OGGffz28vEMyUjyOhxjTDvsa5zxzOqScn735lrOH92fKwuyvA7H\nGHMEliyMJ6pq67lt3jJ8veK47/LxNvDOmDBnzVDGE79duJbPd1bw9PUn0TspzutwjDEBWM3CdLl3\n1+3mycUbufbUIZx9Qj+vwzHGBMGShelS+w/WcNfzyzm+XzKzLrRR2sZECksWpsuoKj9+ZSV7K2qY\nMy3XHmFqTASxZGG6zEufbOP1lTu44/wRjM1M8zocY8xRsGRhusTWfYe4Z8EqJub05j/OtFHaxkQa\nSxYm5OoblNvnFSHAA1dNsKfaGROB7NZZE3J/+vcXFG7ez4NXTWBw715eh2OM6QCrWZiQWllcxkNv\nfc6Xxw/k0rxMr8MxxnSQJQsTMpU19Xxv3jL6JMfzy0vG2ihtYyJYSJOFiEwRkbUisl5EZrWxPVtE\nFonIMhFZISIX+W37obvfWhG5IJRxmtD41etr2LD7IA9cNQFfLxulbUwkC1mfhYjEAI8A5wHFwBIR\nWaCqq/2K3Q3MV9VHRWQ08DqQ476eDowBBgH/JyIjVLU+VPGazrXos1389cPNfPv0oZx2fB+vwzHG\nHKNQ1iwmAutVdYOq1gBzgamtyiiQ6r5OA0rc11OBuaparaobgfXu8UwE2FtRzfdfWMHIASncdcEJ\nXodjjOkEoUwWmcBWv+Vid52/e4FrRKQYp1Zxy1Hsa8KQqjLrpZWUV9by0LRcEmJtlLYx0SBgshCR\nm0UkvQPHbqs3U1stzwCeVtUs4CLgryLSI8h9EZGZIlIoIoW7d+/uQIims81bspW3Vu/kB1NOYNTA\n1MA7GGMiQjA1iwE4/Q3z3Q7rYG9pKQYG+y1n0dzM1OhbwHwAVf0ASAD6BLkvqvqYqhaoakHfvn2D\nDMuEysY9B/nZ31czaVgG3zxtqNfhGGM6UcBkoap3A8OBvwDXAetE5FciEmjOhiXAcBEZKiJxOB3W\nC1qV2QKcAyAio3CSxW633HQRiReRoe77fxz0WZkuV1ffwO3zioiNER64agI9bJS2MVElqD4LVVVg\nh/tTB6QDL4jIb4+wTx1wM/AmsAbnrqdVIjJbRC52i90JfEdElgPPAdepYxVOjWM1sBC4ye6ECm9/\nXLSeoq2l/PLScQxMS/Q6HGNMJxMnDxyhgMitwLXAHuAJ4BVVrXX7FtapaljMCldQUKCFhYVeh9Et\nfbJlP1f+6QMunjCIh6bleh2OMeYoiMhSVS0IVC6YcRZ9gMtUdbP/SlVtEJGvdDRAEx0OVtdx+7wi\nBqQm8LOpY7wOxxgTIsE0Q70O7GtcEJEUETkZQFXXhCowExl+/tpqtuw7xINXTSA1IdbrcIwxIRJM\nsngUqPBbPuiuM93cP1btYO6SrfzHmcM4+bgMr8MxxoRQMMlC1K9jQ1UbsKnNu71dB6qY9dJKRg9M\n5Y7zRngdjjEmxIJJFhtE5FYRiXV/vgdsCHVgJnypKj94YQUHq+t4eHoucT1t8mJjol0w/8tvACYB\n23AGy50MzAxlUCa8/e3Dzfxr7W5+dNEohvdP8TocY0wXCNicpKq7cAbUGcP6XRX84n/XcNaIvnzj\n1CFeh2OM6SIBk4WIJOBMyzEGZ4Q1AKr6zRDGZcJQTV0Dt81bRq+4GH53xXh7mJEx3UgwzVB/xZkf\n6gLg3zjzNB0IZVAmPD389ud8uq2cX182jn6pCYF3MMZEjWCSxfGq+hPgoKo+A3wZGBfasEy4+Xjj\nPv7rX19wVUEWU8YO9DocY0wXCyZZ1Lq/S0VkLM5DinJCFpEJOweqarl9XhGD03vx06/aKG1juqNg\nxks85j7P4m6c2WCTgZ+ENCoTVu5dsJrtZZU8f8MkkuNtiI0x3dER/+e7kwWWq+p+4B3guC6JyoSN\n/12xnRc/KebWLx3PiUM68gwsY0w0OGIzlDta++YuisWEmR1lVfzo5ZVMyErjlnOGex2OMcZDwfRZ\nvCUid4nIYBHp3fgT8siMpxoalLueX05NXQMPTcslNsZGaRvTnQXTAN04nuImv3WKNUlFtafe38R7\n6/fwq0vHcVzfZK/DMcZ4LJgR3PYw5W5m7Y4D/GbhZ5wzsh8zJg4OvIMxJuoFM4L7G22tV9X/7vxw\njNeq6+r53txlpCb05Dc2StsY4wqmGeokv9cJwDnAJ4Aliyj0wD8+57MdB/jLtQX0SY73OhxjTJgI\nphnqFv9lEUnDmQLERJn3v9jD4+9u4OqTszlnVH+vwzHGhJGO3OJyCLD7KKNM2aFa7py/nJyMJH78\n5VFeh2OMCTPB9Fn8HefuJ3CSy2hgfiiDMl3vJ69+yq4D1bx04yR6xdkobWNMS8FcFe73e10HbFbV\n4hDFYzzwatE2Fiwv4c7zRjBhsM/rcIwxYSiYZLEF2K6qVQAikigiOaq6KaSRmS6xrbSSu1/5lBOH\npHPj2cO8DscYE6aC6bN4HmjwW65315kIV9+g3DGviIYG5aGrculpo7SNMe0I5urQU1VrGhfc13Gh\nC8l0lSfe3cBHG/dxz8VjyM7o5XU4xpgwFkyy2C0iFzcuiMhUYE/oQjJdYVVJGff/Yy1TxgzgyhOz\nvA7HGBPmgumzuAF4VkT+6C4XA22O6jaRoaq2ntvmFpHeK45fXTbORmkbYwIKZlDeF8ApIpIMiKoG\n/fxtEZkCPAzEAE+o6n2ttj8ETHYXewH9VNXnbqsHVrrbtqjqxZhOcd8bn7FuVwXPfHMivZOsRdEY\nE1gw4yx+BfxWVUvd5XTgTlW9O8B+McAjwHk4tZElIrJAVVc3llHV2/3K3wLk+R2iUlVzj+ZkTGDv\nfL6bp9/fxHWTcjhrRF+vwzHGRIhg+iwubEwUAO5T8y4KYr+JwHpV3eB2is8Fph6h/AzguSCOazpo\n/8Ea7np+Ocf3S2bWhSO9DscYE0GCSRYxItI0o5yIJALBzDCXCWz1Wy521x1GRIYAQ4F/+q1OEJFC\nEflQRC4J4v3MEagqP3xpJfsP1TBnWi4JsTFeh2SMiSDBdHD/DXhbRJ5yl68Hngliv7Z6TbWNdQDT\ngRdUtd5vXbaqlojIccA/RWSl23/S/AYiM4GZANnZ2UGE1H29sLSYhat2MOvCkYzNTPM6HGNMhAlY\ns1DV3wK/AEbhzAu1EBgSxLGLAf8n52QBJe2UnU6rJihVLXF/bwD+Rcv+jMYyj6lqgaoW9O1r7e/t\n2bL3EPcuWMXJQ3vznTPsAYfGmKMX7JDdHTijuC/HeZ7FmiD2WQIMF5GhIhKHkxAWtC4kIicA6cAH\nfuvSG5u+RKQPcBqwuvW+JrC6+gbumF9EDxEeuGoCMT3sNlljzNFrtxlKREbgXOBnAHuBeTi3zk5u\nbx9/qlonIjcDb+LcOvukqq4SkdlAoao2Jo4ZwFxV9W+iGgX8WUQacBLaff53UZng/enfX1C4eT9z\npuWSlW6jtI0xHSMtr9F+G5wL9bvAt1R1vbtug6qGZTtGQUGBFhYWeh1GWFm+tZTLH32fC8cN5PfT\nc23wnTHmMCKyVFULApU7UjPU5TjNT4tE5HEROYe2O61NGDpUU8ft84romxLPL6aOtURhjDkm7SYL\nVX1ZVacBI3E6mG8H+ovIoyJyfhfFZzroV6+vYcOegzxw5QTSesV6HY4xJsIFczfUQVV9VlW/gnNH\nUxEwK+SRmQ7752c7+duHW/jOGUOZdHwfr8MxxkSBo3qAgaruU9U/q+qXQhWQOTZ7Kqr5wQsrGDkg\nhbsuOMHrcIwxUcIethxFVJVZL66gvKqOv337ZOJ72ihtY0znsEejRZG5S7byf2t28Z9TRjJyQKrX\n4Rhjoogliyixcc9BZv99Nacdn8H1k3K8DscYE2UsWUSB2voGbptXRFzPHtx/5QR62ChtY0wnsz6L\nKPCHf65n+dZSHvlaPgPTEr0OxxgThaxmEeGWbt7PH/+5jsvyM/ny+IFeh2OMiVKWLCJYRXUdd8wv\nYmBaIvdePMbrcIwxUcyaoSLYz/++mi37DjFv5qmkJtgobWNM6FjNIkIt/HQH8wq3cuNZw5g4tLfX\n4Rhjopwliwi0q7yKH760grGZqdx27givwzHGdAOWLCKMqvL9F1ZQWVvPnGl5xPW0j9AYE3p2pYkw\nf/1wM//+fDc/umgUx/dL9jocY0w3YckigqzfdYBf/u8azhrRl6+fEsxj0I0xpnNYsogQNXUNfG9u\nEUnxPfndlePtYUbGmC5lt85GiDn/9zmrSsr589dPpF9KgtfhGGO6GatZRICPN+7j0X9/wfSTBnPB\nmAFeh2OM6YYsWYS58qpabp9XRHbvXvzkK6O9DscY001ZM1SYu/fVVewor+L5G04lKd4+LmOMN6xm\nEcZeW1HCS8u2cfPk48nPTvc6HGNMN2bJIkxtL6vkxy9/Su5gHzd/6XivwzHGdHOWLMJQQ4Ny1/PL\nqalr4KFpucTG2MdkjPGWXYXC0JOLN7J4/V5++tXRDO2T5HU4xhhjySLcfLajnN8uXMu5o/oz/aTB\nXodjjDGAJYuwUlVbz21zi0hNjOU3l4+zUdrGmLAR0mQhIlNEZK2IrBeRWW1sf0hEityfz0Wk1G/b\ntSKyzv25NpRxhosH/rGWz3Yc4HdXjCcjOd7rcIwxpknIbtwXkRjgEeA8oBhYIiILVHV1YxlVvd2v\n/C1Anvu6N3APUAAosNTdd3+o4vXa4vV7ePzdjVxzSjaTR/bzOhxjjGkhlDWLicB6Vd2gqjXAXGDq\nEcrPAJ5zX18AvKWq+9wE8RYwJYSxeqrsUC13zl/OcX2T+PFFNkrbGBN+QpksMoGtfsvF7rrDiMgQ\nYCjwz6PdN9KpKj9+ZSV7KqqZMy2XxLgYr0MyxpjDhDJZtNU7q+2UnQ68oKr1R7OviMwUkUIRKdy9\ne3cHw/TWq0UlvLZiO7efN4LxWT6vwzHGmDaFMlkUA/73fmYBJe2UnU5zE1TQ+6rqY6paoKoFffv2\nPcZwu17x/kP85JVPKRiSzg1nDfM6HGOMaVcok8USYLiIDBWROJyEsKB1IRE5AUgHPvBb/SZwvoik\ni0g6cL67LmrUNyh3zF+OAg9NyyWmh90ma4wJXyG7G0pV60TkZpyLfAzwpKquEpHZQKGqNiaOGcBc\nVVW/ffeJyM9xEg7AbFXdF6pYvfDYOxv4eOM+7r9yAoN79/I6HGOMOSLxu0ZHtIKCAi0sLPQ6jKB8\nuq2MS/9rMeeN7s8jX8u3wXfGGM+IyFJVLQhUzkZwd7Gq2npum1dE76Q4fnmJjdI2xkQGe5pOF7vv\njc9Yv6uCv35rIulJcV6HY4wxQbGaRRf619pdPP3+Jq4/LYczhkfe3VvGmO7LkkUX2Xewhu+/sIIR\n/ZP5zykjvQ7HGGOOijVDdQFV5UcvraTsUC3PXD+RhFgbpW2MiSxWs+gCzy8tZuGqHdx5/ghGD0r1\nOhxjjDlV6GhhAAAQ8klEQVRqlixCbPPeg/xswSpOOa433z7jOK/DMcaYDrFkEUJ19Q3cPq+IHj2E\nB66yUdrGmMhlfRYh9Oi/vuCTLaU8PD2XTF+i1+EYY0yHWc0iRIq2ljLn7XVMzR3E1NyonF3dGNON\nWLIIgUM1ddw+r4j+KfHMnjrW63CMMeaYWTNUCPzif9ewae9Bnv32yaQlxnodjjHGHDOrWXSyt9fs\n5H8+2sLMM45j0rA+XodjjDGdwmoWnWj3gWp+8MIKRg1M5Y7zR3gdjjERq7a2luLiYqqqqrwOJWok\nJCSQlZVFbGzHWjssWXQSVWXWiys4UF3Hc9Nzie9po7SN6aji4mJSUlLIycmxmZk7gaqyd+9eiouL\nGTp0aIeOYc1QneR/Pt7C25/tYtaUkYzon+J1OMZEtKqqKjIyMixRdBIRISMj45hqapYsOsGG3RX8\n4rU1nDG8D9dNyvE6HGOigiWKznWsf09LFseo1h2lHR/bg/uvnEAPG6VtTMTbu3cvubm55ObmMmDA\nADIzM5uWa2pqgjrG9ddfz9q1a49Y5pFHHuHZZ5/tjJBDzvosjtEf3l7H8uIy/uvqfPqnJngdjjGm\nE2RkZFBUVATAvffeS3JyMnfddVeLMqqKqtKjR9vfuZ966qmA73PTTTcde7BdxGoWx2Dp5n38cdF6\nLs/P4qJxA70OxxgTYuvXr2fs2LHccMMN5Ofns337dmbOnElBQQFjxoxh9uzZTWVPP/10ioqKqKur\nw+fzMWvWLCZMmMCpp57Krl27ALj77ruZM2dOU/lZs2YxceJETjjhBN5//30ADh48yOWXX86ECROY\nMWMGBQUFTYmsK1nNooMqquu4fd5yBvkSuffi0V6HY0zU+tnfV7G6pLxTjzl6UCr3fHVMh/ZdvXo1\nTz31FH/6058AuO++++jduzd1dXVMnjyZK664gtGjW14TysrKOOuss7jvvvu44447ePLJJ5k1a9Zh\nx1ZVPv74YxYsWMDs2bNZuHAhf/jDHxgwYAAvvvgiy5cvJz8/v0NxHyurWXTQ7L+vonj/IR6alktK\ngo3SNqa7GDZsGCeddFLT8nPPPUd+fj75+fmsWbOG1atXH7ZPYmIiF154IQAnnngimzZtavPYl112\n2WFl3nvvPaZPnw7AhAkTGDOmY0nuWFnNogMWfrqd+YXF3Dz5eE7K6e11OMZEtY7WAEIlKSmp6fW6\ndet4+OGH+fjjj/H5fFxzzTVt3p4aFxfX9DomJoa6uro2jx0fH39YGVXtzPA7zGoWR2lneRWzXlrJ\nuMw0vnfucK/DMcZ4qLy8nJSUFFJTU9m+fTtvvvlmp7/H6aefzvz58wFYuXJlmzWXrmA1i6PQ0KDc\n9fxyqmrreWhaLrExlmuN6c7y8/MZPXo0Y8eO5bjjjuO0007r9Pe45ZZb+MY3vsH48ePJz89n7Nix\npKWldfr7BCLhUsU5VgUFBVpYWBjS93h68Ubu/ftqfn7JWL5+ypCQvpcx3dmaNWsYNWqU12GEhbq6\nOurq6khISGDdunWcf/75rFu3jp49j/67flt/VxFZqqoFgfa1mkWQ1u08wK/f+IzJJ/TlmpOzvQ7H\nGNNNVFRUcM4551BXV4eq8uc//7lDieJYWbIIQk1dA9+bW0RyfE9+e8UEm4bAGNNlfD4fS5cu9TqM\n0HZwi8gUEVkrIutF5PCbip0yV4nIahFZJSL/47e+XkSK3J8FoYwzkAff+pzV28u57/Lx9E2J9zIU\nY4zxRMhqFiISAzwCnAcUA0tEZIGqrvYrMxz4IXCaqu4XkX5+h6hU1dxQxResDzfs5c/vfMGMiYM5\nb3R/r8MxxhhPhLJmMRFYr6obVLUGmAtMbVXmO8AjqrofQFV3hTCeo1ZeVcud85czpHcv7v6yjdI2\nxnRfoUwWmcBWv+Vid52/EcAIEVksIh+KyBS/bQkiUuiuvySEcbbrnldXsaO8ioem5ZIUb907xpju\nK5TJoq1e4Nb36fYEhgNnAzOAJ0TE527Ldm/n+howR0SGHfYGIjPdhFK4e/fuzoscWLC8hJeXbePW\nLw0nLzu9U49tjAlvZ5999mED7ObMmcN3v/vddvdJTk4GoKSkhCuuuKLd4wa6xX/OnDkcOnSoafmi\niy6itLQ02NBDJpTJohgY7LecBZS0UeZVVa1V1Y3AWpzkgaqWuL83AP8C8lq/gao+pqoFqlrQt2/f\nTgu8pLSSu19eSV62j5smH5ajjDFRbsaMGcydO7fFurlz5zJjxoyA+w4aNIgXXnihw+/dOlm8/vrr\n+Hy+I+zRNUKZLJYAw0VkqIjEAdOB1nc1vQJMBhCRPjjNUhtEJF1E4v3WnwZ0yRj3hgblzvnLqWtQ\nHroql542StuYbueKK67gtddeo7q6GoBNmzZRUlJCbm4u55xzDvn5+YwbN45XX331sH03bdrE2LFj\nAaisrGT69OmMHz+eadOmUVlZ2VTuxhtvbJra/J577gHg97//PSUlJUyePJnJkycDkJOTw549ewB4\n8MEHGTt2LGPHjm2a2nzTpk2MGjWK73znO4wZM4bzzz+/xft0lpA1xKtqnYjcDLwJxABPquoqEZkN\nFKrqAnfb+SKyGqgHvq+qe0VkEvBnEWnASWj3+d9FFUpPLt7IBxv28pvLx5HTJynwDsaY0HpjFuxY\n2bnHHDAOLryv3c0ZGRlMnDiRhQsXMnXqVObOncu0adNITEzk5ZdfJjU1lT179nDKKadw8cUXtzv2\n6tFHH6VXr16sWLGCFStWtJhe/Je//CW9e/emvr6ec845hxUrVnDrrbfy4IMPsmjRIvr06dPiWEuX\nLuWpp57io48+QlU5+eSTOeuss0hPT2fdunU899xzPP7441x11VW8+OKLXHPNNZ3zt3KF9Guzqr6u\nqiNUdZiq/tJd91M3UaCOO1R1tKqOU9W57vr33eUJ7u+/hDLORmu2l/PbhWs5f3R/rioYHHgHY0zU\n8m+KamyCUlV+9KMfMX78eM4991y2bdvGzp072z3GO++803TRHj9+POPHj2/aNn/+fPLz88nLy2PV\nqlUBJwh87733uPTSS0lKSiI5OZnLLruMd999F4ChQ4eSm+uMNDjSFOjHwm7xcVXV1nPb3CJSE2P5\n9WXjbJS2MeHiCDWAULrkkku44447+OSTT6isrCQ/P5+nn36a3bt3s3TpUmJjY8nJyWlzSnJ/bV1L\nNm7cyP3338+SJUtIT0/nuuuuC3icI83j1zi1OTjTm4eiGcoa5F2/e3Mta3ce4HdXjicj2UZpG9Pd\nJScnc/bZZ/PNb36zqWO7rKyMfv36ERsby6JFi9i8efMRj3HmmWfy7LPPAvDpp5+yYsUKwJnaPCkp\nibS0NHbu3Mkbb7zRtE9KSgoHDhxo81ivvPIKhw4d4uDBg7z88succcYZnXW6AVnNAli8fg9/eW8j\n3zh1CJNP6Bd4B2NMtzBjxgwuu+yypuaoq6++mq9+9asUFBSQm5vLyJEjj7j/jTfeyPXXX8/48ePJ\nzc1l4sSJgPPEu7y8PMaMGXPY1OYzZ87kwgsvZODAgSxatKhpfX5+Ptddd13TMb797W+Tl5cXkian\ntnT7KcpLD9UwZc67JMXH8NotZ5AYFxOC6IwxR8OmKA8Nm6L8GNQ1KGMz07jt3OGWKIwxph3dPln0\nSY7niWsDJlVjjOnWrIPbGGNMQJYsjDFhKVr6U8PFsf49LVkYY8JOQkICe/futYTRSVSVvXv3kpCQ\n0OFjdPs+C2NM+MnKyqK4uJjOnk26O0tISCArK6vD+1uyMMaEndjYWIYOHep1GMaPNUMZY4wJyJKF\nMcaYgCxZGGOMCShqpvsQkd3AkWf1OrI+wJ5OCsdL0XIeYOcSrqLlXKLlPODYzmWIqgZ81GjUJItj\nJSKFwcyPEu6i5TzAziVcRcu5RMt5QNecizVDGWOMCciShTHGmIAsWTR7zOsAOkm0nAfYuYSraDmX\naDkP6IJzsT4LY4wxAVnNwhhjTEDdKlmIyBQRWSsi60VkVhvb40Vknrv9IxHJ6foogxPEuVwnIrtF\npMj9+bYXcQYiIk+KyC4R+bSd7SIiv3fPc4WI5Hd1jMEK4lzOFpEyv8/kp10dYzBEZLCILBKRNSKy\nSkS+10aZiPhcgjyXSPlcEkTkYxFZ7p7Lz9ooE7prmKp2ix8gBvgCOA6IA5YDo1uV+S7wJ/f1dGCe\n13Efw7lcB/zR61iDOJczgXzg03a2XwS8AQhwCvCR1zEfw7mcDbzmdZxBnMdAIN99nQJ83sa/r4j4\nXII8l0j5XARIdl/HAh8Bp7QqE7JrWHeqWUwE1qvqBlWtAeYCU1uVmQo8475+AThHRKQLYwxWMOcS\nEVT1HWDfEYpMBf5bHR8CPhEZ2DXRHZ0gziUiqOp2Vf3EfX0AWANktioWEZ9LkOcSEdy/dYW7GOv+\ntO50Dtk1rDsli0xgq99yMYf/o2kqo6p1QBmQ0SXRHZ1gzgXgcreJ4AURGdw1oXW6YM81UpzqNiO8\nISJjvA4mELcZIw/nW6y/iPtcjnAuECGfi4jEiEgRsAt4S1Xb/Vw6+xrWnZJFW9m1dVYOpkw4CCbO\nvwM5qjoe+D+av21Emkj5TILxCc7UChOAPwCveBzPEYlIMvAicJuqlrfe3MYuYfu5BDiXiPlcVLVe\nVXOBLGCiiIxtVSRkn0t3ShbFgP+36yygpL0yItITSCM8mxUCnouq7lXVanfxceDELoqtswXzuUUE\nVS1vbEZQ1deBWBHp43FYbRKRWJyL67Oq+lIbRSLmcwl0LpH0uTRS1VLgX8CUVptCdg3rTsliCTBc\nRIaKSBxO58+CVmUWANe6r68A/qluT1GYCXgurdqPL8Zpq41EC4BvuHffnAKUqep2r4PqCBEZ0Nh+\nLCITcf7/7fU2qsO5Mf4FWKOqD7ZTLCI+l2DOJYI+l74i4nNfJwLnAp+1Khaya1i3eVKeqtaJyM3A\nmzh3Ez2pqqtEZDZQqKoLcP5R/VVE1uNk4+neRdy+IM/lVhG5GKjDOZfrPAv4CETkOZy7UfqISDFw\nD07HHar6J+B1nDtv1gOHgOu9iTSwIM7lCuBGEakDKoHpYfpl5DTg68BKt30c4EdANkTc5xLMuUTK\n5zIQeEZEYnAS2nxVfa2rrmE2gtsYY0xA3akZyhhjTAdZsjDGGBOQJQtjjDEBWbIwxhgTkCULY4wx\nAVmyMCYAEan3m5G0SNqY5fcYjp3T3iy1xoSTbjPOwphjUOlOsWBMt2U1C2M6SEQ2ichv3GcMfCwi\nx7vrh4jI2+4kjm+LSLa7vr+IvOxOWLdcRCa5h4oRkcfdZxT8wx2di4jcKiKr3ePM9eg0jQEsWRgT\njMRWzVDT/LaVq+pE4I/AHHfdH3Gm7x4PPAv83l3/e+Df7oR1+cAqd/1w4BFVHQOUApe762cBee5x\nbgjVyRkTDBvBbUwAIlKhqsltrN8EfElVN7iT1e1Q1QwR2QMMVNVad/12Ve0jIruBLL8JHhunzX5L\nVYe7y/8JxKrqL0RkIVCBMwvqK37PMjCmy1nNwphjo+28bq9MW6r9XtfT3Jf4ZeARnBmDl7qziBrj\nCUsWxhybaX6/P3Bfv0/zBG5XA++5r98GboSmh9iktndQEekBDFbVRcAPAB9wWO3GmK5i31SMCSzR\nb8ZSgIWq2nj7bLyIfITzxWuGu+5W4EkR+T6wm+YZWb8HPCYi38KpQdwItDetdwzwNxFJw3mgzUPu\nMwyM8YT1WRjTQW6fRYGq7vE6FmNCzZqhjDHGBGQ1C2OMMQFZzcIYY0xAliyMMcYEZMnCGGNMQJYs\njDHGBGTJwhhjTECWLIwxxgT0/8JXmCG/empdAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "network_history = model.fit(X_train, Y_train, batch_size=128, \n",
+ " epochs=4, verbose=1, validation_data=(X_val, Y_val))\n",
+ "plot_history(network_history)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* If you continue training, at some point the validation loss will start to increase: that is when the model starts to **overfit**. \n",
+ "\n",
+ "It is always necessary to monitor training and validation loss during the training of any kind of Neural Network, either to detect overfitting or to evaluate the behaviour of the model **(any clue on how to do it??)**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# %load solutions/sol23.py"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Inspecting Layers"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "_________________________________________________________________\n",
+ "Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ "dense_4 (Dense) (None, 512) 401920 \n",
+ "_________________________________________________________________\n",
+ "dropout_1 (Dropout) (None, 512) 0 \n",
+ "_________________________________________________________________\n",
+ "dense_5 (Dense) (None, 512) 262656 \n",
+ "_________________________________________________________________\n",
+ "dropout_2 (Dropout) (None, 512) 0 \n",
+ "_________________________________________________________________\n",
+ "dense_6 (Dense) (None, 10) 5130 \n",
+ "=================================================================\n",
+ "Total params: 669,706\n",
+ "Trainable params: 669,706\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "# We already used `summary`\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `model.layers` is iterable"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Model Input Tensors: Tensor(\"dense_4_input:0\", shape=(?, 784), dtype=float32)\n",
+ "\n",
+ "Layers - Network Configuration:\n",
+ "\n",
+ "dense_4 True\n",
+ "Layer Configuration:\n",
+ "{'kernel_constraint': None, 'use_bias': True, 'trainable': True, 'dtype': 'float32', 'units': 512, 'name': 'dense_4', 'bias_constraint': None, 'kernel_regularizer': None, 'bias_initializer': {'config': {}, 'class_name': 'Zeros'}, 'kernel_initializer': {'config': {'seed': None, 'mode': 'fan_avg', 'distribution': 'uniform', 'scale': 1.0}, 'class_name': 'VarianceScaling'}, 'activity_regularizer': None, 'batch_input_shape': (None, 784), 'activation': 'relu', 'bias_regularizer': None}\n",
+ "----------------------------------------\n",
+ "dropout_1 True\n",
+ "Layer Configuration:\n",
+ "{'name': 'dropout_1', 'rate': 0.2, 'trainable': True}\n",
+ "----------------------------------------\n",
+ "dense_5 True\n",
+ "Layer Configuration:\n",
+ "{'bias_constraint': None, 'use_bias': True, 'trainable': True, 'units': 512, 'name': 'dense_5', 'kernel_constraint': None, 'kernel_regularizer': None, 'bias_initializer': {'config': {}, 'class_name': 'Zeros'}, 'kernel_initializer': {'config': {'seed': None, 'mode': 'fan_avg', 'distribution': 'uniform', 'scale': 1.0}, 'class_name': 'VarianceScaling'}, 'activity_regularizer': None, 'activation': 'relu', 'bias_regularizer': None}\n",
+ "----------------------------------------\n",
+ "dropout_2 True\n",
+ "Layer Configuration:\n",
+ "{'name': 'dropout_2', 'rate': 0.2, 'trainable': True}\n",
+ "----------------------------------------\n",
+ "dense_6 True\n",
+ "Layer Configuration:\n",
+ "{'bias_constraint': None, 'use_bias': True, 'trainable': True, 'units': 10, 'name': 'dense_6', 'kernel_constraint': None, 'kernel_regularizer': None, 'bias_initializer': {'config': {}, 'class_name': 'Zeros'}, 'kernel_initializer': {'config': {'seed': None, 'mode': 'fan_avg', 'distribution': 'uniform', 'scale': 1.0}, 'class_name': 'VarianceScaling'}, 'activity_regularizer': None, 'activation': 'softmax', 'bias_regularizer': None}\n",
+ "----------------------------------------\n",
+ "Model Output Tensors: Tensor(\"dense_6/Softmax:0\", shape=(?, 10), dtype=float32)\n"
+ ]
+ }
+ ],
+ "source": [
+ "print('Model Input Tensors: ', model.input, end='\\n\\n')\n",
+ "print('Layers - Network Configuration:', end='\\n\\n')\n",
+ "for layer in model.layers:\n",
+ " print(layer.name, layer.trainable)\n",
+ " print('Layer Configuration:')\n",
+ " print(layer.get_config(), end='\\n{}\\n'.format('----'*10))\n",
+ "print('Model Output Tensors: ', model.output)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Extract hidden layer representation of the given data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "One simple way to do it is to use the weights of your model to build a new model that's truncated at the layer you want to read. \n",
+ "\n",
+ "Then you can run the `._predict(X_batch)` method to get the activations for a batch of inputs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "model_truncated = Sequential()\n",
+ "model_truncated.add(Dense(512, activation='relu', input_shape=(784,)))\n",
+ "model_truncated.add(Dropout(0.2))\n",
+ "model_truncated.add(Dense(512, activation='relu'))\n",
+ "\n",
+ "for i, layer in enumerate(model_truncated.layers):\n",
+ " layer.set_weights(model.layers[i].get_weights())\n",
+ "\n",
+ "model_truncated.compile(loss='categorical_crossentropy', optimizer=SGD(), \n",
+ " metrics=['accuracy'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Check\n",
+ "np.all(model_truncated.layers[0].get_weights()[0] == model.layers[0].get_weights()[0])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "hidden_features = model_truncated.predict(X_train)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(45000, 512)"
+ ]
+ },
+ "execution_count": 36,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "hidden_features.shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(45000, 784)"
+ ]
+ },
+ "execution_count": 37,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "X_train.shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.manifold import TSNE\n",
+ "\n",
+ "tsne = TSNE(n_components=2)\n",
+ "X_tsne = tsne.fit_transform(hidden_features) "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "colors = np.ar"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "colors = 'bgrcmykpoa'\n",
+ "color_classes = np.argmax()\n",
+ "plt.scatter(X_tsne[:,0], X_tsne[:, 1], c=colors[Y_train])"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb b/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb
deleted file mode 100644
index feb5bc0..0000000
--- a/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb
+++ /dev/null
@@ -1,254 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "# Fully Connected Feed-Forward Network\n",
- "\n",
- "In this notebook we will play with Feed-Forward FC-NN (Fully Connected Neural Network) for a *classification task*: Image Classification on MNIST Dataset\n",
- "\n",
- "**RECALL**\n",
- "\n",
- "In the FC-NN, the output of each layer is computed using the activations from the previous one, as follows:\n",
- "\n",
- "$$h_{i} = \\sigma(W_i h_{i-1} + b_i)$$\n",
- "\n",
- "where ${h}_i$ is the activation vector from the $i$-th layer (or the input data for $i=0$), ${W}_i$ and ${b}_i$ are the weight matrix and the bias vector for the $i$-th layer, respectively. \n",
- "
\n",
- "$\\sigma(\\cdot)$ is the activation function. In our example, we will use the *ReLU* activation function for the hidden layers and *softmax* for the last layer.\n",
- "\n",
- "To regularize the model, we will also insert a Dropout layer between consecutive hidden layers. \n",
- "\n",
- "Dropout works by “dropping out” some unit activations in a given layer, that is setting them to zero with a given probability.\n",
- "\n",
- "Our loss function will be the **categorical crossentropy**."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Model definition\n",
- "Keras supports two different kind of models: the [Sequential](http://keras.io/models/#sequential) model and the [Graph](http://keras.io/models/#graph) model. The former is used to build linear stacks of layer (so each layer has one input and one output), and the latter supports any kind of connection graph.\n",
- "\n",
- "In our case we build a Sequential model with three [Dense](http://keras.io/layers/core/#dense) (aka fully connected) layers, with some [Dropout](http://keras.io/layers/core/#dropout). Notice that the output layer has the softmax activation function. \n",
- "\n",
- "The resulting model is actually a `function` of its own inputs implemented using the Keras backend. \n",
- "\n",
- "We apply the binary crossentropy loss and choose SGD as the optimizer. \n",
- "\n",
- "Please remind that Keras supports a variety of different [optimizers](http://keras.io/optimizers/) and [loss functions](http://keras.io/objectives/), which you may want to check out. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- }
- ],
- "source": [
- "from keras.models import Sequential\n",
- "from keras.layers.core import Dense, Dropout\n",
- "from keras.optimizers import SGD\n",
- "\n",
- "nb_classes = 10\n",
- "\n",
- "# FC@512+relu -> DropOut(0.2) -> FC@512+relu -> DropOut(0.2) -> FC@nb_classes+softmax\n",
- "# ... your Code Here"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "# Decomment and Execute this cell to get the solution\n",
- "# %load solutions/sol_221_1.py"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Data preparation (`keras.dataset`)\n",
- "\n",
- "We will train our model on the MNIST dataset, which consists of 60,000 28x28 grayscale images of the 10 digits, along with a test set of 10,000 images. \n",
- "\n",
- "\n",
- "\n",
- "Since this dataset is **provided** with Keras, we just ask the `keras.dataset` model for training and test data.\n",
- "\n",
- "We will:\n",
- "\n",
- "* download the data\n",
- "* reshape data to be in vectorial form (original data are images)\n",
- "* normalize between 0 and 1.\n",
- "\n",
- "The `binary_crossentropy` loss expects a **one-hot-vector** as input, therefore we apply the `to_categorical` function from `keras.utilis` to convert integer labels to **one-hot-vectors**."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from keras.datasets import mnist\n",
- "from keras.utils import np_utils\n",
- "\n",
- "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
- "X_train = X_train.reshape(60000, 784)\n",
- "X_test = X_test.reshape(10000, 784)\n",
- "X_train = X_train.astype(\"float32\")\n",
- "X_test = X_test.astype(\"float32\")\n",
- "X_train /= 255\n",
- "X_test /= 255\n",
- "\n",
- "# convert class vectors to binary class matrices\n",
- "Y_train = np_utils.to_categorical(y_train, 10)\n",
- "Y_test = np_utils.to_categorical(y_test, 10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Training\n",
- "Having defined and compiled the model, it can be trained using the `fit` function. We also specify a validation dataset to monitor validation loss and accuracy."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "scrolled": true
- },
- "outputs": [],
- "source": [
- "network_history = model.fit(X_train, Y_train, batch_size=128, \n",
- " epochs=100, verbose=1, validation_data=(X_test, Y_test))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "### Plotting Network Performance Trend\n",
- "The return value of the `fit` function is a `keras.callbacks.History` object which contains the entire history of training/validation loss and accuracy, for each epoch. We can therefore plot the behaviour of loss and accuracy during the training phase."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "%matplotlib inline\n",
- "\n",
- "plt.figure()\n",
- "plt.xlabel('Epochs')\n",
- "plt.ylabel('Loss')\n",
- "plt.plot(network_history.history['loss'])\n",
- "plt.plot(network_history.history['val_loss'])\n",
- "plt.legend(['Training', 'Validation'])\n",
- "\n",
- "plt.figure()\n",
- "plt.xlabel('Epochs')\n",
- "plt.ylabel('Accuracy')\n",
- "plt.plot(network_history.history['acc'])\n",
- "plt.plot(network_history.history['val_acc'])\n",
- "plt.legend(['Training', 'Validation'], loc='lower right')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "After `100` epochs, we get a `98.8%` validation accuracy. \n",
- "\n",
- "If you continue training, at some point the validation loss will start to increase: that is when the model starts to **overfit**. \n",
- "\n",
- "It is always necessary to monitor training and validation loss during the training of any kind of Neural Network, either to detect overfitting or to evaluate the behaviour of the model **(any clue on how to do it??)**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# %load solutions/sol_221_2.py"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.5.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/2.1 Convolutional Neural Networks.ipynb b/Intro cnn.ipynb
similarity index 90%
rename from 2.1 Convolutional Neural Networks.ipynb
rename to Intro cnn.ipynb
index bc08ba0..53358e9 100644
--- a/2.1 Convolutional Neural Networks.ipynb
+++ b/Intro cnn.ipynb
@@ -3,8 +3,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -16,8 +14,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -31,8 +27,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -44,8 +38,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -59,8 +51,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -73,8 +63,6 @@
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "-"
}
@@ -88,8 +76,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -103,8 +89,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -118,8 +102,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -131,8 +113,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -143,10 +123,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/cnn1.png\")
\n",
"\n",
@@ -156,8 +133,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -173,8 +148,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -185,10 +158,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"What we want the computer to do is to be able to differentiate between all the images it’s given and figure out the unique features that make a dog a dog or that make a cat a cat. "
]
@@ -196,8 +166,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -211,8 +179,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -223,10 +189,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"> A more detailed overview of what CNNs do would be that you take the image, pass it through a series of convolutional, nonlinear, pooling (downsampling), and fully connected layers, and get an output. As we said earlier, the output can be a single class or a probability of classes that best describes the image. \n",
"\n",
@@ -236,8 +199,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -249,8 +210,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -261,10 +220,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"![](\"imgs/conv.png\")
"
@@ -273,8 +229,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -286,10 +240,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"A Convolutional Filter much like a **kernel** in image recognition is a small matrix useful for blurring, sharpening, embossing, edge detection, and more. \n",
"\n",
@@ -301,8 +252,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -314,10 +263,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/cnn2.png\")
"
]
@@ -325,8 +271,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -338,8 +282,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -351,8 +293,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -366,8 +306,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -379,8 +317,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -392,8 +328,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -404,10 +338,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/cnn4.png\")
"
]
@@ -415,8 +346,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -428,8 +357,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -441,8 +368,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -454,8 +379,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -467,8 +390,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -482,8 +403,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -495,8 +414,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -511,8 +428,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -524,8 +439,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -537,8 +450,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -556,8 +467,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -571,8 +480,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -594,8 +501,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -607,8 +512,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -620,8 +523,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -633,8 +534,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -645,10 +544,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/MaxPool.png\")
"
]
@@ -656,8 +552,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -669,8 +563,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -686,8 +578,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -701,8 +591,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -714,8 +602,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -729,8 +615,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -742,8 +626,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -755,8 +637,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -767,10 +647,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/ConvNet)
"
]
@@ -801,8 +678,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -814,8 +689,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -838,8 +711,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -905,8 +776,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -918,8 +787,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -944,8 +811,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1023,8 +888,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1051,8 +914,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1064,8 +925,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1077,8 +936,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1090,8 +947,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1111,8 +966,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1162,9 +1015,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/solutions/sol11.py b/solutions/sol11.py
new file mode 100644
index 0000000..e957570
--- /dev/null
+++ b/solutions/sol11.py
@@ -0,0 +1,2 @@
+y = w * x + b
+loss = K.mean(K.square(y-target))
\ No newline at end of file
diff --git a/solutions/sol12.py b/solutions/sol12.py
new file mode 100644
index 0000000..70ad4a6
--- /dev/null
+++ b/solutions/sol12.py
@@ -0,0 +1,2 @@
+grads = K.gradients(loss, [w,b])
+updates = [(w, w-lr*grads[0]), (b, b-lr*grads[1])]
\ No newline at end of file
diff --git a/solutions/sol13.py b/solutions/sol13.py
new file mode 100644
index 0000000..181d8e2
--- /dev/null
+++ b/solutions/sol13.py
@@ -0,0 +1,5 @@
+plt.plot(range(len(loss_history)), loss_history, 'o', label='Linear Regression Training phase')
+plt.ylabel('cost')
+plt.xlabel('epoch')
+plt.legend()
+plt.show()
\ No newline at end of file
diff --git a/solutions/sol21.py b/solutions/sol21.py
new file mode 100644
index 0000000..1f54912
--- /dev/null
+++ b/solutions/sol21.py
@@ -0,0 +1,11 @@
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout
+from keras.optimizers import SGD
+
+model = Sequential()
+model.add(Dense(512, activation='relu', input_shape=(784,)))
+model.add(Dense(512, activation='relu'))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer=SGD(),
+ metrics=['accuracy'])
\ No newline at end of file
diff --git a/solutions/sol_221_1.py b/solutions/sol22.py
similarity index 100%
rename from solutions/sol_221_1.py
rename to solutions/sol22.py
diff --git a/solutions/sol_221_2.py b/solutions/sol23.py
similarity index 100%
rename from solutions/sol_221_2.py
rename to solutions/sol23.py
diff --git a/1.1 Introduction - Deep Learning and ANN.ipynb b/to remove/1.1 Introduction - Deep Learning and ANN.ipynb
similarity index 98%
rename from 1.1 Introduction - Deep Learning and ANN.ipynb
rename to to remove/1.1 Introduction - Deep Learning and ANN.ipynb
index 19e169d..c75b7e0 100644
--- a/1.1 Introduction - Deep Learning and ANN.ipynb
+++ b/to remove/1.1 Introduction - Deep Learning and ANN.ipynb
@@ -3,8 +3,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "24ca50a0-b9ad-425d-a28f-6bd3ca3ac378"
},
@@ -19,8 +17,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "3896ecb2-47fa-4dbc-a2ff-9c76e9dfe93e"
},
@@ -35,8 +31,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "8c3060aa-fee9-438c-bc60-4685c0eb4750"
},
@@ -51,8 +45,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "6287766f-972f-4b4d-bee7-5418f0af74de"
},
@@ -67,8 +59,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "2f1c6299-954a-461a-b5c1-a86a1d12ad15"
},
@@ -83,8 +73,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -101,8 +89,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "5e13607b-3ec5-4a95-a2d8-f898f20748da"
},
@@ -117,8 +103,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "4fa2e86a-be32-4e78-96d9-f511a07e3908"
},
@@ -133,8 +117,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "df0121bc-10f1-4ace-840e-6fc89c6fdc7f"
},
@@ -149,8 +131,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "c25d7194-10bd-4196-9d4c-592bf6e188f9"
},
@@ -165,8 +145,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "15260f90-13d1-4fcc-afc6-379c507cb950"
},
@@ -181,8 +159,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "92d4603e-7e39-4156-818c-785df6189fe8"
},
@@ -198,8 +174,6 @@
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "356d5ec7-3392-4daa-9671-4cc7111c5c91"
},
@@ -216,8 +190,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -228,10 +200,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
""
]
@@ -239,8 +208,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -251,10 +218,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/single_layer.png\")
\n",
"\n",
@@ -271,8 +235,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -304,10 +266,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"In **gradient descent optimization**, we update all the **weights simultaneously** after each epoch, and we define the _partial derivative_ for each weight $w_j$ in the weight vector $w$ as follows:\n",
"\n",
@@ -359,8 +318,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -372,8 +329,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -556,8 +511,6 @@
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "5678486b-caf4-440b-be62-2f1286982c71"
},
@@ -581,8 +534,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -593,10 +544,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
""
]
@@ -604,8 +552,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -619,8 +565,6 @@
"execution_count": 1,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -640,8 +584,6 @@
"execution_count": 2,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -659,8 +601,6 @@
"execution_count": 3,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -676,8 +616,6 @@
"execution_count": 4,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -691,9 +629,6 @@
"cell_type": "code",
"execution_count": 5,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -718,9 +653,6 @@
"cell_type": "code",
"execution_count": 6,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -745,9 +677,6 @@
"cell_type": "code",
"execution_count": 7,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -783,8 +712,6 @@
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -798,8 +725,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -810,10 +735,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/mlp_details.png\")
"
]
@@ -823,8 +745,6 @@
"execution_count": 8,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -842,8 +762,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -859,8 +777,6 @@
"execution_count": 9,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -875,8 +791,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -889,9 +803,7 @@
"cell_type": "code",
"execution_count": 10,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -904,8 +816,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -920,9 +830,7 @@
"cell_type": "code",
"execution_count": 11,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -934,8 +842,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -951,8 +857,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -988,8 +892,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1000,10 +902,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"def activate(self, inputs):\n",
@@ -1037,8 +936,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1049,10 +946,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"def backPropagate(self, targets, N, M):\n",
@@ -1103,9 +997,6 @@
"cell_type": "code",
"execution_count": 12,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1236,8 +1127,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1249,11 +1138,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1276,8 +1161,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1289,11 +1172,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1311,9 +1190,6 @@
"cell_type": "code",
"execution_count": 15,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1385,9 +1261,6 @@
"cell_type": "code",
"execution_count": 16,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1411,8 +1284,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1426,8 +1297,6 @@
"execution_count": 17,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -1458,11 +1327,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -1493,8 +1358,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1511,9 +1374,7 @@
"cell_type": "code",
"execution_count": 19,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1525,9 +1386,6 @@
"cell_type": "code",
"execution_count": 20,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1540,8 +1398,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1558,9 +1414,7 @@
"cell_type": "code",
"execution_count": 21,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1571,9 +1425,6 @@
"cell_type": "code",
"execution_count": 22,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
@@ -1587,8 +1438,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1619,7 +1468,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
},
"nbpresent": {
"slides": {
@@ -1944,5 +1793,5 @@
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/1.2 Introduction - Tensorflow.ipynb b/to remove/1.2 Introduction - Tensorflow.ipynb
similarity index 94%
rename from 1.2 Introduction - Tensorflow.ipynb
rename to to remove/1.2 Introduction - Tensorflow.ipynb
index 8629f51..c631dff 100644
--- a/1.2 Introduction - Tensorflow.ipynb
+++ b/to remove/1.2 Introduction - Tensorflow.ipynb
@@ -3,8 +3,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -16,8 +14,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -34,8 +30,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -46,10 +40,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"* Defining, optimizing, and efficiently calculating mathematical expressions involving multi-dimensional arrays (tensors).\n",
"\n",
@@ -63,8 +54,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -76,8 +65,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -90,9 +77,6 @@
"cell_type": "code",
"execution_count": 1,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -117,9 +101,6 @@
"cell_type": "code",
"execution_count": 2,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -133,9 +114,6 @@
"cell_type": "code",
"execution_count": 3,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -159,8 +137,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -172,8 +148,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -188,9 +162,6 @@
"cell_type": "code",
"execution_count": 4,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -204,9 +175,6 @@
"cell_type": "code",
"execution_count": 5,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -229,8 +197,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -242,8 +208,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -257,8 +221,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -273,8 +235,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -285,10 +245,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"* **Node**: In TensorFlow, each node represents the instantion of an operation. \n",
" - Each operation has inputs (`>= 2`) and outputs `>= 0`.\n",
@@ -303,8 +260,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -315,10 +270,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"* **Operation**: This represents an abstract computation, such as adding or multiplying matrices. \n",
" - An operation manages tensors, and It can just be polymorphic: the same operation can manipulate different tensor element types. \n",
@@ -332,8 +284,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -344,10 +294,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"**Session**: When the client program has to establish communication with the TensorFlow runtime system, a session must be created. \n",
" \n",
@@ -361,8 +308,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -373,10 +318,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"**TensorBoard** is a visualization tool, devoted to analyzing Data Flow Graph and also to better understand the machine learning models. \n",
"\n",
@@ -386,8 +328,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -399,8 +339,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -411,10 +349,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```shell\n",
"tensorboard --logdir=/tmp/tf_logs\n",
@@ -424,8 +359,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -444,11 +377,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -479,8 +408,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -492,8 +419,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -506,9 +431,6 @@
"cell_type": "code",
"execution_count": 7,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -524,9 +446,6 @@
"cell_type": "code",
"execution_count": 8,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -552,8 +471,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -566,9 +483,6 @@
"cell_type": "code",
"execution_count": 9,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -600,8 +514,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -614,9 +526,6 @@
"cell_type": "code",
"execution_count": 10,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "-"
}
@@ -630,11 +539,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"tf_mat1 = tf.constant(matrix1) \n",
@@ -645,9 +550,6 @@
"cell_type": "code",
"execution_count": 12,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -662,9 +564,6 @@
"cell_type": "code",
"execution_count": 13,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -679,8 +578,6 @@
"execution_count": 14,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -697,9 +594,6 @@
"cell_type": "code",
"execution_count": 15,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -731,8 +625,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -755,11 +647,7 @@
{
"cell_type": "code",
"execution_count": 17,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import matplotlib.image as mp_image\n",
@@ -771,9 +659,6 @@
"cell_type": "code",
"execution_count": 18,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -799,9 +684,6 @@
"cell_type": "code",
"execution_count": 19,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -826,10 +708,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"### Slicing"
]
@@ -839,8 +718,6 @@
"execution_count": 20,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "-"
}
@@ -855,9 +732,6 @@
"cell_type": "code",
"execution_count": 21,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -881,9 +755,6 @@
"cell_type": "code",
"execution_count": 22,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -908,8 +779,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -921,11 +790,7 @@
{
"cell_type": "code",
"execution_count": 23,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"x = tf.Variable(input_image,name='x')\n",
@@ -941,9 +806,6 @@
"cell_type": "code",
"execution_count": 24,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -968,8 +830,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -983,11 +843,7 @@
{
"cell_type": "code",
"execution_count": 25,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1009,8 +865,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -1023,9 +877,6 @@
"cell_type": "code",
"execution_count": 26,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -1067,8 +918,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1079,10 +928,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"This algorithm has nothing to do with the canonical linear regression, but it is an algorithm that allows us to solve problems of classification(supervised learning). \n",
"\n",
@@ -1103,9 +949,7 @@
{
"cell_type": "code",
"execution_count": 27,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"name": "stderr",
@@ -1122,9 +966,7 @@
{
"cell_type": "code",
"execution_count": 29,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1154,9 +996,7 @@
{
"cell_type": "code",
"execution_count": 30,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -1177,8 +1017,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1192,8 +1030,6 @@
"execution_count": 31,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1211,8 +1047,6 @@
"execution_count": 32,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1226,10 +1060,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Model"
]
@@ -1238,9 +1069,7 @@
"cell_type": "code",
"execution_count": 33,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1254,8 +1083,6 @@
"execution_count": 34,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1271,8 +1098,6 @@
"execution_count": 35,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1289,8 +1114,6 @@
"execution_count": 36,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1305,9 +1128,6 @@
"cell_type": "code",
"execution_count": 37,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1321,8 +1141,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1335,9 +1153,7 @@
"cell_type": "code",
"execution_count": 38,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1353,8 +1169,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1367,9 +1181,7 @@
"cell_type": "code",
"execution_count": 39,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1384,8 +1196,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1397,11 +1207,7 @@
{
"cell_type": "code",
"execution_count": 40,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1453,8 +1259,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1466,8 +1270,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -1478,10 +1280,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"On a typical system, there are multiple computing devices. \n",
"\n",
@@ -1497,8 +1296,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1512,8 +1309,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1526,10 +1321,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"# Creates a graph.\n",
@@ -1546,8 +1338,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1568,8 +1358,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1580,10 +1368,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"# Creates a graph.\n",
@@ -1605,8 +1390,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1632,10 +1415,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on Tensorflow\n",
"\n",
@@ -1659,9 +1439,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/additional materials/1.5.1 Introduction - Theano.ipynb b/to remove/1.5.1 Introduction - Theano.ipynb
similarity index 95%
rename from additional materials/1.5.1 Introduction - Theano.ipynb
rename to to remove/1.5.1 Introduction - Theano.ipynb
index fabb440..6631871 100644
--- a/additional materials/1.5.1 Introduction - Theano.ipynb
+++ b/to remove/1.5.1 Introduction - Theano.ipynb
@@ -27,7 +27,6 @@
"cell_type": "code",
"execution_count": 1,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -61,7 +60,6 @@
"cell_type": "code",
"execution_count": 2,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -75,7 +73,6 @@
"cell_type": "code",
"execution_count": 3,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -111,7 +108,6 @@
"cell_type": "code",
"execution_count": 4,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "-"
}
@@ -147,7 +143,6 @@
"cell_type": "code",
"execution_count": 5,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "-"
}
@@ -195,7 +190,6 @@
"cell_type": "code",
"execution_count": 6,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -217,7 +211,6 @@
"cell_type": "code",
"execution_count": 7,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -242,7 +235,6 @@
"cell_type": "code",
"execution_count": 8,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -283,9 +275,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -328,7 +318,6 @@
"cell_type": "code",
"execution_count": 11,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -394,9 +383,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [],
"source": [
"x = T.scalar()\n",
@@ -407,7 +394,6 @@
"cell_type": "code",
"execution_count": 14,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -446,9 +432,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -458,9 +442,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -492,7 +474,6 @@
"cell_type": "code",
"execution_count": 17,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "-"
}
@@ -517,9 +498,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [],
"source": [
"x.set_value(values)"
@@ -540,7 +519,6 @@
"cell_type": "code",
"execution_count": 19,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "-"
}
@@ -575,9 +553,7 @@
{
"cell_type": "code",
"execution_count": 20,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -598,9 +574,7 @@
{
"cell_type": "code",
"execution_count": 21,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -636,7 +610,6 @@
"cell_type": "code",
"execution_count": 22,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "slide"
}
@@ -654,7 +627,6 @@
"cell_type": "code",
"execution_count": 23,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -679,7 +651,6 @@
"cell_type": "code",
"execution_count": 24,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -704,7 +675,6 @@
"cell_type": "code",
"execution_count": 25,
"metadata": {
- "collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
@@ -746,9 +716,7 @@
{
"cell_type": "code",
"execution_count": 27,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -788,9 +756,7 @@
{
"cell_type": "code",
"execution_count": 28,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [],
"source": [
"import os\n",
@@ -803,9 +769,7 @@
{
"cell_type": "code",
"execution_count": 29,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"name": "stderr",
@@ -822,9 +786,7 @@
{
"cell_type": "code",
"execution_count": 30,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -882,9 +844,7 @@
{
"cell_type": "code",
"execution_count": 31,
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -981,9 +941,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}