diff --git a/0. Preamble.ipynb b/0. Preamble.ipynb index 3547455..19b5a52 100644 --- a/0. Preamble.ipynb +++ b/0. Preamble.ipynb @@ -3,57 +3,32 @@ { "cell_type": "markdown", "metadata": { - "slideshow": { - "slide_type": "-" - } + "deletable": true, + "editable": true }, "source": [ - "
\n", - "

Deep Learning with Keras

\n", - " \n", - "
\n", - "\n", - "
\n", - "

Tutorial @ EuroScipy 2016

\n", - " \n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### Yam Peleg, Valerio Maggio" + "# Deep Learning Tutorial with Keras and Tensorflow" ] }, { "cell_type": "markdown", "metadata": { - "slideshow": { - "slide_type": "-" - } + "deletable": true, + "editable": true }, "source": [ - "# Goal of this Tutorial\n", - "\n", - "- **Introduce** main features of Keras\n", - "- **Learn** how simple and Pythonic is doing Deep Learning with Keras\n", - "- **Understand** how easy is to do basic and *advanced* DL models in Keras;\n", - " - **Examples and Hand-on Excerises** along the way." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Source\n", + "```shell\n", "\n", - "https://github.com/leriomaggio/deep-learning-keras-euroscipy2016/" + "git clone -b pydatait https://github.com/leriomaggio/deep-learning-keras-tensorflow.git\n", + "```" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---" ] @@ -61,105 +36,124 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "source": [ - "# (Tentative) Schedule \n", - "\n", - "## Attention: Spoilers Warning!\n", + "# Tentative Outline\n", "\n", + "- **Warmup**\n", "\n", - "- **Setup** (`10 mins`)\n", + "- **Part I**: **Introduction**\n", "\n", - "- **Part I**: **Introduction** (`~65 mins`)\n", - "\n", - " - Intro to ANN (`~20 mins`)\n", + " - Intro to ANN\n", " - naive pure-Python implementation\n", " - fast forward, sgd, backprop\n", " \n", - " - Intro to Theano (`15 mins`)\n", - " - Model + SGD with Theano\n", - " \n", - " - Introduction to Keras (`30 mins`)\n", + " - Intro to Tensorflow \n", + " - Model + SGD with Tensorflow\n", + " \n", + " - Introduction to Keras\n", " - Overview and main features\n", - " - Theano backend\n", " - Tensorflow backend\n", + " - Theano backend\n", + " - Keras Backend\n", + " - Overview of the main layers\n", " - Multi-Layer Perceptron and Fully Connected\n", " - Examples with `keras.models.Sequential` and `Dense`\n", - " - HandsOn: MLP with keras\n", + " - HandsOn: FC with keras\n", + " \n", + " - Extra Material:\n", + " - Intro to Theano\n", + " - Alternative ANN implementation for MNIST\n", + " \n", " \n", - "- **Coffe Break** (`30 mins`)\n", + "- **Break**\n", "\n", - "- **Part II**: **Supervised Learning and Convolutional Neural Nets** (`~45 mins`)\n", + "- **Part II**: **Supervised Learning and Convolutional Neural Nets**\n", " \n", - " - Intro: Focus on Image Classification (`5 mins`)\n", + " - Intro: Focus on Image Classification\n", "\n", - " - Intro to CNN (`25 mins`)\n", + " - Intro to ConvNets\n", " - meaning of convolutional filters\n", " - examples from ImageNet \n", " - Meaning of dimensions of Conv filters (through an exmple of ConvNet) \n", " - Visualising ConvNets\n", " - HandsOn: ConvNet with keras \n", "\n", - " - Advanced CNN (`10 mins`)\n", + " - Advanced CNN\n", " - Dropout\n", " - MaxPooling\n", " - Batch Normalisation\n", " \n", - " - Famous Models in Keras (likely moved somewhere else) (`10 mins`)\n", - " (ref: https://github.com/fchollet/deep-learning-models)\n", + " - Famous Models in Keras\n", + " (ref: `keras.applications`)\n", " - VGG16\n", " - VGG19\n", " - ResNet50\n", " - Inception v3\n", + " - Transfer Learning\n", " - HandsOn: Fine tuning a network on new dataset \n", " \n", - "- **Part III**: **Unsupervised Learning** (`10 mins`)\n", + "- **Part III**: **Unsupervised Learning**\n", "\n", " - AutoEncoders (`5 mins`)\n", - " - word2vec & doc2vec (gensim) & `keras.datasets` (`5 mins`)\n", + " - word2vec & doc2vec (gensim) & `keras.datasets`\n", " - `Embedding`\n", " - word2vec and CNN\n", " - Exercises\n", "\n", - "- **Part IV**: **Advanced Materials** (`20 mins`)\n", + "- **Part IV**: **Advanced Materials**\n", " - RNN and LSTM (`10 mins`)\n", " - RNN, LSTM, GRU \n", - " - Example of RNN and LSTM with Text (`~10 mins`) -- *Tentative*\n", + " - Example of RNN and LSTM with Text\n", " - HandsOn: IMDB\n", + " - **Multi-Input/Multi-Output Network Topologies**\n", "\n", - "- **Wrap up and Conclusions** (`5 mins`)" + "- **Wrap up and Conclusions**" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "# Requirements" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "This tutorial requires the following packages:\n", "\n", - "- Python version 3.4+ \n", - " - likely Python 2.7 would be fine, but *who knows*? :P\n", + "- Python version 3.5.x\n", + " - Python 3.4 should be fine as well\n", + " - likely Python 2.7 would be also fine, but *who knows*? :P\n", + " \n", "- `numpy` version 1.10 or later: http://www.numpy.org/\n", "- `scipy` version 0.16 or later: http://www.scipy.org/\n", "- `matplotlib` version 1.4 or later: http://matplotlib.org/\n", "- `pandas` version 0.16 or later: http://pandas.pydata.org\n", "- `scikit-learn` version 0.15 or later: http://scikit-learn.org\n", - "- `keras` version 1.0 or later: http://keras.io\n", - "- `theano` version 0.8 or later: http://deeplearning.net/software/theano/\n", + "- `keras` version 2.0 or later: http://keras.io\n", + "- `tensorflow` version 1.0 or later: https://www.tensorflow.org\n", "- `ipython`/`jupyter` version 4.0 or later, with notebook support\n", "\n", "(Optional but recommended):\n", @@ -174,21 +168,30 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "### Python Version" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "I'm currently running this tutorial with **Python 3** on **Anaconda**" ] @@ -197,7 +200,9 @@ "cell_type": "code", "execution_count": 1, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -214,347 +219,94 @@ }, { "cell_type": "markdown", - "metadata": {}, - "source": [ - "# How to set up your environment" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ - "The quickest and simplest way to setup the environment is to use [conda](https://store.continuum.io) environment manager. \n", + "### Configure Keras with tensorflow\n", + "\n", + "1) Create the `keras.json` (if it does not exist):\n", + "\n", + "```shell\n", + "touch $HOME/.keras/keras.json\n", + "```\n", + "\n", + "2) Copy the following content into the file:\n", "\n", - "We provide in the materials a `deep-learning.yml` that is complete and **ready to use** to set up your virtual environment with conda." + "```\n", + "{\n", + " \"epsilon\": 1e-07,\n", + " \"backend\": \"tensorflow\",\n", + " \"floatx\": \"float32\",\n", + " \"image_data_format\": \"channels_last\"\n", + "}\n", + "```" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "metadata": { "collapsed": false, - "scrolled": false + "deletable": true, + "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "name: deep-learning\r\n", - "channels:\r\n", - "- conda-forge\r\n", - "- defaults\r\n", - "dependencies:\r\n", - "- accelerate=2.3.0=np111py35_3\r\n", - "- accelerate_cudalib=2.0=0\r\n", - "- bokeh=0.12.1=py35_0\r\n", - "- cffi=1.6.0=py35_0\r\n", - "- backports.shutil_get_terminal_size=1.0.0=py35_0\r\n", - "- blas=1.1=openblas\r\n", - "- ca-certificates=2016.8.2=3\r\n", - "- cairo=1.12.18=8\r\n", - "- certifi=2016.8.2=py35_0\r\n", - "- cycler=0.10.0=py35_0\r\n", - "- cython=0.24.1=py35_0\r\n", - "- decorator=4.0.10=py35_0\r\n", - "- entrypoints=0.2.2=py35_0\r\n", - "- fontconfig=2.11.1=3\r\n", - "- freetype=2.6.3=1\r\n", - "- gettext=0.19.7=1\r\n", - "- glib=2.48.0=4\r\n", - "- h5py=2.6.0=np111py35_6\r\n", - "- harfbuzz=1.0.6=0\r\n", - "- hdf5=1.8.17=2\r\n", - "- icu=56.1=4\r\n", - "- ipykernel=4.3.1=py35_1\r\n", - "- ipython=5.1.0=py35_0\r\n", - "- ipywidgets=5.2.2=py35_0\r\n", - "- jinja2=2.8=py35_1\r\n", - "- jpeg=9b=0\r\n", - "- jsonschema=2.5.1=py35_0\r\n", - "- jupyter_client=4.3.0=py35_0\r\n", - "- jupyter_console=5.0.0=py35_0\r\n", - "- jupyter_core=4.1.1=py35_1\r\n", - "- libffi=3.2.1=2\r\n", - "- libiconv=1.14=3\r\n", - "- libpng=1.6.24=0\r\n", - "- libsodium=1.0.10=0\r\n", - "- libtiff=4.0.6=6\r\n", - "- libxml2=2.9.4=0\r\n", - "- markupsafe=0.23=py35_0\r\n", - "- matplotlib=1.5.2=np111py35_6\r\n", - "- mistune=0.7.3=py35_0\r\n", - "- nbconvert=4.2.0=py35_0\r\n", - "- nbformat=4.0.1=py35_0\r\n", - "- ncurses=5.9=8\r\n", - "- nose=1.3.7=py35_1\r\n", - "- notebook=4.2.2=py35_0\r\n", - "- numpy=1.11.1=py35_blas_openblas_201\r\n", - "- openblas=0.2.18=4\r\n", - "- openssl=1.0.2h=2\r\n", - "- pandas=0.18.1=np111py35_1\r\n", - "- pango=1.40.1=0\r\n", - "- path.py=8.2.1=py35_0\r\n", - "- pcre=8.38=1\r\n", - "- pexpect=4.2.0=py35_1\r\n", - "- pickleshare=0.7.3=py35_0\r\n", - "- pip=8.1.2=py35_0\r\n", - "- pixman=0.32.6=0\r\n", - "- prompt_toolkit=1.0.6=py35_0\r\n", - "- protobuf=3.0.0b3=py35_1\r\n", - "- ptyprocess=0.5.1=py35_0\r\n", - "- pygments=2.1.3=py35_1\r\n", - "- pyparsing=2.1.7=py35_0\r\n", - "- python=3.5.2=2\r\n", - "- python-dateutil=2.5.3=py35_0\r\n", - "- pytz=2016.6.1=py35_0\r\n", - "- pyyaml=3.11=py35_0\r\n", - "- pyzmq=15.4.0=py35_0\r\n", - "- qt=4.8.7=0\r\n", - "- qtconsole=4.2.1=py35_0\r\n", - "- readline=6.2=0\r\n", - "- requests=2.11.0=py35_0\r\n", - "- scikit-learn=0.17.1=np111py35_blas_openblas_201\r\n", - "- scipy=0.18.0=np111py35_blas_openblas_201\r\n", - "- setuptools=25.1.6=py35_0\r\n", - "- simplegeneric=0.8.1=py35_0\r\n", - "- sip=4.18=py35_0\r\n", - "- six=1.10.0=py35_0\r\n", - "- sqlite=3.13.0=1\r\n", - "- terminado=0.6=py35_0\r\n", - "- tk=8.5.19=0\r\n", - "- tornado=4.4.1=py35_1\r\n", - "- traitlets=4.2.2=py35_0\r\n", - "- wcwidth=0.1.7=py35_0\r\n", - "- wheel=0.29.0=py35_0\r\n", - "- widgetsnbextension=1.2.6=py35_3\r\n", - "- xz=5.2.2=0\r\n", - "- yaml=0.1.6=0\r\n", - "- zeromq=4.1.5=0\r\n", - "- zlib=1.2.8=3\r\n", - "- cudatoolkit=7.5=0\r\n", - "- ipython_genutils=0.1.0=py35_0\r\n", - "- jupyter=1.0.0=py35_3\r\n", - "- libgfortran=3.0.0=1\r\n", - "- llvmlite=0.11.0=py35_0\r\n", - "- mkl=11.3.3=0\r\n", - "- mkl-service=1.1.2=py35_2\r\n", - "- numba=0.26.0=np111py35_0\r\n", - "- pycparser=2.14=py35_1\r\n", - "- pyqt=4.11.4=py35_4\r\n", - "- snakeviz=0.4.1=py35_0\r\n", - "- pip:\r\n", - " - backports.shutil-get-terminal-size==1.0.0\r\n", - " - certifi==2016.8.2\r\n", - " - cycler==0.10.0\r\n", - " - cython==0.24.1\r\n", - " - decorator==4.0.10\r\n", - " - h5py==2.6.0\r\n", - " - ipykernel==4.3.1\r\n", - " - ipython==5.1.0\r\n", - " - ipython-genutils==0.1.0\r\n", - " - ipywidgets==5.2.2\r\n", - " - jinja2==2.8\r\n", - " - jsonschema==2.5.1\r\n", - " - jupyter-client==4.3.0\r\n", - " - jupyter-console==5.0.0\r\n", - " - jupyter-core==4.1.1\r\n", - " - keras==1.0.7\r\n", - " - mako==1.0.4\r\n", - " - markupsafe==0.23\r\n", - " - matplotlib==1.5.2\r\n", - " - mistune==0.7.3\r\n", - " - nbconvert==4.2.0\r\n", - " - nbformat==4.0.1\r\n", - " - nose==1.3.7\r\n", - " - notebook==4.2.2\r\n", - " - numpy==1.11.1\r\n", - " - pandas==0.18.1\r\n", - " - path.py==8.2.1\r\n", - " - pexpect==4.2.0\r\n", - " - pickleshare==0.7.3\r\n", - " - pip==8.1.2\r\n", - " - prompt-toolkit==1.0.6\r\n", - " - protobuf==3.0.0b2\r\n", - " - ptyprocess==0.5.1\r\n", - " - pygments==2.1.3\r\n", - " - pygpu==0.2.1\r\n", - " - pyparsing==2.1.7\r\n", - " - python-dateutil==2.5.3\r\n", - " - pytz==2016.6.1\r\n", - " - pyyaml==3.11\r\n", - " - pyzmq==15.4.0\r\n", - " - qtconsole==4.2.1\r\n", - " - requests==2.11.0\r\n", - " - scikit-learn==0.17.1\r\n", - " - scipy==0.18.0\r\n", - " - setuptools==25.1.4\r\n", - " - simplegeneric==0.8.1\r\n", - " - six==1.10.0\r\n", - " - tensorflow==0.10.0rc0\r\n", - " - terminado==0.6\r\n", - " - theano==0.8.2\r\n", - " - tornado==4.4.1\r\n", - " - traitlets==4.2.2\r\n", - " - wcwidth==0.1.7\r\n", - " - wheel==0.29.0\r\n", - " - widgetsnbextension==1.2.6\r\n", - "prefix: /home/valerio/anaconda3/envs/deep-learning\r\n", - "\r\n" + "{\r\n", + "\t\"epsilon\": 1e-07,\r\n", + "\t\"backend\": \"tensorflow\",\r\n", + "\t\"floatx\": \"float32\",\r\n", + "\t\"image_data_format\": \"channels_last\"\r\n", + "}" ] } ], "source": [ - "!cat deep-learning.yml" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Recreate the Conda Environment" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### A. Create the Environment\n", - "\n", - "```\n", - "conda env create -f deep-learning.yml # this file is for Linux channels.\n", - "```\n", - "\n", - "If you're using a **Mac OSX**, we also provided in the repo the conda file \n", - "that is compatible with `osx-channels`:\n", - "\n", - "```\n", - "conda env create -f deep-learning-osx.yml # this file is for OSX channels.\n", - "```\n", - "\n", - "#### B. Activate the new `deep-learning` Environment\n", - "\n", - "```\n", - "source activate deep-learning\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Optionals" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1. Enabling Conda-Forge" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is strongly suggested to enable [**conda forge**](https://conda-forge.github.io/) in your Anaconda installation.\n", - "\n", - "**Conda-Forge** is a github organisation containing repositories of conda recipies.\n", - "\n", - "To add `conda-forge` as an additional anaconda channel it is just required to type:\n", - "\n", - "```shell\n", - "conda config --add channels conda-forge\n", - "```" + "!cat ~/.keras/keras.json" ] }, { "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2. Configure Theano\n", - "\n", - "1) Create the `theanorc` file:\n", - "\n", - "```shell\n", - "touch $HOME/.theanorc\n", - "```\n", - "\n", - "2) Copy the following content into the file:\n", - "\n", - "```\n", - "[global]\n", - "floatX = float32\n", - "device = gpu # switch to cpu if no GPU is available on your machine\n", - "\n", - "[nvcc]\n", - "fastmath = True\n", - "\n", - "[lib]\n", - "cnmem=.90\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**More on [theano documentation](http://theano.readthedocs.io/en/latest/library/config.html)**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3. Installing Tensorflow as backend " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "```shell\n", - "# Ubuntu/Linux 64-bit, GPU enabled, Python 3.5\n", - "# Requires CUDA toolkit 7.5 and CuDNN v4. For other versions, see \"Install from sources\" below.\n", - "export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp35-cp35m-linux_x86_64.whl\n", - "\n", - "pip install --ignore-installed --upgrade $TF_BINARY_URL\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**More on [tensorflow documentation](https://www.tensorflow.org/versions/r0.10/get_started/os_setup.html)**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "# Test if everything is up&running" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## 1. Check import" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -567,17 +319,18 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "Using Theano backend.\n", - "Using gpu device 0: GeForce GTX 760 (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)\n" + "Using TensorFlow backend.\n" ] } ], @@ -587,16 +340,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## 2. Check installeded Versions" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -607,7 +365,7 @@ "scipy: 0.18.0\n", "matplotlib: 1.5.2\n", "iPython: 5.1.0\n", - "scikit-learn: 0.17.1\n" + "scikit-learn: 0.18\n" ] } ], @@ -632,16 +390,18 @@ "cell_type": "code", "execution_count": 6, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "keras: 1.0.7\n", - "Theano: 0.8.2\n", - "Tensorflow: 0.10.0rc0\n" + "keras: 2.0.2\n", + "Theano: 0.9.0\n", + "Tensorflow: 1.0.1\n" ] } ], @@ -649,17 +409,20 @@ "import keras\n", "print('keras: ', keras.__version__)\n", "\n", + "# optional\n", "import theano\n", "print('Theano: ', theano.__version__)\n", "\n", - "# optional\n", "import tensorflow as tf\n", "print('Tensorflow: ', tf.__version__)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "
\n", "

If everything worked till down here, you're ready to start!

" @@ -667,65 +430,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---\n" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Consulting Material" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You have two options to go through the material presented in this tutorial:\n", - "\n", - "* Read (and execute) the material as **iPython/Jupyter** notebooks\n", - "* (just) read the material as (HTML) slides" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the first case, all you need to do is just execute `ipython notebook` (or `jupyter notebook`) depending on the version of `iPython` you have installed on your machine\n", - "\n", - "(`jupyter` command works in case you have `iPython 4.0.x` installed)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the second case, you may simply convert the provided notebooks in `HTML` slides and see them into your browser\n", - "thanks to `nbconvert`.\n", - "\n", - "Thus, move to the folder where notebooks are stored and execute the following command:\n", - "\n", - " jupyter nbconvert --to slides ./*.ipynb --post serve\n", - " \n", - " \n", - "(Please substitute `jupyter` with `ipython` in the previous command if you have `iPython 3.x` installed on your machine)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## In case..." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "..you wanna do **both** (interactive and executable slides), I highly suggest to install the terrific `RISE` ipython notebook extension: [https://github.com/damianavila/RISE](https://github.com/damianavila/RISE)" - ] } ], "metadata": { diff --git a/1.1 Introduction - Deep Learning and ANN.ipynb b/1.1 Introduction - Deep Learning and ANN.ipynb index 9232a07..8639b59 100644 --- a/1.1 Introduction - Deep Learning and ANN.ipynb +++ b/1.1 Introduction - Deep Learning and ANN.ipynb @@ -3,6 +3,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "24ca50a0-b9ad-425d-a28f-6bd3ca3ac378" }, @@ -17,6 +19,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "3896ecb2-47fa-4dbc-a2ff-9c76e9dfe93e" }, @@ -31,6 +35,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "8c3060aa-fee9-438c-bc60-4685c0eb4750" }, @@ -45,6 +51,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "6287766f-972f-4b4d-bee7-5418f0af74de" }, @@ -59,6 +67,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "2f1c6299-954a-461a-b5c1-a86a1d12ad15" }, @@ -73,6 +83,26 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## This Section will cover:\n", + "\n", + "* Getting a conceptual understanding of multi-layer neural networks\n", + "* Training neural networks for image classification\n", + "* Implementing the powerful backpropagation algorithm\n", + "* Debugging neural network implementations" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "5e13607b-3ec5-4a95-a2d8-f898f20748da" }, @@ -81,12 +111,14 @@ } }, "source": [ - "# Artificial Neural Networks (ANN)" + "# Building Blocks: Artificial Neural Networks (ANN)" ] }, { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "4fa2e86a-be32-4e78-96d9-f511a07e3908" }, @@ -101,6 +133,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "df0121bc-10f1-4ace-840e-6fc89c6fdc7f" }, @@ -115,6 +149,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "c25d7194-10bd-4196-9d4c-592bf6e188f9" }, @@ -129,6 +165,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "15260f90-13d1-4fcc-afc6-379c507cb950" }, @@ -143,6 +181,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "nbpresent": { "id": "92d4603e-7e39-4156-818c-785df6189fe8" }, @@ -158,6 +198,8 @@ "cell_type": "markdown", "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "nbpresent": { "id": "356d5ec7-3392-4daa-9671-4cc7111c5c91" }, @@ -174,6 +216,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -184,7 +228,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "" ] @@ -192,50 +239,141 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true, - "nbpresent": { - "id": "5678486b-caf4-440b-be62-2f1286982c71" - }, + "deletable": true, + "editable": true, "slideshow": { - "slide_type": "subslide" + "slide_type": "slide" } }, "source": [ - "The weights of each neuron are learned by **gradient descent**, where each neuron's error is derived with respect to it's weight." + "# Single Layer Neural Network" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Weights Update Rule" ] }, { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } }, "source": [ - "Optimization is done for each layer with respect to the previous layer in a technique known as **BackPropagation**." + "- We use a **gradient descent** optimization algorithm to learn the _Weights Coefficients_ of the model.\n", + "

\n", + "- In every **epoch** (pass over the training set), we update the weight vector $w$ using the following update rule:\n", + "\n", + "$$\n", + "w = w + \\Delta w, \\text{where } \\Delta w = - \\eta \\nabla J(w)\n", + "$$\n", + "\n", + "

\n", + "\n", + "In other words, we computed the gradient based on the whole training set and updated the weights of the model by taking a step into the **opposite direction** of the gradient $ \\nabla J(w)$. \n", + "\n", + "In order to fin the **optimal weights of the model**, we optimized an objective function (e.g. the Sum of Squared Errors (SSE)) cost function $J(w)$. \n", + "\n", + "Furthermore, we multiply the gradient by a factor, the learning rate $\\eta$ , which we choose carefully to balance the **speed of learning** against the risk of overshooting the global minimum of the cost function." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "" + "### Gradient Descent" ] }, { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "In **gradient descent optimization**, we update all the **weights simultaneously** after each epoch, and we define the _partial derivative_ for each weight $w_j$ in the weight vector $w$ as follows:\n", + "\n", + "$$\n", + "\\frac{\\partial}{\\partial w_j} J(w) = \\sum_{i} ( y^{(i)} - a^{(i)} ) x^{(i)}_j\n", + "$$\n", + "\n", + "**Note**: _The superscript $(i)$ refers to the ith sample. The subscript $j$ refers to the jth dimension/feature_\n", + "\n", + "\n", + "Here $y^{(i)}$ is the target class label of a particular sample $x^{(i)}$ , and $a^{(i)}$ is the **activation** of the neuron \n", + "\n", + "(which is a linear function in the special case of _Perceptron_)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We define the **activation function** $\\phi(\\cdot)$ as follows:\n", + "\n", + "$$\n", + "\\phi(z) = z = a = \\sum_{j} w_j x_j = \\mathbf{w}^T \\mathbf{x}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Binary Classification" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "While we used the **activation** $\\phi(z)$ to compute the gradient update, we may use a **threshold function** _(Heaviside function)_ to squash the continuous-valued output into binary class labels for prediction:\n", + "\n", + "$$\n", + "\\hat{y} = \n", + "\\begin{cases}\n", + " 1 & \\text{if } \\phi(z) \\geq 0 \\\\\n", + " 0 & \\text{otherwise}\n", + "\\end{cases}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "slide" } }, "source": [ - "# Building Neural Nets from scratch \n" + "## Building Neural Nets from scratch \n" ] }, { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -253,9 +391,219 @@ "Libraries like `theano` have highly optimized code." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + " Perceptron and Adaline " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you want a sneak peek of alternate (production ready) implementation of _Perceptron_ for instance try:\n", + "```python\n", + "from sklearn.linear_model import Perceptron\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introducing the multi-layer neural network architecture" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we will see how to connect **multiple single neurons** to a **multi-layer feedforward neural network**; this special type of network is also called a **multi-layer perceptron** (MLP). \n", + "\n", + "The figure shows the concept of an **MLP** consisting of three layers: one _input_ layer, one _hidden_ layer, and one _output_ layer. \n", + "\n", + "The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer, respectively. \n", + "\n", + "If such a network has **more than one hidden layer**, we also call it a **deep artificial neural network**.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Notation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we denote the `ith` activation unit in the `lth` layer as $a_i^{(l)}$ , and the activation units $a_0^{(1)}$ and \n", + "$a_0^{(2)}$ are the **bias units**, respectively, which we set equal to $1$. \n", + "

\n", + "The _activation_ of the units in the **input layer** is just its input plus the bias unit:\n", + "\n", + "$$\n", + "\\mathbf{a}^{(1)} = [a_0^{(1)}, a_1^{(1)}, \\ldots, a_m^{(1)}]^T = [1, x_1^{(i)}, \\ldots, x_m^{(i)}]^T\n", + "$$\n", + "

\n", + "**Note**: $x_j^{(i)}$ refers to the jth feature/dimension of the ith sample" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Notes on Notation (usually) Adopted\n", + "\n", + "The terminology around the indices (subscripts and superscripts) may look a little bit confusing at first. \n", + "

\n", + "\n", + "You may wonder why we wrote $w_{j,k}^{(l)}$ and not $w_{k,j}^{(l)}$ to refer to \n", + "the **weight coefficient** that connects the *kth* unit in layer $l$ to the jth unit in layer $l+1$. \n", + "

\n", + "\n", + "What may seem a little bit quirky at first will make much more sense later when we **vectorize** the neural network representation. \n", + "

\n", + "\n", + "For example, we will summarize the weights that connect the input and hidden layer by a matrix \n", + "$$ W^{(1)} \\in \\mathbb{R}^{h×[m+1]}$$\n", + "\n", + "where $h$ is the number of hidden units and $m + 1$ is the number of hidden units plus bias unit. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Forward Propagation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* Starting at the input layer, we forward propagate the patterns of the training data through the network to generate an output.\n", + "\n", + "* Based on the network's output, we calculate the error that we want to minimize using a cost function that we will describe later.\n", + "\n", + "* We backpropagate the error, find its derivative with respect to each weight in the network, and update the model." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Sigmoid Activation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Backward Propagation" + ] + }, { "cell_type": "markdown", "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "nbpresent": { + "id": "5678486b-caf4-440b-be62-2f1286982c71" + }, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "The weights of each neuron are learned by **gradient descent**, where each neuron's error is derived with respect to it's weight." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "_(Source: Python Machine Learning, S. Raschka)_" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "Optimization is done for each layer with respect to the previous layer in a technique known as **BackPropagation**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "skip" } @@ -269,6 +617,8 @@ "execution_count": 1, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "skip" } @@ -288,13 +638,15 @@ "execution_count": 2, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ - "# Display plots inline \n", + "# Display plots in notebook \n", "%matplotlib inline\n", "# Define plot's default figure size\n", "matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)" @@ -305,6 +657,8 @@ "execution_count": 3, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "skip" } @@ -320,6 +674,8 @@ "execution_count": 4, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -335,6 +691,8 @@ "execution_count": 5, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -349,6 +707,8 @@ "execution_count": 6, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -374,6 +734,8 @@ "execution_count": 7, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -399,6 +761,8 @@ "execution_count": 8, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -434,6 +798,8 @@ "cell_type": "markdown", "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -447,6 +813,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -457,7 +825,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "" ] @@ -465,6 +836,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -480,6 +853,8 @@ "execution_count": 9, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -496,6 +871,8 @@ "execution_count": 10, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -510,6 +887,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -522,7 +901,9 @@ "cell_type": "code", "execution_count": 11, "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -535,6 +916,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -549,7 +932,9 @@ "cell_type": "code", "execution_count": 12, "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -561,6 +946,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -576,6 +963,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -611,6 +1000,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -621,7 +1012,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "```python\n", "def activate(self, inputs):\n", @@ -655,6 +1049,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -665,7 +1061,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "```python\n", "def backPropagate(self, targets, N, M):\n", @@ -717,6 +1116,8 @@ "execution_count": 13, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -847,6 +1248,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -859,7 +1262,9 @@ "cell_type": "code", "execution_count": 14, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -883,6 +1288,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -895,7 +1302,9 @@ "cell_type": "code", "execution_count": 15, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -915,6 +1324,8 @@ "execution_count": 16, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -987,6 +1398,8 @@ "execution_count": 17, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "fragment" } @@ -1010,6 +1423,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -1023,6 +1438,8 @@ "execution_count": 18, "metadata": { "collapsed": true, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "skip" } @@ -1054,7 +1471,9 @@ "cell_type": "code", "execution_count": 19, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -1086,6 +1505,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -1102,7 +1523,9 @@ "cell_type": "code", "execution_count": 20, "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -1115,6 +1538,8 @@ "execution_count": 21, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -1162,6 +1587,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -1178,7 +1605,9 @@ "cell_type": "code", "execution_count": 22, "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -1190,6 +1619,8 @@ "execution_count": 23, "metadata": { "collapsed": false, + "deletable": true, + "editable": true, "scrolled": false, "slideshow": { "slide_type": "subslide" @@ -1355,6 +1786,8 @@ { "cell_type": "markdown", "metadata": { + "deletable": true, + "editable": true, "slideshow": { "slide_type": "subslide" } @@ -1362,7 +1795,7 @@ "source": [ "# Addendum\n", "\n", - "There is an additional notebook in the repo, i.e. [A simple implementation of ANN for MNIST](1.4 (Extra) A Simple Implementation of ANN for MNIST.ipynb) for a *naive* implementation of **SGD** and **MLP** applied on **MNIST** dataset.\n", + "There is an additional notebook in the repo, i.e. [A simple implementation of ANN for MNIST](1.1.2 MLP and MNIST.ipynb) for a *naive* implementation of **SGD** and **MLP** applied on **MNIST** dataset.\n", "\n", "This accompanies the online text http://neuralnetworksanddeeplearning.com/ . The book is highly recommended. " ] @@ -1385,7 +1818,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.5.2" + "version": "3.5.3" }, "nbpresent": { "slides": { diff --git a/1.1.1 Perceptron and Adaline.ipynb b/1.1.1 Perceptron and Adaline.ipynb new file mode 100644 index 0000000..0fc0a7f --- /dev/null +++ b/1.1.1 Perceptron and Adaline.ipynb @@ -0,0 +1,976 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "##### (exceprt from Python Machine Learning Essentials, Supplementary Materials)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "### Sections\n", + "\n", + "- [Implementing a perceptron learning algorithm in Python](#Implementing-a-perceptron-learning-algorithm-in-Python)\n", + " - [Training a perceptron model on the Iris dataset](#Training-a-perceptron-model-on-the-Iris-dataset)\n", + "- [Adaptive linear neurons and the convergence of learning](#Adaptive-linear-neurons-and-the-convergence-of-learning)\n", + " - [Implementing an adaptive linear neuron in Python](#Implementing-an-adaptive-linear-neuron-in-Python)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "# Display plots in notebook \n", + "%matplotlib inline\n", + "# Define plot's default figure size\n", + "import matplotlib" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Implementing a perceptron learning algorithm in Python" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "\n", + "class Perceptron(object):\n", + " \"\"\"Perceptron classifier.\n", + "\n", + " Parameters\n", + " ------------\n", + " eta : float\n", + " Learning rate (between 0.0 and 1.0)\n", + " n_iter : int\n", + " Passes over the training dataset.\n", + "\n", + " Attributes\n", + " -----------\n", + " w_ : 1d-array\n", + " Weights after fitting.\n", + " errors_ : list\n", + " Number of misclassifications in every epoch.\n", + "\n", + " \"\"\"\n", + " def __init__(self, eta=0.01, n_iter=10):\n", + " self.eta = eta\n", + " self.n_iter = n_iter\n", + "\n", + " def fit(self, X, y):\n", + " \"\"\"Fit training data.\n", + "\n", + " Parameters\n", + " ----------\n", + " X : {array-like}, shape = [n_samples, n_features]\n", + " Training vectors, where n_samples is the number of samples and\n", + " n_features is the number of features.\n", + " y : array-like, shape = [n_samples]\n", + " Target values.\n", + "\n", + " Returns\n", + " -------\n", + " self : object\n", + "\n", + " \"\"\"\n", + " self.w_ = np.zeros(1 + X.shape[1])\n", + " self.errors_ = []\n", + "\n", + " for _ in range(self.n_iter):\n", + " errors = 0\n", + " for xi, target in zip(X, y):\n", + " update = self.eta * (target - self.predict(xi))\n", + " self.w_[1:] += update * xi\n", + " self.w_[0] += update\n", + " errors += int(update != 0.0)\n", + " self.errors_.append(errors)\n", + " return self\n", + "\n", + " def net_input(self, X):\n", + " \"\"\"Calculate net input\"\"\"\n", + " return np.dot(X, self.w_[1:]) + self.w_[0]\n", + "\n", + " def predict(self, X):\n", + " \"\"\"Return class label after unit step\"\"\"\n", + " return np.where(self.net_input(X) >= 0.0, 1, -1)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "### Training a perceptron model on the Iris dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "#### Reading-in the Iris data" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
01234
1456.73.05.22.3Iris-virginica
1466.32.55.01.9Iris-virginica
1476.53.05.22.0Iris-virginica
1486.23.45.42.3Iris-virginica
1495.93.05.11.8Iris-virginica
\n", + "
" + ], + "text/plain": [ + " 0 1 2 3 4\n", + "145 6.7 3.0 5.2 2.3 Iris-virginica\n", + "146 6.3 2.5 5.0 1.9 Iris-virginica\n", + "147 6.5 3.0 5.2 2.0 Iris-virginica\n", + "148 6.2 3.4 5.4 2.3 Iris-virginica\n", + "149 5.9 3.0 5.1 1.8 Iris-virginica" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "df = pd.read_csv('https://archive.ics.uci.edu/ml/'\n", + " 'machine-learning-databases/iris/iris.data', header=None)\n", + "df.tail()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "#### Plotting the Iris data" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2UFPW95/H3lwGD49VohHWvcmE0iY/ADA8BIYmgoibq\ncj0bjSYkgtHDqjcBY7zXmJwNHA3J5nqzihv1Bh+iNxDBzJr14WgeRIkmcRVGhwfjJZorKOBGxEAw\ngArz3T+qG3qGeajq7l93VffndU6dmaqurv5W1cx3an7f+v3K3B0REal9/aodgIiIVIYSvohInVDC\nFxGpE0r4IiJ1QglfRKROKOGLiNQJJXwRkTqhhC8iUieU8EVE6kT/agdQaNCgQd7U1FTtMEREMqOt\nre0tdx8cZ91UJfympiZWrFhR7TBERDLDzNbHXVdNOiIidUIJX0SkTijhi4jUiVS14Xfn/fffZ8OG\nDezatavaoWTewIEDGTJkCAMGDKh2KCJSBalP+Bs2bODggw+mqakJM6t2OJnl7mzZsoUNGzZw9NFH\nVzscEamC1Dfp7Nq1i8MPP1zJvkRmxuGHH67/lGpI12cX6VlG0pfUJ3xAyb5MdBxrx9y58NWv7kvy\n7tH83LnVjErSLhMJX0T2cYetW2H+/H1J/6tfjea3btWVvvQs9W34WXLPPfdw5plncuSRR1Y7FKlh\nZnDTTdH38+dHE8Ds2dFy/SMnPdEVfhndc889bNq0qdphSB0oTPp5SvbSl6AJ38zWmdlqM2s3s8qM\nmbBoETQ1Qb9+0ddFi0ra3F//+lfOOeccmpubGT58OEuWLKGtrY1JkyYxZswYzjrrLN544w1aW1tZ\nsWIF06ZNo6WlhZ07d7J06VJGjRrFiBEj+NKXvsS7774LwNe//nVOPPFERo4cyTXXXAPAww8/zPjx\n4xk1ahRTpkzhT3/6U4kHQmpZvhmnUGGbvvSsrovd7h5sAtYBg+KuP2bMGO/q97///X7LerRwoXtj\no3t0DqOpsTFaXqTW1la/7LLL9s5v3brVJ0yY4G+++aa7uy9evNgvueQSd3efNGmSL1++3N3dd+7c\n6UOGDPG1a9e6u/sXv/hFv+mmm3zLli1+7LHHekdHh7u7//nPf3Z397fffnvvsjvuuMOvvvrqomPu\nTaLjKanU0eE+e3b04z17dvfz0r05czofo/yxmzOnmlGVBljhMXNsbTXpfPObsGNH52U7dkTLizRi\nxAgef/xxrr32Wp5++mlef/111qxZwxlnnEFLSwvf/va32bBhw37vW7t2LUcffTTHHnssANOnT+ep\np57ikEMOYeDAgVx22WU88MADNDY2AlF/g7POOosRI0Zw44038uKLLxYds9Q2Mzj00M5t9jfdFM0f\neqiadXqiYnf4oq0DvzQzB37o7guCftprryVbHsOxxx5LW1sbjz76KNdddx1nnHEGJ510Es8880yv\n7/Mefnr69+/Pc889x9KlS1m8eDE/+MEPeOKJJ/jKV77C1VdfzdSpU1m2bBlzdX+d9GLu3ChB5ZN7\nPukr2fdMxe7wRduPu/to4NPAP5jZKV1XMLOZZrbCzFZs3ry5tE8bOjTZ8hg2bdpEY2MjX/jCF7jm\nmmt49tln2bx5896E//777++9Gj/44IPZvn07AMcffzzr1q3jlVdeAeDHP/4xkyZN4p133mHbtm2c\nffbZ3HzzzbS3twOwbds2jjrqKADuvffeouOV+tE1QVUjYWWtPTxksTsLxyJownf3TbmvbwI/A8Z1\ns84Cdx/r7mMHD441hn/P5s2DXBPJXo2N0fIirV69mnHjxtHS0sK8efO4/vrraW1t5dprr6W5uZmW\nlhZ+97vfATBjxgwuv/xyWlpacHd+9KMfccEFFzBixAj69evH5Zdfzvbt2zn33HMZOXIkkyZN4qbc\nT9/cuXO54IIL+OQnP8mgQYOKjlekUrLY+StUsTszxyJuY3/SCTgIOLjg+98Bn+rtPSUXbd2jAu2w\nYe5m0dcSCra1SEVbKYcsFo5DxVztY0GCom3INvwjgJ/luvP3B37i7j8P+HmRadOiSUSCyWJ7eE/F\nbiit2J2lY2GeooamsWPHetdHHL700kuccMIJVYqo9uh4Sjm5R11e8jo60pXgulNY7O5uvpTtVuNY\nmFmbu4+Ns25t3ZYpknFZKPzlZbXzV5Jid9zzkZVjoYQvkhKZKfzR+R722bOjq9nZszvf4551cc9H\nlo6FBk8TSYHCTkEQtf0WJpFyNTuUS6j28LRIcj6ydCzUhl9ndDzTq/BKMS+Nhb9CodrD0yDp+ajW\nsVAbfsp961vf4vHHH0/8vmXLlnHuuecGiEjSIIsjYKah81coSc9HFo5FzSX8tBS93J2Ojo5uX7v+\n+uuZMmVK8Bh2794d/DOkfEIW/rr+KPbwo5lYWn7fkkhDIbZax62mEn6Iote1117LbbfdVvAZc/n+\n97/PjTfeyMc+9jFGjhzJnDlzAFi3bh0nnHACV155JaNHj+b1119nxowZDB8+nBEjRuztVTtjxgxa\nW1sBWL58ORMnTqS5uZlx48axfft2du3axSWXXMKIESMYNWoUTz755H5xvf3225x33nmMHDmSk08+\nmVWrVu2Nb+bMmZx55plcfPHFxe+4VFTIwt/kyTBmzL4k39ERzU+eXFrMWSoy56WhEFvN41YzCT/U\nSHgXXXQRS5Ys2Tt///33M3jwYF5++WWee+452tvbaWtr46mnngKiUTIvvvhiXnjhBd566y02btzI\nmjVrWL16NZdcckmnbb/33ntceOGFzJ8/n5UrV/L4449z4IEHcuuttwLRsA733Xcf06dP3+/h43Pm\nzGHUqFGsWrWK73znO52Se1tbGw8++CA/+clPittpqbhQI2B2dMC2bdDevi/pjxkTzW/bVvyVfhZH\nnkwSc6jzUfXjFrdLbiWmUodWKOzSnJ/K0bX5+OOP940bN3p7e7tPnDjRv/a1r/mwYcO8ubnZm5ub\n/cMf/rDfeeed/uqrr3pTU9Pe97399tt+zDHH+Je//GV/7LHHfM+ePe7uPn36dP/pT3/qq1at8okT\nJ+73eeedd54vXbp07/wnPvEJX7lypT/55JN+zjnnuLt7S0uL//GPf9y7zpAhQ3zr1q0+Z84cnzt3\nbo/7oqEV0q3rz2o5uuXv2ePe0tL596KlJVpeilC/byEljTnE+Sj3caNex8MPVfQ6//zzaW1tZcmS\nJVx00UW4O9dddx3t7e20t7fzyiuvcOmllwJw0EEH7X3fYYcdxsqVK5k8eTK33norl112WaftujvW\nTXAe4898d+vkt1UYg2RLiMJfv37Q1tZ5WVtb516hxchqkbnahdhqHreaSvihiiwXXXQRixcvprW1\nlfPPP5+zzjqLu+++m3feeQeAjRs38uabb+73vrfeeouOjg4+85nPcMMNN/D88893ev34449n06ZN\nLF++HIDt27eze/duTjnlFBblHs34hz/8gddee43jjjuu03sL11m2bBmDBg3ikEMOKW1HpSZ1dMDo\n0Z2XjR5deuE2dO/SJIXNNBRi46pqDHH/FajEVEqTTugR64YPH+6TJ0/eO3/zzTf78OHDffjw4X7y\nySf7K6+84q+++qqfdNJJe9dpb2/3UaNG7W36efTRR919X5OOu/tzzz3n48eP95EjR/r48eN9+/bt\nvnPnTp8+fboPHz7cW1pa/IknnnB379Sks2XLFp86daqPGDHCx48f7ytXrnR39zlz5viNN97Y436o\nSae+7NnjfsQR0e9Bc3M039wczR9xRPHNOqF/35I8ijDuutUe1TJUDCRo0ql6ki+cSm3Dr8XnVZab\nEn596ehwP+qo6Dd91qxoftasaP6oo0pLcqF+35IkxaQJNA05otwx1G3Cdw9TZKklSvj1pzDJ56d8\n8i/HtnubL2W7cQubaSjEJlXOGJIkfA2tUGd0POuTV2no3lIkiTmL+1cuNTe0Qpr+KGVZLRzHrrtQ\nrl1Kst1QMSQVt/esJywSpuUYx4056f7Vs9Qn/IEDB7Jly5aaSFbV5O5s2bKFgQMHVjuUooXqoZhk\nu2npXRq392w+vrg9RtNwjJPEnHT/6l3qh0ceMmQIGzZsYPPmzdUOJfMGDhzIkCFDqh1GUdzDDB+c\nZLuhYkiqa+/ZtrZ9vWdbWqLX880bSYbuTcMxThpzloYmToW4jf2VmLor2orkherZGbJAGErS3rNx\ni4RpOMZJY066bq2hloq2IoWSFvLijk+exQJhRwc0NOyb37On9N6zkI5jnFTSOGppDP+aK9qKQLLi\nXDFtxnG2m5YCYb7NvlBhm36x0nCMk8piDaZq4v4rUIlJTTrSk1CdcUJ28gmlsDkn34zTdb4YWTxu\naYmjmqiljlcieUl6KCZpMw7RjT+0SZM6J/d80p80qbTtpuEYJ5XFGkw5JUn4asOXTPGA7fJZawMu\nvBunu/lipeEYJ5U0jjTUYMpFbfhSs+IOV+sJ24yTDIOblmeXdk3u5Uj2SSQ9xmmIIy0xV03cfwUq\nMalJR8qhVttqKyHUyJNZHGgtK0jQpJP6jlciSakzTnE8QQepNHToShqHfi5QG77UrrS0tWdJvskj\nn5yhc4Lsbv04xzjpdouJO2s1mHJJ0oavhC8inYQqatZasTQtVLQVSajrdU9v10FxR6kMHUcIoYqa\ndV8sTQklfKl7SXpfxh2lMnQcIRQ2u5Rz5MlQ25XklPClrhUWFPPJJ5+ctm7tnIy6jlKZT/bt7dHy\nUq70k8QRSk9FzdmzSytqhtquJKc2fKl7SQqKhUk+r6UlGqK41PvgQxc2k8QRoqhZa8XStFDRViSh\nJAXFUKNUJo1DBFS0FUnEHa66qvOyq67qvhmlowNGj+68bPTo3h8v2Nt819dU2JSQlPClrrnDhAlw\nyy0wa1aUuGfNiuYnTNi/Df/II2HlSmhujq7sm5uj+SOP3D/ph3qsn0ixlPBFYjKD/rm+6ZMmRfOT\nJkXz/fv33Ls0ThFWhU2piLhjMFRi0lg6Ug0dHe6zZnmnIXNnzep5eN0k64Z8rJ+Ie7KxdFS0FSHc\n8LoqwkpoqSramlmDmb1gZo+E/iyRQnF7xCYpliZdN24xuBhJC8Jx15XaVYk2/NnASxX4HJG94vaI\nTVIsTbpu3GJwMfQcVylG0IRvZkOAc4A7Q36OSKEkPWKTFEvTUlhNUhBOQw9eSZG4jf3FTEArMAaY\nDDzS1/oq2kq5FD7YOz/19oDvJMXSuOsmKfAmVe/PcZV9SEPR1szOBc529yvNbDJwjbuf2816M4GZ\nAEOHDh2zfv36IPFI5XhKutAn6REbKua09MpV8bh2paVo+3FgqpmtAxYDp5nZwq4rufsCdx/r7mMH\nDx4cMByphLS0F+ebcQoVtukXChXznDndxzBnTmnbhXCFZqlxPV36Ax+KMR0a598I1KRTF9LyzNDC\n5px8M07X+dAxJ4khKT3HVQpRpmfabspNvf3j1wAMLekvjtSMwmeEzp+/b9THSo/42K8ffPCDnUex\nbGuLrq4/+MHOTRuhYu7XD6ZOjb5vb9/XrNPSEi0vpVlHz3GVYvXYhm9mL7j7qF7fHGOdJNTxqjak\npb24o2P/OHprww8Rc+g2/Hp9jqvsU642/Akx3h9nHakjaWov7prQekuGoR7rd/XVnZddfXX5jkXc\n/Uu6rtSuHhO+u+/Kf29mh5nZSDMbnZ+6riOST5xpGPExbiE2VMxpOhYieb214QNgZjcAM4A/Avkf\nUwdOCxeWZFFa2ou9oLMRRDEUJt/C5oxQMaflWIgU6vM+fDNbC4xw9/dCB6M2/NqQhvbiwivsvN4K\nsaFiTsOxkNpW7vvw1wCHlhaS1JM0tBcXXlHn9XbXTaiY03AsRPLiJPzvAi+Y2S/M7KH8FDowkVJ4\n4JEqRbKozzZ84F7ge8BqoIcBZkXSw3MjVT77bDRC5c03R8n+lluiZc88oyttqU9xEv5b7n5L8EhE\nRCSoOE06bWb2XTOb0PW2TJE0Mouu4vPjz/frt29cel3dSz2Lc5fOk90sdncv+22ZuktHyiktPX5F\nQkpyl06fTTrufmrpIYlUVk+9Zyt9W6ZImvTZpGNm3zGzQwvmDzOzb4cNS6R4SXu5pmVIZ5HQ4rTh\nf9rdt+Zn3P3PwNnhQhIpTZJHERb2ytUjAKXWxWnDXwV8zN3fzc0fSDT+8knlDkZt+FJOcZtpkvbK\nFUmTcve0XQgsNbNLzexLwK+I7s0XSbW4vVyT9soVyao+E767/zPwbeAE4CTghtwyybiu/9zVa/NF\n0uGRddwkq2I9isHdf+7u17j719z9F6GDkvBUqIyowCv1pMeEb2aP9PXmOOtI+qhQuY8KvFJPenvE\n4Vbgqd7eC5zk7seUKxgVbStHhcrOVOCVrEpStO0t4U+K8f733P2ZJMH1Rgm/stQTtTg6bpImZelp\n6+6/Ll9IkjZJe6JKRMdNsixW0VZqi563WhwdN8m6OMMjS43R81aLo+MmWddnT9tKUht+ZWnAsOLo\nuEmalHW0TDP7ODAXGJZb34iGRy7b3TlSPkpG4ek5tZJVcZp07gK+CrQBe8KGI6WYOze6Hzzf3JBv\ncz700P07BiVZV0RqQ5yi7TZ3f8zd33T3LfkpeGSSSJJOQepAJFKfersPP/8Yw88CDcADwLv51939\n+XIHozb80iTpFKQORCK1oVwdr7p7tGGeHnGYUkk6BakDkUj2lWV4ZHc/Nfd4w0vz3xcsu6xcwUr5\nJBn1MekIkSKSfXHa8Fu7WfbTcgcipUnSKUgdiETqU4936ZjZ8UTj33/QzP5rwUuHAANDBybJJOkU\npA5EIvWptzb8vwfOA6YCDxW8tB1Y7O6/K3cwasMvXZL78HXPvkj2lWvwtAeBB81sQjlHxJSwknQK\nUgcikfoSp+PV583sc12WbSN6kPmDAWISEZEA4hRtPwC0AC/nppHAh4BLzezmgLGJiEgZxbnC/whw\nmrvvBjCz24FfAmcAqwPGJiIiZRTnCv8o4KCC+YOAI919DwU9b0VEJN3iXOH/M9BuZsuIRso8BfiO\nmR0EPB4wNkkR3dEjkn19Jnx3v8vMHgXGESX8b7j7ptzL/9jT+8xsINFD0D+Q+5xWd59TeshSaRpZ\nU6Q2xH3EYT9gM/A28BEzOyXGe94lavtvJir6fsrMTi4uTKkWjawpUjviPADle8CFwItAR26xE129\n98ijHl3v5GYH5Calh4wp7IU7f/6+0TU1sqZI9vT5iEMzWwuMdPfEBVozayB6cMpHgFvd/dpu1pkJ\nzAQYOnTomPXr1yf9GKkAjawpkk5lGS2zwH8QXZ0n5u573L0FGAKMM7Ph3ayzwN3HuvvYwYMHF/Mx\nEphG1hSpDXES/g6iu3R+aGa35KckH+LuW4FlwKeKiFGqSCNritSOOLdlPkTnwdNiMbPBwPvuvtXM\nDgSmAN9Luh2pLo2sKVI7+mzDB8gl7KHuvjb2hs1GAvcSPR6xH3C/u1/f23s0WmZ66T58kXQqy2iZ\nBRv7L8C/AAcAR5tZC3C9u0/t7X3uvgoYFScIST+NrCmSfXHa8OcSdbraCuDu7cDRAWMSEZEA4iT8\n3e6+rcsylepERDImTtF2jZl9Hmgws48Cs4CyP+1KRETCinOF/xWiZ9u+C9wH/AW4KmRQIiJSfnEG\nT9sBfDM3iYhIRvWY8M3sYXppq+/rLh0REUmX3q7w/6ViUYiISHA9Jnx3/3UlAxERkbDijocvIiIZ\np4QvIlInlPBFROqE7tIREakTuktHRKRO6C4dEZE6EWd45I8C3wVOBAbml7v7MQHjEhGRMotTtP0R\ncDuwGzgV+DfgxyGDEhGR8ouT8A9096VET8da7+5zgdPChiUiIuUWZ3jkXWbWD3jZzL4MbAT+U9iw\nRESk3OJc4V8FNBKNgz8G+CIwPWRQIiJSfnGGR14OkLvKn+Xu24NHJSIiZdfnFb6ZjTWz1cAqYLWZ\nrTSzMeFDExGRcorThn83cKW7Pw1gZp8gunNnZMjARESkvOK04W/PJ3sAd/8NoGYdEZGMiXOF/5yZ\n/ZDoebYOXAgsM7PRAO7+fMD4RESkTOIk/Jbc1zldlk8k+gOge/JFRDIgzl06p1YiEBERCSvOXTpH\nmNldZvZYbv5EM7s0fGgiIlJOcYq29wC/AI7Mzf+BqDOWiIhkSJyEP8jd7wc6ANx9N7AnaFQiIlJ2\ncRL+X83scHJPvzKzk4FtQaMSEZGyi3OXztXAQ8CHzey3wGDg/KBRiYhI2cW5S+d5M5sEHAcYsNbd\n3w8emYiIlFWcu3QuIBoT/0XgPGBJvtOViIhkR5w2/P/u7ttzY+icBdxL9AQsERHJkDgJP39HzjnA\n7e7+IHBAuJBERCSEOAl/Y24snc8Cj5rZB2K+T0REUiRO4v4sUcerT7n7VuBDwD8GjUpERMouzl06\nO4AHCubfAN4IGZSIiJSfmmZEROpEsIRvZn9nZk+a2Utm9qKZzQ71WSIi0rc4PW2LtRv4Wq7j1sFA\nm5n9yt1/H/AzRUSkB8Gu8N39jfzTsNx9O/AScFSozxMRkd5VpA3fzJqAUcCz3bw208xWmNmKzZs3\nVyIcEZG6FDzhm9nfAP8buMrd/9L1dXdf4O5j3X3s4MGDQ4cjIlK3giZ8MxtAlOwXufsDfa0vPVi0\nCJqaoF+/6OuiRfUZg4iUJFjR1swMuAt4yd3/Z6jPqXmLFsHMmbBjRzS/fn00DzBtWv3EICIlM3cP\ns+FosLWngdXknpYFfMPdH+3pPWPHjvUVK1YEiSezmpqiBNvVsGGwbl39xCAi3TKzNncfG2fdYFf4\n7v4bovHzpRSvvZZsea3GICIlU0/btBs6NNnyWo1BREqmhJ928+ZBY2PnZY2N0fJ6ikFESqaEn3bT\npsGCBVF7uVn0dcGCyhZL0xCDiJQsWNG2GCraiogkk6Roqyt8EZE6oYQvIlInlPAlnrT0tL3ySujf\nP6ol9O8fzVdaWo6FSEIhh0eWWpGWnrZXXgm3375vfs+effO33VaZGNJyLESKoKKt9C0tPW3794+S\nfFcNDbB7d2ViSMuxEMlR0VbKKy09bbtL9r0tDyEtx0KkCEr40re09LRtaEi2PIS0HAuRIijhV0PS\nol+oQuWUKdE289OUKd2vN28eDBjQedmAAZXvaZtvK4+7PAT1OpYsc/fUTGPGjPGat3Che2OjO+yb\nGhuj5d254orO6+anK64oLY7TT+9+u6ef3n3MBxzQeb0DDug55pCuuMK9oSGKoaGh9ONQjIUL3YcN\nczeLvlbjOIjkACs8Zo5V0bbSkhb9QhUqrZeBTLv+TKhQKZJaKtqmWdKinwqVIlImSvjlErddPmnR\nL2mhMm67fBJJY05ac0hS01DHK5HixW37qcSU2Tb8JO3yIdvwk7TLn3hi9+ueeGJp6yatOSQ5HqHq\nGUkkPX8igZGgDb/qSb5wymzCHzas+0Q0bFj36yct+sUtVHYXQ34qJeYk283H2XVqaOg+5iRxJN12\nCEnPtUhgSRK+irbl0K/f/oVOiJodOjr2Xx5KkkJskpiTbDfJuiHjCCUt51okR0XbSstiZ5xQMSet\nOSSJQx2vREpSfwk/RMFt3rxoe4X69StfZ5y4hcrTT4+/fN68/RNlQ0P3MSfZbtLOUUk6MiXddqhz\nrY5XklVx234qMQVvww9VcAtZTEyy7YULu1+31AJo0hi6trU3NPR+jJPUNOLWM0IWV9XxSlIEteH3\nIFQHopCjOCbZdpL9S7LdUDGElJY4RAJL0oZfXwk/VMEtZDExa4XYtBQ10xKHSGAq2vYkLYXKUNsO\nVQANFUNIaYlDJEXqK+EnLa7GLfqFLCYm2XaoAmioGIoR99ilZYRPkTSJ29hfiSl40TZp8TFpj9hQ\nxcQkI0TGLSgmLa6GiCGppD2a0zLCp0hAqGjbgzQUH9NSTExLHEkkiTmL+ydSBBVte5KG4mNaiolp\niSOJJDFncf9EiqCibU/SUHxMSzExLXEkkSTmLO6fSGD1lfDTUHxMS0/NefPggAM6LzvggHQXNZMc\nu7QcZw2lLGkSt7G/ElNFRstMS/Gx2j01Fy50HzCgc1FzwID0FzWTHLtqH2cNpSwVgIq20icVNcPT\nMZYKUBu+9E2PLQxPx1hSpjYSvtpJk1NRMzwdY0mZ7Cf8RYuiouv69VEr6fr10bySfu/SUtSsZTrG\nkjLZT/jf/Cbs2NF52Y4d0XLp2bRpsGBB1J5sFn1dsCBaLuWhYywpk/2irTrYiEgdq6+irdpJRURi\nCZbwzexuM3vTzNaE+gwgPe2kKhyLSMqFvMK/B/hUwO1H0tBOqsKxiGRA0DZ8M2sCHnH34XHWz2zH\nK3WwEZEqyVQbvpnNNLMVZrZi8+bN1Q6nOOpgIyIZUPWE7+4L3H2su48dPHhwtcMpjgrHIpIBVU/4\nNSEthWMRkV4o4ZdDGgrHIiJ96B9qw2Z2HzAZGGRmG4A57n5XqM+rumnTlOBFJNWCJXx3/1yobYuI\nSHJq0hERqRNK+CIidUIJX0SkTijhi4jUCSV8EZE6karx8M1sM9DNoDRVNwh4q9pBBKT9yzbtX3aV\nY9+GuXusYQpSlfDTysxWxB2cKIu0f9mm/cuuSu+bmnREROqEEr6ISJ1Qwo9nQbUDCEz7l23av+yq\n6L6pDV9EpE7oCl9EpE4o4XdhZg1m9oKZPdLNazPMbLOZteemy6oRY7HMbJ2Zrc7Fvt+zJC1yi5m9\nYmarzGx0NeIsVoz9m2xm2wrO37eqEWexzOxQM2s1s383s5fMbEKX1zN7/mLsW2bPnZkdVxB3u5n9\nxcyu6rJORc5dsNEyM2w28BJwSA+vL3H3L1cwnnI71d17uu/308BHc9N44Pbc1yzpbf8Annb3cysW\nTXnNB37u7ueb2QFAl6fuZPr89bVvkNFz5+5rgRaILiiBjcDPuqxWkXOnK/wCZjYEOAe4s9qxVMnf\nA//mkf8LHGpmf1vtoATM7BDgFOAuAHd/z923dlktk+cv5r7VitOBP7p71w6mFTl3Svid3Qz8E9DR\nyzqfyf3L1Wpmf1ehuMrFgV+aWZuZzezm9aOA1wvmN+SWZUVf+wcwwcxWmtljZnZSJYMr0THAZuBH\nuSbHO80NOoK6AAAFIElEQVTsoC7rZPX8xdk3yO65K3QRcF83yyty7pTwc8zsXOBNd2/rZbWHgSZ3\nHwk8DtxbkeDK5+PuPpro38d/MLNTurxu3bwnS7dx9bV/zxN1Q28G/hfwfyodYAn6A6OB2919FPBX\n4Otd1snq+Yuzb1k+dwDkmqqmAj/t7uVulpX93Cnh7/NxYKqZrQMWA6eZ2cLCFdx9i7u/m5u9AxhT\n2RBL4+6bcl/fJGpDHNdllQ1A4X8tQ4BNlYmudH3tn7v/xd3fyX3/KDDAzAZVPNDibAA2uPuzuflW\noiTZdZ0snr8+9y3j5y7v08Dz7v6nbl6ryLlTws9x9+vcfYi7NxH92/WEu3+hcJ0ubWpTiYq7mWBm\nB5nZwfnvgTOBNV1Wewi4OHfHwMnANnd/o8KhFiXO/pnZfzYzy30/jujnf0ulYy2Gu/8/4HUzOy63\n6HTg911Wy+T5i7NvWT53BT5H9805UKFzp7t0+mBm1wMr3P0hYJaZTQV2A28DM6oZW0JHAD/L/c70\nB37i7j83s8sB3P1fgUeBs4FXgB3AJVWKtRhx9u984Aoz2w3sBC7ybPU8/AqwKNc08B/AJTV0/vra\nt0yfOzNrBM4A/lvBsoqfO/W0FRGpE2rSERGpE0r4IiJ1QglfRKROKOGLiNQJJXwRkTqhhC+ZYtGI\npUfGWO8eMzs/7vIyxPWNgu+bzKxrH4eeYnw1f3teiZ9/YW6kxf1GeRXJU8KXrJkB9Jnwq+Abfa/S\nrX/M3YddEndfAmRquG6pPCV8qZrclfC/m9m9BQPSNeZeG2Nmv84NhPYLM/vb3JX5WKIOOu1mdqCZ\nfcvMlpvZGjNbkO+NGfPz9/uM3PJlZvY9M3vOzP5gZp/MLW80s/tzsS4xs2fNbKyZ/Q/gwFxMi3Kb\nbzCzO8zsRTP7pZkdGCOeI8zsZ7kBwlaa2cSCY3Rnbh8XmdkUM/utmb2c63UqEosSvlTbccCC3IB0\nfwGuNLMBRANkne/uY4C7gXnu3gqsAKa5e4u77wR+4O4fc/fhwIFArPHSe/qMglX6u/s44CpgTm7Z\nlcCfc7HeQG4sJXf/OrAzF9O03LofBW5195OArcBnYoR1C/Dr3ABho4EXc8s/QjRe/EjgeODzwCeA\nayj+PwupQxpaQartdXf/be77hcAs4OfAcOBXuQv2BqCncUVONbN/InpgxoeIkuTDMT73uD4+44Hc\n1zagKff9J4gSL+6+xsxW9bL9V929vZtt9OY04OLc9vcA28zssNy2VgOY2YvAUnd3M1sdc7sigBK+\nVF/XsT2caKjYF919Qjfr72VmA4HbgLHu/rqZzQUGxvzcvj4jPyrqHvb9nsRuLip4f34bfTbpxNxW\nR8F8B/odlgTUpCPVNtT2Pb/0c8BvgLXA4PxyMxtg+x54sR04OPd9Prm/ZWZ/QzTAVly9fUZPfgN8\nNrf+icCIgtfezzUTlWIpcEVu+w0WPQlKpGyU8KXaXgKm55pHPkT0EIz3iJL398xsJdAOTMytfw/w\nr2bWTnSlewewmuiBGMvjfmgfn9GT24j+SKwCrgVWAdtyry0AVhUUbYsxm6iJajVRM1BWn+okKaXR\nMqVqzKwJeCRXcE09ix5APcDdd5nZh4muyI/N/fEoZnv3EO1/a5nimwxck8UHfUtlqP1PJL5G4Mlc\n040BVxSb7HO2ATeY2aBS78U3swuJ7ibq7RGdUud0hS8iUifUhi8iUieU8EVE6oQSvohInVDCFxGp\nE0r4IiJ1QglfRKRO/H9dfFAdFqad1gAAAABJRU5ErkJggg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# select setosa and versicolor\n", + "y = df.iloc[0:100, 4].values\n", + "y = np.where(y == 'Iris-setosa', -1, 1)\n", + "\n", + "# extract sepal length and petal length\n", + "X = df.iloc[0:100, [0, 2]].values\n", + "\n", + "# plot data\n", + "plt.scatter(X[:50, 0], X[:50, 1],\n", + " color='red', marker='o', label='setosa')\n", + "plt.scatter(X[50:100, 0], X[50:100, 1],\n", + " color='blue', marker='x', label='versicolor')\n", + "\n", + "plt.xlabel('petal length [cm]')\n", + "plt.ylabel('sepal length [cm]')\n", + "plt.legend(loc='upper left')\n", + "\n", + "# plt.savefig('./iris_1.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "#### Training the perceptron model" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfWd//HXJwshrGFfwhLW4C4aFEURFAlqq9apox3b\nUbtYW1uxRVzmMVOnnc6vFVypTi3ujlbbWkZtqwQQBHEPKoJK2HeUALInEJLP74970RizHCA359zc\n9/PxOI9777nnnvPmAvnke873fL/m7oiIiERNWtgBREREaqMCJSIikaQCJSIikaQCJSIikaQCJSIi\nkaQCJSIikaQCJSIikaQCJSIikaQCJSIikZQRdoBD1blzZ8/Lyws7hoiIHKYFCxZscfcuDW2XdAUq\nLy+P4uLisGOIiMhhMrM1QbbTKT4REYkkFSgREYkkFSgREYkkFSgREYkkFSgREYmkhPXiM7OWwDwg\nK36cZ939thrbZAFPACcDW4HL3H11ojJJcM+9t4HJRSVs3F5Gz5xsJhbmc/HQ3LBjiUgKSWQLah9w\ntrufAJwIjDOz4TW2+R7wmbsPBO4Gbk9gHgnoufc2cOu0RWzYXoYDG7aXceu0RTz33oawo4lICklY\ngfKY3fGXmfGl5vzyFwGPx58/C5xjZpaoTBLM5KISyioqv7SurKKSyUUlISUSkVSU0GtQZpZuZu8D\nm4GZ7v5WjU1ygXUA7n4A2AF0qmU/15hZsZkVl5aWJjKyABu3lx3SehGRREhogXL3Snc/EegFnGJm\nx9bYpLbWUs1WFu4+1d0L3L2gS5cGR8eQI9QzJ/uQ1ouIJEKT9OJz9+3AK8C4Gm+tB3oDmFkG0B7Y\n1hSZpG7/PKzXV9ZlZ6YzsTA/hDQikqoSVqDMrIuZ5cSfZwNjgCU1NnsBuDL+/JvAbHf/SgtKmo67\n8+aKbbRukU6P9i0/X//Ds/qrF5+INKlEtqB6AHPM7APgHWLXoP5uZr8yswvj2zwMdDKz5cDPgVsS\nmEcCmL98C2+s3MqEsfm8ces5fPjLQjq3acEbK7ai3x1EpCkl7D4od/8AGFrL+l9Ue14OXJqoDHJo\nqqqcSdNLyM3J5orhfQBonZXBT88exG0vfMi8ZVs4a7CuAYpI09BIEvK5lxZ/wqINO/j5uYPJykj/\nfP23TulD747ZTJq+hKoqtaJEpGmoQAkAFZVV3DGjhMHd2nzlWlOLjDQmnJvPhxt38o9Fm0JKKCKp\nRgVKAHh2wXpWbdnDxMIhpKd9tff/hSf0ZEj3ttw5o4SKyqoQEopIqlGBEsorKrln1lJO6pPDmKO6\n1rpNWppx07h8Vm/dy5+L1zVxQhFJRSpQwmOvr+bTnfu4edwQ6htpanR+V4bldeDeWcso219Z53Yi\nIo1BBSrF7dhbwf/MWc6o/C6c2v8ro0x9iZlx07ghbN61j0dfX9VECUUkValApbg/zFvBzvIDgUeJ\nGJbXkXOGdOWBV1awY29FgtOJSCpTgUphm3eW88hrq7joxJ4c07N94M/dWJjPrn0H+P3cFQlMJyKp\nTgUqhU2ZvYwDlc7Pzx18SJ87qkc7Lj4xl0dfW8UnO8oTlE5EUp0KVIpavWUPz7y9jm+d0oe+nVof\n8ud/fu5gqtyZMntZAtKJiKhApay7Zi4lMz2Nn5498LA+37tjK644tS9/emcdq7bsaeR0IiIqUCnp\nw407eGHhRr57Rh5d27Vs+AN1uG70QLIy0rhzhmbaFZHGpwKVgiYXldA+O5NrRg44ov10aZvF98/o\nx98/2MTiDTsaKZ2ISIwKVIp5c+VWXikp5cejBtA+O/OI9/f9kf3p0CqTSUVqRYlI41KBSiHuzu3T\nl9CtXRZXnp7XKPts1zKT60YPZN7SUl5fsaVR9ikiAipQKWXmR5/y3trt3DBmMC0z0xv+QEDfHt6X\nHu1bcvv0Ek1qKCKNRgUqRVRWOZOLSujfuTWXntyrUffdMjOdn40ZzMJ12yn68NNG3beIpC4VqBTx\nf+9tYNnm3dxYmE9GeuP/tV9yUi4DurTmjhklHNB0HCLSCFSgUsC+A5XcPXMpx+W257xjuyfkGBnp\naUwsHMLyzbuZ9t6GhBxDRFKLClQKeOrNtWzYXtbgdBpHqvCYbpzQO4d7Zi6lvELTcYjIkVGBauZ2\n7zvAfXOWM2JgJ84Y1DmhxzIzbh6Xz8Yd5Tz55pqEHktEmj8VqGbuoVdXsm3Pfm4qHNIkxzt9QGfO\nHNSZ++csZ1e5puMQkcOnAtWMbdm9jwfnreS8Y7tzQu+cJjvuTYVD+GxvBQ++qkkNReTwqUA1Y/fP\nWU5ZRSUTxgabjLCxHNerPRcc34OHXl1J6a59TXpsEWk+VKCaqXXb9vLUm2v554LeDOzapsmPP+Hc\nwew7UMX9c5Y3+bFFpHlQgWqm7pm1DAzGjxkUyvH7d2nDPxf05qm31rBu295QMohIcktYgTKz3mY2\nx8w+NrMPzWx8LduMMrMdZvZ+fPlFovKkkpJPdjHtvfVcdXoePdpnh5Zj/DmDSDPj7plLQ8sgIskr\nkS2oA8AEdz8KGA5cZ2ZH17Ldq+5+Ynz5VQLzpIw7ZpTQpkUGPzrryKbTOFLd27fkqhF5/N/7G1jy\nyc5Qs4hI8klYgXL3Te7+bvz5LuBjIDdRx5OYBWs+Y+ZHn/LDs/rToXWLsOPw47MG0jYrgzs0HYeI\nHKImuQZlZnnAUOCtWt4+zcwWmtlLZnZMU+Rprg5Op9G5TRZXj+gXdhwA2rfK5NpRA5j18WaKV28L\nO46IJJEGC5SZDTCzrPjzUWZ2vZkFvqnGzNoAfwVucPea53neBfq6+wnA74Dn6tjHNWZWbGbFpaWl\nQQ+dcl5ZWsrbq7Zx/TkDaZ2VEXacz119ej+6ts3i9ulLNB2HiAQWpAX1V6DSzAYCDwP9gD8G2bmZ\nZcY//5S7T6v5vrvvdPfd8ecvAplm9pXxeNx9qrsXuHtBly5dghw65VRVOZOml9CnYysuH9Yn7Dhf\nkt0inevPGcQ7qz/jlRL9giEiwQQpUFXufgD4BnCPu/8M6NHQhyw2KunDwMfuflcd23SPb4eZnRLP\nszVoePnC3z7YyMebdjJh7GBaZETv7oHLhvWmb6dW3D59CVVVakWJSMOC/CSrMLNvAVcCf4+vywzw\nuRHAd4Czq3UjP9/MrjWza+PbfBNYbGYLgSnA5a5zQIds/4Eq7pyxlCHd2/L143uGHadWmelpTBib\nz5JPdvHCwo1hxxGRJBDkQsXVwLXAf7v7KjPrBzzZ0IfcfT5Q79wO7n4fcF+QoFK3PxWvY+22vTx6\n1TDS0hI3ncaR+tpxPXjglRXcObOE84/rEcmWnohER4M/Idz9I3e/3t2fjr9e5e6/TXw0CWLv/gNM\neXkZp+R1ZFR+tK/PpaUZN43LZ922Mp55Z23YcUQk4oL04hthZjPNbKmZrTSzVWa2sinCScMefW01\npbv2cdO4/IRORthYzhrchVP7dWTKy8vZs+9A2HFEJMKCnGN5GLgLOAMYBhTEHyVk2/fu54G5Kxhz\nVFcK8jqGHScQM+Pm84awZfc+Hn1N03GISN2CFKgd7v6Su292960Hl4Qnkwb9fu4Kdu87wI2FTTud\nxpE6qU8Hxh7djT/MXclne/aHHUdEIipIgZpjZpPN7DQzO+ngkvBkUq9NO8p47LXVfOPEXIZ0bxd2\nnEN2Y2E+e/Yf4PdzV4QdRUQiKkgvvlPjjwXV1jlwduPHkaCmvLyMKnd+du7gsKMclsHd2nLJSb14\n7PXVXD0i3FHXRSSagvTiG13LouIUohWlu/lz8XquOLUvvTu2CjvOYbthzCBwuGfmsrCjiEgEBenF\n197M7jo4Fp6Z3Wlm7ZsinNTurhlLycpI4ydnDww7yhHp1aEV3x7el78sWMfyzbvDjiMiERPkGtQj\nwC7gn+PLTuDRRIaSun2wfjv/WLSJ75/Zn85tssKOc8SuGz2A7Mx07pyh6ThE5MuCFKgB7n6bu6+M\nL78E+ic6mNRuclEJHVpl8oMzozGdxpHq1CaLH4zsz0uLP2Hhuu1hxxGRCAlSoMrM7IyDL8xsBFCW\nuEhSl9eWb+HVZVu4bvRA2rYMMhxicvj+mf3p2LoFk4qWhB1FRCIkSIH6EXC/ma02szXExs67toHP\nSCNzdyZNX0LP9i359vC+YcdpVG2yMvjJ6IG8tnwr85dtCTuOiEREkF5878cnFDweOM7dh7r7wsRH\nk+qKPvyEhet3cMO5g2mZmR52nEZ3xfA+5OZka1JDEflcnfdBmdm33f1JM/t5jfUA1DXHkzS+A5VV\nTCoqYWDXNlwyNDfsOAmRlZHOz88dzIS/LOSlxZ9w/nENTjkmIs1cfS2o1vHHtrUsbRKcS6r567vr\nWVm6h4mF+WSkN98pKi4emsvgbm24o6iEA5VVYccRkZDV2YJy9z/En85y99eqvxfvKCFNoLyikntm\nLePE3jmMPbpb2HESKj3NmFg4hB88UcxfFqznW6dEa+p6EWlaQX4d/13AdZIA//vGGjbtKOfmcUOS\nYjqNIzXmqK6c1CeHe2YtpbyiMuw4IhKi+q5BnQacDnSpcR2qHdD8rtJH0M7yCu5/ZTkjB3fhtAGd\nwo7TJMyMm8cN4bKpb/L466v54VkDwo4kIiGprwXVgti1pgy+fP1pJ/DNxEeTB+etZPveCm5Ksuk0\njtSp/TsxKr8L//PKCnaUVYQdR0RCUt81qLnAXDN7zN3XNGEmAUp37eOhV1fxteN7cGxu6g19OLEw\nnwumzGfqvBVMLBwSdhwRCUGQa1B74/NBvWhmsw8uCU+W4u6bvYz9lVVMGJtaraeDjunZngtP6Mkj\n81ezeWd52HFEJARBCtRTwBKgH/BLYDXwTgIzpby1W/fyx7fXctmw3vTr3LrhDzRTE8YOpqKyit/N\nXh52FBEJQZAC1cndHwYq3H2uu38XGJ7gXCntrpklpKcZ488ZFHaUUPXt1JpvndKHp99ey5qte8KO\nIyJNLEiBOniVepOZXWBmQ4FeCcyU0j7auJPnF27k6hH96NauZdhxQvfTsweSmZ7GXTOXhh1FRJpY\nkAL16/gEhROAG4GHgJ8lNFUKu2NGCW2zMrh2pLpXA3Rt15LvnpHH8+9v5MONO8KOIyJNKMhgsX93\n9x3uvjg+3fvJ7v5CU4RLNW+v2sbsJZv50aiBtG/VfKbTOFLXjBxA++xMJhdpUkORVBJkyvfHzSyn\n2usOZvZIYmOlnoPTaXRtm8VVp+eFHSdS2mdn8uNRA3ilpJQ3V24NO46INJE674Oq5nh3/3yqU3f/\nLH4dql5m1ht4AugOVAFT3f3eGtsYcC9wPrAXuMrd3z2E/Ifkufc2MLmohI3by+iZk83EwnwuDnl0\n8IOZNmyPzQF5aUEvsltooI6arjw9j/vnLOM7D7/FgUqPzN+fiCROkAKVZmYd3P0zADPrGPBzB4AJ\n7v6umbUFFpjZTHf/qNo25wGD4supwO/jj43uufc2cOu0RZTFx3fbsL2MW6Z9QEVlFV8/oWciDtmg\nvy3cyH88v5jyiqovrRsxoLN+8NYwffEnlFVUUVEZmytqw/Yybp22CEDflUgzFaTQ3Am8bmbPxl9f\nCvx3Qx9y903ApvjzXWb2MZALVC9QFwFPeGyGujfNLMfMesQ/26gmF5V8XpwOKq+oYuKzHzDx2Q8a\n+3CHrbyiislFJfqhW8PkopLPi9NBZRWV+q5EmrEGC5S7P2FmxcDZgAGX1GgFNcjM8oChwFs13soF\n1lV7vT6+7ksFysyuAa4B6NPn8KZg2Bg/hVabm8eFM5TO7dOX1Lq+vqypqq7vRN+VSPNV32jm7dx9\nZ/yU3ifAH6u919HdtwU5gJm1Af4K3ODuO2u+XctHvjLft7tPBaYCFBQUHNZ84D1zsj+/zlNdbk42\nPxoVTpfuJ99cU2umnjnZIaSJtrr+/vRdiTRf9fXiO1iQFgDF1ZaDrxtkZpnEitNT7j6tlk3WA72r\nve4FbAyy70M1sTCf7Mwvdz7IzkxnYogjhUcxU1TV9l1lppu+K5FmrL5TfL+NPx7l7oc8Wme8h97D\nwMfuflcdm70A/MTMniHWOWJHIq4/wRcX0qPUiy+KmaKq5nfVIiONjDRjTDOfZVgklVmsf0Itb5gt\ncPeTzexddz/pkHdsdgbwKrCIWDdzgH8D+gC4+wPxInYfMI5YN/Or3b3e1llBQYEXFwdqwEkztnDd\ndi66/zVuGDOIG8YMDjuOiByCeH0paGi7+lpQFWb2KNDLzKbUfNPdr69vx+4+n9qvMVXfxoHrGgop\nUtMJvXM479juPDhvJd8Z3pdObbLCjiQijay+a1BfA4qAMmLXnWouIqGaMDafsopK7p+zIuwoIpIA\n9c2ouwV4xsw+dveFTZhJJJCBXdtw6cm9efLNNXz3jDx6dWgVdiQRaUR1tqDM7Kb40++b2ZSaSxPl\nE6nX+DGDwOCeWcvCjiIijay+a1Afxx/VI0Eiq2dONlee1peH56/impH9GdytbdiRRKSR1HeK72/x\nx8cPrjOzNKBNLTfcioTmx6MG8szb67ijqISp/9pgxyARSRJBptv4o5m1M7PWxMbRKzGziYmPJhJM\nh9Yt+OFZ/Znx0acsWPNZ2HFEpJEEmVH36HiL6WLgRWL3MX0noalEDtHVI/rRuU0Wt09fQl339olI\ncglSoDLjQxZdDDzv7hXUMl6eSJhaZ2Vw/TkDeXvVNuYuLQ07jog0giAF6g/AaqA1MM/M+gK6BiWR\nc/mwPvTumM2k6SVUVel3KJFk12CBcvcp7p7r7ud7zBpgdBNkEzkkLTLSmHBuPh9t2snfFyVkSEcR\naUJBOkmMj3eSMDN72MzeJTY3lEjkXHhCT4Z0b8udM0qoqKxq+AMiEllBTvF9N95JYizQBbiaL0Y6\nF4mUtDTjpnH5rNm6lz+9s67hD4hIZAUpUAcHfD0feDQ+7FG9g8CKhGl0fleG5XXg3peXUba/Muw4\nInKYghSoBWY2g1iBKjKztnwxfYZI5JgZN48bQumufTzy2qqw44jIYQpSoL4H3AIMc/e9QAtip/lE\nIqsgryNjjurKA3NXsH3v/rDjiMhhCNKLrwpYBQw2s5HAMUBOooOJHKkbC/PZve8Av5+r6ThEklGQ\nXnzfB+YRmxvql/HH/0xsLJEjN6R7O75xYi6PvbaaT3aUhx1HRA5RkFN844FhwBp3Hw0MBXSrviSF\nn507mCp37n1Z03GIJJsgBarc3csBzCzL3ZcA+YmNJdI4endsxRWn9uXPxetYWbo77DgicgiCFKj1\nZpYDPAfMNLPngY2JjSXSeK4bPZCsjDTunLk07CgicgiCdJL4hrtvd/f/BP4DeJjYwLEiSaFL2yy+\nf0Y//vHBJhat3xF2HBEJqL4p3zvWXIBFwHygTZMlFGkEPxjZnw6tMplUtCTsKCISUH1Tvi8gNq1G\n9VEjDr52oH8Cc4k0qrYtM7lu9EB+/Y+PeX35Fk4f2DnsSCLSgDpbUO7ez937xx/71Xit4iRJ59vD\n+9KzfUtNaiiSJILcB/UNM2tf7XWOmekalCSdlpnp3HDuYBau30HRh5+EHUdEGhCkF99t7v75lWV3\n3w7clrhIIolzydBcBnZtw+SiEg5oOg6RSAtSoGrbpr5rVwCY2SNmttnMFtfx/igz22Fm78eXXwTI\nInJEMtLTuHFsPitK9zDt3Q1hxxGRegQpUMVmdpeZDTCz/mZ2N7EOFA15DBjXwDavuvuJ8eVXAfYp\ncsQKj+nGCb1zuHvWUsorNB2HSFQFKVA/BfYDfwL+ApQD1zX0IXefB2w7onQiCRCbjiOfTTvKefLN\nNWHHEZE6BLlRd4+73+LuBcApwG/cfU8jHf80M1toZi+Z2TF1bWRm15hZsZkVl5ZqGEA5cqcP6MyZ\ngzpz/5zl7CyvCDuOiNQiSC++P5pZOzNrDXwIlJjZxEY49rtAX3c/AfgdsaGUauXuU929wN0LunTp\n0giHFoGbxw3hs70VPDhvZdhRRKQWQU7xHe3uO4kNb/Qi0Af4zpEe2N13uvvu+PMXgUwz092T0mSO\nzW3P147vwUOvrqJ0176w44hIDUEKVKaZZRIrUM+7ewWxkSSOiJl1NzOLPz8lnmXrke5X5FBMGJvP\n/soq7put6ThEoiZIgfoDsBpoDcwzs77AzoY+ZGZPA28A+Wa23sy+Z2bXmtm18U2+CSw2s4XAFOBy\n1+390sT6dW7NZcN688e317J2696w44hINXY4NcHMMtz9QALyNKigoMCLi4vDOLQ0U5/uLGfkpDmc\nf1wP7r7sxLDjiDR7ZrYg3vGuXnXecGtm33b3J83s53VsctdhpxOJkG7tWnL1iH78Yd4KrhnZn6N6\ntAs7kohQ/ym+1vHHtnUsIs3Gj84aQNusDO4oKgk7iojE1dmCcvc/xB9/2XRxRMLRvlUm144awKTp\nJbyzehvD8jqGHUkk5QW5D6pffKijaWb2wsGlKcKJNKWrT+9H17ZZ3P6SpuMQiYIGB30ldgPtw8Df\nAA3/LM1Wdot0rj9nEP/+3GJmL9nMOUd1CzuSSEoLUqDK3X1KwpOIRMBlw3rz0KsrmTS9hFH5XUlP\ns4Y/JCIJEeQ+qHvN7DYzO83MTjq4JDyZSAgy09OYMDafkk938cJCTcchEqYgLajjiA1tdDZfnOLz\n+GuRZueC43rwwNwV3DljKRcc15MWGUF+jxORxhbkf943gP7ufpa7j44vKk7SbKWlGTeNG8L6z8p4\n+u21YccRSVlBCtRCICfRQUSiZOSgzgzv35HfzV7Gnn2hDJoikvKCFKhuwBIzK1I3c0kVZrFW1Jbd\n+3lk/qqw44ikpCDXoG5LeAqRCDqpTwfGHt2NqfNWcsXwvnRs3SLsSCIpJciMunNrW5oinEjYbizM\nZ8/+A/zPnOVhRxFJOeqeJFKPwd3acslJvXjizTVs2F4WdhyRlKICJdKAn507GBzunbU07CgiKaXO\nAmVmL8cfb2+6OCLRk5uTzXdO68uzC9azfPOusOOIpIz6WlA9zOws4EIzG1p9FAmNJCGp5sejBtCq\nRQZ3FKkVJdJU6uvF9wvgFqAXX52cUCNJSErp1CaLH5zZn7tnLeX9dds5sbduDRRJtDpbUO7+rLuf\nB0yqNoKERpKQlPW9M/vRqXULTcch0kSCdDP/LzO70MzuiC9fa4pgIlHTJiuDn5w9kDdWbmX+8i1h\nxxFp9oJMWPgbYDzwUXwZH18nknL+5dQ+5OZkc/v0JVRVqRUlkkhBuplfAJzr7o+4+yPAuPg6kZST\nlZHOz88dzOINO3lx8aaw44g0a0Hvg6p+Rbh9IoKIJIuLh+aS360td85YSkWlJpkWSZQgBeo3wHtm\n9piZPQ4sAP5fYmOJRFd6mjGxMJ9VW/bwl+L1YccRabaCdJJ4GhgOTIsvp7n7M4kOJhJl5xzVlZP7\nduDel5dStr8y7DgizVKgU3zuvsndX3D35939k0SHEok6M+PmcUP4dOc+Hn9jddhxRJqlhI3FZ2aP\nmNlmM1tcx/tmZlPMbLmZfaDRKSTZnNKvI0d1b8vtLy2h3y3/YMRvZ/PcexvCjiXSbCRysNjHiPX4\nq8t5wKD4cg3w+wRmEWl0z723gZVb9uDEhlbZsL2MW6ctUpESaST1FigzS6urBdQQd58HbKtnk4uA\nJzzmTSDHzHoczrFEwjC5qIR9B77ci6+sopLJRSUhJRJpXuotUO5eBSw0sz4JOHYusK7a6/XxdV9h\nZteYWbGZFZeWliYgisih21jH/FB1rReRQxNkyvcewIdm9jaw5+BKd7/wCI9ttayr9dZ8d58KTAUo\nKCjQ7fsSCT1zsmudxLBnTnYIaUSanyAF6pcJOvZ6oHe1172AjQk6lkijm1iYz63TFlFW8UU383SL\n3SMlIkcuyH1Qc4HVQGb8+TvAu41w7BeAf4335hsO7HB3jR0jSePiobn85pLjyM3JxoC2WRlUujOw\na5uwo4k0Cw22oMzsB8R62XUEBhC7TvQAcE4Dn3saGAV0NrP1wG1AJoC7PwC8CJwPLAf2Alcf7h9C\nJCwXD83l4qGxS6c7yysYOWkOk4tKePy7p4ScTCT5BTnFdx1wCvAWgLsvM7OuDX3I3b/VwPse37dI\ns9CuZSY/HjWA//fiEt5YsZXTBnQKO5JIUgtyH9Q+d99/8IWZZVBHZwaRVPevp+XRvV1LJhVpUkOR\nIxWkQM01s38Dss3sXOAvwN8SG0skObXMTOeGMYN4b+12Zn70adhxRJJakAJ1C1AKLAJ+SOza0b8n\nMpRIMvvmyb3o37k1k4tKqNSkhiKHLUgvvirgceC/iHU5f9x17kKkThnpadxYmM+yzbuZ9q6m4xA5\nXEGmfL8AWAFMAe4DlpvZeYkOJpLMzju2O8f3as89s5ZRXqHpOEQOR5BTfHcCo919lLufBYwG7k5s\nLJHkdnA6jg3by3jqrbVhxxFJSkEK1GZ3X17t9Upgc4LyiDQbIwZ25oyBnbl/znJ2lVeEHUck6dRZ\noMzsEjO7hNg4fC+a2VVmdiWxHnzvNFlCkSQ2sTCfbXv289Crq8KOIpJ06mtBfT2+tAQ+Bc4iNjJE\nKdAh4clEmoETeudw/nHdeejVlWzZvS/sOCJJpc6RJNxdQw+JNIIJY/Mp+vBT7p+znNu+fkzYcUSS\nRpCx+PoBPwXyqm/fCNNtiKSEAV3acOnJvXjqzbV8d0Q/endsFXYkkaQQpJPEc8RGM/8dsR59BxcR\nCWj8mEFgcPespWFHEUkaQQaLLXf3KQlPItKM9WifzVWn5/Hgqyv54cgB5HdvG3YkkcgL0oK618xu\nM7PTzOykg0vCk4k0Mz86awBtWmQwuagk7CgiSSFIC+o44DvA2UBVfJ3HX4tIQB1at+DaUQOYXFTC\ngjXbOLlvx7AjiURakBbUN4D+7n6Wu4+OLypOIofh6hF5dG6Txe0vlWg6DpEGBClQC4GcRAcRSQWt\nWmQw/pyBvL16G68sLQ07jkikBSlQ3YAlZlZkZi8cXBIdTKS5umxYH/p0bMWk6SVUaToOkToFuQZ1\nW8JTiKSQFhlpTBg7mPHPvM/fPtjIRSfmhh1JJJIaLFDuPrcpgoikkq8f35Pfv7KCO2cs5bxje9Ai\nI8jJDJHHFPzeAAAMO0lEQVTUEmQ+qF1mtjO+lJtZpZntbIpwIs1VWlpsOo612/byp3c0HYdIbYLM\nqNvW3dvFl5bAPxGbuFBEjsCo/C6ckteRe19ezt79B8KOIxI5h3xewd2fQ/dAiRwxM+Pm8/LZsnsf\nj762Ouw4IpETZLDYS6q9TAMKiN2oKyJH6OS+HRlzVDceeGUF/3JKHzq0bhF2JJHICNKC+nq1pRDY\nBVyUyFAiqWRiYT679x/ggbkrwo4iEilBevFpXiiRBMrv3pZvDM3lsddXc9WIPHq0zw47kkgk1Fmg\nzOwX9XzO3f2/Gtq5mY0D7gXSgYfc/bc13r8KmAxsiK+6z90fami/Is3Nz8YM5m8LNzLl5WX85pLj\nw44jEgn1neLbU8sC8D3g5oZ2bGbpwP3AecDRwLfM7OhaNv2Tu58YX1ScJCX17tiKK07ty5+L17Oi\ndHfYcUQioc4C5e53HlyAqUA2cDXwDNA/wL5PAZa7+0p33x//nK5didThJ2cPJCsjjbtmaFJDEWig\nk4SZdTSzXwMfEDsdeJK73+zumwPsOxdYV+31+vi6mv7JzD4ws2fNrHcdOa4xs2IzKy4t1QCb0jx1\nbpPF98/szz8WbeKD9dvDjiMSujoLlJlNBt4h1mvvOHf/T3f/7BD2bbWsq9k9/W9AnrsfD8wCHq9t\nR+4+1d0L3L2gS5cuhxBBJLn84Mx+dGzdgknTNamhSH0tqAlAT+DfgY3VhjvaFXCoo/VA9RZRL2Bj\n9Q3cfau774u/fBA4OXh0keanbctMrhs9kPnLtzB/2Zaw44iEqr5rUGnunl1jqKN2B18H2Pc7wCAz\n62dmLYDLgS9N02FmPaq9vBD4+HD+ECLNyRWn9iE3J5tJRUs0qaGktIQNoezuB4CfAEXECs+f3f1D\nM/uVmV0Y3+x6M/vQzBYC1wNXJSqPSLJomZnODWMG8cH6HUxf/EnYcURCY8n2G1pBQYEXFxeHHUMk\noSqrnHH3zKPSnRk3jCQjXdNxSPNhZgvcvaCh7fSvXiSC0tOMGwvzWVm6h7++uz7sOCKhUIESiaix\nR3fjxN453DNrGeUVlWHHEWlyKlAiEWUWm9Rw045ynnhjddhxRJqcCpRIhJ02oBMjB3fh/jkr2FFW\nEXYckSalAiUScTcV5rOjrIIH560MO4pIk1KBEom4Y3Pb8/UTevLw/FVs3lUedhyRJqMCJZIEJpw7\nmIrKKu6bvTzsKCJNRgVKJAnkdW7NZcN688e31rJ2696w44g0CRUokSRx/TmDyEg37pqpgWQlNahA\niSSJbu1acvWIfjy/cCMfbQwyXrNIclOBEkki144cQNusDO6YoVaUNH8qUCJJpH2rTH40aiCzl2zm\n7VXbwo4jklAqUCJJ5qrT8+jWLovbp2s6DmneVKBEkkx2i3TGnzOYBWs+4+WPN4cdRyRhVKBEktCl\nBb3o17k1k4tKqKxSK0qaJxUokSSUmZ7GhLGDKfl0F8+/vyHsOCIJoQIlkqTOP7YHx+a2466ZS9l3\nQNNxSPOjAiWSpNLSjJsKh7D+szKefmtt2HFEGp0KlEgSO3NQZ07r34nfzV7O7n0Hwo4j0qhUoESS\nmJlx07h8tu7ZzyPzV4UdR6RRqUCJJLmhfTpQeEw3ps5bydbd+8KOI9JoVKBEmoGJhfns3X+A/3ll\nRdhRRBqNCpRIMzCwa1u+eXIv/veNNWzYXhZ2HJFGoQIl0kyMHzMYDO6ZuTTsKCKNQgVKpJnIzcnm\nX4f35a/vrmfZp7vCjiNyxFSgRJqRH48eSKsWmo5DmoeMRO7czMYB9wLpwEPu/tsa72cBTwAnA1uB\ny9x9dSIziTRnHVu34JqR/blr5lKG/XoWW3bvo2dONhML87l4aG6o2Z57bwOTi0rYuL0sMpmimkuZ\nYhJWoMwsHbgfOBdYD7xjZi+4+0fVNvse8Jm7DzSzy4HbgcsSlUkkFXRrmwVAabzL+YbtZdw6bRFA\naD/knntvA7dOW0RZRWVkMkU1lzJ9IZEtqFOA5e6+EsDMngEuAqoXqIuA/4w/fxa4z8zMNcmNyGGb\nMnv5V9aVVVTyH88vZs3WvSEkgofmr/z8h1tUMkE0cyVTpslFJUlboHKBddVerwdOrWsbdz9gZjuA\nTsCW6huZ2TXANQB9+vRJVF6RZmFjHd3Md5Uf4O5Z0erhF8VMEM1cUcxU17+1xpLIAmW1rKvZMgqy\nDe4+FZgKUFBQoNaVSD165mTXei9Uz5yWzL/p7BASwRmTZrNxe/lX1oeZCaKZK7kyZSf0uInsxbce\n6F3tdS9gY13bmFkG0B7YlsBMIs3exMJ8sjPTv7QuOzOdmwqHkJZmoSw3FQ6JXKao5kqmTBML8xvx\nX+5XJbIF9Q4wyMz6ARuAy4F/qbHNC8CVwBvAN4HZuv4kcmQOXhOIUi+wKGaKai5l+oIlsh6Y2fnA\nPcS6mT/i7v9tZr8Cit39BTNrCfwvMJRYy+nyg50q6lJQUODFxcUJyywiIollZgvcvaCh7RJ6H5S7\nvwi8WGPdL6o9LwcuTWQGERFJThpJQkREIkkFSkREIkkFSkREIkkFSkREIkkFSkREIimh3cwTwcxK\ngTVh50iAztQY4knqpO8qOH1Xwem7CqYxvqe+7t6loY2SrkA1V2ZWHOS+ANF3dSj0XQWn7yqYpvye\ndIpPREQiSQVKREQiSQUqOqaGHSCJ6LsKTt9VcPqugmmy70nXoEREJJLUghIRkUhSgRIRkUhSgQqZ\nmfU2szlm9rGZfWhm48POFGVmlm5m75nZ38POEmVmlmNmz5rZkvi/rdPCzhRVZvaz+P+9xWb2dHwa\nIAHM7BEz22xmi6ut62hmM81sWfyxQ6KOrwIVvgPABHc/ChgOXGdmR4ecKcrGAx+HHSIJ3AtMd/ch\nwAnoO6uVmeUC1wMF7n4ssbnrLg83VaQ8Boyrse4W4GV3HwS8HH+dECpQIXP3Te7+bvz5LmI/SMKd\nZjSizKwXcAHwUNhZoszM2gEjgYcB3H2/u28PN1WkZQDZZpYBtAI2hpwnMtx9HrHJZKu7CHg8/vxx\n4OJEHV8FKkLMLI/Y7MJvhZsksu4BbgKqwg4Scf2BUuDR+OnQh8ysddihosjdNwB3AGuBTcAOd58R\nbqrI6+bumyD2CzbQNVEHUoGKCDNrA/wVuMHdd4adJ2rM7GvAZndfEHaWJJABnAT83t2HAntI4GmY\nZBa/fnIR0A/oCbQ2s2+Hm0oOUoGKADPLJFacnnL3aWHniagRwIVmthp4BjjbzJ4MN1JkrQfWu/vB\nlvizxAqWfNUYYJW7l7p7BTANOD3kTFH3qZn1AIg/bk7UgVSgQmZmRuxawcfuflfYeaLK3W91917u\nnkfsIvZsd9dvurVw90+AdWaWH191DvBRiJGibC0w3Mxaxf8vnoM6lDTkBeDK+PMrgecTdaCMRO1Y\nAhsBfAdYZGbvx9f9m7u/GGImSX4/BZ4ysxbASuDqkPNEkru/ZWbPAu8S61H7Hhry6HNm9jQwCuhs\nZuuB24DfAn82s+8RK/CXJuz4GupIRESiSKf4REQkklSgREQkklSgREQkklSgREQkklSgREQkklSg\nRBqRmVWa2fvVlkYbwcHM8qqPKi3S3Ok+KJHGVebuJ4YdQqQ5UAtKpAmY2Wozu93M3o4vA+Pr+5rZ\ny2b2QfyxT3x9NzP7PzNbGF8ODr+TbmYPxucvmmFm2fHtrzezj+L7eSakP6ZIo1KBEmlc2TVO8V1W\n7b2d7n4KcB+xkdmJP3/C3Y8HngKmxNdPAea6+wnExtH7ML5+EHC/ux8DbAf+Kb7+FmBofD/XJuoP\nJ9KUNJKESCMys93u3qaW9auBs919ZXxw4E/cvZOZbQF6uHtFfP0md+9sZqVAL3ffV20fecDM+ERx\nmNnNQKa7/9rMpgO7geeA59x9d4L/qCIJpxaUSNPxOp7XtU1t9lV7XskX15EvAO4HTgYWxCffE0lq\nKlAiTeeyao9vxJ+/zhdTjF8BzI8/fxn4EYCZpcdnya2VmaUBvd19DrEJHXOAr7TiRJKNfssSaVzZ\n1UalB5ju7ge7mmeZ2VvEfjH8Vnzd9cAjZjaR2Cy4B0cdHw9MjY8YXUmsWG2q45jpwJNm1h4w4G5N\n8S7Nga5BiTSB+DWoAnffEnYWkWShU3wiIhJJakGJiEgkqQUlIiKRpAIlIiKRpAIlIiKRpAIlIiKR\npAIlIiKR9P8BFlwC8omIyqkAAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ppn = Perceptron(eta=0.1, n_iter=10)\n", + "\n", + "ppn.fit(X, y)\n", + "\n", + "plt.plot(range(1, len(ppn.errors_) + 1), ppn.errors_, marker='o')\n", + "plt.xlabel('Epochs')\n", + "plt.ylabel('Number of misclassifications')\n", + "\n", + "plt.tight_layout()\n", + "# plt.savefig('./perceptron_1.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "#### A function for plotting decision regions" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "from matplotlib.colors import ListedColormap\n", + "\n", + "\n", + "def plot_decision_regions(X, y, classifier, resolution=0.02):\n", + "\n", + " # setup marker generator and color map\n", + " markers = ('s', 'x', 'o', '^', 'v')\n", + " colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')\n", + " cmap = ListedColormap(colors[:len(np.unique(y))])\n", + "\n", + " # plot the decision surface\n", + " x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n", + " x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n", + " xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),\n", + " np.arange(x2_min, x2_max, resolution))\n", + " Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n", + " Z = Z.reshape(xx1.shape)\n", + " plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)\n", + " plt.xlim(xx1.min(), xx1.max())\n", + " plt.ylim(xx2.min(), xx2.max())\n", + "\n", + " # plot class samples\n", + " for idx, cl in enumerate(np.unique(y)):\n", + " plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n", + " alpha=0.8, c=cmap(idx),\n", + " marker=markers[idx], label=cl)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUXWWZ5/HvU0mkCCkoA4lKJyTCcOkQ5FYEMVoRggwo\nrY3IalyrxwYyK0A7DA5jOyZZs+z2QuGtW4bGxAwlLQMNLWhsFo0IBKEkKoQgyC0ETUADQgAlqRBC\nSOqZP/Y+VafqXPY+dfY+e59zfp+1alXtffblPYdUPez3ed/nNXdHREQkbzqyboCIiEg5ClAiIpJL\nClAiIpJLClAiIpJLClAiIpJLClAiIpJLClAiIpJLClAiIpJLClAiIpJLE9O8uJl1A9cAcwEHLnD3\nX1Q6fsqUA3z//Wen2SQREcnY73637hV3nxZ1XKoBCrgSuMPdP2FmbwMmVzt4//1ns2zZQyk3SURE\nsnThhfZcnONSC1Bmti/QC5wH4O67gF1p3U9ERFpLmjmog4GXgWvN7Fdmdo2Z7TP2IDNbbGYPmdlD\n27e/nGJzRESkmaQZoCYCxwHL3f1Y4HXg82MPcveV7t7j7j1TpkR2SYqISJtIMwe1Gdjs7g+E27dQ\nJkBFmTjxLQ4+eDOTJ+9MtHFp2rGjk40bZ7B796SsmyIi0rRSC1Du/qKZ/d7MDnf3p4GFwJO1Xufg\ngzczc2YXXV2zMbPkG5owd2dw8FVgMxs2vDvr5oiINK20R/FdAtwQjuDbCJxf6wUmT97ZNMEJwMzo\n6tqfyZOVTxMRqUeqAcrdHwF66r1OswSngmZrr4hIHqmShIiI5JICVA2eeWY9Z5xxEjNm7MXVV38j\n6+aIiLS0tHNQDdV92gl0vLKlZP/QAdN57c619V+/eyqXX/5/+PGPf1T3tUREpLqWClAdr2xhaP/S\nuVTlgtZ4TJs2nWnTpnPXXf+RyPVERKQydfGJiEguKUCJiEguKUBF6O+/mpNPPoaTTz6GF198Ievm\niIi0jZbKQaVh0aJPs2jRp7NuhohI22mpADV0wPSKo/iS8NJLL3LaaT0MDm6jo6ODlSu/xf33P0lX\n176JXF9EREa0VIBKYih5Ne94xzt59NHNqd5DREQCykGJiEguKUCJiEguKUCJiEguKUCJiEguKUCJ\niEguKUCJiEgutVyAcq++PR6XXnoBc+ZMp7d3bv0XExGRWFoqQF13HSxfPhKU3IPt666r77rnnnse\nN910R/0NFBGR2FomQLnD9u2watVIkFq+PNjevr2+J6mTTuqlu3tqco0VEZFILVNJwgwuvjj4edWq\n4AvgrLOC/WbZtU1ERGrXMk9QMDpIFSg4iYg0p5YKUIVuvWLFOSkREWkeLROginNOZ50Fd94ZfC/O\nSYmISPNomQBlBlOmjM45XXxxsD1lSn3dfBde+Ek+/OGT+M1vnuboo2dwww39yTVcRETKaplBEgCf\n+lTwpFQIRoUgVW8O6jvfubH+xonkQPHvR7ltkTxpmSeogrG/bPrlEwkMDMDdd4+eJ3j33cF+kTxq\nuQAlIqXc4c034cEHR4LU3XcH22++qRyt5FNTdPG5O9ZEj0Ku33bJGTM49dTg5wcfDL4A5s0L9jfR\nr5e0kdw/Qe3Y0cng4KtN80ff3RkcfJUdOzqzborIKMVBqqDVglMatTglO6k+QZnZs8AgsAfY7e49\ntV5j48YZwGYmT3454dalZ8eOzrDdIvlR6NYrdvfdrROkBgaC7srC+ym83732gt7erFsn49GILr6T\n3f2V8Z68e/ckNmx4d5LtEWk7xTmnQrdeYRuaP0gV59hg9PubN0+jFZtVU+SgRKQ+ZsGTRHHOqdDd\nt9dezf/HWzm21pR2gHLgTjNz4DvuvnLsAWa2GFgMMHXqQSk3R6R99faWzhNM+o93lvOsCu+nEJxg\nfO9Pc8XyI+1BEvPd/TjgDODTZlbSE+zuK929x917pkyZlnJzRNpbmvMEs55nVSnHVstAiazfg4yW\naoBy9xfC71uAVcC8NO8nItnIep7V2Bzb0qXB9+L25P09SKnUuvjMbB+gw90Hw59PA76Y1v1EJDtZ\n54CSyLFl/R6kVJo5qHcAq8IJthOBf3V3rZsukpKscydJ5YDGK06OLeozyvo9yGipdfG5+0Z3Pzr8\nOtLdv5LWvUTaXR5yJ0nkgOpVLccW5zPKw3uQEbmvJCEi1eUhd5JEDijt9kV9Rnl/D+1I86BEmlwe\ncid5n2cV9zPK83toRwpQIi0gqdzJ0BB0dFTerqa3Nzi+OAe0cGH885NQLccU5zNKIo8lyVEXn0gL\nSCJ30t8Py5cHQQaC78uXB/vjGBiA1atH53hWr25cHiwqxxT3M6o3jyXJUYASaXJJ5E6GhmDnTli/\nfiRILV8ebO/cORK0qrUh63lQ1e4/NKR5Us1IXXwiTS6J/E9HB1x8MXz720FQuvTSYP/hhwf7o7rp\nksqDRXWfVdqOc3/Nk2o+eoISaQG9vaP/SBb+mNayzMT998Nhh43uvjrssGB/HPWuNxXVfRb1etT9\nk/iM2mFNrTxRgBJpEfXU2XMPuvJuvz3oDnMPvt9+e7A/bhfYePNgcbro4g4Tr3b/emsRap5UY6mL\nT0Rwhw0bYNcueNvbYNo0ePnlYHvDhuinhHrXm4rTfVbtdUh/vatWX1MrjxSgRGKqd3jxePMrSao0\njLyjA/beG+bMga1bg9emTYP99gv2F86plgOKk+OpZxh41Otpz2HK+1yvVmSeo2fTWbN6fNmyh7Ju\nhkiJepcTjzq/EcuV9/cH3XWFQQ+FkXqdnbBoUXDPu+6CtWtHzjnhBPjQh4I2xWljtQAUdX7xE0pB\ncTCIej3q/knRPKj6XXihrXP3nqjjlIMSiVDv8OIk8iv1ihpGvmdPcM+1a0cPw167Ntgft42Vcjz1\nDgOPO0w8zfWuKl1TwSk96uITiVDv8OJ68ytJ/AEsDCMvBKXCMPIjjhh5oqrWfdXRkf5nEHV/da+1\nHz1BicQQZ3jx2CedsaPHqp3fiOHLhSBVrHiOU9Qw7LQ/g6j79/YGpZPGllIq7gKtdn9pPgpQIjFE\nDS+ut8xOI4YvF7r1ihWXNoLq3VdpfwZR948qpaQyRK1HAUokQlQpoaj8TFL5lXoU55yOOAKuvDL4\nXpyTyvIzaIY8njSeclAiEeLkR/KeX+noCEbrFeecCjmpzs54pYzS/AyaIY8njadh5iIxxZm3dPnl\nI9tLl9Y376mR86CSOj/qM6j3/nE+42qvSz5omLlIwurJz0SdH2c7CWODQS3BISoHFCdHVc9yHHnI\n40ljqYtPpE7tUAKnOAcEo9/jvHnBk9Dq1ZU/g4ULq58f9bQY9RkvXFj9/q3w36AdKUCJ1KkdSuAk\nMY+p3nlUWefxpPGUgxJJSL05piTyM40o81Nvnq2eHFEe8nhSP+WgRBosag5PtTk6SSy3nvYcoHrz\nbEnkiPKQx5PGUYASSVnUHJ49e/K/3HrUPKg485jSnuslrUc5KJGUFfIh7qPzLyecMJIvibPcerWl\nLtKeA5TEPCbliKRWClAiDfCzn1Xe39s7stz6+vUjS0sUlluPsxxH4Q9+pbWSklBYEmNsrby496j3\nfGk/6uITSZl70FW3ejUMDgb7BgeD7UIXXrXl1pNa7jwJ9eZ4lCOSWugJSqRBCk8+ha/CH2f36OXW\ns17uXCQLFQOUmd0a4/w/uvt5yTVHpHlVGiZuFtS7O+WUIGiYQVdXEEw6O2HChHjLrS9cOLoLr3jp\nibj5HQ3TlmZS7Qnqz4H/WuV1A66OuoGZTQAeAp539zNra55Ic4haTv0DHwiWUy/Ov0CwH+CCC0qX\nWz/ssGC5dQhyUI89NhIw3IPrH3VUkNuJk9/Jw7LzIrWoloNa5u73Vfm6F/iHGPe4FHgqkdaK5FAS\ny6lXe33PniA4rV8Pb387LFkSfF+/PthfGIYeNQdJy1VIs6n4BOXu3486OeoYM5sBfAT4CnBZza0T\naQJJLKde7fUJE2Du3GD7tdegr2/k+nPnxqs2oeUqpBlFDpIwsx5gGTArPN4Ad/f3xLj+t4DPAV1V\nrr8YWAwwdepBMS4pMj5p5lcKQaoQnKB0OfWhodFdcAsXjn69WhfdggXw/vfDFVeMXP+ii4LgFVfU\nUPRGDFUXqUWcYeY3ANcCZwN/AZwZfq/KzM4Etrj7umrHuftKd+9x954pU6bFaI5I7dIuBRS1nHqc\npSaqddHddx+sWDH69RUrgv1xabkKaTZxAtTL7n6ru29y9+cKXzHOmw981MyeBW4CTjGz6+tprMh4\npF0KKGo59T176rv/0BA8/nhwve7uIAfV3R1sP/54dCmkwmeQ9bLzIrWKMw/qC2Z2DbAaeLOw091/\nWO0kd18CLAEwsw8Cn3X3vx5/U0XGJ+38StRy6hMm1Hf/jo5gtB7An/4U5KDcg/sddVT8HJSWq5Bm\nEydAnQ8cAUwCCv+v5kDVACWSJ2nnVxYtCp6UCsGio2N0jqje+/f2BjmowgCJQv2+WpbjiMpzqRSR\n5E2cf95Hhzmiv3H388OvC2q5ibvfqzlQkqW08ysDA3DPPaPzOffcE3859CiFnFWx4pxWXFquQppJ\nnCeoX5rZHHd/MvXWiKQg7SXZ610OPer+7bCkvEg5cQLU+4G/MbNNBDmoWoaZi2Qu7aUe6l0OXUtV\niJQXJ0CdnnorRFKWdn6lMK+pUq08LVUhUrs4AepdwBPuPghgZl3AHCDOUHOR3EgzvxJVKy+J+ys/\nJO0mziCJ5cD2ou3Xw30iQpBjilMrT0RqE+cJytxHxgq5+5CZaR0pkVBHR7xaeVrKQqQ2cZ6gNprZ\nfzezSeHXpcDGtBsm0kwWLAjmJRW7+OJgP6RfakmkFcUJUBcB7wOeBzYDJxIWdxWRQLV5SmmXWhJp\nVZFdde6+BTi3AW0RaUpx5ilpKQuR2lV8ggqXwagqzjEiSRj7lJGnp45K85TmzRuZp1Q8d6lgbHDK\n83sUyUK1J6jPm9krVV43gtVyVybbJJHRmmEp8qh5SpVKHRWOaYb3KNJo1QLUfUSv+3RXgm0RKRFV\nRihPI+EqzVOK6gJcuLB53qNII1Vb8v38RjZEpJxWWIo8zlIXzf4eRdKg+UySe62wFHlUF2ArvEeR\npNWwmoxINlplKfJqpYpa5T2KJElPUJJr7bDURDu8R5HxiAxQZrYXcDYwu/h4d/9ies0SCbTDUhPt\n8B5FxiPOE9S/A1uBdQTrQYkkrlqdut7eoOBqcb5m4cLaljvPOy2nIVIqToCa4e5aE0pSEzUHqNzr\nq1e33hwhLachMlqc/wf9uZkdlXpLpC1F1akbGlIdO5F2VfEJysweAzw85nwz24iWfJeExZnnpDlC\nIu2pWhffmQ1rhbS1qDlAmiMk0p4qdvG5+3Pu/hzw5cLPxfsa10RpdVFzgDRHSKQ9xRkkcWTxhplN\nAI5PpznSbuLUqVu9WnOERNpRtRzUEmApsLeZbSvsBnahCuaSkDh16jRHSKS59fcH37f8dhu89lrs\n88wj+knMrM/dl9TTuLhmzerxZcseasStJGeqzYOKsy0i+dC3dBvsfGNkx85g+uz8znUAXH/kFdiD\nD65z956oa8Xp4rvZzI4bs28r8Jy7747ZZpGqouYAaY6QSL4MDMCam5+HPXtKXtt0/DmjdyxaVLwx\nesRTFXEC1LeB44BfE3TxHQU8CuxvZhe5+52x7iQiIk1lYACeeWZke8sTLw0/EUHwVHT9VS+XOXNR\nmX21ixOgngUWufsTAGY2B/g74EvADwEFKMkFdQOKjF9fHzC4bWTH4CDs2TPcNQdwKHD9uT9oWAmX\nOAHqiEJwAnD3J83sWHffaFV++82sExgA9grvc4u7f6HeBouUoyXTRWrT3w9bHhndRbdp5oKRA7qA\nJeWGHzTuFypOgHrazJYDN4XbfwVsCKucv1XlvDeBU9x9u5lNAu43sx+7+y/ra7LIaM20LLxIowwM\njPy8Zg3w+9+VHLOkewWLL59dvCftZtUkToA6D/hb4DMEOaj7gc8SBKeTK53kwfDA7eHmpPBLUysl\ncSqHJFLURbfzDdj5JtMnvDr82nTggeMvGjNYAYJVlPIrMkC5+xvAN8OvsbaX2TcsnNS7DvhPwNXu\n/kCZYxYDiwGmTj0oRpNFSqkckrSTvstegreKBlGH3XSbZi4Iuubmzy/Tt53MwIVGirNg4Xzg74FZ\njF6w8OCoc919D3CMmXUDq8xsrrs/PuaYlYQTf2fN6tETloxLpXJIClLSrAoj6LasK+2amz7hVR44\nZ8wzQ28veeuiq1ecLr5+4H8QPAmVDniPwd1fM7N7gdOBxyMOF6mJlkyXZtbfD1u2FO14IRi4UOii\nW9J985g8UUHrj/6JE6C2uvuPa72wmU0D3gqD097AqcBXa72OSBQtmS7NolKVhSXdK0b2dTEmIBX/\n3F7iBKifmtnXCeY8Dc/QcveHI857F/C9MA/VAXzf3W8bd0tFqtCS6ZJHfX2UjJ7bdPw5cOihIzt6\ne2nnIFRNnAB1Yvi9uG6SA6dUO8ndfw0cO852idRM5ZCkkUqqLFQohLpp5oIx84mab7BCVuKM4qs4\nlFxEpB3094dlfgre2j1cZaG3c+3w7sXn/q7M6LnWGrjQSHFG8b0DuBw40N3PCEsdneTu/am3TkQk\nA+UKoW6auQCmTx85aHhO0eyiM4t/lnrF6eL7F+BaYFm4vQH4N4LRfSIiTae/6K/Xli2UrbJQWghV\nT0KNFidAHeDu3w8XMMTdd5vZuIabi4hkoVyVhUMnPQuEBVBnXlah7pxkKU6Aet3M9icsU2Rm7yVY\nD0pEJHf6Lhu9JERB9SoLCk55FCdAXQbcChxiZmuAacAnUm2ViEgVkVUWzv2mglALiDOK72EzWwAc\nTlAs9ml3r1bFXEQkEf394fDtgjGFUNu5ykI7qBigzOzjFV46zMxw9x+m1CYRaUN9S7cFi+QVhCPo\nlnSvYHHXjcG+kjWKZjeqeZKBak9Qf1HlNSeoLCEiElvxGkXluugqV1lQ91w7qhig3P38RjZERFpL\nX1/RxuBIlYXidYo2zfy4qixIRXEGSYiIRBrVRVeuysLxq8ssmKcnI6lMAUpEalKuykLBcBfd8Ai6\n2UWv6ulIaqMAJSIlBgZgzZpwI1aVhQIFIUnOeEbxAWgUn0iL6OsDXq5cCHXxzBtVZUEyoVF8Im2k\napWFioVQFZwkGxrFJ9JiqhVCVZUFaSaxclBm9hHgSKCzsM/dv5hWo0Qk2sAArPnRS6N3liuEevwV\nY0bPqcqCNIc460GtACYDJwPXENThezDldolIGWO76JZ0r2DxIatHDhg1gq5AAxekOcV5gnqfu7/H\nzH7t7v9gZt9E+SeRxPWPWWEtXiHU2SgASauKE6DeCL/vMLMDgVeBd6fXJJH2MKoQalhlYX7nuuHX\nF3WvVSFUaWtxAtRtZtYNfB14mGAE3zWptkqkxfQtHSn1U2y4EOoh08tUWZjdkLaJ5FWcAPU1d38T\n+IGZ3UYwUGJnus0SaT6FQqhVqyyUDUIaQSdSTpwA9QvgOIAwUL1pZg8X9om0m4EBWHPHttE7iwqh\nzp/0rKosiCSgWiWJdwJ/BuxtZscSLFYIsC/BqD6RltffD1seeX70zrDKwvXTLhvZd3q5ZcRFpB7V\nnqD+M3AeMAP4x6L924ClKbZJpKGK1yiC0i66TTMXwPz5IwcMByJ1zYmkqVolie8B3zOzs939Bw1s\nk0hqRq1RBMNVForXKCrtolMgEslCnBzUGjPrBw509zPMbA5wkrv3R50okpXhKgtv7R7ZuWcP0ye8\nyqKum4d3qRCqSH7FCVDXhl/Lwu0NwL8BClCSK2OrLMzvXMf154x5+B9eQrxAwUkkr+IEqAPc/ftm\ntgTA3XebWekYWpEU9feHhU8LyqxRVL4QqgYuiDSrOAHqdTPbn2CCLmb2XmBr1ElmNhO4DngnMASs\ndPcr62irtJH+ftjyRFgINXwqWtK9YuSAblRlQaTFxQlQlwG3AoeY2RpgGkHB2Ci7gf/p7g+bWRew\nzszucvcnx99caUV9fZR9IhpVCHXRIlRZQaS9RAaoMMAsAA4nmAv1tLu/FeO8PwB/CH8eNLOnCOZV\nKUC1oUIh1C1PVFgwr2KVBU1uFWlXcZbb6AT+Fng/QTffz8xshbvHLndkZrOBY4EHxtdMaSZ9S7fB\nzjdGdoQBaX7numB9om+ryoKIRIvTxXcdMAhcFW5/Evh/wDlxbmBmU4AfAJ9x921lXl8MLAaYOvWg\nOJeUnOjvL78kxPzOdVx/5BUjO554Avbaa2S7MHCuqwuWJjTn+/LLYXCwdH+S9xCRhooToA5396OL\ntn9qZo/GubiZTSIITje4e9k1pNx9JbASYNasHo9zXWmcgQF45pmR7S2PjK6ysKR7RYXBCkVPREuW\nwJQppYeUCyjjNTiY/j1EpKHiBKhfmdl73f2XAGZ2IrAm6iQzM4K5Uk+5+z9GHS/Z61tavgBq8RpF\nh05iTJWF2ek3TETaUpwAdSLwKTMr9OUcBDxlZo8B7u7vqXDefOC/AI+Z2SPhvqXufntdLZbE9PUB\nL4SFUMOnok0zF4wc0IWqLIhIZuIEqNPHc2F3v5+RCujSYCUFUH9UYfTczAVjgpACkojkQ5xh5s81\noiFSv+FCqC88P1x3rmA6lKmyAApIIpJXcZ6gJGcGBsIlIYqFAemBAz8OB1Khay6jKgtdXZVH2DXT\nPUSkoRSgcq5qlYXTx+zv7SWXT0SNGOatoeQiLUcBKgcqrVEEYQHUmR8v80Q0G42gK5L2PKhLLoHd\nu0v3T5wIV11Vur9WmsclUkIBqsH6Lntp9I5YhVBz+FSUN2nPg9q9OwhG5fYnQfO4REooQKWsb+m2\n4flEED4RHXPRyAGHHlpmjSIREVGAqkP/mCUbx1ZZKCgthKq6cyIiURSgajC8jDiMKoBasKhrbXTZ\nn1YUlT+pN39z0UXgZapgmcGKsGt069bgS0RahgJUBbEKoZZdHqINReVP6s3fuAfBqNz+4p/TzBGJ\nSMO1dYAaXqPot6PzRAWlVRYKWvyJqBmZle1eLRvYxmPixMpPgUnQPC6REm0ToKoVQj0UuP7cH6jK\nQjPbb7/yT3Hbtydz/SSGklejoeQiJVouQA1XWRjzf9PDVRYKDpk+posuoyoLzSDtOTpxckzlXi/2\nxz8GX5VE5cHSzqNF0TwokRJNGaCiCqHO71w3ZkmIAj0RjUvac3Ti5JjqFZUHSzuPFkXzoERK5D5A\n9fUBg0Xdc2HXXHEh1PmTnq3QRScNEZU/STt/IyItKVd/IV7c/BZ9lxQVQa1pjSIFp4rq7T6K6j5L\nogsqyaelcoaGYNeu9K6vLjqRxOUqQE30t9h0zv8avTOvBVCbSRLdR/V0wSXRPZZEF2Ca3YjqohNJ\nXK4C1J/PfF3ddCIiAuQsQEmOpd0F14j7V7tG1nk0zYMSKaEA1Q6ickhR+ZOOjupddPXmX6ZOrX8O\nU1Qbo+4R1c53vat6AKm31JLyVCIlFKDaRbX8S735k0bkX9Iehh4l6j2q1JJI4hSg2kU9f8zr7d6K\nOj9O91ZU+5O4Rz3SLrUk0oYUoNpBVPdXlHorJUSdH9W9FacLsN571CvtUksibUgBSvJPS2lonpW0\nJQUoyT/ldzTPStqSAlQ7SDs/0w75naj3qGHiIolTgGoHaedn2iG/E/Ue1c0mkjgFqFag/ISItCAF\nqFag/ISItCAFKMk/5Xf0GUhbSi1Amdl3gTOBLe4+N637SBtQN6U+A2lLaT5B/Qvwz8B1Kd5DGkE5\nLhHJQGoByt0HzGx2WteXBlKOS0QyoBxUK1B+QkRaUOYByswWA4sBDpo6NePWNCl1s4lIC+rIugHu\nvtLde9y9Z1q5biQREWlLmQcoERGRclILUGZ2I/AL4HAz22xmi9K6l6SsqysoKzT2SzkuEUlRmqP4\nPpnWtaXBlOMSkQyoi09ERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpE\nRHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJ\nAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpERHJJAUpE\nRHJJAUpERHJJAUpERHJJAUpERHIp1QBlZqeb2dNm9hsz+3ya9xIRkdaSWoAyswnA1cAZwBzgk2Y2\nJ637iYhIa0nzCWoe8Bt33+juu4CbgI+leD8REWkhE1O89p8Bvy/a3gycOPYgM1sMLA43t9uFFz6d\nYpuydgDwStaNaHL6DJOhz7F++gzHb1acg9IMUFZmn5fscF8JrEyxHblhZg+5e0/W7Whm+gyToc+x\nfvoM05dmF99mYGbR9gzghRTvJyIiLSTNALUWONTM3m1mbwPOBW5N8X4iItJCUuvic/fdZvbfgJ8A\nE4DvuvsTad2vSbRFV2bK9BkmQ59j/fQZpszcS9JCIiIimVMlCRERySUFKBERySUFqAYyswlm9isz\nuy3rtjQjM3vWzB4zs0fM7KGs29OMzKzbzG4xs/Vm9pSZnZR1m5qJmR0e/vsrfG0zs89k3a5WleY8\nKCl1KfAUsG/WDWliJ7u7JkeO35XAHe7+iXB07eSsG9RM3P1p4BgYLuf2PLAq00a1MD1BNYiZzQA+\nAlyTdVukPZnZvkAv0A/g7rvc/bVsW9XUFgK/dffnsm5Iq1KAapxvAZ8DhrJuSBNz4E4zWxeWyJLa\nHAy8DFwbdjVfY2b7ZN2oJnYucGPWjWhlClANYGZnAlvcfV3WbWly8939OIIK+Z82s96sG9RkJgLH\nAcvd/VjgdUDL4IxD2D36UeDmrNvSyhSgGmM+8FEze5agqvspZnZ9tk1qPu7+Qvh9C0G//7xsW9R0\nNgOb3f2BcPsWgoAltTsDeNjdX8q6Ia1MAaoB3H2Ju89w99kE3QL3uPtfZ9yspmJm+5hZV+Fn4DTg\n8Wxb1Vzc/UXg92Z2eLhrIfBkhk1qZp9E3Xup0yg+aRbvAFaZGQT/bv/V3e/ItklN6RLghrCLaiNw\nfsbtaTqVqCQcAAADwklEQVRmNhn4EHBh1m1pdSp1JCIiuaQuPhERySUFKBERySUFKBERySUFKBER\nySUFKBERySUFKJEamNkHy1Wjr7Q/gfv9pZnNKdq+18x6YrRxq5ndnsD99w6rdu8yswPqvZ5ILRSg\nRPLtL4E5kUeV+pm7f7jem7v7G+5+DPBCvdcSqZUClLSUsOLEf5jZo2b2uJn9Vbj/eDO7Lyw0+xMz\ne1e4/14z+5aZ/Tw8fl64f16471fh98Or3bdMG75rZmvD8z8W7j/PzH5oZneY2TNm9rWicxaZ2Yaw\nPf/XzP7ZzN5HUO/t6+FTzCHh4eeY2YPh8R+I2abPhWtpPWpmVxS9938ys4FwbagTwvY9Y2Zfjvt+\nRdKiShLSak4HXnD3jwCY2X5mNgm4CviYu78cBq2vABeE5+zj7u8Li89+F5gLrAd63X23mZ0KXA6c\nHbMNywjKWV1gZt3Ag2Z2d/jaMcCxwJvA02Z2FbAH+N8EdfEGgXuAR93952Z2K3Cbu98Svh+Aie4+\nz8w+DHwBOLVaY8zsDIInsRPdfYeZTS16eZe795rZpcC/A8cDfwR+a2b/5O6vxnzPIolTgJJW8xjw\nDTP7KsEf9p+Z2VyCoHNX+Ad+AvCHonNuBHD3ATPbNwwqXcD3zOxQgmU+JtXQhtMIigN/NtzuBA4K\nf17t7lsBzOxJYBZwAHCfu/8x3H8zcFiV6/8w/L4OmB2jPacC17r7DoDCfUK3ht8fA55w9z+EbdgI\nzAQUoCQzClDSUtx9g5kdD3wY6DOzOwkqnz/h7pWWNx9b78uBLwE/dfezzGw2cG8NzTDg7HD11ZGd\nZicSPDkV7CH4HbQark3RNQrnx2lPpZpmhWsNjWnbUMxri6RGOShpKWZ2ILDD3a8HvkHQbfY0MM3M\nTgqPmWRmRxadVshTvR/YGj7h7EewnDfAeTU24yfAJRY+rpnZsRHHPwgsMLO3m9lERnclDhI8zdXj\nTuCCsMgpY7r4RHJLAUpazVEEOZ9HCHJBX3b3XcAngK+a2aPAI8D7is75k5n9HFgBLAr3fY3gCWwN\nQZdgLb5E0CX4azN7PNyuyN2fJ8hxPQDcTbAExtbw5ZuAvwsHWxxS4RJVhVXfbwUeCj+Xz0acIpIL\nqmYubc3M7gU+6+4PZdyOKe6+PXyCWgV8191XjfNaHyR4T2cm2L5ngR53fyWpa4pE0ROUSD78ffh0\n8ziwCfhRHdfaBcxNcqIuwRPhUL3XE6mFnqBERCSX9AQlIiK5pAAlIiK5pAAlIiK5pAAlIiK5pAAl\nIiK59P8BVGnEoi1X/oUAAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plot_decision_regions(X, y, classifier=ppn)\n", + "plt.xlabel('sepal length [cm]')\n", + "plt.ylabel('petal length [cm]')\n", + "plt.legend(loc='upper left')\n", + "\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Adaptive linear neurons and the convergence of learning" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "### Implementing an adaptive linear neuron in Python" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "class AdalineGD(object):\n", + " \"\"\"ADAptive LInear NEuron classifier.\n", + "\n", + " Parameters\n", + " ------------\n", + " eta : float\n", + " Learning rate (between 0.0 and 1.0)\n", + " n_iter : int\n", + " Passes over the training dataset.\n", + "\n", + " Attributes\n", + " -----------\n", + " w_ : 1d-array\n", + " Weights after fitting.\n", + " errors_ : list\n", + " Number of misclassifications in every epoch.\n", + "\n", + " \"\"\"\n", + " def __init__(self, eta=0.01, n_iter=50):\n", + " self.eta = eta\n", + " self.n_iter = n_iter\n", + "\n", + " def fit(self, X, y):\n", + " \"\"\" Fit training data.\n", + "\n", + " Parameters\n", + " ----------\n", + " X : {array-like}, shape = [n_samples, n_features]\n", + " Training vectors, where n_samples is the number of samples and\n", + " n_features is the number of features.\n", + " y : array-like, shape = [n_samples]\n", + " Target values.\n", + "\n", + " Returns\n", + " -------\n", + " self : object\n", + "\n", + " \"\"\"\n", + " self.w_ = np.zeros(1 + X.shape[1])\n", + " self.cost_ = []\n", + "\n", + " for i in range(self.n_iter):\n", + " output = self.net_input(X)\n", + " errors = (y - output)\n", + " self.w_[1:] += self.eta * X.T.dot(errors)\n", + " self.w_[0] += self.eta * errors.sum()\n", + " cost = (errors**2).sum() / 2.0\n", + " self.cost_.append(cost)\n", + " return self\n", + "\n", + " def net_input(self, X):\n", + " \"\"\"Calculate net input\"\"\"\n", + " return np.dot(X, self.w_[1:]) + self.w_[0]\n", + "\n", + " def activation(self, X):\n", + " \"\"\"Compute linear activation\"\"\"\n", + " return self.net_input(X)\n", + "\n", + " def predict(self, X):\n", + " \"\"\"Return class label after unit step\"\"\"\n", + " return np.where(self.activation(X) >= 0.0, 1, -1)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4VGXax/HvnRBIaIlAEBKqNEV6R7AXVBARRZoCFnDt\nbXFl3WJZF3d5sSsKUiwgKlIUC6ILKio99F6FhA6hJpByv3+cEwyQkEnI5MxM7s91zZXMmXNmfmkn\n9zznKaKqGGOMMcaEkjCvAxhjjDHGFDYrcIwxxhgTcqzAMcYYY0zIsQLHGGOMMSHHChxjjDHGhBwr\ncIwxxhgTcqzA8SMRGSAic3zc91kR+cj9vIaIHBGRcP8mDFwi8lcRec/rHMYEAjuXFJydS4ovK3AK\nQERmi8gBESnlj+dX1d9VtayqZvjj+c9GRMaJyL+K+nVPp6r/VtV7vc4Bp/7DOIfneFxEdorIQREZ\nc7bfHRG5WkTWiMgxEZklIjWzPXa7iPzqPjb7XDIZ79m5xP/sXJLruaSUe/wh9/meyMexQXEesgIn\nn0SkFnApoEBXT8MEKREp4XWGLEWRRUQ6AU8DVwO1gAuA53LZtxIwGfg7UAFYCHySbZf9wKvAS/5L\nbIqCnUvOnZ1Lzulc8ixQD6gJXAk8JSLX+3hsUJyHrMDJv37AXGAc0D/7AyJSUUS+cCvi+UCd0x5/\nTUS2uY8vEpFLc3oBEaklIpr1B+O+y3tBRH4RkcMi8p37C5i1fzu3mk4WkaUickXhfsknX+dCEZkp\nIvtFZK2I3J7tsc4ikuB+bdtE5Nkcvp57ROR34H/ZtvUXkd9FZK+IPJPtmOzN7HntGyUi77vvhFeL\nyFMisv0sX4eKyIMish5Y727L8Wfj/sH/FegpTlP/Und7tIiMFpEdIpIoIv+S3C8D9AdGq+pKVT0A\nvAAMyGXf7sBKVf1MVVNxTkJNReRCAFX9XlU/BZJy+/pM0LBziZ1LPDuX4Pz+vaCqB1R1NTAq23OF\nxHnICpz86weMd2+dROT8bI+9BaQCVYG73Vt2C4BmOBXxBOAzEYn08XX7AHcBlYGSwJ8BRCQe+Ar4\nl/u8fwY+F5HYfH9lZyEiZYCZbu7KQG/gbRG52N3lKM73JgboDNwvIt1Oe5rLgYuATtm2dQQa4Lwj\n+YeIXHSWGLnt+0/+eDdzLXCHD19SN6At0NC9n+PPRlW/Bf4NfOI29Td1938fSAfqAs2B64DcmsEv\nBpZmu78UOF9EKua1r6oeBTa6201osXOJnUvAg3OJiJwHxOXwXBfndWwuuQKSFTj5ICIdcZrzPlXV\nRTg/8D7uY+HArcA/VPWoqq7A+cU9SVU/UtV9qpquqsOBUjh/ZL4Yq6rrVDUF+BTnDwicP8CvVfVr\nVc1U1Zk4zYk3nttXe4YuwBZVHevmXwx8DtwGoKqzVXW5m2EZ8DHOSSi7Z93vTUq2bc+paoqqLsX5\ng2pK7nLb93bg3+47ke3A6z58PUNVdX9Wlvz8bNx/RDcAj7lfz27gFaBXLq9VFjiY7X7W5+V82Ddr\n/5z2NUHKziV2LgFPzyVlTzs++2N5HRs0rMDJn/7Ad6q6170/gT+almOBEsC2bPtvzX6wiDzpNnse\nFJFkIBqohG92Zvv8GH/8gtYEerhNysnu83bEeed3ChHp6zaNHhGRb3x83Sw1gbanvU5foIr73G3F\n6Yi2R0QOAn/K4Wvbxply+7pyktu+cac9d06vc7pT9snnz6YmEAHsyPa9eBfn3WhOjgDls93P+vyw\nD/tm7Z/TviZ42bnEziXg3bnkyGnHZ38sr2ODRsB00Ap0IhKFU92Hi0jWH0cpIEZEmgIrcJoZqwNr\n3MdrZDv+UuAvOE2iK1U1U0QOAHKO0bYBH6rqwLx2VNWs5vCCvs6PqnptLo9PAN4EblDVVBF5lTP/\nqP21dP0OoBqwyr1f3YdjTmbx4Wdzeu5twHGgkqqm+/BaK3HeIX7q3m8K7FLVfbnse7I/htucX8fd\nbkKAnUvsXJKNJ+cSVT0gIjvc42dme66VeR3rQ8aAYS04vusGZOBcZ23m3i4Cfgb6ucMwJwPPikhp\nEWnIqR0Hy+GctPYAJUTkH5xZIRfER8BNItJJRMJFJFJErhCRaufwnFnPk3UrCUwH6ovInSIS4d5a\nZ7t2XQ7Y756Q2uA2txeRT4EhInKe24/goXwen9fPZhdQS0TCAFR1B/AdMFxEyotImIjUEZHTm9Gz\nfADcIyIN3Wvff8PpWJqTKUAjEbnV7VPxD2CZqq4B5/KFu70EEOb+fCLy+fUab9m5xM4lnp9L3Of6\nm/u1XggMzPZcIXEesgLHd/1xrl3/rqo7s2447zT6ijNK4SGcps6dOL8oY7MdPwP4BliH09ycim/N\nn2elqtuAm3F65+9xn3Mw5/azfRpIyXb7n6oexun81gun5/xO4D847zwBHgCeF5HDOH8Mn57+pH70\nPLAd2Ax8D0zCeVfkq7x+Np+5H/eJyGL38344HTRXAQfc1zyjKR/A7Vz4X2CW+/xbcTozAiAiK0Wk\nr7vvHpz+Fy+6z9uWU6/H34nzMxmBM8Q4BWf0gwkedi6xc0kgnEv+idP3ayvwIzDMff6QOQ+Jqr9a\n+ozxhojcD/RS1dzeBRljTJ7sXBLcrAXHBD0RqSoiHdzm3QbAkzhNrMYY4zM7l4QW62RsQkFJnJEH\ntYFkYCLwtqeJjDHByM4lIcQuURljjDEm5PjtEpXbq3q+ONN9rxSR59zttUVknoisF5FP3F71xhhj\njDGFxm8tOCIiQBlVPeIOH5sDPAo8AUxW1Yki8g6wVFVHnO25KlWqpLVq1fJLTmNM4Vm0aNFeVS3U\nqf2Lip1njAkOvp5n/NYHR53KKWu2xAj3psBV/DGvwfs4i3idtcCpVasWCxcu9E9QY0yhEZGtee8V\nmOw8Y0xw8PU849dRVO5kQEuA3TizJW4EkrPN2LgdiM/l2EEislBEFu7Zs8efMY0xxhgTYvxa4Khq\nhqo2w5n6ug3ObJ1n7JbLsSNVtZWqtoqNDcoWb2OMMcZ4pEjmwVHVZGA20A5nvZWsS2PVcGayNMYY\nY4wpNP4cRRUrIjHu51HANcBqnCmmb3N36w9M81cGY4wxxhRP/pzoryrwvoiE4xRSn6rqdBFZBUwU\nkX8BCcBoP2YwxhhjTDHkz1FUy4DmOWzfhNMfxxgTwKYmJDJsxlqSklOIi4licKcGdGue45gAz4nI\nFuAwzird6araSkQqAJ8AtYAtwO2qeuBcXieYvifGFHe2FpUx5gxTExIZMnk5ickpKJCYnMKQycuZ\nmpDodbSzuVJVm6lqK/f+08APqloP+MG9X2BB+j0xptiyAscYc4ZhM9aSkpZxyraUtAyGzVjrUaIC\nuRlnri3cj93O5clC5HtiTLFhBY4x5gxJySn52h4AFPhORBaJyCB32/mqugPA/Vj59IPyM99WEH5P\njCnWrMAxxpwhpnREjtvjYqKKOInPOqhqC+AG4EERucyXg/Iz31ZuX3sAf0+MKdaswDHGnOLLpUkk\nH0sjTE7dHhURzuBODbwJlQdVTXI/7gam4Axk2CUiVQHcj7vP5TUGd2pAVET4KdtKhocF7PfEmOLO\nChxjzEnTliTy6MQEWteuwNBbGhMfE4UA8TFRDO3eOCBHDIlIGREpl/U5cB2wAvgCZ64tKIQ5t7o1\nj2do9z++JyXChDKlwrm+UZVzeVpjjJ/4cx4cY0wQmZKwnSc/XUqb2hUYM6A1pUuWoGebGl7H8sX5\nwBQRAeecNkFVvxWRBcCnInIP8DvQ41xfqFvz+JNF3q8b99Jn1DxGzN7I49fWP9enNsYUMitwjDFM\nWrSdwZOW0v6Ciozu35qokuF5HxQg3Lm1muawfR9wtb9e95I6lbipaRwjftxI9xbx1KxYxl8vZYwp\nALtEZUwx9+mCbQyetJQOdSoFXXHjtWduvIiIMOG5L1d5HcUYcxorcIwpxibO/52nPl9Gx7qVeK9/\nKytu8qlKdCSPXlOP/63ZzferdnkdxxiTjRU4xhRT4+dt5enJy7miQSyj+rUiMsKKm4K4q0Nt6lUu\ny7NfriT1tIkAjTHesQLHmGLow9+28MyUFVx1YWXevbOlFTfnICI8jOduvpjtB1IYMXuj13GMMS4r\ncIwpZsb9spm/T1vJNRdVZsQdLShVwoqbc5W9w/HWfUe9jmOMwQocY4qV0XM28+yXq7iu4fm83bel\nFTeFKKvD8fPW4diYgGAFjjHFxKifNvHC9FVcf3EV3urbgpIl7M+/MGV1OP7BOhwbExDsDGdMMfDO\njxt58evVdG5clTf6NCci3P70/SGrw/Fz063DsTFes7OcMSHurVkbeOmbNXRpUpXXejWz4saPsjoc\nb9tvHY6N8Zqd6YwJYW/8sJ5hM9Zyc7M4Xu3ZjBJW3PiddTg2JjDY2c6YEPXq9+sYPnMdtzSP5+Xb\nrbgpStbh2Bjv2RnPmBCjqrw8cx2vfr+eW1tU4/96NCU8TLyOVaxYh2NjvGcFjjEhRFUZ/t06Xv9h\nPT1aVuO/tzWx4sYj1uHYGG9ZgWNMiFBV/jtjLW/O2kCv1tX5z61W3HjJOhwb4y0rcIwJAarKS9+s\nYcTsjfRpW4N/39KYMCtuPJe9w/Hv+455HceYYqWE1wGMMQUzNSGRYTPWkpScQulS4Rw9nsGd7Wry\nXNeLrbgJIM/ceBH/W72L575cyegBrb2OY0yxYS04xgShqQmJDJm8nMTkFBQ4ejyD8DChRY0YK24C\njHU4NsYbVuAYE4SGzVhLymkdVzMylf/7bp1HiczZ3NWhNnWtw7ExRcoKHGOCUFJySr62G29FhIfx\nfFfrcGxMUbICx5ggk5mpRJXMeRXwuJioIk5jfHVJXetwbExR8luBIyLVRWSWiKwWkZUi8qi7/VkR\nSRSRJe7tRn9lMCbUZGYqf52ynGMnMihxWl+bqIhwBndq4FEy44usGY6f+3Kl11GMCXn+bMFJB55U\n1YuAdsCDItLQfewVVW3m3r72YwZjQkZGpvKXz5cxccE2Hr6qLsNua0J8TBQCxMdEMbR7Y7o1j/c6\npjkL63BsTNHx2zBxVd0B7HA/PywiqwE7+xpTABmZyuBJS5m8OJFHr67HY9fUQ0S4pUU1r6OZfLqr\nQ20+Xbid56avpGO9SkRG5Hy50RhzboqkD46I1AKaA/PcTQ+JyDIRGSMi5+VyzCARWSgiC/fs2VMU\nMY0JSBmZyp8/c4qbx6+pz+PX1kfEhoIHq+wdjt/50TocG+Mvfi9wRKQs8DnwmKoeAkYAdYBmOC08\nw3M6TlVHqmorVW0VGxvr75jGBKT0jEwe/2QJUxIS+fN19Xn0mnpeRzKF4JK6lejSpCpvz7YOx8b4\ni18LHBGJwCluxqvqZABV3aWqGaqaCYwC2vgzgzHBKj0jk8c+WcIXS5N46voGPHSVFTeh5G+dG1Ii\nTHh+unU4NsYf/DmKSoDRwGpVfTnb9qrZdrsFWOGvDMYEq7SMTB6ZmMD0ZTsYcsOFPHBFXa8jmUJW\nJTqSx66px/errcOxMf7gzxacDsCdwFWnDQn/r4gsF5FlwJXA437MYEzQOZGeycMTEvh6+U7+1vki\n7ru8jteRjJ/YDMfG+I8/R1HNAXLqCWnDwo3JxYn0TB6csJiZq3bx9y4Nuadjba8jGT/K6nDc5715\nvPPjRh67pr7XkYwJGTaTsTEB4nh6Bg+MX8TMVbt49iYrbooL63BsjH9YgWNMAEhNy+D+jxbz/erd\nvHDzxQzoYMVNcWIdjo0pfH67RGWM8U1qWgb3fbiIH9ft4cVbGtG3bU2vI5kiltXh+N9fr6HlCzPZ\nf/QEcTFRDO7UwGanNqaArAXHGA+lpmUw8IOF/LR+Dy91b2zFTQGJSLiIJIjIdPf+1SKy2B3cMEdE\nAn4YWsUyJRFg39ETKJCYnMKQycuZmpDodTRjgpIVOMZ4JOVEBve+v5A5G/byn1ub0KtNDa8jBbNH\ngdXZ7o8A+qpqM2AC8DdPUuXDyzPXo6dtS0nLYNiMtZ7kMSbYWYFjjAeOnUjnnvcX8MvGvQy7rSm3\nt6rudaSgJSLVgM7Ae9k2K1De/TwaSCrqXPmVlJySr+3GmLOzPjjGFLFjJ9K5e9wC5m/ez8u3N+WW\n5rZg5jl6FXgKKJdt273A1yKSAhwC2uV0oIgMAgYB1KjhbQtaXEwUiTkUM3ExUR6kMSb4WQuOMUXo\n6PF0BoxxiptXejaz4uYciUgXYLeqLjrtoceBG1W1GjAWePmMgwmsNe8Gd2pAVA4ri/dsbb8jxhSE\nFTjGFJEjx9MZMHY+i34/wGu9mnNzMxsdUwg6AF1FZAswEWfm9K+Apqo6z93nE+ASj/L5rFvzeIZ2\nb0x8TBQCVCkfSYUyEbz/61a27jvqdTxjgo5dojLGj6YmJDJsxlqSklOICA8jLSOTN/u0oHOTqnkf\nbPKkqkOAIQAicgXwZ6AbsFNE6qvqOuBaTu2AHLC6NY8/ZVj4xj1HuHXErwwYu4DP77+ECmVKepjO\nmOCSZwuOiLQXkbdEZJmI7BGR30XkaxF5UESiiyKkMcFoakIiQyYvJzE5BQVOZGQSES6kZWR6HS2k\nqWo6MBD4XESW4qyJN9jbVAVTJ7Yso/u3Iik5hXvfX2DrVRmTD2ctcETkG5zOejOA64GqQEOcIZeR\nwDQR6ervkMYEo2Ez1pJy2j+kExlqw379RFVnq2oX9/MpqtpYVZuq6hWqusnrfAXVsmYFXuvVjIRt\nyTw6MYGMzNMHkxtjcpJXC86dqnqPqn6hqkmqmq6qR1R1saoOV9UrgF+LIKcxQceG/ZrCcn2jqvyj\nS0NmrNzF81+uRNWKHGPyctYCR1X3ujOEfn+2fQo/ljHBLfnYCUqES46P2bBfUxB3dajNwEtr8/5v\nWxn1c9A2SBlTZPLsZKyqGSJyTESiVfVgUYQyJpgdOHqCvu/NIzNTKRkexolsfW6iIsIZ3KmBh+lM\nMBtyw0UkHUzl31+voUp0FF2bxnkdyZiA5esoqlRguYjMBE6OV1TVR/ySypggte/Icfq+N4/Ne48y\n5q42HDh64uQoKls80ZyrsDBheI+m7Dl8nD9/upTYsqVoX6ei17GMCUi+FjhfuTdjTC72HjlO31Hz\n2LLvKKP7t6ZjvUoAVtCYQhUZEc6oO1tx6zu/MujDhXx+/yXUP79c3gcaU8z4NNGfqr4PfAwscm8T\n3G3GGGDP4eP0HjmXrfuPMnbAH8WNMf4QXTqCcXe1JjIinAFj5rPrUKrXkYwJOD4VOO4EWuuBt4C3\ngXUicpkfcxkTNHYfTqX3qLlsP5DC2AFtuKSuFTfG/6qdV5qxA1pzMCWNAWMXcDg1zetIxgQUX5dq\nGA5cp6qXq+plQCfgFf/FMiY47DqUSq+Rc0lKTmHcXa2tP4QpUo3ioxlxR0vW7zrM/R8t5kS6TSJp\nTBZfC5wIVT05O5k7/XmEfyIZExx2HnSKm10HU3n/7ja0vcCKm4Jyp6P4yOscweiy+rEM7d6YORv2\n8vTkZTZHjjEuXzsZLxSR0cCH7v2+OH1xjCmWdhxMoffIuew9coIP7mlDy5oVvI4U1NzpKGJFpKSq\nnvA6T7Dp0ao6Ow6m8vLMdcTHRPHkdTYVgTG+Fjj3Aw8CjwAC/ITTF8eYYicx2SluDhx1ipsWNc7z\nOlKo2AL8IiJfcOp0FC97liiIPHxVXZKSU3jjfxuoGh1Fn7Y1vI5kjKfyLHBEJBwYrap3AHaiMcXa\n9gPH6D1qLsnH0vjw3rY0qx7jdaRQkuTewgAb95xPIsIL3Rqx81Aqf5+2girRpbjqwvO9jmWMZ3yd\nydiajk2xt23/MXqNnMvh1DTG39uWJtWsuClMqvocgIiUc+7qEY8jBZ2I8DDe6tOCniN/48HxCXxy\nXzv7PTXFlq+djLfgNB3/XUSeyLr5MZcxAWXrvqP0GjmXI8fTmTDQ/mn4g4g0EpEEYAWwUkQWicjF\nXucKNmVKlWDMgNZULFuSu8ct4Pd9x7yOZIwnfC1wkoDp/NF0nHUzJuRt2esUN0dPpDP+3rY0io/2\nOlKoGgk8oao1VbUm8CQwyuNMQalyuUjev7sN6ZlK/7Hz2X/UGt9N8ZNngeP2wSmrqs+dfsvjuOoi\nMktEVovIShF51N1eQURmish696P10DQBa9OeI/Qc+RupaRlMuLedFTf+VUZVZ2XdUdXZQBnv4gS3\nOrFlea9fKxKTU7j3/QWkpmV4HcmYIpVngaOqGUCLAjx3OvCkql4EtAMeFJGGwNPAD6paD/jBvW9M\nwNmw+wi9Rs4lPUP5eFA7GsaV9zpSqNvkXgav5d7+Bmz2OlQwa1WrAq/1bEbCtmQenZhARqbNkWOK\nD1+HiS9xh25+xqnDNyfndoCq7gB2uJ8fFpHVQDxwM3CFu9v7wGzgL/kNbow/bdh9mF4j5wFOcWOL\nGRaJu4HngKzzyk/AXd7FCQ03NK7K3zs35Pnpqxgwdj6b9hwhKTnVVrc3Ic/XAqcCsA+4Kts25Y8T\n0VmJSC2gOTAPON8tflDVHSJS2dewxhSFdbsO02fUXED4eGA76llx43fupfC/quojXmcJRXd3rM1P\n63Yze93ek9sSk1MYMnk5YCvem9DkU4GjqgV+FyUiZYHPgcdU9ZCI+HrcIGAQQI0aNmGVKRprdh6i\n76h5hIcJEwa2o27lsl5HKhbc6Shaep0jlK3bfeao+5S0DIbNWGsFjglJvq4mXl9EfhCRFe79Ju71\n8byOi8ApbsZnu5y1S0Squo9XBXbndKyqjlTVVqraKjY21peYxpyTVUmH6DNqHiXChYmDrLjxQIKI\nfCEid4pI96yb16FCxY7k1By3JyWnFHESY4qGr5eoRgGDgXcBVHWZiEwA/pXbAeI01YwGVp821foX\nQH/gJffjtALkNqZQTE1IZNiMtc5JXqBcqRJ8fn9HaleywTseOKdL4ebs4mKiSMyhmImLifQgjTH+\n52uBU1pV5592eSk9j2M6AHcCy0VkibvtrziFzacicg/wO9AjH3mNKTRTExIZMnk5KVnDZxWOp2ey\ndFuyFThFzO2Ds0xVX/E6S6ga3KnBqb/vrhoVS5ORqYSH+dZ9wJhg4etEf3tFpA7OuylE5DbcEVK5\nUdU5qiqq2kRVm7m3r1V1n6perar13I/7z/FrMKZAhs1Ye8bJ/nh6JsNmrPUoUfHlTkfR1escoaxb\n83iGdm9MfEwUAsTHRHL1RZX5beN+Hp2YwIn0TK8jGlOofG3BeRBnltELRSQRZ26Kvn5LZUwRyK3v\ngfVJ8MyvIvIm8AmnTkex2LtIoaVb8/gzOhS/8+NGXvpmDUeOpzOib0uiSoZ7lM6YwuXrKKpNwDUi\nUgYIU9XD/o1ljH8t/v1Aro/FxUQVYRKTzSXux+ezbVNO7ZNjCtmfLq9DdFQEf52ynH5j5vFe/9ZE\nR0V4HcuYc+ZrCw4AqnpURKYDXfyUxxi/W7R1P/3HLKBCmQiOHM/geLam+aiIcAZ3auBhuuJLVa/0\nOkNx1btNDcpFluDxT5bQe+RcPrinDZXKlvI6ljHnxNc+ONnZhAkmaC3Ysp9+o+cTW64U0x+5lP/c\n2iRbn4QohnZvbHOCeEREzheR0SLyjXu/oTsYwRSBLk3iGNWvFZv2HuH2d37LccSVMcEkXy04roRC\nT2FMEZi3aR93jVtAlfKRfDyoHeeXj8yxT4LxzDhgLPCMe38dTn+c0V4FKm6uaFCZD+9py93jFtBj\nxK98eG9b6sTafFAmOOW7BUdV7/ZHEGP86beN+xgwdgFVoyOZ6BY3JuBUUtVPgUwAVU0HbAnsIta6\nVgUmDmrHiYxMerzzGysSD3odyZgCOWuBIyLLRWRZbreiCmnMufhlw17uGjefaudF8fGgdlS24iZQ\nHRWRivwxHUU7wP67euDiuGg+va89URHh9B45l3mb9nkdyZh8y6sFpwtwE/Cte+vr3r4GJvk3mjHn\n7uf1e7h73AJqVijjFDflrLgJYE/gzHReR0R+AT4AHvY2UvF1QWxZPvtTe2LLl6LfmPnMWpPjqjrG\nBKyzFjiqulVVtwIdVPUpVV3u3p4GOhVNRGMK5sd1e7jn/YXUrlSGCQPb2qiQAOfOd3M5znDx+4CL\nVdWnlmIRCReRBHeUJ+J4UUTWichqEbFVygsgLiaKz+5rT73zyzLwg4VMW5LodSRjfOZrH5wyItIx\n646IXALYXPYmYM1au5uBHyykbmxZJgxsR0UrboKCqqar6krgEVVNy8ehjwKrs90fAFQHLlTVi4CJ\nhZeyeKlYthQTBrajRc3zeOyTJXw0d6vXkYzxia8Fzj3AWyKyRUQ2A28D1tnYBKQfVu/ivg8WUa9y\nWSYMbEuFMiW9jmTyr5WvO4pINaAz8F62zfcDz6tqVodlu75yDspHRvDB3W24skFl/jZ1BW/N2oCq\neh3LmLPyqcBR1UWq2hRoAmStK2XTp5uAM3PVLv700SIaVCnHhHvbEVPaipsglZ+C5FXgKdzRV646\nQE8RWSgi34hIvZwOFJFB7j4L9+zZcw5xQ19kRDjv3tmSm5vFMWzGWl76Zo0VOSag+VTgZE3ABXyi\nqgdtAi4TiGas3MkD4xfRsGp5Prq3LdGlbbr5YKWq1/uyn4h0AXar6qLTHioFpKpqK2AUMCaX1xmp\nqq1UtVVsbOw5ZS4OIsLDeOX2ZtzRrgbv/rSJv05ZTkamFTkmMPk60d84bAIuE8C+Wb6Dhz9OoFF8\nNB/c04bykVbcBAsR+RJ3aHhOVPVsq4x3ALqKyI1AJFBeRD4CtgOfu/tMwTl/mUIQFia8cHMjoqMi\neGvWRg6lpPNKz2aULFGQifGN8R9ffyNtAi4TsL5atoOHPk6gSbVoPrTiJhj9HzAc2Ayk4LS4jAKO\nACvOdqCqDlHVaqpaC+gF/E9V7wCm8scinZfjvCkzhUREGNzpQv5644V8tXwH936wkGMn0r2OZcwp\nfG3BsQm4TED6cmkSj32yhObVYxh3dxvKlirI6iPGS6r6I4CIvKCql2V76EsR+amAT/sSMF5EHscp\nlO49x5gmB4Muc1YiHzJ5OZ1f+5nU9Ex2HkwlLiaKwZ0a2DIoxlO+/jc4fQKuWOA2v6UyxgfTliTy\n+CdLaFXGgz+VAAAgAElEQVSzAmPuam3FTfCLFZELVHUTgIjUxjnX+ERVZwOz3c+TcUZWGT/r2boG\nq3ccYtyvfwwfT0xOYcjk5QBW5BjP5PkfQUTCcK5tXw40AARYm885KowpVFMStvPkp0tpXasCYwa0\npowVN6HgcWC2iGxy79fCmfDPBLiZq84c9JaSlsGwGWutwDGeyfO/gqpmishwVW0PrCyCTMac1aRF\n2xk8aSntaldk9IBWlC5pxU0oUNVv3eHcF7qb1qjqcS8zGd8kJafka7sxRcHXTsbficitIiJ+TWNM\nHj5dsI3Bk5bSoU4lxgxobcVNCBGR0sBg4CFVXQrUcIeBmwAXFxOV43abqsF4KT99cMoA6SKSinOZ\nSlW1vN+SGQNMTUhk2Iy1JCWnEB0VQXJKGpfWq8Sofq2IjAj3Op4pXGOBRUB79/524DNgumeJjE8G\nd2rAkMnLSUn7Y3BtmEDysTSGzVjDk9c2ICzM3h+bouVTgaOq5fwdxJjTTU1IPOWkmZySRphA16Zx\nVtyEpjqq2lNEegOoaoq1GgeHrH42WW9G4mKieOLaeizceoC3Zm1k896jvHx7M/u7NUXK5/Z9ETkP\nqIfT4RgAVS3oEE5j8jRsxtpT3hECZCq8+v16erSq7lEq40cnRCSKP6ajqANYH5wg0a15/Bkdiru3\nqMYFlcry729Wk5g8l1H9WlK5XGQuz2BM4fJ1qYZ7gZ+AGcBz7sdn/RfLGOu4WAz9E/gWqC4i44Ef\ncNaYMkFKRBh42QW8c0dL1u08zC1v/cqanYe8jmWKCV87GT8KtAa2quqVQHPAVqYzflU+KucGxtw6\nNJrg5V6KWgN0BwYAHwOt3LltTJDrdHEVPvtTe9IzM7ltxG/MXmuLuxv/87XASVXVVAARKaWqa3Dm\nxDHGL0b9tImDKemc3i8xKiKcwZ3sVy/UqLMs9VRV3aeqX6nqdFXd63UuU3gaxUcz9cEO1KhQmrvH\nLeCD37Z4HcmEOF8LnO0iEoOzvstMEZkGJPkvlinO3vlxIy9+vZrOjasy7LYmxMdEIUB8TBRDuze2\nicNC11wRae11COM/VaOj+OxP7bnqwsr8Y9pKnv1iJekZmV7HMiHK11FUt7ifPisis4BonGvlxhSq\nt2ZtYNiMtXRpUpVXezajRHgYt7a0DsXFxJXAfSKyFTjKH9NRNPE2lilMZUqV4N07WzH069W8N2cz\nW/cd5Y0+LWypFVPofPqNEpEa2e5udj9WAX4/yzFjgC7AblVt5G57FhjIH/13/qqqX+czswlRb/yw\nnuEz13FzsziG92hKiXBfGxhNiLjB6wCmaISHCX/r0pDasWX4x7SV3DbiV0YPaE289a8zhcjX/yBf\n4Uy29RXOyIZNwDd5HDMOuD6H7a+oajP3ZsWNAeDV79cxfOY6bmkez8u3N7PiphhS1a2quhVIwRkq\nnnUzIapv25qMu6s1iQdSuPnNX1i6LdnrSCaE+PRfRFUbq2oT92M9oA0wJ49jfgL2F0JGE8JUlZdn\nruPV79dza4tq/F+PpoTbjKfFkoh0FZH1OK3EPwJbyPuNlAlyl9aLZfIDlxAZEUbPkb/xzfIdXkcy\nIaJAb5NVdTHOsPGCeEhElonIGHfywByJyCARWSgiC/fssRHpoUhVGf7dOl7/YT09Wlbjv7c1seKm\neHsBaAesU9XawNXAL95GMkWh3vnlmPpgBxpWLc/94xfz9uwNOAPrjCk4Xyf6eyLb7c8iMoGCzYMz\nAqgDNAN2AMNz21FVR6pqK1VtFRsbW4CXMoFMVfnvjLW8OWsDvVpX5z+3WnFjSFPVfUCYiISp6iyc\nc4UpBiqVLcWEge3o2jSO/367lqcmLeNEuo2wMgXna7f17GtRpeP0xfk8vy+mqruyPheRUdgiesWS\nqvLSN2t496dN9Glbg3/d3MgW4jMAySJSFmfW9PEishvnfGOKiciIcF7r1Yzalcrw2g/r2XbgGO/c\n0ZKY0iW9jmaCkK/DxJ8rjBcTkaqqmnWB9RZgRWE8rwkeqsqLXznDQ+9oV4Pnu1pxY066GUgFHgf6\n4kxH8byniUyRExEev7Y+tSuV4alJy+j+tjPCqnalMl5HM0HG12HiX5ztcVXtmsMxHwNXAJVEZDvO\nOjNXiEgznJERW4D78pnXBDFV5fnpqxj7yxb6t6/Js10vxhaLNllU9Wi2u+97FsQEhG7N44k/L4r7\nPlzELW//wh1tazAlIenkauWDOzWwST/NWfl6iWozzrw3H7n3e+MUKDNyO0BVe+eweXR+wpnQoao8\n9+Uqxv26hbs61OIfXRpacWNOISKH+WNYeEkgAjiqquW9S2W81LpWBaY8cAk93vmVN2dtPLk9MTmF\nIZOXA1iRY3Lla4HTXFUvy3b/SxH5SVX/6o9QJrRkZir//GIlH87dyr0da/NM54usuDFnUNXsff0Q\nkW44U1KYYqxmxTKEh505HiYlLYNhM9ZagWNy5esw8VgRuSDrjojUBmxok8lTZqbyt2kr+HDuVu67\n7AIrbozPVHUqcJXXOYz3dh5MzXF7UnJKEScxwcTXFpzHgdkissm9XwvrP2PykJmp/HXKciYu2Mb9\nV9ThqU4NrLgxuRKR7tnuhgGtsJmMDRAXE0ViDsVMbLlSHqQxwcLXUVTfikg94EJ30xpVPe6/WCbY\nZWYqT09exqcLt/PQlXV58rr6VtyYvNyU7fN0nH5+N3sTxQSSwZ0aMGTyclLSMk7ZfvDYCWat2c2V\nF1b2KJkJZL5O9NcDKKmqS3FOQh+LSAu/JjNBKyNTGTzJKW4eubqeFTfGJ6p6V7bbQFV9UVV3e53L\neK9b83iGdm9MfEwUAsTHRPHPmxpSr0o57n5/AW/NspmPzZl8vUT1d1X9TEQ6Ap2A/8OZlbit35KZ\noJSRqQz+bCmTExJ57Jp6PHZNfa8jmSAhIq+f7XFVfaSospjA0615/Bkdinu1rsFfPl/GsBlrWbXj\nEMNua0Lpkr7+WzOhztffhKx2wc7ACFWdJiLP+ieSCSZTExIZNmMtSckpVI2JpEr5SBb/nsyT19bn\n4avreR3PBJdIoCHwiXu/B7AIWOJZIhPQoko6Mx83ii/PS9+sYdOeo4y8syXVK5T2OpoJAL4WOIki\n8i5wDfAfESlFARfqNKFjakLiKdfFk5JTSUpOpXPjKlbcmIKoB1ypqmkAIvIO8J2qPu5tLBPIRIRB\nl9WhQZXyPDxhMV3fnMNbfVtwSZ1KXkczHvO1SLkdZ1K/61U1GagADPZbKhMUhs1Ye0anP4Al2w56\nkMaEgDhOXfeurLvNmDxdXj+WaQ91pGLZUtw5ej7jftls/XKKOZ8KHFU9pqqTVXW9iAxS1R2q+p2/\nw5nAltscFDY3hSmgl4AEERknIuOAxcC/vY1kgkntSmWY8sAlXNmgMs9+uYqnJi3jePqZb8JM8VCQ\ny0x/KvQUJihVjY7McXtcTFQRJzGhQFXH4gxcmOLe2quqrUll8qVcZAQj72zJI1fX47NF2+n57lx2\nHcp5okAT2gpS4Nh4X8Px9AzOK1PyjO1REeEM7tTAg0Qm2IlIB+Cwqk7DuVT1lIjU9DiWCUJhYcIT\n19bnnTtasG7XYW56Yw6Lfz/gdSxTxApS4NyU9y4mlB1Pz+D+jxazMukQt7WIP2VuiqHdG9vaMKag\nRgDHRKQpTh+/rcAHvhwoIuEikiAi00/b/oaIHCn8qCYYXN+oKlMe6EBkRDi93p3Lpwu3eR3JFCGf\nRlGJSAzQD2eJhhJZk7bZvBTFT2paBn/6aBGz1+7hxVsa0betvcE2hSZdVVVEbgZeV9XRItLfx2Mf\nBVYDJ1ceF5FWQIwfcpog0qBKOb54qAMPf5zAU5OWsSrpEM90voiIcBsIHOp8/Ql/jVPcLMeZlyLr\nZoqR1LQMBn3oFDdDuze24sYUtsMiMgS4A/hKRMKBiLwOEpFqOHN0vZdtWzgwDHjKT1lNEIkpXZKx\nA1pzb8fajPt1C/1Gz2f/0RNexzJ+5us8OJGq+oRfk5iAlnIig0EfLmTOhr3899Ym3N66uteRTOjp\nCfQB7lHVnSJSA6dIycurOIVM9iHmDwFfqOqOsy0TIiKDgEEANWrUKGhuEwRKhIfxty4NaRhXnqcn\nL+emN+Ywsl9LLo6L9jqa8RNfW3A+FJGBIlJVRCpk3fyazASMlBMZ3PP+AuZs2Muw25pacWP8QlV3\nqurLqvqziHRR1d9V9ax9cESkC7BbVRdl2xaHMwvyGz685khVbaWqrWJjY8/5azCBr3uLakz6U3sy\nVbltxG9MX5bkdSTjJ7624JzAeSf1DJA1c5ICF/gjlAkcx06kc/e4BczfvJ/hPZrSvUU1ryOZ4uF5\nYHqee0EHoKuI3Iiz1EN5YCVwHNjgtt6UFpENqlrXX2FNcGlSLYZpD3XggY8W89CEBFYlHaJubFmG\nz1xHUnIKcTFRDO7UwAZMBDlfC5wngLqqutefYUxgOXo8nbvGLWDhlv280rMZNzezP3ZTZHyajkJV\nhwBDAETkCuDPqtrllCcSOWLFjTld5XKRTBjYjn9+sZK3Z28kTCDTffuemJzCkMnLAazICWK+XqJa\nCRzzZxATWI4cT2fA2Pks2nqAV3s1t+LGFLX7vA5gQl/JEmEM7d6Y6KiIk8VNlpS0DIbNWOtNMFMo\n8rOa+BIRmYXT9AvYMPFQdTg1jQFjF7BkWzKv92pO5yZVvY5kigF35FNn/piOoiOAqr7sy/GqOhuY\nncP2soUW0oSkQylpOW63ZWeCm68FzlT3ZkLcodQ0+o+Zz/LtB3mzd3NuaGzFjSkyXwKpONNRZHqc\nxRQjcTFRJOZQzFSNyXk5GhMcfCpwbD2Y4uFgShr9xsxnZeJB3uzTgusbVfE6kileqqlqE69DmOJn\ncKcGDJm8nJS0UxfmLB0Rzt4jx6lUtpRHycy58KkPjohsFpFNp9/8Hc4UnYPH0rhz9DxWJR1kxB0t\nrbgxXvhGRK7zOoQpfro1j2do98anLDvTp011th1I4aY35pBg61gFJV8vUbXK9nkkzhwTNg9OiEg+\ndoI7Rs9j3c4jvHtnS6668HyvI5niaS4wRUTCgDSckVSqquXPfpgx565b8/gzRkz1aVuT+8cvoue7\nc/ln14b0aVODs00caQKLTy04qrov2y1RVV8FrvJzNlMEDhw9QZ9R81i36wjv9rPixnhqONAeKK2q\n5VW1nBU3xkuN4qP58qGOtK9TkWemrOCpSctIPe0ylglcvi622SLb3TCcFp1yuexugsT+oyfo+948\nNu05wqh+rbi8vs3kajy1HlihqprnnsYUkZjSJRkzoDWv/bCe139Yz+qdhxjRtyXVK5T2OprJg6+X\nqIZn+zwd2ALcXuhpTJHZd+Q4fd+bx+a9RxndvzUd61XyOpIxO4DZIvINp05H4dMwcWP8JTxMeOLa\n+jStFs3jnyzhpjfn8Fqv5vamMMD5eonqymy3a1V1oKqedQYkERkjIrtFZEW2bRVEZKaIrHc/nneu\nX4DJvz2Hj9N71Fy27DvK2AFW3JiAsRn4ASiJ00KcdTMmIFx90fl8+XBHqpSPZMDY+bzxw3oyT58h\n0ASMsxY4InKTiNTMdv8fIrJURL4Qkdp5PPc44PrTtj0N/KCq9XBOZE8XILM5B7sPp9J71Fy27U9h\n7IA2XFLXihsTGFT1uZxuXucyJruaFcsw5YEO3Nw0juEz1zHow4UczGWiQOOtvC5RvQi0g5Or9t4B\n9AaaA+8AnXI7UFV/EpFap22+GbjC/fx9nFlH/5K/yCa/piYkMmzGWpKSUwgPE0Tgo3va0vaCil5H\nM+Ykd6b0M94Oq6oNaDABJapkOK/0bEbzGufxwvRV3PzmHN65syUXVrE+8YEkrwJHVTVrDaruwGhV\nXQQsEpEHCvB656vqDveJd4hI5dx2FJFBwCCAGjVqFOClDDjFTfYJrNIzlZIlwthxMNXjZMac4c/Z\nPo8EbsXp82dMwBER+l9Si4vjyvPA+MXc8tavvHRrY1u3L4Dk1QdHRKSsOy/F1TiXlbL4dQ5rVR2p\nqq1UtVVsrHXkKqhhM9aeMTvnifRMW0TOBBxVXZTt9ouqPgG09TqXMWfTqlYFpj/Skcbx0Tw6cQnP\nfrGStAxbaSQQ5FXgvAosARYCq1V1IYCINMcZ8ZBfu0SkqvscVYHdBXgOkw+5LRZni8iZQOMOQsi6\nVRKR6wGbUtsEvMrlIhk/sC13d6jNuF+30GfUXHYfslZyr521wFHVMcDlwD3Ajdke2gncVYDX+wLo\n737eH5hWgOcwPtp+4BhhYTnPuhkXE1XEaYzJ0yKcN1MLgV+BJ3DOPcYEvIjwMP5xU0Ne792cFYmH\n6PzGHBZu2e91rGItr1FUtdyZixNU9WSbm6ruUNXfxVEtl2M/Bn4DGojIdhG5B3gJuFZE1gPXuveN\nH2zbf4ye784lIgxKlTj1xxwVEc7gTg08SmbMqUSktYhUUdXaqnoB8Bywxr2t8jadMfnTtWkcUx/s\nQNlSJeg1ci5jf9mMzV3pjbw6GQ9z+99Mw3l3tQen701d4Eqcfjn/BLaffqCq9s7lOa8ucFrjk9/3\nHaP3qLkcOZ7OpPs7sGH3kZOjqOJiohjcqcEZa64Y46F3gWsAROQyYCjwMNAMGAnc5l00Y/KvQZVy\nTHuoA098spTnvlzFl0uT2HEwlZ0HU+0cXITOWuCoag8RaQj0Be4GqgIpwGrgK+BFVbULjQFky96j\n9B41l5S0DMbf25ZG8dE0io+2PyYTyMJVNastvycwUlU/Bz4XkSUe5jKmwMpHRjDyzpY8/PFivlq+\n8+T2xOQUhkxeDmDnZT/Lc6kGVV0FPFMEWcw52rz3KL1HzuV4egYT7m1Hwzibk8EEhXARKaGq6Tgt\nvIOyPebrcjLGBJywMGHJtoNnbE9Jy2DYjLVW4PiZr4ttds9h80FguaraSKgAsHHPEXqPnEtGpvLx\noHY24ZQJJh8DP4rIXpwW4p8BRKQuznnGmKBlI1m94+u7o3uA9sAs9/4VwFygvog8r6of+iGb8dGG\n3YfpPWoeqk5xU/98W77HBA9VfVFEfsC5BP5dttXEw3D64hgTtOJiokjMoZiJCA9j58FUqkT7dUq5\nYs2nxTaBTOAiVb1VVW8FGuKs9tsWW2rBU+t3HabXyHmowscDrbgxwUlV56rqFFU9mm3bOlVd7GUu\nY87V4E4NiIoIP2VbRLigKJ1f/5lfNuz1KFno87XAqaWqu7Ld3w3UdzsG2ipjHlm78zC9Rs4lTGDi\noHbUs+LGGGMCSrfm8Qzt3pj4mCgEiI+JYthtTfnm0UupUKYkd4yeZ6uS+4mvl6h+FpHpwGfu/duA\nn0SkDJDsl2TmrFbvOETf9+YRES58PLAdF8SW9TqSMcaYHHRrHp9jh+JpD3XgmSkrGD5zHQu3HuCV\nns2oUKakBwlDk68tOA8CY3HmpWiOsxL4g6p6VFWv9Fc4k7OVSQfpM2ouJcPDmDiovRU3xhgThEqX\nLMHLtzfl37c05reN++j8+s8s/v2A17FChk8Fjtvpbw7wP+B74Ce1qRk9sSLxIH3fm0dURDif3NeO\n2pXKeB3JGGNMAYkIfdrWYPIDl1AiXLj9nd8YM8dmPy4MPhU4InI7MB/n0tTtwDwRsdlFi9jy7U7L\nTZmSJZg4qD01K1pxY4wxoaBRfDTTH7qUKxpU5vnpq3hwwmIOp1oX13Ph6yWqZ4DWqtpfVfsBbYC/\n+y+WOd3Sbcn0eW8u5aMimDioHTUqlvY6kjHGmEIUXTqCUf1aMuSGC5mxchdd3/yF1TsOeR0raPla\n4ISdNqHfvnwca85Rwu8HuOO9ecSUdoqb6hWsuDHGmFAkItx3eR0+HtiOo8fT6fbWL3y2cJvXsYKS\nr0XKtyIyQ0QGiMgAnHWovvZfLJNl0dYD3Dl6PhXKluSTQe2pdp4VN8YYE+ra1K7AV49cSsua5zF4\n0jKemrSU1LQMr2MFFV87GQ/GWdW3CdAUZzE8m+DPzxZu2U+/0fOILVeKiYPaERcT5XUkY4wxRSS2\nXCk+vKctD19Vl08XbueWt39l896jeR9ogHxcZlLVz1X1CVV9XFWn+DOUgfmb99NvzHzOLx/JxEHt\nqBptxY0xxhQ34WHCk9c1YOxdrdlxMIWb3pjDN8t3eB0rKJy1wBGRwyJyKIfbYRGxnk9+MnfTPgaM\nnU/VaKe4Ob+8rVVijDHF2ZUNKvPVI5dSp3JZ7h+/mOe/XMWJ9EyvYwW0s85krKo2938RmJqQyLAZ\na0lKTqFi2ZIcPJZGrUplGD+wLZXLWXFjTF5EJBxYCCSqahcRGQ+0wllKZj5wn6ramFsT1OJjovjs\nvvb8++vVjPllM0u2HeDNPi2s+0IufF2qwfjJ1IREhkxeTorbeWzvkRMI0O+SmlbcGOO7R4HVQHn3\n/njgDvfzCcC9wAgPchlTqEqWCOPZrhfTqtZ5/GXSMjq//jO3t67O9KU7SEpOIS4misGdGuS4NERx\nY0O9PTZsxtqTxU0WBd6ZvcmbQMYEGRGpBnQG3svapqpfqwunBaeaV/mM8YcuTeL44uGOlCoRxrs/\nbiIxOQUFEpNTGDJ5OVMTEr2O6DkrcDyWlJySr+3GmDO8CjwFnNEhQUQigDuBb3M6UEQGichCEVm4\nZ88e/6Y0ppDViS1LmMgZ21PSMhg2Y60HiQKLFTgey23lWLumakzeRKQLsFtVF+Wyy9s4a+f9nNOD\nqjpSVVupaqvY2Fi/5TTGX3YcTM1xu71JtgLHU9+v2kXysROcXoBHRYQzuFMDb0IZE1w6AF1FZAsw\nEbhKRD4CEJF/ArHAE97FM8a/cnszXDayBBmZxXvBTitwPPLdyp3cP34RjeKjebFbI+JjohCcXvJD\nuze2DmLG+EBVh6hqNVWtBfQC/qeqd4jIvUAnoLeq2lhaE7IGd2pAVET4KdvCRTicms4d781j96Gc\nW3iKAxtF5YFvV+zgoQkJNIqP5v272xAdFUGftjW9jmVMKHkH2Ar8Jk4T6WRVfd7bSMYUvqw3w1lT\njWSNokrLyOTv01Zw4+s/80rPZlxar/hdgrUCp4h9vXwHD3+cQNNq0Yy7uw3lIyO8jmRMSFDV2cBs\n93M7t5lio1vz+Bxb/ZtVj+GB8YvpN2Y+D15Rl8euqUeJ8OJz4ab4fKUB4MulSTz8cQLNq8fwwT1t\nrbgxxhjjN/XOL8cXD3WkR8tqvDlrA33em8fOXDolhyIrcIrItCWJPDoxgZY1zmPc3W0oW8reYBpj\njPGvqJLh/Pe2prx8e1NWJB7kxtd/Zvba3V7HKhKeFDgiskVElovIEhFZ6EWGojQlYTuPf7KE1rUq\nMPau1lbcGGOMKVLdW1Tji4c6UrlcKQaMXcB/vl1DekZo97/3sgXnSlVtpqqtPMzgd58v2s4Tny6l\nbe2KjL2rNWWsuDHGGOOBupXLMvXBDvRuU50RszfSa+TckJ4vxy5R+dGnC7fx50lL6VCnEmMGtKZ0\nSStujDHGeCcyIpyh3ZvwWq9mrN5xiBtf/5kfVu/yOpZfeFXgKPCdiCwSkUEeZfCrifN/56lJy+hY\ntxLv9W9FVMnwvA8yxhhjisDNzeKZ/silxEVHcc/7C3nxq1WkhdglK68KnA6q2gK4AXhQRC47fYdg\nXiNm/LytPD15OZfXj2VUv1ZERlhxY4wxJrDUrlSGyQ9cwp3tajLq5830eOc3th845nWsQuNJgaOq\nSe7H3cAUoE0O+wTlGjEf/raFZ6as4KoLK/PunS2tuDHGGBOwIiPCeaFbI97q04KNu49w42s/893K\nnV7HKhRFXuCISBkRKZf1OXAdsKKoc/jD+79u4e/TVnLNRZUZcUcLK26MMcYEhc5NqjL9kY7UrFiG\nQR8u4rkvV3IiPbgvWXnRgnM+MEdElgLzga9U9VsPchSqMXM2888vVnJdw/N5u29LSpWw4sYYY0zw\nqFmxDJPub8+AS2ox9pct9HjnV7btD95LVkU+rEdVNwFNi/p1/em9nzfxr69Wc/3FVXijT3MiitFU\n2MYYY0JHqRLhPNv1YtpdUIHBk5Zx4+s/c1vLeL5bufuUta6CYUFoG7d8jt79cSNDv1lD58ZVebVX\nMytujDHGBL3rG1Xl4rho+oyay9hftp7cnpicwpDJywECvsix/8bn4K1ZGxj6zRq6NKnKa1bcGGOM\nCSHVK5QmI1PP2J6SlsGwGWs9SJQ/1oJTQG/8sJ7hM9dxc7M4hvdoWqxWaDXGGFM87Mhlcc5gmAHZ\n/isXwKvfr2P4zHXc0jyel29vZsWNMcaYkBQXE5Xj9siIcA6nphVxmvyxFhwfTE1IZNiMtSQlp1A2\nsgSHU9O5tUU1/ntbE8LDxOt4xhhjjF8M7tSAIZOXk5KWcXJbiTAhNS2Dzq/P4Y3ezWlaPcbDhLmz\npoc8TE1IZMjk5SQmp6DA4dR0wsOEDnUqWnFjjDEmpHVrHs/Q7o2Jj4lCgPiYKP6vR1M++1N7MjKV\nW0f8yrs/biQzh746XrMWnDwMm7H2lMoVICNTGT5zHd1bVvMolTHGGFM0ujWPz3HE1NePXMrTk5cx\n9Js1/LJxH8N7NCW2XCkPEubMWnDOYvWOQyTm0pEqGDpYGWOMMf4SXTqCt/u24MVbGjFv0z5ueO0n\nfloXOGtHWoGTg/W7DvPg+MXc8NrP5HYRKreOV8YYY0xxISL0bVuTLx/uSIUyJek3Zj5Dv14dEMs8\n2CWqbDbvPcpr369j2tIkSkeE89CVdYmLieSF6atPuUwVFRHO4E4NPExqjDHGBI7655fji4c68sL0\nVbz70ybmbtrH672bU7NiGc8yWYEDbNt/jNd/WM/khEQiwoVBl13AfZfVoUKZkgCULlni5CiqYJqm\n2hhjjCkqkRHhvHhLYzrWrcRfPl9G59fn8OItjbi5mTf/L4t1gZOUnMIb/9vAZwu3ERYm9G9fi/uv\nqHNGJ6ncOlgZY4wx5lQ3NK5Kk+oxPPpxAo9OXMLP6/fyXNeLKVOqaEuOYlng7DqUyluzNjBx/jYU\npSCLdTwAAAjiSURBVE/bGjxwRV2qREd6Hc0YY4wJevExUUwc9P/t3XuMXGUdxvHvwy6VXmwrbUHa\nBVrqBuRWiqsBqsYUomIJJRUCiIqExGiUFjAU8B+NURExAlVEAYESCGAqgmkIQkrDrVoCFIRSDLcW\ni4vdhl64tbTLzz/mVAbYJTPb2X3nPfN8ksmc83Z25pnt5Hd+e86Z8x7BgiXP8pulz/HYmg0sOHU6\nB08aM2QZWqrB6XltK7+/73lu/Mcaet8JTurq4PszO5nkE4bNzMwaqr1tF8794v4cOXU859z6OHN+\nt4wLjj2AM2ZMRhr868i1RIPz6htv84f7n+eGZWvYur2XOYd3MHdmJ/uMG5E6mpmZWakdOXUcd877\nHPMXPcFPFj/Ng8+t55ITD2XcqMG9Zk6pG5xNb27j6gde4LqHXuTNbb3MnjaRuUd3st+EUamjmZmZ\ntYzdRw7j6m92sXDZan5+5zMce/kDXHbKYRw1dfygvWZpGpzq+aI+PmY3pnWM5aHn1/Palu3MOmQv\nzj6mk849P5o6ppmZWUuSxLdmTOHTU3bnrJtXcNo1yznmgD1Y2b2Z7o1bGv4t5VI0ODvmi9pxrZru\nTVvo3vQKB08czS9PnMaBE0cnTmhmZmYAB00cw+KzPssZ1z3MPavW/X/85Y1vceFtTwI0pMkpxZWM\n+5ovCmDDm9vc3JiZmTWZEcPaWbthywfG39rWyyV/+1dDXqMUDU5/80J5vigzM7PmNNjb7lI0OP3N\nC+X5oszMzJrTYG+7S9HgnPel/Rm+a9t7xjxflFnrkNQmaYWkxcX6FEnLJT0r6VZJw1JnNLP3Guxt\ndykanBOmT+KiOYcwaexwROUKihfNOcTTK5i1jnnAqqr1i4FLI6IT2ACcmSSVmfVrsLfdpfgWFXi+\nKLNWJakDmAX8DDhXlUukzgS+VjxkIfBj4MokAc2sX4O57S7FHhwza2mXAfOBd4r1ccDGiNherK8F\n+qygkr4t6RFJj/T09Ax+UjMbMm5wzCxbko4D1kXEo9XDfTw0+vr5iLgqIroiomvChAmDktHM0ijN\nISoza0kzgOMlfQXYDRhNZY/OWEntxV6cDuA/CTOaWQLeg2Nm2YqICyOiIyImA6cA90bEacBS4MTi\nYacDdySKaGaJuMExszI6n8oJx89ROSfnj4nzmNkQU0Sfh6abiqQeYE3qHMB4YH3qEDsh5/w5Z4e8\n89eTfd+IyPJkliaqM9A6n5dmlHP+VsleU53JosFpFpIeiYiu1DkGKuf8OWeHvPPnnD1XOf/Oc84O\need39vfyISozMzMrHTc4ZmZmVjpucOpzVeoAOynn/Dlnh7zz55w9Vzn/znPODnnnd/YqPgfHzMzM\nSsd7cMzMzKx03OCYmZlZ6bjBqYGkvSUtlbRK0kpJ81JnqpekNkkrJC1OnaVeksZKWiTpmeL/4MjU\nmWol6ZziM/OUpJsl7ZY604eRdK2kdZKeqhrbXdI9kp4t7j+WMmNZlaHOQL61Juc6A3nVmqGqM25w\narMd+EFEfBI4AviepAMTZ6rXPGBV6hADdDlwV0QcAEwjk/chaRIwF+iKiIOBNirTCTSz64Evv2/s\nAmBJRHQCS4p1a7wy1BnIt9ZkWWcgy1pzPUNQZ9zg1CAiuiPisWL5NSof/ElpU9VOUgcwC7gmdZZ6\nSRoNfJ7iUvsR8XZEbEybqi7twHBJ7cAImnzSx4i4H3j1fcOzgYXF8kLghCEN1SJyrzOQb60pQZ2B\njGrNUNUZNzh1kjQZmA4sT5ukLpcB84F3UgcZgP2AHuC6Yrf3NZJGpg5Vi4h4GfgV8BLQDWyKiLvT\nphqQPSOiGyobYWCPxHlKL9M6A/nWmmzrDJSm1jS8zrjBqYOkUcCfgbMjYnPqPLWQdBywLiIeTZ1l\ngNqBw4ErI2I68AaZHCIpjiHPBqYAE4GRkr6eNpU1uxzrDGRfa7KtM+Ba0x83ODWStCuVonNTRNyW\nOk8dZgDHS1oN3ALMlHRj2kh1WQusjYgdf8kuolKIcnAM8GJE9ETENuA24KjEmQbiv5L2Aiju1yXO\nU1oZ1xnIu9bkXGegHLWm4XXGDU4NJInKsdlVEfHr1HnqEREXRkRHREymctLZvRGRTWcfEa8A/5a0\nfzF0NPB0wkj1eAk4QtKI4jN0NBmduFjlr8DpxfLpwB0Js5RWznUG8q41mdcZKEetaXidad/ZJ2gR\nM4BvAE9KerwY+2FE3JkwUys5C7hJ0jDgBeCMxHlqEhHLJS0CHqPyDZkVNPml1CXdDHwBGC9pLfAj\n4BfAnySdSaWQnpQuYam5zqSVZZ2B/GrNUNUZT9VgZmZmpeNDVGZmZlY6bnDMzMysdNzgmJmZWem4\nwTEzM7PScYNjZmZmpeMGx+omqVfS41W3hl3xU9Lk6hlmzax1udbYzvB1cGwg3oqIw1KHMLPSc62x\nAfMeHGsYSaslXSzp4eL2iWJ8X0lLJP2zuN+nGN9T0l8kPVHcdlxavE3S1ZJWSrpb0vDi8XMlPV08\nzy2J3qaZJeZaY7Vwg2MDMfx9u41Prvq3zRHxGeC3VGYWpli+ISIOBW4CFhTjC4D7ImIalXlfVhbj\nncAVEXEQsBH4ajF+ATC9eJ7vDNabM7Om4VpjA+YrGVvdJL0eEaP6GF8NzIyIF4pJA1+JiHGS1gN7\nRcS2Yrw7IsZL6gE6ImJr1XNMBu6JiM5i/Xxg14j4qaS7gNeB24HbI+L1QX6rZpaQa43tDO/BsUaL\nfpb7e0xftlYt9/LuuWKzgCuATwGPSvI5ZGaty7XGPpQbHGu0k6vu/14sL6MyuzDAacCDxfIS4LsA\nktokje7vSSXtAuwdEUuB+cBY4AN/2ZlZy3CtsQ/lrtQGYnjVbMcAd0XEjq9vfkTScirN86nF2Fzg\nWknnAT28O0vvPOCqYvbYXioFqLuf12wDbpQ0BhBwaURsbNg7MrNm5FpjA+ZzcKxhiuPiXRGxPnUW\nMysv1xqrhQ9RmZmZWel4D46ZmZmVjvfgmJmZWem4wTEzM7PScYNjZmZmpeMGx8zMzErHDY6ZmZmV\nzv8AazTyZ2LDK04AAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))\n", + "\n", + "ada1 = AdalineGD(n_iter=10, eta=0.01).fit(X, y)\n", + "ax[0].plot(range(1, len(ada1.cost_) + 1), np.log10(ada1.cost_), marker='o')\n", + "ax[0].set_xlabel('Epochs')\n", + "ax[0].set_ylabel('log(Sum-squared-error)')\n", + "ax[0].set_title('Adaline - Learning rate 0.01')\n", + "\n", + "ada2 = AdalineGD(n_iter=10, eta=0.0001).fit(X, y)\n", + "ax[1].plot(range(1, len(ada2.cost_) + 1), ada2.cost_, marker='o')\n", + "ax[1].set_xlabel('Epochs')\n", + "ax[1].set_ylabel('Sum-squared-error')\n", + "ax[1].set_title('Adaline - Learning rate 0.0001')\n", + "\n", + "plt.tight_layout()\n", + "# plt.savefig('./adaline_1.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "#### Standardizing features and re-training adaline" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "# standardize features\n", + "X_std = np.copy(X)\n", + "X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()\n", + "X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYHHW1//H3JwQJmEBYEhTIAlwWERUlbCJhVYHrBspV\nrxuCN+K+XhW4j+JyjfuuIBpRAUW8iiICAhEIoEIAQUAg8mMJYZFFlkQEspzfH99q0pnMdH8nU9Vd\nPf15PU8/01VdXXWqM+kzderUtxQRmJmZ1c2YbgdgZmY2GCcoMzOrJScoMzOrJScoMzOrJScoMzOr\nJScoMzOrJSco6ypJh0u6NHPZ4ySdUjyfKmmJpLWqjbB7JP1Q0meK53tJurnbMZl1khOUVULSRZIe\nkrROFeuPiIURMT4illex/nYkzZB0VrGPD0v6q6T/lbRhFduLiEsiYrsy1iXpdkkHtHh9H0krij8A\nlkhaJOl0SbuUsf0qFL9vb+t2HFYuJygrnaTpwF5AAK/oajAVkPRC4CLgMmD7iJgIHAgsA543xHvG\ndizActwdEeOBCcDuwE3AJZL2725Y1k+coKwKbwb+BPwQeEvzC5I2lnSmpEclXQFsPeD1r0u6s3j9\nKkl7DbYBSdMlReOLv/gL+tOSLpO0WNJ5kjZpWn53SX8ojnaulbTPCPbvC8BJETE7Iv4OTx3RfSIi\nLiq2d3gRy1cl/QM4TtLWkn4v6UFJD0g6VdLEphifL+nqIv6fAeOaXttH0qKm6c0k/ULS/ZJuk/Te\npteOK454flys6wZJM4rXTgamAr8pjo4+0mpHI1kUER8Hvg98vmk720s6X9I/JN0s6T+aXju4OKpc\nLOkuSR9ueu2Vkq4p/o3/n6QDi/kbSJoj6Z7iPZ9plHAbpWBJXyqOWm+TdFDx2v+S/iD6VrFP38r9\nh7Saiwg//Cj1AdwCvBPYGVgKbNr02mnA6cDTgR2Bu4BLm15/I7AxMBb4EHAvMK547TjglOL5dNIR\n2thi+iLg/wHbAusW058rXtsceBA4mPRH2YuL6UlrsG9PB5YD+7RZ7nDSEdV7in1ZF/i3YtvrAJOA\necDXiuWfBtwBfABYG3hN8dl9pnh9H2BR8XwMcBXw8eJ9WwG3Ai9t+pweL/Z3LWA28Kem2G4HDmgR\n+1PbGjB/P2BF8Rk8HbgTeGuxfy8AHgCeXSx7D7BX8XxD4AXF812BR4rPYUzxb7N98dqvgO8W654M\nXAG8venzXAr8V7FP7wDuBtT07/+2bv/u+1Huw0dQVipJLwKmAadHxFWkpPGfxWtrAa8GPh4R/4yI\n64EfNb8/Ik6JiAcjYllEfJn0ZZ577uWkiFgQEf8iJcGdivlvBM6OiLMjYkVEnA9cSfoCH64NSV+s\n9zbt8xeKI7N/SvqfpmXvjohvFvvyr4i4JSLOj4gnIuJ+4CvA3sWyu5MS09ciYmlE/B8wf4gYdiEl\n109FxJMRcSvwPeB1TctcWuzvcuBkhig9DtPdgICJwMuA2yPipGL/rgZ+QUqskJLJDpLWj4iHitcB\njgR+UHwOKyLiroi4SdKmwEHA+4vfjfuArw7Ypzsi4nvFPv0IeCawaQn7ZTXlBGVlewtwXkQ8UEz/\nhJVlvkmkv7bvbFr+juY3S/qQpBslPSLpYWADYBPy3Nv0/DFgfPF8GnBYkUQeLtb7ItIX3CokvaGp\nOeCcQbbxEOko4qn3RsRHIp2HOqPYv4bm/UTSZEmnFeWrR4FTmvZtM+CuiGgevXmVz6bJNGCzAftz\nDKt+WQ/8LMaVcB5sc9JR68NFDLsNiOENwDOKZV9N+gPgDkkXS9qjmD+F9EfLYPu0NnBP0/q+SzqS\nWm2fIuKx4ul4bNTqtRO3VmOS1gX+A1hLUuPLZB1goqTnAdeTyl5TSCfdIZ0Pabx/L+CjwP7ADRGx\nQtJDpL/aR+JO4OSI+K92C0bEqcCpLV7/p6TLgUOBC9utbsD07GLecyPiQUmvAhrnS+4BNpekpiQ1\nlcG/zO8EbouIbdpsPzeuXIcAVxefwZ3AxRHx4kE3EDEfeKWktYF3k45op5Bi33qQt9wJPAFsEhHL\n1iA235ZhFPIRlJXpVaTzMzuQyms7Ac8CLgHeXJRmfklqGFhP0g6s2kQxgZTA7gfGSvo4sH4JcZ0C\nvFzSSyWtJWlc0XSwxRqu7yPAEZI+JmkyQLGuLdu8bwKwBHhY0ubAfze99kfSvr9X0lhJh5LO1wzm\nCuBRSR+VtG6xTzsqvw3876TzVm0p2VzSJ4C3kY7UAM4CtpX0JklrF49dJD1L0tOKI9ENImIp8Cjp\n9wJgDvBWSftLGlOse/uIuAc4D/iypPWL17aWtPfAmEa6T9Y7nKCsTG8hnQdaGBH3Nh6ko4Q3FCWm\nd5PKMveSuvxOanr/74BzgAWk8tbjDCiTrYmIuBN4JenL9f5inf/NGv7+R8SlpIaBmcCCohx1LulE\n/TdbvPWTpGaCR4DfkpJ1Y51Pko7KDieVEV/b/PqA7S8HXk76A+A2UnPC90nl0Byzgf8pSmkfHmKZ\nzSQtISXU+cBzSI0h5xUxLAZeQjpHdDfp3/PzpCNmgDcBtxelzKNI5wGJiCtIjRVfLT6Hi0nlPUjd\nn08D/lp8Bv/HIGXYIXwdeE3R4feNzPdYzWnVkreZmVk9+AjKzMxqyQnKzMxqyQnKzMxqyQnKzMxq\nqaeugxo/fpPYeOPp3Q7DzMxGYOHCqx6IiEntluupBLXxxtM59tgrux2GmZmNwNvfrqFGSVmFS3xm\nZlZLTlBmZlZLTlBmZlZLPXUOajBjxy5lq60Wsd56j3c7lGyPPTaOW2/dgmXL1u52KGZmtdXzCWqr\nrRYxZcoEJkyYjjTSQa+rFxEsXvwgsIgFC9qNLWpm1r96vsS33nqPM2HCxj2RnAAkMWHCxj11xGdm\n1g09n6CAnklODb0Wr5lZN4yKBGVmZqOPE1SJ/va3mzjooD3YYot1+Pa3v9TtcMzMelrPN0nUycSJ\nG/HZz36Dc875VbdDMTPreX2VoCa+ZBfGPHDfavNXbDKZh8+bP+L1T5o0mUmTJnP++b8d8brMzPpd\nXyWoMQ/cx4qNVx+fcLCkZWZm3eVzUGZmVktOUCM0Z8632Xffndh335249967ux2Omdmo0Vclvioc\neeS7OPLId3U7DDOzUccJqkR///u9vOQlM1i8+FHGjBnDiSd+jUsv/SsTJqzf7dDMzHpOXyWoFZtM\nHrKLrwybbvoMrr12USnrMjPrd32VoMpoJTczs85wk4SZmdWSE5SZmdVS1xKUpCmSLpR0o6QbJL2v\nW7GYmVn9dPMc1DLgQxFxtaQJwFWSzo+Iv3YxJjMzq4muHUFFxD0RcXXxfDFwI7B5t+IxM7N6qcU5\nKEnTgecDlw/y2ixJV0q6csmS+zsdWpb3ve8IdthhMjNn7tjtUMzMRo2uJyhJ44FfAO+PiEcHvh4R\nJ0bEjIiYMX786gO9DldE6+k18brXHc5pp5078hWZmdlTupqgJK1NSk6nRsQvq97ej38Mxx+/MilF\npOkf/3hk691jj5lMnLjRyAM0M7OndLOLT8Ac4MaI+ErV24uAJUvgjDNWJqnjj0/TS5aUcyRlZmbl\n6WYX357Am4DrJF1TzDsmIs6uYmMSvOMd6fkZZ6QHwCGHpPlSFVs1M7M11bUEFRGXAh1NC40k1UhO\n4ORkZlZXXW+S6KRGWa9Z8zkpMzOrj75JUM3nnA45BM47L/1sPie1pt7+9tdz8MF7cMstN/O8523B\nqafOKS9wM7M+1TejmUswfvyq55wa56TGjx9Zme+73/1pOUGamdlT+iZBAbz5zelIqZGMGknK56DM\nzOqnb0p8DQOTkZOTmVk9jYoEFT3W5dBr8ZqZdUPPJ6jHHhvH4sUP9syXfkSwePGDPPbYuG6HYmZW\naz1/DurWW7cAFrHeevUcSHYwjz02rojbzMyG0vMJatmytVmwYMtuh2FmZiXr+RKfmZmNTk5QZmZW\nS05QZgZUc680s5FwgjIz5s2DCy5Y9V5pF1yQ5pt1y5BNEpLOzHj/PyLi8PLCMbNOi4AnnoArrkjT\nBxyQktMVV8Cuu646+opZJ7Xq4nsW8LYWrwv4drnhmFmnSSkpQUpKjUS1665pfr8lp4EJ2Qm6e1ol\nqGMj4uJWb5b0yZLjMbMuaCSpRnKC/kxO8+alo8nGvjdKneusAzNndju6/jPkOaiIOL3dm3OWMbP6\na3wRN2s+J9UPmkudjX1vlDqfeKK/Pou6aHUO6jfAkP8kEfGKSiIys45q/iJulPUa01DtkVSdymlV\nlDrrtH+9qFWJ70vFz0OBZwCnFNOvB26vMCYz6yAplbCav4gbX9TrrFPdF2ody2llljrruH+9ZsgE\n1Tj/JOnTEdH8cf5GkptPzUaRmTNXv1da1UdOdewcHKrUOdzPoq7712tyxuKbJGmriLgVQNKWwKRq\nwzKzMuWUmjp5r7Q6dg4Op9TZ7vOs4/71opwLdT8AXCTpIkkXARcC7680KjMrTV0vwm3+Em/o5pf3\nUKXOXXddtdSZ+3nWbf96UdsjqIg4V9I2wPbFrJsi4olqwzKzMtS51FRWOa1M7Uqdw/k867h/vaZt\ngpK0HvBBYFpE/JekbSRtFxFnVR+emY1EFaWmFStgzJihp3N0s3NwJHI/zzLLhf0s59fqJOBJYI9i\nehHwmcoiMrNSlVlqmjMHjj8+JSVIP48/Ps0fbkwLF8LEibD//ml6//3T9MKF3fuCzinf5XyeZZcL\n+1VOgto6Ir4ALAWIiH+Rhjkysx5Q1kW4K1bA44/DTTetTFLHH5+mH398ZdLKjWnqVHjoIZg7N03P\nnZump07tzkWxuRfq5n6eM2eumrgaSarRYu4Lg9vL6eJ7UtK6FBftStoa8Dkosx5QZiltzBh4xzvg\nO99JSel970vzt9suzR9OmW+oUtluu63Z0V1umazVcjnlu+F+nq06I93p117Or9RxwLnAFEmnAnOB\nj1QZlJmVI7fUlOvSS2HbbVc9mth22zR/TWIro/SYWyYro3xX9ufpTr/W2iaoiDiPNJrE4cBPgRkR\ncVG1YZlZWdqVmnJFpFLe2Wencl5E+nn22Wn+cEtSZZQeh1OWK6t8V9bnmbu9fpbTxTcX+HJE/LZp\n3okRMavSyMxqpsxuq5x1lbm9nItw23XnRcCCBfDkk/C0p8GkSXD//Wl6wYLhdaaV1eWWWyYru3xX\nxkXNvdrJ2Ek5Jb4tgY9K+kTTvBkVxWNWS2V2W+Wsq9PdXTndeWPGwLrrwg47pOQE6ecOO6T5jWSW\nW0orq8stt0zW6fJdO53eXi/KSVAPA/sDm0r6jaQNKo7JrFbK7LbKWVenu7uG0513xBHpnFOzbbdN\n83P3r6GsLrfcMlmny3c5Or29XpPTxaeIWAa8U9LhwKXAhpVGZVYjZXZb5a6rk91dje68RlJqdOdt\nv/2q3XmNL/j581cvSTWPuDCc2Efa5ZZbJut0+W44Or29XpJzBHVC40lE/JDULHFeRfGY1dJwuq0G\n+8t9uOvqdHdXI0k1G9g6nluS6uRnNZyY1lkHdtll1eV22WX1clq7mKxzhkxQktYvnv5c0kaNB3Ab\n8OGORGdWE7llpJxzJjnr6nR3V6Os16z5nFRDTkmq059VmWUyj+xQL62OoH5S/LwKuLL4eVXTtFlf\nGFgeOuaY9LP53EhjuZzzS+3Wlbu9sjSfc9p+e/j619PP5nNSzVqVpDr9WeXE1Ly9+fNX3d78+d07\n92ftDXkOKiJeVvzcsqqNS/oB8DLgvojYsartmI3EUGUkGLq01er8S866OnmH2zFjYNy4Vc85Nc5J\njRs3/BEiBiulRaz5Z9VuXblx1e3cn7U3ZIKS9IJWb4yIq0vY/g+BbwE/LmFdZpXJveNsY36rW4bn\nrKvTd7g98shVr3tqJKnhjlI+HDmfVae31+mYrLVWv35fLh7fBi4HTgS+Vzz/Rhkbj4h5wD/KWJdZ\n1XK6rXLPv+Ssq9PdXQOT0Zokp5xSWvOyrT6r4awrN7a6nfuz1lqV+PYFkHQaMCsiriumd8RNEmar\nKWt0hIYy7rs01PpHMipFK7mltNzPqqySW872wCM71E3OdVDbN5ITQERcL2mnCmNahaRZwCyAjTaa\n2qnNmg1b7rmqefPSEcDAa3TWWWdl59mcOeki2UaZrdHIMG5cKscNR872ypRbShvOeb2Rltxyt9fJ\nc3/WXk6CuknS94FTgADeCNxYaVRNIuJEUnmRadNm+EDbaq2MW4ZHrDqyQ/NFtNtvP7wjqZztlf3F\nO1SZbE3Os+WuK0cdz/1ZazkJ6nDgHUBxfTnzgOOHXNqsz+WMjhCxatmquVNNGt59l8oYTLUswx0A\ndTgt62WU3Op47s+G1vLvMElrAd+PiK9GxCHF46sR8XgZG5f0U+CPwHaSFkkaZvHCrPdcckn7+bn3\nXSpzMNUy5I7s0Ol1WW9qeQQVEcslTZL0tIh4suyNR8Try16nWZ01yndz56Yv2AkTYPHiNL3//isT\nTfN9lxrnoM4+Gw4+eOURUm75rswyWY4yy2QuufW3nBLf7cBlks4E/tmYGRFfqSoos16W033XSByN\nx8DSVs59l3LKhWV3FuYuV2aZzCW3/pVzqvVu4Kxi2QlNDzMboN19laTUibfffunoqXEUtd9+ab6U\nf98lWFkWbC7xNc/PLZOVedt0s7K0PYKKiE92IhCzXjfwvkpDdd/ttRecf/6qZStI8xuOOCItM3/+\nynnbbgsvfvHK6Ua58Pe/T88b5cLf/z4lvMaRTRmdhcMpKZqVpW2CkjQJ+AjwbGBcY35E7FdhXGal\n6dRFqjn3VWoeDWGoeypB3n2XGhqJr1H2Gzi4K+R1FsLIb5tuVqacEt+pwE3AlsAnSeek5rd6g1ld\ndLok1e6+ShIsXAgTJ6amCCn9nDgxzW8kmdx7HN15J0ydmo6eIP2cOjXNH27HXE6nXyc7As1yEtTG\nETEHWBoRF0fEEcDuFcdlNmLduH1Cu/sqRaQE8tBDqXMvIv186KE0vxFT7n2XpkxJyWjx4jRv8eI0\nPWXK8PYvdww6j1VnnZTTxbe0+HmPpH8nNU1sUV1IZuXodEmqkZxuvBGe9ayV5b4bb1x5TmrMmMFj\n2m234V3EOnD+UB2BOXI7/aq4cNaslZwjqM9I2gD4EGmQ2O8DH6g0KrOSdLIkNWYMPPoobLIJHHVU\nmj7qqDT96KOrlvnKiCmnIzB3PcO5bbovnLVOyeniO6t4+giwb7XhmJWrkxepRsCMGXD55amT7oAD\n0s+xY9P8Ki6czekIzJF7QawvnLVOanXDwm+SBocdVES8t5KIzErS6ZLUUCXF5vJdmTHldAQOt1Gi\n1fRwlzMbqVZHUFcWP/cEdgB+VkwfBlxVZVBmZRiqJAXVlaQaXXnNt4dodOuVHVM39s+skxRt2m8k\nXQi8JCKWFtNrA+c1bmjYSdOmzYhjj72y/YJmTTp1HRSk9vXrrktdeY0jpg03hOc8Z/UOvLJi6uT+\nmZXh7W/XVRExo91yOU0Sm7Hq0Ebji3lmPaFTJakVK1JyuummlJSOPjr9vOmmNL/5AlqPVWfWXk6b\n+eeAPxdHUgB7A8dVFpFZjxozBnbcMT1/+GGYPTs93377NL95DD0f9Zi1l9PFd5Kkc4Ddilkfi4h7\nqw3LrDftvXfqoGskJ1j9JoOdvgW7Wa/KvHE0awH3Aw8B20ryfyOzQTRGhmjWGDGi8XqnR7cw61U5\ng8V+HngtcAPQqKIH6dbvZpXoxRJYbgt57ugWvfgZmJUp5xzUq4DtIuKJqoMxg94tgeW2fTfmN7ei\nD0xOvfoZmJUpp8R3K7B21YGYQe+XwHIHeW014GqvfwZmZck5gnoMuEbSXOCpoyiPJGFVGA33HGrV\n9l12GdBsNMtJUGcWD7OOyCmB9aoyy4Bmo11Om/mPOhGIWUMnB3jthpwBV0f7Z2CWI6eLbxtgNmk8\nvuZbvm9VYVw2irXqTmt8MV9++cpBVhvTMHq+oMsoA5rVybx5cNmv/g5Ll5W2zpwS30nAJ4Cvkm63\n8VbA/z1sjbTrTmvcEn3DDVe9JfqCBStviT7aeRBYq5vZ77mr/ULLl3P0xBOY9aqFbRfVqXnbzUlQ\n60bEXEmKiDuA4yRdQkpaZtmau9Ng1SODXXdd2Z02dWqaN3duWmbu3DR0UGOZfviC9n2XrGqzP/h3\neDz/6qHbXndM+4VmzgSmt1/u1LwMlZOgHpc0BvibpHcDdwGTs9Zu1iS3O80dbIkHgbXhmn3Mo+mv\nuUy3ve6YYVxY1/kL8HIS1PuB9YD3Ap8mlfneXGVQNnrldKe5g836ybw2Y/JcdhlwZ/uyWcNtOx8G\nRx6ZuXS9r/rOSVDTI2I+sIR0/glJhwGXVxmYjU453WnuYLNeN2dO3nL3XZUSz+S1HhxymcnA5VMO\nTfdvyZKbnOovJ0EdDfw8Y55ZSzndaeAONqun4SadPce1v/H4NuPglK/cn7HW3OQ0ugyZoCQdBBwM\nbC7pG00vrQ+U10dofSO3O80dbNYpzbdFaenuu57qUmtrIsz67PSRhGWFVkdQdwNXAq8Amv8UWAx8\noMqgbPTK6U5zB5uVYfYxj7Ze4PF/weNP5CWdCY2kM72EyCzXkAkqIq4FrpX0k4hYCiBpQ2BKRDzU\nqQBt9MnpTnMHmw1m9gf/nrfg40+w57irOGXSB4deZgKw554eHr7Gcs5BnS/pFcWy1wD3S7o4Ilr8\ny5tVz/dLGh1mfzBz9IHly5m81oNcvtNReSs+8kj69dzNaJGToDaIiEclvQ04KSI+IekvVQdm1orv\nl1Rva3Q9To6ZMxlNXWrWWk6CGivpmcB/AMdWHI9ZWzkjUvhIqhpz5qzsUmun7heBWv3lJKhPAb8D\nLo2I+ZK2Av5WbVhmQ/P9kso1bx5cdlr+haBHTzwhs0vNScdGJud2Gz+n6ZqniLgVeHWVQZm108+j\nTbQbeaChmqQD7mSzTml1HdSsiDix1Ztzlmnz/gOBrwNrAd+PiM+t6bqsv4zG0SZyLgS975q7nmoW\naGfPcbdnXgQKTjpWR62OoD4m6YEWrwt4H7BGCUrSWsC3gRcDi4D5ks6MiL+uyfqsf/TS/ZLmzSvG\nUmtncWoqaDf6wDZrwynfzE06Zr2tVYK6GHh5m/efP4Jt7wrcUpQMkXQa8ErACcpaqsP9ktpeBNpQ\nJJ2Z4+a3XXTWznOHMcin2ejX6kLdt1a87c2BO5umFwG7DVxI0ixgFsBGG02tOCTrFVWMNjGcpDN5\nrQe5fLND2y87gWKQz+kZK3ZyMmuW08VXlcG+SmK1Gekc14kA06bNWO1161+5o03Mng3c32YEguLG\nbbdN2bv9hree7ItAzTqgmwlqETClaXoL0vh/Zm3Nnk0awDPH8uXpHjntOOmY1Uo3E9R8YBtJW5Lu\n0vs64D+7GI912XAuAoXiaGfPPdsv6NEHzHpS2wQlaR3SdU/Tm5ePiE+NZMMRsay4hfzvSG3mP4iI\nG0ayTquf6i4CBR/tmI1uOUdQvwYeId1y44kyNx4RZwNnl7lO64zZ78xPOnuOu8rX45jZsOUkqC0i\n4sDKI7FKZY8+cO6jT3WptTJ5Lbj8m+1bp83M1lROgvqDpOdExHWVR2PDljX6wH3AnQuzRh+YDFz+\nui97SHAz67pWQx1dR2r7Hgu8VdKtpBKfgIiI53YmxP6TfRvqOxe2HXkAYBvglJ0/N4yLQJ2czKz7\nWh1BvaxjUfSJrAtBi/LakRN+3nbRWVN+WlwEmsNdbGbWW1qNJHEHgKSTI+JNza9JOhl406Bv7DNZ\nF4FCumPo8uXtLwQd1sgD7mIzs9Er5xzUs5snikFed64mnHqYMwfuuyEj6QA8/kRKOpMnt1/WF4Ka\nmWVrdQ7qaOAYYF1JjdqUgCdZwxHMu2nePLjs58MYeSAn6WyzTdFM4KRjZla2ViW+2cBsSbMjotbf\nwLPfk+6R087RE09g1oGZ1+/MrPUum5mNejklvp9LesGAeY8Ad0TEsgpiGtK9C58c8gLR275zbsYa\npuMLQc3MekNOgvoO8ALgL6QS33OAa4GNJR0VEedVGN8qnjP1Ea48NicRmZlZrxuTscztwPMjYkZE\n7AzsBFwPHAB8ocLYzMysj+UkqO2bB3Etbsn+/MadcM3MzKqQU+K7WdLxwGnF9GuBBcUo50sri8zM\nzPpazhHU4cAtwPuBDwC3FvOWAvtWFZiZmfW3tkdQEfEv4MvFY6AlpUdkZmZG3g0L9wSOA6ax6g0L\nt6ouLDMz63c556DmkEp7VwHtr4Y1MzMrQU6CeiQizqk8EjMzsyY5CepCSV8EfknTLd8j4urKojIz\ns76Xk6B2K37OaJoXwH7lh2NmZpbkdPG5ldzMzDoup4tvU+CzwGYRcZCkHYA9ImJO5dFZ7/jsZ2Hx\n4tXnT5gAxxzT+9szs47LuVD3h8DvgM2K6QWki3bNVlq8GMaPX/0xWBLpxe2ZWcflJKhNIuJ0YAVA\ncYsNt5ubmVmlchLUPyVtTGqMQNLupPtBmZmZVSani++DwJnA1pIuAyYBr6k0KjMz63s5XXxXS9ob\n2I50w8KbI8KjmJuZWaWGTFCSDh3ipW0lERG/rCgm60UTJgzdVVeFJUvg4YdXnz82pyiwBt7zHli2\nbPDtffOb5W/PXYpmLY+gXt7itSCNLGGWdPpLs9G1N9CSigbYX7Zs8OQ3WNIqQ6NLcbD5Zn1iyAQV\nEW/tZCBmZmbNcrr4zMzMOq6igr2NKjnnQ8o8R3PUURCx+nwJTjghPX/kkfQws1HLCcrayzkfUuY5\nmoiUjAab3/y8k+eEzKzj1qSLD8BdfNZdEiwfZECTwRJbGcaOHfoIsQqd7oo0qyF38fW7Trcz55Tv\nYPBlmg2WnAaTU3p0S7dZLbmLr991up05p3xXppzSY6dLmDncZm6Wdw5K0r8DzwbGNeZFxKfWdKOS\nDgOOA54F7BoRV67puszMbHTKuR/UCcB6wL7A90nj8F0xwu1eDxwKfHeE6+lPZZak/vGP9Ggl53zI\nihXw5JN526zqaGkww4mrDC4XmpUm5wjqhRHxXEl/iYhPSvoyIzz/FBE3AqiqE9qjXdnln3Ylt5wv\n1jFjyi0FFpVQAAAMj0lEQVSBlVkG7GRJ0aU5s9LkXKj7r+LnY5I2A5YCW1YX0qokzZJ0paQr769q\nGBszM6udnCOosyRNBL4IXE3q4Pt+uzdJugB4xiAvHRsRv84NMCJOBE4EmDFtWgdrQzWWU5YbTqmp\n3dFEN8pWZR7htFtXTgkzYvBS4cCjs7IuIHabuVlWgvpCRDwB/ELSWaRGicfbvSkiDhhpcNZCu7JV\nbqkppzRXZtlqo43yBnltt3+5JcWc7eUk2Q03zIu7rAuIfb7KLKvE98fGk4h4IiIeaZ5nZmZWhVYj\nSTwD2BxYV9LzSTcrBFif1NW3xiQdAnyTdHfe30q6JiJeOpJ19p2ySmBljZCQu57c0lW7/St7e2Xp\n9AgX7hq0UazVt9BLgcOBLYCvNM1/FBjRb35EnAGcMZJ19LUyO+bKutle7nrK6ggsc3tl2mCDzt6n\nyl2DNoq1GkniR8CPJL06In7RwZjMzMyymiQukzQH2CwiDpK0A7BHRMypOLb+lFOyySlvlVnaGu1l\nshy5n4G778xKk5OgTioexxbTC4CfAU5QVcgp2eSUt8osbY32MlmO3M/A533MSpPTxbdJRJwOrACI\niGVA5lDSZmZmaybnCOqfkjYmXaCLpN0B38p0tKhjF5jLZPn8WdkolpOgPgicCWwt6TJSa/hrKo3K\nOqeOXWAuk+XzZ2WjWNsEFRFXS9ob2I50LdTNEbG08sjMzKyv5dxuYxzwTuBFpDLfJZJOiIi2wx3Z\nGnDJxswMyCvx/RhYTBr5AeD1wMnAYVUF1ddcsjEzA/IS1HYR8bym6QslXVtVQGZmZpCXoP4safeI\n+BOApN2Ay6oNyzrGJUUzq6mcBLUb8GZJC4vpqcCNkq4DIiKeW1l0Vj2XFM2spnIS1IGVR2FmZjZA\nTpv5HZ0IxMzMrFnOUEdmZmYd5wRlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARl\nZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma15ARlZma1\n5ARlZma15ARlZma15ARlZma15ARlZma15ARlZma11JUEJemLkm6S9BdJZ0ia2I04zMysvrp1BHU+\nsGNEPBdYABzdpTjMzKymupKgIuK8iFhWTP4J2KIbcZiZWX3V4RzUEcA53Q7CzMzqZWxVK5Z0AfCM\nQV46NiJ+XSxzLLAMOLXFemYBswCmbrRRBZGamVkdVZagIuKAVq9LegvwMmD/iIgW6zkROBFgxrRp\nQy5nZmajS2UJqhVJBwIfBfaOiMe6EYOZmdVbt85BfQuYAJwv6RpJJ3QpDjMzq6muHEFFxL91Y7tm\nZtY76tDFZ2ZmthonKDMzqyUnKDMzqyUnKDMzqyUnKDMzqyUnKDMzqyUnKDMzqyUnKDMzqyW1GAav\ndiTdD9zR7ThKsAnwQLeD6JB+2lfw/o5m/bSvUO3+TouISe0W6qkENVpIujIiZnQ7jk7op30F7+9o\n1k/7CvXYX5f4zMyslpygzMyslpyguuPEbgfQQf20r+D9Hc36aV+hBvvrc1BmZlZLPoIyM7NacoIy\nM7NacoLqAklflHSTpL9IOkPSxG7HVCVJh0m6QdIKSaO2TVfSgZJulnSLpI91O54qSfqBpPskXd/t\nWKomaYqkCyXdWPwev6/bMVVJ0jhJV0i6ttjfT3YrFieo7jgf2DEingssAI7ucjxVux44FJjX7UCq\nImkt4NvAQcAOwOsl7dDdqCr1Q+DAbgfRIcuAD0XEs4DdgXeN8n/bJ4D9IuJ5wE7AgZJ270YgTlBd\nEBHnRcSyYvJPwBbdjKdqEXFjRNzc7TgqtitwS0TcGhFPAqcBr+xyTJWJiHnAP7odRydExD0RcXXx\nfDFwI7B5d6OqTiRLism1i0dXuumcoLrvCOCcbgdhI7Y5cGfT9CJG8ZdYv5I0HXg+cHl3I6mWpLUk\nXQPcB5wfEV3Z37Hd2Gg/kHQB8IxBXjo2In5dLHMsqXxwaidjq0LO/o5yGmSer+EYRSSNB34BvD8i\nHu12PFWKiOXATsX58TMk7RgRHT/f6ARVkYg4oNXrkt4CvAzYP0bBxWjt9rcPLAKmNE1vAdzdpVis\nZJLWJiWnUyPil92Op1Mi4mFJF5HON3Y8QbnE1wWSDgQ+CrwiIh7rdjxWivnANpK2lPQ04HXAmV2O\nyUogScAc4MaI+Eq346mapEmNzmJJ6wIHADd1IxYnqO74FjABOF/SNZJO6HZAVZJ0iKRFwB7AbyX9\nrtsxla1oenk38DvSSfTTI+KG7kZVHUk/Bf4IbCdpkaQjux1ThfYE3gTsV/x/vUbSwd0OqkLPBC6U\n9BfSH17nR8RZ3QjEQx2ZmVkt+QjKzMxqyQnKzMxqyQnKzMxqyQnKzMxqyQnKzMxqyQnKRjVJ+0ha\nrUV2qPklbO9VzQOJSrqo3QjuRSyPSDq7zXLHlBVnsb4l7Zdq+f7DJX2reH6UpDeXENPtkjaRtG7R\nzv2kpE1Gul7rTU5QZuV6FWk08+G6JCLaXVtTaoIaDiVDfl9ExAkR8eOythcR/4qInfBoHH3NCcq6\nStLTJf22uPfM9ZJeW8zfWdLFkq6S9DtJzyzmXyTpa5L+UCy/azF/12Len4uf2w0zhh9Iml+8/5XF\n/MMl/VLSuZL+JukLTe85UtKCIp7vSfqWpBcCrwC+WPz1v3Wx+GHF/XUWSNorI55nSppXrON6SXtJ\n+hzQOKo4tVjuV8Xnc4OkWU3vXyLpf4vP9E+SNi3mbynpj8V+frpp+fGS5kq6WtJ1Tfs/XekeSN8B\nrgamSHprsR8Xky5gbazjOEkflrRZ08Ws10haLmlaMTrBL4ptz5e0Z/G+jSWdV3zu32XwMQ2tX0WE\nH3507QG8Gvhe0/QGpOH9/wBMKua9FvhB8fyixvLATOD64vn6wNji+QHAL4rn+wBnDbLdp+YDnwXe\nWDyfSLpH19OBw4Fbi5jGAXeQxtvbDLgd2KiI9RLgW8X7fwi8pmk7FwFfLp4fDFzQKpZi+kOkQXYB\n1gImFM+XDHjfRsXPdUnjpG1cTAfw8uL5F4D/KZ6fCby5eP6uxvpIY3KuXzzfBLiFlCimAyuA3YvX\nngksBCYBTwMua9rv44APD4jvXaQRNQB+AryoeD6VNGwQwDeAjxfP/72IfZOmddzePO1Hfz08WKx1\n23XAlyR9nvQlfYmkHYEdSUNBQfqSvqfpPT+FdE8iSesrjRs2AfiRpG1IX3JrDyOGlwCvkPThYnoc\n6UsUYG5EPAIg6a/ANNKX+MUR8Y9i/s+BbVusvzG46FWkL/125gM/UBqg9FcRcc0Qy71X0iHF8ynA\nNsCDwJNA4/zaVcCLi+d7kv4gADgZ+HzxXMBnJc0kJaTNgU2L1+6IiD8Vz3cDLoqI+wEk/Ywh9rs4\nQnob0DhiPADYofj3BFhf0gTSHxmHAkTEbyU9NMS+Wh9ygrKuiogFknYmHV3MlnQecAZwQ0TsMdTb\nBpn+NHBhRByidM+ei4YRhoBXx4CbKkrajXR30YblpP8zwy1DNdbReH9LReKdSTqiOFnSF2PA+R1J\n+5C+9PeIiMeURpweV7y8NCIan9HAbQ42ttkbSEdFO0fEUkm3N63rnwPDaxd/UY6dQxoMudGIMaaI\n9V8Dls1ap/Unn4OyrpK0GfBYRJwCfAl4AXAzMEnSHsUya0t6dtPbGuepXgQ8UhzhbADcVbx++DDD\n+B3wHhXflpKe32b5K4C9JW0oaSwrj0oAFpOO5taYpGnAfRHxPdIX/QuKl5YWR1WQ9vehIjltT7oV\neTuXkUZZh5SUGjYotrdU0r6ko8TBXA7sU5w3Whs4bJDY1wZOBz4aEQuaXjqPNJhuY7mdiqfzGrFI\nOgjYMGM/rE84QVm3PQe4QununccCn4l0y/TXAJ+XdC1wDfDCpvc8JOkPwAlAYxTtL5COwC4jlQSH\n49OkkuBfJF1fTA8pIu4inbe6HLgA+CvwSPHyacB/Fyf9tx5iFe3sA1wj6c+k5Pf1Yv6JRYynAucC\nY5VGnP408KfBVjTA+4B3SZpPSkoNpwIzJF1JShaD3lohIu4hnWv6I2m/rx5ksRcCuwCfbGqU2Ax4\nb7GNvxSl0qOK5T8JzJR0NanUujBjP6xPeDRz6ylFKevDEXFll+MYHxFLiiOoM0hNHGes4br2Ie3T\ny8qMcTQoyo0zIuKBbsdinecjKLM1c1xx1Hc9cBvwqxGs60lgR7W5ULefqLhQl3Rku6Lb8Vh3+AjK\nzMxqyUdQZmZWS05QZmZWS05QZmZWS05QZmZWS05QZmZWS/8f4u+vh0ZpMloAAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfWd//HX52YhCUvCEiAJyC5VAUHj7s/WrVihSrWt\n7aily692saPVDq3M/NrpMjO2Q6vVqbWltVPbWrVVRAtVUMS9omyCiCyyJyxhCVsSsn1+f9wTDZqQ\nm5CTc2/yfj4e53HvOTm5930fLJ/7Xc73mLsjIiKSbGJRBxAREWmKCpSIiCQlFSgREUlKKlAiIpKU\nVKBERCQpqUCJiEhSUoESEZGkpAIlIiJJSQVKRESSUnrUARLRr18/Hzp0aNQxRESkHSxZsmS3u+e3\ndF5KFKihQ4eyePHiqGOIiEg7MLPNiZynLj4REUlKKlAiIpKUVKBERCQpqUCJiEhSUoESEZGkFOos\nPjPbBBwE6oBady82sz7Aw8BQYBPwaXffF1aG2ctKmDFvDaXllRTmZTNt4mimTCgK6+1ERKSddEQL\n6kJ3H+/uxcH+bcACdx8FLAj2QzF7WQnTZ62kpLwSB0rKK5k+ayWzl5WE9ZYiItJOoujiuxK4P3h+\nPzAlrDeaMW8NlTV1Rx2rrKljxrw1Yb2liIi0k7ALlAPzzWyJmd0QHBvg7tsBgsf+Tf2imd1gZovN\nbHFZWVmb3ry0vLJVx0VEJHmEXaDOc/fTgI8BN5rZBYn+orvPdPdidy/Oz29xRYwmFeZlt+q4iIgk\nj1ALlLuXBo+7gMeAM4GdZlYAEDzuCuv9p00cTXZG2lHHsjPSmDZxdFhvKSIi7SS0AmVm3c2sZ8Nz\n4KPAm8ATwNTgtKnA42FlmDKhiNuvGkthbhYAOZlp3H7VWM3iExFJAWFOMx8APGZmDe/zZ3d/ysxe\nB/5iZl8CtgCfCjEDUyYUMWVCEbc8vJxn397F5WMLwnw7ERFpJ6EVKHffAJzaxPE9wMVhvW9zJo8r\n4LFlJbz8zm4uHN3kvAwREUkiXWYlifNH9aNnVjpzV2yPOoqIiCSgyxSobulpfPTkgcxbtYMjtXUt\n/4KIiESqyxQogMmnFnCwqpaX1u2OOoqIiLSgSxWo80b0Izc7Q918IiIpoEsVqMz0GBNPGcD8t3ZS\nVaNuPhGRZNalChTA5HGFHDpSywtr27Z8koiIdIwuV6DOGdGX3jkZzF2pbj4RkWTW5QpURlqMy8YM\n5Bl184mIJLUuV6Ag3s13uLqO59aEtgygiIgcpy5ZoM4a1oe+3TOZo9l8IiJJq0sWqPSgm2/B6l1U\nVNdGHUdERJrQJQsUwKRxBVTW1LHwbc3mExFJRl22QJ01rC/9enRj7srSqKOIiEgTumyBSosZl48d\nyLNv7+LwEXXziYgkmy5boAAmjS2gqqaeBW9rNp+ISLLp0gXqjKF96N+zG3NXqJtPRCTZdOkCFYsZ\nl48tYOGaMg6pm09EJKl06QIF8TvtVtfW88xbO6OOIiIijXT5AnXaCb0pyM3SRbsiIkmmyxeohm6+\nF9aWcaCqJuo4IiIS6PIFCuIX7VbX1fP0KnXziYgkCxUoYMLgPIrysnULDhGRJKICBZgZk8YV8OK6\nMvZXqJtPRCQZqEAFJo0toKbOmffWjqijiIgIKlDvGjcol8F9spmr2XwiIklBBSpgZkwaW8jL63ez\n73B11HFERLo8FahGJo8roLbema9uPhGRyKlANXJKYS+G9M3RRbsiIklABaoRM2PyuAJeeWcPew4d\niTqOiEiXpgL1PpPGFlJX78zTRbsiIpFSgXqfkwp6Mrxfd+boFhwiIpFSgXqfhm6+VzfsoeyguvlE\nRKKiAtWESeMKqXd4apVm84mIREUFqgknDujByP49mPOGuvlERKKiAtWEhm6+1zbtZdeBqqjjiIh0\nSSpQzZg0tgB3ePJNdfOJiEQh9AJlZmlmtszM5gT7w8xskZmtM7OHzSwz7AxtMWpAT0YP6KnZfCIi\nEemIFtTNwOpG+z8B7nT3UcA+4EsdkKFNJo8r4PVN+9ixX918IiIdLdQCZWaDgEnAb4N9Ay4CHglO\nuR+YEmaG43H5uAIA/q4bGYqIdLiwW1A/B74N1Af7fYFyd68N9rcBRU39opndYGaLzWxxWVlZyDGb\nNiK/BycV9NKddkVEIhBagTKzycAud1/S+HATp3pTv+/uM9292N2L8/PzQ8mYiMnjCliyeR+l5ZWR\nZRAR6YrCbEGdB1xhZpuAh4h37f0cyDOz9OCcQUBSz0KYNFbdfCIiUQitQLn7dHcf5O5Dgc8Az7r7\ntcBC4JPBaVOBx8PK0B6G9uvOmKJeugWHiEgHi+I6qO8At5rZeuJjUvdFkKFVJo0tZPnWcrburYg6\niohIl9EhBcrdn3P3ycHzDe5+pruPdPdPuXvSr8g6WbP5REQ6nFaSSMDgPjmcOihXs/lERDqQClSC\nJo0rYMW2/WzeczjqKCIiXYIKVIIuD2bzqRUlItIxVKASNKh3DhNOyGOuZvOJiHQIFahWmDS2gFWl\nB9i4W918IiJhU4FqhUnBbL65WuFcRCR0KlCtUJCbTfGQ3rpoV0SkA6hAtdKkcQW8veMg63cdijqK\niEinpgLVSpePLcAMTZYQEQmZClQrDeiVxRlD+zB3pcahRETC1GKBCm7Z/qeOCJMqJo8rYO3OQ6zd\neTDqKCIinVaLBcrd64B8M8vsgDwp4bIxA4kZmiwhIhKi9JZPAWAT8LKZPQG8exGQu98RRqhk179n\nFmcN68vcFaXccsko4neyFxGR9pToGFQpMCc4v2ejrcuaNK6Ad8oOs0bdfCIioUioBeXuPwAws57x\nXe/yc6wvGzOQ7z3+JnPe2M6HBvaKOo6ISKeTUAvKzMaY2TLgTWCVmS0xs1PCjZbc+vXoxrkj+jF3\n5XbcPeo4IiKdTqJdfDOBW919iLsPAb4F/Ca8WKlh0rgCNu4+zFvbD0QdRUSk00m0QHV394UNO+7+\nHNA9lEQpZOIpA0mLmWbziYiEINFZfBvM7LvAH4P964CN4URKHX26ZzKyfw9mvrCBXz33DoV52Uyb\nOJopE4qijiYikvISbUF9EcgHZgVbP+ALYYVKFbOXlbCh7BB19Y4DJeWVTJ+1ktnLSqKOJiKS8lps\nQZlZGvCv7n5TB+RJKTPmraGm7ugJEpU1dcyYt0atKBGR45ToShKnd0CWlFNaXtmq4yIikrhEx6CW\nBatI/JWjV5KYFUqqFFGYl01JE8WoMC87gjQiIp1LomNQfYA9wEXAx4NtclihUsW0iaPJzkg76lhW\nRoxpE0dHlEhEpPNIdAxqhbvf2QF5UkrDONOMeWsoLa/EgQtH99f4k4hIO2ixQLl7nZldAahANWHK\nhKJ3C9KX/7CYl9bvZn9lDbnZGREnExFJbYl28b1iZr8ws/9jZqc1bKEmS0G3XHIiB6tque/FDVFH\nERFJeYlOkjg3ePxho2NOfExKAicX9uLysQP53cub+MJ5w+jdXbfQEhFpq4RaUO5+YRObilMTvnnJ\niRyuruXXL6gVJSJyPBJdzXyAmd1nZk8G+yeb2ZfCjZaaThzQkytOLeT+Vzax+9CRqOOIiKSsRMeg\nfg/MAwqD/bXAN8MI1BncfPEojtTW8avn3ok6iohIykq0QPVz978A9QDuXgvUhZYqxQ3P78EnJgzi\nj69uZueBqqjjiIikpEQL1GEz60t8YgRmdjawP7RUncDNF4+irt755cL1UUcREUlJiRaoW4EngBFm\n9jLwB+CfQ0vVCZzQN4dPFQ/iwde2NrkckoiIHFuis/iWAh8mPt38K8Ap7r4izGCdwTcuGoXj3KNW\nlIhIqyXagsLda919FXCTu9e0dL6ZZZnZa2b2hpmtMrMfBMeHmdkiM1tnZg+bWae9WKgoL5vPnHEC\nf3l9K1v3VkQdR0QkpSRcoBopTvC8I8BF7n4qMB64LBi7+glwp7uPAvYBnXq6+o0XjiQWM+5esC7q\nKCIiKaUtBWpXIid53KFgNyPYGlafeCQ4fj8wpQ0ZUsbA3CyuO2sIs5aVsHH34ZZ/QUREgDYUKHe/\nLNFzzSzNzJYTL2pPA+8A5cE0dYBtQJNLf5vZDWa22MwWl5WVtTZmUvnaR0aQmRbjrmfWRh1FRCRl\nHHMtPjP7G8HU8qa4+xXH+v3gbrzjzSwPeAw4qanTmvndmcBMgOLi4mYzpIL8nt343LlDmPnCBr5x\n0UhG9u8ZdSQRkaTXUgvqp8DPgI1AJfCbYDsEvJnom7h7OfAccDaQZ2YNhXEQUNq6yKnpKxeMICcj\njTuf0ViUiEgijlmg3P15d38emODu17j734Ltn4Dzj/W7ZpYftJwws2zgEmA1sBD4ZHDaVODx4/0Q\nqaBP90y+cN4w5q7YzurtB6KOIyKS9BIdg8o3s+ENO2Y2DMhv4XcKgIVmtgJ4HXja3ecA3wFuNbP1\nQF/gvtbHTk1f/j/D6ZmVzp1PayxKRKQlid4P6hbgOTNruIfEUOIX7DYruJB3QhPHNwBntiJjp5Gb\nk8H/PX84dz6zlpXb9jN2UG7UkUREklaiK0k8BYwCbg620e4+L8xgndUXzh9KbnYGd2pGn4jIMSV6\nP6gcYBrwDXd/AzjBzCaHmqyT6pWVwQ0XDOfZt3exdMu+qOOIiCStRMeg/heoBs4J9rcB/xFKoi7g\n8+cOpW/3TI1FiYgcQ6IFaoS7/zdQA+DulYCFlqqT694tna9+eAQvrtvNaxv3Rh1HRCQpJVqgqoOp\n4g33gxpBfK09aaPrzh5Cfs9u3PH0mqijiIgkpUQL1L8DTwGDzewBYAHw7dBSdQHZmWl8/SMjeHXD\nXl5ZvzvqOCIiSafFAmVmBrwNXAV8HngQKHb350JN1gV89swTKMjN4mdPr8U9pVdzEhFpdy0WKI//\nzznb3fe4+1x3n+Pu+srfDrIy0rjxwpEs2byP59em9oK4IiLtLdEuvlfN7IxQk3RRny4eTFFeNneo\nFSUicpREC9SFwD/M7B0zW2FmK4MljOQ4ZabHuOnikazYtp8FqxO61ZaISJeQ6FJHHws1RRd31WmD\n+OVz73DH02u56EP9icU0g19EJNGljja7+2bit9zwRpu0g4y0GDdfPIq3th9g3qodUccREUkKiS51\ndIWZrSN+X6jngU3AkyHm6nKuHF/E8Pzu3PnMWurqVftFRBIdg/oR8ZsNrnX3YcDFwMuhpeqC0mLG\nNy85kbU7DzFnRZe4h6OIyDElWqBq3H0PEDOzmLsvBMaHmKtLmjy2gNEDenLXM+uorauPOo6ISKQS\nLVDlZtYDeAF4wMzuAmrDi9U1xWLGLZeOYsPuwzy+XK0oEenaEi1QVxKfIHEL8SWP3gE+Hlaormzi\nKQM5pbAXdy1YR41aUSLShSU6i++wu9e5e6273+/udwddftLOzIxbLjmRLXsreHTJtqjjiIhEJtFZ\nfAfN7ECwVZlZnZkdCDtcV3XxSf05dXAe//Pseqpr1YoSka4p0RZUT3fvFWxZwNXAL8KN1nWZGbde\neiIl5ZU8vHhr1HFERCKR6BjUUdx9NnBRO2eRRi4Y1Y/iIb2559n1VNXURR1HRKTDJdrFd1Wj7ZNm\n9mO0kkSozIxbP3oiOw5U8edFW6KOIyLS4RJdi6/xjL1a4itJXNnuaeQo547ox4j87vzH3Lf40Zy3\nKMzLZtrE0UyZUBR1NBGR0CVUoNz9C2EHkQ+avayErfsqaVj5qKS8kumzVgKoSIlIp5dQgTKzu4/1\nc3e/qX3iSGMz5q35wCy+ypo6ZsxbowIlIp1eopMksoDTgHXBNh6oA5YEm4SgtLyyVcdFRDqTRMeg\nRgEXunsNgJn9Cpjv7reElkwozMumpIliVJiXHUEaEZGOlWgLqhDo2Wi/R3BMQjRt4miyM9I+cPyj\np/SPII2ISMdKtAX1Y2CZmS0M9j8MfD+URPKuhnGmGfPWUFpeSUFuFmlpxl8Xl3D92UMZnt8j4oQi\nIuEx98QuZzKzgcBZwe4id++wW78WFxf74sWLO+rtklpJeSWT736R/j2zeOzGc8nJTPQ7hohIcjCz\nJe5e3NJ5iV6oex5w0N0fJ97V920zG3KcGaUNivKyueszE1i76yD/OmsliX7BEBFJNYmOQd0LVJjZ\nqcA0YDPwh9BSyTFdcGI+t15yIrOXl/KnVzdHHUdEJBSJFqhaj39VvxK4293v4uhJE9LBbrxwJBeO\nzueHc95i2ZZ9UccREWl3iRaog2Y2HbgOmGtmaUBGeLGkJbGYcec14xnQK4uvP7CUPYeORB1JRKRd\nJVqgrgGOAF8KJkcUATNCSyUJycvJ5N5rT2fP4Wq++fBy6uo1HiUinUei94Pa4e53uPuLZjbZ3be4\nu8agksDYQbn88IpTeHHdbu56Zm3UcURE2k1b7gf1w0ROMrPBZrbQzFab2Sozuzk43sfMnjazdcFj\n7zZkkEauOWMwnzp9EHc/u55n394ZdRwRkXbRlgJlCZ5XC3zL3U8CzgZuNLOTgduABe4+ClgQ7Mtx\nMDN+NGUMJxf04paH32Dr3oqoI4mIHLe2FKivJHKSu29396XB84PAauJjV1cC9wen3Q9MaUMGeZ+s\njDR+dd3puDtfe2CJ7sIrIikv0Qt108zsCjO7CTjfzG41s1sTfRMzGwpMABYBA9x9O8SLGNDkwnJm\ndoOZLTazxWVlZYm+VZd2Qt8c7vj0eN4sOcD3n1gVdRwRkeOSaAvqb8Dngb7Er39q2FpkZj2AR4Fv\nuvuBRIO5+0x3L3b34vz8/ER/rcu75OQB3HjhCB56fSt/eX1r1HFERNos0YXcBrn7uNa+uJllEC9O\nD7j7rODwTjMrcPftZlYA7Grt68qx3XrpaJZvLee7j7/JyYW9GFOUG3UkEZFWS7QF9aSZfbQ1L2xm\nBtwHrHb3Oxr96AlgavB8KvB4a15XWpYWM+76zAR652Ty9QeWsr+iJupIIiKtlmiBehV4zMwqzeyA\nmR00s5a6684DrgcuMrPlwXY58Vt3XGpm64BLg31pZ/16dOOea0+jtLySb/11OfW6iFdEUkyiXXw/\nA84BVnqCy2e7+0s0PyX94gTfV47D6UN68/8mncT3//YW9z7/DjdeODLqSCIiCUu0BbUOeDPR4iTJ\nY+q5Q7ni1EJ+Nn8NL6/fHXUcEZGEJdqC2g48Z2ZPEl+TD4D3jS1JEjIzbr9qLKu3H+CmB5cx56bz\nKcjNjjqWiEiLEm1BbSS+6kMmrZxmLtHr3i2de687naqaOr7+wFKqa+ujjiQi0qKEWlDu/oOwg0i4\nRvbvwX9/8lRu/PNS/uvvq/n+FadEHUlE5JgSKlBmthD4wPiTu1/U7okkNJPGFbB0yzDue2kjE07I\n48rxRVFHEhFpVqJjUP/S6HkWcDXxxWAlxdz2sQ/xxtZybnt0JScV9OLEAeqpFZHklOj9oJY02l52\n91uBs0LOJiHISItxz7Wn0b1bGl/90xIOHdH3DBFJTokuFtun0dbPzC4DBoacTUIyoFcW//PZ09i8\np4LvPLICXT0gIsko0S6+Jbw3BlULbAK+FEYg6RjnjOjLtyeO5vYn3ybjYeP1TfsoLa+kMC+baRNH\nM2WCxqdEJFrHLFBmdgaw1d2HBftTiY8/bQLeCj2dhOqGC4bztzdKmb289N1jJeWVTJ+1EkBFSkQi\n1VIX36+BagAzuwC4nfhNBvcDM8ONJmEzM/Ycrv7A8cqaOmbMWxNBIhGR97TUxZfm7nuD59cAM939\nUeBRM1sebjTpCDv2VzV5vLS8soOTiIgcraUWVJqZNRSxi4FnG/0s0fErSWKFeU0ve9TccRGRjtJS\ngXoQeN7MHgcqgRcBzGwk8W4+SXHTJo4mOyPtA8fHD87V7D4RidQxW0Hu/p9mtgAoAOY3Ws08Bvxz\n2OEkfA0TIWbMW0NpeSUFuVkMzM1i7sodZP11BbdfNZbM9ESXbBQRaT8tdtO5+6tNHFsbThyJwpQJ\nRUfN2HN37lqwjp8/s45t+yr49fWnk5eTGWFCEemK9NVYPsDM+OYlJ/Lza8azbEs5n/jlK2zcfTjq\nWCLSxahASbOmTCjigS+fRXlFNZ/45cu8tnFvy78kItJOVKDkmM4Y2ofZN55Hn+6ZXPvbV5m1dFvU\nkUSki1CBkhYN6dudx752HsVD+nDrX97gjvlrNMNPREKnAiUJyc3J4P4vnsmniwdx97Pruemh5VTV\n1EUdS0Q6MV1sKwnLTI/xk6vHMaxfD37y1NuU7KvgN58rpm+PblFHE5FOSC0oaRUz42sfGcG9157G\nqtIDTPnly6zbeTDqWCLSCalASZt8bGwBD3/lHCqr67nq3ld4ad3uqCOJSCejAiVtNn5wHrNvPJfC\n3Gym/u9rPPjalqgjiUgnogIlx2VQ7xwe+do5nD+yH9NnreT2v6+mvl4z/ETk+KlAyXHrmZXBfVOL\nuf7sIfz6hQ187YElVFTXRh1LRFKcCpS0i/S0GD+88hS+N/lk5r+1k2t+/So7DzR9rykRkUSoQEm7\nMTO+eP4wfnN9Me+UHWLKPS/zVumBqGOJSIqyVFgRoLi42BcvXhx1DGmFVaX7+dLvF3OwqoZrzzqB\nuSt3UFpeSWFeNtMmjj5q9XQR6VrMbIm7F7d0nlpQEopTCnN5/Bvn0Tsng5kvbqSkvBIHSsormT5r\nJbOXlUQdUUSSnAqUhGZAryzqmmigV9bUMWPemo4PJCIpRQVKQrVjf9MTJUrLKzs4iYikGhUoCVVh\nXnaTx83gj//YRG1dfccGEpGUoQIloZo2cTTZGWlHHeuWHmNYv+589/FVXHbXiyx8e5du3yEiHxBa\ngTKz35nZLjN7s9GxPmb2tJmtCx57h/X+khymTCji9qvGUpSXjQFFedn85OpxPHPrh5l5/enU1Ttf\n+P3rfO53r/H2Dk1JF5H3hDbN3MwuAA4Bf3D3McGx/wb2uvuPzew2oLe7f6el19I0886ruraeP726\nmbsWrONgVQ3XnDGYWy49kf49s6KOJiIhiXyaubu/AOx93+ErgfuD5/cDU8J6f0kNmekxvnj+MJ6f\n9hE+f+4w/rp4GxfOeI57Fq7XDRFFuriOHoMa4O7bAYLH/s2daGY3mNliM1tcVlbWYQElGnk5mXzv\n4ycz/5YLOG9kP2bMW8PFP3uex5eXaHxKpItK2kkS7j7T3YvdvTg/Pz/qONJBhuf3YObninnwy2eT\nl5PBzQ8t5xO/fIUlm9/fGBeRzq6jC9ROMysACB53dfD7S4o4Z0Rf/vaN8/npp05l+/5Krr73H9z4\n56Vs3VsRdTQR6SAdXaCeAKYGz6cCj3fw+0sKicWMT54+iIX/8hG+eckonl29i4t/9jy3P7maA1U1\nUccTkZCFOYvvQeAjQD9gJ/DvwGzgL8AJwBbgU+7eYt+NZvEJxFel+On8NTy6dBu9czK55dIT+ewZ\ng5mzYjsz5q3RYrQiKSLRWXxazVxSzpsl+/nRnLdYtHEvA3p2Y19FDdWNVqTIzkjj9qvGqkiJJKnI\np5mLhGVMUS4P3XA2v77+dHYfrj6qOIEWoxXpLFSgJCWZGRNPGUh9fdM9AFqMViT1qUBJSmtuMVoH\npv7uNZ54o1QX/IqkKBUoSWnNLUZ76Un9WbfzIDc9uIwz/vMZps9ayZLNe3XRr0gKSY86gMjxaJgI\n0dQsvvp65x8b9vDokm3MXlbCg69tYVi/7lw1oYhPnFbEoN45EacXkWPRLD7pEg4dqeXvK7fz6JJt\nLNoYv7LhnOF9ufr0QXxszEC6d9N3NZGOomnmIs3YureCWUtLmLVsG5v3VJCTmcZlYwbyydMGcfbw\nvsRiFnVEkU5NBUqkBe7O4s37eHTJNuau2M7BI7UU5WXziQlFXH36IIb16w7A7GUluhBYpB2pQIm0\nQlVNHfNW7eDRpSW8tK6MeofTTshjRP8e/O2NUqpqdCGwSHtRgRJpo50HqnhsWQmPLtnGul2Hmjyn\nKC+bl2+7qIOTiXQOWklCpI0G9Mriqx8ewfxbLqC50aiS8krmrdrBvsPVHZpNpCvR1CWRZpgZhXnZ\nlDSzKsVX/rgEgNEDenLW8D6cNawvZw7rQ37Pbh0ZU6TTUoESOYZpE0czfdZKKhutRpGdkcaPrjyF\nIf26s2jDHhZt3MsjS7bxh39sBmB4fnfOGtaXs4OiNTA3K6r4IilNBUrkGI51ITDAGUP78A2gpq6e\nN0v2s2jjXhZt2MOcN0p58LUtAJzQJ4ezhvXhrOF9OWtYHwb3OfoCYc0SFGmaJkmIhKCu3lm9/QCv\nBi2s1zftpbwifpPForxszhzWh7OG9eFgVS13PL2GSs0SlC5Es/hEkkh9vbN210EWbdjLaxv3smjj\nHnYfan6ChWYJSmemAiWSxNydd8oOc8kdzzd7zpnD+jCqfw9G9u/BqP49Gdm/BwN6dcNMK11Iaku0\nQGkMSiQCZsbI/j0oamaWYE5mGvX1zpwV29lfWfPu8Z7d0hnxbtF6r3gV9c4m7RhLNGmcS1KRCpRI\nhJqbJfhfn4iPQbk7uw9Vs27XQd7ZdYj1uw6xbtchXlhbxiNLtr37O93SYwzPb1y04o9D+nbn7yu3\nH/UeJeWVTJ+1EkBFSpKauvhEItbW1s3+yhrW7zrEO7sOsW7XwXeL17Z977XI0oNWVW0Tdx4uyM3i\nldsuUpehdDiNQYl0URXVtWwoO8z6oMX1i4Xrmz23W3qMgblZDOiVxcBeWRQ0PM8Ntl5Z9O/ZjfS0\nYy86oy5EaQ2NQYl0UTmZ6YwpymVMUS4Ajy0raXKcKzc7nWvOOIEd+6vYsb+K5VvLeWpVFdW19Ued\nZwb5Pbq9W8gKGhW0gblZvFm6nzufXvvugrrqQpT2ogIl0sk1N871gyvGfKCAuDv7KmrYsb+KnQeq\n2L6/ih0Hqti5v4rtB6rYsqeCRRv2cKCq9pjvWVlTx/RZK1m+tZy8nAzysjPIy8kkt9HzvOwMemVn\nHHNyRwO10LomFSiRTq6l1TAaMzP6dM+kT/dMTi7s1exrVlbXseNAFdv3V/JPv1nU9Dk1dTy6dBsH\nWyhmvbLS4wUrJ4PcRsWrYf+dskM8uqSE6rr3Wmi3zVpBbV09V58+qF3G0DqiAKrItp7GoETkuJz3\n42eb7EIFNi6uAAAIT0lEQVRsuNi4tq6eA1W1lFdUU15Zw/6KGsorqymvqKG8oob9lTXv/qzx/v7K\nGpqY23GU9JiRk5lG927pRz9mppPTLZ2cjDRyujXsB4/vO//VDXv4xbPrOVLbeDWPGLdfNa7dCsjs\nZSVNtmLbc8WQsAtge76+JkmISIcI6z/f+nrn4JFaxv9gPs39L/X1j4ygorqOiupaDlfXUXEkeKyu\npeJIHYcbPbZU7N7PgF7ZGXRLj9EtI0a39LT48/TgeUaMzLQY3TI+ePzd5+kxMtNj/HT+mneXumqs\nX49Mfjv1DNJjRkZajPQ0IzN4TI/FyEgz0tNipMfix2PNdIeGXQDb+/VVoESkw4T57b2lFloi3J0j\ntfUcPlIbFLT3itd19zXdRQkw9ZwhHKmtD7Y6qhue18T33/1ZTfx5w88buiPbW8wgPS1GRixeuDLS\n4oVt18Ej1DVRgTPTYowfnEcsBumxeIFLjxkxiz+mNdoajjWc0/hnf160hUNHPthV29YluTSLT0Q6\nzJQJRaGNpzQ3yWPaxNEJv4aZkZWRRlZGGn3f97PmVvMoysvmB1eOaVPm+nqnuu69QvbxX7zEzgNH\nPnBe3+6ZzPjUOGrqnNo6p7Y+XuRq653auvr48fr4Y01dPbV1Tk19/LG2rp6a4Ly/LN7WRAqorqsn\nFoP6eqioraXO49lq6z14rKfeiT/Wxx/r6qHe46/b8LOqmqYLbmkz90prLypQIpLUWjPJoy3aowC+\nXyxmZMXiBREymP6xk5p8j+9OPpmLPjTgeOID8PL6Pc0W2YduOOe4X7+5VmxhXvZxv/axqECJSNIL\ns4UWdgHsiPcIo8h25Os3R2NQIiKdgGbxRUQFSkSk80i0QB17gS0REZGIqECJiEhSUoESEZGkpAIl\nIiJJSQVKRESSkgqUiIgkpZSYZm5mZcDmqHO0oB+wO+oQxynVP4PyRy/VP0Oq54fU+AxD3D2/pZNS\nokClAjNbnMi8/mSW6p9B+aOX6p8h1fND5/gMDdTFJyIiSUkFSkREkpIKVPuZGXWAdpDqn0H5o5fq\nnyHV80Pn+AyAxqBERCRJqQUlIiJJSQVKRESSkgrUcTKzwWa20MxWm9kqM7s56kxtYWZpZrbMzOZE\nnaUtzCzPzB4xs7eDP4vjv41oBzKzW4K/P2+a2YNmlhV1ppaY2e/MbJeZvdnoWB8ze9rM1gWPvaPM\neCzN5J8R/B1aYWaPmVlelBlb0tRnaPSzfzEzN7N+UWRrDypQx68W+Ja7nwScDdxoZidHnKktbgZW\nRx3iONwFPOXuHwJOJYU+i5kVATcBxe4+BkgDPhNtqoT8HrjsfcduAxa4+yhgQbCfrH7PB/M/DYxx\n93HAWmB6R4dqpd/zwc+AmQ0GLgW2dHSg9qQCdZzcfbu7Lw2eHyT+H2M496YOiZkNAiYBv406S1uY\nWS/gAuA+AHevdvfyaFO1WjqQbWbpQA5QGnGeFrn7C8De9x2+Erg/eH4/MKVDQ7VCU/ndfb671wa7\nrwKDOjxYKzTzZwBwJ/BtIKVnwalAtSMzGwpMABZFm6TVfk78L3N91EHaaDhQBvxv0E35WzPrHnWo\nRLl7CfBT4t92twP73X1+tKnabIC7b4f4lzegf8R5jscXgSejDtFaZnYFUOLub0Sd5XipQLUTM+sB\nPAp8090PRJ0nUWY2Gdjl7kuiznIc0oHTgHvdfQJwmOTuWjpKME5zJTAMKAS6m9l10abq2szs34h3\n3z8QdZbWMLMc4N+A70WdpT2oQLUDM8sgXpwecPdZUedppfOAK8xsE/AQcJGZ/SnaSK22Ddjm7g0t\n10eIF6xUcQmw0d3L3L0GmAWcG3GmttppZgUAweOuiPO0mplNBSYD13rqXSg6gvgXnTeCf9ODgKVm\nNjDSVG2kAnWczMyIj32sdvc7os7TWu4+3d0HuftQ4gPzz7p7Sn17d/cdwFYzGx0cuhh4K8JIrbUF\nONvMcoK/TxeTQpM83ucJYGrwfCrweIRZWs3MLgO+A1zh7hVR52ktd1/p7v3dfWjwb3obcFrwbyTl\nqEAdv/OA64m3PJYH2+VRh+qC/hl4wMxWAOOB/4o4T8KClt8jwFJgJfF/l0m/XI2ZPQj8AxhtZtvM\n7EvAj4FLzWwd8VlkP44y47E0k/8XQE/g6eDf8q8iDdmCZj5Dp6GljkREJCmpBSUiIklJBUpERJKS\nCpSIiCQlFSgREUlKKlAiIpKUVKBE2omZ1TW61GC5mbXbahZmNrSpFatFOrP0qAOIdCKV7j4+6hAi\nnYVaUCIhM7NNZvYTM3st2EYGx4eY2YLg3kMLzOyE4PiA4F5EbwRbw7JHaWb2m+C+UfPNLDs4/yYz\neyt4nYci+pgi7U4FSqT9ZL+vi++aRj874O5nEl+p4OfBsV8AfwjuPfQAcHdw/G7geXc/lfiagquC\n46OAe9z9FKAcuDo4fhswIXidr4b14UQ6mlaSEGknZnbI3Xs0cXwTcJG7bwgWFt7h7n3NbDdQ4O41\nwfHt7t7PzMqAQe5+pNFrDAWeDm4EiJl9B8hw9/8ws6eAQ8BsYLa7Hwr5o4p0CLWgRDqGN/O8uXOa\ncqTR8zreG0OeBNwDnA4sCW56KJLyVKBEOsY1jR7/ETx/hfdu7X4t8FLwfAHwNQAzSwvuGNwkM4sB\ng919IfGbTuYBH2jFiaQifdMSaT/ZZra80f5T7t4w1bybmS0i/qXws8Gxm4Dfmdk04ncE/kJw/GZg\nZrAydR3xYrW9mfdMA/5kZrmAAXem4O3uRZqkMSiRkAVjUMXuvjvqLCKpRF18IiKSlNSCEhGRpKQW\nlIiIJCUVKBERSUoqUCIikpRUoEREJCmpQImISFL6//RX9JmEIDCkAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ada = AdalineGD(n_iter=15, eta=0.01)\n", + "ada.fit(X_std, y)\n", + "\n", + "plot_decision_regions(X_std, y, classifier=ada)\n", + "plt.title('Adaline - Gradient Descent')\n", + "plt.xlabel('sepal length [standardized]')\n", + "plt.ylabel('petal length [standardized]')\n", + "plt.legend(loc='upper left')\n", + "plt.tight_layout()\n", + "# plt.savefig('./adaline_2.png', dpi=300)\n", + "plt.show()\n", + "\n", + "plt.plot(range(1, len(ada.cost_) + 1), ada.cost_, marker='o')\n", + "plt.xlabel('Epochs')\n", + "plt.ylabel('Sum-squared-error')\n", + "\n", + "plt.tight_layout()\n", + "# plt.savefig('./adaline_3.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "### Large scale machine learning and stochastic gradient descent" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "from numpy.random import seed\n", + "\n", + "class AdalineSGD(object):\n", + " \"\"\"ADAptive LInear NEuron classifier.\n", + "\n", + " Parameters\n", + " ------------\n", + " eta : float\n", + " Learning rate (between 0.0 and 1.0)\n", + " n_iter : int\n", + " Passes over the training dataset.\n", + "\n", + " Attributes\n", + " -----------\n", + " w_ : 1d-array\n", + " Weights after fitting.\n", + " errors_ : list\n", + " Number of misclassifications in every epoch.\n", + " shuffle : bool (default: True)\n", + " Shuffles training data every epoch if True to prevent cycles.\n", + " random_state : int (default: None)\n", + " Set random state for shuffling and initializing the weights.\n", + " \n", + " \"\"\"\n", + " def __init__(self, eta=0.01, n_iter=10, shuffle=True, random_state=None):\n", + " self.eta = eta\n", + " self.n_iter = n_iter\n", + " self.w_initialized = False\n", + " self.shuffle = shuffle\n", + " if random_state:\n", + " seed(random_state)\n", + " \n", + " def fit(self, X, y):\n", + " \"\"\" Fit training data.\n", + "\n", + " Parameters\n", + " ----------\n", + " X : {array-like}, shape = [n_samples, n_features]\n", + " Training vectors, where n_samples is the number of samples and\n", + " n_features is the number of features.\n", + " y : array-like, shape = [n_samples]\n", + " Target values.\n", + "\n", + " Returns\n", + " -------\n", + " self : object\n", + "\n", + " \"\"\"\n", + " self._initialize_weights(X.shape[1])\n", + " self.cost_ = []\n", + " for i in range(self.n_iter):\n", + " if self.shuffle:\n", + " X, y = self._shuffle(X, y)\n", + " cost = []\n", + " for xi, target in zip(X, y):\n", + " cost.append(self._update_weights(xi, target))\n", + " avg_cost = sum(cost)/len(y)\n", + " self.cost_.append(avg_cost)\n", + " return self\n", + "\n", + " def partial_fit(self, X, y):\n", + " \"\"\"Fit training data without reinitializing the weights\"\"\"\n", + " if not self.w_initialized:\n", + " self._initialize_weights(X.shape[1])\n", + " if y.ravel().shape[0] > 1:\n", + " for xi, target in zip(X, y):\n", + " self._update_weights(xi, target)\n", + " else:\n", + " self._update_weights(X, y)\n", + " return self\n", + "\n", + " def _shuffle(self, X, y):\n", + " \"\"\"Shuffle training data\"\"\"\n", + " r = np.random.permutation(len(y))\n", + " return X[r], y[r]\n", + " \n", + " def _initialize_weights(self, m):\n", + " \"\"\"Initialize weights to zeros\"\"\"\n", + " self.w_ = np.zeros(1 + m)\n", + " self.w_initialized = True\n", + " \n", + " def _update_weights(self, xi, target):\n", + " \"\"\"Apply Adaline learning rule to update the weights\"\"\"\n", + " output = self.net_input(xi)\n", + " error = (target - output)\n", + " self.w_[1:] += self.eta * xi.dot(error)\n", + " self.w_[0] += self.eta * error\n", + " cost = 0.5 * error**2\n", + " return cost\n", + " \n", + " def net_input(self, X):\n", + " \"\"\"Calculate net input\"\"\"\n", + " return np.dot(X, self.w_[1:]) + self.w_[0]\n", + "\n", + " def activation(self, X):\n", + " \"\"\"Compute linear activation\"\"\"\n", + " return self.net_input(X)\n", + "\n", + " def predict(self, X):\n", + " \"\"\"Return class label after unit step\"\"\"\n", + " return np.where(self.activation(X) >= 0.0, 1, -1)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8HFWZ//HPNwkSMIFASEDIBgwQERUlbCJh1UHGDZQR\nfjNiBCeijrujAvMaYXSIu+MKMkYZhBF1EIdBkE0ggAohCAICgWEJQWQPScRAluf3x6kmnZvb3efm\nVndX3/t9v179ul1LVz1VN+nnnqdOnVJEYGZmVjUjuh2AmZlZf5ygzMyskpygzMyskpygzMyskpyg\nzMyskpygzMyskpyghjFJsyRdl7nuKZLOKd5PkbRc0sj2Rtg9ks6S9LkO7OfvJF3W7v1sqPrzIGl/\nSXd3OyYbPpyghiBJV0t6WtLG7dh+RCyKiDERsbod229G0iRJ50t6QtIzkm6TNKtYNk1SSBrV6bhy\n9BdfRJwbEa/fwO3NkHRR8bteIukPkv5N0hblRb1WRFwbEbuUsS1JD0g6tMnyAyWtKf4QWi5psaSf\nSNqzjP23Q/H/7j3djmMocYIaYiRNA/YHAnhzV4Npjx8CDwFTgfHAscCjXY2oCyS9BrgauB6YHhHj\ngMOAVcArG3ymkom7iT9GxBhgLLAPcBdwraRDuhuWdYoT1NBzLPBb4CzgXfULJI2XdKGkpZJuBHbs\ns/zrkh4qli+QtH9/O+jbEij+cvyspOslLZN0maSt6tbfR9Kvi7/yb5V04CCOb0/grIj4c0Ssiojf\nRcQlxbJ5xc8lxV/d+0oaIemfJT0o6TFJZ0vavC6219bF9lCtNVbYQtIvimO6QdKOdZ9reK4k7SXp\npmLZo5K+2iS+dcqskl4m6XJJTxWfPanBefgi8IOImBMRj8ILLdvPRMTVxbZmFb+Tr0l6CjhF0o6S\nfiXpyaIVeq6kcXX7f5Wkm4tj/jEwum7ZgZIW101vW7RmH5d0v6QP1S07pWjxnF1s6w5JM4plPwSm\nAP9bnIdPNjhGiuOKiFgcEf8CfA/4Qt1+ptedr7sl/W3dssOLVuUySQ9L+kTdsrdIuqX4Hf2fpMOK\n+ZtLmivpkeIzn1NRyq79riR9WanVer+kNxTL/o30h+G3imP6VrNjskwR4dcQegH3Au8H9gBWAlvX\nLTsP+AnwYmA34GHgurrlf09qlYwCPg78CRhdLDsFOKd4P43UQhtVTF8N/B+wM7BJMf35Ytl2wJPA\n4aQ/iF5XTE/YwOO7gtRqOBqY0mfZOnEV844rzskOwBjgZ8APi2VTgGXAMcBGxbHvXiw7C3gK2Ks4\nH+cC52Weq98A7yzejwH2aRLfrNrvgNRSeKTY3uhieu9+zsGLgdXAgS3O1SxSi+qDRZybAH9V/A42\nBiaQkua/F+u/CHgQ+GhxPt5e/Bv6XLH8QGBx8X4EsAD4l+JzOwD3AX9d9+9lRfF7HwnMAX5bF9sD\nwKFNYn9hX33mHwysKc7Bi0mt6XcXx/dq4AngZcW6jwD7F++3AF5dvN8LeKY4DyNI/0anF8t+Dny3\n2PZE4EbgvXXncyXwD8UxvQ/4I6C6/wfv6fZ3wFB6dT0Av0r8ZcJri/9AWxXTdwEfLd6PLJZNr1v/\nNOoSVD/bexp4ZfH+FJonqH+u+9z7gV8W7z9FkRDqll8KvGsDj3EL4PPAHaQv6VuAPfuLq5h3JfD+\nuuldivMwCjgRuKDBfs4Cvlc3fThwV+a5mgecWvs91K3TX3yzWJugjgF+l3EOJhXbqf9dfhFYAvy5\n9rsotr2oxbbeWtsnMLP+C7eY92v6T1B79912cT5/UPfv5Yq6ZbsCf6mbfoANS1DTi2PfDngHcG2f\n5d8FPlO8XwS8F9isn3W+1s+2twaeAzapm3cMcFXd+by3btmmRSzb1P0/cIIq8eUS39DyLuCyiHii\nmP4v1pb5JpC+lB+qW//B+g9L+rikO5U6HywBNge2Is+f6t4/S2o5QLpWdFRRQltSbPe1wEv6bkCp\nR1vtovglfZcDRMTTEfHpiHgZ6QvlFuDnktQgrm37HOeDpPOwNTCZ1PIb6DG1OlfHk1qTd0maL+mN\nTfZRr1U8NU+TWhEvnMOI+GSk61AXkI6vpv73jaSJks4ryldLgXPq4t4WeDiKb9vCOv9G6kwFtu3z\nez2JdF5r+p6/0Rr8dbDtSElhSRHD3n1i+Dtgm2Ldt5H+sHhQ0jWS9i3mNzrPU0ktx0fqtvddUktq\nvWOKiGeLt2Owtui1i6bWgKRNgL8FRkqq/SfaGBgn6ZXA7aRyz2RSywpSiav2+f1JrZ1DgDsiYo2k\np4FGX/y5HiK1oP6h1YoRcS6plJYlIp6Q9GVSEt6S9MXV1x9JXzw1U0jn4dEitr1y91fT6lxFxD3A\nMZJGAEcC/y1pfIP46j1E+ou9qYj4s6Qbim1f1Wr1PtNzinmviIgnJb0VqF0veQTYTpLqktQU+v8y\nfwi4PyJ2ahVvZly5jgBuLs7BQ8A1EfG6fncQMR94i6SNgH8klbcnk2LfsZ+PPERqQW0VEas2IDY/\nGqJkbkENHW8llbx2BXYvXi8FrgWOjdQl/GekC+WbStqVdTtRjCV9cT8OjJL0L8BmJcR1DvAmSX8t\naaSk0cXF9kkbsjFJX5C0m6RRksaSrgPcGxFPFrGvIV0PqfkR8FFJ20saQypr/rj4AjoXOFTS3xbb\nGy9p94wwmp4rSX8vaUJErCH9pQ/pd9NffPUuAraR9BFJG0saK2nvBut+EjhO0qclTSz2OwnYPiP2\n5aSOGtsB/1S37DfFcX2oOB9H0jiB3wgslfQpSZsUv9vdlN8N/FEan4d1KNlO0meA95BaapDO186S\n3ilpo+K1p6SXSnpR0SLfPCJWAktJvwOAucC7JR2i1IlmO0nTI+IR4DLgK5I2K5btKOmAso/J8jhB\nDR3vItX/F0XEn2ov0l/Hf1eUVv6RVI74E+kayw/qPn8pcAmwkFTWWUGf8tCGiIiHgLeQvlQeL7b5\nT2z4v71NSWWsJaSL8lMputMXJZd/A64vSjT7AN8ndU2fB9xPOq4PFusvIpWAPk7qEHELDbpo99Hq\nXB0G3CFpOfB14OiIWNEgvhdExDLShfs3kX5H9wAH9RdARFxH6jAwE1hYlKN+SboO8s0msZ9K6kzw\nDPAL0h8ttW0+T2qVzSKVEd9Rv7zP/lcXce5OOq9PkHrYbd7f+v2YA/xzcR4+0WCdbYtzuByYD7yc\n1DHksiKGZcDrSR1m/kg6Z18gVQ4A3gk8UJQyTyB1bCEibiR1rPhacR6uYW0r+1hSp48/FOfgv+mn\nHN3A14G3K/Xw+0bmZ6wJRbhVamZm1eMWlJmZVZITlJmZVZITlJmZVZITlJmZVVJP3Qc1ZsxWMX78\ntG6HYWZmg7Bo0YInImJCq/V6KkGNHz+Nk0++qdthmJnZILz3vWo0Qsk6XOIzM7NKcoIyM7NKcoIy\nM7NK6qlrUP0ZNWolO+ywmE03XdHtULI9++xo7rtvEqtWbdTtUMzMKqvnE9QOOyxm8uSxjB07jcZP\nXKiOiGDZsieBxSxc2GpcTzOz4avnS3ybbrqCsWPH90RyApDE2LHje6rFZ2bWDT2foICeSU41vRav\nmVk3DIkEZWZmQ48TVInuuecu3vCGfZk0aWO+/e0vdzscM7Oe1vOdJKpk3LgtOe20b3DJJT/vdihm\nZj1vWCWoca/fkxFPPLbe/DVbTWTJZfMHvf0JEyYyYcJELr/8F4PelpnZcDesEtSIJx5jzfj1xyfs\nL2mZmVl3+RqUmZlVkhPUIM2d+20OOmh3Djpod/70pz92OxwzsyFjWJX42uH44z/A8cd/oNthmJkN\nOU5QJXr00T/x+tfPYNmypYwYMYIzz/x3rrvuD4wdu1m3QzMz6znDKkGt2Wpiw158Zdh662249dbF\npWzLzGy4G1YJqoyu5GZm1hnuJGFmZpXkBGVmZpXUtQQlabKkqyTdKekOSR/uVixmZlY93bwGtQr4\neETcLGkssEDS5RHxhy7GZGZmFdG1FlREPBIRNxfvlwF3Att1Kx4zM6uWSlyDkjQNeBVwQz/LZku6\nSdJNy5c/3unQsnz4w8ex664TmTlzt26HYmY2ZHQ9QUkaA5wPfCQilvZdHhFnRsSMiJgxZsz6A70O\nVETz6Q1x9NGzOO+8Xw5+Q2Zm9oKuJihJG5GS07kR8bN27+/ss+H009cmpYg0ffbZg9vuvvvOZNy4\nLQcfoJmZvaCbvfgEzAXujIivtnt/EbB8OVxwwdokdfrpaXr58nJaUmZmVp5u9uLbD3gncJukW4p5\nJ0XExe3YmQTve196f8EF6QVwxBFpvtSOvZqZ2YbqWoKKiOuAjqaFWpKqJSdwcjIzq6qud5LopFpZ\nr179NSkzM6uOYZOg6q85HXEEXHZZ+ll/TWpDvfe9x3D44fty771388pXTuLcc+eWF7iZ2TA1bEYz\nl2DMmHWvOdWuSY0ZM7gy33e/+6NygjQzsxcMmwQFcOyxqaVUS0a1JOVrUGZm1TNsSnw1fZORk5OZ\nWTUNiQQVPdbLodfiNTPrhp5PUM8+O5ply57smS/9iGDZsid59tnR3Q7FzKzSev4a1H33TQIWs+mm\n1RxItj/PPju6iNvMzBrp+QS1atVGLFy4fbfDMDOzkvV8ic/MzIYmJygzM6skJygzA9rzrDSzwXCC\nMjPmzYMrrlj3WWlXXJHmm3VLw04Ski7M+PxTETGrvHDMrNMi4Lnn4MYb0/Shh6bkdOONsNde646+\nYtZJzXrxvRR4T5PlAr5dbjhm1mlSSkqQklItUe21V5o/3JJT34TsBN09zRLUyRFxTbMPSzq15HjM\nrAtqSaqWnGB4Jqd581JrsnbstVLnxhvDzJndjm74aXgNKiJ+0urDOeuYWfXVvojr1V+TGg7qS521\nY6+VOp97bnidi6podg3qf4GGv5KIeHNbIjKzjqr/Iq6V9WrT0N6WVJXKae0odVbp+HpRsxLfl4uf\nRwLbAOcU08cAD7QxJjPrICmVsOq/iGtf1Btv3L4v1CqW08osdVbx+HpNwwRVu/4k6bMRUX86/1eS\nO5+aDSEzZ67/rLR2t5yq2HOwUalzoOeiqsfXa3LG4psgaYeIuA9A0vbAhPaGZWZlyik1dfJZaVXs\nOTiQUmer81nF4+tFOTfqfhS4WtLVkq4GrgI+0taozKw0Vb0Jt/5LvKabX96NSp177bVuqTP3fFbt\n+HpRyxZURPxS0k7A9GLWXRHxXHvDMrMyVLnUVFY5rUytSp0DOZ9VPL5e0zJBSdoU+BgwNSL+QdJO\nknaJiIvaH56ZDUY7Sk1r1sCIEY2nc3Sz5+Bg5J7PMsuFw1nOP6sfAM8D+xbTi4HPtS0iMytVmaWm\nuXPh9NNTUoL08/TT0/yBxrRoEYwbB4cckqYPOSRNL1rUvS/onPJdzvksu1w4XOUkqB0j4ovASoCI\n+AtpmCMz6wFl3YS7Zg2sWAF33bU2SZ1+eppesWJt0sqNacoUePppuPLKNH3llWl6ypTu3BSbe6Nu\n7vmcOXPdxFVLUrUu5r4xuLWcXnzPS9qE4qZdSTsCvgZl1gPKLKWNGAHvex985zspKX34w2n+Lruk\n+QMp8zUqle2994a17nLLZM3WyynfDfR8NusZ6Z5+reX8kzoF+CUwWdK5wJXAJ9sZlJmVI7fUlOu6\n62DnnddtTey8c5q/IbGVUXrMLZOVUb4r+3y6p19zLRNURFxGGk1iFvAjYEZEXN3esMysLK1KTbki\nUinv4otTOS8i/bz44jR/oCWpMkqPAynLlVW+K+t85u5vOMvpxXcl8JWI+EXdvDMjYnZbIzOrmDJ7\nW+Vsq8z95dyE26p3XgQsXAjPPw8vehFMmACPP56mFy4cWM+0snq55ZbJyi7flXFTc6/2ZOyknBLf\n9sCnJH2mbt6MNsVjVkll9rbK2Vane3fl9M4bMQI22QR23TUlJ0g/d901za8ls9xSWlm93HLLZJ0u\n37XS6f31opwEtQQ4BNha0v9K2rzNMZlVSpm9rXK21eneXQPpnXfccemaU72dd07zc4+vpqxebrll\nsk6X73J0en+9JqcXnyJiFfB+SbOA64At2hqVWYWU2dsqd1ud7N1V651XS0q13nnTp6/bO6/2BT9/\n/volqfoRFwYS+2B7ueWWyTpdvhuITu+vl+S0oM6ovYmIs0idJS5rUzxmlTSQ3lb9/eU+0G11undX\nLUnV69t1PLck1clzNZCYNt4Y9txz3fX23HP9clqrmKxzGiYoSZsVb38qacvaC7gf+ERHojOriNwy\nUs41k5xtdbp3V62sV6/+mlRNTkmq0+eqzDKZR3aolmYtqP8qfi4Abip+LqibNhsW+paHTjop/ay/\nNlJbL+f6Uqtt5e6vLPXXnKZPh69/Pf2svyZVr1lJqtPnKiem+v3Nn7/u/ubP7961P2ut4TWoiHhj\n8XP7du1c0veBNwKPRcRu7dqP2WA0KiNB49JWs+svOdvq5BNuR4yA0aPXveZUuyY1evTAR4jor5QW\nseHnqtW2cuOq2rU/a61hgpL06mYfjIibS9j/WcC3gLNL2JZZ2+Q+cbY2v9kjw3O21ekn3B5//Lr3\nPdWS1EBHKR+InHPV6f11OiZrrtk/v68Ur28DNwBnAv9RvP9GGTuPiHnAU2Vsy6zdcnpb5V5/ydlW\np3t39U1GG5Kcckpp9es2O1cD2VZubFW79mfNNSvxHQQg6TxgdkTcVkzvhjtJmK2nrNERasp47lKj\n7Q9mVIpmcktpueeqrJJbzv7AIztUTc59UNNryQkgIm6XtHsbY1qHpNnAbIAtt5zSqd2aDVjutap5\n81ILoO89OhtvvLbn2dy56SbZWpmt1pFh9OhUjhuInP2VKbeUNpDreoMtueXur5PX/qy1nAR1l6Tv\nAecAAfw9cGdbo6oTEWeSyotMnTrDDW2rtDIeGR6x7sgO9TfRTp8+sJZUzv7K/uJtVCbbkOtsudvK\nUcVrf9ZcToKaBbwPKO4vZx5wesO1zYa5nNERItYtW9X3VJMG9tylMgZTLctAB0AdSJf1MkpuVbz2\nZ401/TtM0kjgexHxtYg4onh9LSJWlLFzST8CfgPsImmxpAEWL8x6z7XXtp6f+9ylMgdTLUPuyA6d\n3pb1pqYtqIhYLWmCpBdFxPNl7zwijil7m2ZVVivfXXll+oIdOxaWLUvThxyyNtHUP3epdg3q4ovh\n8MPXtpByy3dllslylFkmc8lteMsp8T0AXC/pQuDPtZkR8dV2BWXWy3J639USR+3Vt7SV89ylnHJh\n2T0Lc9crs0zmktvwlXOp9Y/ARcW6Y+teZtZHq+cqSakn3sEHp9ZTrRV18MFpvpT/3CVYWxasL/HV\nz88tk5X52HSzsrRsQUXEqZ0IxKzX9X2uUqPed/vvD5dfvm7ZCtL8muOOS+vMn7923s47w+tet3a6\nVi781a/S+1q58Fe/Sgmv1rIpo2fhQEqKZmVpmaAkTQA+CbwMGF2bHxEHtzEus9J06ibVnOcq1Y+G\n0OiZSpD33KWaWuKrlf36Du4KeT0LYfCPTTcrU06J71zgLmB74FTSNan5zT5gVhWdLkm1eq6SBIsW\nwbhxqVOElH6OG5fm15JM7jOOHnoIpkxJrSdIP6dMSfMH2mMup6dfJ3sEmuUkqPERMRdYGRHXRMRx\nwD5tjsts0Lrx+IRWz1WKSAnk6adTz72I9PPpp9P8Wky5z12aPDklo2XL0rxly9L05MkDO77cMeg8\nVp11Uk4vvpXFz0ck/Q2p08Sk9oVkVo5Ol6RqyenOO+GlL11b7rvzzrXXpEaM6D+mvfce2E2sfec3\n6hGYI7enXztunDVrJqcF9TlJmwMfJw0S+z3go22NyqwknSxJjRgBS5fCVlvBCSek6RNOSNNLl65b\n5isjppwegbnbGchj033jrHVKTi++i4q3zwAHtTccs3J18ibVCJgxA264IfWkO/TQ9HPUqDS/HTfO\n5vQIzJF7Q6xvnLVOavbAwm+SBoftV0R8qC0RmZWk0yWpRiXF+vJdmTHl9AgcaEeJZtMDXc9ssJq1\noG4qfu4H7Ar8uJg+CljQzqDMytCoJAXtK0nVeuXVPx6i1luv7Ji6cXxmnaRo0f1G0lXA6yNiZTG9\nEXBZ7YGGnTR16ow4+eSbWq9oVqdT90FB6r5+222pV16txbTFFvDyl6/fA6+smDp5fGZleO97tSAi\nZrRaL6eTxLasO7TRmGKeWU/oVElqzZqUnO66KyWlE09MP++6K82vv4HWY9WZtZbTzfzzwO+KlhTA\nAcApbYvIrEeNGAG77ZbeL1kCc+ak99Onp/n1Y+i51WPWWk4vvh9IugTYu5j16Yj4U3vDMutNBxyQ\netDVkhOs/5DBTj+C3axXZT44mpHA48DTwM6S/N/IrB+1kSHq1UaMqC3v9OgWZr0qZ7DYLwDvAO4A\nalX0ID363awterEEltuFPHd0i148B2ZlyrkG9VZgl4h4rt3BmEHvlsByu33X5td3Re+bnHr1HJiV\nKafEdx+wUbsDMYPeL4HlDvLabMDVXj8HZmXJaUE9C9wi6UrghVaUR5KwdhgKzxxq1u277DKgWa+p\nPV06R06CurB4mXVETgmsV5VZBjSrmjknLW290pIl2dvL6Wb+n9lbMytBJwd47YacAVeH+jmw3jLn\npKWw4i/NV1rxHBNHPskN2x7ZfL0dJ6Ibm69Sk9OLbydgDmk8vvpHvu+QtwuzdTXrnVb7Yr7hhrWD\nrNamYeh8QZdRBjQrw7x5cP3PH4WVqxqvtHo19+9xVOuNHX88cGLr9W7My1A5Jb4fAJ8BvkZ63Ma7\nAf/3sA3Sqnda7ZHoW2yx7iPRFy5c+0j0oc6DwFpZ5nysReIBWL2a/UYv4Jyjzm+8zsyZwPGlxpYj\nJ0FtEhFXSlJEPAicIulaUtIyy1bfOw3WbRnstdfa3mlTpqR5V16Z1rnyylS2rq0zHL6g/dwla2bO\nHOChRVnr3n/0Sa1XmjkTqN79CzkJaoWkEcA9kv4ReBiY2N6wbCjK7Z3mHmyJB4EdvuZ88GFYvbrp\nOvdPPiCNSNxS9RJPrpwE9RFgU+BDwGdJZb5j2xmUDV05vdPcg8161bwW4+tc/8ulWb3YJo58khu+\nM7/FWjnJqbflJKhpETEfWE66/oSko4Ab2hmYDU05vdPcg82qqNX9O48tWMTEkU82XWcicMMeJxSd\nCayVnAR1IvDTjHlmTeX0TgP3YLPOa5l87ngUVjzHfqMbP0z8+HHzmX3atIy9OTnlapigJL0BOBzY\nTtI36hZtBrToFmK2vtzeae7BZmWpf+xJQw8tYr/RC5g5uklJbTTMfuuiFgMhThtgdNZKsxbUH4Gb\ngDcD9X82LAM+2s6gbOjK6Z3mHmzWypw5wLIWoxYU13pOHHdG09VmT/5R0dlgWou9tlpuZWuYoCLi\nVuBWSf8VESsBJG0BTI6IpzsVoA09Ob3T3INteJvzsUebr7DiudSLrZmxZCaeod/ZoFflXIO6XNKb\ni3VvAR6XdE1EfKy9oZk15+cl9Z45Jy2FZcuar7R6derFtvsJjdfZaSeY6cQy1OUkqM0jYqmk9wA/\niIjPSPp9uwMza8bPS6qWefPg+vMybxzd46iUYJrp0sgFVi05CWqUpJcAfwuc3OZ4zFrKGZHCLaly\nzflY6sXWzInjznAvNitVToL6V+BS4LqImC9pB+Ce9oZl1pifl1SeuXPT/Ts57j/6JPdis47KedzG\nT6m75yki7gPe1s6gzFrxaBMZoxbkltyGwZA51pua3Qc1OyLObPbhnHVafP4w4OvASOB7EfH5Dd2W\nDS9DfbSJljeOPgY81Hzkgv1GP8A5X308Y2/ubGDV1KwF9WlJTzRZLuDDwAYlKEkjgW8DrwMWA/Ml\nXRgRf9iQ7dnw0cvPS8q9cXTiyCfZaaMHGq6yE3DOHp/3kDk2pDVLUNcAb2rx+csHse+9gHuLkiGS\nzgPeAjhBWVNVfF7SvHnFQKDNLFsGq1e3vHGUcbizgRnNb9R9d5v3vR3wUN30YmDvvitJmg3MBthy\nyyltDsl6RadHm5hzUutRC04cdwazx/6o8TrZN46aGeT14muX/r5KYr0Z6RrXmQBTp85Yb7kNX2WM\nNjF3bjEQaDNF9+qmIxfsODH/cddmlqWbCWoxMLluehJp/D+zUsz54MOtV1q9OiWeiS2ewenkY9Zx\n3UxQ84GdJG1Pekrv0cD/62I81kNyHnm93+gFnPPW81tvzEPmmFVSywQlaWPSfU/T6tePiH8dzI4j\nYlXxCPlLSd3Mvx8Rdwxmm9b7BjxkTstebL53x6xX5bSg/gd4hvTIjeZjnQxQRFwMXFzmNq265nzw\nYVi9uuV6HjLHzCAvQU2KiMPaHolVWqtRC+65p/UjrycCN3ynyUPhXjBtIKGZ2RCVk6B+LenlEXFb\n26Oxjms1YgGsHautWfIBuH/ykZlD5piZtdZsqKPbSN2+RwHvlnQfqcQnICLiFZ0J0TZUy1ELli2F\nJUvYb/SCpqvtNBoPmWNmHdesBfXGjkVhA9bykddLljBx5JMcP/anjdcBZh+9yA9QMrNKajaSxIMA\nkn4YEe+sXybph8A7+/2gDcq8eXD9z1vcOAqtH3mdPWpBq+VmZt2Rcw3qZfUTxSCve7QnnKFtzsfy\nEs9+oxdwzstaDOzuG0fNbIhrdg3qROAkYBNJtVqSgOfZwBHMh7K5c+GxW5qMXFB0r75/j6Oab2in\nnfy4azMzmpf45gBzJM2JiGH9p/qc9+fdOHriuDOYfViTdZ14zMyy5ZT4firp1X3mPQM8GBGr2hBT\nx8w5KfVia2XiyCe54Zu59+9MG2RUZmYGeQnqO8Crgd+TSnwvB24Fxks6ISIua2N8DbW6cRTyhsy5\n/+iT3IvNzKyCchLUA8DxtXHyJO0K/BPwWeBnQMcS1BNP1D0eYcVzLW8czXvktZOTmVkV5SSo6fWD\nuEbEHyS9KiLuU4cfXTp6xRJ2uuOC9Ljr7+TcOGpmZr0qJ0HdLel04Lxi+h3AwmKU85Vti6wf209a\nyTknOzGZmQ0HIzLWmQXcC3wE+ChwXzFvJXBQuwIzM7PhrWULKiL+AnylePW1vPSIzMzMyHtg4X7A\nKcBU1n1v6H1aAAAPD0lEQVRg4Q7tC8vMzIa7nGtQc0mlvQVA66fNmZmZlSAnQT0TEZe0PRIzM7M6\nOQnqKklfIt3z9MIj3yPi5rZFZWZmw15Ogtq7+Dmjbl4AB5cfjpmZWZLTi89dyc3MrONyevFtDZwG\nbBsRbyiGOto3Iua2PTrrHaedBsuWrT9/7Fg46aTe35+ZdVzOjbpnAZcC2xbTC0k37ZqttWwZjBmz\n/qu/JNKL+zOzjstJUFtFxE+ANQDFIzbc3dzMzNoqJ0H9WdJ4UscIJO1Deh6UmZlZ2+T04vsYcCGw\no6TrgQnA29salZmZDXs5vfhulnQAsAvpgYV3R0RHRzE3M7Php2GCknRkg0U7SyIiftammKwXjR3b\nuFddOyxfDkuWrD9/VE5RYAN88IOwalX/+/vmN8vfn3spmjVtQb2pybIgjSxhlnT6S7PWa6+v5W0a\nYH/Vqv6TX39Jqwy1Xor9zTcbJhomqIh4dycDMTMzq5fTi8/MzKzj2lSwtyEl53pImddoTjgBItaf\nL8EZZ6T3zzyTXmY2ZDlBWWs510PKvEYTkZJRf/Pr33fympCZddyG9OIDcC8+6y4JVvczoEl/ia0M\no0Y1biG2Q6d7RZpVkHvxDXed7s6cU76D/tep119y6k9O6dFdus0qyb34hrtOd2fOKd+VKaf02OkS\nZg53MzfLuwYl6W+AlwGja/Mi4l83dKeSjgJOAV4K7BURN23otszMbGjKeR7UGcCmwEHA90jj8N04\nyP3eDhwJfHeQ2xmeyixJPfVUejWTcz1kzRp4/vm8fbartdSfgcRVBpcLzUqT04J6TUS8QtLvI+JU\nSV9hkNefIuJOALXrgvZQV3b5p1XJLeeLdcSIcktgZZYBO1lSdGnOrDQ5N+r+pfj5rKRtgZXA9u0L\naV2SZku6SdJNj7drGBszM6ucnBbURZLGAV8Cbib14Pteqw9JugLYpp9FJ0fE/+QGGBFnAmcCzJg6\ntYO1oQrLKcsNpNTUqjXRjbJVmS2cVtvKKWFG9F8q7Ns6K+sGYnczN8tKUF+MiOeA8yVdROoosaLV\nhyLi0MEGZ020KlvllppySnNllq223DJvkNdWx5dbUszZX06S3WKLvLjLuoHY16vMskp8v6m9iYjn\nIuKZ+nlmZmbt0GwkiW2A7YBNJL2K9LBCgM1Ivfo2mKQjgG+Sns77C0m3RMRfD2abw05ZJbCyRkjI\n3U5u6arV8ZW9v7J0eoQL9xq0IazZt9BfA7OAScBX6+YvBQb1Lz8iLgAuGMw2hrUye8yV9bC93O2U\n1SOwzP2VafPNO/ucKvcatCGs2UgS/wn8p6S3RcT5HYzJzMwsq5PE9ZLmAttGxBsk7QrsGxFz2xzb\n8JRTsskpb5VZ2hrqZbIcuefAve/MSpOToH5QvE4uphcCPwacoNohp2STU94qs7Q11MtkOXLPga/7\nmJUmpxffVhHxE2ANQESsAjKHkjYzM9swOS2oP0saT7pBF0n7AH6U6VBRxV5gLpPl87myISwnQX0M\nuBDYUdL1pK7hb29rVNY5VewF5jJZPp8rG8JaJqiIuFnSAcAupHuh7o6IlW2PzMzMhrWcx22MBt4P\nvJZU5rtW0hkR0XK4I9sALtmYmQF5Jb6zgWWkkR8AjgF+CBzVrqCGNZdszMyAvAS1S0S8sm76Kkm3\ntisgMzMzyEtQv5O0T0T8FkDS3sD17Q3LOsYlRTOrqJwEtTdwrKRFxfQU4E5JtwEREa9oW3TWfi4p\nmllF5SSow9oehZmZWR853cwf7EQgZmZm9XKGOjIzM+s4JygzM6skJygzM6skJygzM6skJygzM6sk\nJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygz\nM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6skJygzM6ukriQoSV+SdJek30u6\nQNK4bsRhZmbV1a0W1OXAbhHxCmAhcGKX4jAzs4rqSoKKiMsiYlUx+VtgUjfiMDOz6qrCNajjgEu6\nHYSZmVXLqHZtWNIVwDb9LDo5Iv6nWOdkYBVwbpPtzAZmA0zZcss2RGpmZlXUtgQVEYc2Wy7pXcAb\ngUMiIpps50zgTIAZU6c2XM/MzIaWtiWoZiQdBnwKOCAinu1GDGZmVm3dugb1LWAscLmkWySd0aU4\nzMysorrSgoqIv+rGfs3MrHdUoRefmZnZepygzMyskpygzMyskpygzMyskpygzMyskpygzMyskpyg\nzMyskpygzMysktRkGLzKkfQ48GC34yjBVsAT3Q6iQ4bTsYKPdygbTscK7T3eqRExodVKPZWghgpJ\nN0XEjG7H0QnD6VjBxzuUDadjhWocr0t8ZmZWSU5QZmZWSU5Q3XFmtwPooOF0rODjHcqG07FCBY7X\n16DMzKyS3IIyM7NKcoIyM7NKcoLqAklfknSXpN9LukDSuG7H1E6SjpJ0h6Q1koZsN11Jh0m6W9K9\nkj7d7XjaSdL3JT0m6fZux9JukiZLukrSncW/4w93O6Z2kjRa0o2Sbi2O99RuxeIE1R2XA7tFxCuA\nhcCJXY6n3W4HjgTmdTuQdpE0Evg28AZgV+AYSbt2N6q2Ogs4rNtBdMgq4OMR8VJgH+ADQ/x3+xxw\ncES8EtgdOEzSPt0IxAmqCyLisohYVUz+FpjUzXjaLSLujIi7ux1Hm+0F3BsR90XE88B5wFu6HFPb\nRMQ84Klux9EJEfFIRNxcvF8G3Als192o2ieS5cXkRsWrK73pnKC67zjgkm4HYYO2HfBQ3fRihvCX\n2HAlaRrwKuCG7kbSXpJGSroFeAy4PCK6cryjurHT4UDSFcA2/Sw6OSL+p1jnZFL54NxOxtYOOcc7\nxKmfeb6HYwiRNAY4H/hIRCztdjztFBGrgd2L6+MXSNotIjp+vdEJqk0i4tBmyyW9C3gjcEgMgZvR\nWh3vMLAYmFw3PQn4Y5disZJJ2oiUnM6NiJ91O55OiYglkq4mXW/seIJyia8LJB0GfAp4c0Q82+14\nrBTzgZ0kbS/pRcDRwIVdjslKIEnAXODOiPhqt+NpN0kTaj2LJW0CHArc1Y1YnKC641vAWOBySbdI\nOqPbAbWTpCMkLQb2BX4h6dJux1S2otPLPwKXki6i/yQi7uhuVO0j6UfAb4BdJC2WdHy3Y2qj/YB3\nAgcX/19vkXR4t4Nqo5cAV0n6PekPr8sj4qJuBOKhjszMrJLcgjIzs0pygjIzs0pygjIzs0pygjIz\ns0pygjIzs0pygrIhTdKBktbrIttofgn7e2v9QKKSrm41gnsRyzOSLm6x3kllxVlsb3nrtZp+fpak\nbxXvT5B0bAkxPSBpK0mbFN25n5e01WC3a73JCcqsXG8ljWY+UNdGRKt7a0pNUAOhpOH3RUScERFn\nl7W/iPhLROyOR+MY1pygrKskvVjSL4pnz9wu6R3F/D0kXSNpgaRLJb2kmH+1pH+X9Oti/b2K+XsV\n835X/NxlgDF8X9L84vNvKebPkvQzSb+UdI+kL9Z95nhJC4t4/kPStyS9Bngz8KXir/8di9WPKp6v\ns1DS/hnxvETSvGIbt0vaX9LngVqr4txivZ8X5+cOSbPrPr9c0r8V5/S3krYu5m8v6TfFcX62bv0x\nkq6UdLOk2+qOf5rSM5C+A9wMTJb07uI4riHdwFrbximSPiFp27qbWW+RtFrS1GJ0gvOLfc+XtF/x\nufGSLivO+3fpf0xDG64iwi+/uvYC3gb8R9305qTh/X8NTCjmvQP4fvH+6tr6wEzg9uL9ZsCo4v2h\nwPnF+wOBi/rZ7wvzgdOAvy/ejyM9o+vFwCzgviKm0cCDpPH2tgUeALYsYr0W+Fbx+bOAt9ft52rg\nK8X7w4ErmsVSTH+cNMguwEhgbPF+eZ/PbVn83IQ0Ttr4YjqANxXvvwj8c/H+QuDY4v0Hatsjjcm5\nWfF+K+BeUqKYBqwB9imWvQRYBEwAXgRcX3fcpwCf6BPfB0gjagD8F/Da4v0U0rBBAN8A/qV4/zdF\n7FvVbeOB+mm/htfLg8Vat90GfFnSF0hf0tdK2g3YjTQUFKQv6UfqPvMjSM8kkrSZ0rhhY4H/lLQT\n6UtuowHE8HrgzZI+UUyPJn2JAlwZEc8ASPoDMJX0JX5NRDxVzP8psHOT7dcGF11A+tJvZT7wfaUB\nSn8eEbc0WO9Dko4o3k8GdgKeBJ4HatfXFgCvK97vR/qDAOCHwBeK9wJOkzSTlJC2A7Yulj0YEb8t\n3u8NXB0RjwNI+jENjrtoIb0HqLUYDwV2LX6fAJtJGkv6I+NIgIj4haSnGxyrDUNOUNZVEbFQ0h6k\n1sUcSZcBFwB3RMS+jT7Wz/Rngasi4gilZ/ZcPYAwBLwt+jxUUdLepKeL1qwm/Z8ZaBmqto3a55sq\nEu9MUovih5K+FH2u70g6kPSlv29EPKs04vToYvHKiKido7777G9ss78jtYr2iIiVkh6o29af+4bX\nKv6iHDuXNBhyrSPGiCLWv/RZN2ubNjz5GpR1laRtgWcj4hzgy8CrgbuBCZL2LdbZSNLL6j5Wu071\nWuCZooWzOfBwsXzWAMO4FPigim9LSa9qsf6NwAGStpA0irWtEoBlpNbcBpM0FXgsIv6D9EX/6mLR\nyqJVBel4ny6S03TSo8hbuZ40yjqkpFSzebG/lZIOIrUS+3MDcGBx3Wgj4Kh+Yt8I+AnwqYhYWLfo\nMtJgurX1di/ezqvFIukNwBYZx2HDhBOUddvLgRuVnt55MvC5SI9MfzvwBUm3ArcAr6n7zNOSfg2c\nAdRG0f4iqQV2PakkOBCfJZUEfy/p9mK6oYh4mHTd6gbgCuAPwDPF4vOAfyou+u/YYBOtHAjcIul3\npOT39WL+mUWM5wK/BEYpjTj9WeC3/W2ojw8DH5A0n5SUas4FZki6iZQs+n20QkQ8QrrW9BvScd/c\nz2qvAfYETq3rKLEt8KFiH78vSqUnFOufCsyUdDOp1Loo4zhsmPBo5tZTilLWJyLipi7HMSYilhct\nqAtInTgu2MBtHUg6pjeWGeNQUJQbZ0TEE92OxTrPLSizDXNK0eq7Hbgf+PkgtvU8sJta3Kg7nKi4\nUZfUsl3T7XisO9yCMjOzSnILyszMKskJyszMKskJyszMKskJyszMKskJyszMKun/A1shS+ntQQQ3\nAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8XWWd7/HPd+fSpAm90ATohdIEKlgRCgRQUWYEgToi\n4AwOIKPoOIdxjhydozLCOOoMoyPKOHgZjoKKiKOiImBnACtyHQWZplAot0JvlLRFUtrSNk3TXH7n\nj71SNmnS7FxW9t7p9/167dfe61lrPfltaPvLs9ZvPY8iAjMzs2KTKXQAZmZm/XGCMjOzouQEZWZm\nRckJyszMipITlJmZFSUnKDMzK0pOUGZmVpScoMzMrCg5QZmZWVEqL3QAY6Guri7mzJlT6DDMzAxY\nsmTJxoioH+y4fSJBzZkzh+bm5kKHYWZmgKTn8znOl/jMzKwoOUGZmVlRcoIyM7Oi5ARlZmZFyQnK\nzMyK0j5RxTdctz26jqsWLWf9lnZmTKnm0jMO55xjZhY6LDOzfYIT1ABue3Qdl9+yjPbObgDWbWnn\n8luWAThJmZmNgVQv8UlaIGm5pBWSLutn/yckPSXpcUl3SzokZ99Fkp5LXhfltB8naVnS5zckKY3Y\nr1q0fHdy6tXe2c1Vi5an8ePMzKyP1BKUpDLgGuCdwDzgAknz+hz2KNAUEUcBNwNfSc7dH/g8cCJw\nAvB5SVOTc74FXAzMTV4L0oh//Zb2IbWbmdnoSnMEdQKwIiJWRcQu4Cbg7NwDIuLeiNiRbP4emJV8\nPgO4KyI2RcRm4C5ggaTpwKSIeCgiArgROCeN4GdMqR5Su5mZja40E9RM4IWc7ZakbSAfBu4c5NyZ\nyed8+xy2S884nOqKste0VVeUcekZh6fx48zMrI80iyT6uzcU/R4o/QXQBPzRIOcOpc+LyV4KZPbs\n2YPFuofeQogv3P4UG7fvYlpNJZ89c54LJMzMxkiaI6gW4OCc7VnA+r4HSXoH8BngrIjoGOTcFl69\nDDhgnwARcV1ENEVEU339oJPm9uucY2byq789GYCPvv0wJyczszGUZoJaDMyV1CCpEjgfWJh7gKRj\ngGvJJqeXcnYtAk6XNDUpjjgdWBQRG4Btkt6UVO99APhlit+BaTWV7FdVzuqNbWn+GDMz6yO1S3wR\n0SXpErLJpgy4PiKelHQF0BwRC4GrgFrg50m1+NqIOCsiNkn6Z7JJDuCKiNiUfP4b4Aagmuw9qztJ\nkSQa62tZtXF7mj/GzMz6SPVB3Yi4A7ijT9vncj6/Yy/nXg9c3097M3DkKIY5qMa6Gh5e9fJY/kgz\ns32e5+LLQ2NdDetf2cmOXV2FDsXMbJ/hBJWHhvoaANZs3DHIkWZmNlqcoPLQWFcL4PtQZmZjyAkq\nD3PqJgKwutWVfGZmY8UJKg8TK8uZMbmKVS41NzMbM05QeWqor3GCMjMbQ05QeWqsq2VV63ayc9Sa\nmVnanKDy1FBXw7adXbzctqvQoZiZ7ROcoPLUmJSar3KhhJnZmHCCylNvqflql5qbmY0JJ6g8zZxa\nTWVZxiMoM7Mx4gSVp7KMOGTaRFfymZmNESeoIWisr2FVqy/xmZmNBSeoIWioq2Xtph10dfcUOhQz\ns3HPCWoIGutr6OwOWja3FzoUM7NxzwlqCBrrsqXmXl3XzCx9TlBD0FifLTVf6ftQZmapc4IagqkT\nK5hcXeERlJnZGHCCGgJJSSWfE5SZWdqcoIaooa7GIygzszHgBDVEh9bX8uLWnbR1dBU6FDOzcS3V\nBCVpgaTlklZIuqyf/SdLekRSl6Rzc9rfLmlpzmunpHOSfTdIWp2zb36a36GvBlfymZmNidQSlKQy\n4BrgncA84AJJ8/octhb4IPDj3MaIuDci5kfEfOAUYAfw65xDLu3dHxFL0/oO/XGCMjMbG+Up9n0C\nsCIiVgFIugk4G3iq94CIWJPs29vUDOcCd0bEjvRCzV9vgnKhhJlZutK8xDcTeCFnuyVpG6rzgZ/0\nafuipMclXS1pQn8nSbpYUrOk5tbW1mH82P5VVZQxc0q1l90wM0tZmglK/bQNab10SdOBNwKLcpov\nB44Ajgf2Bz7d37kRcV1ENEVEU319/VB+7KAa62s8q7mZWcrSTFAtwME527OA9UPs48+BWyOis7ch\nIjZEVgfwfbKXEsdUQ10Nq1vbiBhSvjUzsyFIM0EtBuZKapBUSfZS3cIh9nEBfS7vJaMqJAk4B3hi\nFGIdksa6GrZ1dNG6vWOsf7SZ2T4jtQQVEV3AJWQvzz0N/CwinpR0haSzACQdL6kFeC9wraQne8+X\nNIfsCOz+Pl3/SNIyYBlQB3whre8wkIZkTr7VLpQwM0tNmlV8RMQdwB192j6X83kx2Ut//Z27hn6K\nKiLilNGNcuh6ZzVftbGNExunFTgaM7PxyTNJDMOMKdVUlmf8LJSZWYqcoIahLCMapnn5dzOzNDlB\nDVNDnUvNzczS5AQ1TI31Nax9eQed3XubBMPMzIbLCWqYGupq6OoJWja3FzoUM7NxyQlqmHqXf/d9\nKDOzdDhBDVOjZzU3M0uVE9QwTa2pZOrEClb6YV0zs1Q4QY1Advl3X+IzM0uDE9QINNbXel0oM7OU\nOEGNQENdDS9t62B7R1ehQzEzG3ecoEbg0PqkUMKjKDOzUecENQINdUmpue9DmZmNOieoEThk2kQk\nfB/KzCwFTlAjUFVRxswp1X4WyswsBU5QI9RYX+tLfGZmKXCCGqHGuhpWt7YREYUOxcxsXHGCGqHG\n+hradnXz0raOQodiZjauOEGNUEPv8u8ulDAzG1VOUCO0e1Zz34cyMxtVTlAjNH1SFVUVGT+sa2Y2\nylJNUJIWSFouaYWky/rZf7KkRyR1STq3z75uSUuT18Kc9gZJD0t6TtJPJVWm+R0Gk8mIOdO8/LuZ\n2WhLLUFJKgOuAd4JzAMukDSvz2FrgQ8CP+6ni/aImJ+8zspp/zJwdUTMBTYDHx714Ieosb7Gz0KZ\nmY2yNEdQJwArImJVROwCbgLOzj0gItZExONATz4dShJwCnBz0vQD4JzRC3l4GutqWbtpB7u68voa\nZmaWhzQT1EzghZztlqQtX1WSmiX9XlJvEpoGbImI3unDB+xT0sXJ+c2tra1DjX1IGupq6O4JXti8\nI9WfY2a2L0kzQamftqE8zTo7IpqA9wFfk3ToUPqMiOsioikimurr64fwY4eusd6l5mZmoy3NBNUC\nHJyzPQtYn+/JEbE+eV8F3AccA2wEpkgqH06faWlMZjX36rpmZqMnzQS1GJibVN1VAucDCwc5BwBJ\nUyVNSD7XAScBT0V2PqF7gd6Kv4uAX4565EM0eWIF02oqXShhZjaKUktQyX2iS4BFwNPAzyLiSUlX\nSDoLQNLxklqA9wLXSnoyOf31QLOkx8gmpCsj4qlk36eBT0haQfae1PfS+g5D0VBXw0pf4jMzGzXl\ngx8yfBFxB3BHn7bP5XxeTPYyXd/zHgTeOECfq8hWCBaVxvoa7l2ebjGGmdm+xDNJjJKGulpat3Ww\nbWdnoUMxMxsXnKBGSW8ln+9DmZmNDieoUdLoWc3NzEaVE9QomT1tIhnhOfnMzEaJE9QomVBexqyp\nE1nV6mehzMxGgxPUKPKksWZmo2fQBCXp4/m0WfZZqNUb28g+T2xmZiORzwjqon7aPjjKcYwLjfW1\n7NjVzR+2dhQ6FDOzkjfgg7qSLiA7UWtD7oKBwCTg5bQDK0WvVvJt56DJVQWOxsystO1tJokHgQ1A\nHfDVnPZtwONpBlWqds9qvrGNtxxWV+BozMxK24AJKiKeB56X9A6yq9v2SHodcASwbKwCLCUH7ldF\ndUWZn4UyMxsF+dyDeoDs4oEzgbuBDwE3pBlUqcpklBRKuNTczGyk8klQiogdwJ8C34yI9wDz0g2r\ndDXU1/hhXTOzUZBXgpL0ZuBC4PakLdVZ0EvZoXU1vLBpB7u6egodiplZScsnQf0tcDlwa7KeUyPZ\nNZqsHw31NfQErN3kUZSZ2UgMOhKKiPuB+yXtJ6k2WY/pY+mHVpp6l39f1drGYQfsV+BozMxKVz4z\nSbxR0qPAE8BTkpZIekP6oZWmhpxSczMzG758LvFdC3wiIg6JiNnAJ4HvpBtW6ZpUVUFd7QRWu9Tc\nzGxE8klQNRGx+55TRNwH1KQW0TjQWFfDKpeam5mNSD4JapWkz0qak7z+AViddmClzLOam5mNXD4J\n6i+BeuCW5FVH9mFdG0BDXQ0bt+/ilfbOQodiZlayBkxQkqok1UfE5oj4WEQcGxHHAl8C2vPpXNIC\nScslrZB0WT/7T5b0iKQuSefmtM+X9JCkJyU9Lum8nH03SFotaWnymj+0r5y+xvpsJZ9HUWZmw7e3\nEdQ3gLf10/4O4OrBOpZUBlwDvJPszBMXSOo7A8Vaskt3/LhP+w7gAxHxBmAB8DVJU3L2XxoR85PX\n0sFiGWsNObOam5nZ8OwtQb01Im7p2xgRPwJOzqPvE4AVEbEqInYBNwFn9+lrTUQ8DvT0aX82Ip5L\nPq8HXiJ7mbEkzN5/ImUZeQRlZjYCe0tQGuZ5vWYCL+RstyRtQyLpBKASWJnT/MXk0t/VkiYMcN7F\nkpolNbe2tg71x45IZXmGg6dWe1ZzM7MR2FuieSlJDq8h6Xggn3/x+0twQ1oLXdJ04IfAhyKid5R1\nOdklP44H9gc+3d+5EXFdRDRFRFN9/dgPvhrra/2wrpnZCOxtqqNLgZ9JugFYkrQ1AR8Azs+j7xbg\n4JztWcD6fAOTNIns5LT/EBG/722PiA3Jxw5J3wc+lW+fY6mhroYHV26kpyfIZPY2GDUzs/4MOIKK\niP8hex9JZAsZPph8PjEiHs6j78XAXEkNkirJJrWFg5wDQHL8rcCNEfHzPvumJ+8CziE7BVPRaayv\nYWdnDy9u3VnoUMzMStJeJ4uNiJeAzw+n44joknQJsAgoA65PZkO/AmiOiIXJ5cJbganAuyX9U1K5\n9+dkCzGmSfpg0uUHk4q9H0mqJ5sslwIfGU58aXu1kq+NGVOqCxyNmVnpSXVdp4i4A7ijT9vncj4v\nJnvpr+95/wH8xwB9njLKYaaid1bz1Ru389a5dQWOxsys9ORTjWfDcOCkCUysLGOlK/nMzIYl7wQl\nyRPEDoEkGuo8J5+Z2XDlsx7UWyQ9BTydbB8t6f+lHtk40Fhf6wRlZjZM+YygrgbOAF4GiIjHyG8m\niX1eQ10NLZt30NHVXehQzMxKTl6X+CLihT5N/hc3D4fW19ATsPblHYUOxcys5OSToF6Q9BYgJFVK\n+hTJ5T7bu95ScxdKmJkNXT4J6iPAR8nOo9cCzE+2bRC9Ccr3oczMhm7Q56AiYiNw4RjEMu7sV1VB\n/X4TvOyGmdkwDJqgJH2jn+ZXyM4G8cvRD2l8aXSpuZnZsORzia+K7GW955LXUWRnEf+wpK+lGNu4\n0Fhf41nNzcyGIZ+pjg4DTomILgBJ3wJ+DZwGLEsxtnGhsa6WTW0vsGXHLqZMrCx0OGZmJSPfhQdz\nZ5GoAWZERDfQkUpU48juSWM9ijIzG5J8RlBfAZZKuo/sDOInA/+STH30mxRjGxca65NKvtY2jp09\ntcDRmJmVjnyq+L4n6Q5eXRvq7yOid+HBS9MMbjw4eP+JlGfEqo2u5DMzG4p8J4vdCWwANgGHSfJU\nR3mqKMswe/+JruQzMxuifMrM/wr4ONl1m5YCbwIeAkpiXaZi0FBXwyrPJmFmNiT5jKA+DhwPPB8R\nbweOAVpTjWqcaazPPgvV0xOFDsXMrGTkk6B2RsROAEkTIuIZ4PB0wxpfGupq6ejqYf0r7YUOxcys\nZORTxdciaQpwG3CXpM3A+kHOsRy7K/k2tjFr6sQCR2NmVhryqeJ7T/LxHyXdC0wGfpVqVONMY++z\nUK1tvG1ufYGjMTMrDXtNUJIywOMRcSRARNw/JlGNM/X7TaB2Qrkr+czMhmCv96Aiogd4TNLs4XQu\naYGk5ZJWSLqsn/0nS3pEUpekc/vsu0jSc8nropz24yQtS/r8hiQNJ7axJImGuhpWelZzM7O85VMk\nMR14UtLdkhb2vgY7SVIZcA3wTmAecIGkeX0OWwt8EPhxn3P3Bz4PnEj2AeHPS+qdhuFbwMXA3OS1\nII/vUHC9lXxmZpaffIok/mmYfZ8ArIiIVQCSbgLOBp7qPSAi1iT7evqcewZwV0RsSvbfBSxIplua\nFBEPJe03AucAdw4zxjHTUFfDwsfWs7Ozm6qKskKHY2ZW9AYdQSX3ndYAFcnnxcAjefQ9E3ghZ7sl\nacvHQOf2ruo7aJ+SLpbULKm5tbXwj2011tcSAc+/vKPQoZiZlYRBE5Sk/wXcDFybNM0kW3I+6Kn9\ntOX7pOpA5+bdZ0RcFxFNEdFUX1/4yrlXK/l8H8rMLB/53IP6KHASsBUgIp4DDsjjvBbg4JztWeT/\n/NRA57Ykn4fTZ0F52Q0zs6HJJ0F1RMSu3g1J5eQ3EloMzJXUIKkSOB8YtLgisQg4XdLUpDjidGBR\nRGwAtkl6U1K99wGgJJadr5lQzoGTJnhOPjOzPOWToO6X9PdAtaTTgJ8D/znYSckKvJeQTTZPAz+L\niCclXSHpLABJx0tqAd4LXCvpyeTcTcA/k01yi4EregsmgL8BvgusAFZSAgUSvRrralntZTfMzPKi\niL0PhpKHdT9MdhQjsgnnuzHYiUWkqakpmpubCx0Gf3/rMu5YtoGlnzu90KGYmRWMpCUR0TTYcfmU\nmZ8N3BgR3xl5WPu2xroatuzoZHPbLqbWVBY6HDOzopbPJb6zgGcl/VDSu5J7UDYMvZPGenVdM7PB\n5fMc1IeAw8jee3ofsFLSd9MObDxqrKsFcKGEmVke8hoNRUSnpDvJVu9Vk73s91dpBjYezZpaTUWZ\nXGpuZpaHfB7UXSDpBrJVc+eSraCbnnJc41J5WYbZ+09ktUdQZmaDymcE9UHgJuCvI6Ij3XDGv4a6\nWt+DMjPLQz73oM6PiNt6k5OkkyRdk35o49Oh9TWseXkH3T0lU6VvZlYQ+VTxIWm+pK9IWgN8AXgm\n1ajGsYa6GnZ19bB+S3uhQzEzK2oDXuKT9Dqy0xNdALwM/JTsg71vH6PYxqXG+qSSb2MbB+8/scDR\nmJkVr72NoJ4BTgXeHRFvjYhvAt1jE9b41Ttp7GrPam5mtld7S1B/BrwI3CvpO5JOpf/lLmwI6mor\n2a+q3KXmZmaDGDBBRcStEXEecARwH/B/gQMlfUuSJ5MbJkk01nn5dzOzweRTxdcWET+KiDPJrr+0\nFLgs9cjGscb6Ws8mYWY2iLyq+HpFxKaIuDYiTkkroH1BQ10N67a0s7PTt/TMzAYypARlo6N30lhf\n5jMzG5gTVAHsruRzgjIzG5ATVAH0JqhVLjU3MxuQE1QBTKwsZ/rkKpeam5nthRNUgTTW17iSz8xs\nL5ygCqShroZVrduJ8KSxZmb9cYIqkMa6Wrbu7GJT265Ch2JmVpRSTVDJYofLJa2QtMfDvZImSPpp\nsv9hSXOS9gslLc159Uian+y7L+mzd98BaX6HtDS41NzMbK9SS1CSyoBrgHcC84ALJM3rc9iHgc0R\ncRhwNfBlgGTmivkRMR94P7AmIpbmnHdh7/6IeCmt75CmQ+uSWc19H8rMrF9pjqBOAFZExKqI2EV2\nVd6z+xxzNvCD5PPNwKmS+k5IewHwkxTjLIiZU6upLMu4ks/MbABpJqiZwAs52y1JW7/HREQX8Aow\nrc8x57Fngvp+cnnvs/0kNAAkXSypWVJza2vrcL9Dav7zsfX0RPDt+1dy0pX3cNuj6wodkplZUUkz\nQfWXOPqWrO31GEknAjsi4omc/RdGxBuBtyWv9/f3wyPiuohoioim+vr6oUWestseXcfltyyjK1n2\nfd2Wdi6/ZZmTlJlZjjQTVAtwcM72LGD9QMdIKgcmA5ty9p9Pn9FTRKxL3rcBPyZ7KbGkXLVoOe19\nJopt7+zmqkXLCxSRmVnxSTNBLQbmSmqQVEk22Szsc8xC4KLk87nAPZE8GCQpA7yX7L0rkrZySXXJ\n5wrgTOAJSsz6Le1Dajcz2xeVp9VxRHRJugRYBJQB10fEk5KuAJojYiHwPeCHklaQHTmdn9PFyUBL\nRKzKaZsALEqSUxnwG+A7aX2HtMyYUs26fpLR1IkVBYjGzKw4aV+YyaCpqSmam5sLHcZuvfegci/z\nZQQ9AV98z5FceOIhBYzOzCxdkpZERNNgx6U2grKBnXNMtpjxqkXLWb+lnRlTqvn4qXP51ZMv8plb\nn2DT9l1ccsphDFCgaGa2T3CCKpBzjpm5O1H1es+xM/n0Lx7nq3c9y8ttu/jcmfPIZJykzGzf5ARV\nRCrKMvzruUczraaS7/z3aja17eJf33s0leWeMtHM9j1OUEUmkxGfedc86mon8KU7n2Hzjl18+y+O\no2aC/1eZ2b7Fv5oXqb/+o0P5yrlH8bsVG3nfdx/2rOdmts9xgipif950MNe+v4lnNmzl3G8/2G9p\nupnZeOUEVeROm3cgN/7lCbRu6+Dcbz3Ic3/YVuiQzMzGhBNUCTixcRo/vfjNdPUE7732IR5Zu7nQ\nIZmZpc4JqkTMmzGJX3zkLUypruDC7zzMfctLchksM7O8OUGVkNnTJvLzj7yFxvoa/uoHzfxyqWc/\nN7PxywmqxNTvN4GbLn4Tx8/Zn4/ftJTv/251oUMyM0uFE1QJ2q+qgu9/6HgWvOEg/uk/n+JfFy1n\nX5hT0cz2LU5QJaqqooxrLjyWC06Yzb/fu4K/v3UZ3T1OUmY2fnh6ghJWlhH/8p4jqaut5Jv3rGBT\n2y6+fv4xVFWUFTo0M7MR8wiqxEnik6cfzuffPY9FT/6Bi67/H7bu7Cx0WGZmI+YR1DjxoZMa2L+m\nkk/+7DEWXP0APQF/2LqTGVOqufSMw/eYOd3MrNg5QY0jZ8+fyZPrX+G6B16t7Fu3pZ3Lb1kG4CRl\nZiXFl/jGmdsff3GPtvbObq5atLwA0ZiZDZ8T1DizfoAJZddtaeeVHb43ZWalwwlqnJkxpXrAfW+5\n8m6+8F9PseEVz4puZsUv1QQlaYGk5ZJWSLqsn/0TJP002f+wpDlJ+xxJ7ZKWJq9v55xznKRlyTnf\nkOQ10XNcesbhVPcpM6+uKOPSMw7nHfMO5PsPruFtX76XT/7sMc+MbmZFLbUiCUllwDXAaUALsFjS\nwoh4KuewDwObI+IwSecDXwbOS/atjIj5/XT9LeBi4PfAHcAC4M6UvkbJ6S2EuGrRctZvad+jiu9T\npx/O9367mpsWr+UXj7Rw6hEH8JE/PpTj5+xfyLDNzPagtKbIkfRm4B8j4oxk+3KAiPhSzjGLkmMe\nklQOvAjUA4cA/xURR/bpczpwb0QckWxfAPxxRPz13mJpamqK5ubm0fty48Cmtl3c+NAafvDgGjbv\n6OS4Q6by1yc38o7XH0gm40GpmaVH0pKIaBrsuDQv8c0EXsjZbkna+j0mIrqAV4Bpyb4GSY9Kul/S\n23KObxmkT8vD/jWV/O07XseDl53KP531Bv6wdScX/3AJp119Pz9b/AIdXd2FDtHM9nFpJqj+fg3v\nO1wb6JgNwOyIOAb4BPBjSZPy7DPbsXSxpGZJza2trUMIe99SXVnGRW+Zw32f+mO+fv58JpSX8Xe/\neJyTv3Iv196/0rNSmFnBpJmgWoCDc7ZnAesHOia5xDcZ2BQRHRHxMkBELAFWAq9Ljp81SJ8k510X\nEU0R0VRfXz8KX2d8Ky/LcPb8mdz+sbdy41+ewGEH1PKlO5/hpC/dw5V3PsNLW3cWOkQz28ekOZPE\nYmCupAZgHXA+8L4+xywELgIeAs4F7omIkFRPNlF1S2oE5gKrImKTpG2S3gQ8DHwA+GaK32GfI4mT\nX1fPya+r5/GWLVz7wCque2Al1/92NX967Ez+18mNLGt5ZcAiDDOz0ZJagoqILkmXAIuAMuD6iHhS\n0hVAc0QsBL4H/FDSCmAT2SQGcDJwhaQuoBv4SERsSvb9DXADUE22es8VfCk5atYUrnnfsazZ2MZ3\nf7uKnze3cNPiF8gIelf28FRKZpaW1Kr4iomr+EbHxu0dvP2q+9jW0bXHvplTqvndZacUICozKzXF\nUMVn40xd7QS295OcIDuS+sbdz7Hipe1jHJWZjVeezdyGZMaUatb1M99fZVmGq3/zLP9217MccdB+\n/Mkbp/Ouo6ZzaH1tAaI0s/HAIygbkoGmUvrKuUfx0GWn8vl3z2O/qnL+7a5nOfWr97Pgaw/wzbuf\nY2WrR1ZmNjS+B2VDdtuj6wat4tvwSjt3LnuRO5ZtoPn5zQAccdB+nHnUdP7kjdNp9MjKbJ+V7z0o\nJyhLXW+yun3ZBpYkyer10yftTlYNdTW7j80n+ZlZaXOCyuEEVTzWb2nnzide5PbH1/PI2i0AzJs+\niXcdNZ3Ksgz/dteztHe+Os1SdUUZX/rTNzpJmY0jTlA5nKCK0/ot7dyxbAO3L9vAo0my6o9L2M3G\nF5eZW9GbMaWav3pbI7f+75P2moDWbWnnv59rHbDE3czGJ5eZW1GYOaWamQOUsAO8/3v/Q0ZwxEGT\naJozleMOyb5mTqnGa1aajU9OUFY0Lj3jcC6/Zdke96A+9+7XM2PKRJas2cSStZu5eUkLNz70PAAH\nTaranaya5kzl9dMnUVHmCwNm44ETlBWNwVYD/qPXZWel7+ru4ZkXt7Hk+c00P7+ZJWs2cfuyDUA2\noR198GSaDtmf4w6ZyrGzpzJ5YgXgCkGzUuMiCRsX1m9pZ8nzm3e/ntqwle5kRtvXHVjLtJpKmp/f\nTGf3q3/eXSFoVhj5Fkl4BGXjwowp1cyYUs27j54BQFtHF4+9sCU7wnp+Mw8827rHypbtnd18fuET\nTJ5YwdwDapkxudrL3ZsVEY+gbJ/QcNnt/S+9nKO6oozDDqjd43XI/hMp930ts1HjEZRZjoEmuT1o\nchVfP28+K1q3s+Kl7Ov3q17m1kfX7T6msizDnLqJzD1gPw49oJa5SeJqqKuhKpmX0Pe3zEafE5Tt\nEwaqELxswRGc2DiNExunveb4bTs7WdnaxoqXtvPcS9tY+dJ2nlj/Cnc+sWH3Yo0Zwez9J1IzoYzl\nL26nK9mI8SybAAAMFUlEQVSRXcTxccCLOJqNhC/x2T5jNEY5Ozu7Wb2xjede6h1xbePXT/5hd3LK\nVZ4Rbz50GrOmZp/xmjm1mplTJjJrajUHTqqiLM/7XR6d2XjjqY5yOEFZmvZ2f+voWZNZt6Wdjdt3\nvaa9PCMOmlzFzCnVzJo6kZlTq5mVJLFZU6uZPrmayvIMtz26rt+Rn6sPrZT5HpTZGBno/tbMKdX8\n8pK3AtC+q5t1W9qzr83ttGzesfvzgys38uLWneT+rijBAftNYFPbrteUxkO2+vCLtz/NUbMmM6m6\ngklVFVSWD7+IwyM0K1ZOUGYjNND9rUvPOPzV7cpXKwT7s6urhxdf2UnLlh20bM4mrnVb2rl5SUu/\nx7du7+CUr96/e7uqIsOkqgomV1ckSat8d/KaVF2eba+q2KPtt89t5J9vf4qdnT1A7/2zZcDo3D9z\n8rORcIIyG6HBZsDIR2V5htnTJjJ72sTXtD+08uV+R2fTair57Jnz2Lqzk63tnWzd2cXW9k5eae9k\n685ONm7fxaqNbbv3dfdzj2wg7Z3d/N0vHue/Hl9PzYRyaiaUUzuhnJrKcmomlGU/97ZN2LNtQnkG\nSXtcniyl5OfEWhxSvQclaQHwdaAM+G5EXNln/wTgRuA44GXgvIhYI+k04EqgEtgFXBoR9yTn3AdM\nB3r/1p4eES/tLQ7fg7JSNRr3oCKCHbu6dyevre1dSeLq5BM/e2zA8+ZNn0Tbri7aOrrY3tG1e5Q1\nmLKMqKkso62ji+5+/nmprijj7PkzqKooY0JFhqryV9+rKsqoqsgwoTz73nd7QnLM3U+/yD/+51Ov\niWm07s2lfd8v7eRXCom74EUSksqAZ4HTgBZgMXBBRDyVc8z/Bo6KiI9IOh94T0ScJ+kY4A8RsV7S\nkcCiiJiZnHMf8KmIyDvjOEFZKUvzH5yTrrxnwPtnfZdA6eruoW1XN20dryatto7u5L2Ltl1dr37u\n6OaGB9cM+HMP2G8CHV097OzspqMrv8SXj4zYXWBSnhEVZRkqykR58p7dTvaVZ6jI9O57df9PF7/Q\n79Iuk6sr+LsFh1ORyVCWEeVlojyTSd6z/ZRnej9n973muIy455k/8JVfLWdnzneuqshwxVlHcvYx\nM8hIlEnDntEkzeQ6mn0XQ4J6M/CPEXFGsn05QER8KeeYRckxD0kqB14E6iMnKGXXUtgIzIiIDico\ns9GT5j9o+Sa/np5gV3cPHZ097OzqZmdnNzs7e+joyr5nt7t3J7SdXT189rYnBvy5f3bsLDq7e+jq\n6WFXV9DV00Nndw+d3UFX8p7d7qGrJ+js6qGzJ9vW1R1Fs+5YRtnRaEbZV1lGKGkrk5BEWYacz2Ld\nlvZ+L+dWlIn5B0/JJsxMhkySSMuS99ztMmWT6u5jk+0fP7y23/82w1lQtBiq+GYCL+RstwAnDnRM\nRHRJegWYRjYh9foz4NGI6Mhp+76kbuAXwBeinywr6WLgYoDZs2eP8KuYjU+jcf9sIPkUjwBkMqIq\nk710N5mKvPr+9n0rB0x+X/3zo0cU90CJ9aDJVfzyoyfR1ZNNdNn3bALMvmfbu3uCzp6guyebDLuT\n5NfdE3u9pHrpGYfT0xN0R9AT5HwOenqybd09yXYE3T0k7clxPcHaTTv67buzOyjPZOiOoL2zm64k\nvu4e6O7JfpeenkjaY4/t7p54zf/HXOsHWMNtNKSZoPobo/ZNJHs9RtIbgC8Dp+fsvzAi1knaj2yC\nej/Z+1iv7STiOuA6yI6ghha62b7jnGNmplIAUAzJbzT7vmzBERw4qWpEfX/1188OmFg/+vbDRtQ3\nwOI1mwfs/ycXv2lEfQ+UuGdMqR5Rv3uT5gyYLcDBOduzgPUDHZNc4psMbEq2ZwG3Ah+IiJW9J0TE\nuuR9G/Bj4ISU4jezETrnmJn87rJTWH3lu/jdZaeMWiI855iZfOlP35hdUZnsP8CjVcSQZt+XnnE4\n1cn8jb1GK7Gm3X/asfcnzRHUYmCupAZgHXA+8L4+xywELgIeAs4F7omIkDQFuB24PCJ+13twksSm\nRMRGSRXAmcBvUvwOZlak0hr5pdl3mqPKtPtPO/b+pF1m/ifA18iWmV8fEV+UdAXQHBELJVUBPwSO\nITtyOj8iVkn6B+By4Lmc7k4H2oAHgIqkz98An4iI/i+OJlwkYWZWPApexVdMnKDMzIpHvgnKq7CZ\nmVlRcoIyM7Oi5ARlZmZFyQnKzMyKkhOUmZkVpX2iik9SK/B8oeMYQB2vndqplJRq7I577JVq7KUa\nNxR37IdERP1gB+0TCaqYSWrOp9yyGJVq7I577JVq7KUaN5R27L18ic/MzIqSE5SZmRUlJ6jCu67Q\nAYxAqcbuuMdeqcZeqnFDaccO+B6UmZkVKY+gzMysKDlBmZlZUXKCKhBJB0u6V9LTkp6U9PFCxzQU\nksokPSrpvwody1BImiLpZknPJP/t31zomPIh6f8mf06ekPSTZKmaoiTpekkvSXoip21/SXdJei55\nn1rIGPszQNxXJX9WHpd0a7JWXVHpL+6cfZ+SFJLqChHbSDlBFU4X8MmIeD3wJuCjkuYVOKah+Djw\ndKGDGIavA7+KiCOAoymB7yBpJvAxoCkijiS7Ftr5hY1qr24AFvRpuwy4OyLmAncn28XmBvaM+y7g\nyIg4CniW7Dp1xeYG9owbSQcDpwFrxzqg0eIEVSARsSEiHkk+byP7D2V6S1OOIkmzgHcB3y10LEMh\naRJwMvA9gIjYFRFbChtV3sqB6mRV6YnA+gLHM6CIeIDsAqS5zgZ+kHz+AXDOmAaVh/7ijohfR0RX\nsvl7YNaYBzaIAf57A1wN/B1QspVwTlBFQNIcsqsKP1zYSPL2NbJ/8HsKHcgQNQKtwPeTy5PflVRT\n6KAGExHrgH8l+5vwBuCViPh1YaMasgMjYgNkfzkDDihwPMPxl8CdhQ4iH5LOAtZFxGOFjmUknKAK\nTFIt8AvgbyNia6HjGYykM4GXImJJoWMZhnLgWOBbEXEM0EZxXmp6jeR+zdlAAzADqJH0F4WNat8i\n6TNkL8v/qNCxDEbSROAzwOcKHctIOUEVkKQKssnpRxFxS6HjydNJwFmS1gA3AadI+o/ChpS3FqAl\nInpHqjeTTVjF7h3A6ohojYhO4BbgLQWOaaj+IGk6QPL+UoHjyZuki4AzgQujNB4cPZTsLzOPJX9P\nZwGPSDqooFENgxNUgUgS2XshT0fEvxU6nnxFxOURMSsi5pC9UX9PRJTEb/MR8SLwgqTDk6ZTgacK\nGFK+1gJvkjQx+XNzKiVQ3NHHQuCi5PNFwC8LGEveJC0APg2cFRE7Ch1PPiJiWUQcEBFzkr+nLcCx\nyZ//kuIEVTgnAe8nOwJZmrz+pNBB7QP+D/AjSY8D84F/KXA8g0pGfDcDjwDLyP69LdppbCT9BHgI\nOFxSi6QPA1cCp0l6jmxl2ZWFjLE/A8T978B+wF3J39FvFzTIfgwQ97jgqY7MzKwoeQRlZmZFyQnK\nzMyKkhOUmZkVJScoMzMrSk5QZmZWlJygzMaIpO6cRwqWShq1WSwkzelvNmuzUlZe6ADM9iHtETG/\n0EGYlQqPoMwKTNIaSV+W9D/J67Ck/RBJdydrEd0taXbSfmCyNtFjyat32qMySd9J1o36taTq5PiP\nSXoq6eemAn1NsyFzgjIbO9V9LvGdl7Nva0ScQHbmgq8lbf8O3JisRfQj4BtJ+zeA+yPiaLJzCT6Z\ntM8FromINwBbgD9L2i8Djkn6+UhaX85stHkmCbMxIml7RNT2074GOCUiViUTCL8YEdMkbQSmR0Rn\n0r4hIuoktQKzIqIjp485wF3JgoBI+jRQERFfkPQrYDtwG3BbRGxP+auajQqPoMyKQwzweaBj+tOR\n87mbV+8xvwu4BjgOWJIsemhW9JygzIrDeTnvDyWfH+TVpd0vBH6bfL4b+BsASWXJSsH9kpQBDo6I\ne8kuMjkF2GMUZ1aM/JuU2diplrQ0Z/tXEdFbaj5B0sNkf2m8IGn7GHC9pEvJrgT8oaT948B1yazV\n3WST1YYBfmYZ8B+SJgMCri6hZe5tH+d7UGYFltyDaoqIjYWOxayY+BKfmZkVJY+gzMysKHkEZWZm\nRckJyszMipITlJmZFSUnKDMzK0pOUGZmVpT+Px5Bt3iiAZtQAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ada = AdalineSGD(n_iter=15, eta=0.01, random_state=1)\n", + "ada.fit(X_std, y)\n", + "\n", + "plot_decision_regions(X_std, y, classifier=ada)\n", + "plt.title('Adaline - Stochastic Gradient Descent')\n", + "plt.xlabel('sepal length [standardized]')\n", + "plt.ylabel('petal length [standardized]')\n", + "plt.legend(loc='upper left')\n", + "\n", + "plt.tight_layout()\n", + "#plt.savefig('./adaline_4.png', dpi=300)\n", + "plt.show()\n", + "\n", + "plt.plot(range(1, len(ada.cost_) + 1), ada.cost_, marker='o')\n", + "plt.xlabel('Epochs')\n", + "plt.ylabel('Average Cost')\n", + "\n", + "plt.tight_layout()\n", + "# plt.savefig('./adaline_5.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "<__main__.AdalineSGD at 0x110dfc9e8>" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ada.partial_fit(X_std[0, :], y[0])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/1.1.2 MLP and MNIST.ipynb b/1.1.2 MLP and MNIST.ipynb new file mode 100644 index 0000000..e21c231 --- /dev/null +++ b/1.1.2 MLP and MNIST.ipynb @@ -0,0 +1,1503 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "##### (exceprt from Python Machine Learning Essentials, Supplementary Materials)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Sections" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "- [Classifying handwritten digits](#Classifying-handwritten-digits)\n", + " - [Obtaining the MNIST dataset](#Obtaining-the-MNIST-dataset)\n", + " - [Implementing a multi-layer perceptron](#Implementing-a-multi-layer-perceptron)\n", + "- [Training an artificial neural network](#Training-an-artificial-neural-network)\n", + "- [Debugging neural networks with gradient checking](#Debugging-neural-networks-with-gradient-checking)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "# Classifying handwritten digits" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Obtaining the MNIST dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of the following four parts:\n", + "\n", + "- Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 samples)\n", + "- Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels)\n", + "- Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 samples)\n", + "- Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels)\n", + "\n", + "In this section, we will only be working with a subset of MNIST, thus, we only need to download the training set images and training set labels. After downloading the files, I recommend unzipping the files using the Unix/Linux gzip tool from the terminal for efficiency, e.g., using the command \n", + "\n", + " gzip *ubyte.gz -d\n", + " \n", + "in your local MNIST download directory, or, using your favorite unzipping tool if you are working with a machine running on Microsoft Windows. The images are stored in byte form, and using the following function, we will read them into NumPy arrays that we will use to train our MLP.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import os\n", + "import struct\n", + "import numpy as np\n", + " \n", + "def load_mnist(path, kind='train'):\n", + " \"\"\"Load MNIST data from `path`\"\"\"\n", + " labels_path = os.path.join(path, \n", + " '%s-labels-idx1-ubyte' \n", + " % kind)\n", + " images_path = os.path.join(path, \n", + " '%s-images-idx3-ubyte' \n", + " % kind)\n", + " \n", + " with open(labels_path, 'rb') as lbpath:\n", + " magic, n = struct.unpack('>II', \n", + " lbpath.read(8))\n", + " labels = np.fromfile(lbpath, \n", + " dtype=np.uint8)\n", + "\n", + " with open(images_path, 'rb') as imgpath:\n", + " magic, num, rows, cols = struct.unpack(\">IIII\", \n", + " imgpath.read(16))\n", + " images = np.fromfile(imgpath, \n", + " dtype=np.uint8).reshape(len(labels), 784)\n", + " \n", + " return images, labels" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Rows: 60000, columns: 784\n" + ] + } + ], + "source": [ + "X_train, y_train = load_mnist('mnist', kind='train')\n", + "print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Rows: 10000, columns: 784\n" + ] + } + ], + "source": [ + "X_test, y_test = load_mnist('mnist', kind='t10k')\n", + "print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "Visualize the first digit of each class:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHnJJREFUeJzt3Xm81XMex/H3IWUXjYx5YJBEMynL2IaHpt0yGSUykhJD\nZaxRmploG4yhIipbCzN2ZpgxlkqMBllHC7KMXYoSUhFn/uDzvZ/j/u695957lu859/X8x+fxPcv9\n9XXu/Zzv9/f9fr6pdDotAABis16xLwAAgCQkKABAlEhQAIAokaAAAFEiQQEAotSougdTqRRL/LKU\nTqdTNT2H/sxeNv0p0afZoj9zi/7MvaQ+rTZBffei/FxNGUmlsvqsSqI/s1Gb/pTo05rQn7lFf+Ze\nVX3KFB8AIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAA\ngCiRoAAAUSJBAQCiRIICAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIA\nRIkEBQCIEgkKABAlEhQAIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAg\nSiQoAECUSFAAgCg1KvYF1Mc777wT4gkTJkiSxo0bF9rOPvvsEJ955pmSpO23375AVwcAqA9GUACA\nKJGgAABRSqXT6aofTKXS1T1eDO+9916I27ZtG+JPPvmk2tdtueWWkqRly5bl/JpSqZTS6XQqi+dF\n15+1cd1114X4tNNOkyR98803oe2VV14J8a677lrnn5Ntf3733Kj7dO3atZKkr776KrQ9/vjjkjI/\nyyeeeGKIGzXK7cx77P350UcfhXjdunWSpHnz5oW2I488MsTrrVf779T9+/eXJE2ZMiW0rb/++rV+\nHxN7f+bbSy+9FOJOnTqF+IUXXpAkbb311rV+z6r6lBEUACBKJCgAQJRKZhXfW2+9JUlq3759aFux\nYkWIU6lvR4dbbLFFaGvSpEmIly5dKkl64403QtuPf/zjENdnyF/OZs2aFeJzzjknxElTLfb/oCHy\nU8yXX355iGfPni1Jeuqpp6p9vZ/uGzFiRI6vLh5LliwJ8YwZMyRJ1157bWizKeO33347tPnPWl0+\nY9OmTZNUMc0vSWPGjAmx/ztRTK+++mqI7W/bvvvuW6zLqZL/LHfs2DGvP4sRFAAgSlGOoOyGso2a\nJKlbt26SMvc+JWnXrl2Ix44dG+KDDjpIktSyZcvQ5r+5DRgwoB5XXL4WL14c4jVr1hTxSuLhF9rY\n/jv7ryStXr06xHaDfKeddgptzZo1kyQ9++yzoc3fwB84cKCkut1sjt2wYcNCfPPNNxf0Z/s9krbI\nR5JatGhR0Ouoip+tePnllyXFNYKyz7If6fm/D/nACAoAECUSFAAgSlFO8Z133nmSpIkTJ9b6tY8+\n+miIV61aFeKjjjpKknT33XeHtueff76ul1j2Fi1aJEm66KKLEh/fa6+9JEkPPfRQaNtkk03yfl2F\n5qc17cb6pEmTQtvKlSurfX2bNm0kZX4uba/PNttsE9o+/PDDSu9ZjlN8v/zlL0OcNMX3ox/9SJI0\nZMiQ0Ob32iUtzvn3v/8tSbrnnntydp3FcOWVV4a4S5cuRbySZJ9//rkk6eKLLw5tVkJOys/nlREU\nACBK0Yyg/OIH+2aVtAPbRkKS1LNnzxD36dNHUmYx2N133z3EQ4cOlSTdeeedoa3cdnjX12uvvRbi\nww47TJK0fPnyxOdecsklkjKX9ZejuXPnhtj+zTVp3bp1iB977DFJ0uabbx7aPv744xxdXenxv79J\nny0bIW266aZZv+epp54qKfP33S9TNyeddFKI/RaTWHz99dfFvoRq+YUlxvd5PjCCAgBEiQQFAIhS\nUaf4/O75PffcM8S2K9/vGj/++OMlZRYstRv5vr13796hbeONNw6x3Xz1N1lvuummENv+jIZ8XtT1\n118f4qT9Zj169AjxL37xi4JcU7FZFYKqWFHcDh06hDa//85P7Rm/v6+h8b9/SX1TF88995ykzKKz\nSXbYYYcQ57ogb129//77IfZ/D2OUNCXbuXPnvP5MRlAAgCiRoAAAUSrKONeG4pdeemlo84VfbX+I\nLw9j5V8aN24c2nxZIx9n64svvgjxZZddJilzL0JDkNQHUsVUjJXlkaTRo0cX7sIicc0114T4gAMO\nkFRRdkuq+KzWZg+YFS5G3dmZWlJFmSn/WU5i+ytj4vcR1nT9xeD3ks6fP7/S4/7vQz4wggIARKlg\nIyjbPS9V7BL3O8n9fpoHH3xQkrTLLruENn8iaT7873//y+v7x8YWovjTSpP4ShK77bZbPi8pSptt\ntlmIBw0alJP3tCM4kB3bSyZJ5557riRp4cKFoe3LL7+s9vUHH3ywpLqdxptvCxYsSGyvy4xQPvzu\nd78LsS3o2GOPPUKbn9HKh/j+jwEAIBIUACBSBZvi86VHkopEPvnkkyG2vSXeRhttlJ8La6CswOZ/\n/vOfxMd79eolSerXr1+hLqmk+RJan376aYitnJbf0+fPgTKHH354iHfeeed8XGIU/MnDt99+uyTp\n/vvvr/Y19913X4irO1G3adOmIbbTeqWKs+A22GCD2l1sEe23334F+Tlr164NsX0u/Tl5t912W6XX\n+IVkG264YR6vjhEUACBSJCgAQJQKNsU3ePDgENu0h69snDStlw9VnS3TECqbP/300yE+8cQTKz3u\nz+qx0lH5HsKXEltJ6svTjBgxQlLVx5fb5y1pBZkvqzV16tQQx7jarD4++OCDELdv3z7Er7/+ek5/\njv/8WjX+UuWnQqvjP4v+b5udP+ZXJ9tqx6uuuiq0+QrqtpfPn0Xlf//t85/vCuZeef0mAADKRl5H\nUP7EWr+XwW502o34QvLfTv0N13322afg11Io9m1s//33r/Z5ft9ZOZ6Omy3/rfLdd98NsX3794V0\nrSCxHw0deuihIb7lllskVZxG6vm9gf/85z9D/Otf/1qStP7669fp+mPmZyqynbWo6URd4xdG+JNe\nY9lTlMQXtPZ/j7p37y5JatWqVbWvf+KJJ0Ls+9OK4fpztWzhhT+t2PaISRX95H/3/efaqkoU8qRn\nRlAAgCiRoAAAUcrrFN+aNWtC7Nfb29lMfu9HPvgplKQisEcffXSIhw8fntdrKabLL79cUs0334cO\nHVqIy4mWTe298MILoS1pP4ovINuxY0dJUosWLULb6tWrQ/ziiy9Kkp566qlK77NkyZIQ9+/fP8S2\nD8r/7FjOL6qLbbfdNsR+oc4dd9whKfOmfLalc2644YYQX3jhhfW9xKIZNWpUiP1naM6cOVm9vmXL\nliG2qWGpYrreF9zOlt+X5j+jxSh1xggKABClonwts6WL/gZervhR06RJk0J8/vnnS5J23HHH0OYL\nIea76GGh+dM5fZWD7/Pf3At58zMWfkGEHdtgn5Xvs2+offv2DW32WfZHJRxxxBEhtgopTZo0CW12\nrIkfqfll5occcogk6ZhjjglttpxdSv692W677RKvOTa+KPTJJ59c5/exorFSaY+gPL/1I2kbSKH8\n4x//SGw/6aSTCnwljKAAAJEiQQEAolSUKb4TTjgh5+9pU1r+lF5/M9umsqxCQrnz+7rsBGOva9eu\nkqSJEycW7Jpi4ffVjB8/PsS2SMSfATVt2rQQW5/53fVvvfWWJOmUU04JbX7PX5s2bSRJt956a2iz\nm81+4dBvf/vbEN94442SpOnTp4c2K6zq+aKyixcvrvR4OXvuueeKfQkNTo8ePQr+MxlBAQCiRIIC\nAEQpr1N8VZU1sWmTP/zhD/V6fysjI1VMkaxYsSK0nXHGGSEeN25cvX5WqVm6dGmIk/Y/2XRWua1e\nzIZfpeT3ftnqOH/+0N577x3iV155RZI0efLk0GZFYv3eJz9taiv/Nt9880rX4Vf2+WO0bdqxZ8+e\noS1pajrGz7RfFTl//nxJ0k9+8pPQVt8zmR5++GFJxSmThsJjBAUAiFJeR1C++KGPrQCn30U9YMCA\nENtN6oULF4a2KVOmSKo4CVaS3nzzzRDbLuzevXuHNj+Cagh8EUi/ECCJ/8be0AwaNCix3fbQ+f1x\nK1euDPGCBQuqfE+/585/lutzdIYv5Onj2Lz66qshvuiii0Jsp7EuX748tGU7gvIj0nnz5oXYfr+T\niu/6wqscE5MbfubLFgQV8sRnRlAAgCiRoAAAUSrKPii7keqn+Hzxx6222kpSxU3Wqvhzd7p16yZJ\nOv3003N2naXC9oD5kkZ+asluxvuSMA35vCdf7soXw7TixnPnzk18XZ8+fSRJnTt3Dm32GWzatGlo\nK7cTcWvSr1+/ECcVxfWLOZIWiyTxC1XsdFgp81aBsf05vvxRMQqbliPf3zXdNsiHhvWbBAAoGXkd\nQfnlpZ06dQrxzJkzKz3Xn1zqC52a5s2bS5IGDhwY2uq7TL1c2A3jpH6TKkYMDf04DTNr1qwQ+xNJ\nbeTkj4c49thjQ2w33svxpNt8Gj16dM7ey47q8dVoRo4cKam0jyQpBbNnz5ZUccRMITCCAgBEiQQF\nAIhSXsfE/oaov4E/Y8YMSTXvUxozZkyIrRhns2bNcnmJaIB8BYf27dsnxsie7XeSMk+uvuKKK2r9\nXq1bt5aU+bfDn7hrfwf8NCzyx++DKgZGUACAKJGgAABRSlU3hEulUuliD/FKQSqVUjqdrrxBo/Lz\n8tKftorP9ulImftIWrVqJUlatGhRzn92PmTbn989l89oDQrZn1YuSpIeeOABSZlHu/uzyewI8e7d\nu4c2m2ZNOtY+FuX++fSrXP2evwsuuECSNHbs2Jz/zKr6lBEUACBKjKByoNgjqHJT7t9QC43+zC36\nM/cYQQEASgoJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABAlEhQA\nIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAA\nUSJBAQCiRIICAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCI\nEgkKABAlEhQAIEqNanpCKpUqxHU0GPRn7tGnuUV/5hb9WXepdDpd7GsAAKASpvgAAFEiQQEAokSC\nAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUaq21FEqlaLMRJbS6XSN9Uzoz+xl058S\nfZot+jO36M/cS+rTGmvxUQqpZrWptUV/1qy2tcvo0+rRn7lFf+ZeVX3KFB8AIEokKABAlEhQAIAo\nkaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUSJBAQCiRIICAESJ\nBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABAlEhQAIEok\nKABAlEhQAIAokaAAAFEiQQEAokSCAgBEqVGxLwBx+uijjyRJP//5z0PbunXrQvz6668X/JoANCyM\noAAAUSJBAQCixBQfgpEjR4Z48uTJkqRly5aFtr59+xb8mgA0XIygAABRSqXT6aofTKXS1T2Ob6VS\nKaXT6VQWz4umP1etWiVJ6tWrV2h78MEHQ5xKffvP2W+//ULbzJkzQ7zRRhvl7dqy7c/vnhtNn8aK\n/swt+jP3qupTRlAAgCiRoAAAUYp6kcQ333wT4rVr11b73OnTp0uqmLqSpEWLFoV4/PjxkqThw4eH\ntokTJ4bYpqwuv/zy0DZw4MC6XHa0bG+TJA0ZMkSS9NBDDyU+d+rUqZKkn/3sZ6Etn9N6QC59+eWX\nIe7WrZukzL17//3vf0PctGnTwl0YaoURFAAgSiQoAECUijLFt3LlSknS119/Hdr8kNumnT755JPQ\ndu2119b65+y4444hPvfccyVJN9xwQ2jbYostQnzwwQdLkjp06FDrn1MqPv300xDffPPN1T7X+m63\n3XbL5yUBtfLZZ59l/NfbZJNNQvzss8+GeM6cOZKktm3bhjamq0sDIygAQJQKNoJ69913Q9yuXTtJ\n0ooVK3L+c9ZbryLn+tGSfWMaMGBAaGvevHmIN910U0nS1ltvnfNrKjZbHHHooYeGtqS9GU899VSI\n99lnn/xfWAPw17/+VZK0Zs2a0DZ//nxJ0pVXXpn4mj333FOS9Mwzz+T56uLwwQcfhNj65M0330x8\nro2MkooV+wVO1sdSxWe9ZcuWoc0vwCpnvh+nTZsmSXrggQdC29NPP13pNX/5y19CvP3224f44Ycf\nliT169cvtPlZqnxgBAUAiBIJCgAQpYJN8TVr1izE22yzjaS6TfF16dIl8T3vvvtuSVKTJk1CW/v2\n7Wv9/uXolltukZQ5LdKnTx9JmXvBNttss8JeWBlYvHixpMw9d75k1PXXXy8peUrVykl934svvihJ\n2muvvULbc889V/+LjdTcuXND/Kc//ana52644YaSpDPPPDO02e++LYT6PuvnwYMHh7ZyXyRhfXrM\nMceEtg8//FBS5mexR48eIX7nnXckVfxt+D57nS8gffXVV+foipMxggIARKlgIyj/jcVu1t15552h\n7YADDghxz549K73+oIMOkiT9/e9/D22NGzcO8ZIlSyRJEyZMyM0Flzi/IOKxxx6TJO26666h7Yor\nrpDEqCnJ559/HuITTjghxH4rhLFZAL/s2X9DtVH8o48+mvXPtxv4th2jXF1zzTWSpPPPP7/SY+ec\nc06IbcZFkgYNGiRJ2njjjUObjZx81RMbLUjSD3/4Q0mZp0OXC7/Ywy+IOPzwwyVlfpZ/9atfSZLG\njBkT2vzCEdv2c9JJJ4W2W2+9tdLPPPDAA+t51dljBAUAiBIJCgAQpaJUkrCh+B577BHa/HSdDfn9\nDdPRo0dXep5nw/iLL744txdbQvy+GV8E1m4Sn3zyyaFtgw02KNyFlQhb6GBTIZL0xhtv1Pp9bLpZ\nqthf56daPv74Y0nSEUccEdqS9v3sv//+tf7ZpcT65Isvvghtu+yyiyTpwgsvDG3Wh97y5ctDbFNW\nvt99VYlJkyZJkho1iro2dp088sgjIe7atWulx4899tgQ33jjjZIyF5J5jz/+uKTkaT2pYs/TUUcd\nVadrrQtGUACAKJGgAABRKuqYt6qh5pZbblmpzUqgWFFXqep9JA2NldGZNWtWtc/7wQ9+EOLNN988\nq/e+4447Qpw03TV06NCs3qcUjBo1SlLN03q2F0eSZsyYIUnae++9Q1tSuSy/ivWqq66SVHU5H1tt\ned1112Vx1aXL9uj4z5jt9xoxYkRou+SSS0Js58L5VX433XSTpMx+96t5jzzyyFxedhTs7+HZZ58d\n2vzfQ+s///tZ1d9bc9ZZZ1X7+G233SYpcwVlvjGCAgBEKcq7hpbJ582bF9ruueceSdLChQtD209/\n+tPCXlik7JuT7y+/P8IK6PrRZxKrOOHf09+sfu211yq9ZtiwYSH2x3mUyv6qBQsWhNgX0UzSokUL\nSdL9999fqa023n777Wof79u3r6TCflMthu22206S1LFjx9BmIyirDiFJxx13XIiPP/54ScnFYm1f\nlZS8l7LUTZ48OcQ2cvKjot69e4f4ggsukJS8GGrdunUh9nv7Xn31VUmZ+/h8QeNiFJBmBAUAiBIJ\nCgAQpSin+Gyvkz9F1xYA+Buefr+KlTHxa/QbyiIK27/jy0D5c7FsGippYcR7770XYr/IwspReX7a\nbuedd5aUOUXQq1evENsNVX9qcYzGjh0bYr9XyVjJGKniZn1tpvVsAYuffr333nur/TnleFM/ie1L\natq0aaXHrHCplLkfzKaf/O+27Zvs3LlzXq6zmPw5YrYXVKr49/tpPdvnVBXbO+b3Rvl9VObUU08N\n8SmnnFLLK84tRlAAgCiRoAAAUYpyis9stdVWIbYzdrp16xbaxo8fXyn2w1y/kiepXEops/0gUvK+\nHX9U8xlnnCEp8/wsOwb+0ksvDW1Tp04NsVWQ9tN25513XoitPM3uu+8e2pYuXVrLf0Xx+b0f77//\nvqTM/TR+qrMunyE78v03v/lNpcd89W1/zHa5fVZrYuWNasOfWWTVzLPd21dKrMK4lFmh3YwbNy7E\nq1atCrGdFGFT7ZL0xBNPSMpcbeunSpNKolVVWq5QGEEBAKIU9QjK23fffSVl7oPyu6htN7o/y8Tv\nlbBv/6WyP6cmL7/8coj9TU/j9yeddtppkjK/YQ0ZMkSSdPPNN4c2v6DBvvH//ve/D2026vI/07+m\ne/fuie8Vs/322y/EtTmzqTr+9NvTTz+90uO2N8X/P2pooyapYq/eww8/HNqSTh727Hyu6dOn5+/C\nIrL++uuH2ApiSxWFcf0sU02LwnbYYQdJmYtS/GIUmzXxJzkXGyMoAECUSFAAgCiVzBSf2XbbbUPs\nb2DbNFanTp1Cm9/j8sorr0jKvGlYyl544YVqH7f+8PyCB39elHnyySdDbAVL/QIMf2S88X1cToVj\n68MvfkiadrnrrrskSYcddljBrilGAwcOlCRdf/31oa2maaqGsrfR+MLEdl6TVLE3bNmyZaGtdevW\nIbapUCubJVWckWWPSZlTfPb/IyaMoAAAUSq5EZTnv120b99eUuZNRV8U8W9/+5ukipGUJLVq1SrP\nV5g/diqrVHFjuX///onPtWoRfoGJvcaWQUuZIyRbEHHooYdWeo1/XdICjYbIL/dNKtTr+RFWQ/DZ\nZ5+F2M9g2HEiflR0yCGHSMrsoz//+c8htq0ADZGdaCtlnh6cLSsGa38LpczP52677Vb3i8sTRlAA\ngCiRoAAAUSq5KT4/xPdnxtguaT+t59mUQdKN/lJnUyQ13UD2w3l77jPPPBPa7AwZSVq9erWkzDO3\n/HNrOp2zobCd/r5vkvrZdvZLmScbNwTPPvtsiH0hUuNPDrbznuz3Wcqc4mvbtm0+LrFBsMKzSZ9P\nKXM6PxaMoAAAUSJBAQCiFPUUn1/jf/XVV0vKLGj67rvvVvt6v6LPVsCUyz4KfxaWnYfj+8ZP19nq\nvZUrV1Z6H7/6zK/Ss7Inl112WWgrlzJR9fXVV1+F2Mr0VLW/zkod+SLH5fIZrImtmK3q+HWb+mvT\npk1oszO5Bg8enPia2pzFhUy+n0sFIygAQJSiGUH500zvu+8+SdKoUaNC2+LFi7N6nw4dOoTYTkCV\npL333ru+lxgVKzgqVRQa9X3YsmXLEGf7jT2pWGy7du3qdZ3lxI44Oeecc0LblClTKj3Pj6Zs9NBQ\nRk3ev/71L0nSihUrQps/8XrPPfeUlHmkxOzZsyVVnP4qZY7sfSUZ1M78+fOLfQm1xggKABAlEhQA\nIEpFmeKzc4l8oUJ/Qubzzz+f1ft06dIlxCNHjpRUc6HOcuFPzJ0zZ46kzMKtfo9YEpum8lOfNuUi\nled+sfqyRSZJ03q+UOfRRx9dsGuKme23STq1VaqY2ps3b15os4LGfq+YL0J85JFH5udiG4Ckk7dj\nxwgKABClvI6grBqBJJ111lkhtrLx/lTYmtjRBCNGjAht/ga+XzTQ0Fg/2KnCyB2/1eGKK66o9Pge\ne+whSXrkkUcKdk2l4sMPP6zU1rx58xDbSPPee++t9DxbYCHFdcJrKbNTyWsqZhyTuK8OANBgkaAA\nAFHK2RTfm2++GeI//vGPkqSZM2eGtrfeeivr99p4440lSaNHjw5tgwYNkiQ1bty4PpcJ1Ir/DF5z\nzTWVHr/wwgslZe4hw7ds+tPzC0xsf9PWW28d2mwKvxSrHsTO9pD5AtAvvfRSiG1KdqeddirshVWD\nERQAIEokKABAlHI2xXfXXXeF+IYbbqj2ubYq57jjjqu4kEYVl2JldvyR7kCh+OO0kwrsDh8+PMQH\nHnhgQa6pFNmeJV/E2IrnSlLnzp0lVex9kqTevXsX6OoarvHjx4e4a9euIbai0xMnTgxtVjS6WBhB\nAQCilPKFGCs9mEqlq3sc30qlUkqn0zWWraA/s5Ntf3733Jz3qT/BddiwYSG2AryPPfZYaPM3+GNV\n7P4sN6Xen1b0WJL69+8f4ttvv12SdMopp4S2CRMmhDifC9Sq6lNGUACAKJGgAABRYoovB5jiy61i\nT6H4vSF+P86TTz4pSdpnn31y+vPyrdj9WW7KqT/9dJ+dn+f3/r333nshzueCCab4AAAlhQQFAIgS\nU3w5wBRfbpXTFEoM6M/coj9zjyk+AEBJIUEBAKJEggIARIkEBQCIEgkKABAlEhQAIEokKABAlEhQ\nAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUSJBAQCiRIIC\nAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABClRjU9\nIZVKFeI6Ggz6M/fo09yiP3OL/qy7VDqdLvY1AABQCVN8AIAokaAAAFEiQQEAokSCAgBEiQQFAIjS\n/wHTK4KPiGYqugAAAABJRU5ErkJggg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True,)\n", + "ax = ax.flatten()\n", + "for i in range(10):\n", + " img = X_train[y_train == i][0].reshape(28, 28)\n", + " ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n", + "\n", + "ax[0].set_xticks([])\n", + "ax[0].set_yticks([])\n", + "plt.tight_layout()\n", + "# plt.savefig('./figures/mnist_all.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "Visualize 25 different versions of \"7\":" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmAzdX/x/HnHdmy7/TNnqVEhG8bX9VXG0lUtiJfpZQl\n9YtUKipU0kZUIimUIkskCtW3JHvlK3xR9qUsgxnDmPn98fmec2fMdmfc5dyZ1+MfM5/7udfb2733\nfM75nPM+vuTkZERERFwTE+kARERE0qMGSkREnKQGSkREnKQGSkREnKQGSkREnHROZg/6fD5N8QtQ\ncnKyL6tzlM/ABZJPUE4DpXwGl/IZfOnlNNMG6n9PCk00uYjPF9B7FVA+A5GdfIJymhXlM7iUz+DL\nKKca4hMRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESdl\nWUnCVVOnTgXgxIkT/PLLLwC88cYb9vFGjRoBsHLlyvAHJ7lCYmIicXFx9veff/4Z8FcGWL9+fabP\nL1CgALfffjsARYsWJSZG14PZYfJ87NgxABYsWMDkyZMBWLdunf3clyhRIjIBOsTkaubMmZk+7vP5\n2Lt3LwD9+vUD4OOPP7aVHFq2bAm4k1NfZmU4fD5fsgtlOjZt2gTAf/7zH7788ksA3n33XSDjMiLm\ny+Diiy8GYPXq1SGLz+fzBVyLz4V8ZiYpKYlDhw6lOW7esOecE/prmkDz+b9zQ5bT3r17M27cuKC8\nVvfu3XnzzTcBKFy4cFBeM1Cu5DMQsbGxACxevJgJEyYAMG/evDTnFSlShH379gFw7rnnhi9A3Mln\nXFwcy5cvB+Chhx4CvO/I9KRsoNJ7zByvXLky4DVeXbp0AaBChQrBDTwdGeVUl3QiIuIkp3pQx44d\no2vXroDXhTfMFf3Ro0ftlcDVV18NwDfffJPpa1atWhWALVu2BDtcK1w9qFWrVgHQuHHjHL8GQEJC\nAgBr1qxhzJgxAJw8edL+OWfOnDTPefvttwHo2bPnWf3dgXDlCvWee+5h69atQM56PYmJiXz11Vf2\nd9OLb9iwYXACDJAr+czI7t27ARg+fLjtNSUkJFCrVi0A7rzzTsDL5/PPPw9Ajx497ChKuLmSzx49\netghz6wE2oNKqWbNmoD3OQDo378/BQoUyGm4mcoop040UKZbeuutt9ovhIzs2bMH8Mb0wWvU/vrr\nLwBuvvlmAH7//Xd7fseOHQGYMmVKUGNOKVwN1KlTpwDInz9/wM85fvw44G/cfv75Z/ulmbIhatGi\nBQCdO3fmk08+AeDrr7+2j5svaPN6oeTKF8CpU6fsUHG+fPmy/fykpCTat28PeLl+4YUXABg4cGDw\nggyAK/lM6bfffuOWW24BYNeuXQDEx8fz+OOPA96QaLVq1QD/+33Xrl12CGr06NH07t075HGmx5V8\nNmjQIN0hPXMvvmLFigG/Vt++fQHssCmkbdTWrFlD/fr1cxxvZjTEJyIiUcWJWXzPPvssQLq9p0KF\nCtlubOPGjSlXrlyqxwsXLszo0aOB1D2n2rVrAzB+/PhQhBwR2ek5Gc2bNwdg7dq1QOrufHJyMvfe\ney8Ar732GuDdfK5evTrg70ElJyfz8MMPn13wUSgn+U7pzOFS05sXb9jevDfNaMhdd91FkyZNgKz3\nXCpSpEhoA4wCixYtYuTIkYD/8wvY70szoSyQGXmmp29Go8IxMSIQ6kGJiIiTItqD+vXXXwFvfcOZ\nzA26+fPn258zsn379jTHunXrBoR/CqprzD0Uk4fmzZszYsQIAKpXr07x4sUB/xXrBx98wAMPPJDq\nNerWrcugQYPCFbLkAVdccQVXXHFFtp7z2GOP2Z87deoU7JCiToUKFXjxxRcB7L2hcePG2bWf06dP\nB6BDhw5nta7J3MsqWbLk2YSbIxFtoIYNGwb4F+IBtG7dGsDeUM6ocTpx4gQAP/30U5pZZ61bt6Zt\n27ZBjzcaTZw4EYDzzz8fgNKlS6d7nplE0atXL+Lj4wHsLKply5bZYRgJ3A8//BDpEHKVlEP44jGT\nd+6++27Am2h21113Af5JUB06dAj49dKbGdm5c2fAv0YqnDTEJyIiTopoD6p///6Afx1EuXLlmDRp\nEkCWV+ym1NF9991njzVt2hTwppTrit/ToEGDTB//4IMPAOyU3fj4eDtJwvQAItG1zw0OHDhgf65R\no0aaG89r1661a9LSc9lll4Ustmh1zTXXAIRsPU60K1GiBHPnzs3288z0cvP9C15vDOC5554LSmw5\noR6UiIg4yYmFutlhVuM3a9YM8KbymunAH330EUDY7z9Fay2+1atX2xvVZhFw3bp1mTZtGgCXXHJJ\nROJyZSFkduzfv99W4zh69CgAN954Izt27AC8ySrFihVL9ZzY2Fi7GLJ48eK2OoqpnHDHHXcEJbZo\nzGdKpj5fhQoV6N69O0DQaiTmRLTnMz1mkkXKhb/ff/89AJdffnnI//6McurEOqjsMMN4KddJzJgx\nA4BWrVpFJKZoY9ZEXXvttbZhqlGjBgDfffcdZcqUiVhsrjp58qT98G7cuBHwipiatXurV6+2E3cy\nYt6zJtft2rWzwyiVKlXKcAJLXrd48WLAK3/0yCOPRDia3GfJkiW2FJx5jzZp0iQsDVNWNMQnIiJO\niqoe1KuvvkpSUhJAqr11TK9KsrZ27Vqbr9OnT3PhhRcC/u68JkSkZvaDuuCCC+w+Olkxk0y2bdtm\njy1btkzv0xwyFU1iYmIoX758hKPJPcx7e+DAgXZ4ukqVKoC/CkWkqQclIiJOiooe1OnTpwFvd1zT\nczJjpZ9++illy5aNWGzRwkwuufbaa20+AZYuXQqo55QRsxCyTJkytgf18ssvA94KezMhJ2WP3lyN\nNmjQwE6SMBtnSvaZZShXXnmlMzu95gavvPIK4H03mO9Ts/THlTw730CdOnWKRYsWAd7WxEafPn0A\nb6ZUVoUl8zIzA8qsc4qNjaVSpUqAtz20GvfMFSxYEPDWi5lcXnnllUDGuwubrUkqV65sGygRV5jh\nu2eeeQbwtoUxxbVdKyGlIT4REXGSsz0os8L+kUcesbu5gr8XddtttwFZl+XPy+Li4uzuraaOWY0a\nNZg/fz7g35JEshbuXXDF/x1gikm3adMmkuHkCnFxcXaI2nx3xsTE2F6Va5NQ1IMSEREnOduDOnLk\nCECq3tNFF13E7bffHqmQos7OnTttz8ncL1m4cKFdKCqhVbZsWaKhioCrli9fDmCr6w8cODCS4eQK\ny5cvZ8mSJamOde/enb/97W8RiihzzjVQpsCmmWEC/oKnZyZWMle0aFG786jZzVWNU/h07tw5zVYw\nErj3338/1e+u7PIazR566KE0xwYNGnTWu0eHiob4RETESc71oExp97Fjx9pjZjqkK3Pzo8V5553H\nzp07AShUqFCEoxHJGbNGz+z+LNl3//33A7B+/Xp7zGxVtG7dOruEonHjxuEPLhNOVTPfu3ev3dZ5\nypQpADzxxBP07dsX8PaLclG0VjN3VW6pFh0fH28XmVavXj3VYt5witZ8NmrUCPCX5DFFeiMtGvNp\nFpynN+s5OTnZFoutVq1aOMOyMsqphvhERMRJTg3xffjhh7bnVKtWLQD69u3rbM9JJDOFCxemZs2a\nkQ4jKk2dOtVuC/Pkk09GOJrcbeDAgc7O4lMPSkREnORUD6p169YMGjQI8Gqfgbv3nUQkdPbt22d/\n7tKlSwQjyb3MxIkRI0ZEOJKMOTVJIlppkkRwReNNaJcpn8GlfAafJkmIiEhUyXKIT8VYg0v5DD7l\nNLiUz+BSPnMu0yE+ERGRSNEQn4iIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiI\nOEkNlIiIOEkNlIiIOCnTUkc+n09lJgIUaLHYcMSSG2SnGGeoY8kNlM/gUj6DL72cZlmLT6WQspad\nWlvKZ9ayW7tMOc2c8hlcymfwZZRTDfGJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiT\n1ECJiIiT1ECJiIiTslyoK3nP4cOHAZgyZQoA/fr145ZbbgFg2rRpABQqVCgywYVQXFwcAM2aNQNg\n3bp1PP7449l+neHDhwPe4kOzSPPee++lfPnyAHTs2BGA+vXrn3XMItmRmJjI5s2bAZg9ezYAX3zx\nBd9++22q85544gmGDRsW9vjOpB6UiIg4yZdZGQ6fz5fsUpmO06dPc/LkSQDeeuste3zPnj0AjBw5\n0h5r2LBhqvMuu+yykMX1vyvlgGrxuZTPuLg4li9fDsBDDz0EeGVZjh07BsCOHTvsuSbuhx9+GICX\nX345ZHEFms//nRu0nJory86dOwOwdu1a++9O2RsyZVmSk5PT/dm897J6/vfffx/S96URqXwGy8SJ\nE+3Pjz76KABHjhzhoosuAvzvSYBWrVoBULFixZDFEy35NN+L8fHxLF26FIAFCxYwY8aMVOelfO+m\ndPr06ZDHaGSUU+eH+E6dOsXo0aMBWLRoEQsWLADSr92U8ti6desA7H9GOL4IXGbebIcPH7aN0b59\n+1iyZEmq8zJ6sxp33nln6IKMsFq1agGwcuVKAPbv329/Tk+ZMmWy9b7avn07ANWrVwfgr7/+ymmo\nudKJEyfs53v69On25yNHjgCpP98+n48NGzYAcN9999njderUAWD9+vVhidk1n3zyiW3Qf/zxRwBi\nY2PTXBwBNGjQwP754Ycfpnqd3r17hyPcLGmIT0REnORcDyo2NhbwD7cMGTKEefPmpTkvX758ANSt\nW9ceM0NTf/zxR6jDjBrmyumDDz4AvJv16V1NBapq1arBC85x5cuXt0NGwTBr1iwgZ3nPjdauXQvA\nqlWrABg6dCi7d+8+q9fcu3cvANu2bbM91bykZ8+eHD16NM3xdu3aAdC+fXvatGkDQIECBQD49ddf\n0/SgnnrqqRBHGhj1oERExElO9KBOnToFwBtvvGEnOuzfvz/dcwsXLgx4N5gBGjVqZJ9vbgTecMMN\n9vxKlSqFJGaXbdq0CYBffvmFfv36Ad79pvR0794d8N8X+de//kXTpk3TPGfo0KEAlC5dOiQx5wXm\n3l9MTN68Lly9ejXLli0D4L333mPjxo2AdxM/u5o2bcqKFSvSHC9ZsiRAnuw9ASxfvpzVq1enOta+\nfXsKFiyY4XMWLVpkR1WuvPJKAIoXLx66ILMhog2UaVgGDx4MpJ6FZ1SvXt3evL7++utt41OvXj17\nzs8//wykbpjMup0+ffqEIHI3mYkhLVu2BODQoUNpzmnSpAm1a9cGvHU4/fv3ByB//vwAnHfeeWku\nDpo0acIjjzwSsrjzgv3799uGyQzxXX311RGMKPxatmxph/Cz0rBhQ/u5f+GFF9I8vn37dq655po0\nx10ZmoqUOnXq2IkiWTFDgdOmTbPvyZtuugkg0wYtnPLmpZyIiDjPiSG+UqVKAd4U5vvvvx/wr2Mo\nW7YsxYoVA/wTI1LatGkTrVu3TnWsb9++tldmegZ5gbkKMlfq9evX5/XXXwf8Oa5SpQolSpRI81xT\nRWHfvn32dcxV1Isvvsi5554b2uBzueHDh9thlNdeew0gT+a0QoUKgDe1+bHHHgPgs88+A6BFixZ2\nbdN5552X6TBTekPWNWvW5Pbbbw92yLnWSy+9BHiTJMz/S69evSIZUhrqQYmIiJOiqpJESub+VefO\nnZk5cybg9ZzAu5dlplCGg2uVJMwUfTOGnxnTc7rxxhsBb/KJmSTx4osvAt6VbThFy0r9QJj7eRdc\ncAHHjx8H4PfffwegcuXKYYnBlXyuWbOGsmXLAmf/b7///vuZMGEC4O+JTpo0ifbt259dkAFwJZ9n\nq0aNGoD3fjSf//nz50cklqitJHEmU+rIFNrcvHkzjRs3Bvzld/LSsF56AmmYDLMK38yKvPzyy1m0\naBGQN4eggm3x4sUAHD9+3M4sK1OmTCRDiphGjRqd9Ws8//zzAKnK9ZhyZuFonHKDXbt2AXDw4EHA\nuyVgChy7RkN8IiLipKjrQW3duhXwD2OVKlWKjz76CFDPKbvmzJnD559/DvgnWCxatEg9pyDZv3+/\nnZ7v8/mYOnUqoJ5pdiUmJgLw22+/MWrUKMCr22dGTG6++eaIxRaNXnnlFcA/zfzaa6+1BY5dE1UN\n1JYtW2yBw6JFiwJeYc+8uigvp0wB1G7dutnyUGbWpL48g2fOnDm29E7FihX5+9//HuGIotPkyZOB\n1EVhe/ToYdfwSeD27NmTqjo8eLl0lYb4RETESVHRgzIlUVLuQPrrr78C3roeyZ4RI0YAXnFd03P6\n6aefIhlSrmJmRr700kt26NSsR5PAmc/9PffcA3g3800v1Kwlk+yJj48PuJqHC9SDEhERJznfgzp1\n6pStuZWYmGgLbqrnlD0nTpwAvGodZt1Y7dq1+e677wBvawkJDjNld8uWLfb+6JnVTiRzCQkJXHrp\npYC/MsoVV1zBwoULAd0rza6kpCTA277IrMsylTrMDtIucraBMns69enTx95o7tq1K6+++mokw4pa\nZofR2bNn2w/8vffeq4YpBMxNaJ/PZ/fe0RdqYMwC/L59+5KQkABgh/WGDRumPObQjh07AJgyZYod\ndn7zzTcjGVJANMQnIiJOcrYHZbbemDdvnr2RP2zYsEiGFJVMQc6UO2aaPaA0TTe4TJkY0+P3+Xw8\n/vjjkQwpqiQkJNh1YymnQpvtNsJdcis3mTZtWppjpnfvMucaqOXLlwNeVxS8qsY//PADAOeff37E\n4opGmzZtsh/0lHtDDRo0CNDC5mAz71kzhKIh1MCYBv22226zn/9ixYrZLeGrVq0asdhyo27dugH+\ntaQu0xCfiIg4yakeVHx8vN0B1+wB9d1332nGXjaZWToXXnihPWZ2IDa7D0twnTx50q7bMfm/+OKL\nIxmS88zMUjN0v3z5cpo3bw7AO++8o55TEH3xxReA9958+umnAf/sSJe5H6GIiORJTvSgzJXU+++/\nz6pVqwAYPXo0oPHnnBg/fjzg3QsxE0zM/REJjX379tl7Jibn3bp1s/tB6V5UagkJCQwYMADwPvcA\nbdu25b333gPIdDddyZ6jR4/a9Y4+n48DBw4AREUNUyc2LNy5cyfgLb4dOnQogJ39dM45TrShmXJl\nw0JTndisG9m8eTMrVqwAgrMXT7hE44ZwO3bsoFq1aoB/iO/SSy+lTp06QGQvEFzKp1nn1L9/f2bP\nng3AddddB2AbJ9e5lM9AHDt2zDb4Pp+PcuXKAd5CcoAiRYpELDYjo5xqiE9ERJwU0e6JGdrr1asX\n4PWgzCSJaOg5ucbsNmz2ymrXrl2qiRISOsWLF7dDeynXQZnhVoHTp0/Tu3dvAD755BM7Yedst3+X\nzBUtWtSuIdu2bRtjxowBoHDhwpEMKyARHeJ78sknAf8Y9MqVK+2HPJq4MsSXW0TbEIrrXMnnv/71\nL7Zv3w7A559/HhVfkOlxJZ+5iYb4REQkqmTZgwpjLFEt0B5UOGLJDbJzhRrqWHID5TO4lM/gSy+n\nmTZQIiIikaIhPhERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIa\nKBERcVKmJcNVpiNwKnUUXColE1zKZ3Apn8GXXk6z3NNCpZCy5vMF9F4FlM9AZCefoJxmRfkMLuUz\n+DLKqYb4RETESWqgRETESWqgRETESWqgRETESWqgRETESVnO4hMRCYVJkyYBcM8999hjw4YNS3Ne\njx49AChfvnxY4hJ3qAclIiJOynTLd5/Pl6w5/Fnz+XwBL9RVPrMWaD7/d65ymgVX87lt2zYAZs+e\nbY/NmTMHgG+//TbN+XPnzuWmm24KS2yZcTWf0SyjnKoHJSIiTsqVPaikpCSSkpJSHYuJiSEmJjTt\ncTT3oEw8J0+eBGDFihX2KnbkyJEAPPTQQ9x2220ANG/ePOQx6Qo1uKIpn4mJiQDs2LGDF154AYAJ\nEyYAUKBAAT7++GMA2rRpE5kAia58ZtdPP/0EwKOPPsp3330H+O8Bmv+HUMgop1HbQJkG6NSpU0yb\nNg2AP//8E4CVK1cyffr0VOePHDmS//u//wtJLNHQQG3evBnA5sowDdPw4cMzfX69evUAWL58OQDn\nnntusEO0ouUL4M8//+Srr75Kc9zE07dvXw4dOpTmcfPebd68uc17s2bNQhZntOTzTKdOnQJg/Pjx\ngJfPQoUKATBjxgxuvPHGiMQVrfk807FjxwCYOXMm7777LgA//PADQKoL/BIlSgCk+14OFg3xiYhI\nVImKHtTp06cB2LlzJx9++CEAmzZtAuCDDz5Ic35ycnKa4oO33HILs2bNCkl8rvWgzJXRzp07bbd8\n8uTJABw4cCDVuSYek6/8+fNTu3ZtAH7//XcAjh8/bs83vdTSpUuHKHo3r1C3bt3Kvn37AFi4cCEA\nb731Vpp8QtqcZvZ4wYIFAfj+++9p2LBh0OM2f49r+cyJdu3a2eHnXr168eabb0YkjmjM5+HDhwEY\nPXo0o0aNAvzDqXFxcZk+t1OnTgBMnTo1ZPFllFOn1kElJSXZZJnu5LvvvsvOnTsBeO+993L82t26\ndTv7AB1nZkP1798fgD/++CPT8wcPHmy/IP/5z38C3tDdhRdeCHhfCADz58+3wylFixYNfuCO2rBh\ngx0WXrNmjW2MsmqAzEyzjB6fP3++/TkhIQGAEydOBCfoXGzEiBHMnTsXgC+//JL4+HgAChcuHMmw\nnGXuIS1ZsoRXX30VgCNHjgT8/HLlygH+i9tI0BCfiIg4yYke1Ndffw3Ap59+yttvv53puebmfNu2\nbYHUPaMff/wRgOeee87e5GvVqhUAt956a3CDdpDpaabsOV1xxRUAlC1bFoBLL73U9owuvvjidGc2\nmtdJeaXfokULwJtJlVccPnzYDuelVK1aNQDy5cvH0KFDAahevbp9/PLLL0/39UwvqUiRIvZYgwYN\nALjooouCEnNuVrVqVZv733//3b7P69atG8Go3PLnn38yduxYAJ5//nnAP5R3pgoVKgDQunVrJk6c\nmObxl19+GYBzzolcM6EelIiIOCmsTWNSUpKttXXw4EF73FylbtiwIc1zChcubNc89OrVi4oVKwLp\nXzWZK6qCBQva8ek77rgDyP4umNHovvvuA+CGG26wxypXrgxg7zUFwvSgjIoVK9rXzktq1apFrVq1\nAG/dTePGjQHo2LFjtl/rxIkTtjefkrlfWLx48bOING8oXLgw119/PQDvvPOOnTChHpR/8tJVV11l\nl5SkVKVKFcAbSapRowYAHTp0ALzp+2fq1KmTfTySwtpALVu2jJkzZwKwbt26dM8xQ0mme1qqVKks\nhz92794NYGfpxcfHc9dddwFw3XXXAXmjgcqfPz8AF1xwwVm9zvvvv5/q988++4xSpUqd1WtGo7Jl\ny6Z70ZQdR48eBbyJE8uWLQP8a0wefPDBPDF5J1iSkpLsjNLk5GQ7sUdg7969gHdxab77zISmevXq\n0bVrV8C7EDIT0cxa0RkzZtjX6dy5M+Atys3ORW2oaIhPREScFNZ1UL1792bcuHEADBgwgNtvvz3N\nOXXq1AGyN+Tx1ltvAV55DvDW7WzZsgXAdmdDybV1UGdj1KhR9sa/+T9Yt24dZcqUCVsM0bjOJCNm\nWrSZmAJQv359wJv+W7JkyZDHkFvyGRsba9ff3XjjjXz22WeAf+QgXFzO5/79++1nNV++fOmes2DB\nAoBUQ85mKNuMbJmKHeGiShIiIhJVwnoPasyYMfTs2RPwpuoG4+px79699r6WGVv95ptvqFq16lm/\ndl5iavJ9+eWXduza1OUKZ+8pN9mwYQN33323/d1U6Fi6dCngr3EmgTH3msGbch7unlM0yGpTx2+/\n/dbeZzLuvPNOWxg63D2nrIS1gfL5fEEv5/Lqq6/aL1JTafuyyy7LsHsrqZmGyTRCx48f5+mnnwb8\nM38ke2JjYwGvAK/5+aKLLmLJkiWAGqacMmv6wPuMS+DMrOYBAwbYahLmfTh06FA7O9o1GuITEREn\nOVFJIidMCf6xY8faoT2zf0wkVz5HG1OI1xSYbdy4MUOGDIlgRNHPTMlPubThnXfe0VBpDpnC0IcP\nH7ZVOtq3bx/JkKJKfHy8vbWyYsUK/v73vwP+ShHhmEiWU+pBiYiIk6Kuq2Gmj48ePRrw7pmYzbaa\nNm0KZDy9UlI7efKknWpqcvb6669HMqSotWbNGru43CzEbdiwIU8++SSQcX0+ydrgwYMBr5KE2dEg\nL1XVP1sLFy60W2WUKFGCTz75BPBXmXFZ1DVQplTSr7/+ao+ZyREa2sue1157zc4oM2V8rrrqqghG\nFH3MzeennnrKri8xBXj79u2b7lo/CYxZQzZv3jzA2/lVZY0CZ3a/7t69uz02ffr0qGiYDA3xiYiI\nk6JiR13jkUce4Z133gH8K59nzJhhp0NHqgcVbZUk9uzZA3g5NMNRpkBsKHfKDZTLK/VT2rBhAz16\n9AC8m8+GKdZZpUoVJ4aboyWf4N/AsWvXrrbnZIZHzbY8keZ6Pk3PKWXRaLPd0Lhx45zc4FGVJERE\nJKpExU0bs43G22+/bcf8zVh0lSpVdO8pm8xkkri4OAYMGAC40XOKFikX4pqe00UXXWR79yk3L5Ts\nufHGGwGv4sHNN98M+KtuS9Y2btxod3AwS0e6d++e7oaE0SAqvtnNflGmcQLsWh01Ttnz6aefcuDA\nAcDbi8jMMpPAZbTOSTP1csZ8rp955hm+/fZbwJttZnbXdmHbh2gxfPhw2zBVqlQJSD1JItpoiE9E\nRJzkfPdj+/btPPHEE/b3sWPHAme/KV9eY6bld+3alVOnTgHQtm1b7eQaoISEBLp06QKkXuekwq85\nt3XrVgC7LueVV16xj40fP97Z+nAuMttkzJw50xZ8/eWXX4DoHr53voFaunQpf/31l/3dzEBxYXZU\nNDFDogkJCUyZMgWAf/zjHxGMKLoMGTKE2bNnA6nXOalhypk1a9bY3bNNqTKAn376CYAGDRpEJK5o\nY0qVtWyEYHr+AAAWO0lEQVTZEvAW3y9evBiI7obJ0BCfiIg4yfke1JmuvPLKSIcQlfbu3Wt/btOm\nDZD6Jr+kz6zLMcNR4O06DKTa60myp1GjRnY2pORMUlIS3bp1A/wz9pYuXZpqW5Jopx6UiIg4yflK\nEsnJyfamNPjH/126+o+GShLNmjUDvF1yzZWrqwU3XVqpb3pOZjdcgMTExJD9faHgUj5zA1fyOXny\nZDuF/Pjx4wBOVokIREY5db6BigbR0EBFE1e+AEANlKSlfAafSh2JiEhUyXKShEtDabmB8hl84c5p\nbv8/zO3/vnBTPnMu0yE+ERGRSNEQn4iIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkN\nlIiIOEkNlIiIOEkNlIiIOCnTUkc+n09lJgIUaLHYcMSSG2SnGGeoY8kNlM/gUj6DL72cZlmLT6WQ\nspadWlvKZ9ayW7tMOc2c8hlcymfwZZRTDfGJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiToqqB\n2rVrFz179qRnz57ExMQQExNDx44dOXr0KEePHo10eCIiEkRR1UCJiEjekemW7z6fL9mFOfyzZs0C\noH379pQrVw6Aq6++GoC1a9dy8OBBADZs2ABA2bJlwxqfz+cLeKGuC/l0XaD5/N+5YclpbGwspUqV\nAiApKQmAmJgYnn32WQBq1qxpzzXxNGvWjO+//94ev+qqqwCoXLlyyONNycV8RjPlM/gyyqnzDdRv\nv/1Go0aNAChUqBDLli0DoG7dugDs27eP8847D4AFCxYAcN1114U1RlcaqO3btwNQvXp1AE6fPh2y\nvyuUXPwCiI+Ptw3MunXrzN+d7rkmntKlS3Po0CF73DRws2fPBqBp06bkz58/ZDEbLuYzmuWFfCYk\nJPD++++nOpYvXz7uueeekPx9GeVUQ3wiIuKkLEsdRYoZRpk1axbnnOOFuWrVKmrUqJHqvGLFilGo\nUCEAWrduDcDhw4c599xzwxitG8wVvflz+fLlXHbZZZEMKdcoXLgwTzzxBAAdO3YM6Dkpe08pf2/e\nvDngvU/D0YOKRkePHuXHH39MdSwxMZHu3btn+rxevXrZPytVqhSq8HKFY8eOATBjxgx7bPLkyQB8\n9913nDp1Ckg9UrB06VIAPvjgg7DE6OwQ3/r16wFo0KABEydOBODuu+9O99y+ffsCMHbsWMC7X1Ck\nSJEwROlxdYgvKSmJxo0bAzB//nwAypcvn+3XXb58OcuXLwfggQceAAjpF6vrQygpvzi3bt0KwJAh\nQ+wx0xCd2UAZJt7Dhw9TrFixEEXp53o+Dxw4AHhDSNOnTwfgpZdeYtu2bTl+zYoVK7Jnz56gxHcm\n1/N5pqVLl7Jv3z4ARo4cyaZNmwB/JyAuLi7br2meGywa4hMRkaji7BBfyi5k27ZtIxhJ9DhziK9x\n48asXbsWgAsuuACAJk2a2Ekkt99+O7Vq1UrzOqa3NG3aNAAmTpzI8ePHAWjXrh0Q/ploLrn88svT\n/NylSxd77KOPPgLgrrvuSvf5JocFCxYMVYhOMT3706dPM3r0aAA2b95sHzejJfnz57dX92erZ8+e\nQXmdaLN48WIWL14MwPjx4wE4ePBg0CZMvfLKK0F5nUCpByUiIk5ytgdl3HHHHZmO0ycnJ5OYmAh4\nvQMI7f0Rl5mxbvNn8eLFbW7MPSiAhQsXAl4Py9wovffeewF49913bQ/M3MuqWLGiHXMuU6ZMqP8Z\nUa9z586At04qPYMGDQKgQIECYYspEn766SfAPynk5MmT2Xq+6aVXq1YN8O6fpPTNN98A8Nhjj6V5\nbnojA7nN7t27ef311wH/aMfu3buzfX+oQ4cONG3aFID7778f8N6j5p6+ccMNN9h70OHibANVpUoV\nwLuBar5k8+XLl+a8uLg43nnnHQB69+4N5P4PfkbOHOJLqVWrVml+fvLJJzlx4kSq8ypUqGBnqZkZ\nk0uXLqV///4hiTk3eeuttwB/w3Tm/4MZ3jIXUrndqFGjAOxssIyYWbgFCxa0OSpVqhT/+Mc/AO9C\n60ybNm1i9erVqY7FxMQwYMAAAP75z3+eXfBRoEmTJuzduzfL87p27cozzzwDeGvzzlSkSBF7UR8b\nGwvAhx9+mOa8Ll26hH1YWkN8IiLiJGd7UA8++GBA55npkwB33nlnqMKJCmZIxFx5Hj582F69pjfs\nacpGpfTcc8+l+9rnn38+QJ5cXxaoI0eOZPhYoUKFaNCgQRijibyPP/4YgFtvvRXwr7s50zXXXAP4\nJ/IEYuPGjfb1jXLlytlRFFNdJjd6+umnAdLtPd199922t2SW2pQqVcquJU3PiRMn7C0AM/wcGxtL\nixYtABg8eDAAV155ZZD+BYFTD0pERJzkbA8qUObqC7z7J+LvSd533332KutspoVPmTIlKHHlZvHx\n8XZ6eXouueQSW8svrzGTRoLh999/B1JP+jFuuummqK0/mR1mosKePXto1qwZANdffz3gLcRP7159\nZg4ePGiXPhiFCxe2EzAi2fN3roFKucr+THFxcbz33nsA/PLLL4C3xsLcjN6/fz/gNVSFCxcOR7hO\nMiVekpOT7QSSjIbuAhGqFfm5gXmfjhw50r4n02MmDEjOJSUl0adPHwDmzZtnj9euXRvwcpzeJIDc\nxny+zTqnnDIXrynXmZoh/FmzZjkxJK0hPhERcZITPSgz1Xnbtm289tprgLce50zJyclppu7GxMRQ\nsmRJwH+VOmrUKHtTPy8ya0Ay2g4iUKZG186dO20hXknN7EX2wgsv2GMp94u66aabgNTVJyR7du/e\nDXgjJKYyCvh7EnfccQdAWOtvRjPzfWsmW6Qswr1kyRLAnUoxEW2gPvnkEwA762Tjxo32MbPpYI8e\nPVI9x1Te3bJlC+DN3DEbFZr9dvI600DNnTv3rBYs/vXXX4CX65wUmc3tYmNjbfXslBcDKddBne1F\ngvjLnpkZZsaXX34JQP369cMeUzTbuXMngN1bD/wzLLN7/yrUNMQnIiJOimgPqn379oC/asT69evp\n1KkT4F+3k3L9TlJSEn/88Qfg70EtXrxYPacMpKwekRNmBX/FihW55JJLghFSrjJ79mxbmDMjZ+5f\nJp5Dhw7ZCSampFZKiYmJ9ko/5ciKUaVKFfu9IYHbv38/9913H+Av0luvXj2GDx8OuLd+TD0oERFx\nkrMbFqZnzJgxPPTQQwDcdtttgLe1QUZFOcPFlQ0LQ+Xaa6+1P2fVYwiGaNkQrmbNmrZHn5KJp1ix\nYvz6668AEZ2042I+//vf/2ZaOeKZZ57h2WefTXUsX7589n5Uhw4dIna/xMV8ZiUhIQGA119/3d7L\nM1PKR4wYYTd9jZSMcurELL6smB03+/XrZ286Dx06FMi4YrRIqJg9izLaMdeYO3dunp5NmpmMGiez\n5m7ChAnpPt6wYUPAvZv5LktISLAXloMGDbINk1nQH+nGKTP6dhcRESc534NKSkqya0nA644C1KlT\nJ1Ih5Rlm/57Y2Nh0tzzIq1atWgVkXBzWrIMyV/sSmN27d9utSPbs2WOXNgwcOBCAPn365JldiIPB\nfH4ffvhhuxVM69at7XfoxRdfHLHYAuV8A7VmzRrWrFkDeCWMzIZaGtoLPVMpfu3atbZCumS+7xbo\nvZlTTZs2TVVWq0uXLgD83//9X6RCikpbt24F/A37zJkz7Yzejz76KKoWNOuTJCIiTnK+B/Xf//7X\n/vzWW29RokSJCEaTt5hJAMnJyVx33XURjsYdZuJDwYIF7ewoyT7TW+rWrZv93ZQvmj17tipE5EB8\nfLwtuzVz5kzA20X37bffBqKvHJR6UCIi4iTne1AVK1a0e56kLAsvoTd9+nTAu9eiShJ+5v3YoEED\nVqxYkeZx08tXHb7MjRkzBoCvvvrKHrv99tsB736UZN8ll1ySatQJYPXq1fztb3+LUERnx/kGqkWL\nFnzzzTeRDiNPS05OPuuySbnR3Llzad68OQCbN2+2x3/88UcAihYtGpG4okWxYsWA1BuNdu/ePULR\n5A5jx46lTZs2gL+C/t69e6O2LJSG+ERExElRVerIVbm11NHgwYMBb1uD9IayQiUaS8m4TPkMLuUz\n+DLKqRqoIMitDVSk6AsguJTP4FI+gy+jnGqIT0REnJTlJAnNRAou5TP4lNPgUj6DS/nMuUyH+ERE\nRCJFQ3wiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuKk\nTEsd+Xw+lZkIUKDFYsMRS26QnWKcoY4lN1A+g0v5DL70cpplLT6VQspadmptKZ9Zy27tMuU0c8pn\ncCmfwZdRTjXEJyIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTspyHZTkLsnJ\nyYwbNw6A3r172+PTpk0DoFOnThGJS0TkTOpBiYiIk9SDymPeffdd+vbtC0BMjP/6ZPHixYB6UNnx\nxx9/8J///AeAQ4cOceeddwLQpEkTAM477zx69eplzz/nHO/jdt1114U5Urdt27YNgBo1alCyZEnA\ny2egtm7dCsC///1vOnToAEChQoWCHGX0SUpKAuDUqVNMnz4dgL/++ss+vnDhQgAWLFhgj91zzz0A\n1KtXjwsvvBDwv19Tfl+Eiy+zMhw+ny852GU6Dh8+DMDXX3+d7uN79+4FoF+/foCX5PQSY5IfExND\nxYoVAbj88ssB6Ny5MzfffDMQnjeqz+cLuBZfpMqe/PnnnwA0b96cTZs2pXn8oosuAmDZsmUULVo0\nrLGdKdB8/u/coOV0zJgxALRv3x6Ae++9l8TExAzP37RpEzt27Aj49QsWLAjAvn37AChWrFhOQ82W\nSOUzUKaBuuCCC2zJmz59+tjPcJUqVQCoVKkSx44dA2D9+vWcOnUKgI4dOwJw/PhxypYtC8DKlSup\nXLlySOJ1PZ/x8fEAvPTSSwA8++yz6Z5n4sqqdNOkSZMA6Nq1a5AiTCujnGqIT0REnBT2HtS6desA\naNy4cUDnJycnp9vCZ9b6Jycnc+DAAQBKly6d01AD5nIPyvSc/vGPfwCwceNG6tWrB3hXRE8//TQA\nJ0+eBGDXrl22RxopkbhC3bJlCy1atABgz549Z/16mbniiisA+OKLL8LSi3L9ij9lDyo9ZtivVKlS\nnDhxAsj6/2jr1q1UrVo1iFH6uZzPXbt20bx5c8AbggaoWbMm5557rj1nxIgRAOTPn9/EaB/bvHkz\nkHoC1ZNPPglk3BMLhoxyGvZ7UIULFwa84Y2jR4+medwMg9SuXRuA06dPky9fvjTnnT59GoBjx46x\nffv2UIUb1Q4cOGC/dDdu3Ah4wyRDhgwBvKEsM9S6aNEiAOLi4sIfqAOqV69OnTp1gNRffubLsXv3\n7mf1+pMmTbLD28uWLQO8i4avvvoKgDJlypzV6+dmJm+HDx+mSJEigJe7b7/9Ns257dq1A4j4RVak\nTJgwwTZMXbp0AWD8+PEB3+ooV66c/dm8980960jQEJ+IiDgp7D0o0zOaPn06N910U4aPr169Gsj6\nBt6RI0e4/vrrAVi1alUwQ41aphfUokUL23MyV1CLFy+2OU7Pq6++yujRo0MfpGNiYmIYOXIk4E2O\nAJg6daodGjE36nPq+uuvp1WrVqmO/fzzz6xcuRKAG2644axeP5q9//77aY716dPH9gBSKlGiBODN\nkCxVqlSaxx9//HHAPxKT1zz22GPceuutgP+7NJDek5lw8txzz9ljDz74IJC6VxVu6kGJiIiTIrYO\n6tJLL033+C+//ALAZ599Bvin/GbkxIkTxMbGpjluelN5cc2JWW/TsmVLLr74YgA7GSKz3lNeZ96T\npvceTDVq1Aj6a+YWZuJUSvXr1+eyyy7L8DmffvppmmMFChSgZs2aQY0t2hQsWJAGDRpk6zmJiYkM\nHjwYgJkzZwJerynlRIlIiVgDVbJkSSZMmAD4F4eBf21ItWrVAnqdChUq2IWRZgYKwNq1a4G82UAV\nKFAAgDfeeCPCkUhmrr76alq2bBnpMCLOrHeaM2eOHY66+uqr0z3XrIN6+eWX0zw2YMAAe2NfAjdp\n0qQ0+Zw6daoTE000xCciIk6KWA8qX758dOvWDfAPrTzxxBO2B5XREGCgUvbKRCLFrNt5/vnn0zyW\nP3/+dJdQ5DW33XYb4FWRMaWKMloTdf/99wOwYsUKe8yMkphhKgmMKRE1aNAge8wM65m1VJGmHpSI\niDgposVizRTy+vXrAzB37tygvbZ5rbvvvjtor5kX9OjRI9Ih5CozZswA4MMPP4xwJO4y941MxYKM\nbN68mVmzZtnfTY1Os/Dc3HuVwDRs2BDwahiaReqvvPIK4J9oFWluRHEWDh8+zG+//Qb4yx8lJSXR\npk2bSIYlwokTJ+jTp0+a4+bL4L333gt3SFHJTIx49NFH7ZBpoUKF7B5mpki0BKZnz56AP68lSpRg\n3rx5gDsNk6EhPhERcZJbzWUO/PHHH6xZswbwDxlGYt+S3GLixIl5spJEqKS3Rq9z586AVxdRsjZ7\n9mwAPv/8c3usQoUK3HLLLZEKKWp9+umndnmPGXGaPHky1atXj2RYGdI3uYiIOCnqe1Aikjs98sgj\ngFeN2zATIQYMGBCRmKKVqc85Z84cO9LUv39/AKfv16uBklTSu6kvOXPm2icz9JzdUjR5UVJSEuPG\njQP8e5WBf5bZAw88EJG4olFCQoKdsbdlyxZb/NXk0mUa4hMRESflih7UmTtWJiUlRSiS6Ge2M5Cc\nM5tpnrnrq9kCom3btmGPKVokJiYC0KlTp1Q9J4C6devSsWPHSIQVlRISEgCvOsSWLVsAbyPH5cuX\nRzKsbFEPSkREnBT1PajHH388zaaGkyZNUk8gh8aMGZNu3TgJ3IYNGwDvfZiSrv6z9sILLwD+7XYA\n6tWrB8DSpUspXbp0ROKKRqaAQcoF4S+99BLnn39+pELKtqhtoL788ksAfvjhB3vMlIdXeaOcO3Lk\nSKRDiHqbNm1Kc6xUqVJ2Vpqk78iRI+neuDefdTVO2WPWO4F/+yJTmDdaaIhPREScFLU9qFatWgGk\nGt7TBn1n7+eff05V70yyb+DAgWmOtWjRwg5VSWrmZv6QIUNS9eBHjRoFQPny5SMSV7TasWMH4FWI\nMBYtWgRE32c6ahsoM1NPZY2C69///jdDhw4FYMSIERGORvKC3bt3A2kvMPv27QugPbOyyeyOe/To\nUcArTlyhQoVIhpRj+nYXEREnRW0PyvSczpzBJ9ljhgHMnjqrVq3iqaeeimBEudOwYcMiHUJUMTtr\nS/bExcWlmjgG3t54RYoUiVBEZ0c9KBERcVLU9qBSMiv0y5QpE+FIoo+5AT127NgIR5J7PPzwwwD0\n69fPbq1hpvlK5sxN/NWrV+veUw4cPHiQ1atXA141DsDZrTQC4TuzTFCqB32+5Mwej6SZM2cCMHXq\nVHsztUWLFhGJxefzkZycnOVYo8v5dEmg+fzfucppFpTP4FI+gy+jnGqIT0REnJRlDyqMsUS1QHtQ\n4YglN8jOFWqoY8kNlM/gUj6DL72cZtpAiYiIRIqG+ERExElqoERExElqoERExElqoERExElqoERE\nxEn/D4pZX87LFMIKAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n", + "ax = ax.flatten()\n", + "for i in range(25):\n", + " img = X_train[y_train == 7][i].reshape(28, 28)\n", + " ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n", + "\n", + "ax[0].set_xticks([])\n", + "ax[0].set_yticks([])\n", + "plt.tight_layout()\n", + "# plt.savefig('./figures/mnist_7.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "Uncomment the following lines to optionally save the data in CSV format. \n", + "However, note that those CSV files will take up a substantial amount of storage space:\n", + "\n", + "- train_img.csv 1.1 GB (gigabytes)\n", + "- train_labels.csv 1.4 MB (megabytes)\n", + "- test_img.csv 187.0 MB\n", + "- test_labels 144 KB (kilobytes)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "#np.savetxt('train_img.csv', X_train, fmt='%i', delimiter=',')\n", + "#np.savetxt('train_labels.csv', y_train, fmt='%i', delimiter=',')\n", + "X_train = np.genfromtxt('train_img.csv', dtype=int, delimiter=',')\n", + "y_train = np.genfromtxt('train_labels.csv', dtype=int, delimiter=',')\n", + "\n", + "#np.savetxt('test_img.csv', X_test, fmt='%i', delimiter=',')\n", + "#np.savetxt('test_labels.csv', y_test, fmt='%i', delimiter=',')\n", + "X_test = np.genfromtxt('test_img.csv', dtype=int, delimiter=',')\n", + "y_test = np.genfromtxt('test_labels.csv', dtype=int, delimiter=',')\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "source": [ + "## Implementing a multi-layer perceptron" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from scipy.special import expit\n", + "import sys\n", + "\n", + "\n", + "class NeuralNetMLP(object):\n", + " \"\"\" Feedforward neural network / Multi-layer perceptron classifier.\n", + "\n", + " Parameters\n", + " ------------\n", + " n_output : int\n", + " Number of output units, should be equal to the\n", + " number of unique class labels.\n", + "\n", + " n_features : int\n", + " Number of features (dimensions) in the target dataset.\n", + " Should be equal to the number of columns in the X array.\n", + "\n", + " n_hidden : int (default: 30)\n", + " Number of hidden units.\n", + "\n", + " l1 : float (default: 0.0)\n", + " Lambda value for L1-regularization.\n", + " No regularization if l1=0.0 (default)\n", + "\n", + " l2 : float (default: 0.0)\n", + " Lambda value for L2-regularization.\n", + " No regularization if l2=0.0 (default)\n", + "\n", + " epochs : int (default: 500)\n", + " Number of passes over the training set.\n", + "\n", + " eta : float (default: 0.001)\n", + " Learning rate.\n", + "\n", + " alpha : float (default: 0.0)\n", + " Momentum constant. Factor multiplied with the\n", + " gradient of the previous epoch t-1 to improve\n", + " learning speed\n", + " w(t) := w(t) - (grad(t) + alpha*grad(t-1))\n", + " \n", + " decrease_const : float (default: 0.0)\n", + " Decrease constant. Shrinks the learning rate\n", + " after each epoch via eta / (1 + epoch*decrease_const)\n", + "\n", + " shuffle : bool (default: False)\n", + " Shuffles training data every epoch if True to prevent circles.\n", + "\n", + " minibatches : int (default: 1)\n", + " Divides training data into k minibatches for efficiency.\n", + " Normal gradient descent learning if k=1 (default).\n", + "\n", + " random_state : int (default: None)\n", + " Set random state for shuffling and initializing the weights.\n", + "\n", + " Attributes\n", + " -----------\n", + " cost_ : list\n", + " Sum of squared errors after each epoch.\n", + "\n", + " \"\"\"\n", + " def __init__(self, n_output, n_features, n_hidden=30,\n", + " l1=0.0, l2=0.0, epochs=500, eta=0.001, \n", + " alpha=0.0, decrease_const=0.0, shuffle=True, \n", + " minibatches=1, random_state=None):\n", + "\n", + " np.random.seed(random_state)\n", + " self.n_output = n_output\n", + " self.n_features = n_features\n", + " self.n_hidden = n_hidden\n", + " self.w1, self.w2 = self._initialize_weights()\n", + " self.l1 = l1\n", + " self.l2 = l2\n", + " self.epochs = epochs\n", + " self.eta = eta\n", + " self.alpha = alpha\n", + " self.decrease_const = decrease_const\n", + " self.shuffle = shuffle\n", + " self.minibatches = minibatches\n", + "\n", + " def _encode_labels(self, y, k):\n", + " \"\"\"Encode labels into one-hot representation\n", + "\n", + " Parameters\n", + " ------------\n", + " y : array, shape = [n_samples]\n", + " Target values.\n", + "\n", + " Returns\n", + " -----------\n", + " onehot : array, shape = (n_labels, n_samples)\n", + "\n", + " \"\"\"\n", + " onehot = np.zeros((k, y.shape[0]))\n", + " for idx, val in enumerate(y):\n", + " onehot[val, idx] = 1.0\n", + " return onehot\n", + "\n", + " def _initialize_weights(self):\n", + " \"\"\"Initialize weights with small random numbers.\"\"\"\n", + " w1 = np.random.uniform(-1.0, 1.0, size=self.n_hidden*(self.n_features + 1))\n", + " w1 = w1.reshape(self.n_hidden, self.n_features + 1)\n", + " w2 = np.random.uniform(-1.0, 1.0, size=self.n_output*(self.n_hidden + 1))\n", + " w2 = w2.reshape(self.n_output, self.n_hidden + 1)\n", + " return w1, w2\n", + "\n", + " def _sigmoid(self, z):\n", + " \"\"\"Compute logistic function (sigmoid)\n", + "\n", + " Uses scipy.special.expit to avoid overflow\n", + " error for very small input values z.\n", + "\n", + " \"\"\"\n", + " # return 1.0 / (1.0 + np.exp(-z))\n", + " return expit(z)\n", + "\n", + " def _sigmoid_gradient(self, z):\n", + " \"\"\"Compute gradient of the logistic function\"\"\"\n", + " sg = self._sigmoid(z)\n", + " return sg * (1 - sg)\n", + "\n", + " def _add_bias_unit(self, X, how='column'):\n", + " \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n", + " if how == 'column':\n", + " X_new = np.ones((X.shape[0], X.shape[1]+1))\n", + " X_new[:, 1:] = X\n", + " elif how == 'row':\n", + " X_new = np.ones((X.shape[0]+1, X.shape[1]))\n", + " X_new[1:, :] = X\n", + " else:\n", + " raise AttributeError('`how` must be `column` or `row`')\n", + " return X_new\n", + "\n", + " def _feedforward(self, X, w1, w2):\n", + " \"\"\"Compute feedforward step\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ----------\n", + " a1 : array, shape = [n_samples, n_features+1]\n", + " Input values with bias unit.\n", + "\n", + " z2 : array, shape = [n_hidden, n_samples]\n", + " Net input of hidden layer.\n", + "\n", + " a2 : array, shape = [n_hidden+1, n_samples]\n", + " Activation of hidden layer.\n", + "\n", + " z3 : array, shape = [n_output_units, n_samples]\n", + " Net input of output layer.\n", + "\n", + " a3 : array, shape = [n_output_units, n_samples]\n", + " Activation of output layer.\n", + "\n", + " \"\"\"\n", + " a1 = self._add_bias_unit(X, how='column')\n", + " z2 = w1.dot(a1.T)\n", + " a2 = self._sigmoid(z2)\n", + " a2 = self._add_bias_unit(a2, how='row')\n", + " z3 = w2.dot(a2)\n", + " a3 = self._sigmoid(z3)\n", + " return a1, z2, a2, z3, a3\n", + "\n", + " def _L2_reg(self, lambda_, w1, w2):\n", + " \"\"\"Compute L2-regularization cost\"\"\"\n", + " return (lambda_/2.0) * (np.sum(w1[:, 1:] ** 2) + np.sum(w2[:, 1:] ** 2))\n", + "\n", + " def _L1_reg(self, lambda_, w1, w2):\n", + " \"\"\"Compute L1-regularization cost\"\"\"\n", + " return (lambda_/2.0) * (np.abs(w1[:, 1:]).sum() + np.abs(w2[:, 1:]).sum())\n", + "\n", + " def _get_cost(self, y_enc, output, w1, w2):\n", + " \"\"\"Compute cost function.\n", + "\n", + " y_enc : array, shape = (n_labels, n_samples)\n", + " one-hot encoded class labels.\n", + "\n", + " output : array, shape = [n_output_units, n_samples]\n", + " Activation of the output layer (feedforward)\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ---------\n", + " cost : float\n", + " Regularized cost.\n", + "\n", + " \"\"\"\n", + " term1 = -y_enc * (np.log(output))\n", + " term2 = (1 - y_enc) * np.log(1 - output)\n", + " cost = np.sum(term1 - term2)\n", + " L1_term = self._L1_reg(self.l1, w1, w2)\n", + " L2_term = self._L2_reg(self.l2, w1, w2)\n", + " cost = cost + L1_term + L2_term\n", + " return cost\n", + "\n", + " def _get_gradient(self, a1, a2, a3, z2, y_enc, w1, w2):\n", + " \"\"\" Compute gradient step using backpropagation.\n", + "\n", + " Parameters\n", + " ------------\n", + " a1 : array, shape = [n_samples, n_features+1]\n", + " Input values with bias unit.\n", + "\n", + " a2 : array, shape = [n_hidden+1, n_samples]\n", + " Activation of hidden layer.\n", + "\n", + " a3 : array, shape = [n_output_units, n_samples]\n", + " Activation of output layer.\n", + "\n", + " z2 : array, shape = [n_hidden, n_samples]\n", + " Net input of hidden layer.\n", + "\n", + " y_enc : array, shape = (n_labels, n_samples)\n", + " one-hot encoded class labels.\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ---------\n", + "\n", + " grad1 : array, shape = [n_hidden_units, n_features]\n", + " Gradient of the weight matrix w1.\n", + "\n", + " grad2 : array, shape = [n_output_units, n_hidden_units]\n", + " Gradient of the weight matrix w2.\n", + "\n", + " \"\"\"\n", + " # backpropagation\n", + " sigma3 = a3 - y_enc\n", + " z2 = self._add_bias_unit(z2, how='row')\n", + " sigma2 = w2.T.dot(sigma3) * self._sigmoid_gradient(z2)\n", + " sigma2 = sigma2[1:, :]\n", + " grad1 = sigma2.dot(a1)\n", + " grad2 = sigma3.dot(a2.T)\n", + "\n", + " # regularize\n", + " grad1[:, 1:] += (w1[:, 1:] * (self.l1 + self.l2))\n", + " grad2[:, 1:] += (w2[:, 1:] * (self.l1 + self.l2))\n", + "\n", + " return grad1, grad2\n", + "\n", + " def predict(self, X):\n", + " \"\"\"Predict class labels\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " Returns:\n", + " ----------\n", + " y_pred : array, shape = [n_samples]\n", + " Predicted class labels.\n", + "\n", + " \"\"\"\n", + " if len(X.shape) != 2:\n", + " raise AttributeError('X must be a [n_samples, n_features] array.\\n'\n", + " 'Use X[:,None] for 1-feature classification,'\n", + " '\\nor X[[i]] for 1-sample classification')\n", + "\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, self.w1, self.w2)\n", + " y_pred = np.argmax(z3, axis=0)\n", + " return y_pred\n", + "\n", + " def fit(self, X, y, print_progress=False):\n", + " \"\"\" Learn weights from training data.\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " y : array, shape = [n_samples]\n", + " Target class labels.\n", + "\n", + " print_progress : bool (default: False)\n", + " Prints progress as the number of epochs\n", + " to stderr.\n", + "\n", + " Returns:\n", + " ----------\n", + " self\n", + "\n", + " \"\"\"\n", + " self.cost_ = []\n", + " X_data, y_data = X.copy(), y.copy()\n", + " y_enc = self._encode_labels(y, self.n_output)\n", + "\n", + " delta_w1_prev = np.zeros(self.w1.shape)\n", + " delta_w2_prev = np.zeros(self.w2.shape)\n", + "\n", + " for i in range(self.epochs):\n", + " \n", + " # adaptive learning rate\n", + " self.eta /= (1 + self.decrease_const*i)\n", + "\n", + " if print_progress:\n", + " sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n", + " sys.stderr.flush()\n", + "\n", + " if self.shuffle:\n", + " idx = np.random.permutation(y_data.shape[0])\n", + " X_data, y_data = X_data[idx], y_data[idx]\n", + "\n", + " mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n", + " for idx in mini:\n", + "\n", + " # feedforward\n", + " a1, z2, a2, z3, a3 = self._feedforward(X[idx], self.w1, self.w2)\n", + " cost = self._get_cost(y_enc=y_enc[:, idx],\n", + " output=a3,\n", + " w1=self.w1,\n", + " w2=self.w2)\n", + " self.cost_.append(cost)\n", + "\n", + " # compute gradient via backpropagation\n", + " grad1, grad2 = self._get_gradient(a1=a1, a2=a2,\n", + " a3=a3, z2=z2,\n", + " y_enc=y_enc[:, idx],\n", + " w1=self.w1,\n", + " w2=self.w2)\n", + "\n", + " delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2\n", + " self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))\n", + " self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))\n", + " delta_w1_prev, delta_w2_prev = delta_w1, delta_w2\n", + "\n", + " return self" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Training an artificial neural network" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "nn = NeuralNetMLP(n_output=10, \n", + " n_features=X_train.shape[1], \n", + " n_hidden=50, \n", + " l2=0.1, \n", + " l1=0.0, \n", + " epochs=1000, \n", + " eta=0.001,\n", + " alpha=0.001,\n", + " decrease_const=0.00001,\n", + " minibatches=50, \n", + " random_state=1)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Epoch: 1000/1000" + ] + }, + { + "data": { + "text/plain": [ + "<__main__.NeuralNetMLP at 0x109d527b8>" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "nn.fit(X_train, y_train, print_progress=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNW9//H3lx1FJYiyK6ioYFS8RDAxUTSKRvNTExM1\nJsYYbhKjcX1uFI33kcQsGHNjjAYSr1s0iRHNFVFRcRuXRMUFBEEEoqjsuLC4sM18f3+cartnpmeY\nYbq6qqs/r+fpp6tOVVd9+zDMd07VqXPM3REREUmjdkkHICIi0hQlKRERSS0lKRERSS0lKRERSS0l\nKRERSS0lKRERSa3YkpSZDTCzx81sjpm9YmbnRuU9zOxhM5tvZtPMrHvBZy4xswVmNs/MRheUDzez\n2dG2a+KKWURE0iXOltQm4AJ33wc4CDjbzIYAY4GH3X1P4NFoHTMbCpwMDAWOBiaYmUXHmgiMcffB\nwGAzOzrGuEVEJCViS1LuvtzdZ0bLHwCvAv2A44A/R7v9GTghWj4euN3dN7n7ImAhMNLM+gDbufv0\naL9bCz4jIiIZVpZ7UmY2EDgAeA7o5e4rok0rgF7Rcl9gccHHFhOSWsPyJVG5iIhkXOxJysy6Af8A\nznP3dYXbPIzJpHGZRESkqA5xHtzMOhIS1G3uPjkqXmFmvd19eXQpb2VUvgQYUPDx/oQW1JJoubB8\nSZFzKdmJiCTI3W3Le7VOnL37DLgRmOvuvyvYNAU4PVo+HZhcUH6KmXUys0HAYGC6uy8H1prZyOiY\npxV8pgH/5OWuV+51+eWXJx5DGl+qF9WN6qZ0r7jE2ZI6GPgWMMvMZkRllwDjgUlmNgZYBJwE4O5z\nzWwSMBfYDJzl+W9+FnAL0BWY6u4Pxhi3iIikRGxJyt2fpumW2hFNfOaXwC+LlL8I7Fu66EREpBJk\ndsSJ2tqkI0iPUaNGJR1CKqlemqa6aZrqprwszmuJ5RQ6TuS/y1tvwYABzXxARERKxszwSuo4ISIi\n0laZTVL33JN0BCIi0laZvdwHkJGvJiKSerrcJyIiVUdJSkREUktJSkREUktJSkREUitTSer73086\nAhERKaVMJakdd0w6AhERKaVMJamGrOSdIUVEpJwylaT0XJSISLZkKknV1SUdgYiIlFKmkpRaUiIi\n2ZKpJLXNNklHICIipZSpJLXLLo3Lnnmm/HGIiEhpZCpJFbvc9/rr5Y9DRERKI1NJSkREsiXzSWr+\n/KQjEBGRrZX5JPWznyUdgYiIbK1MJammuqBv3lzeOEREpDQylaQ+85ni5U8/Xd44RESkNDKVpIYN\nK16uh3xFRCpTppJUU5Yvh+efTzoKERFpLfOMNDPMzN296MjnXbvCxx+rRSUiEhczw91LPvdEVbSk\nPv446QhERGRrVEWSEhGRylR1Scodli1LOgoREWmJqkpS//43DBkCffsmHYmIiLREVSWpSZPgtdeS\njkJERFqqqpKUiIhUlqpKUuqCLiJSWaoqSRX63OeSjkBERLakKh7mbUpGvrqISOL0MK+IiFQdJSkR\nEUktJSkREUktJSkREUmtqk9SdXVJRyAiIk2p+iTVvr16+YmIpFXVJykREUmvqk5SuRaUWlIiIulU\n1Ukqp2fPpCMQEZFiqjpJXXFFeH///WTjEBGR4qp6WKRCGakGEZFEaFgkERGpOplLUhMmJB2BiIiU\nSuaS1N57b93nXnghv/z66zB1amniERGRrZe5JLW1zjgjv3zOOXDsscnFIiIiQaxJysxuMrMVZja7\noGycmS02sxnR60sF2y4xswVmNs/MRheUDzez2dG2a+KI9ZVXYPbsLe8nIiLlE3dL6mbg6AZlDvzW\n3Q+IXg8AmNlQ4GRgaPSZCWaf9NWbCIxx98HAYDNreMyS2G+/xmXPPKOefyIiSYk1Sbn7U0Cxp5CK\ndVM8Hrjd3Te5+yJgITDSzPoA27n79Gi/W4ETmj5n22JueIzPfS4kKhERKb+k7kmdY2Yvm9mNZtY9\nKusLLC7YZzHQr0j5kqi8bGpry3k2ERHJSSJJTQQGAcOAZcD/JBBDi+gyn4hIsjqU+4TuvjK3bGY3\nAPdGq0uAAQW79ie0oJZEy4XlS4ode9y4cbz+em5tVPQSEZFSq6mpoaamJvbzxD4skpkNBO51932j\n9T7uvixavgA40N1PjTpO/A0YQbic9wiwh7u7mT0HnAtMB+4Hfu/uDzY4j7s7jz4KRxyx9fHW1UG7\ndvWXn3wSvvCFrT+miEjWxTUsUqwtKTO7HTgU6GlmbwOXA6PMbBihl98bwA8A3H2umU0C5gKbgbM8\nn0HPAm4BugJTGyaoQnvu2baY2+nJMRGR1MjcALNhuTTHvOEG+M//VEtKRGRL4mpJKUk1o2dPeOcd\n2GMPWLCgNMcUEckiJaktiCNJFcpINYmIxEJTdYiISNXJZJKKoyX14oulP6aIiDQvk5f7Bg+GhQtL\nf46MVJWISMnpcl8rfPazSUcgIiKlkMkkdcstMGJEPMe+7jq1qEREyiWTSapdu3juS0GYEFFERMoj\nk0kK1NoREckCJSkREUmtzCapOMbgW7lyy/uIiEjpZDZJde1a+mPeeGPpjykiIk3LbJIaOrT0x5w1\nq/THFBGRpmXyYV6AzZuhY8d4zjVuHJx2GnTpAn37xnMOEZFKogFmt6Bhkgpl8Z6zf394++14zyEi\nUgk04kQKrV2bX3aH9euTi0VEJIuUpNqgsOF27bXxdNYQEalmSlJttHRpeJ8/P9k4RESyKNNJatWq\neI+/bh306wdz58KGDfGeS0SkGmU6SfXsCc8/H/95HnoIliyJ/zwiItUm07378tviP/8xx8DUqRqO\nSUSqk3r3pVxziXDmTPif/ylPshQRyRK1pErMPXRN37wZevQIZTvtBO+8k98uIpI1aklViFmz4NBD\nYY898mW5BCUiIq2jllQMuneH1avzrabC82ekukVE6lFLSkREqo6SVAz0zJSISGkoScXg44+b3rbP\nPmEUdRER2bKquCf1zjuhh125bdgAnTsX35aRahcRAXRPqk169kzmvLvvnsx5RUSyoipaUmF7GYNp\ngYxUu4gIoJZUm/3ud0lHUN/hh+eXN22CM85o3efvvBNGjChtTCIiaVM1LSl3aJeylHzddXD22WFw\n2v79m29dvfYa7LVXfv300+HWW9UiE5F0UEuqjdJ2uQ/gRz9qett3vwv3359f33tvqKuLPyYRkTTp\nkHQA1e4734GPPqpfdt99cPPNsGYNHHtsvjyNiVZEJE5Vc7kvvx+cdBJMmlSGoFqp4TBKX/lKGAPw\npJPgwANDSyq3rVOncC8rI/98IlLh4rrcV5UtqV12STqClrvqquITKm7aVP5YRETKrWruSRUaMiTp\nCFom10ravLn+eiEz3asSkeyquiS1eXPru3uX03vvNS4rTE7vvAMXX9z09oa+8AUlMRGpXFV3Tyq/\nf3gfPBgWLIgpqFZ66qmQVBoaMADefhtqa+GOO+DUU+tv37wZ2rcvfkwz2LgROnYsfbwiIjnqgh6T\niROTjiCvWIKCkKDaIiN/h4hIFaraJPXoo2FsvUobX2/9+tZ/RklKRCpV1V7uK5Trzp12O+8MK1cW\n3/a//ws1NfCTn+Q7huRG2Vi/vunR2EVESkGX+2I0bVrSEbRMUwkK4Hvfg7/+Fe66K9yHmjkz3IsC\ntaREpHIpSQH9+iUdQenkOoScfDLMmxeWly9PLh4RkbbYYpIys9taUibpMn8+DBsWljVauohUqpa0\npD5duGJmHYDh8YSTjO22SzqC0il2b+3dd8sfh4hIKTSZpMzsUjNbB+xrZutyL2AlMKVsEZZB795b\n12sujX72s8ZldXUwdWr9so8+ggsvLE9MIiJba4u9+8xsvLuPLVM8W60tvftypk+Hiy6CJ54oUVAp\n0qcPLF2aX582DY46Sp0qRKQ0kuzdd5+ZdYuCOM3Mfmtmu5Y6kDQYMSJ0477ppqQjidcTT4QEJSKS\ndi1JUhOBj8xsf+BC4HXg1pYc3MxuMrMVZja7oKyHmT1sZvPNbJqZdS/YdomZLTCzeWY2uqB8uJnN\njrZd0+JvJ58obDE115VdRCRNWpKkNrt7HXAC8Ad3vw5oaVeDm4GjG5SNBR529z2BR6N1zGwocDIw\nNPrMBLNPpvmbCIxx98HAYDNreMySOuwwOPhg+NrXwth+WdDUZb2hQ8sbh4hIa7QkSa0zs0uBbxEu\n/bUHWjRcqbs/BbzfoPg44M/R8p8JyQ/geOB2d9/k7ouAhcBIM+sDbOfu06P9bi34TCwGDoSnn4Y7\n7wzLWfDOO8XLX321vHGIiLRGS5LUycAG4LvuvhzoB1zVhnP2cvcV0fIKoFe03BdYXLDf4uhcDcuX\nROVlkZWOBbW1cNll4WHf669POhoRkZbZ4sy87r7MzP4KHGhmXwamu3uL7km14NhuZiVLA+PGjftk\nedSoUYwaNarNx9xvvzAYbRaS1S9+Ed4feaTxtltuCfNsTZsGa9fCiSfCjBlwwAFlDVFEKkRNTQ01\nNTWxn6clXdBPIrScch2zDwF+7O53tugEZgOBe91932h9HjDK3ZdHl/Ied/e9zWwsgLuPj/Z7ELgc\neDPaZ0hU/g3gUHc/s8F52twFvZi6uvCAbJcuJT90asybB3vvXb/sgw+gWzfYsCEMwCsi0pwku6Bf\nBhzo7t92928DBwL/3YZzTgFOj5ZPByYXlJ9iZp3MbBAwmNBqWw6sNbORUUeK0wo+E7t27cII4hdd\nVK4zlt/q1Y3LdtopvO+7b3ljEREp1JIkZcCqgvV3o7Itf9DsduBfwF5m9raZnQGMB440s/nA4dE6\n7j4XmATMBR4AzipoGp0F3AAsABa6+4MtOX8pXXFFuc+YrI8/Du/z5+fLamrg3nsTCUdEqlRLLvdd\nBewP/I2QnE4GZrl7qtoWcV3uq3+OWA+fmP33h5dfbnp7rlp79gzjAGbh/pyIlFZcl/uaTFJmNpjQ\nE+9pMzsRODjatBr4m7svLHUwbaEkFZ9cte68M6xapSQlIo0lkaTuBy5x91kNyvcDfuHu/6/UwbSF\nklR8ctWa+/7FqnnmzNAiq9Y6Eql2SXSc6NUwQQFEZYNKHUil6NYt6QjSZ+nS0FV9xozQG/Dxx5OO\nSESyorkk1b2ZbRnukN28akxSCxeGZ8WacuON4X3jxtDB5PDDyxOXiGRfc0nqBTP7fsNCM/se8GJ8\nIaXfSSeF98WLm98vKw44AI44Ir++eDHcdlvx7//ee43LNm3KT2XfErW1rY9RRLKpuREnzgfuNrNv\nkk9Kw4HOwFfiDiyNnn8ett0WBg2Cjh2hX79weatz56Qji9cHH9RfHzAgv/zcc/nlK6+Evn0bf37i\nRDjvvJZ3uOjQIbTKOrZohEgRybImk1Q0IsTngMMIU8g7cJ+7P1au4NLmM5/JL//lL+G92kdjmDED\nNm8Oy5Mnw1lnNd5n3brWH7e2VklKRLYwdl/UXe6x6CXSyJln1l/P9e5bujQk8J49yx+TiGRHS0ac\nEGmxP/whvPfrF+blKvTaay0/jp7FEhFQkiqJ1nQKqCYN57B6663SHPd3vwsD/4pI9ilJlcBee4V3\nzdNU3/Ll4X3ZsvBuFlpIDzwAs2c33r/w3lVzLakLLgjDM4lI9ilJldCOO4ZBWK8qmBLy2msTCyc1\ncpf5zj4bvvlNOOYY+Pa3G++3/fb5Luy63Cci0IIBZitFOYZFav78cM89cNxx+XWAl14KXar32y+x\n0BLl3vRQScuWQe/e+XUzWLIk3M/64IPQ3b8YM1ixIowlKCLpkOR8UtICTzwBxx6bXz/qqDAn07Bh\n1T0nU3Nj+Y0c2bhsxYr4YhGRyqMkVSKHHALt2+fXH3wQVq7M/5IePjy8b7NN+WNLq7feyo8uMXBg\neD/++PB+yinNfzYjFwBEZAuUpMrkhRfg5pvDqBUbN8IzzyQdUTqsWhVG7XjzzbD+9tvh/b77Gu/7\nj3/kk9OaNS07/urVGpldpJLpnlSC9Msz3Jfq06f4tob/nGYhme26a1i///7QCaM5b7wBu+2mlpdI\n3HRPSjKpuSRTVwfTpuWXof7gs7p/JZJ9SlKSqBkzmt72y1+GDig77AAnnBDKFi3Kb1dLVCT7lKQk\ntf77v8P72rVw772Nt+eS1EUX5Qf8hdB9fePG+vvU1MQWpojESEkqQbvtBkOGbP3nx48vXSyV4sMP\nG5dddVX+AeqZM2G77Ro/LDxnTvyxiUjpqeNEwtauDZezirngArj66uLbunQJg7mOGRNfbJXgpZfg\nP/4jLNfVQbvoz66BA0O3/09/Gn7601BWgT8eIhUjro4TSlIJc8//Yi109dXwgx+E3m+77954+4kn\nhk4H1Z6kWqMCfzxEKoZ692VU7p7JiSfCddfVL+/aNVwSnDMHbrghmfhERJKkJJUCv/pVuPnfpUtY\nP+EEOPro/PahQ/PPBuXkRhSXttl1V42oLpJmSlIpMHYsjBgB++8f1u++Oz/9R06PHvnlM8+E732v\n+GVCadr994f3P/4xP6LFW2/B7bfn91myBD7+uPyxiUhxuidVQcxCMnvuubC+YQOMG5fv5Xf55flO\nAgB77AELF5Y9zFTbZZf85Itr14bpQXK++EV49FH44Q9hwoRk4hOpVLonJUBodeV07gyHHx6W3UPC\ngvAQLIQZbAEuu6xs4aVe4ezAhQkKQoKCMJ4ghES1yy6wfn391pV7fo4sEYmXklQF+eij/MgLOXvu\nWX/0dQjzV0GYOsQdrriiPPFlzWOPhQFv99oLBgzI/xHwz3/C3nsnGppI1VCSqiBduzYeCmjXXWHz\n5vz6okX171/l3HcfzJoVa3iZk2t1vfVW6FyRu5Sqe1Yi5aMklTENewHmHHts48kXP/wQvvWt+GOq\nNHfdBf37h2lVGho/HkaPDstmcOihxY9x/PHw/vvxxShSLZSkMqhhz8Bi3MMEjLvtli8bPx523DG+\nuCrJkiXFyy+5pP76k08W32/KFHjlldLGJFKNlKQy6POfb/kzVBdckF+++OIwX9OUKWH9Zz8rfWxZ\n9MUvhgktRaT01AW9ytxxR+i6XjgAa+4+V2H1TZ0KBx2kllVrFNafGTzxBBxySHLxiJSTuqBLSZx8\ncuMRwgF69qy/fswxoQPGjBlh/EDZsnXrwgPDuWT11FNhypAFC2DlylC2Zo3mwRJpDbWkBLPQlb25\nZ39yv1i7dQvzNUnrHHQQPPMMPPBA+AOgqR/VuXNDL85Bg8obn0hbqSUlsXnzzaY7ADT0+OPxxpJV\na9aEJPWNb4T1p54qvt8++4ThsTZsKF9sImmmlpS0SOF9K12uarvrrw/jLzZUWLf6cZZKopaUJOrz\nn4eHH046iuz4/vfDw9Xnnx8ewC52309/DIioJSVb4ZxzwtxXalXFb+HC4pNeiqSNWlKSGtde2/hS\nVG5cOymtH/0o6QhEkqWWlLTJ+PFhFIYNG8Ko7KAJGUvt61+HSZOgrg5+//vQuvrud2Hp0vAc2xtv\nwCmn1J9fbP368O9hFpY7dMgPPCwSh7haUkpS0ia1teGXX+GlvzFj4MYbk40r6zp1Cs9g9euXH8Jp\nyhQ48sjw75Cb5XnhQhg5MpQXTu4oUmpKUlugJJW8yZNh9eow6GpuTMBevWDFimTjqjZ9+4ZWFoQO\nGtdfD5/6VCjLJa8nn4TFi+HUU5OLU7JFSWoLlKTS5cUXYfjwMP7f5ZeHMQI/+gj+9KekI6teP/xh\nmAjz/fehd+9Qpv8yUirqOCEVZfjw8J77Jfjb38LPf15/n4YTOObkZhaW0po4MdynyiWoQgsXlj8e\nkZZQkpJY5TpTQOPxASdMgO98p/Fniv0SlXg88EDoWDF4MMyenXQ0Io3pcp/Eav16mDMn37IqfK7q\n/fdDj7QddsiX/elP4Ub/sGHljVPguefCQ8VHHAHbbpt0NFJpdLlPKlKXLvkEBWEa+69+Fdauhe7d\nYfvt88MDLV0abvTvv3/oNdhwgkGJ18iR4RLsVVeFPy42bICxY3XfSpKVWEvKzBYBa4FaYJO7jzCz\nHsAdwK7AIuAkd18d7X8J8N1o/3PdfVqD46klVaHmzw+zCRf75/v73+Hcc2HVqvLHVe1OPBH+8Y+w\n/PHH4Q8Os3A5dtmy8G+y007JxijpkcWWlAOj3P0Adx8RlY0FHnb3PYFHo3XMbChwMjAUOBqYYGZq\nBWZEw3tVhU45JT8XE4RLUXfcAdOnxx9XtcslKAjTh1x2WVhevjx0hNl55zDtyJVXhulbDjqoZcdt\n3x5uu6308Uo2JdmSegP4jLu/W1A2DzjU3VeYWW+gxt33jlpRde5+ZbTfg8A4d3+24LNqSWVYsdmD\nx4yBm25KJh6p7+CD4Z//hDvvhK99LZRdfDH8+tfhMuKzz+b3LfZvKZUvqy2pR8zsBTPLTVrQy91z\nj36uAHpFy32BxQWfXQz0K0+YkhYnnVR//YgjkolDGvvnP8P7178ektCYMSFBQeiQMWcOHHZY/bEI\n164NsxZv3Jifp+zZZ8MAxiI5Sbak+rj7MjPbCXgYOAeY4u6fKtjnPXfvYWbXAs+6+1+j8huAqe7+\nfwX7qiWVYWvXhu7shV3aIawfdRR85SvwzW+G9f/6L/jNb8L2Ll1CJ4BPfxpeeaX8cUvL5VpjUL+V\n9f77YcQMSbe4WlKJDTnp7sui91VmdjcwAlhhZr3dfbmZ9QFydyOWAAMKPt4/KqtnXMFQ3KNGjWLU\nqFHxBC9lt/32xcsbzmC7dGm4x3X11eGv+smTQ/ns2aETRm5mXEmfXIIC2LQJOnYMyz16wEMPwejR\nycQlxdXU1FBTUxP7eRJpSZnZNkB7d19nZtsC04CfAkcA77r7lWY2Fuju7mOjjhN/IySyfsAjwB6F\nTSe1pKTQBx+EQVjPOw/WrYO//CWUa/6ryrFwIWy3XRj/8brr4Oyzk45ImpOpsfvMbBBwd7TaAfir\nu/8q6oI+CdiFxl3QLyV0Qd8MnOfuDzU4ppKUbNGXvxwGwP3xj/MJ6zvfgVtuSTIqaam1a8OAuS++\nCH/7W9LRSKFMJak4KElJa/XvDz/5CRx4IFx4ITz11Jb3X7y4+X2kfEaPhmnTQrLae2/Yb7/Qvb3Q\nxo3hEYb+/ZOJsZooSW2BkpS01ezZ4Rddod/8JnTEAN3ArwTnnw9DhoSZopcty5f/5jfwpS/BmWfm\n/xh57z39e5aSktQWKElJKRx6KPz0p6G7dO7H6a23wgOrf/hD6Kjx0kuw556hM8e770KfPsnGLKX3\n73+H7vGrV4dHH+bPD2NMLlsGBxwQ9sn9fOg+Z6AktQVKUlIq69aFHoINew425d//hj32yN/bKhxO\nSCRnm23CwL3dusHAgeEPoW23DY9NdOwYLlV27hxG9+jUKcx4nSvv1Cksd+gQ1nOv3D4dOjTe1r59\nGMC52CsOSlJboCQlSVm/Pvxief310F26c2c4/HD48EOYNSvp6ETKRUmqWUpSkiabNoXLQXV1oUW2\nww75y0K77BIuIUK4R/LHPyYXp0jpZG9YJJHM6tgxXKLp0iU/X5Z7GE387ujhi/vuC9O5L14cLgHl\nfPWr5Y9XJK3UkhIps9raMAp4w1mJcy2tWbPCQK1Dh2qEDKkkutzXLCUpqXRm8PDD+YFz3WHFinCj\ne8kSOPLIcBmxRw9YtAgGDIC33040ZJECutwnkmnu9Ud2z00wuPPOodvzokWhC/Rrr8HEifDmm/mB\ndO+/P5GQRWKnlpRIBaurC12N3fW8jiRNLSkRaaBdu/xDpS+9FEZVyOnSJb9cVxfezz8/vEQqhVpS\nIhn2wQcheR1ySHhdc024dJhrdY0fH0ZU2G23ZOOULFDHiWYpSYm0XG1tGKFgzhzYa68waeS99xbf\nd889w7BAIs3T5T4RKZH27cMEkUOHhuUpU8I4hPvsE0bQgNBN3h0mTEg2VqluakmJSCPTpsGoUeGB\nZAjJ6sMP4dVXYeTI+tO7iwRqSYlImYwenU9QEO5hdesW5t7KDem0dm2YNTfnmGMaH2f77eONU7JP\nSUpEWqV/f3jllTC1+/LlcPXV8PTT4VmtTZvCA8jf+lYYk3DNGnjkkTC5JITR4gE++9n88d59t/zf\nQSqHLveJSOLuuQeOOy7f67CuLrw+/BB22il09Pj1r/MTUEoaqXdfs5SkRKrDRx/Bv/4FZ58NF14Y\nOn8cckjSUYmS1BYoSYlUt8KZctesge7dw/ruu4eJKSVuSlLNUpISkdZYvjxMVnnrrfDEE6Hr/aWX\nwsEHJx1ZpVKSapaSlIjEYfPmcE+sU6fQSqutDSN5/N//wV13wdSpSUeYFkpSzVKSEpFqVFsbEumm\nTfmEWlsb1jdsCGW5jii5bbmy3OzRuW2bN+cvm9bW1v8MNN4/Vw5w6qlKUs1SkhIRSY6ZHuYVEZEq\noyQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQl\nIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKp\npSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKpVTFJysyONrN5ZrbAzC5OOh4REYlf\nRSQpM2sPXAccDQwFvmFmQ5KNqnLU1NQkHUIqqV6aprppmuqmvCoiSQEjgIXuvsjdNwF/B45POKaK\nof9Uxalemqa6aZrqprwqJUn1A94uWF8clYmISIZVSpLypAMQEZHyM/f0//43s4OAce5+dLR+CVDn\n7lcW7JP+LyIikmHubqU+ZqUkqQ7Aa8AXgaXAdOAb7v5qooGJiEisOiQdQEu4+2Yz+xHwENAeuFEJ\nSkQk+yqiJSUiItWpUjpONKsaHvQ1s5vMbIWZzS4o62FmD5vZfDObZmbdC7ZdEtXHPDMbXVA+3Mxm\nR9uuKSjvbGZ3ROXPmtmu5ft2W8/MBpjZ42Y2x8xeMbNzo3LVjVkXM3vOzGaa2Vwz+1VUXvV1A+H5\nSzObYWb3RuuqF8DMFpnZrKhupkdlydWNu1f0i3D5byEwEOgIzASGJB1XDN/zC8ABwOyCsl8DF0XL\nFwPjo+UOXTAmAAAFyUlEQVShUT10jOplIflW83RgRLQ8FTg6Wj4LmBAtnwz8Penv3MJ66Q0Mi5a7\nEe5dDlHdfFI/20TvHYBngc+rbj6pmwuBvwJTonXVS4j3DaBHg7LE6ibxCilBhX4WeLBgfSwwNum4\nYvquA6mfpOYBvaLl3sC8aPkS4OKC/R4EDgL6AK8WlJ8C/LFgn5HRcgdgVdLfdyvraDJwhOqmUb1s\nAzwP7KO6cYD+wCPAYcC9UVnV10sU7xvAjg3KEqubLFzuq+YHfXu5+4poeQXQK1ruS6iHnFydNCxf\nQr6uPqlHd98MrDGzHjHFHQszG0hobT6H6gYAM2tnZjMJdfC4u89BdQNwNfBjoK6gTPUSOPCImb1g\nZt+LyhKrm4ro3bcF6vkBuLtX87NiZtYN+AdwnruvM8s/rlHNdePudcAwM9sBeMjMDmuwverqxsy+\nDKx09xlmNqrYPtVYLwUOdvdlZrYT8LCZzSvcWO66yUJLagkwoGB9APUzeJatMLPeAGbWB1gZlTes\nk/6EOlkSLTcsz31ml+hYHYAd3P29+EIvHTPrSEhQt7n75KhYdVPA3dcA9wPDUd18DjjOzN4AbgcO\nN7PbUL0A4O7LovdVwN2EsVMTq5ssJKkXgMFmNtDMOhFuxE1JOKZymQKcHi2fTrgfkys/xcw6mdkg\nYDAw3d2XA2vNbKSFpsZpwD1FjvU14NFyfIG2ir7HjcBcd/9dwSbVjVnPXC8sM+sKHAnMoMrrxt0v\ndfcB7j6IcK/kMXc/jSqvFwAz28bMtouWtwVGA7NJsm6SvklXoht9XyL06loIXJJ0PDF9x9sJo21s\nJFzPPQPoQbj5Ox+YBnQv2P/SqD7mAUcVlA+PfugWAr8vKO8MTAIWEHqBDUz6O7ewXj5PuK8wk/AL\neAZhShfVDewLvBTVzSzgx1F51ddNQfyHku/dV/X1AgyKfl5mAq/kfp8mWTd6mFdERFIrC5f7REQk\no5SkREQktZSkREQktZSkREQktZSkREQktZSkREQktZSkRIows9poqoLc66ISHnugFUy5spXHuLyZ\nbYWxTy4oH2Rh6o4FZvb3aKQOkVTTc1IiRZjZOnffLqZjDySMvL3vVnz2F4QBdI8kPMR8k7u/3GCf\norGb2STgLnefZGYTgZfd/Y9b8RVEykYtKZFWiCaEuzKaFO45M9s9Kh9oZo+Z2ctm9oiZDYjKe5nZ\n3RYmHpxpZgdFh2pvZtdbmKjxITPrEu1/roUJHF82s9sbnt/df0IYYeWbwHUNE1QzcRthWoq7oqI/\nAye0pS5EykFJSqS4rg0u9309KndgtbvvB1wH5MYLvBa42d33J0yk9/uo/PeEKTKGAf8BzI3KBxOS\nzKeB1cCJUfnFhEkc9wd+0DAoM7sCeAD4C/AjM9uvSOxdzOxFM3vGzI6PynaM4s5NTVE4dYJIauly\nn0gRzVwyewM4zN0XRfd0lrl7TzNbBfR299qofKm772RmK4F+7r6p4BgDgWnuvme0fhHQ0d1/YWYP\nAB8QBvCc7O4fNhHf5e7+0ya29fEw1cIg4DHgcGAd8Iy7D472GQBM3ZpLjiLlpJaUSNsU/pVnTexT\nrHxDwXIt+bndjgX+QGh1PW9m7YuetIkEFW3LTbXwBlBDmAjyXaC7meX+z/cntKZEUk1JSqT1Ti54\n/1e0/C/CtA8Q7hc9GS0/CvwQwMzam9n2TR00um+0i7vXAGOBHYBtWxOYmXU3s87Rck/gYMI0Jg48\nDuQuWxZOtyCSWlmYmVckDl3NbEbB+gPufmm0/CkzexlYD3wjKjsHuNnMfkyYEO6MqPw84HozG0No\nMZ1JmH674XV2B9oDt1mYRdeAa9x9bSvjHgL8yczqCH+E/srdczOrXgz83cx+TpjC48ZWHluk7HRP\nSqQVontSw71CZlkVqXS63CfSOvqrTqSM1JISEZHUUktKRERSS0lKRERSS0lKRERSS0lKRERSS0lK\nRERSS0lKRERS6/8D1eIjaZLpzHsAAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "%matplotlib inline\n", + "import matplotlib.pyplot as plt\n", + "plt.plot(range(len(nn.cost_)), nn.cost_)\n", + "plt.ylim([0, 2000])\n", + "plt.ylabel('Cost')\n", + "plt.xlabel('Epochs * 50')\n", + "plt.tight_layout()\n", + "# plt.savefig('./figures/cost.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "batches = np.array_split(range(len(nn.cost_)), 1000)\n", + "cost_ary = np.array(nn.cost_)\n", + "cost_avgs = [np.mean(cost_ary[i]) for i in batches]" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xn8VHW9x/HXh32RNVkFxIUQl1AwTM2kFNPM5VqBpqVm\n5XbTFhfw3pS2q3hvZaa5pYZe18I9NTTE5RrihqCAgUgKsiiKgAvyg8/943N+zjD+fvADZubMnHk/\nH495zJnvOTPznaPw5nvOdzF3R0REpBI1S7sCIiIijVFIiYhIxVJIiYhIxVJIiYhIxVJIiYhIxVJI\niYhIxSpZSJlZXzN7xMxeMrMXzeyMpLyrmT1kZv80s4lm1jnvPWPMbI6ZzTazg/LKh5rZjGTf70pV\nZxERqSylbEmtAX7k7rsAnwNON7NBwGjgIXf/NPD35DVmtjMwCtgZOBj4g5lZ8llXACe5+wBggJkd\nXMJ6i4hIhShZSLn7YneflmyvAmYB2wCHA+OTw8YDRybbRwC3uPsad58PzAX2MrNeQAd3n5ocd0Pe\ne0REJMPKck/KzPoDewBPAT3cfUmyawnQI9nuDSzIe9sCItQKyxcm5SIiknElDykz2wqYAJzp7ivz\n93nMyaR5mUREpEEtSvnhZtaSCKgb3f2upHiJmfV098XJpbylSflCoG/e2/sQLaiFyXZ++cIGvkth\nJyJSQdzdNn7UhpWyd58B1wIz3f2SvF33AMcn28cDd+WVH21mrcxsO2AAMNXdFwMrzGyv5DO/lfee\n9fitt+IjR+LueuQ9LrjggtTrUMkPnR+dG52b4j+KpZQtqX2B44DpZvZ8UjYGuAi43cxOAuYDIwHc\nfaaZ3Q7MBOqA0zz3S08D/gS0Be539wcb/MY2beDDD0vyY0REpPxKFlLu/gSNt9QObOQ9/wX8VwPl\nzwK7bfRLFVIiIpmSrRknFFINGj58eNpVqGg6P43TuWmczk15WDGvHabJzNynTIEzzoCnnkq7OiIi\nNc3M8EruOJEKtaRERDJFISUiIhUrWyHVtq1CSkQkQ7IVUmpJiYhkSvZC6oMP0q6FiIgUSfZCSi0p\nEZHMyFZItWwJdXWwdm3aNRERkSLIVkiZRWtq9eq0ayIiIkWQrZACXfITEckQhZSIiFSs7IWUxkqJ\niGRG9kJKLSkRkczIXkh16ADLl6ddCxERKYLshVT//vDqq2nXQkREiiB7IbXDDvDKK2nXQkREiiB7\nIdW9O7z1Vtq1EBGRIsheSLVvD++9l3YtRESkCLIXUu3awfvvp10LEREpguyFlFpSIiKZkb2QUktK\nRCQzshdSakmJiGRG9kJKLSkRkczIXki1b6+QEhHJiOyF1FZbwTvvgHvaNRERkS2UvZDq2RM6doQX\nX0y7JiIisoWyF1JmsOuuMG9e2jUREZEtlL2QAujcWTOhi4hkQDZDqkuXuC8lIiJVTSElIiIVSyEl\nIiIVK5sh1bmzQkpEJAOyGVJqSYmIZIJCSkREKlZ2Q0pd0EVEql42Q6pzZ5g7F555Ju2aiIjIFjDP\nyBx3ZuYf/5b334+JZkFz+ImIpMDMcHfb0s/JZkuqXTu4+27Yfvu0ayIiIlsgmy0pgFWroHv3WADR\ntjjMRURkE6gltTFbbQWtW8Pbb6ddExER2UzZDSmAvn1hwYK0ayEiIpsp2yHVpw+8/nratRARkc2U\n7ZAaOBAmTUq7FiIispmy23ECYPFi2GkneO21WK1XRETKQh0nmqJnz3gsXJh2TUREZDNkO6RA8/iJ\niFSx7IdUu3Zwxx1p10JERDZDtu9JxY54zsjvFBGpBronJSIimVfSkDKz68xsiZnNyCsba2YLzOz5\n5HFI3r4xZjbHzGab2UF55UPNbEay73ebVIlf/CKe1ZISEak6Jb3cZ2b7AauAG9x9t6TsAmClu/+m\n4NidgZuBzwLbAA8DA9zdzWwq8O/uPtXM7gcudfcHC97f8OU+iCmS3nhD3dBFRMqkKi73ufvjQENd\n6xqq+BHALe6+xt3nA3OBvcysF9DB3acmx90AHLlJFVEPPxGRqpTWPakfmNkLZnatmXVOynoD+RPt\nLSBaVIXlC5PypuveHRYt2oLqiohIGtIIqSuA7YDdgUXAr0v+jXvuCU89VfKvERGR4mpR7i9096X1\n22b2R+De5OVCoG/eoX2IFtTCZDu/vMEpJMaOHfvx9vDhwxk+fHi8GDQI5s2L6ZGmT4evfnVLf4aI\niOSZPHkykydPLvrnlnyclJn1B+7N6zjRy90XJds/Aj7r7t/M6zgxjFzHiR2TjhNPAWcAU4G/sqkd\nJ667Dp54IpaVv+029fQTESmxYnWcKGlLysxuAfYHtjaz14ELgOFmtjvgwKvAyQDuPtPMbgdmAnXA\naXmpcxrwJ6AtcH9hQG1Ux46wYoVW6BURqTLZn3ECYOJE+PKXYxHE119XS0pEpMSqogt6xagfH1W/\nAOLy5bBqVXr1ERGRJqmNkNpqq/Vfd+kChx6aTl1ERKTJaiOkOnX6ZNn06eWvh4iIbJLaCKm+fWHK\nlPXLmjdPpy4iItJktRFSAJ/6VG77uONinSkREalotRlSAwZoLj8RkSpQG13Q602fHmOl+vSBbt3g\ngw+gZcvyVFBEpIYUqwt6bYVUvm7d4KWXYvJZEREpKo2T2lJvvQVjxqRdCxER2YDaDalLLonBvRMm\npF0TERFpRO1e7ps7NzpQADz5JOy9d2kqJiJSg3S5b0vtuCOMGwcHHADPPZd2bUREpAFlX0+qopxz\nTvT2e+WVtGsiIiINqN2WVL2ddtIUSSIiFap270nVW7kSevWK3n5t2hS/YiIiNUj3pIqlQwfYbjuY\nOTPtmoiISAGFFMDgwXDXXbBoUdo1ERGRPAopiJD6xS/goIPSromIiORRSAF85jPxrElnRUQqikIK\nYNCgeLYG7vHtvntu2XkRESkrhRTErOgACxZEJ4p8L7wA06aVv04iIqKQAqBZM3j22diePx+uvhq2\n3z63f926VKolIlLranvGiXxDhsTlPnc4+eQoe+yxeF6zJr16iYjUMLWk8l1++fqvTzopnu++u/x1\nERERzTjRwAfF8wUXwEUXwerV8Xrt2rgsKCIiG6UZJ0rlww/jkt9JJ+UCCuJelYiIlJVCqlDr1vHc\ntWs8jx4Nhx2mHn4iIilQSDWmXbt4PuqoGCulkBIRKTuFVGPMYMSIWMpDISUikgqF1IZMnBizpO++\ne2713vHjoxOFiIiUnEKqKbbbLjpRTJ0KJ5ygaZJERMpEIdUUZrDnnnDNNfF68eJ06yMiUiMUUk3V\np09umXmFlIhIWSikmqpXL3jxxdjW4ogiImWhkGqqHj3g/fehZUu1pEREykTTIjXVggXQty80bx69\n+5YtiwG/ixbBpz4FrVqV7rtFRKqMpkUqtz59okv6z38er8eMiYlne/eGP/0p1aqJiGSVQmpTjBgB\nhx4a21dfDUceGdtr18Z8f7NmpVc3EZEMUkhtqsGD4bLL1i9bsQIeeQR23jmdOomIZJRCanN06bL+\n65Ur414VaBVfEZEiUkhtjsKQWrEiN1XSm2+Wvz4iIhmlkNocXbpEV/R6v/89LF0a2wsXplMnEZEM\nUkhtjj59YK+94Kyz4DOfibJnn43nZcuiE0XhJLRPPFHeOoqIZIDGSW2pZctilvTtt4fHHoNbbonB\nvj/6UYQVwDvvxJiqceNgv/1g773LX08RkTIq1jgphVQxXHopnHlm7nW/fvDaaxFS558Pu+wCRx8d\n+775TbjppnTqKSJSJmUbzGtmNzalrKYdccT6r197LZ5Hj4Zf/AK+//3cPjPYddfy1U1EpIq1aMIx\n6/2NamYtgKGlqU6V2nbbhsuffjqeV6zIlT3yCLzxRunrJCKSAY22pMzsPDNbCexmZivrH8BS4J6y\n1bBaPP88zJgBl1+eK2vf/pPHvftuPGt1XxGRjdroPSkzu8jdR5epPpst1XtS+ZYtgwsvjM4UF10E\n//oXrFr1yeMWL46Z1UVEMqhsHSfM7PPANHdfZWbfAvYAfufu/9rSLy+migmpekuWQM+eudejRsWs\nFDffHK+ffjpW+xURyaByzoJ+BfC+mQ0GfgzMA25oyoeb2XVmtsTMZuSVdTWzh8zsn2Y20cw65+0b\nY2ZzzGy2mR2UVz7UzGYk+37X5F+Xph494OWXo0s6xLiqAw6I7VGj4IEH4Kc/Ta9+IiJVoCktqefd\nfQ8zuwBY6O5/NLPn3H3IRj/cbD9gFXCDu++WlF0MvOXuF5vZuUAXdx9tZjsDNwOfBbYBHgYGuLub\n2VTg3919qpndD1zq7g8WfFdltaTyzZsX3dLXrIGZM6PzxNlnx75166LHn4hIhpSzJbXSzM4DjgPu\nM7PmQMuNvAcAd38ceKeg+HBgfLI9HkjWu+AI4BZ3X+Pu84G5wF5m1gvo4O5Tk+NuyHtPddh+e2jR\nAtq2haFD4Sc/iZYVxGq/TeGeGxwsIlIjmhJSo4DVwHfcfTHRyvnvLfjOHu6+JNleAtT3HugNLMg7\nbkHyXYXlC5Py6mUG994b2/W9/TamSxf45S9LVycRkQq00XFS7r7IzG4CPmtmXwWmunuT7kk14bPd\nzIrWPBg7duzH28OHD2f48OHF+uji69YNPv3pGEPVu/fGj3/3XXjuudLXS0RkM0yePJnJkycX/XOb\nck9qJNFyejQp+gJwtrv/uUlfYNYfuDfvntRsYLi7L04u5T3i7juZ2WgAd78oOe5B4ALgX8kxg5Ly\nY4D93f2Ugu+p3HtSjRk2LDpYrFoVIXTffTHH34QJ0YX9y1+O2SuaNYvW19FH5zpiiIhUsGLdk2rK\njBP/CXzW3ZcmX9wN+DvQpJBqwD3A8cC45PmuvPKbzew3xOW8AUSrzc1shZntBUwFvgVcupnfXVk6\ndoxgqjd7NixaBMcdB9/7Xiz7MW5crgXVqtX677/tNthjj2iRiYhkUFPuSRmQv5LfsqRs4280uwV4\nEhhoZq+b2YnARcAIM/sn8KXkNe4+E7gdmAk8AJyW1zQ6DfgjMAeYW9izr2rtuOP6r994I0KqWTO4\n5pooO+88+MtfYnvlytyxU6ZEy+r888tTVxGRFDTlct9/A4OJ7uFGdKSY7u7nlL56TVeVl/vefjum\nUzrwwHg9ciRMmwbHHAN33gnz58c9q0sugc9/Hk44IaZe+te/oH//eM/3vw9XXZXSDxARaVjJL/eZ\n2QCiJ97ZZvY1YN9k15NEYMmW6to1N8B3l13g9ttju08feOGFWO23R4/oer7rrjB3Lnz4YS6gAFq3\nLnu1RUTKZUOX+y4BVgC4+wR3/7G7/5i4h/TbclSuZsycCZMm5V7Xh1D37jHo96ijIoy6d4f77499\np54K//u/MGsW3LiBlVPeeQdefbVkVRcRKaVGL/eZ2TPu3uDkcmb2ortX1KJIVXm5r9ATT8R8fm3a\nNLw/f2YK9wi2+pbYySfDlVd+8j3HHAO33qqBwCJSVuWYcaLzBvY18reobJHPf77xgMo3Z04851/2\na+y+lKZcEpEqtqGQesbMvl9YaGbfA54tXZWkUZMmRceK+l6Bffuuv78+kNatg7feiu36buvLl5en\njiIiRbShy309gTuBj8iF0lCgNfBv7r6oLDVsokxc7tscDz8MI0bkXj/2WExoe/HFMHBgjL2aNQue\neSbmDRQRKYOyrCdlZgZ8kVhC3oGX3H1So29IUc2GFGz8kt4OO8CvfhVLhIiIlEFZZpxI/taflDyk\nUj3xBCxYEIN7G7LfftF9vdCjj8Ylw+23L239REQ200YH81aLmm5J1XOHa6+N0OnUCY49NhZevPJK\nOOWUGHfVrVvueDPYe2948sn06iwimVS25eOrhUKqAV/5SqwAvHRpjLGCGJM1aFBsm8HgwdEZQ0Sk\niMq56KFUq4MOghNPjNbTfvtF2d/+FhPXfuMb8Tp/0UV3qKsrfz1FRBqhllSteOMNOO00uPvumHZp\nQd46klddFXMA/vKX8fjww/TqKSKZoJaUbJrevSOkYP2A6tEjd0/quedg9WrYbbfc/g8/1GwVIpIa\nhVQtOeAAuOKK9csuvTQWXQRo3jyeX3wRfvvbmE6pbVsttCgiqWnKooeSFc2bR4+/++6Dli3hrrtg\n221jmqXVq+Hxx3PHPv00vPRSbOevYyUiUkYKqVrToUOE1AcfwJtvxmXARYvgz3+Oe1VTpsQ6Vmee\nGeE0alT0DhQRSYEu99Wqtm2hXz9o0QL23Rd+/evoAdi/f6xttWABLFkSXdQXL8697847YyFGEZEy\nUEgJDBkSY6Xqu6lvvXWsGvz++7D77jH3X72jjoLrr0+nniJSc3S5TyKkIJYKgRjku/XW8bzPPnF/\nas6cuFQI0Z1dRKQMFFISUyN997u5WSkAnnoqQqpTp7g/ddllcMghsU8r/YpImWgwr2zcyy/DTjvF\no0uXmLFi3rxcl3URkQIazCvlM3AgDBgQa1MddRS89lp0uDCLmdQB7rgDvvOd2P7mN+HUU9Orr4hk\nhlpS0jS33grHHBMT1P7kJzFxbT13OP54uOGG2K5f30r/PURqllpSUl6f+xwcdljMoH7jjTHQd+ed\nY9911+WCSQN/RaSI1JKSzbduXe6+VPfuMej36aejIwbEvH/1+++/H1q3jqmZRCTz1JKS9DVrBo88\nEttLl8KwYbEUyJAhEVotWsBf/hIzWxx6KBx4YNzDqp8rUERkI9SSki3329/C889HaI0fD+eeGy2q\nSZNyx7RoAW3a5I7p2BF23DFmvRCRzFFLSirHj34UnSbGjYvX3/52BFC+ujr40pdiSqV//CMu+40f\n3/hnnn46vPJK6eosIlVBLSkprrq6aDW9+WZMp7T//rDrrtHJ4vDDo9X17LNxbLt2MHYsnH127v2r\nV8d9rJYt4eKL198nIlWjWC0pzTghxdUi+V+qW7d4PPdcTFLbLGm09+wZ96a+/vW4X3XOOfDFL8Ke\ne8b4qltuiWNA0y+JiC73SYntsUcuoCAGBUOMuYK4f3XTTbFdv7ji4sXRrX3WLJg+HbbZBt59d/3P\nPfHEaHWJSKbpcp+Ulzs8/DCMGBGvFyyAvn1h+HCYPDlmXZ82LS4JDh2ae9+kSdHighiL1bEjPPRQ\n3NuyLb6iICJFpo4TUp3McgEFsdAiREAB/PCH8J//CdtvH69POinGXT30ELz3XpTNnRvPI0bEwOKm\nuPvuWOhRRKqKWlKSvmnToqPE3LlwxBGf3P+HP0RvP4CrropWVP3lwiOPjIUY8736aizemN/CMoOb\nb869T0RKSi0pyY7dd4/VgBsKKIAvfCHubQGcfHIEzZAhcN998Nhj63ewuPLKaIUddljMiJFP/4gR\nqToKKal8u+4avQTdo1UFMGpUzGIxYgRMnBhhdf318Pjjsf+vf41ygI8+iuc1a8pfdxHZIuqCLtXl\n1FPh2GNhq63i9aGHxuDhhjz6KBx8cO5+19tvl6WKIlI8CimpPh075raPPTZ6B7ZtGy2pq66CCRMi\noJ54Av78Zxg5Mo5dsiSd+orIZlPHCckO95jotkePmMS2Q4co32cfOPPMCLC//z3dOorUCHWcEClk\nFgEFcTmwXTs47jh48MG47DdvHlxxRQwWPvTQmEPQPcZgLVuWbt1FpEFqSUl2rV4NrVrluqLPnRtz\nCLZokRszdfnl0b39/PNjNozly+Hee2PJERHZbMVqSSmkpLaMHw/XXgtXXx2ztz/4YCwhsvXWEWjz\n5sVxo0bBrbfGrO3bbAO33x6zYrRtm2r1RaqFQqqAQko2y5FHRlj9+tfRVf3ss2NKpp/+FGbOhGee\ngaOPzq08vGBBhBbEe7797ZhIV0TWo5AqoJCSolm9OsLrwQfj9QUXwM9+ltt//vnwk59Ap04xhdMR\nR8Qs7iLyMYVUAYWUFJU7XHNNzHAxY0Y83n47Lg1+97sx+W39ulj1x4vIxxRSBRRSUhLLl0PnzuuX\nPfAA/OAHsR5W/XRNQ4bAbrvBn/5U9iqKVCKFVAGFlJTVunWxTpZ7XAqsvxx4zjnRxf3002NC2xNO\niLkJV6+Oe1v1k+GuXRvPzZunUn2RUlNIFVBISaruuWf9CXK7d4e33oJhw2JWjF/9KsLrzTejJ2GP\nHvCNb8Bll6VXZ5ES0mBekUpy+OG5RRkhevzdfDNMmRKT3y5eHOUHHBAT3y5dCi+/nE5dRapIai0p\nM5sPrADWAmvcfZiZdQVuA7YF5gMj3X15cvwY4DvJ8We4+8SCz1NLStK1dGlcArz+evja16B37+hc\nse++sf/rX495BXfZJZYXad8evve9CLQvfQkGDvzkZ86YEc+77Va+3yFSBFV/uc/MXgWGuvvbeWUX\nA2+5+8Vmdi7Qxd1Hm9nOwM3AZ4FtgIeBT7v7urz3KqSksi1dGp0tTj8dLrwQxoxZf/9JJ8H//E90\n1PjZzyKgJkyA1q3hww/TqbPIZipWSKU9C3rhDzgc2D/ZHg9MBkYDRwC3uPsaYL6ZzQWGAVPKVE+R\nLde9O5x2GhxyCGy7bbSeunWLAcMjR8ZMGN26xXF/+AMcf3y8b/XqeLRunW79RVKQZktqHvAucfnu\nKne/xszecfcuyX4D3nb3Lmb2e2CKu9+U7Psj8IC7T8j7PLWkpLq98EIMCq6rg0suiZnb163L9QA8\n4QQ48cRYBLJr1yhbty6mbirsJi+Ssix0nNjX3fcADgFON7P98ncmibOh1FEiSbYMHhwDiF9/PQIK\n4h7Xf/xHBNPEibD//vCpT0XPwPnz4xJhly5w223Qrx/ccUe0ukQyIrXLfe6+KHl+08zuJC7fLTGz\nnu6+2Mx6AUuTwxcCffPe3icpW8/YsWM/3h4+fDjDhw8vTeVFSuWEEz5Z9stfxsMdTjkFDjssAmvQ\noGhRDRsWY7B69YqOGKefDj//eUyWe/rpMHYs7LVXuX+J1JjJkyczuX4V7CJK5XKfmbUDmrv7SjNr\nD0wEfgYcCCxz93FmNhroXNBxYhi5jhM75l/f0+U+qTlr10ZLa9myWFrk2GPhuediyqZBgyLUZs+O\nY5s1y82Q0b9/qtWW2lDVvfvMbDsgGXpPC+Amd78w6YJ+O9CPT3ZBP4/ogl4HnOnufyv4TIWUyLp1\nMQFumzYxk/spp8T8g3vuGaHWoUN00nj//RhYfOaZ8NWvwjvvxCKRa9bEgpEiW6iqQ6oUFFIiBerq\nYoHHfLNmwQ9/GMH03nvwm9/EPa76FteAATHI2Lb47xapcQqpAgopkc3w+ONw7rlw1lkwaRI89BD0\n7AkrV8ZaWSNHRkeMf/wDvvIV9SKUJlNIFVBIiRTBX/8Ko0dHr8Hx42HatBi31a9fDCiePDkuJU6Y\nEDNp7LBD2jWWCqWQKqCQEimB2bNj/NbIkXF/67rr4jIiRG/CCy+ERx+Ne1uDB6dbV6koCqkCCimR\nMpg9O2Zwb98+gunKK6O8S5cYwzV0KNx6a4TZ0KHRq3DJEpgzJ3oV9u27wY+X7FBIFVBIiaSgri5m\nvHj5Zfi//4twOvxwGDcu9vfrB6+9ljt+4cLoVbhkSXTqaNMG9tknnbpLSSmkCiikRCrI/PnRurr/\n/hhUvMsucMMNsehj167Qpw9Mnx5BddZZMZ9h27aw3XbR2/D116PVpV6GVUshVUAhJVLh3KMF1bNn\nvK6ri96FDz0Ev/89rFoV5e3axTiugQNjOqizzooZ4du1i9k2OnZM7zdIkymkCiikRKqcO3z0UW5g\n8W23wYMPxnyEX/hCHPPYY3DUUXGJ8Gtfi8HHO+6oFlcFUkgVUEiJZJx79DR89tkIsIcfjkuHbdvG\npcEOHWIw8kEHxSrJc+ZES6xVq7RrXpMUUgUUUiI1xD1aT2vWwNy58Mor8MEHMa5rypQYz9WlS8xr\nOHhwPOrqYO+9Yy2vfv2i04ZZBJtaYkWnkCqgkBKRj61bF8Gzdm3MljF7dtznmj07ehu+9lquh+HK\nlbGgZLt2MaN8t27QsiVsvXV0+GjePI4bODBaZe++GzNv9OypVtoGKKQKKKREZJO5w/Ll0cpavhym\nTo0QWrMmQmzmzDhm1arosVhXF2G2alXsb98+gqpDB+jdO8KtY8d4dOgQ4deqVTxatMg96oMvf7uh\nsg1tN2uWe5htfLupx21ou/4B679uoCWaleXjRUTSYxaXBSFaUAMGNP29a9dGsK1ZE509li6N7ZUr\nY+zYihXREeSjj2L+w9WrozVXVxePtWs3vL2h/WvWRGvRPZ6bur2576kvq28I1G8XNgzqA+vf/q04\n/31QS0pERLZUfmglfw9by5ZqSYmISAVo5JJfMTQryaeKiIgUgUJKREQqlkJKREQqlkJKREQqlkJK\nREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQq\nlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJK\nREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQq\nlkJKREQqlkJKREQqVtWElJkdbGazzWyOmZ2bdn1ERKT0qiKkzKw5cBlwMLAzcIyZDUq3VtVj8uTJ\naVehoun8NE7npnE6N+VRFSEFDAPmuvt8d18D3AockXKdqob+MG2Yzk/jdG4ap3NTHtUSUtsAr+e9\nXpCUiYhIhlVLSHnaFRARkfIz98r/+9/MPgeMdfeDk9djgHXuPi7vmMr/ISIiNcTdbUs/o1pCqgXw\nMnAA8AYwFTjG3WelWjERESmpFmlXoCncvc7M/h34G9AcuFYBJSKSfVXRkhIRkdpULR0nNqjWB/qa\nWV8ze8TMXjKzF83sjKS8q5k9ZGb/NLOJZtY57z1jkvM128wOSq/25WFmzc3seTO7N3mtcwOYWWcz\n+4uZzTKzmWa2l85NSH7rS2Y2w8xuNrPWtXxuzOw6M1tiZjPyyjb5fJjZ0OSczjGz3230i929qh/E\n5b+5QH+gJTANGJR2vcp8DnoCuyfbWxH37wYBFwPnJOXnAhcl2zsn56llct7mAs3S/h0lPkc/Bm4C\n7kle69zmdM6OAAAEvElEQVTE7x0PfCfZbgF00rlxkt83D2idvL4NOL6Wzw2wH7AHMCOvbFPOR/2V\nu6nAsGT7fuDgDX1vFlpSNT/Q190Xu/u0ZHsVMIsYR3Y48ZcQyfORyfYRwC3uvsbd5xP/Aw0ra6XL\nyMz6AF8B/gjU9zaq+XNjZp2A/dz9Ooh7v+7+Ljo3ACuANUC7pONWO6LTVs2eG3d/HHinoHhTzsde\nZtYL6ODuU5Pjbsh7T4OyEFIa6JvHzPoT/9p5Cujh7kuSXUuAHsl2b+I81cv6OfstcDawLq9M5wa2\nA940s+vN7Dkzu8bM2qNzg7u/DfwaeI0Ip+Xu/hA6N4U29XwUli9kI+cpCyGlnh8JM9sKmACc6e4r\n8/d5tK03dK4yeR7N7KvAUnd/nlwraj21em6Iy3tDgD+4+xDgPWB0/gG1em7MbAfgh8Slqt7AVmZ2\nXP4xtXpuGtOE87FZshBSC4G+ea/7sn5S1wQza0kE1I3ufldSvMTMeib7ewFLk/LCc9YnKcuifYDD\nzexV4BbgS2Z2Izo3EH9OFrj708nrvxChtVjnhj2BJ919mbvXAXcAe6NzU2hT/hwtSMr7FJRv8Dxl\nIaSeAQaYWX8zawWMAu5JuU5lZWYGXAvMdPdL8nbdQ9zsJXm+K6/8aDNrZWbbAQOIm5mZ4+7nuXtf\nd98OOBqY5O7fQucGd18MvG5mn06KDgReAu6lxs8NMBv4nJm1Tf58HQjMROem0Cb9OUr+n1uR9CI1\n4Ft572lY2j1GitTr5BCiR9tcYEza9Unh93+euN8yDXg+eRwMdAUeBv4JTAQ6573nvOR8zQa+nPZv\nKNN52p9c7z6dm/itg4GngReI1kInnZuPf+s5RGjPIDoFtKzlc0NciXgD+IjoB3Di5pwPYGhyTucC\nl27sezWYV0REKlYWLveJiEhGKaRERKRiKaRERKRiKaRERKRiKaRERKRiKaRERKRiKaREisTM1ibL\ngdQ/ziniZ/fPXyJBpFZUxcq8IlXifXffI+1KiGSJWlIiJWZm881snJlNN7OnkslL61tHk8zsBTN7\n2Mz6JuU9zOxOM5uWPD6XfFRzM7vaYmHLv5lZm+T4M5LF+V4ws1tS+pkiJaGQEimetgWX+76RlDux\n1MNngMuA+vkVfw9c7+6DiQUZL03KLwUecffdiQlfZyblA4DL3H1XYDnwtaT8XGLRy8HAySX8fSJl\np2mRRIrEzFa6e4cGyl8Fvuju85PZ6he5+9Zm9ibQ093XJuVvuHs3M1sKbOOxiGf9Z/QHJrr7p5PX\n5wAt3f1XZvYAsIqYqPMud3+v1L9VpFzUkhIpv/x/GTa4xlUj5avztteSu6d8KHA50ep62syab3EN\nRSqEQkqkPEblPT+ZbD9JLB8CcCzwWLL9d+BUADNrbmYdG/vQZLmDfu4+mViwsBPQvqg1F0mReveJ\nFE9bM3s+7/UD7n5est3FzF4APgSOScp+AFxvZmcTi8WdmJSfCVxtZicRLaZTiKW5C6/NO9AcuNHM\nOhGtr9+5+4oi/y6R1OielEiJJfekhrr722nXRaTa6HKfSOnpX4Iim0ktKRERqVhqSYmISMVSSImI\nSMVSSImISMVSSImISMVSSImISMVSSImISMX6f+5I6h05kVFUAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(range(len(cost_avgs)), cost_avgs, color='red')\n", + "plt.ylim([0, 2000])\n", + "plt.ylabel('Cost')\n", + "plt.xlabel('Epochs')\n", + "plt.tight_layout()\n", + "plt.savefig('./figures/cost2.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training accuracy: 97.74%\n" + ] + } + ], + "source": [ + "y_train_pred = nn.predict(X_train)\n", + "acc = np.sum(y_train == y_train_pred, axis=0) / X_train.shape[0]\n", + "print('Training accuracy: %.2f%%' % (acc * 100))" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training accuracy: 96.18%\n" + ] + } + ], + "source": [ + "y_test_pred = nn.predict(X_test)\n", + "acc = np.sum(y_test == y_test_pred, axis=0) / X_test.shape[0]\n", + "print('Training accuracy: %.2f%%' % (acc * 100))" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEZCAYAAAA32jqgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYFNXVh9/TA8POsDnIvqgICIyAioQACmrQoBIlRlAR\nRI1JBI2CRFQCiAvGaFxQQdwQP0WDGwIGZBkQCQRlF1D2fWeYYRmY5Xx/VHXb03T3dPdUzXQP932e\nerq77nbq1/fWXeuWqCoGg8FgMMQDnpI2wGAwGAwGL6ZSMhgMBkPcYColg8FgMMQNplIyGAwGQ9xg\nKiWDwWAwxA2mUjIYDAZD3OBopSQibURkkZNxns0YPZ3HaOosRk9nMXrGUCmJyP0iskxEskXkHX83\nVV0FZIhIzzDh54vIwBhsRUT6iUh+rOEjTONiEfleRI7b15nmYlrJIvKWiGwVkUwRWS4iPbzubugp\nIs1E5AsR2S8ih0TkaxFpVsRLiSRd1/87v7Qmi8geW9PNIvKY182tPCoi14vIGhHJEpFFItKiCJcQ\nLp2advwHReSonWd6uZFWkLQvsMv9+95zpaTMJ4nIGBHZZeeZH0QkxcX05ovISTuvZInIOq+bi/mz\n2K6xqPfQWHpKu4AngbdDuH8A/DFM+Jie1hWR6sBwYE2scUSQRjLwBTAJqAa8B3whImXdSA8oA2wH\nuqhqVeBx4GMRaeTnx2k9U4DPgWZAbWAp1jW7RnH8dwE8AzSxNb0WGORf2eOwpiJyATAZuBdL32nA\nlyKSFJXVkXEMuAtIVdUUYCRWnqnsQlqBjMPKL4H6JGyZtxkFXA5cbueZ24FsF9NT4C+qWsU+Ahsw\nbuhZLNfoyD1UVWM6sCqmd4KcrwecAMoGcXsKyAVOAlnAy1Gk9wbwJ2AecFcYf+/afmcBmcB8oGGE\naVwD7Aw4tw34Taw6xaDrSuB3buvpF0cNIB+o7rSe0f53Lul5IbATaOeWpsD9wFd+v8WOv5tbmtrx\neIDrgd1Asss63gpMAf4OvB/glshlvrptV5NizJPzgIFh3J3On1FdYxH1LPI9tChzShLspKruAnKw\nbgaBbo8BC/mllTAYQESmicgjIRMSuQxohyVUJPQFRgO1gBVYLQ9vXOHSughYFXBupX3edUSkNlYP\nZq33nBt6BtAF2KOqR8L4iVXPWP47RxCR10TkOJaWY1T1B6+bC5oqBcuDx/4dLt/ErKntZxXWjeld\nrEbM6XD+i4KIVMVqaf+VIOU+wct8a6yb/O/tId8NIvLnCNMsCs+IyAER+VZEuvo7uKBnLNdYYvfQ\nMpF6DEK4LmQWVtctFAUytqpeH9KjNQQyDutPUJGgdWEgX6nqt3b4x4CjIlJPVXeFSwuoDBwNOJcJ\nVIkk0aJgd28/AN5V1Z8CnB3TMyDN+sCrwEOFeI1Jzxj/O0dQ1T+LyF+ArsC/ReQHVV3q58VJTb8B\nxto3l8XAMCAZqBgmTKx51GtPG3uo5I/AVBFprqrHCgsXI08CE1V1t4iEKveJWubrYw25XgA0xmoU\nzhGRn1T1m0gSjoFhWI2l00AfYJqIXKyqm/38OJk/Y7nGEruHOt5TsqkCZIRxj2ZM9M/AqoAbSri0\nFWu4xvqhehw4DNSNIK0soGrAuRQsUV1DRDzA+1hjvPcH8eKknt40z8Hqno9T1SmFxB2rntH+d46i\nFvOBT7AKvz+OaaqqG4A7sSr43UBN4Ef8dAsSd6ya+qd7WlVfwcq33aMJGykicrEd97+8p0J4TdQy\nf9L+HK2qp1R1NfARcF0U9kaFqi5V1eOqmqOqk4BFQdJzUs9or7FE76GO95REpB5WK3FDNOHC0A3o\nKiJeAWsAbUUkzdt1DTQBaOBnT2U7zO4I0loLPBxwrg3wSpQ2R4xYzcC3gHOA61Q1L8DdaT29E8iz\ngM9V9ZnCvBO7ntH+d25RFjjk/eGGpqo6FZhqx18NGAj8L4T3omgajDLA8RjDFkZXrNb1drvHUhlI\nEpEWqnoJJHyZDxxq8lJir09wQc9or7FE76GxLAlPEpHyWAUhSUTKBawy6grMUdWcEFHsA86LIsn+\nQHMgDbgYWIa14uix0EG4TkQ62cMbTwKL7XHawpgP5InIYPu6BmMtApgbhb3R8jrW9d2gqqeCuDuq\npz0/8B/gW1UdHmGwWPXsT/T/XZEQkXNE5FYRqWTn1d8Av6fgCkOn8ygi0t5O7xxgAvBFkGFYf2LS\nVEQ6iMivxXqcoIKIDAPKA/+Nxt4omAA05Zf/8A1gOvAbPz8JW+ZVdRPWHM1jtqYtgD8AX0Vhb8SI\nSIqI/EZEyotIGRG5DegMfO3nzVE9Y7zGkruHRroiwm8lxUg7Ef9jhJ/7dKBnmPCXY7UADgP/ss/N\nAP4WYfqFrcR5B+tGPwurKzkfaOTnHjYtfikEJ+zPtGg1ikLLRrZ+J2xbvUcft/TEGmbKx1pa7E0v\nE6jvhp7R/HcOaVrLtvEI1vDHUqwK39+P43kUq9BnYvXIXgcquJFHsRamrLDTOmBfy0VuahqQ/t+B\nSW7rGU2+caDM1wVm2mE3Afe4nD+X2v/fEeA7oHsx5M+Ir9EBPYt0DxU7EkcQkTbA66raybFIo7fh\nHawliU+UlA1OYfR0HqOpsxg9ncXoWbQ5pTNQ62nkEhPTpviWeLmM0dN5jKbOYvR0FqNn6dyQVSnB\nScpSiNHTeYymzmL0dJYS1dPR4TuDwWAwGIpC1MN3YR6eK9WoqitdWqOnsxg9neds1NTo6TyRahrT\nnNLZ1rtyeycCo6ezGD2dpyQ1zcnJ4Z577uH888/n8ccfdz290q5nSRCNplEP34mIno2CutmyN3o6\nGrfR0/n4S1TTgwcPkpqaSsWKFTl48CDly5d3Nb3SpueePXto27Yt+/btY9u2bTRs2LDY0vYSjaal\ncaGDwWAwGBKUuK6UsrOzz7pursFgCE7dunXxeOL6lhV3DBs2jEaNGnHgwAFat25NtWrh9niND+L6\nH77zzjsZOHAgy5cvjyrc8ePHyc/Pd8mqxCc/P9/o4wBZWVm88sortGzZEhFBRBg6dCgLFiwgLy+v\n8AgMUXHrrbeSnJxc0mYkFFOmTCEvL4/WrVuzYMECqlYN3Cs1/ojrSunSSy/l3XffZcOGUPsSBmfC\nhAk8//zzLlmV2CxfvpzU1FRat27NN998Y3qiUaKqpKenk56eTpcuXXjggQdYv369r1L65z//yRVX\nXMGll15KVlYWWVlZJW1ywjN+/HjKlSvH4MHFuYdvYjNixAhGjBjBrl27aNOmDfPnz0+IXhLE+UKH\nTz/9lN69e/N///d/3HrrrRGFWbt2LZdccgmnTp1yrDdQmibmMzIyGD9+PBMmTGDr1q307NmTt956\ni1q1ahWbDYms54wZM+jZs6fvd0pKCg8++CAtW7YErPz31FNPkZeXx5/+9CcAXnnlFVeHnUrbxLyX\no0et1/Kcd955iAgHDhwolnRLg57nnnsuAAcOHGDnzp3UqVPH1fQKIypNY9hQUIuL9u3bq4johx9+\nGHGYF198UUVEr7jiCsfssK/ZrQ0aHbMzWlavXq01a9bUKlWqaGZmZrGlm6h6Lly4UNu2basioiKi\naWlpeuTIkTP8fffdd9qwYUOfv4yMDNdsUnVXTy3BPDpr1iydNWuWioi+++67xZZuous5a9YsLVOm\njJYpU0b/8pe/aF5ens8tMzNT58+fr/Pnz4/bMu/o3ndOsXfvXgA2b95ciM8zmTZtGoAZvouAVq1a\nMX36dDp16sSWLVto06ZNSZsU18yaNYsVK1ZQt671rrN58+YFHRLp2LEjqamp7Nixo7hNLDUcP36c\noUOHAtYCh969e5ewRYnD4cOHfaNEnTt3xuPxsHLlSkaOHMnKlSvZtm0bAI0aNSIlJQWA+vXrM3z4\ncFq0aFHiw3xxWSl5FzZkZFgvXox0ci43N5dTp05RsWJFLroo4lfCn9V06NCBNm3asGzZMlMpRcgD\nDzwAQPXq1UvYktLLkiVLWLXKejfdH//4RypVqkROTg75+fmUK1euhK2Lb0aPHu37/tvf/pbly5fT\nqVMnTp0q+Lo2b+UEsGrVKmbMmEG7du2YPXt2iVZMcb3QwWAwGAxnF3FXKZ0+fZpHH32URx99FICB\nAwdy3XWhXiVfkKVLl/Ldd9/xwAMPuP7UdyLhXQX24YcfMmDAAB5++GFWrFhBdnY2e/fuJSMjg+uv\nv76kzUwY6tevT/369cP6ufPOO33fZ86c6bZJpYrTp0/zxhtv+H4/+uij5Ofnc++993L99ddz8uTJ\nErQuvpk7dy4//fTLC4/HjRtH3759OXXqFD179mTp0qVs3br1jOPZZ5+lQYMG/PDDD4wfP74Er4D4\nW+gwbNgw3wSxiOiuXbsiDtunTx8VER0+fLijNpGgE/PZ2dk6a9YsrVatmlarVk1TUlK0WbNm2qRJ\nEwW0QYMGetVVV+m9997rmg3BSFQ9n3jiCRURbd26tbZu3Tqs37vvvtuXh4cNG+aaTaqJPzEfyIQJ\nE1REtHfv3tq7d29VtRblxHJPiIVE1vO9995Tj8dzxlGvXj09fPhw2LAbNmxQj8ejycnJ+t133zlq\nVzSaxtWc0vfff8+ECRN8vy+44AJSU1MjCpudnc3OnTvdMi0hWbNmDZ999hljxowB4PbbbyclJYW8\nvDxWrlxJ+/bt2bFjB7fccgtHjx71TXoawrN161YAfv75Zy644IKgflasWFGMFpUe9u7dyzPPPAPA\noEGDADhx4gR9+/YFoEGDBiafRsm5557L3LlzC50DrVOnDnXq1GHPnj0sWLCAjh07FpOFAURae6nL\ntXx2drZeeeWVBXpJO3fujDj8zp07feG2bt3qqG0kYMt+48aNWq1aNV2+fHlQ9wULFiigzZs31+Tk\nZK1bt65u2bJFt2zZ4oo9/iSinqqq27Zt0wYNGvjyWYcOHXT69Om6YcMG3bBhg37zzTd6yy236C23\n3KJVqlTx+evXr5/m5ua6Zpebemox9ZRyc3P1+++/10aNGvl0O336tJ4+fVqXLFniO9ekSRPNzMzU\nnJwc12xJZD0De0p16tTRcePGRRze21uqXr26o3ZFo2ncCHro0CFfxrv77rv17rvvLrC+vjD8K6VD\nhw45alsi3kSvvvpq7dy5s2ZnZ5/htmLFCm3YsKF27txZjx07psuWLdNmzZpphQoVtEKFCrp//35X\nbPKSiHp6mThxYoGGU6RHVlaWazYl8k1UVfXkyZN68803R6Vn06ZNdc2aNa7Yk8h6BlZK69atiyp8\nZmZmiVdKcTF8t3r1at8WIhdddBEvvvgiAB6PB1UlJyfH59fj8fjW4JctWxYo+K6OgQMHmqW6QOPG\njenRo8cZy2ePHj3KjTfeSMWKFZkxYwaVKlWiffv2LF682Nddv/rqq0lPTzfDJEG48847ad++PQAv\nvPACq1atok2bNsyePRuwtBs8eDDLly/nj3/8Y0mamhBkZ2czcuRIPv30UwAqV67M2LFjqV69Om++\n+SZgPQ8WSLly5Vi2bJl59KMQCluQE0g8bIsVF5XSJ598Qnp6OmCNH/s/+JqXl8dTTz0FWL26Ll26\nsHDhQgDmz58PWM/azJkzB7CeHSmOl3TFO4899hjdu3encePGtGvXDrDmOfr370+TJk2YOnUqlStX\n9vmvUaMGixcvBqyHP7t27crSpUvNBpgBlClThosvvhiASZMmcezYMSpXrszhw4cBS0ewbrb+7Nq1\niwsvvLB4jY1z8vLyGDlyJM8995zv3Pjx4+nTpw+5ubm+OSWwGp4dO3bk+uuvp1evXjRu3Ng8r+Qw\neXl5vvnne+65p8TsiLsl4QaDwWA4i4l0nE9dHA+tU6dOxGPJrVq10qZNm+pll13mO3fllVfqeeed\n58oiB9XEnQN55ZVXVET0nnvu0XvuuUfLly+vzZo1K3TPq0OHDmlKSorecccdevr0acftSlQ9o2Hh\nwoUF8u3nn3/uWlpu6qkuarp3794CGj344IO+eeQ5c+YooICKiN5+++2u2BCMRNVTVXX37t1asWJF\n35zS5MmT9eTJkxGF3bVrl3o8Hq1WrZru3bvXUbui0TQuBP3ss898GfO8887Tq666Sq+66irfcwpv\nvvmmrlq1SletWqUnT570rb5Zs2aNrlmzRu+8804VEe3UqVPEf0A0JOpN9NSpU9qrVy/t0KGDdujQ\nQb/55pugCx+CsWrVKq1cubJefvnljq90SlQ9jx07pq1atdKxY8fq2LFjg27G6iWwUjILHc7koYce\n8unTvHnzAhvXTp482ef217/+VfPz812xIRiJqqeXwYMHF1jskJaWpp999lnQhWN5eXmal5en+/fv\n17S0NPV4PPrII484blM0msbFnNINN9zAli1bAGtOqEKFCsAvCxkC8e7W4J3kbNy4MQDdunUzOzn4\nkZyczCeffOKbY0tKSoo4bOvWrdm0aRNNmjThtttu46OPPjrr5+oqVKjAkCFDGDBgAACZmZkMGTKk\nxDewTEROnjzJ559/DliLFmbOnFlgYc3WrVt9Zblfv35nfd6LhqFDh/r0mjhxIqtXr+bmm29m0KBB\nnHPOOT5/1157LZMmTQKs16ucc845TJkyhU6dOpWI3T4irb20mGr5WDh+/LgeOnTIlaEm1cRt2TtB\nenq6JiUl6SeffOJYnIms54kTJzQtLU3T0tJURLRt27b6+eef69KlSwv4Gzt2rK+l3759+4h7qLHg\npp7qkqZZWVm+Yfsff/wxqPuOHTt0x44djqddGImoZyh2796tDzzwgCYnJwfd6cF7NGzYUKdPn+6a\nHdFoGtcv+YsXEvmldE7wySef0KdPH7777jsuu+yyIseX6Hp6Xz7XtWtX307WZcqUKdDSz8jI8L0S\nfcaMGfTo0cM1e0rDS+niidKo5759+3jmmWd45ZVXfOcaNWrE8OHDAWu0KtLdc2IhGk1NpRQBiX4T\nLSr5+fnccMMNrFu3jk2bNhU5vtKiZ2ZmJqNHj+aFF14I6n7ppZcCkJ6e7uqwcmm8iZYkRk/nMZWS\nw5SWm2hR8HatnXitd2nSU1XJz89n3rx5rF+/nn/+85+kpaXRunVr3073FStWdNUGcxN1FqOn80Sj\nqXlOyWAwGAxxg+kpRUBpatnHA0ZPZzEte2cxejqP6SkZDAaDISExlZLBYDAY4oaYHp41D7I5i9HT\nWYyezmM0dRajZ2iinlMyGAwGg8EtzPCdwWAwGOIGUykZDAaDIW4wlZLBYDAY4gZTKRkMBoMhbjCV\nksFgMBjiBlMpGQwGgyFuMJWSwWAwGOIGUykZDAaDIW4wlZLBYDAY4gZHKyURaSMii5yM82zG6Ok8\nRlNnMXq6i4g8LyL3lbQdxUlMlZKI3Coi60TkmIhsFJFfA6jqKiBDRHqGCTtfRAZGmd6vReR/InJU\nRDaJyD2x2B1lmv1EJD9aW2NI55iIZPkduSLyMrijp4g0E5EvRGS/iBwSka9FpJkDlxIqvYtF5HsR\nOS4iy0Qkza20/NKsLyLT7OvbIyKviEgSJL6mItI5IL9k2fn0d26kZ6fZQkTmikiGiPwsIr28bm6V\neb+wrpdDEbleRNbYWi4SkRYOx3+/nfezReSdIO7dRWS9XUbmikhDP+fngeEiUjZE3I1tfSK+l4tI\nfxHJC8hDXaK/sojSutW+tqMiclBEPhWRuuHCRF0picjVwLPAnapaGegMbPbz8gHwxzBRRLXZnn0z\n+QyYoKopwB+AF0SkTVSGR5dmdWA4sIYo7Y0WVa2sqlVUtQpwLnAS+NjPi6N6AinA50AzoDawFPgi\nyjgiQkSS7bgnAdWA94AvQhUwB3kZOAjUAS4GugJ/9nNPWE1VdaE3v9h5pidwDPjajfREpAzWtXwJ\nVAfuBSaLyAV+3pzW05u26+XQvo7JWNeVAkwDvvQ2YhxiF/Ak8HaQ9GsBU4HHsPRdBkzxuqvqXmA9\ncEMhaUS7w+si/3ykqguiDB9xOkAX+97dCDgBvBA2hPc115EewHfAgDDu9eyEywZxewrIxbrxZgEv\nR5BeXSAfKO93binwhxD+3wXeAGYBmcB8oGGU1/gG8CdgHnBXtBrFegB3Ahvd1DNIHDVsfas7rSdw\nDbAz4Nw24Dcu67gB6OH3+zngjdKgaZC43gHeclHLVkBWwLn/AKPd1jPScljEPHo/8JXfb7GvpZsL\nWj4JvBNw7l7gW7/fFe30m/mdGw68HSLO7XZey7KPDhHY0R9YGIXd+cAgYBNwwC5PEsP1V8ZqmL4Y\nzl9UPSW79dAeSLW78TvsoZHyXj+qugvIAS4MDK+qjwELgb+oVTsPtuOdJiKPBEtTVXcDq4C7RCRJ\nRH6FVeN+G8bUvsBooBawAqsl572GkGnZ7pcB7bAyeXFzJ1avwofTegahC7BHVY+E8ROrnhdh/Xf+\nrLTPu8l/gL4iUkFE6gHXAjO9jgmuKX7+KgE3YxX04sSDVVkB7ugZQzmMVU+lYC/DY/92I48G681c\nhFUmLGNUTwAb8dMXq6cUati7s/2ZYuu7REQaisgREakfIowCbUXkgIhsEJHHI+gZ9sK697cDbgTu\nAoggLe/0SwZWg6EhMCxsSlHWdN5ey1KsYYqaWJXDmAB/O4Ffh4hjHjAwynQvw6qhc+wjZHisVtP/\n+f2uhNVSqxdBOknA/4DL/Gwtlp4SVkWbCzQK4uaonn5h69txB+11OqDnE8CHAecmA393WcsawA92\nXsknSCszUTUNiOcOYJPLWpbFaiEPtb9fA5wCZrqlZ7TlsIh59EKs4c+uQLKdZ/OAYS5oGaynNBF4\nJuDct0A/v99Xh/qfgcZ2HvdEYUcT730Gq/JbC/wtjP984Bq/338Cvonh+uti9WZfCucv2jmlk/bn\nK6q6T1UPYY0PXhfgrwqQESaeiMeH7ZbuV0BfVS2L1bIYJiKBafrHvdP3Q/U4cBhLkML4M7BKVZf6\nmxCprUXkDqwu9bYgbo7p6UVEzsHKIONUdUoYr0XRMwuoGnAuBavF5AoiIlg9pU+whkJqATVEZGyA\n10TV1J8zetZOo6o5WK3k3wJ7gL9izXnuDPDqpJ7RlsOY9VTVDVg6vgrsxmpo/8iZ1+cEwa7hGMHL\nSJbf78K0jQpV3eK9z6jqGqweZu9Cgu3w+76d6PMqao16PQH0C+cvqkpJreGIsH+WXYkkY43rB40m\nmjSBX2HNS8y2bfgJmI41JBPUBKCBnz2VsVrOuyNIqxvwO3vF1h477X+KvRrOZfoRZBjGBT29E8iz\ngM9V9ZnCvBO7nmuBwAUpbezzblELa5jhVVXNUdXDWC1pXyMmwTX1hmmA1bp3tVICUNXVqnqFqtZS\n1WuB87BGS7y2OK1ntOWwSHqq6lRVba2qtYCRWL2P/0Vpc0RJBTm3Fr+hOXtI9jwKlpEWWEOSkcYZ\nC4U1vhsGfN8VYzplsebMQhLLkvB3gEEico5dEP+KtWLFS1dgjt3CCsY+LNEjZQ1woYhcKRbnYa04\nWhkmzHUi0sle/fUksFitce/C6A80x8okF2OthBmJtTLGNex5srpYrftAHNVTRKpi9SS+VdXhEQaL\nVc/5QJ6IDBaRciIyGGsoYG6k9sbAQawW/Z/sOchqWC1h//ySyJp6uQNrBdWWKMLEhIi0FpHyIlJR\nRIZgDd2/6+fF6TLfn+jLYcx6ikh7O6+cA0wAvrAbv45gx10eKAMk2WXBO4fzGdBKRG6y/fwdWBGQ\nflf85kQDOIBVpqLJr9eKSG37e3PgcazVo+EYIiLV7MbQYPxWCBaSVl87DCLSCGvhy9SwgWIYFywD\njAOOYBX+fwHJfu7TgZ5hwl+O1aI6DPzLPjeD8GOa/bC61JlY3chnwvh9B3gdq9WahXVjbOTnHjat\ngLiKZU4JazL3vRBujuqJdYPOxxo28K7YyQTqu6Env9xUTtifacWgZwesyfUjWIX2I+Cc0qKp7Wcd\nYVbBOqznc7YWWbZ2Td3Mo0HCFzanVNQ8utD+vw7Z8VRwWL+Rdv7wP0b4uXe3/88TWA22hn5udbDu\neWXCxD8K2G/n98uwejJZYfLfP4C9dn7dZNuXFCb+fKxVipuwGn3/wJ7DiiCtMbb9x4AtWI8TlQ+V\nlqpay/qcwn526HVV7eRYpNHb8A7WcN8TJWWDUxg9ncdo6ixGT3cRkeexHhMpidXAXhvygfNVdXOh\nnh2gjJORqfV0d4llTpviWpjgOkZP5zGaOovR011UdUhJ21DclMYNWRWXd2E4yzB6Oo/R1FmMnu5S\nrNo6OnxnMBgMBkNRiHr4TkTOylpMVV0ZIjB6OovR03nORk2Nns4TqaYxzSmdbb0r63lM9zB6OovR\n03nOJk2Nns4TjaalcU7JYDAYDCE4ePAgN998MzNnhnr0qWSJek5JRPRsrOXdHG5yQ8/c3Fzeeecd\nduzYwdatWwF4//33vdfia7nUrFmTl156iVtvvRWPp3jaKImoZzzjpp52/GeVpqVZz4MHD9K1a1c2\nbNhA06ZN+eknx54RDks0mpqeksFgMBjihxieTtbi5LLLLlNAb7755mJN1x/7mt16Wt4Vm4cNG6Ye\nj+eMo1evXvrQQw9p69atC5z//e9/r5mZmZqZmemKPf4kop7xjJt6aglpunv3bm3QoIE2aNBAX3nl\nlWJNuzTqqar68ccfa7NmzTQpKUmTkpK0V69exZZ2NJo6+vCs04wZM4Zly5YhIlx55ZUlbU7CMGrU\nKF566SWefvppbrrpJho3buxzS0pKwuPxkJeXR35+PtnZ2fTr14/PPvuMn3/+GYD09HSqVg3cuNgQ\nSHZ2Nvv27WPy5MkAPPHEEwUmdOvXt14xs2rVKlJSUkrExkQkNzeXjh07snOntfdzcSw8KO3k5+ez\nYsUKNm7ciIjQpk0b3nuvuF/DFSGR1l5ajLX86NGjdfTo0ZqUlKQion/4wx80Ly/P5z5jxgxNSUkp\ncDzyyCOu2UOCtewfeOABbdiwoR47diziMF26dPH1mi655BJXe0yJpmcg6enpmp6eru3bt/e1OpOS\nktTj8Wjnzp31kksuKXB+y5Ytrtrjpp5aAi37n3/+WUXEdyxZsqRY0y9teqqqfvbZZ748mpSUpD17\n9izW9KMIDAMUAAAgAElEQVTRNO4E3bdvn6ampmpqaqqKiDZr1kw/++wzn/u0adM0OTm5QKbt3bu3\nrlq1yjWbEu0mumvXLv3xxx81Nzc34jBZWVnaunVr37DerbfeWqAh4CSJpqc/6enpWqVKFa1SpYom\nJSVp7dq1dfjw4fr555/rnj17NDs7W0+ePKmVK1f2VUqjR4921abSdBPNzs7W3r17q4hot27dtFu3\nbnr69OliS1+1dOmpqpqZmamdOnVSj8ejgHo8Ht28eXOx2pDQlVLr1q19lU3z5s113759Prcvv/xS\nK1SooCKiM2bM0KysLM3KynI90ybyTTQaFi5cqAsXLtSKFSuqx+PRdevWuZJOoup5+vRp7d27t/bq\n1Ut79eqla9euDZr3Tp8+7au0kpKSdOPGja7ZpFq6bqL//e9/VUS0Ro0aunbtWl27dm1Iv5s3b9ZJ\nkyZpdna2ozaUJj1VVTdt2lSgNz9gwICoGqxOkLCV0oEDB7RmzZq+SmnixIkF3G+++WYVEa1Zs6Zu\n3brVNTsCSdSbaKz069dPPR6P1qtXz5X4S7ueEyZM0KSkJE1LS9O0tDQ9ceKEq+mVlptofn6+3nbb\nbSoi+sQTT4T1O2/ePE1JSdGkpKQCDVcnKC16etmzZ4/WqVPHVynVqVNH9+zZU6w2JGylNGLECBUR\nHTBgwBm1+cyZM7VSpUoqIjp79mzXbAhGab+JBjJ+/Hj1eDxatmxZ/emnnxyPvzTruXXrVl8v6eOP\nP9aPP/7Y9TRLy0103bp1KiJauXLlsP6++OILTU1N1eTk5AJD+05RWvT0Z/jw4QXmlEaMGFGs6Uej\nadw8p3T48GFeeuklAKpWrUrVqlVJSkpi/fr1fPDBB/z+97/nxIkTNGrUiIsvvriErU1s8vLyChxW\nnvmFm266yedv165Y33p89qGqzJ49mxMnTlC9enU6depEp04l/VaHxMG7ivHee+8N6Sc9PZ17772X\nAwcOcP3119OrV6/iMi+hGTZsWIHfkyZNYvfuiN4WX+zETaU0e/ZsMjMzAdi1axe7du2ib9++XHLJ\nJdxxxx0cP34cgG3bttG1a1fWrVtXkuYmFNnZ2WRnZzNjxgyGDRtG7dq1SU5O9h39+/dn2LBhLFmy\nhCVLllChQgVq1KhR0mYnHF9//TX33XcfAO+99x5169albt26JWxV4vHYY2e+9Tw/P5+vvvqKa6+9\nlv3793PNNdfwwQcflIB1iUnVqlUZM2YM+fn5qCrbt2+nc+fOJW1WUOKmUjIYDAaDIW7GQz/66KMC\ny7wLO9LS0vTLL7/UL7/80hV7/CGB50B++umnAku9Izk6deqktWrV0kqVKrkySZ/Ieoajf//+mpSU\npM2aNdOcnJxiS9dNPbUYNX3sscdURM54Rm7Tpk3at29fX9kfNWqUq4tHSouegZw8edL3SENSUpIm\nJyfrqFGjdNSoUbpz505X045G07jZkHXKlCn06dOnwLmuXbtSuXJlFi5cSGZmJpdddhllypThu+++\nA6BBgwYA/Pjjj1SqVMlxm7wk6gaip06dolGjRmRnZwPQv39/X5f9V7/6FapKSkoKX3/9NRkZGTz8\n8MMAZGVlAVCpUiXfkKqTJKqe4di5cyeNGjVCRJg6dSo33nhjsaVdWjYQffzxx3n66ad55JFHSEtL\n4+233waseaScnBxEhIcffpjRo0dToUIF1+woLXqGYuDAgXz++eccPXrUd65evXqsW7eOihUrupJm\nVJpGWnupy7V8RkaGrlq1SseOHaurVq3SVatW6alTp1RV9corr1QR0XXr1unJkyd1zpw5WqtWLV/L\nacKECa7Y5IUEbdk/+OCD6vF49Pbbb9fbb7+9UP/ff/+9fv/991qpUiX1eDxapUoVV+xKVD1DcerU\nKe3fv78CessttxR7+m7qqcWo6ZgxY8KOjvTo0aNYeqClRc9wTJs27YzdSOJlF5e42fsuJSWF1q1b\n07p16zPcvHuIAZQvX55u3bpx00038eabbwLw1FNPcc899xSbrYlCtKtrvIsbypSxssUf//hHx20q\njcyYMYP333+fSpUq8Ze//KWkzUlYHn74Yc4991w+//xzfvjhB/bs2eNzu+iii/j00099edNQNFq2\nbFnSJoQkIRY6DBky5Ixzd911l+/7gQMHWLNmTXGaVOrIyclhwIABDBgwgKysLGrXrs3KlStL2qy4\n59ChQwwYMACAsWPH0qVLlxK2KHEpX768b2ipXbt2vvO//vWvee+991wdsitN/PTTT74jGDNnzuTa\na68t0DvJz88vZitDk1DNDv/5jQsuuIDKlSsDcOzYMbZs2UKrVq1KyrS45vTp04C1rDbYi/yys7Np\n164dGzZsAOCcc85h8eLF1KlTp1jtTDRUlWeffdY3B9ezZ88Stqh0kJ2dzfTp033zG6+99pop2xGS\nlZXF5Zdfzg033AAUHO2YMmUKX375Jbt27SIvLw8R8c0xf/nll67Oy0dFpON8WoLjoUePHtWWLVtq\n3bp1dfz48b7zgwcP1sGDB6uIuLoKjwSdA5kzZ46WL1/et6pu3LhxZ/jZvHmznnvuuQVW302fPt01\nm1QTV89AFi9e7BuTd3OX+sJwU08tZk2PHz+uV111lYqIzp07V+fOnVtsaXtJZD2PHj3qmyPy7t4Q\nOHeUlJSkNWrU0OnTp/v2D3WbaDRNiOG7qlWr8vjjj3Pw4EFGjhzJxo0bycnJ8T0UaghOt27dfC0m\ngAceeIARI0awZs0aVqxYwd/+9jcuvfRS9u/fj4gwe/ZsZs+eTY8ePUrQ6sRh7dq1vu+PPvpoAbdF\nixYVtzmlgkmTJjFnzhxSU1NJS0sjLS2tpE1KKEQk7Lu7GjduTL9+/Vi0aBHXXXcdlStX9o04xQ2R\n1l5aDLV8Yfg/q9CjR48CK3NMTyk4O3fu9K2+C/dsUnp6uqt2+JPIevozcOBATUpK0nHjxmleXp7m\n5OToggULtH379vrjjz8Wmx1u6qnFqGlubq726NFDy5cvr7t37y6WNIOR6Hpu2LBBP/roI9/RvHlz\nHTp0qH700UeuphuOaDSNO0HDkZmZqaNHjw66XNRUSqHJzs7W7OxsnTNnjt51110FKqMXXnhBd+3a\n5dq7k4KR6Hp6qVevniYlJemoUaN09+7d2qFDB61WrZq+9957pUZPLUZN9+zZE9GGrG5TWvSMJ6LR\nNCGG7wwGg8FwlhBp7aVxUstnZmbqyJEjC/SS7r77bt2+fbtraVJKWvbxQmnRc8SIEWdMIj/22GPF\nlr4XN/XUYtQ0JydHhw8frtdee22xpBeK0qJnPBGNpnGzzVA8Uxq3xSlJSoueJ06coHv37vzvf/+j\nc+fOjB07lrZt21K2bNliSd9Lad8Wp7gxejpPNJqaSikCSstNNF4wejqLuYk6i9HTeaLR1MwpGQwG\ngyFuiGlHBxHXGhFnJUZPZzF6Oo/R1FmMnqGJevjOYDAYDAa3MMN3BoPBYIgbTKVkMBgMhrjBVEoG\ng8FgiBtMpWQwGAyGuMFUSgaDwWCIG0ylZDAYDIa4wVRKBoPBYIgbTKVkMBgMhrjBVEoGg8FgiBtc\nrZRE5HkRuc/NNM42jKbOYvR0FqOns5yVehb2bgvgfmAZkA28E+BWFvg3sAXIB7oGuJ8LbAfKhoi7\nsR3OE+m7NoBbgfXAUeAg8ClQN9Lw0RzAG0CW35ENZDoQbzhNLwdmA4eA/cDHwLluamqHewLYAWQA\n84CWLml6p33tR+30xgJJLurZ0nY7bF/bIuDXbuvpF35OUcJHmEZT4CsgEzgAjHVLzwB/I+xr61aK\n8qfjZb6Q/OnVwz/Nx4pBT0fzTCFp/RXYY5f5t4DkcP4j6SntAp4E3g7hvgC4HdgLFNhIT1X3YlUg\nNxSSRjS7Ey4CuqhqCtAIOAG8EEX4iFHV+1S1ivcAPsSqJIpKOE2rYRWMRvaRBbzjZ5PjmorIDcB9\nQGegBrAYeD/S8FFSAXgAqAl0ALoDQ4oYZzg9dwG/t9OrDnyE1ZACXMujVgCR27A2PXZtg0kRScZq\nxHwD1AbqAZOLGG1hZR4ROQ/oDez2P5/o+dOlMl+onkBVv3Sf8rPHDT3dyDOh0voNMAzohnU/awqM\nChem0EpJVT9T1S+wWu6Bbjmq+rKqLgLyQkQxH/htCLcF9meGiGSJSIcI7Nmhqvvtn2KnuyeUfxHJ\nF5FBIrJJRA6IyHMSwxa9IlIJuBl4L9qwgRSi6deqOlVVj6nqSWAc0CnA23wc1BS4CPhWVbeqaj7w\nAVYPIyhF0VRV31DVRaqaq6q77bQCry8qCtHzqKpuUavJloTVqgzML/NxVk9EJAWrJ/EIhdwwiphH\n+wM7VfVfqnpSVU+r6uoIwwYlnJ5+vIp1s8kJ4jafBM2fAfE4UuYj1DPcvXg+zurZnyjyTBH1vBOY\nqKrrVDUDGG2nH5Jo5pRi3Wt9PZAWwq2z/ZlitxCWiEhDETkiIvVDGiLyaxHJwOp6NsQqHOHoBbQH\n2gE3AnfZ8RSalh83A/tVdWEEfiMlEk27AGsCzjmt6Rygo4hcICJlsTLSzELsckJTgK6ceX2xElJP\nO7+cxKokegc4O55HgaeB14B9kZkes56XA9tEZIZ9w5gnIq0iTLMwguopIr8HslU1VB4pLfnT6TIf\nrrxvE5EdIvK2iNQMcHNaz1jyTKx6tgRW+v1eBdQWkeqhEoqmUop1CCILa0gqGGf8Saq6XVWrq+rO\nkIaofquq1YD6WC21fxRiw1hVzVDVHcC/gD6RpuXHncCkCPxFQ1hNRaQN1lj60AAnRzVV1aVYrcEN\nWMOhNwMPhTe96JqKyF1Ymfz5wvxGSEg97fySgjV890lAS89RPUXkEqAj8EqkhhO7nvWx5llfAuoA\n04Ev7Jt3UTlDTxGpAjyFNQQbilKRP3G+zAfLnweAS7Aa1+2BKlg9QX+cvofGkmdi1bMy1lySl0z7\ns0qohIqjp1QFa3LScezhnyeAfoV43eH3fTtQN5p0RKQhVove6UopXMv+fGAGMNgeHvXHUU1F5H6s\nuZ36QDmsLvZcEakQJlhRNe2F1Zu4VlUPR2dx6GjDOarqCeBvQDOgtZ+TY3qKiAerh/SgPdQUkW3E\nrucJYKGq/sceEn0ea/6seaQ2hyGYzSOB91V1exh/pSF/ulHmg1Ugx1X1B1XNt6cl7geusYcOvTh9\nD40lz8Sq5zGgqt/vFPszK1SA4ugptQBWOBynP2WxRA5Hw4Dvu6JM4w7sMe0owxVG0OsXkUZYE5Gj\nVTWw1QTOa9oD+FBVd9uF4z2sRQEtwoSJWVMR6QFMAHqq6toY7A1FJNeehJXv/fOMk3pWxWrxThGR\nPcBS+/xOEQk3dxarnqv8f8QydxKGYNfeDRgsInvs62sAfCwi/r35hM6fNm6U+Wiu3f/e7LSeseSZ\nWPVcC1zs9zsN2KeqR0IFKLRSEpEkESmPtYooSUTKiUiSn3s52x3A/7uXroQe/z2ANfF8XmF2+KXX\nV0Qa2N8bYQ0lTC0k2BARqWaHGwxMiTQ9m37Au1GGCUk4TUWkHjAXeFVVJ4SIwlFNsTLpLSKSKiIe\nEbnDtm1jmDAxaSoi3bCGJ25S1WVR2BguznB6XiUiF9t+qmKt1Nygqv7X5pie9mRuHazClwZcZzu1\n45cKKhix5tHJwOUi0t2+5gdtm9dFGP4MCinz3bEWHqRh3Wx2A/di9Q69JGz+9MOxMl9I/rxMRC60\nr6sm8DIwT1X9exJO6xlLnolVz0nAQBFpYc8jPYHfauKgaOFrzEdiXbT/McLPfat9Ls/vs6HtVger\n21cmTPyjsJ7HOQJchlULZwH1Q/gfY8d5DOv5qGeB8mHiz8fqEm/Ceq7pH9hr+gtLy/bT0fZTqTCt\nIj3CaQr8nTOfW8j0C+uGphWBiVjL+o9iPVNxjRuaYlW4pwOub7qLevbGKmxZWKvuPgQauKlnQNjG\ndpkI+RyJA3n0d8DP9n83F2jhlp5B/G6h4HNKCZ0/bT+OlvlC8uetwGas+9lurIow1e38GU2ecUDP\nv/r9d28R4pkr7yF2IFcQkeeBjar6hmuJFG5DPnC+qm4uKRucxGjqLEZPZzF6OsvZqKerlVI8UJoy\naLxgNHUWo6ezGD2dpbj1PBs2ZC3dtW7JYDR1FqOnsxg9naVY9Sz1PSWDwWAwJA5log0gImdlLaaq\nTi619WH0dBajp/OcjZoaPZ0nUk2jrpTsyGMJlrA4++jHmRg9ncXo6Txnk6ZGT+eJRtOzYU7JYDAY\nDAmCqZQMBoPBEDeYSslgMBgMcYOplM5i7r//fjweDyKCx+PxHRUrVmTlypWFR3CWU6tWLUSErKyQ\ne0saYuTrr7/mkUce4ZFHHvHl0fr167N///7CAxsK8O9//7twT3FE1EvCRUTPxkk6N1eLFaee2dnZ\nvPzyy0ycOJFt27aRm5trbe0RMBGZlpbGt99+S8WKFR23obToec4553Do0CGOHj1KlSohd+J3HTf1\ntOMvNk2feuopxowZQ05OTtDFAJdccglLlixx1YbSoOfp06cBeOaZZ9i0aROTJjn9goPoiEbTmFbf\nOc2JEyd48sknmTdvHu3atePhhx8GrBUqtWvXLtECXxrIz7feoLBu3Tp++9vfsmPHL7vQp6Sk8Oc/\n/5nU1FQWLrTeZfbpp5+ycuVKhg4dyrhx40rEZsPZR4cOHfjhhx98+bVx48aAVVHddNNN3H333axZ\n49T7IEs3hw9bb4MZPXo0W7ZsKWFroqPEKqUVK1Zwzz338PPPP5OTk8PJkyepW7cuS5cuZeLEiYB1\nMy1Xrhxly5Zl4MCBDBgwAIDzzz+f8uUDNyM3hOLECestDW3atPH1iv7+97+TmppK9+7dueCCCwDo\n06cPAEuWLGHXrl3Urx/py2MNhqIxZswYX4XUrFkz5s6dS40aNQAoV64cYA03N2vWjLfeeosJEya4\n3mNKZAYPHgzAZZddRnJycglbEx0lVik9+uijfP/991x66aU0bdqUdu3acd1115GTk0PVqtY7oXJz\nc/nvf//LtGnT2L59O507d+bo0aOkpqbyj3/8gx49egDWMIohNK+++mqB3/fddx8PP/wwlSpVKnDe\nW9F7K7FOncK9/sdgcI6JEyf6ekipqakcPHiQOnXqFPDTqFEjLrroIvbu3cudd95ZEmYmBGvXriUn\nJweAxYsXB/Vz+PBhjh8/ToMGDQBYv369b6QEoH379rRr1859Y4NQYnNK+/bto2/fvlx88cX885//\njChMdnY2O3fu5KOPPuLFF1+kadOmAHzwwQc0a9asyDaFIpHnQI4dO+brCe3bt4+OHTuSnp5OmTJn\ntkd++uknAJo3t15AOXToUMaOHeu4TYmsJ1j5EKBOnTocPXrUN6fUvXt3Bg8ezI033uhq+oGUhjmQ\nZcuWcdVVV/kWjVSsWJG6da2Xmz788MNceOGFjBs3jqlTp1K1alWWLl3qy9dOk+h6jh07lv/9739A\nwUUOWVlZ9OrVC4BDhw5x4sQJ331z586drF692ue3du3atGjRgjlz5jhiUzSamtV3BoPBYIgfYnhh\nlTrFyZMn9fTp01GFyc3N1QMHDuhdd92lIqIion379tUPP/xQT5w44Zht/tjXXOSXfQU7nNQzGEeP\nHtVq1apptWrV1OPx6O233x7S77hx43TcuHHq8Xj0qquu0v3797tiUyLrqaq6evVqXb16tS//ZWZm\nqqpqixYtNDk52TXdQuGmnlpMmqqqHj58WD/++GP1eDxhj08++cRVOxJZz8zMTL3lllv08OHDevjw\nYd22bZseO3ZMVVX79OmjaWlp2rp1a5+W3jL/8ssvn6Fzv379HLMrGk1LdPVdtIsVVqxYQa9evTh0\n6BDHjx/3nZ81axZTpkxh9erVtGjRwmkzE5qqVavSoUMHAGbPns3KlSvJycmhbNmyBfzl5ub6lo3e\ncMMNTJ48+Yw5J0N4OnXqxPr165kxY4aZ84iB6tWrc/3117Nz507Gjx/P999/D8CMGTN8fm677TZ6\n9uxZUibGPXfddReffvqp7/e6dev46quvqFSpEpUrV2bmzJmoKkeOHAHgwgsvBCAzM5O3336b9evX\nc/r0afr06cPrr79eItcQV7V8YcyYMcPXOr388ss1NTVVU1NT9YsvvtD09HTX0iXBW/bp6emanp6u\nHo9HRUTHjRt3hp+5c+dq2bJltWzZsjp9+nRX7Ul0PUP1lH7++WcVEb3yyitdt8EfN/XUEizz69at\n03Xr1vla7g0aNNB9+/a5nm6i6rllyxZNSUnRK664Qrdv367bt2/XQYMG6eHDhyMKn52drQ8++KB6\nPB595JFHHLUtGk3jRtBIyM3N1QceeEArVKigK1as0KysLM3KynI93US/iZ46dUpPnTqlN9xwg4qI\n1qpVSzdt2lTAz4cffqjvvPOOvvPOO67bk+h6btq0STdt2qRlypRREdF7771Xt27dqtnZ2dq8eXNt\n3ry55uTkuG6Hl0S9iRbGzJkzdebMmQWGlBYvXux6uomq55AhQ9Tj8eiHH34YU/hDhw75dC7JSinh\ndnQ4efIkzz77LB988AFXXHEFAK+99pqra/ETfbWYl4yMDKpXr46IkJKS4uoKpnCUFj29Ozp4adWq\nFRs3biQ7O5sRI0Zw7rnnAlCpUiV69OiBx+OhZs2ajtuR6KvFgrF48WLfMF1GRobv/KJFi7j88std\nTTsR9XzjjTcYPHgw1113HZ9++ikeT3Rr2F577TWGDBnCE088wdChQxERkpKSHLMvGk0TrlIC66Ha\n1atX+56jufbaa3nxxRdde9iztNxEAZYvX07Xrl05fvw41apVY+bMmaSmppKRkcG0adN8/jZv3kzT\npk05//zzfQ/VOkVp0XPOnDkMGjSI9evXR+S/TJkybNiwgSZNmjhqRyLeRAuje/fuzJ8//4zzplIK\njnd/wF69ejF16tSIw7344osATJs2jd69e9OjRw/fozZOknDbDEWLx+MhLS2NTZs2AXDFFVcwcuRI\nXn75ZVf2aistnDp1iiNHjniHEMjIyKBjx45Wl9lv77saNWpQr149MjMz6d27d0mZG/d0796d5cuX\n89NPP/Gf//yHI0eO8Mknn7Bt2zbfw4sASUlJNGnShHLlyvkeDDeEZtmyZUE3BK5QoYLZciwE3jJ8\n+PBhjh07RuXKlcP63759O+PGjfNVSg0aNKBPnz5Ur169OMwNi3lOyWAwGAxxQ0IO34G1C+6iRYsA\n6N27N4cPH6Zx48aubD6Y6MNNGzduBOB3v/sda9euPWNHcFVlxowZvuX01apVIyUlxTV7El3Pwrjn\nnnt46623mD17NgDNmjXzbefiBok43BSKzZs3c+mll5KXl8e9994L4NvxpWPHjnz77beu25CIenqH\n7wBuueUW3n77bcDqXfqze/dupk6dypAhQ7j99tv51a9+BVijTeedd56jNvlTqofvsrKy+PLLL3n6\n6adZt26d73yFChXo27dvCVoWn2RlZfnG4L3PJtSuXZs//OEPrFmzhrlz5wLWMyKNGjUqMTtLE/36\n9eOtt97y/XazQipt7N27l4yMDPr27cvNN98M/FIpmWe/QtOiRQv27NnD0aNH+fjjj31D9K+99hpD\nhw717a5+8OBBnnzySbZt20ZKSsoZlVZcEOkyPe9BCS0P3bx5s44ZM0arVavmez7Ee9SoUUNnzZrl\nWtok8BLmgwcPFtDqtttuK7Bc+bbbblNA//3vf7tqhz+JrGck7N+/X0VEmzZtqk2bNtW8vDxX03NT\nTy1mTQcNGqQej0ebNGmi9erV03r16qnH49FOnTppRkZGsdiQqHoOGTJER40apd26dQu6E0ZycrI+\n99xzunPnTlfSD0c0msZ9T2nBggVMnz6d8ePHk5mZCVg72Aa+csEscAhO2bJlfUNxmZmZ3HjjjQU2\nYxURRISvv/7a1zI1FA2Px0NSUpJvKHnbtm2Or7grrXh3qt62bVuB86+//rqrQ8qlgWHDhlGjRg0G\nDRrENddcA1Dg3Wnjx48v9s2CYyHuK6UOHTqQnp7O1VdfTcuWLbnppps499xzqV27dkmblhBUrVqV\n++67D4DnnnuOgQMHkpubyzXXXMP27dt9q5zOP//8kjSzVFGzZk06dOjAd999V9KmlAratm1Lw4YN\nS9qMuKdWrVqANRTv3SU8EUnYhQ7FSaJPzHsf8Kxfvz6nTp06Y6FDcnIy33//PS1btnTVDi+Jrmck\nPPnkk/z9738HYNOmTa72lBJxYj4Ubdu2ZdWqVQCkpaUBMG/evGLtJZUmPeMF8+oKQwFq1qxJzZo1\n2bBhA4MHD/a9yROsDRnnzZtXbBXS2cJDDz1E48aNady4sXk2KQZat27N/PnzmT9/vhm2O8swlZLB\nYDAY4gYzfBcBZ8NwU3Fi9HQWM9zkLEZP5zHDdwaDwWBISEylZDAYDIa4IaYl4YGrtwxFw+jpLEZP\n5zGaOovRMzRRzykZDAaDweAWZvjOYDAYDHGDqZQMBoPBEDeYSslgMBgMcYOplAwGg8EQN5hKyWAw\nGAxxg6mUDAaDwRA3mErJYDAYDHGDqZQMBoPBEDeYSslgMBgMcYOrlZKIPC8i97mZxtmG0dRZjJ7O\nYvR0lrNST1UNewD3A8uAbOCdIO4VgdeAA0AGkO7ndi6wHSgbIu7GQD7gKcwOvzDlgBeBXcBhYBxQ\nJtLw0R7AE8AO+9rmAS0diDOkpsBtQJbfcdzWqG1p0BS4FVgPHAUOAp8Cdd3S03bvBawFMu3PG93M\no27lmxDp3Glf+1E7vbFAkst63g38bOfPmUAdN/UE+gN5AeWiSyLkTyAZeAvYaue/5UCPAD/d7TSP\nA3OBhi7rWaz3UL9050RiayQ9pV3Ak8DbIdwnANWA5kB14EGvg6rutcW+oZA0otmd8G9AO+AioJn9\n/fEowkeMiNwA3Ad0BmoAi4H3HYg6pKaq+oGqVvEewJ+BTaq63HZPaE2BRVg3lBSgEXACeKGIcYbU\nUyuddrAAACAASURBVERSgQ+Ah1S1KjAU+D8ROQfc0dPFfBOMCsADQE2gA9YNbkgR4wyn5xXAU1h6\n1QC2AB963V3KnwCL/MuFqi6IMnzE6eBs/iyDVal0sfPf48DHItIIQERqAVOBx7Dun8uAKd7ApaC8\nAyAit2FpUfhmq1HUck9yZqu+OVaLonKYcMOBt0O4bceqOb2tnw4R2PE/oLff7z7A9jD+84FBwCas\n3txz2BvRRpDWo8AUv98XAScdbDmcoWkQP/OAJ0qLpgHxVAbeA150S0/gV8C+gHP7/XVxQc+o8o1T\netpx/RX40kU9nwde9ftdx7a/iYt69gcWRmF3XOZPv3hXAr+zv98LfOvnVhGrImzmop7FWt6BFGAD\nVqPJkZ6Sl2A18WXANmC0iBwQkVUiclOAn/VAWog4O9ufKWq1fpaISEMROSIi9SO0xQPUF5EqYfz3\nAtpjtQhuBO4CiCCtOUBHEblARMpiDZXMDJNOtIRt3ditqc7ApACnRNYUEfm1iGRgDWc0BIaFSSca\ngum5EsgVkZ4ikiQivbCGpVb5+XFaz1jyTcx6BtAVWBOh38IIpqdyZl4BaOV3zmk9FWhr32M2iMjj\nIpJUiO3xmD8RkdpYvZO19qmLsPIoAKp6AtiIu3pCMZZ34GmsKZ59Yfz8QhFbTcOxar4RWF2zLli1\ndXM/P1djDT8Fi7Mx0Y+HPgl8C9TCGm9dgjXeXDtMLX+N3+8/Ad9EmV4+kIPVUmjsYIspbE8Ja15i\nbpDzCa2pX7i6wCzgJTf1BHpijdfn2J/XuqlntPnGQT3vwmo513BLT6zhwf1Aa6yhw/F2XvmDi/mz\nCdDI/t4K64b+t2LQ0+n8WRb4Bnjd79xE4JkAf98C/VzUs9jKO3AJ8ANWxReRrUXtKZ3EKnRjVDVX\nrXHeecA1fn6qYE32OsVTWJOFK7CE/QzIVdVwtfAOv+/bsTJboYjI/ViFsD7W5OBoYK6IVIjB7qBJ\nFOLeD2v4IJCE1dQfVd2NVfH2izZsCM7QU0TaYc17dlbVslg9ibdExL/l6aieMeabIulp9wCfxqpw\nD0dncehoA0+o6hxgJNY8yBb7yAJ2+nlzVE9V3aKq2+zva7D07F1IsLjKnyLiwZpXzMZaSOLlGFA1\nwHsKlqZeErK829f8GvCgqub7O4ULF02lFGyCyjsEEpiIv98WWBcfaZzhjVDNVtVBqlpfVc/HWj2y\nrJBgDQO+74owuR7Ah6q6W1XzVfU9rMnIFtHaHYKQ1y8inbDG6/8dxDmRNQ2kLNYYuhMEu/buwH9V\n9QcAVV2G1TK8ys+Po3oSW76JWU8R6YFV8fZU1bWF+Y+CoNeuqq+pajNVPRdrdVoZCg4ZOq1nMApr\n0MVN/hTrNbNvAecAN6tqnp/zWvyG5kSkEnAevwzvQeKW96pYQ35TRGQPsNQ+v9O+v4U0sLDuVxJQ\nHngGa26jHPaSU6zM+DPWyo0yQCescVj/SbpZ+E2qBcRdEcgFLoiyS10XK1NejlVrXxXGfz4wG2uF\nYANgHXB3hGk9DSwEUrEq8DuwWjBVI7U3Wk39/EwA3g0RPpE17Qs0sL83AtKBl93SE6vXfgBIs3+3\nxVrqe5VfeKf1jCrfFFHPbsAh4NdF0TAKPcthDaEJ1s1pPtZIiZv581rsoSWsxVWrCVj8E+f58w2s\nFZiVgrjVwuoF3WRr/hzwnct6Fmd5T/U7LrHjqkOIJe6qGlGlNNKOyP8Y4efeEvgOqxu6hoLPgNTB\n6vaFXAMPjMIaoz6CtXCioV2A64fw3xlryOC4LU6fQuzPx+oub8K6Gf0De0wzgrQqYo357sVaZbgM\nv7HVImTSwjQtb+txZZCwia7pGNv+Y3aazwLlXdZzqG1rlv35V5f1jCrfFFHPucBpCj7DM90tPbFu\nTCvt/28P1lCQ+IV1Q89/2FoeszUaSZhnseIpf2JVbPlYvS3//6iPn5/uWOXuBGc+p5TQ5T0gnsZY\nc1dh55TE9uwKIvI8sFFV33AtkcJtyAfOV9XNJWWDkxhNncXo6SxGT2c5G/V0tVKKB0pTBo0XjKbO\nYvR0FqOnsxS3nmfDhqylu9YtGYymzmL0dBajp7MUq56lvqdkMBgMhsShTLQBROSsrMVUNdq9uiLC\n6OksRk/nORs1NXo6T6SaRl0p2ZHHEixhsR4zcA+jp7MYPZ3nbNLU6Ok80Wh6NswpGQzFxo4dO6hV\nqxYXXnghBw8e5ODBgyVtksGQUEQ9pyQiejbW8m4ONxk9HY27RPTMyclh6NChTJ48mYwMa0eYSy+9\nFIDFixe7mrabetrxn1V51OjpPNFoanpKBoPBYIgbEq5SevPNN/F4PIgIderUoU6dOuzZs6ekzYpr\njh8/zvHjx7n//vvxeDw+/dq2bcuCBQs4depUSZuY0OzZs4ff/OY3vPrqqxw5csR3/rrrruO6664r\nQctKBzt27OChhx5ix44dhXs2hCQ7O5s33niDDh06UKlSJSZOnFjSJgUnhm0ztLg5ePCg/u1vf9Py\n5ctrUlKSejyeAkerVq1cTd++5pi3bQl3uK3npk2btEuXLtqlSxf1eDzauHFj7d+/v/bv39937rbb\nbtOTJ0+6aoc/iaxnIJmZmfrII49oUlKSL296v6ekpGhKSoqmp6e7aoObemoJlXlV1R07duiOHTu0\nUaNGmpSUpLfeequuXr1aVVWzs7N19erVvsNJSqOee/fu1caNGyvWM0e+Y/LkyTp58mTX049G07gX\n9MCBAzps2LAClVCDBg20ZcuWvt9JSUk6YsQI12xI1Jvo8ePHtVWrVj6dOnfurMePH/e5nzp1SmfP\nnq0ej0efffZZzc3N1dzcXNfs8ZKoegZj1KhRvkoosFLyP/6/vTMPj6JK+/ZdFQIhRAIEgaiR+CmM\nKA7Rl4sAIsg2Ok6UyDIoviwOMIIzss44DPuiEDcYUUFlkbAo44agfrixJKAsw7zKMjIOX/wkEIiA\nQDaSEJLn/aO6y07SnXQnXd1dybmvq650V51T58mvn6rnrFWLFi2S7du3W2JDXbyJnj59WmJjYyU2\nNraSlt27d5eEhATze2Jiol/Lrmt6FhYWyj333COAPPvss1JSUiK7du2SFi1ayNy5c2Xu3LmW21Cn\ngtJdd91l3lTHjh0ra9askcLCQnn00UfLBar4+HjLbLDrTfT06dOi67qMHDlSRo4c6TZNfn6+qWFO\nTo7k5ORYZo8Tu+pZkczMTImJiSnnh0Cllrxz/+OPP26JHXXtJnr69GmJi4srV+msGPgTExMlPz9f\n8vPzpbi42K/l1zU9p06dKoC89957UlpaKiIieXl5sn37domLi5O4uDjLbfBF0xqtUwoU58+f54cf\nfgBg0KBBLF++nLAw929B7t27dwAtswdbtmwBoFMnT29SLs+mTZsAGDlypGU21SVefPFFLl68aK7B\nuOuuu9iyZQsZGRmkpKTw7rs/vwpL13XWrVvHgw8+SL9+/TydUgH87W9/Iysri7Iy471wDRs2pHfv\n3gwfPhyARx55JJjm2YotW7awePFipk6dSnJyMrqus2fPHgYOHMiuXbtMTUOJkA5Kq1atIisri/j4\neBYvXuwxIDVt2pS//vWvAbYu9PFmYLhx48aMHDmS1FR3L7hVVMWSJUvMgNS/f3/eeecdoqKiSEhI\nYOHCheWCEkBUVBQtW7YMhqm2Yvfu3Wiahq4b87Cee+45JkyYEGSr7MmqVau44YYbmDdvnqnnt99+\nS3Z2Nhs2bOCBBx4IsoWVCcmgtHfvXgBmzpwJwN133811110HwKVLl/jwww95//33zfTJycm0a9cu\n8IbWAXRdp3Fjf73dvf4yadIkGjRoQHFxMcXFxbz22muV0gwcOJCEhIQgWGcv2rdvb94DACIjI4No\njT358MMPAfjoo49Yvnw5TZo0MY/t32+8APbKlSskJiYGxb6qCMmglJOTAxiiAWRlZXHggPG23iNH\njjB69GjAaCEBqpXkgQ4djLdvr169GoDJkye7TXfHHXcAsH79ekB139WEYcOG0alTJ5o3b87mzZvd\nphk6dGiArbInI0aMYO3ateb3yZMn89VXXzFjxgxuvPHGIFpmH15//XXACOhDhgwx9xcUFJjd+hER\nEUGxrVq8HXySAA7SHTp0SA4dOiRRUVFuB42d2+233y6333675fZg04H50tJSGTJkiKnX/PnzpbCw\nUMrKyqSkpESKiorklVdekcjISNF1Xdq2bStt27a1fAaeXfV0smjRIlm0aJHbSQ3u9um6Lv/4xz8s\ns8dKPSUIA/OZmZkyePDgShMdoqKiZPny5ZaXb3c9i4qKpEWLFtKiRQu55557yh3bsGGDOR38qaee\nstQOV3zR1HaLZxUKhUJRh/E2ekmAorwrrrV8XdfLrU3SdV3efPNNefPNNy23AxvX7C9evGi2KJ26\njRs3Tvr16ydt2rSRAQMGyM6dO6Vdu3bm8S1btlhqk531nDlzpjRv3lyaN2/ucbqyu3VKVmKlnhKE\nlpKTZcuWybJlyyQuLk4aN25s+ueyZcssLdfuej7zzDNma+i1114z9587d07at2+vWkq1YfXq1Ywa\nNYr4+HiSkpL44IMP6NGjh3n8hhtu4IYbbgiihaFPdHQ06enppKen889//pOJEyfSuHFjZsyYwd69\ne3n//ffp1asX06ZNM/OsW7cuiBaHLqdPn2blypXk5uaSm5tr7o+OjmbLli08//zzHvMuXryY0tLS\nQJhZZxg/fjzjx48nMzOT1NRUNE1D0zQWLVoUbNNCGlff/Ne//kVeXh579uyhe/fuXLx4kdatWwfR\nOi/wNnpJgKK8O86dOyciRl9pnz59RNd16dChg+Tl5UleXp7l5WPjmr23nDx5UiIiIiQiIkKGDBli\naVl21XPWrFmVWkVJSUly9OhRM828efOkYcOGbltQK1eutMQuK/WUIPpoSUmJlJSUyOHDh2Xs2LGm\nlhXHSfyN3fX8+OOPy41xNm/eXAAJDw+XvXv3ysqVKwWQsWPHWmqHK75oGpKz7yoSExMDGFMZd+7c\nCUBCQgJRUVFBtKpuce2115oL6Y4dO0ZJSQnh4eFBtip0yMrKctuCfPLJJ7n55pvN77NnzyY1NZXj\nx49XSvv1119baqOdKCoq4sqVKx6v4ezsbO6//36gvG6dO3cmJSUlIDbalfvuu4833ngDgHnz5gFw\n//33M27cOBITEzly5AgQuv4YckGpqKiI2bNnA5CYmMigQYPMY88884z5eerUqQG3ra7jvLmuWrWK\nAwcO0K1btyBbFDocP36czMzMcvt69uxpvjOpuLiYTZs28fDDD5uLFF0pKyujf//+AbHVDqSlpXHd\ndddx6623Vjr2wgsv8PTTT5tLQzRNo127dowePZrx48eXW3OjcM+IESPK/XXH9ddfHyhzfCLkgtJn\nn33GCy+8AMCECRPMoFRSUsKlS5eCaVqdoKSkhLCwMLc3Tld27dqlgpILzvEMV7755hueffZZUlNT\nKS0t5eTJk+ZrQSqSm5urbqYO5syZQ0pKCocPH6akpITi4mIOHjzIpk2bWLJkiZnOqeOUKVOYPn06\nzZo1C5bJdZJTp04F2wS3hFxQcuWLL74gPz+fRo0aMX36dNLS0gDjWViNGjUKsnX249ixYzz99NO8\n+uqrbhfOubYEqqphKQxyc3OZP3++x+NOjZctW8ZVV10VKLNCnqeeeoqWLVvy6quvkpGRwccff4yI\nmIG/ZcuWjB492uy+69q1a5Atrpt899135OfnA4TUUEjIBaXIyMhyz2iaPn0658+f56233gKgQYMG\nJCcn07Fjx2CaaStKSkoA6N69O0OHDnUbkM6ePWs+Gqd///7qGW0VuPXWW7nttts4fPhwleni4+PN\nZzQ6ZzSqAF+exx57jNdff52lS5ea+xo1asSDDz7I8OHDSUhIoE2bNkG0sG7Tu3dvdF3nwoULnDx5\nEqDcuGjQ8XZGhARo5oiISPv27aV9+/ZuV8ZbPTPMHdh0tpiTsrIyKSsrk/nz50tYWJj85je/kdTU\nVLlw4YK5TZgwwdQ4KirKnPFoBXbVc82aNR7XJP35z3+WjRs3WlZ2VVipp1igaV5ennz22Wfy8ssv\ny9GjR+XUqVOSnZ3t1zJqg930rAkLFy4UICTfp6QZ6b1H0zTxNY+vfPrppwCVXiXdqVMnduzYQXR0\ntKXlV0TTNESk8kCBf85tuZ5OioqKmDVrFp988gnffvttpePOFtTWrVvp2bOnZXbUFT1DBSv1dJy/\nXmlaH/Rcu3YtI0eONB8QvHv3bkvHPH3RNKQXzyoUCoWifhGSLaWsrCzAqLHPmjWLxx9/nNjYWH77\n29+aTwYPJHWtZl9YWMjBgwe58847zX1/+MMfzKn4Vo8n1TU9g019qNkHkvqgZ1ZWFgkJCTz22GMA\nTJ8+3dJXhPiiaUgGpVBD3UT9i9LTv9SHm2ggUXr6H9V9p1AoFApbooKSQqFQKEKGGq1TcrdiXVFz\nlJ7+Renpf5Sm/kXp6Rmfx5QUCoVCobAK1X2nUCgUipBBBSWFQqFQhAwqKCkUCoUiZFBBSaFQKBQh\ngwpKCoVCoQgZVFBSKBQKRciggpJCoVAoQgYVlBQKhUIRMqigpFAoFIqQwdKgpGna85qmjbOyjPqE\n0tP/KE39i9LTv9RLPat6LS3QEFgF/ADkAl8D97ocDwfeBf4/UAb0qpC/DZAJhHs4f7wjn+7tq3KB\nV4E8l60IyPU2vy8b8BDwbyAHOAe8D1xTi/NVp2dX4HPgJ+AM8DbQxko9K+TfVpv8Xpy/EbAEyALO\nA68ADWp5zuo0vQU44CjvIvAl0MNiH/Wr31RTll+vh+r0rJB2tkObPnXFRx1l/B/gI8f/fxZ4xkL/\ndOrh+hvOsNg/OwKfOv63Mqt0dClvFnDCcf3tAG6pKn11LaUGDkF6ikhTYCbwtqZpbV3SpAP/DWQD\n5R6kJyLZGBfnA9WU4/XTCUVknIhc5dyAtzBu3lbwJcb/Hg20BS4Bi2txvur0bIZxk2nr2PKAN5yZ\nrdDTzKBpjzjss/JhiNOAO4BbgfaOzzNrec7qNM0ChgAxQHNgI0ZFCrBMU3/7jUcsuB68uebRNO1G\nYDBwqoI9tvZRTdMaYlQMvwBaA9cC62txSq/0BJq6/I5PO3dapOdljOtgtA95aoSmaQ8A44C7gBbA\nHmBdlZlqEPUOAg+62X8CQ/iK+6cDqz2cK5PytYREH21pglH7uKuKNGXAE0AGRs3gWRwPovWxrCgg\nFVji51qEWz0dx+6gQq3XCj2BaOA7IJFqal210RP4BzDY5fvDQKY/9azGRxsAfwC+DqCPVus3fvTR\naq8Hf+kJbAV+jdFL0qfCMTv76O+BNH/7pCc9+bmlE1ZFekv8E7gJL1pKtdTzr8DfXb7fChRWmcdH\nMVsDhUB7N8c8BaWBwD89nK9tRQcDrgcuANd5Yc8I4P95Ieg2jFZInMOxR3tbFtADo9lZhtH0bOhH\n5/Sop+P4JOArq/XE6EabiBddAbXREyMoDXH5/ojjfFdZranjNywBjgM3BkBTr/2mtj7qy/XgDz0x\nWp6bHJ/dBSU7++hqYC3wfzFuwDuAjlbp6fL/nMS4h64GYqzW05HOl6BUUz27YATOdhjDPc8C71dZ\nng9ihmM0aZd7OO4pKPUHMjzkqdbBqrFpGzDbC0F/5fJ9PPBFDcq6BvgMeNFPzlmdnr/EGFu600o9\ngc7A/2BMevH2gq+RnsACYDfQEqOvfB9QCrQOkKaRwDOO/1dz2W+lj1brN3700Wqvh9rqCVwF/Ae4\n3vHdXVCys49+htG9dQ9Gy/pPGC0Et2M6ftCzCUaPiA60At4BPrFST5e8vgSlGvun47ovw6gUZgDx\nVaX3avadpmk6Rj9gEfBHb/K4cBVGjdGvaJp2PdALo1ZTHSdcPmdi3Ch8QkROYQzYjfA1b0Wq01PT\ntJswamoTROTLCof9pqfDjmXAJBEpcz1UTdaa6vk0xkDvNxjBaRNwRUR+9DK/R7zxURG5hDGu1R64\nzeWQJT7qKNNbv6mVj/p4PXhzPk96zgXWiUima/IK2e3so5eAXSLyqYhcEZHnMcYjb/bWZnd40lNE\nCkTkf0SkTETOOI79StO0Ji7ZLfNPH6iRnpqm/RHoC1yHMdFpPrBd07TGnvJUG5Q04xWJq4CrgUEi\nUuqNMS50wLgJuUN8PJcrw4HdIvKDF2mvr/A5q4ZlhmM4bY2pTk/HAOjnwHwR2eDmFP7UsynwX8Df\nNU07Dex37D+padqdVeSrkZ4iUiQiT4jIdSJyE8aMuAM+2lwJH300DMPvXX9Hq3zUiTd+U1sf9eV6\nqJJq9OwDTNA07bTDZ+IwBu7/7JLGtj4KHHL9ovnhFbE1vIe63put9k9vqKme9wJvicgpR+BNxZhw\n1MFjDi+aXq9izJho4uF4IyACI5L2ByLcNIcHe8gbCVwB2tWg6fkdMMrLpufn/NwfehQY42UZw4A4\nx+e2QBqw1FdbvdUTY6ZPBjC1ivx+1ROjy8C5dXboFYvnKai10fMax6ZhTH/PBPrVRk8vNO0HJGAE\no6bAUipPdPC3pj75TW009fV68IOeLVz8pbXjNxzkmtbmPtoeKMCo3YcBk4Fj1GLpQjV6dgF+gRGE\nYoC/A9us9E9HvgiM5RJlGPfwRlb4J7AQ2OX47XSMylMexmxD93mqOaFzEO0S5efRP+yS5gdHmlKX\nv87+5liMYOXxBwXmYazJueD4ga53lFHVoGc3Rxq3gdKNoH/EuNmfA57D0f9aXVnAUw778zH6zlOo\nEHR9dIQq9QTmUHnNQq5Lfkv0dMkb7/j9quuvr6medzl0LHA49sPV2eQHTQc7ysoDTmNMmY6zUlNf\n/aY2mvp6PdRWTzfpy40p2d1HHWkexAhEOcB2oIOF/vkQ8L3DV04Ba4BWFvtnvMMm1/v291boiRE0\nV2IsGcrB6Bn5laeyRMQY7LUKTdOex5gN9KplhVRvQxlwk4h8Hywb/IXS0/8oTf2L0tO/1Ec9LQ1K\noUBdctBQQOnpf5Sm/kXp6V8CrWd9eCBr3Y66gUfp6X+Upv5F6elfAqpnnW8pKRQKhcI+NPA1g6Zp\n9TKKiUitp4a6Q+npX5Se/qc+aqr09D/eaupzUHKcvCbZbIsflipUidLTvyg9/U990lTp6X980bQ+\njCkpFAobc/nyZZYuXcq+ffuCbYoiAKigpFAoFIqQQQUlhUIR0uzZs4cpU6awcePGYJtiC9LS0mjQ\noIG5paenB9sknwi5oPTpp5+iaRqaptGkSRP+8pe/8P33armBIjS4fPky8fHxxMfHk5SUxOXLl4Nt\nUp1n8uTJiAitWrUKtim2oG/fvui6bm79+vXjo48+4tixY8E2zSt8nhKuaZpYOUiXnp7O0KFDAWMw\n8OzZszRq1IihQ4eyYsUKGjSo0dyMWqFpmqWzxQI16FlQUMDcuXPJzs7m5ZdfJjo6moKCArp27Uq/\nfv1YsmRJQOyws55FRUXExcUB8NNPP7FixQpGjhzJyZMnAXj00Uc5evQoPXr0oFOnTma+uLg4RowY\nga77vx5opZ6O8wfMR11xjiF1794dTdPIzc0lMjLS8nLtrmeDBg0q+VlZWRmdO3dm/fr13HTTTZaV\n7QmfNK3Bs5wkUFy5ckXee+89GTp0qOi6LsOGDZPi4uKAle/E8T/X6plinrZA6pmamiq6rkuzZs3k\n22+/FRGRN954Q3RdlzZt2gTMDrvrmZGRIRkZGaJpmjRp0kTat28vuq6LruuiaZpcc8010rp1a3Of\nc/+cOXPkypUrfrfHSj0lwD7qysSJE2XixImiaZokJycHrFy765mWlibh4eHltrCwMAkPD5etW7da\nWrYnfNE08M0OHwgLC2PgwIH8+te/pqSkhLfeeovHHnuMnj17Bts0W1Faajwpf/fu3QDMmzePDh2M\nJ8f/9NNPQbPLjogIe/bsMb+XlJQAhqYAw4cPp2XLlpSVlXH+/Hkz3bBhw1iwYAEJCQkkJycH1mgb\nsnLlSlavXg1AmzZtWLp0aZAtsg/x8fGUlZWV2+f8npSUxPbt20P7Hupt9JIARfmq0DRNunTpEvBy\nsXnNfu3atbJ27Vqz1v7555+bxyZPnqxaSj4wZ86ccq2fjz/+2Kt8H3zwgei6LsuWLfO7TVbqKUG4\n5gsKCuT22283dV64cKF57Mcff5QZM2ZIZmamZeXbXc+LFy/KoEGDZNCgQZVaSs4t0PiiaUi3lCoS\nGxvLkSNH2LZtGzNnzuQXv/gFAGvWrAmuYSHMm2++yYgRxktPu3fvTpcuXejdu7d5fMeOHa4Xi6Ia\nkpKSyMjIAGDGjBncfHP1LyTNyMggOTmZjh07MmDAAKtNtD0vvvgiBw8eZMyYMQBMnDjRPPbQQw+R\nlpaGpmksWLAgWCaGNNHR0Sxfvtz8vmXLliBa4zshN/tOoVAoFPUXW7WUbrnlFrZv387QoUO5cOEC\nYWFhwTYppMnNzWXcuHHmIz4GDhzI1KlTK6VzTsF3JT8/n/DwcBo1ahQQW+1C586dWbdunVdpd+3a\nBcD999+PpmmMHz+ea665xkrzbE9OTg4vvfQSkZGRjB49GsCccXfixAnS0tIQEY4cORJMM0Oeq6++\nGoAOHTrYrqVkq6C0bds2NE3jwoULAPTo0SPIFoU2FQPNc889x/r16+nRoweTJk0CjGniYAwmO7ul\nAMaOHUurVq3UgsUakp6ezn333QdAw4YNWbx4MePGjQuyVaFP3759OXPmDK+99hpdunQpd2z+/Pmm\nT+/fv58+ffrwyCOPEBsba6Zxaq4wWLBgAV9++SU7d+4st/+OO+5gx44dREdHB8ewqvB28EkCNEhX\nFZqmlZtmm5KSIikpKZaXi40H5tPS0kTTtEraVfdd13VJTEy0xCY761kdxcXFMm3aNGnSpInExMRI\nTEyM/Oc//7G0TCv1lABqumLFCrfTv4uLiyUzM9P0Ude/sbGxMmzYMCkoKJCCggK/2FFX9HSybQci\nuQAABWdJREFUatWqShMdwsLCZPbs2QGzwRdNQ66ldO7cOU6cOAFA69at+eCDD9i6dWuldLNnzzZr\n+wrP9OzZk7NnzwKwefNmWrVqxf79+3nppZcoLS0lPz8fgFatWpm10LZt29KtWzdGjRoVLLNtyb//\n/W/uvfdeTpw4QVJSEitWrABQTyLwgn379jFlypRyExjOnDkDwMaNG81jTh8dM2YMM2fOpGnTpqFZ\n2w8hfve73/H73/8+2GZ4j7fRSwIU5T/55BPBeNOhWcN3bs5969evt9SGilAHa/aXLl2S6dOnm62i\nQFLX9CwuLpaNGzdK27ZtBZDFixdLUVFRwMq3Uk8JgKauU8A1TZOxY8eWmxJesYXk7TT8mmJ3Pd3h\nrqV07bXXyv79+2X//v2Wl++LpiHXUrr55pt5/vnnAWNM5OjRo3To0IHCwkJmzZpFVFQU/fr1C7KV\n9qdx48YcOnQo2GbYnsOHD5OcnMwPP/wAGK3MvLw89u7dS69evYJrnE0YO3YsBw8eNFtBq1evRkTK\njYm6tpDuvvvuYJhpa9wtps3Ozg7NxfPeRi8JYpQXEXnllVfMPuRAQx2r2TtJSkoSXdflySefDGi5\ndtczKytLsrKypGPHjhIeHi66rkt4eLh06tRJmjVrJrquS0REhJnOaqzUUyzWdMyYMeVaQb1795aV\nK1fKjz/+6DaNlYtmndhZT3fs3bvXbUspPDxcpkyZIlOmTPHbeJwnfNFUrVNSKBQKRcgQct137rhy\n5QrvvPMOAE888USQrakbFBcXs3//fkREdYf6yPHjxwE4e/Ys06ZN48477+SWW24hLi6OgwcP0q1b\nN4qLi83n4inc43y+naZpjB49mlmzZplPYIefJzo40yhqxvTp0z0ee+mllwDj9SCBeAK7N9giKF24\ncMF8UZVazOkfSktLOXfuHJqmERUVFWxzbEW3bt0AyM7OLrf/4sWLDBgwgOLiYpKSkmjdunUwzLMF\nOTk5zJ49m7KyMmJjY3n99dcrpTlw4ACAs8uLhx56qFzQUnjHxo0by63lCnVs033n7G8cPnx4sE2p\nE3zzzTfm54SEhCBaUjc4d+4cCQkJHD9+nEmTJvH2228TERFBREREsE0LSf70pz9x5swZYmNjzfcm\nVeTgwYPmBAhN08wp9grfuPrqq/nlL39JWVmZuZWWlpb7HkrYoqUEP8++ycjIMB+hoag5zic5KLzn\nzJkzfPXVV2YQ37lzJ82aNePw4cOkpqZy4sQJpkyZwsKFC1WLvhoSExPJyMhg48aNHtdxff7558DP\nLaVQ6V6yI5s3b+bGG28st0/XdfMBwU2bNg2GWW6xTVBykpOTE2wT6gSFhYXmxa7wjnfffbfcmKa4\nTFsOCwtj1apVPPzwwyogecGYMWPMp4B7IiUlBYAPP/wwECbVaWJiYnjiiSfMMSQnzucLhlJQskX3\nXUxMDHPmzAFg8ODB9O7dm8uXLwfZKnuzefNmtw9iVXjmtttu83hswIABjBo1SgUkP9KlSxe6dOnC\nggUL1GsqaklkZCR9+/YNthleYYuWkq7rTJ06lblz53Lp0iW6du2qnhCuCDiJiYns27ePDRs2AEZL\nKT09nV69erFo0aIgW6dQVM19991ni8q8LVpKCoVCoagf2KKlBNCkSZOQmyVSF5gxY4aaIeYlDRs2\npHPnznTu3DnYpigUdRbN18FuTdOkvg2Qa5qGiFgy+KL09Pu5lZ7+P3+90lTp6X980VR13ykUCoUi\nZKhR952aseVflJ7+Renpf5Sm/kXp6Rmfu+8UCoVCobAK1X2nUCgUipBBBSWFQqFQhAwqKCkUCoUi\nZFBBSaFQKBQhgwpKCoVCoQgZ/hfAHLPf42Z3yAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "miscl_img = X_test[y_test != y_test_pred][:25]\n", + "correct_lab = y_test[y_test != y_test_pred][:25]\n", + "miscl_lab= y_test_pred[y_test != y_test_pred][:25]\n", + "\n", + "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n", + "ax = ax.flatten()\n", + "for i in range(25):\n", + " img = miscl_img[i].reshape(28, 28)\n", + " ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n", + " ax[i].set_title('%d) t: %d p: %d' % (i+1, correct_lab[i], miscl_lab[i]))\n", + "\n", + "ax[0].set_xticks([])\n", + "ax[0].set_yticks([])\n", + "plt.tight_layout()\n", + "# plt.savefig('./figures/mnist_miscl.png', dpi=300)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "source": [ + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "source": [ + "# Debugging neural networks with gradient checking" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "[[back to top](#Sections)]" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from scipy.special import expit\n", + "import sys\n", + "\n", + "\n", + "class MLPGradientCheck(object):\n", + " \"\"\" Feedforward neural network / Multi-layer perceptron classifier.\n", + "\n", + " Parameters\n", + " ------------\n", + " n_output : int\n", + " Number of output units, should be equal to the\n", + " number of unique class labels.\n", + "\n", + " n_features : int\n", + " Number of features (dimensions) in the target dataset.\n", + " Should be equal to the number of columns in the X array.\n", + "\n", + " n_hidden : int (default: 30)\n", + " Number of hidden units.\n", + "\n", + " l1 : float (default: 0.0)\n", + " Lambda value for L1-regularization.\n", + " No regularization if l1=0.0 (default)\n", + "\n", + " l2 : float (default: 0.0)\n", + " Lambda value for L2-regularization.\n", + " No regularization if l2=0.0 (default)\n", + "\n", + " epochs : int (default: 500)\n", + " Number of passes over the training set.\n", + "\n", + " eta : float (default: 0.001)\n", + " Learning rate.\n", + "\n", + " alpha : float (default: 0.0)\n", + " Momentum constant. Factor multiplied with the\n", + " gradient of the previous epoch t-1 to improve\n", + " learning speed\n", + " w(t) := w(t) - (grad(t) + alpha*grad(t-1))\n", + " \n", + " decrease_const : float (default: 0.0)\n", + " Decrease constant. Shrinks the learning rate\n", + " after each epoch via eta / (1 + epoch*decrease_const)\n", + "\n", + " shuffle : bool (default: False)\n", + " Shuffles training data every epoch if True to prevent circles.\n", + "\n", + " minibatches : int (default: 1)\n", + " Divides training data into k minibatches for efficiency.\n", + " Normal gradient descent learning if k=1 (default).\n", + "\n", + " random_state : int (default: None)\n", + " Set random state for shuffling and initializing the weights.\n", + "\n", + " Attributes\n", + " -----------\n", + " cost_ : list\n", + " Sum of squared errors after each epoch.\n", + "\n", + " \"\"\"\n", + " def __init__(self, n_output, n_features, n_hidden=30,\n", + " l1=0.0, l2=0.0, epochs=500, eta=0.001, \n", + " alpha=0.0, decrease_const=0.0, shuffle=True, \n", + " minibatches=1, random_state=None):\n", + "\n", + " np.random.seed(random_state)\n", + " self.n_output = n_output\n", + " self.n_features = n_features\n", + " self.n_hidden = n_hidden\n", + " self.w1, self.w2 = self._initialize_weights()\n", + " self.l1 = l1\n", + " self.l2 = l2\n", + " self.epochs = epochs\n", + " self.eta = eta\n", + " self.alpha = alpha\n", + " self.decrease_const = decrease_const\n", + " self.shuffle = shuffle\n", + " self.minibatches = minibatches\n", + "\n", + " def _encode_labels(self, y, k):\n", + " \"\"\"Encode labels into one-hot representation\n", + "\n", + " Parameters\n", + " ------------\n", + " y : array, shape = [n_samples]\n", + " Target values.\n", + "\n", + " Returns\n", + " -----------\n", + " onehot : array, shape = (n_labels, n_samples)\n", + "\n", + " \"\"\"\n", + " onehot = np.zeros((k, y.shape[0]))\n", + " for idx, val in enumerate(y):\n", + " onehot[val, idx] = 1.0\n", + " return onehot\n", + "\n", + " def _initialize_weights(self):\n", + " \"\"\"Initialize weights with small random numbers.\"\"\"\n", + " w1 = np.random.uniform(-1.0, 1.0, size=self.n_hidden*(self.n_features + 1))\n", + " w1 = w1.reshape(self.n_hidden, self.n_features + 1)\n", + " w2 = np.random.uniform(-1.0, 1.0, size=self.n_output*(self.n_hidden + 1))\n", + " w2 = w2.reshape(self.n_output, self.n_hidden + 1)\n", + " return w1, w2\n", + "\n", + " def _sigmoid(self, z):\n", + " \"\"\"Compute logistic function (sigmoid)\n", + "\n", + " Uses scipy.special.expit to avoid overflow\n", + " error for very small input values z.\n", + "\n", + " \"\"\"\n", + " # return 1.0 / (1.0 + np.exp(-z))\n", + " return expit(z)\n", + "\n", + " def _sigmoid_gradient(self, z):\n", + " \"\"\"Compute gradient of the logistic function\"\"\"\n", + " sg = self._sigmoid(z)\n", + " return sg * (1 - sg)\n", + "\n", + " def _add_bias_unit(self, X, how='column'):\n", + " \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n", + " if how == 'column':\n", + " X_new = np.ones((X.shape[0], X.shape[1]+1))\n", + " X_new[:, 1:] = X\n", + " elif how == 'row':\n", + " X_new = np.ones((X.shape[0]+1, X.shape[1]))\n", + " X_new[1:, :] = X\n", + " else:\n", + " raise AttributeError('`how` must be `column` or `row`')\n", + " return X_new\n", + "\n", + " def _feedforward(self, X, w1, w2):\n", + " \"\"\"Compute feedforward step\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ----------\n", + " a1 : array, shape = [n_samples, n_features+1]\n", + " Input values with bias unit.\n", + "\n", + " z2 : array, shape = [n_hidden, n_samples]\n", + " Net input of hidden layer.\n", + "\n", + " a2 : array, shape = [n_hidden+1, n_samples]\n", + " Activation of hidden layer.\n", + "\n", + " z3 : array, shape = [n_output_units, n_samples]\n", + " Net input of output layer.\n", + "\n", + " a3 : array, shape = [n_output_units, n_samples]\n", + " Activation of output layer.\n", + "\n", + " \"\"\"\n", + " a1 = self._add_bias_unit(X, how='column')\n", + " z2 = w1.dot(a1.T)\n", + " a2 = self._sigmoid(z2)\n", + " a2 = self._add_bias_unit(a2, how='row')\n", + " z3 = w2.dot(a2)\n", + " a3 = self._sigmoid(z3)\n", + " return a1, z2, a2, z3, a3\n", + "\n", + " def _L2_reg(self, lambda_, w1, w2):\n", + " \"\"\"Compute L2-regularization cost\"\"\"\n", + " return (lambda_/2.0) * (np.sum(w1[:, 1:] ** 2) + np.sum(w2[:, 1:] ** 2))\n", + "\n", + " def _L1_reg(self, lambda_, w1, w2):\n", + " \"\"\"Compute L1-regularization cost\"\"\"\n", + " return (lambda_/2.0) * (np.abs(w1[:, 1:]).sum() + np.abs(w2[:, 1:]).sum())\n", + "\n", + " def _get_cost(self, y_enc, output, w1, w2):\n", + " \"\"\"Compute cost function.\n", + "\n", + " y_enc : array, shape = (n_labels, n_samples)\n", + " one-hot encoded class labels.\n", + "\n", + " output : array, shape = [n_output_units, n_samples]\n", + " Activation of the output layer (feedforward)\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ---------\n", + " cost : float\n", + " Regularized cost.\n", + "\n", + " \"\"\"\n", + " term1 = -y_enc * (np.log(output))\n", + " term2 = (1 - y_enc) * np.log(1 - output)\n", + " cost = np.sum(term1 - term2)\n", + " L1_term = self._L1_reg(self.l1, w1, w2)\n", + " L2_term = self._L2_reg(self.l2, w1, w2)\n", + " cost = cost + L1_term + L2_term\n", + " return cost\n", + "\n", + " def _get_gradient(self, a1, a2, a3, z2, y_enc, w1, w2):\n", + " \"\"\" Compute gradient step using backpropagation.\n", + "\n", + " Parameters\n", + " ------------\n", + " a1 : array, shape = [n_samples, n_features+1]\n", + " Input values with bias unit.\n", + "\n", + " a2 : array, shape = [n_hidden+1, n_samples]\n", + " Activation of hidden layer.\n", + "\n", + " a3 : array, shape = [n_output_units, n_samples]\n", + " Activation of output layer.\n", + "\n", + " z2 : array, shape = [n_hidden, n_samples]\n", + " Net input of hidden layer.\n", + "\n", + " y_enc : array, shape = (n_labels, n_samples)\n", + " one-hot encoded class labels.\n", + "\n", + " w1 : array, shape = [n_hidden_units, n_features]\n", + " Weight matrix for input layer -> hidden layer.\n", + "\n", + " w2 : array, shape = [n_output_units, n_hidden_units]\n", + " Weight matrix for hidden layer -> output layer.\n", + "\n", + " Returns\n", + " ---------\n", + "\n", + " grad1 : array, shape = [n_hidden_units, n_features]\n", + " Gradient of the weight matrix w1.\n", + "\n", + " grad2 : array, shape = [n_output_units, n_hidden_units]\n", + " Gradient of the weight matrix w2.\n", + "\n", + " \"\"\"\n", + " # backpropagation\n", + " sigma3 = a3 - y_enc\n", + " z2 = self._add_bias_unit(z2, how='row')\n", + " sigma2 = w2.T.dot(sigma3) * self._sigmoid_gradient(z2)\n", + " sigma2 = sigma2[1:, :]\n", + " grad1 = sigma2.dot(a1)\n", + " grad2 = sigma3.dot(a2.T)\n", + "\n", + " # regularize\n", + " grad1[:, 1:] += (w1[:, 1:] * (self.l1 + self.l2))\n", + " grad2[:, 1:] += (w2[:, 1:] * (self.l1 + self.l2))\n", + "\n", + " return grad1, grad2\n", + "\n", + " def _gradient_checking(self, X, y_enc, w1, w2, epsilon, grad1, grad2):\n", + " \"\"\" Apply gradient checking (for debugging only)\n", + "\n", + " Returns\n", + " ---------\n", + " relative_error : float\n", + " Relative error between the numerically\n", + " approximated gradients and the backpropagated gradients.\n", + "\n", + " \"\"\"\n", + " num_grad1 = np.zeros(np.shape(w1))\n", + " epsilon_ary1 = np.zeros(np.shape(w1))\n", + " for i in range(w1.shape[0]):\n", + " for j in range(w1.shape[1]):\n", + " epsilon_ary1[i, j] = epsilon\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, w1 - epsilon_ary1, w2)\n", + " cost1 = self._get_cost(y_enc, a3, w1-epsilon_ary1, w2)\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, w1 + epsilon_ary1, w2)\n", + " cost2 = self._get_cost(y_enc, a3, w1 + epsilon_ary1, w2)\n", + " num_grad1[i, j] = (cost2 - cost1) / (2 * epsilon)\n", + " epsilon_ary1[i, j] = 0\n", + "\n", + " num_grad2 = np.zeros(np.shape(w2))\n", + " epsilon_ary2 = np.zeros(np.shape(w2))\n", + " for i in range(w2.shape[0]):\n", + " for j in range(w2.shape[1]):\n", + " epsilon_ary2[i, j] = epsilon\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, w1, w2 - epsilon_ary2)\n", + " cost1 = self._get_cost(y_enc, a3, w1, w2 - epsilon_ary2)\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, w1, w2 + epsilon_ary2)\n", + " cost2 = self._get_cost(y_enc, a3, w1, w2 + epsilon_ary2)\n", + " num_grad2[i, j] = (cost2 - cost1) / (2 * epsilon)\n", + " epsilon_ary2[i, j] = 0\n", + "\n", + " num_grad = np.hstack((num_grad1.flatten(), num_grad2.flatten()))\n", + " grad = np.hstack((grad1.flatten(), grad2.flatten()))\n", + " norm1 = np.linalg.norm(num_grad - grad)\n", + " norm2 = np.linalg.norm(num_grad)\n", + " norm3 = np.linalg.norm(grad)\n", + " relative_error = norm1 / (norm2 + norm3)\n", + " return relative_error\n", + "\n", + " def predict(self, X):\n", + " \"\"\"Predict class labels\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " Returns:\n", + " ----------\n", + " y_pred : array, shape = [n_samples]\n", + " Predicted class labels.\n", + "\n", + " \"\"\"\n", + " if len(X.shape) != 2:\n", + " raise AttributeError('X must be a [n_samples, n_features] array.\\n'\n", + " 'Use X[:,None] for 1-feature classification,'\n", + " '\\nor X[[i]] for 1-sample classification')\n", + "\n", + " a1, z2, a2, z3, a3 = self._feedforward(X, self.w1, self.w2)\n", + " y_pred = np.argmax(z3, axis=0)\n", + " return y_pred\n", + "\n", + " def fit(self, X, y, print_progress=False):\n", + " \"\"\" Learn weights from training data.\n", + "\n", + " Parameters\n", + " -----------\n", + " X : array, shape = [n_samples, n_features]\n", + " Input layer with original features.\n", + "\n", + " y : array, shape = [n_samples]\n", + " Target class labels.\n", + "\n", + " print_progress : bool (default: False)\n", + " Prints progress as the number of epochs\n", + " to stderr.\n", + "\n", + " Returns:\n", + " ----------\n", + " self\n", + "\n", + " \"\"\"\n", + " self.cost_ = []\n", + " X_data, y_data = X.copy(), y.copy()\n", + " y_enc = self._encode_labels(y, self.n_output)\n", + "\n", + " delta_w1_prev = np.zeros(self.w1.shape)\n", + " delta_w2_prev = np.zeros(self.w2.shape)\n", + "\n", + " for i in range(self.epochs):\n", + " \n", + " # adaptive learning rate\n", + " self.eta /= (1 + self.decrease_const*i)\n", + "\n", + " if print_progress:\n", + " sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n", + " sys.stderr.flush()\n", + "\n", + " if self.shuffle:\n", + " idx = np.random.permutation(y_data.shape[0])\n", + " X_data, y_data = X_data[idx], y_data[idx]\n", + "\n", + " mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n", + " for idx in mini:\n", + "\n", + " # feedforward\n", + " a1, z2, a2, z3, a3 = self._feedforward(X[idx], self.w1, self.w2)\n", + " cost = self._get_cost(y_enc=y_enc[:, idx],\n", + " output=a3,\n", + " w1=self.w1,\n", + " w2=self.w2)\n", + " self.cost_.append(cost)\n", + "\n", + " # compute gradient via backpropagation\n", + " grad1, grad2 = self._get_gradient(a1=a1, a2=a2,\n", + " a3=a3, z2=z2,\n", + " y_enc=y_enc[:, idx],\n", + " w1=self.w1,\n", + " w2=self.w2)\n", + " \n", + " ## start gradient checking\n", + " grad_diff = self._gradient_checking(X=X[idx], y_enc=y_enc[:, idx],\n", + " w1=self.w1, w2=self.w2,\n", + " epsilon=1e-5,\n", + " grad1=grad1, grad2=grad2)\n", + " \n", + " if grad_diff <= 1e-7:\n", + " print('Ok: %s' % grad_diff)\n", + " elif grad_diff <= 1e-4:\n", + " print('Warning: %s' % grad_diff)\n", + " else:\n", + " print('PROBLEM: %s' % grad_diff)\n", + "\n", + " # update weights; [alpha * delta_w_prev] for momentum learning\n", + " delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2\n", + " self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))\n", + " self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))\n", + " delta_w1_prev, delta_w2_prev = delta_w1, delta_w2\n", + "\n", + " return self" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "nn_check = MLPGradientCheck(n_output=10, \n", + " n_features=X_train.shape[1], \n", + " n_hidden=10, \n", + " l2=0.0, \n", + " l1=0.0, \n", + " epochs=10, \n", + " eta=0.001,\n", + " alpha=0.0,\n", + " decrease_const=0.0,\n", + " minibatches=1, \n", + " random_state=1)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ok: 2.56712936241e-10\n", + "Ok: 2.94603251069e-10\n", + "Ok: 2.37615620231e-10\n", + "Ok: 2.43469423226e-10\n", + "Ok: 3.37872073158e-10\n", + "Ok: 3.63466384861e-10\n", + "Ok: 2.22472120785e-10\n", + "Ok: 2.33163708438e-10\n", + "Ok: 3.44653686551e-10\n", + "Ok: 2.17161707211e-10\n" + ] + }, + { + "data": { + "text/plain": [ + "<__main__.MLPGradientCheck at 0x10a13ab70>" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "nn_check.fit(X_train[:5], y_train[:5], print_progress=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/1.2 Introduction - Tensorflow.ipynb b/1.2 Introduction - Tensorflow.ipynb new file mode 100644 index 0000000..f2d774c --- /dev/null +++ b/1.2 Introduction - Tensorflow.ipynb @@ -0,0 +1,1768 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "# Tensorflow\n", + "\n", + ">**TensorFlow** (https://www.tensorflow.org/) is a software library, developed by Google Brain Team within Google's Machine Learning Intelligence research organization, for the purposes of conducting machine learning and deep neural network research. \n", + "\n", + ">TensorFlow combines the computational algebra of compilation optimization techniques, making easy the calculation of many mathematical expressions that would be difficult to calculate, instead.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Tensorflow Main Features" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "* Defining, optimizing, and efficiently calculating mathematical expressions involving multi-dimensional arrays (tensors).\n", + "\n", + "* Programming support of **deep neural networks** and machine learning techniques.\n", + "\n", + "* Transparent use of GPU computing, automating management and optimization of the same memory and the data used. You can write the same code and run it either on CPUs or GPUs. More specifically, TensorFlow will figure out which parts of the computation should be moved to the GPU.\n", + "\n", + "* High scalability of computation across machines and huge data sets.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + ">TensorFlow is available with Python and C++ support, but the **Python API** is better supported and much easier to learn." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "# Very Preliminary Example" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "11\n" + ] + } + ], + "source": [ + "# A simple calculation in Python\n", + "x = 1\n", + "y = x + 10\n", + "print(y)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ + "import tensorflow as tf" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tensor(\"y/read:0\", shape=(), dtype=int32)\n" + ] + } + ], + "source": [ + "# The ~same simple calculation in Tensorflow\n", + "x = tf.constant(1, name='x')\n", + "y = tf.Variable(x+10, name='y')\n", + "print(y)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "**Meaning**: \"When the variable `y` is computed, take the value of the constant `x` and add `10` to it\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## Sessions and Models\n", + "\n", + "To actually calculate the value of the `y` variable and to evaluate expressions, we need to **initialise** the variables, and then create a **session** where the actual computation happens" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "model = tf.global_variables_initializer() # model is used by convention" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "11\n" + ] + } + ], + "source": [ + "with tf.Session() as session:\n", + " session.run(model)\n", + " print(session.run(y))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Data Flow Graph" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "* (**IDEA**) \n", + "_A Machine Learning application is the result of the repeated computation of complex mathematical expressions, thus \n", + "we could describe this computation by using a **Data Flow Graph**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "* **Data Flow Graph**: a graph where:\n", + " - each Node represents the _instance_ of a mathematical operation \n", + " - `multiply`, `add`, `divide`\n", + " - each Edge is a multi-dimensional data set (`tensors`) on which the operations are performed." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Tensorflow Graph Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "* **Node**: In TensorFlow, each node represents the instantion of an operation. \n", + " - Each operation has inputs (`>= 2`) and outputs `>= 0`.\n", + " \n", + "* **Edges**: In TensorFlow, there are two types of edge:\n", + " - Data Edges: \n", + " They are carriers of data structures (`tensors`), where an output of one operation (from one node) becomes the input for another operation.\n", + " - Dependency Edges: These edges indicate a _control dependency_ between two nodes (i.e. \"happens before\" relationship). \n", + " + Let's suppose we have two nodes `A` and `B` and a dependency edge connecting `A` to `B`. This means that `B` will start its operation only when the operation in `A` ends. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Tensorflow Graph Model (cont.)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "* **Operation**: This represents an abstract computation, such as adding or multiplying matrices. \n", + " - An operation manages tensors, and It can just be polymorphic: the same operation can manipulate different tensor element types. \n", + " + For example, the addition of two int32 tensors, the addition of two float tensors, and so on.\n", + "\n", + "* **Kernel**: This represents the concrete implementation of that operation. \n", + " - A kernel defines the implementation of the operation on a particular device. \n", + " + For example, an `add matrix` operation can have a CPU implementation and a GPU one." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Tensorflow Graph Model Session" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "**Session**: When the client program has to establish communication with the TensorFlow runtime system, a session must be created. \n", + " \n", + "As soon as the session is created for a client, an initial graph is created and is empty. It has two fundamental methods:\n", + "\n", + "* `session.extend`: To be used during a computation, requesting to add more operations (nodes) and edges (data). The execution graph is then extended accordingly.\n", + "\n", + "* `session.run`: The execution graphs are executed to get the outputs (sometimes, subgraphs are executed thousands/millions of times using run invocations)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "# Tensorboard" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "**TensorBoard** is a visualization tool, devoted to analyzing Data Flow Graph and also to better understand the machine learning models. \n", + "\n", + "It can view different types of statistics about the parameters and details of any part of a computer graph graphically. It often happens that a graph of computation can be very complex." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Tensorboard Example" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "Run the **TensorBoard** Server:" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "```shell\n", + "tensorboard --logdir=/tmp/tf_logs\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "[Open TensorBoard](http://localhost:6006)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "95\n" + ] + } + ], + "source": [ + "a = tf.constant(5, name=\"a\")\n", + "b = tf.constant(45, name=\"b\")\n", + "y = tf.Variable(a+b*2, name='y')\n", + "model = tf.global_variables_initializer()\n", + "\n", + "with tf.Session() as session:\n", + " # Merge all the summaries collected in the default graph.\n", + " merged = tf.summary.merge_all() \n", + " \n", + " # Then we create `SummaryWriter`. \n", + " # It will write all the summaries (in this case the execution graph) \n", + " # obtained from the code's execution into the specified path”\n", + " writer = tf.summary.FileWriter(\"tmp/tf_logs_simple\", session.graph)\n", + " session.run(model)\n", + " print(session.run(y))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "# Data Types (Tensors)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## One Dimensional Tensor (Vector)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "tensor_1d = np.array([1, 2.5, 4.6, 5.75, 9.7])\n", + "tf_tensor=tf.convert_to_tensor(tensor_1d,dtype=tf.float64)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 1. 2.5 4.6 5.75 9.7 ]\n", + "1.0\n", + "4.6\n" + ] + } + ], + "source": [ + "with tf.Session() as sess: \n", + " print(sess.run(tf_tensor))\n", + " print(sess.run(tf_tensor[0]))\n", + " print(sess.run(tf_tensor[2]))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Two Dimensional Tensor (Matrix)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[ 0 1 2 3]\n", + " [ 4 5 6 7]\n", + " [ 8 9 10 11]\n", + " [12 13 14 15]]\n", + "[[ 0. 1. 2. 3.]\n", + " [ 4. 5. 6. 7.]\n", + " [ 8. 9. 10. 11.]\n", + " [ 12. 13. 14. 15.]]\n" + ] + } + ], + "source": [ + "tensor_2d = np.arange(16).reshape(4, 4)\n", + "print(tensor_2d)\n", + "tf_tensor = tf.placeholder(tf.float32, shape=(4, 4))\n", + "with tf.Session() as sess:\n", + " print(sess.run(tf_tensor, feed_dict={tf_tensor: tensor_2d}))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "# Basic Operations (Examples)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "-" + } + }, + "outputs": [], + "source": [ + "matrix1 = np.array([(2,2,2),(2,2,2),(2,2,2)],dtype='float32') \n", + "matrix2 = np.array([(1,1,1),(1,1,1),(1,1,1)],dtype='float32')" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "tf_mat1 = tf.constant(matrix1) \n", + "tf_mat2 = tf.constant(matrix2)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ + "matrix_product = tf.matmul(tf_mat1, tf_mat2)\n", + "matrix_sum = tf.add(tf_mat1, tf_mat2)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "matrix_det = tf.matrix_determinant(matrix2)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ + "with tf.Session() as sess: \n", + " prod_res = sess.run(matrix_product) \n", + " sum_res = sess.run(matrix_sum) \n", + " det_res = sess.run(matrix_det)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "matrix1*matrix2 : \n", + " [[ 6. 6. 6.]\n", + " [ 6. 6. 6.]\n", + " [ 6. 6. 6.]]\n", + "matrix1+matrix2 : \n", + " [[ 3. 3. 3.]\n", + " [ 3. 3. 3.]\n", + " [ 3. 3. 3.]]\n", + "det(matrix2) : \n", + " 0.0\n" + ] + } + ], + "source": [ + "print(\"matrix1*matrix2 : \\n\", prod_res)\n", + "print(\"matrix1+matrix2 : \\n\", sum_res)\n", + "print(\"det(matrix2) : \\n\", det_res)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "# Handling Tensors" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "%matplotlib inline" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import matplotlib.image as mp_image\n", + "filename = \"imgs/keras-logo-small.jpg\"\n", + "input_image = mp_image.imread(filename)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input dim = 3\n", + "input shape = (300, 300, 3)\n" + ] + } + ], + "source": [ + "#dimension\n", + "print('input dim = {}'.format(input_image.ndim))\n", + "#shape\n", + "print('input shape = {}'.format(input_image.shape))" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ4AAAEACAYAAABCu5jVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcXFW1779r73OqOwMJYcgcBCJX8SkXUUFwAlEEuV54\njgw+BZwHFJwAL2JwQlQUAUElooDz8BT1giDDE1FQvIigDAYQSAiZSUjSna5z9l7vj32604Ru0tVd\n3dXdWV/oT6pOnaqz6lSdX+291tpriapiGIbRCK7VBhiGMfYw4TAMo2FMOAzDaBgTDsMwGsaEwzCM\nhjHhMAyjYYZNOETkUBG5R0T+KSKnDNdxDMMYeWQ48jhExAH/BA4GlgK3Akep6j1NP5hhGCPOcI04\n9gUWqepDqloAPwSOGKZjGYYxwgyXcMwBFve6v6TaZhjGOGC4hEP62Ga57YYxTsiG6XWXALv0uj+X\n5OvoQURMSAyjRahqXz/uA2a4Rhy3Ak8XkaeJSA04CvjlljudfsYZFBqJZSSWgRgiUUNL/8745Bkt\nt8FsGj82tcKuMnZRD4GuGNEiEkOdIgZijEQtm3KBD8uIQ1WDiLwfuIYkTt9S1bu33E9QpATduI6u\nu++kkAzf4nFIsWQJHbf8qbVGbIHZNDBGo00w8naJlKjWKL2jsx7I2yLyzP+FnzAFXHPGCsM1VUFV\nfwM846l3AnHQeedf+eehR+DqBdJiV8hjoeT+y77XUhu2xGwaGKPRJhh5u5yDqI5SIy6vAXX+7fqr\nYd+XoU3yEAybcAyEAw96KTEKodYO9ToUdbTFwvE8QGNoqQ1bYjYNjNFoE4y8XelInjYCZVmQ5210\nMpkJCjIkz8ZmWppyftDLXoHPIr6MeGA0BF5e0GoD+sBsGhij0SYYebscEFHqgGik0Ei71vuOdQ7h\nGIZhGA1hwmEYRsOYcBiG0TAmHIZhNIwJh2EYDWPCYRhGw5hwGIbRMCYchmE0jAmHYRgNY8JhGEbD\nmHAYhtEwJhyGYTSMCYdhGA1jwmEYRsOYcBiG0TAmHIZhNIwJh2EYDWPCYRhGw5hwGIbRMCYchmE0\njAmHYRgNY8JhGEbDmHAYhtEwJhyGYTSMCYdhGA1jwmEYRsMMqXesiDwIrAMiUKjqviIyDfgR8DTg\nQeCNqrpuiHYahjGKGOqIIwIHqupzVXXfatupwLWq+gzgeuC0IR7DMIxRxlCFQ/p4jSOAS6vblwJH\nDvEYhmGMMoYqHApcLSK3isjbq20zVHU5gKouA3Ye4jEMwxhlDMnHARygqstEZGfgGhG5lyQmhmGM\nY4YkHNWIAlVdKSK/APYFlovIDFVdLiIzgRX9PX/BmQtAlGLxYubHyPOHYoxhGH1ya4z8aOFC8quu\nBmnO77qoDu6FRGQi4FR1g4hMAq4BzgQOBtao6tkicgowTVVP7eP5GoOCi3T88WbuP/BgtOgawlsx\nDAOS/6HEAREPaN7G02+8gYn77Q+iOHGoqgzlGEMZccwAfi4iWr3O91T1GhH5C/BjETkBeBh4w1AM\nNAxj9DFo4VDVfwF797F9DfCKoRi1rdA91pN+7vf3nEb2H+t0v9/e73sor9N9u5vxfO6Gk6E6R40h\noiJ4VcAREBwBBbSPr7QiqGi1v6BkKCVedFy6pB1QOhAcREUQ1IFqY+9XxBE14kTx6okIUSKicdhs\nH++YcLQQIfmq0tc3UooHPDUVoOxzf6fdc1hFKPEoXvvae+wTgVwzCi0rMVWcDsa/p/hKaSIRQcg0\nYrIxeEw4WkgEoiQxqAFeAyHJST9fau15XqhURxWijtMht0ChZXLwAbl4VIWSQGxgyKFEAuAERNIJ\nsy/+0LDz10IckKkjOKGLNHR2qgS0n8y8Sh6cw0cFYjUMT7/H440IOPGoBpwImzQg4hBtLHNREXLv\nKUMEiYgA6sDGHIPGVse2EAEyFxARatHhcBTiaSMjTVqe+CcoEYdGj0MQgY4Yifk41X+BUiPee4Km\nsyACDu3z/PT3BxBDpI0MUUE9bPJqsjEExuk3bvSx2aufYgQKxKyd/A1vYN7p72NCPhWngc5MaCsd\nQfrSdCE6IQ91Hr3gYjZcfDHTN21kXVE8Keow1CjESLJlxKM7ipLLJMpXvIR/+9LpbNhuGhOKGqpK\n9N17DgyvDpxS3H0vd334NLZf9HcmoFjW0OAx4Rgh1AsSHaoZjjpBhKmv+09mfXoBsvs8cs3wKmQO\nXCwR/+SPJkUTFI3K3E+cxlIKNlxwEU4hioI6MlFKIujYEQ8Bogio4gScZpRtNSa87vXsfu6nKXac\nw/YKwUW8Ck58Q29MY0zRqFmzmbHXs1m76O/46njG4LCpygiggA8O0YhQoHgmH/tG5p77Fdxuc8nV\nkWmaijhi8uL18zolELxDdtiJ2Wd+hmknfQiItKmiLhCJlCJk5H2GdEcrpSptro0ISK1k2mGvZuZZ\nZxB2mkMWHeoET8QN4j2JKkRPNrGNqe85Hg+UY+fUjEpMOEaAFFyVFEbNHJOPegPzz/syOmMWGTle\nHYUX6l4oY0ny3vX1OgLicF2RTCJuylRmnPM5Jp58MhsmTMLjUUl5C4ECN0YcpirQhiNoF5plcMCr\nmH7JuUycsys+CNEpGmOV5eIa9gOrS69BdEx+6Ytpe96+eDUfx1Aw4RghvEREFD9/PjPPPAO2n06O\n4FV6phU+KhnZU14XWQhILVT3FKcZT/vIh9npefsQo6IquBDTkH8E3lczEByOSNHextTXH80e//e7\ntE+dQxAFr4gEPEqkutgbHS1UUZjoBMlyZpx0EuJrY2g8NvoYK9+tMY0CwSlBoTZpCvmMuQSJ9EQF\nveJCgYuavP79faUFglckRlQ9USTlj+6wA7LD9qgTnCrgaTC5sqUEFQKe7Q89nJlf+jS17XZMPhtR\niBFRoXSekgykRz4GjIgiIaYEMhwTDj6I/H89x4RjCJhwjAAKSIQaGa7swmuJ1wwXIxILQHHeo5lQ\nZtDfT6pExUdPzNpBBOeUKCDq0SgECdUzHSBpWD+K6M42kV63FcgyT37Yocz7+leZMGse6iHJhIDP\ncHjyILQBXmPDF3xdAOfIVHEqZDtuz3avOqhfX5KxdUbXN2uc4kg/nvUqMTyKS/4H5xCfI+JJWRxC\nhvTkHjz5hQTx6XHnIMPhxaEOolPyAAEQ6pRIT5r1aEFFUMl6xMM5T2ybwKQ3HsUzL/kautNsJAqB\ngKahWBIJR4r/ieBcBn2GqvvHqSNKFV2JELOM9vm7IO0Tmv0WtxlMOIwRI1clKjipERyIC0w78CBm\nfW4BYfo8fHRERxU9aR6ZBpwEgs+IXthwzz0s+9olxE2bmniUbQvL4zBGjFS4pUwJ9a6NbL8DmHX5\nRbTtMBeJjlBFT5xzBDy+SYkodXHkXQIuUNz5N5a86Ri4775+VwQZW8dGHMaIkfJLoMw9O/7nkezx\no++T7ziP0kXUB5CAFyVq8t00ixwHLrDxlpt5+Li3ER/4Z4PL5IwtsRGHMWKoCl3ADgcezPSvnIWf\nOYMoZfJZBEVEKMWlJDBiyklpwm+bSJ2ND9zLsve8j66778JpVdvEpGPQmHAYTaf7ctxy/WnmM/ID\nX8a8S76JzJpBIUpW1cbA5anWSAA8xGolbEPH1d4xmyqVPQY6/nYvD73uP6k//CDQRl0DExDCE6w1\nGsGmKkbTSb/mm1PePY6YtzP5yNfy9G9fRJw1BzTHUUIUouQpWbYnesKgoidRI1oWBCJRQWPBut/9\njqXHH03H4odABRe7qAGlqI04hoAJh9F0cpRIxLmc4CBmkWkveSlzz/ksce5uSOkoneJk83rhZuDw\nBB/xEZxEOv9xL8vf/R7W33kXWZU92i0V0Ra4DQmbqhhNJ5IqqBYaIW8jP+ClzPzBxeTT5kAUNNtc\nNxVxTVvGK1FRXyOGks5b/sLDr30t5aplZCIQ0/jCIahgK2OHiI04jKZTItQArXmmveZI9rj8Utp2\nmkfIItHF5BAlphIBKcm1KYSqJMHjN/6Oh975TsKK1QQVYkwikXwuKWrj+llIaAwMG3EYTUdwdBKY\n9qKXMfPcz5PNnAlSpoI6muqGRC8IUhUe1n5XBDeC0zqb7rmf5e9+D/X77ycjI2rKoBURymqUkbn0\ne2nTlcFjwmEMmr6jJ4LLPBMPPoTdLvkmzJxOkFSAB4l0uZy8u2aogGoASSIy8OOmxYC972sIbPjz\n/7DoDUfhli4GgegK8uhSOwSqqvIihBhp4kBnm8SmKsagSVGTnNgTPRFCXmO7I1/Lbt84jzBzNmgO\nBFRT9KQd8JIGGEIqRuwa/RoqaCyJmqInJYH1v/s9S9/xDnTZEiBVjpeY2iGAIqrVmqH+CkEbjWDn\n0Bg0OUog4l2tip4o0/Z/EXPP+Ry6y+7DFj0RFQoXIUZKAuGu+3j07e9k4z33pBXETTyW0Tc2VTEG\nzROjJ+3UXnIgs77/DfIdhjd6gkCuOTEWdP7+Zpa+6U1sWrW8SgPZnABmHozhw0YcxqB5YvTkCJ5+\nycIUPfGR6EJP9ESbHD3R6ugbf3sdS97xTjauWY1XwWvl7yBNo2wJ2/BhIw5j0HRHT7bf78XMPO9s\nsumzQQq8ZqCxj+hJbDgbtE9iJ53/eIBH3vYOipWP0B5zSkhTIu3udrc57dxoPlv9FEXkWyKyXETu\n6LVtmohcIyL3isjVIjK112PnicgiEbldRJ7Uzd4Ye/Su3JXup/KGLsuYctjh7P6Dy2mfOZvgI6gH\nAnWfEX1yfIqkVlLa4FWsaBqtpK4QadVsUWft727mHwcdzKYVj6AKJQUiUjlKUzGj7iiKMTwMRP6/\nDbxqi22nAteq6jOA64HTAETkMGC+qu4BvAv4ehNtNVpE7+hJqmgKIcvZ7sgj2fXCrxBmzQbNEEK6\nwF2eyvwNNXqSFpz0ip4UPH7ttSx/93vwj61ANWWCpja6yTpP+lL3W0XNaApb/SRV9SbgsS02HwFc\nWt2+tLrfvf2y6nl/AqaKyIzmmGq0ihQ9CWS+jeAgZMq0fV/IvK+cjT5tPlKk6AmSpiRN+6UXR10i\nLgSUkvKOe1l2wjvpXLSoat4k1YpYY6QZ7IRzuqouB1DVZcD0avscYHGv/R6pthljmEgq51ePXUje\nTvvBhzPr55eTzZxLLQqSKXlVCBjnKn9Gc46bkRM1sP7a3/LgQQdSX/YIiuJFENWepK7x2XZ79NJs\n52hfPzb9fp4LzlwAohSLFzM/Rp7fZGOM5hAQcpR6zTH1P45gly99Cb/THIKUVTmc1ItB0yqy5o04\nFKLU6bjuOpaefDIb16+jhpBVvo9uulPHzafRN7fGyI8WLiS/6mqapeqDFY7lIjJDVZeLyExgRbV9\nCTCv135zgaX9vciCTy4AF+n4483cf/n30dDfnkYrEYRNwPbP25/ZX/si2U5zRiR64rSTjX+9m6X/\n53jqj60kjx7BEavGDwqVU9SE46l4gXMc/fa3M3G//UGUT535qSG/5kA/XeGJn8svgeOq28cBV/Ta\n/hYAEXkhsLZ7SmOMDXrX0Ird97Ocqf/xGnb/yQ9o23kmwTU7esIToieqEOubWHvltdz3soOpr15Z\nCVIAwhMcn1qlkFtC0siy1RGHiHwfOBDYUUQeBj4JfB74iYicADwMvAFAVa8UkVeLyH3ARuD44TLc\naD7JT+AJRJwkn0XpM7Z/zWuYc85nCbNm46MgUqLqiD5FT3r/pIgMIp5RtZ0L0p3xGVh/5dUsO/lk\nso51KbTay+lqiV2tZ6vCoarH9PPQK/rZ//1DsshoKRGYRM4m6pArU569D/O++iXi3HlI3RHy1Je2\nmdETIXVby0NJdELnbX/n0eOOI65bayIxSrERnrEZETyBLqlDljN5/1cy55c/xM2eRx4FySMZaSSi\nzuGadFVHBE9EQ2Tdr6/koZe/hLhu3eal8NV+vdtGGq3FhMPooQqKELxj+1e8ml0Wfp222U9LjZ5d\nQCXlZKpWBTWa+O0pUdZdfQ2PfvADdHV2bfav9NpHeLKzzWgNtlbF6EE0LVybuOfeTL/4PNzseXSx\nkfY4sSd6ok56Er2aFT2R0EnxP3fzyJuPJa7fQA2h9IqEJ44wTDBGDzbi2EbRXv92tzFwPmfyK17F\n7r/+GfmsuRCVdp2AujpFFT2RIUdPYk/kJCjErk7W/vc1LDrsUFi/HhWlJOJC/0lBNlVpPSYc2yAq\noOLQqnqFB0qXMfHQw9jlwi+j83bBq6S2jBGUdmpUa0/cUCp3VUvsiRQEohas/fVVLP3AB/FrV1EC\n6JO/lL2nJzZVGR3YVGUbxGnKhpgoykYBFWXSns9h3kXnoXPn4UshZBFX/bQ3OrLoD1VXrast8VHZ\n+PtbWXr0scSi60mdXG16MrqxEcc2SERwFHRIifiMKQcdxrwrf46bPZdaBLJIppp6kHiXusY3ARGo\nS0noKlj30yt44DWHEoo6sZosmUiMHUw4tlE8gjph6sGHMeeiC2ifuwvilOhSdiYSibHyYzTpio6S\nmiZtuPIaHv7IyRQdHT2RnC0xIRnd2FRlG0RQunBMfOZeTF94AdnsXYhsJIsTqjmCELxDPPgIEME1\nYe1J6KTjxv/hX296E1lQoqb66BF5Qu/43nkbJh6jExtxjHNUuteeSE/0BN/GlEMPY/df/4LarLlA\nIIsTUKkTvCd4XxXISZf1ILoX9ERPYlW5S+udrP3Jr7jntUfiyjqlFmnNiUhVHujJr2GMXkw4xjMC\niiOQISgZUIpn4qsOYd55X4Rd5qXoiaR2jFHayQQyASfVl8N119RqhFj9n/4jlqz+79+w5MMfprZ2\n9RNHEr2Wx1v0ZOxgU5VxTIqeOCYJdAiUTpn0zOewy0Xnp5BrFT3puUCb5ctQhyPiKNBSePyPf+bR\n178RjWVzDmC0HBtxjGNS9KSkQ0rwGVNefjjzrvoFbs68XtGTWOVsNDF6AqCBsKlk1Q9+zOLXHE6I\nwaYf4wgTjnGNVNETmHLQq5jztfNon7Nl9ER7oifNytdAInjhsZ/9kkdP+xjF+g1EaaQ7rDHasanK\nuCbSJY5Jz3wOMy75WhU96XhC9CR6B02Onki5iTVXXMeytx+P60pO0EIU1c1d1oyxjY04xgOVCCSn\no/REUsTXmHLoq9ntv39BbdY8ID4peiI90ZM+cr0HcFiN3ZW7IhqV2NXFqh/8jCVvO47Y1YUSCUQk\n0mf0xBib2IhjHKCiCEJBxgQCApQ4Jrzylcw99/PovHn4KEQXUwFgl6InT2AwI41YJ5DhAgSvqAgb\nf/VLHv3wKbBuTbKt2rV3noYx9jHhGAcIIFHYzkGXKiETJj1jL3b5+gXoLvPwpSNmsQp9NtHTIA5X\nCvUM2oo6635zJYvfcDRBgg1lxzn2+Y4LHE4iGymRLGPKKw5N0ZN5qXIXWcRrxFc+jaZ1OYse8VAr\nC9b+7McsefM78VU3N2N8Y8IxDtBUJQM8bPfSQ5hz3rlMmDMXQYmu6qZaRU+k2dETKQkIXQ+vZFOx\nsSoDaPGT8Y4JxzhANFIXYeIznsvMb19ENn8PSrrwUfDB4YMjkoHPcRFcbFKxUPFARDLP1LcezcRd\nn0XdK5axMf4x4RgPSKT98Fcz/9e/IJ87Fw9k2ob6vtaeNB496fewQIg1ANqmz2Dm979M26y5ZFRV\nxSSNPNTiKeMOE45xgOQ1ph9zLLW588hUqjUmgLT1rD2RnrUnzasyrIAXTZXANGP7vfdn+ic/issn\nUUoaeZRATTLEmXiMJ0w4xgMi5JMmoKJENLVEHIHinN21OlQipYDENqa89k1MOeLVuEyI4smpsUkL\nAmpftnGEfZbjBAVUBKn+RowoRBRfdWiqTdmJqR87kYmzdiOPASHi8dDEBk5G6zHhGA+IEDU+6coc\n7tUh0t2dXgU0IJVQTNpnP3b6zH+h4il8ABGcOuvKNo4w4RgHiCoZrup1klAd/p941ZSJWoYA4lEc\nqjlIjfY3H8sO73gH4Ai+RHr6shnjAROO8YKkdSM9d0diuuIAlDbJkwFeIAMnyqRYY+eTP8CkvZ7X\n02dWK2epiiP0VOrpVZnMGDNsVThE5FsislxE7ui17ZMiskREbqv+Du312GkiskhE7haRQ4bLcKMX\nksImzrseH8dICIfDIa6qTdoduakO2xUKanvswU4fPZG29gnUHSlJzQGSan9kCpnzFm0ZgwxkxPFt\n4FV9bP+yqu5T/f0GQET2BN4I7AkcBlwoI+qpM0YDApDnqV/L617HhDcexcSYlrnVIsQYUOcpgHos\nsaX2Y4+tCoeq3gQ81sdDfQnCEcAPVbVU1QeBRcC+Q7LQGHMoSlsZcZqT5xOY9/kF+P1fDCLUfVpZ\nWWgEJ2T05IkZY4ih+DjeJyK3i8hCEZlabZsDLO61zyPVNmMbIq2FcdX/Eb/zXHb+3Bm0zZpDTTMC\nSo1ePhkbcIw5BiscFwLzVXVvYBlwTrW9vz7BxjaE4Ci9EClB65QIU/d7EZOOPopSPG0IpSgiUIjF\nWsYig6rHoaore929GPhVdXsJMK/XY3OBpf29zoIzF4AoxeLFzI+R5w/GGKNhYtV00UUBCYASNQMX\ncSoEBK/dI4eQUsobnE9kQEARydLalfZ2Zpz8Xlbf9lfkhutwCrkKdbEiP8PNrTHyo4ULya+6mifE\n7IeA6ACKJ4jIrsCvVPU51f2Zqrqsun0y8AJVPUZEngV8D9iPNEX5LbCH9nEQEdEYFFyk4483c/+B\nB6NFV1Pe1GgkLX1Xanvtxfwbf082ZTKCa8nPbSQSk4uBQiEnsvGm38GkqUx+7nMpSsjyKrGLVGHM\nydCreGiAjffexr0vPpTssZVVTmnobvtkNAlHqgAHEQ9o3sbTb7yBifvtD6I4cagOrbjCQMKx3wf+\nCPybiDwsIscDXxCRO0TkduBlwMkAqnoX8GPgLuBK4L19iYbRWiSA1wBlAUWdDT/6CfceeDhrPvMF\nutasJM8E0dCdx9684zpo23MfZn7mDNyE7dAsojiiTVbGHFudqqjqMX1s/vZT7H8WcNZQjDKGGedQ\nhQJY991vs+Ljn8BLF2tuuIa2X1/JDm8+Di8Z4kCq7ixNoSqivvPr38Cm/3cjHT/9KeqEQpXMfl7G\nFJY5ug0igKhj7S9+zKqTT4EVK9AItccfZ81JH6fjoX+CE4QA2kT/gwQKFLfDNKafchJ+4o5ojE1r\nBGWMHCYc2yBl10ZWfP87LH/j8WzqWEuZ1q9SqqLrlrPiPR9G16xB1dEVmrfGREVpQ4m+xuR9XsiU\nL38GJk7BAm9jDxOOcUwESlLXeCUV1tGii9XfuYxHP3QqQoEPoKQVrApsFCH8/jpWfvtiYlFQ8z7V\n74qaFs5FQEFDiTbo0hRxaUGeRlQdM044lqlHvR4lI1JFYlxaxyLVJMkYnZhwjGNEI76EEEpQxQVY\nfvl3WPrRTyCrlvdc9mmtWlpq1oZDNnWx6usLWf/AIiQ6ogbKqkhQt28EcY2PE2IV1q30QP0kZn30\nvUx+7r9TtmfJjhjJtarxYSORUYsJx3jHB7wXwsYuHvn2N1j9tvfiN6zuc9fk+wgUZJQPPMDKT3yK\netdGUu95JRKryuapEEfDNb0kjXyipFFQFgTdYy+mfeBE8pjTRVoIF7MczyCEyRgxTDjGNQ5Vh5aB\n1QsvZM1pZxDov6BOBIIIdRdQETb97OesPfdcNEby6PB4IoE8Jqepa/TK1pRV6hScU5wouWZs//rX\ns90JbyGKQwXKWKcQrNjgKMY+mXGMAqV41lx4Pqv/63TC6pVEtGpr8GRcVQwoRKVdHeICa8+7gPU3\n/ZF6yifCiQMCuLJxD4QAKgRAYolSgA8wcSKzv3wOE194ADVVsijkNtwY1ZhwjGc6HmflV7/Mwx8+\njXpHFx5P6UOq3NUHkQgZTALqKETP+lUrWfnFc/ArFhMdqDqiCKhvODesFEWjImREl4HLAAGJ1Nom\nsvMZp1DMmkkbjtKRpkXGqMSEYzygCjH5DUKEGKAIdR47/zxWf/osspBGB0ED7UGe0umoQUmrVwIl\nkEel44YbWHPZjwElRkE01dpodFAgVcVi13NP8OoQTZmq2734Zez0lhMoJOIEgkKoOs95IIojOou0\njAZMOMYFkUJSPwRHgQgs+tKXePhTZ6FrVlVLyJLHIDzFgjIBXNVZwSm4bgnpXM+jn/4iHbf+CXHp\nYhap5i4N4HGIF1xVKcx1N3sRQZyQTZrM7I9+kPIZz4IQaXNZyiitbCo1Imqej9GAfQbjgCipX2uh\ngWJjJ0u+dBblxxeQd3Y07RjZ+hWsOukUyuUP4pTUuyU2xxEhlER1gBB2mM78n1yCmzGbUku8q+HJ\nKUV6mmWHphzVGAomHOMAV110vnSsPP9rPHb2F8hioGhWaS2BAmHDbX9m5cXfQTu7AGnKilkAF9Lr\niCq5RrZ7xvPZ4dSPIJOm0BXrFFLiVak61VqYdhRgwjEOUJJPY9nZZ7FmwafgsXWs981bA+LwqaZG\nvWDNhZfQ8cAdSCGUzXl5gk9JZiqBKIHOTJh+9LHUXnIAHmhTxUmq3C6VgBitxYRjHKCxZNVXzmfV\ngtPRok6BMKGJP8uqgULSCtawYjGPnnQa5cZlTbyAFXGp21uIgo/gd9qReaefSjFrD6ITugSiZnhv\nsjEaMOEYDxQFm37/e3xMIVWnkRCb18dNABTqoniBzutv5KGzvoLWN6BEygBoTGtZgtK40zQlk4l4\nMvHUXHKWtr/wAOZ/7gy6XI6PgvpIDFYtbDRgwjEeEI+K9jgNU/Oj5vkCPGk6FFQRhYySjZf+mLU3\n3ABdqQFTQUQJFA4aLi4lDueqfjBOqjYxDi852x91ODuf8FYEyENoWuk7Y2iYcBhbJS2s9+RkoFA4\nyFY8xMpzz6dY/SiFpOZKMUq1YrY5Yx0JULRNZdr738eEZ+/NpsxVXeEsl6PVmHAYWyU5QZUMIYij\nLlB3Sv2GG1n2ja9RIyDRI87hS6o+sUNHfSSLMOnf9mT7D30gNdbO0njKaC0mHMZWSV+SCERKFSYE\nj48eLeus/+w5rL/uWuq+agbruyunD526BATF5Rk7v/UYdnzrcUi52SKjddgnYGwVR5UtSiAj4CsP\nSogCWuepabQQAAAVT0lEQVShj54O/7pv8+J7zZtyXE+WKhyLomTs9oVzkBe+IK2pMVqKCYexVVL1\njTRlEVJVsUDEoXT4jHDXXaxaeDGxszP5H2KAkAr+hJgCLRq14e4pXtOoQ6smCvUdpjL71NORWXMp\nyStPR5Yetz6SI4oJhzEgvG72LHQLiYiSFyVZfRNLz/sm6/5xG5AWopUuhYUFpRykT0Ji6rwiUcki\nFJTscMjL2P51/xtx6VUjJRme3LpwjCgmHMagKdQnhylKrWMtK9/3MerLl6acDDQJh5Ygoaou1miY\nNrVqipJW0k4gp94+kdn/9VHanvVsohfEAdSJTUp/NwaGCYcxaByRIGnRWd3Bpr/8mUdPO5O4qQMJ\nPtXscI4cRQfjMK1W0QanBA8Zjjw6dMZsdv/OQmoz5yIKnQ6izVRGFBMOY9BkKJkqtTzHBYd3kQ1X\n/IzHr7mG0kciQsCniuXSeFO4qCDRUSsURyqW7JyQq6O29/PZ+YPvJ6eGy9qQGC2/YwQx4TAGjZIc\noGUZ0iK0AJs2rmX1F8/DLfkn6mJVuVyQ6BteDi/OEQWiT6UGA4GQih8i3jH12DdTe/Wh5PUuoq8K\ndxgjggmHMWgiVK2NI2hKec+LyPpb/sCjZ1+AU8VpanuvhMaLG1O5OUQQl5Hhu+v+EDWSzZjFTgtO\ngUlT6W51W1VEBVKfFgvcDg8mHMag6c7h9JUztCfaoiWrv/YdVv38F6CR6FKBZNdgDVEhtUtwUq1j\nEUlNndBqO0zd5wXMvuRc2qUGQEfmaCMD53rCxzaBaT4D6VY/V0SuF5G7ROROEflAtX2aiFwjIveK\nyNUiMrXXc84TkUUicruI7D2cb8AYfaSoykZWnf5JOhbdk3wTUZs2k4gEpFoJrOKZcsRRTHnLscS8\nRl4qBQEiFOJSn5bmHNboxUBGHCXwIVV9FrA/8D4ReSZwKnCtqj4DuB44DUBEDgPmq+oewLuArw+L\n5caoJQB4CA8sYsVF38B1laiDpi1+w4NLBZUlQp61sd0H30O22+7glRytKqRLNdcxms1WhUNVl6nq\n7dXtDcDdwFzgCODSardLq/tU/15W7f8nYKqIzGiy3cYoxuMgdlcMW8jKP92EaKOdZvtH1UGMqWCy\nB3UFk/bamxmnnkgWJ6CSwsOZE6JaB9rhoCEfh4jsCuwN3ALMUNXlkMQFmF7tNgdY3Otpj1TbjG0E\nQVEVglNqZSfr3nEinQ8tal5hIVclrysETa0oM8nY+dg3k7/trdQ1pz1CEUpzcAwTAxYOEZkM/BT4\nYDXyeKoq+1ticbJtitTCIIqgAp3338PS0z+HPr4KNFBHCTGiQVENoI0FagXAOZx4PCBkaVu2HbM/\nfToTD9iPqDCx8m/Yl6/5ZAPZSUQykmhcrqpXVJuXi8gMVV0uIjOBFdX2JcC8Xk+fCyzt63UXnLkA\nRCkWL2Z+jDx/UG/BGG0UwEQyQlSilkQHG678NauufhU7v/ZocCkdve49WemImdDYetoUUdl8DxBB\nFSbPmMGcj32IB0/4J2HNCmqaU1Bs0wOPW2PkRwsXkl91ddMqqIkOYHGQiFwGrFLVD/XadjawRlXP\nFpFTge1V9VQReTXwPlU9XEReCJyrqi/s4zU1BgUX6fjjzdx/4MFo0dWUNzUaUQSHUttrL+bf+Huy\nKZMRXFOG0rGrzgNvfD0bf/mrnl/XiCOj8RWpzSAKoJ6AIyeAj7gg+H32YZef/4TJc3ellNTsKY0Z\nXFN8mEFTGny5voNHTv8Ea87/KuIcEsM2JRxpJXPKsPGA5m08/cYbmLjf/iCKE4c2XN/xycd4SkTk\nRcCxwMtF5K8icpuIHAqcDbxSRO4FDgY+D6CqVwL/EpH7gG8A7x2KgcbYwykEAt6l9asuwCandN1+\nG0tOPYPY1YWPDleGKkOrOW5TASQ4/OTJzDjjw0zcf3+yGCxZaRjY6lRFVf8A/VbCf0U/z3n/UIwy\nxj41wMeSoppCTIkZm1xB1w++y9pXHsLktxxDntVSnQ7p/wvWEAKdXmmjoH3KHOYt/AoPvPhw9LFV\nzXh1oxcmxkbT0Srns5DU+qAUpaREo6AOFn/+c4S/3ZkWsaXW0005blRoj0qIHjJom78XO5x6Ejpx\nYlNe39iMCYcxLGjVKTpqmroUKEGUKDXk/vtYfdEFlEWdKA7RAEFRTdmGGgti2XiBwAzACZlLviNX\na2PaW08g3/8FeByO9IXvynPLJh0iJhxG00kdXbTHIdm9XiRXwYU6xJKll3yXTVddlUogqxJ9SUnA\na0TEoyINO3a795fK0ypA+84zmP/FsyimTKEEHBlZEciqtS3G4DDhMEaMiMOJ4IB27eLhd32Qzn/8\nDUGQIKSF96mOh3M0vJq2WzB6RwpFhOzf92XmOWfD5KnUKcldRtSieW9sG8SEwxgxogScKl0KAaFY\nu5gVpy8gPrYqrZ4NnohP9TYG0Yuuv9QCr46d3nAM0444EvEOYkEUayU5FEw4jBEjq7qw5d4j6qjV\nYf0NN7DmJ1fQpQWF91W+C0RR4iCSO6SP59R9STZhEjt9+EQm7b57KgdgqjEkTDiMEUMROkVTzwQi\nIuDWd7D6ggth0R2gqRShUEdUGi9uTN+jjixmSKa07fXvbP/RUyiYgLNVs0PChMMYMRSt2iyk/0oF\niHTddQcPvfsTsOnxpCnkCHFQlTR6jzi6b3uELgmIOHZ885uZ9u7jEfUIWbc7lUiGer9NZZgOBRMO\nY8RI2R2JzV+8VF6w83fXsfKb38ZpnQIh4Br2QfREU6pqYZu3KbWQpW507TVmf+Jk8v2ehydQ+pRF\nUq8FXAgwiONui5hwGKMAh7iCx8+/kI1/uQWqRk5Nq1qukSgpHBw0wox57HzSSegOO5CpUDhPrdQq\n08MYCHamjJZTElF1dD50P2vO/SYSA6qOIa7D6iE6T5QSDQUeT6Y50/7zCCb9nzfhIzgCuXoKFB1M\nReVtEBMOo+Vk4hGNEJXHf/JTll2ykEID0qQl4FUbOcT7qrdLRNsnMOeM/6L27OejkVSNPc8oLNwy\nIEw4jJajGkA8qhC1zpqPfJyuW25qmq9BiWQxJ2WHRKIXMiK1abOY8fWz8PN2pS4RKQtqakXDBoIJ\nx4ihFEDuhBAdqFSthRp7jRBLtFQ0NYRHA8QYyaLS6UDJUte0LFYZEaMfBUqN5EDdQ75+Hcs++wXK\nFUvQIlAqRA0QFQ0BbbC1k0MQlxymThy+KmIsClNesD87vu0E6m2TyIEg3taxDICx8c0aB+QitONY\nD6jrIkrjQ/EYlQJHPQMk4KqGzsFB4RztQBuQK7hAw2s9WoakJrEByCOo82y64SYe++H/RTPBqSJl\nZJNLq2sb7SUpSFXwXFJ6exV1KVyB+HZ2OOa1TJ2zM50K7Wr1OwaCnaMRQIEuVbp8ZLsI7dGnVaAN\nXtcOR00F0ZCuHRFUFR+VPHraI3RRUjpHWpw6Nn47nSqTKlNVoYOAdHWw4ivns/EffwMCZebJguLK\nplW/w6sQUNbd9nfqKx/HC3SJ1SgdCCYcI0RWDY2DCkEVp1nD4b8kFmlILzgInhgzxEGZRQrAu9RS\nJBtDn2zpYD1VGwNxTIypZWTXww/xr6NPQDaswanDR6VwoE3K+nQxg8fX0/H971E8vhG7HAaOnakR\nQBCCOjKFjcuX8tiNNxBifVDjAZHk4EsqouDrrL/1D3Teew8RqGsamdSb/B6GExfBZY4aCqrUUXIF\nT0T+fjuPnvM1KDdRz0qc8zStU4oT1tx8A+uuvgqRLlRT1Q5zjm4dE44RIKIIkUIdbsVKln3kNDqu\nuIJGB8VKKsiLOjRGgivZeNP/45H3n0q8dxEZHl+5ROuud0WM0Y3gcLFy9go4yelE8BopHDz2ne/S\ncdPNKA4XtWm5nfUN61l95meResADGQWqY8Yz1FJMOEaAqkoEOZHSK+5f93Hfu07k8Z99i1JjCkOG\nSMkmQlknpjo4T/4jIhJRPOojxXXX89Drjqe88zaIkZK0bD1SMqGEsTJbVyISIwEQVVRLPEpwGT5C\nXPIgK8/6Am1hA6VUPo4G/so0kCEGJaqimkoZPn7ZRay9425yTetmRMaQQ7nFDKivijF01FVtCxXK\nCFNXr+H+kz7Nbm4y7a86nLZYAwmIRoqs7HPALMTqYnBs+MvNPHLiR3Crl1GoPOFXONWzGDvIk26n\nsZJS4oFCIo9ddz1tXzyfnT5wInVpa6idglOlFMGFlKSxyQeyB1fyyDd+SL6pg4BQutRESiSN5oyn\nxoRjBBDAxfRb1gZ0ShplTHj0EZZ/9HTyX12J8xNQF8hDSBe+SnXxbL6wogiinugyOm/5A8U9d4FX\nYpSGq2WNBbIIiEOip6aBR878AsU995K5NqJsnoh1n6Mt/938mFI6IVNHJFJ4gSUryO5bhEMIEpkA\nJL0YhydyGDDhGAEUKKWaF0aY6GCTRDaKkP1rMcUD3wMfCZoullT76slfYBGhdFAL1etJRj0UbIey\naaTf1AiwyXnqMbCdOELmybo28Pjl3yPSWEhWSJ1bvDiixlSwOAMCRAcTSOe9DqioaccAMOEYIXLS\nPDtU3+IcqKGUFOQ4QhAKSY7C0E9OqaqSB4gInoxCSyY4YYMq2Tj8sudR2Q7o0pJQwgQ8AQEtG45I\nSe4pikibeLo0kJdJyIODGJJolDWPFKFJzRrGNyYcI4SQft00pT5S+O45t1KIEqOSq1BISqd2/WRH\nllV9CVc5EGMcvx9ilEhdoRBPJpGoIbU2FJcWpQ2QTJSuym8RNeWBeCCoIupJBZIVV1jW6EAZr9+5\nUYWQvqRUvU0VkmhAyvDUWN1URDc7CPt8rS0umPHsxhOtphga0mgNBlUVrFTw1fmOBHz1WilQlYS6\nWkBrDBATjhFiy/FDf/e3FiwYS9GSodLXOWnG+x/ouTb6x0ZmhmE0zEC61c8VketF5C4RuVNETqy2\nf1JEllTd67s72Hc/5zQRWSQid4vIIcP5BgzDGHkGMlUpgQ+p6u0iMhn4HxH5bfXYl1X1y713FpE9\ngTcCewJzgWtFZA/tr1uOYRhjjq2OOFR1mareXt3eANwNzKke7muaeATwQ1UtVfVBYBGwb3PMNQxj\nNNCQj0NEdgX2Bv5UbXqfiNwuIgtFZGq1bQ6wuNfTHmGz0BiGMQ4YsHBU05SfAh+sRh4XAvNVdW9g\nGXBO9659PN2mKYYxjhhQOFZEMpJoXK6qVwCo6speu1wM/Kq6vQSY1+uxucDSvl53wZkLUgLU4sXM\nj5HnN2i8YRhb59YY+dHCheRXXd20ZBUZiM9SRC4DVqnqh3ptm6mqy6rbJwMvUNVjRORZwPeA/UhT\nlN8CT3KOiojGoOAiHX+8mfsPPBgtuprypgxjW8ZByrAl4gHN23j6jTcwcb/9QRQnDh1i05qtjjhE\n5EXAscCdIvJX0rTj48AxIrI3KbnvQeBdAKp6l4j8GLgLKID3WkTFMMYXWxUOVf0D9Lnu5zdP8Zyz\ngLOGYJdhGKMYyxw1DKNhTDgMw2gYEw7DMBrGhMMwjIYx4TAMo2FMOAzDaBgTDsMwGsaEwzCMhjHh\nMAyjYUw4DMNoGBMOwzAaxoTDMIyGMeEwDKNhTDgMw2gYEw7DMBrGhMMwjIYx4TAMo2FMOAzDaBgT\nDsMwGsaEwzCMhjHhMAyjYUw4DMNoGBMOwzAaxoTDMIyGGVDv2OEjggai1hEXCK03yDDGPKWAUwEy\nlBInDonNHSO09DoNKjj1eD+JUjMyoCS20iTDGPN4gQIh14B6iETEhaYeo6XC4VQJTimJiEDhSsR0\nwzCGRIiONiKgSISQOaIMqcf0k2ipcJReyFVoKwQXHBI9QnOV0TC2ORzUY6Qd6ATEO1zR3L7vLRWO\nPETUBWLWRayBczVUbchhGENCHJlEigjOZ4gCbWVzD6HaXCUa8IFF9Nprr+PlBx5EZ+c64j/+hoZJ\nOOotsaebG2+7jZfus09LbdgSs2lgjEabYOTtUnE4AhHFB4fmBfme/0623fYgihOHqg5p7tLSEceN\nN/6Ol7/8ICZOmgr7vpQAOJo7F2uUW66+hkP3P6ClNmyJ2TQwRqNNMIrsauIgocXRTyUSK7EQQGmx\nboBo+htNmE0DYzTaBKPKrtikqGVrhaMooKMTtIoxizZTFAdHvUA3dLTYiC0wmwbGaLQJRo1dgqBN\nEo6W+jhacmDDMIbs42iZcBiGMXaxtSqGYTSMCYdhGA3TEuEQkUNF5B4R+aeInNIKGyo7HhSRv4nI\nX0Xkz9W2aSJyjYjcKyJXi8jUEbDjWyKyXETu6LWtXztE5DwRWSQit4vI3iNo0ydFZImI3Fb9Hdrr\nsdMqm+4WkUOGyaa5InK9iNwlIneKyAeq7S07V33YdGK1vaXnathR1RH9I4nVfcDTgBy4HXjmSNtR\n2fIAMG2LbWcDH6tunwJ8fgTseDGwN3DH1uwADgP+u7q9H3DLCNr0SeBDfey7J/BXUpRu1+rzlWGw\naSawd3V7MnAv8MxWnqunsKml52q4/1ox4tgXWKSqD6lqAfwQOKIFdkDKGtnyHBwBXFrdvhQ4criN\nUNWbgMe2YscRvbZfVj3vT8BUEZkxQjZB35k2RwA/VNVSVR8EFpE+52bbtExVb69ubwDuBubSwnPV\nj01zqodbdq6Gm1YIxxxgca/7S9h8okcaBa4WkVtF5O3VthmquhzSlwLYuUW2Td/CjunV9i3P3yOM\n7Pl7XzXsX9hrSjDiNonIrqQR0S08+TNrybnqZdOfqk2j4lwNB60Qjr5UuFUx4QNU9fnAq0kf8kta\naMtAaeX5uxCYr6p7A8uAc1phk4hMBn4KfLD6le/vWCNmVx82jYpzNVy0QjiWALv0uj8XWNoCO7p/\nnVDVlcAvSEPG5d3DWRGZCaxohW1PYccSYF6v/Ubs/KnqSq0m6sDFbB5ij5hNIpKRLtDLVfWKanNL\nz1VfNo2GczWctEI4bgWeLiJPE5EacBTwy5E2QkQmVr8SiMgk4BDgzsqW46rd3gpc0ecLDINJPPHX\nqLcdx/Wy45fAWwBE5IXA2u5h+nDbVF2U3bwW+Hsvm44SkZqI7AY8HfjzMNl0CXCXqn6117ZWn6sn\n2TRKztXw0QqPLHAoyfu8CDi1RTbsRoro/JUkGKdW23cArq3s+y2w/QjY8n3Sr04X8DBwPDCtPzuA\nC0je+L8B+4ygTZcBd1Tn7Rck30L3/qdVNt0NHDJMNr0ICL0+t9uq71K/n9lwn6unsKml52q4/yzl\n3DCMhrHMUcMwGsaEwzCMhjHhMAyjYUw4DMNoGBMOwzAaxoTDMIyGMeEwDKNhTDgMw2iY/w819DdL\nuqg9fwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "plt.imshow(input_image)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "### Slicing" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "-" + } + }, + "outputs": [], + "source": [ + "my_image = tf.placeholder(\"uint8\",[None,None,3])\n", + "slice = tf.slice(my_image,[10,0,0],[16,-1,-1])" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(16, 300, 3)\n" + ] + } + ], + "source": [ + "with tf.Session() as session:\n", + " result = session.run(slice,feed_dict={my_image: input_image})\n", + " print(result.shape)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAAyCAYAAACXroq0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAB25JREFUeJzt3WuMVGcdx/Hvb9kCtdvWrZG2FmGxtNYXNqvWS5AuICmh\nKq7aN+gLpWljoqaamKZATArSmBSiRo3RiEVSmpC+6IuCGtN6CbRVSzHlUi2XbQTrUouNlypREXb/\nvjjPlmGYGWbnsrPn8Pskk5x99sw5z5/n8N//nOecM4oIzMwsH7o63QEzM6ufk7aZWY44aZuZ5YiT\ntplZjjhpm5nliJO2mVmONJW0JS2VdFDSYUkrW9UpMzOrTI1epy2pCzgMLAZeAnYDyyPiYOu6Z2Zm\npbqbeO+HgF7gp8AIcAQYBJy0zczapJmkPQPYERG3SeoBhoBXW9MtMzOrpJmk/SrwV4CIOCHpGHBJ\n+UqSfJ+8mVkDIkLlbc0k7WFgVjq3vR+4HthWacWBmwdYuGgRKFgwbx43HjrMn+9Zxeh//93E7ieH\n7wGf7XQn2sjx5ZvjmxwEBGKELqYwwigwBRhhCt3Tp3L11zaw/7rr2fmrX4MCRoN1991XcVvNJO3d\nwFzgq8DVwCngkUorLlywgLVr14JGiZMn+duhw03s1syseBYODLDoliVZ0h4ZaX3SjogRSRuA7wOj\nZEn7skrr7ti5MyXtVGk3ulMzs4La8cQTZ1Xa1TRTaQN8A/gNsBq4GzhQaaW1997LwsWLC1lp39Tp\nDrSZ48s3x5cfba+0JX2ZbOLxHcDjZKdtBoAfl6+7Y+dOdjz5ZCEr7Xd3ugNt5vjyzfHlR72VdjM3\n12wFPgGcAKaTnVd/OiLmla0Xo/87Dd1TzlTaGx8ozESkmdn51DMRecUdd6Jp01+rtLu6L2rs6hFJ\nm4APA8cj4sbU1gu8Na1yEFgHPAy8UGkba9d9Bbq6Cllpm5m1QssqbUnzyarpLSVJez1wEvgccCnZ\nROS/gF0Rsazs/a60zeyCNqGVdkQ8JWl2WfMgsAy4Ky1/E4j0OocrbTOz2lpZaW8CPgL0RMTFqe0/\nZHdD9pBV2ZcBe4A9EfGZsve70jazC1orK+16Hs26GfhUWdsp4DbgT8Cm9PPvgX3NBGZmZrXVc8nf\n7cDHgGklbceAo2SPZL2brNoWsLHSBnx6xMystlZPRF4LbI6IrtS2HjgNvAv4APCLiLi1yvt9esTM\nLmgTOhFJdoXILYAkvQisAe4nu9TvorTvms9sWXHH7fTNeUshK+3dFOsC/3KOL98cX3607Db2iPik\npHlk1fQsAEnLyCYeLwXeBPyz1jb6ZvcV9oFRv6U4B00lji/fHF9+tPo29tdKdEkXAxvSj/PJ/t3O\nKeFL+YFRZma1tazSTrerLwampdMjG4HryK7JfgmYChyRNDci/lJpG340q5lZbfVW2nU9e0RSH/Cj\niHh7hd8dAd4ZEX+v8l5/c42ZWQMaffbIVmAh8IaxiciI2Fy6XWqcHqm0UzMza0zDT/kzM7OJV88d\nkWZmNkm0NWlLWirpoKTDkla2c18TRdJRSfsk7ZH0TGrrlfS4pEOSHpN0eaf7WS9JmyQdl7S/pK1q\nPJK+LWlI0l5J/Z3pdf2qxLdG0rCkZ9NracnvVqf4Dkha0ple10fSTEm/lPS8pOckfSG1F2L8KsR3\nV2ovxPg1LCLa8iL7g/ACMJvsJpy9wA3t2t9EvYA/AL1lbeuBe9LySuD+TvdzHPHMB/qB/eeLB7gV\n+Elafi/Zl150PIYG4lsDfKnCum8ju/+gG+hLx686HUON2K4C+tNyD3AIuKEo41cjvkKMX6Ovdlba\n7wGGIuKPEXGK7EsSBtu4v4kizv2EMgg8mJYfBD46oT1qQkQ8BZRf+VMez2BJ+5b0vl3A5ZKunIh+\nNqpKfFB58nwQeDgiTkfEUWCI7DielCLi5YjYm5ZPkH1H60wKMn5V4rsm/Tr349eodibta8ieAjhm\nmDP/4HkWwGOSdku6M7VdGRHHITvQgDd2rHetMaMsnhmpvXxMj5HfMf18OkXwQMnpg9zGly7L7Qee\n5tzjMffjVxLfrtRUqPEbj3Ym7Up/CYtwqcq8iLgJ+CDZgXMzxYirHkUZ0+8C10ZEP/Ay8PXUnsv4\nJPUAjwBfTBVptT4XJb5Cjd94tTNpDwOzSn6eSXYHZa6lyoWIeAV4lOzj1/Gxj5mSrgIq3hmaI9Xi\nGQbeXLJeLsc0Il6JdBIU+AFnPkLnLj5J3WQJ7aGI2JaaCzN+leIr0vg1op1JezcwV9JsSVOB5cD2\nNu6v7SS9Lv3VR9IlwBLgObK4VqTVPg1sq7iByUucXaWUxrOCM/FsJ30hhqT3Af8Y+xg+yZ0VX0pk\nYz4O/C4tbweWS5oqaQ4wF3hmwnrZmB8Cz0fEt0raijR+58RXsPEbvzbP/i4lm/EdAlZ1eta1BfHM\nIbsKZg9Zsl6V2q8Afp5i/Rnw+k73dRwxbSWrRk4CL5J96UVvtXiA75DNyu8je3xBx2NoIL4twP40\nlo+SnQMeW391iu8AsKTT/T9PbO8HRkqOyWfT/7mqx2Oexq9GfIUYv0ZfviPSzCxHfEekmVmOOGmb\nmeWIk7aZWY44aZuZ5YiTtplZjjhpm5nliJO2mVmOOGmbmeXI/wHpXEbejJAQCAAAAABJRU5ErkJg\ngg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.imshow(result)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Transpose" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "x = tf.Variable(input_image,name='x')\n", + "model = tf.global_variables_initializer()\n", + "\n", + "with tf.Session() as session:\n", + " x = tf.transpose(x, perm=[1,0,2])\n", + " session.run(model)\n", + " result=session.run(x)" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ4AAAEACAYAAABCu5jVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcHWWVv5/zvlW300kghJEkkIUlKOjoT3RQFEcNO+KC\nqDCI48aICyiLOiO4AC6jooLKvgQRFBXEBXVEEFERWRUjaJA9JCEkJJCEJL3cqvc9vz/eup0mdpab\nvr2lz8Pnhtt1t3PrVn3rvO857zmiqhiGYTSDG2oDDMMYeZhwGIbRNCYchmE0jQmHYRhNY8JhGEbT\nmHAYhtE0AyYcInKwiPxDRB4QkU8M1OcYhjH4yEDkcYiIAx4A9gMWAXcBR6rqP1r+YYZhDDoD5XG8\nHHhQVR9T1QL4AXDoAH2WYRiDzEAJx1RgQa+/F1bbDMPYAhgo4ZA+tlluu2FsIWQD9L4LgRm9/p5G\nmuvoQURMSAxjiFDVvi7um8xAeRx3AbuKyI4iUgOOBH627pM+feqpFBopi0gIkaJQooYhvZ162qlD\nboPZtOXYNCR2xTqhDNRDSQyRELqJZSTGSBnLlpzgA+JxqGoQkQ8DN5DE6VJVvW/d5zlRfABds5Ku\nv80htLXjyjgQJm0yxcKFdNx+x5DasC5m06YxHG2Cwbcrajfej6fUSFsJMauj//oixo7bGtfnLELz\nDNRQBVX9FbDbhp8DoHT9/R7uP+iNUC/xQzwVsjyUPHzFlUNqw7qYTZvGcLQJBt8ucYFSM0TABYfW\nYNffXQd7vpq+px+bZ8CEY1OY9dpXgwileFy9hKIbHWLh+DdAYxhSG9bFbNo0hqNNMPh2BdKJXbgS\niR5xNWIYT7KgNefXkArHPrMOABfxCoIOuWgAvGyoDegDs2nTGI42weDblQElEYkgBFQjju40TGmN\nw2FrVQzDaB4TDsMwmsaEwzCMpjHhMAyjaUw4DMNoGhMOwzCaxoTDMIymMeEwDKNpTDgMw2gaEw7D\nMJrGhMMwjKYx4TAMo2lMOAzDaBoTDsMwmsaEwzCMpjHhMAyjaUw4DMNoGhMOwzCaxoTDMIymMeEw\nDKNpTDgMw2gaEw7DMJrGhMMwjKYx4TAMo2lMOAzDaBoTDsMwmqZfLSBFZB6wEohAoaovF5GJwFXA\njsA84AhVXdlPOw3DGEb01+OIwCxVfYmqvrzadjJwo6ruBtwEnNLPzzAMY5jRX+GQPt7jUODy6v7l\nwJv7+RmGYQwz+iscClwvIneJyPuqbZNVdQmAqi4GtuvnZxiGMczo1xwHsLeqLhaR7YAbROR+kpgY\nhrEF0y/hqDwKVHWpiPwUeDmwREQmq+oSEZkCPLm+15/+2dNBlGLBAmbGyJ79McYwjD65K0aumj2b\n/LrrQVpzXRfVzXsjERkLOFVdLSLjgBuAzwL7AU+r6hki8glgoqqe3MfrNQYFF+m49TYenrUfWnT3\n46sYhgFp/qHEAREPaN7Grjf/lrF7vRJEceJQVenPZ/TH45gM/EREtHqfK1X1BhH5E3C1iBwNzAcO\n74+BhmEMPzZbOFT1UWCPPrY/DezfH6MMwxjeWOaoYRhNY8JhGEbTmHAYhtE0JhyGYTSNCYdhGE1j\nwmEYRtOYcBiG0TQmHIZhNI0Jh2EYTWPCYRhG05hwGIbRNCYchmE0jQmHYRhNY8JhGEbTmHAYhtE0\nJhyGYTSNCYdhGE1jwtEPdJ37cagMMQYEpfEbC/FZf6+9P1pL+ve3PcKoRAH1gg+OEsFLJDqFCJtZ\n+9kYViiZCHVN/w/qeqRDncNFpRTISR3Jgir9qvw7AjHh2AwEkJiqSNcAQZEANTLqlENsndFfCiBX\nAa8EVTICpToyFGJESa66KkTHqHQ7TDg2C0E1Awq6c4ffeSa1cVtD2UU+1KYZ/WacE1YBEyIEdax5\nchHuyaXUfWpL0laNUYKAqsONwkGqCcdmoTjqKJ5t3no4U07/DPmU6XgtiWLTRiOdEB3quhkTPUGV\n5b/7DYv/+5OMmfcwZYROiYxNDiego9HhMOHYHBQIIox/x9vY8eyvIdtMJkgEHZfcWWNEk6kQZQx1\nwGnGVm96Iy4EHv7g8Ux4ejlRoUsiOVB4wQWb4zB60ZAAqe5LdYvZGCa89U1M+9qZlNtsT46QqcfF\nCM48jpFOrLqd5QpOwPs2am87guexkodP/AJjFi+iQ4Qaiguj80JhwrERVASpxrM1dWQukB9+ONt/\n/nR0u+eQKTgFdaAExIYqIx5XXSJEGn8DCOPf9j52clvx5H9/mmzeQkqKNOkxCrXDjvINIECmaSbM\nqRKcUJecyZ86DtllGhker4BqCteZt7FF43G0H/x68le/Eq8lbTicuFE3TAETjk0g7aIaEIn4oLTX\nJpBrTqaO0gtFJoQYUBmNh9DoQWOkLbbhsnbwkUAkVuHZ0YYJxwZQICAU4tMch0aCRJxGfNXsWwEf\nFa8eRuW1Z/QQfR2lRCUQFLolDWtG469uwrERHAFIAuIUCoTOLCX+qFMk1HGqSJVbaGy5SOlwCHkI\nZBHa1BFkdIZjNyocInKpiCwRkXt6bZsoIjeIyP0icr2ITOj12Nki8qCIzBGRf+pmP5JoRFNqKj3C\nMFaFttLhYgkomc9QDyFjlF57Rg/iaxRZY42KUFaXitH4q2+Kx3EZcNA6204GblTV3YCbgFMAROR1\nwExVfS7wAeDCFto6JCiCUhKIpJyfkiAe8RkiHsHhcHgEP9TGGgOKI0XQRAWt/nM6GmVjE4RDVW8B\nlq+z+VDg8ur+5dXfje1XVK+7A5ggIpNbY6phGMOFzZ3jmKSqSwBUdTEwqdo+FVjQ63mPV9sMw9iC\naHUCWF9+23rnjk7/7OkgSrFgATNjZM8WG2MYBtwVI1fNnk1+3fUpYa0FbK5wLBGRyaq6RESmAE9W\n2xcC03s9bxqwaH1vcvppp4OLdNx6Gw9/53to2ExrDMNYLy9zjre/732M3euVIMrnPvu5fr/npg5V\nGss0GvwMeE91/z3Atb22vwtARF4BrGgMaUYq+qyvniIrAqgqURXVVJdBAeJoDMxtiSgxxp7CTKrp\np41EBE3LEGRUZpr3sFGPQ0S+B8wC/kVE5gOnAV8GfigiRwPzgcMBVPWXInKIiDwErAHeO1CGDwYK\nqCiu1xES8EQnoJoyPETIQiD4lAQ2OufYtyxCDBQ4apoKQooIUUEkVQETzSidkAelFEGsAtg/o6pH\nreeh/dfz/A/3y6JhhABelUbpBXUOiZ4s1NGo4B2uHtA84EMkZpmFZLcAXPRIBqUGctLiRYdD1YN6\novPUAsSeTJ/Rh2WObhShJK2Olah4AovPv5RyxVMQFZcrgkOlDbHU0S0DEWoxkqVLB6K1lCosgdV/\nup3OO26jFBDJUl7HUNs7BJhwbIDkqGZpniMVGkVdYNXFF7Po819AV60iqCeooxSIzpRjSyB6QDQt\nq49pWBp9oH7Tb3j8g8fTPfdvODIKTdnDoxETjo2glDgUURBxdEQY37ma1eecz9IvfBFcmYq+VM8x\nRj4uAAoalBAhSMGaP9zEQ0cejd4/F7xS14I2N3pPICvkswEE8KL4xqy6KrU8Z2VR4FRZduZX6NCS\nHT/+MbKJE3HqU9Xr4Y6mSd80QnfQsxInInmNngo21cRf437/Pzf5cFEEpy5NPscSigJk+MwOxRgJ\nLq16Flew6vZbePwjnyI+/QRRhRiFrVBWq47aE2i0fu9NR6GkMY5VfFGkIYwobQrLL7iY7E9/QiZO\nJKqgI8HtkLRojwgOxYvQLUL76w9huyPfTj6uPX3HGMhwrRvEq1KkrgNElCLUWfaNc+i65Y8MJ5c/\ni5G69+TRU/pI1z/mEu5/MM1pkaJsXUA2fEwedEw4NgMBUEe3C4zp7mTlzbegTggSyEdAEpsCBRlb\nOVhDCR7G7vYSZhxwALX2dlQVRHDOpTyGqDjff/VQSb1JCg340rPsjK+x7PRP4yMMp93W6WAMMCam\nHisRyPDWMacXJhybSSZKVFAJIIKPkYzhdQKsD0FoJ9CtSp5lbPWaA5hy2YW4adPSMIaGOFbPd61x\nOaKCo4Q1nSw+5zye/twXkOh6lqcPF9piTg2lkxLvoKiSv9wo9jDWZSSMyIclJZG6CKV4VCGQUVIb\narM2iShKFCVkwtgDDmDKhd8gmzY9FekdwNhiGsXlLLngXJ468yto6KYYhhmYKgWBEpwjRkdNM3JL\n7nsW5nFsBlr9M0YzAgUpkTTNhJQjQItFoUQY97wXs8NFFzFm6nRKuhBtQx04cZXnoagq0qJaqkEK\nHvzqVyk/eTpZDKzySrsqQaVKphoeqI+UAYiRzEE3kbqD9nJ05mz0hQnHZtDIFywISSaqS2ZAhl9D\nJmlU8K9K/hPB12jf/wBmXHQuboepRA3ksYb6bqAtzXFUz96cYUok1WdNpOgJHc/wzDnn0P3Z/yWP\nRZqMDem9M4ZXPzQpfVWmJxJiss/H0Zsl2hcmHJtJOg3TybH2FBleFa9VGuFWh6PEI3SLY+uDDmTa\nN85AZ0zHRSG6gOIIjEkHRD8vq6kaPOACilKKZ+kll/DUV7+B6+okAE7XSsVw2meQUszXRYadlUOL\nCccWjFMIOMYJdAiUThm3+4uYccE56PQZ+FIIWVybotEiP9yrI1DixSFl5Jnzv8GSj59CFiwusaUw\n/AfkxmYTERwlHVKCz9h639cz/bqf4qZOpxaBLJJpTHkV3qXmUq1AFO8dYU2dJ887myc/+WlcKO2a\nvQVhHscWjkconDJhn4OYet7ZZFNnkLI3q2ITIsSo4HuFYftJBFwUFv/g2zz9xTMInV148QQNdqXa\nQjDh2IIRlG7xjNv9RUy+9DyyqTOIdJDF9mpiQYjeId6T1ue1po2lKwqWfOcyVh7zYVLLIk/dB8YE\nIZjfsUVgF4AtBJVGtEfoKSvj29j64EPY+f9+Sm376SiRLLajUid4T/C+6gVTRQyaPBpi9Z+GSNVi\nl7JrDU9eeilPnPCp1CKRFMP0QYbZ1LHRH0w4tgCqttgEcoS08KrEMfaA/Zn+jS+j06eT0pfS8CTK\nGDKBTMBJdRC4FH1p7nOFGAECqhGN8NQ1P2TxZz6LW/1Uz9BHNOCGVcDV6C82VNkCaKydGStKh0Dh\nYNzzX8yMi86D6dPxpSNmrU/rdjEt6tMQKRBW/PT7LHnn+3EUwyyJ3Gg15nFsAaQre8maKnqyzf5V\n9GTadPIokEW8Kp4qetKqD5ZAoVBEZeUV3+Kpo4+j9CYaowHzOLYIBEeqsbH1aw9gh3O+SbbDdCAS\nXaiUpfXRE1BylNU//jFPfvIzsHolDpcWhLXk/Y3hignHFoCg1CVj3PNfxOTLLlgnepJOYfUOvMe3\nMHoSNaPjlt/wwDuOxks3CviqKjhqMxpbMnZhGGFo4yZClBRBEV9jwusOYedfbjh6IpsZPVkvLkL7\n1mz3xjfA+Am0VSmoa9e6NLwb12O3sWVgwjGCSCefJ+BBFa9KBMbssy/TvvFldFojeqL/FD2RfkRP\n1odTYfy/vZTJl57L9l//GrLVc8iAughSlQJUqcKwmTBKG7tvkZhwjCDSjxUZh0cEyswx9oV7sNOl\nFyC77kZWCtEN3rU9IBQltG07iW3fdTQ73P17sv0OZuu2GjVNa4WjQB0lL7V1Ke3GkGPCMYKIgBOl\nS+qQ5Wx7wBuZcf21yLQZ5FHQXtGT4N2A/7heIcsBSrzAVrvszswffJetP3cqxS4zKSUgCOMloxBP\nMI9ji8GEYwShgKggGYzfexZTv3kmbdtPwylEX+IkgCgxpp4gLaq/s357JNkDGeJSWUC37TZMOv4k\npv78R4x982FkUQhSJm9Dh08lc6N/mHCMIASh9Bnj/vXl7PDdi/Ezd6HOGpwovnS40qF48Dk+gsSB\nroBavb9KqlehQj0EXF5j4vP+Hzt959tM/typ1LadRLdTnGV4bDGYcAxDqlbHVccTqh7pDslrbPOm\nQ9np5z+ktv0MlMCYOBaVgthn9GRgXQ6HS5mjLgIOFU8tqyHiiC5Qax/HdqecwuSrr2abAw9B2sdX\n3VtSgzTE98zG+Oq7Vr2QrETfMGejwiEil4rIEhG5p9e200RkoYjcXd0O7vXYKSLyoIjcJyIHDpTh\nWzpCRr26RucoJcrYgw5i6tf+lzB1WhIHAYiUMgY/gNGT9RspOPE48YgITgTXKFUojlIUkYz2Wa9h\nx8vOZcppn0HGb0eNNMTxqmhq70IdCPie0G1dsDmRYcymHFmXAQf1sf0sVX1pdfsVgIg8HzgCeD7w\nOuB8aVWl21FE+lFK2nwtRSXaPNu8Ym9mXnQesvOu+Hp1pUerE3CIDV6HxkpdUNRBHkEmz2Db//kY\nu9z4U9pf8e+4Wk4QqKlLWa2kBfg1PDU8Y3V0Nzwa7mxUOFT1FmB5Hw/1JQiHAj9Q1VJV5wEPAi/v\nl4WjEK3+7dYuqLUz4fVvY+pPvw9TJpNFQfOYOrApRCfDrt+HqOCiI1OHaom4qidJcIx72d5M/dEV\nTDzlE8h2O1CXiBelTdLkaiGBTpIHMsy+ltGL/viyx4nIHBGZLSITqm1TgQW9nvN4tc1oAkXwCN7B\n2H32Y8aZX2PMc6YRxaGuxLkUPUm1eGT4zVQJlA5Kl76HaAlS4J2iouRTdmKH//4kO//6GsbN2geN\nWaqP6qDuIm1ALXV5MYYpm3vInQ/MVNU9gMXAmdX2vn5ru3BsBvUsZ6uXvYYdLz8PP2MquAKvILGK\nnqgHl6VpUB1e/eM0lmQak3eER8UhmqES8ZIW4+nYNsbtvhc7XXMlYz92IsV2O5JFIdM051EyMrri\njVY2a5Gbqi7t9eclwM+r+wuB6b0emwYsWt/7nP7Z00GUYsECZsbInptjzAgm5XgKOVqdLCki4rKc\niW9+M9O//hWybadWeRCpwWT0OQJVqT+A2HTChgLEOoiD6EEiSE+yek9quCioRnApgrLJ+LVrU5xK\n+oau8dkRUcWLo8whn7g9M//3NJbv+2qWfeMc9Pd/QIs6Qav4kCiFejwRXxUDiljUpRnuipGrZs8m\nv+76Rju9fiO6CasYRWQn4Oeq+qLq7ymquri6fxLwMlU9SkReAFwJ7EUaovwaeK728SEiojEouEjH\nrbfx8Kz90KK7JV9qpJB2Sk4gVvW7Uhf0CYe8gRnnngU7zaRW9T1xEeo+o60Vnxs1fWasGi5JCURC\nrOFFqxhwNQSq2tZJC1bTrtceInRDfdnjLL7wPFZ/8SyUgk6fkRclGUIQyFQp6GWW0ScOqo6CEQ9o\n3sauN/+WsXu9EkRx4lDt38qhTQnHfg+4FXieiMwXkfcCXxGRe0RkDvBa4CQAVZ0LXA3MBX4JHNuX\naBiJRvSk5jJUhHpbxjavfi0zL7kAt9NMfOGIUnU5U2ldlEHS5GXdCxoKylKSbDlFJX1m6SMBpTuW\nPUvzB4oQHJoL7DCNGZ/7AtN+dS3xRS8hl4ycVOA4qKJ5DuIsTDsM2CSPY0A+2DyOnqtmcEA2hm3e\neCjbn30GY6bsAJpROiXX5JyreEQd0oJzWDVSomRFnRU/+iH1hUvZ+p1vp23SpPS5QvI8NICk1Cw/\nkCerRgqUTBVRT90JPPIgyy65mEXnXkKtYyV1By64lNOmap3VNsCw8DiMgaUnevKafZhx5pm0T5pR\nRU8KvPSKntDC6ImmTNSV1/2Sxz/4URac+inm7f8mVtzzR3Dd5JrmJnwUXBj4gyRFkVLz6eDAC+S7\nzGTyZ05nl5uupfZve5FHR81Hco2NbrTGEGLCMaQI9bzG1nvvy05XXoifvgP4RvTE40qXFoa1OHqi\nRZ01P7qG+W/+D+SZp2jr6qbjH3/m8Te9k2XfuoTulUvQGFAiqgN/bZeYskajA8oIMRI0QvsYJuz5\nGnb82TVs9Z9HE7faljW5s2jLMMCEYxBorD1Zux5DiHgky5l46GFM/+6ljNl2KlFK0Awkot6jmQfp\ntfZkM6InSiQoRE19ULTs4Kmrr2bBsScQJNBd2SYBdMECnvzQySz90AmsmnM7QVLQBZSoiqoSNVb9\nUmjdkjWfWjrkeHwG3gWcZFXqutI2eQdmXPh1pl16MRP/7VWopDof0AjZup4VOiX0PJYs79VnxmgZ\nJhyDRkY3ngC0oQQC7fvtx7SvfZ44bQZIOvjRSF1yPOnHkSp9W8Q1FxIF0AgBXAxIKBCNPH3tjSw6\n/kR06RO4VLuYVFcsCUhXuYalP72aJ972blZ+/0q6pBuJUC8KugEpFVcIISZPoTU4RCTponggr/q9\nNKqIKa42nvGHHcbOV13Edh88BtFxAHgPSKxETBhLDuopSPut2qtGizHhGATScrO10ZOutoyJ++zL\nrt+6GJm+C75w6EBET3DJ/Xcloaiz7MprWPSOI2DF030+u5GEGktH92MPs+Bd72P5sSfR/fD91Lwj\n0xJ1Hs0EkZIgA9t9vvDVya+KiOKjg+m7MfXcc9jhh98m3+0FOF9DVFCnFB66RAk4MpclMZRoraAG\nABOOQSCSRKFON5q3MfENb2X771wKUybhVYh5TIMXheildXOgjVsJT1/9YxZ94qPQXaz/CqyKKtSi\nx5ERXWTp7It57Kj3svyHV1OWJeqobC0HvES+i1WhIE1L96NLhY8Fx3aHHcaOP7mKce9/P0U2Bh8h\nj4poSU5AYkkha4eHRmsx4RgkPI7MQfver2b618+iffKORPFEV5D1ip6sXS7fChSngWf+eCdLjv8I\nYdGSnuzQvp8ttEtOSZ0I+OiBQMec21lyzDE8cdxH0KVpKVIpbdUAZwCputLVvYBEMilxGlIhZgdt\nu+/Ojl/5ClNv+Cltu+5GbMzJ+NTXxSkUG/i+xuZjwjFIFHnO1q85gJ2vugQ/dXvwJV4VF31K5BiA\n6IkUXTx97S9YsN++hGdWVSeibmCyUCmkBAdBSlQjvgQfoKOjg87Zl/HwYf/J8pt/i+vuGvAT0qlS\ni9AWPahUnpsjSlqE76NDxrQx+bX7Mf36nzDuiHfgxk/ERaHLQRsZbVZafUAw4WghCsTKPXbV/YDg\nfI1tDn0LU791IWMmTquiJz55Gd6nGb7+Rk80ElNP6VRFK3Sy4sc/54kPH4/GEiGSacRF1psHIaR0\n9CxCrmlI0vi3FoWCSNdtf+DJo97DolNPo1z0GNoo2RWAGFBN6evVWd7z/5KU5t4UjVpEIjiyavgk\niGR40arOqaDiaN95d2ZecBY7XPhN/B4vIUPocgUFqSBQKm7oq/vp25onsvmYcLQcTzdZOtmqrNx8\n1iymn/l5dMZO4NIYHY10tzB6olU2ZZ2CEOqsvvlPPPShY4mLFvacII06Hxs6YRpDmbWnuBJInooD\nSqcUixez7Otnc9+7j2HNg3PSyemgS6ozPWoKKROqlXLgS3BN5or3lEGUZFgqi9yoiOjSIr2q8piI\n4Cb8C9u97R3M+PHVtB/+9qosvCKuRkRxEqlpEptCbMq0P5hwtJA0ri6pSTqou2s52x54IM+7fDZM\n27lX9CSCCnkLoydIinRk3QWd117HwwfuT9uKp1ry7pk4PBkFQh6EMpa0lXVqv7mR+/c+kKUXnU9Y\n9iRZKChwxMpLCaJEAoWUBC/oANdArYujcDB+xi4877JLmX7p5cjM3cgIVXhXU96Kd9S0dZPQoxHb\ndy2kcZUupE7wOf/yhrcw+ZILkO0nD2j0BKqhQFRWXvsL5p1wLFnQliVoBSISS5xz1AWcCJ1O6RLI\nli/lqY+dwsKPHE/XPXeTV0UAUYfEDMGTRx2UNHEflVxj6udSqzH+3UfxvO9/n4nveRddUqv60ggx\n0NoEtlGICUdLEcR5cgdjX/Eqpp79DcZM3YlSsgGNnkRANLLm5jtZcMz7qC9aTF03EHZtEk9VuCUG\nMoGoqWxhTVMGbFFfxeofXs38WW/i8dmzCXF1KlRcJYkpjUVpAyseToW694CisU7uHLWXvJhJ557H\nbld/D9luGporGYFc1A7+fmD7rqUoZV5j/L6vY6cfX4ZsP5noSjLigEZPXL2TVT/5BfMPOgBWrSGP\nVUZoq+pEq4DLUNJVPaOqTC6kTrZlytEsVi3jqRM/waIPfRQemENdAkEFdb6abhjgCIeLZCV4POJq\naFScRrJaG+2HHcaOv76WrQ58A8X48ZRqGR79wYRjM+h93VQh1YjA47IaE956OFMuPpe2bacRJRDF\nIbQyepImG2N107KTFb+6gUUnnUQsulEijphCri0qmaBAvWruFCQFULyC17WB3UJjsq9jBU/PvozH\njvoAa668Eu1eQ1RBUSKRoErUAsq0hiaoptT4FiAoMUsiRqXR6rPUsIrI+Be+mJ0vm83UM75K7QUv\n7KnGLqSFcz2NsgGV9CsF1/i9bU1ub0w4NoNUIEuoA04dXlPdg/zAA9n5zM+RTd8ppY4jZEFJy7da\nET1JolFHUQIFJWv+PJd5Rx9DXDi/IRfVUKh1h3lahJ9c+0aBMOgdfYmplkZPhkjJirl/YcH7P8j8\n//k4Rb0TUQcakqhpRnBCjGV6j1Z5IpKRU+1jcXh8sjk6xGkqWbjtJLb+wHvZ+ZpLaH/1gURy1MWq\nRkWqPxIQggg1HCFCyFyqmWL0YLtjc1FlnGtDJFJvb2OrNx/KrpddQDlpOj6mNSKx8ipaVOYRqhJ6\neSiReknHL37FvP1nkT+9lLiOdzHwMwrrpwaM7yqhu5MVF1zMY/u9jhU3XkfZ1YVWC9oc4B04DQPf\n49YJpTrURSRGvGvD7b4nu1z3AyadeQZtO+6KuirvRiM10pCsC6WGMLaM1GK0gU0vTDg2AyXlEnRr\nN0WeMeHNRzD5vLPw2+1AroJ6RTX1EinFtXz6PjphxfW/YtHxHyauXtOT1DRcDuxu5+gEGmlX4dY/\nsOC97+PJ/z0Dli+r0sqStIVBULggkKcy0ASf2i7UFFxtAtt/5FimfvcS/uWwI8jJyVC6fVpUB4qK\n0I1QiCWM9caEYzMQBKcZeQ61V+3PtHPPYuykGYAQXEBIPUQiVUi0hUs6lEDH3X9j0TvfSdfCRQhC\n7OOIHlIh0UiGQJajopQIbYueZPmXv8Rf/30/OubehQboIkvF7QbY5fCxqIZw6YfwKEgd75UyqzHm\nFa9m++84bNuFAAAVUUlEQVR9m23OPxO2msC4AJl4StdI4lN8+kZGhQnHZqAoZVuN8a97G7v8aDb5\nNs8heBBKnApOPVEzouRpwk5bs/xciy5W/eyXzN9vFvrMarKypKwWdCW71pm4bcmnNk+mQkTxZSp0\n7IAulxamufvu5bH9D+OJC88hX7qsyvsY2IwKjZ4Q02SnR1GJKDlQdcQTcL7Gc475ADv+8ie4gw4i\n1MZSU08hSg0otDSPoxcmHBuhZwdJqiwVHWR+HFu/9W1MOvsM2rfenkgg1fVyOPEgPk2WKuSAa7LC\nsKJojGndSXXT0MnK63/N4hNPJD6zklRlQpFeEV151nsMHYFqrQ7VcA1BNFKGiHcZYcnjLP3kqTx8\n9HvouPNWgoYUACrTmptuCkpSxTFCIC3CicQYN2/uJnPreH2u59b41yGMcRlbv2oWu1x+MZO/eDpu\nxo6UZNTFUUPBebR6q8Kl6e5UhX70YcKxAZTUytBLRtRUkSpGodz3Nezy9S+QV9ETL4KPlSvcGCNU\nx6ZU6efNfnBj3QlApM6qOffx6FHvpnxsXk/Nzb6GI7KBxwYL12s9jFThYamizzGW5FIjrH6G7v/7\nJfcffgyrvvtD6F5B3Uc0Qi04fIQ6geBS1/sUMZJUwrDJU9WRikKLCEgSCanC6FKtd5Eq5CUiZNtO\nZbsTT2THn11N/oJ/xWukcCAxkMX0unoMjCGS6fCZWxpMTDg2QDr/HYWW6YKlgTbveMFXP0XxnKm4\nkMJ0kZji/i2MnpQItRIouuj41a95/KCD8auXD6voSbMkLwRiTCqSAbUnHuDh/3ov80/8BF1//xta\nhiqeGsnJqmI+nlIcElJIdcAnU7O0jH/cC1/Mi269gW0/8xnapu6Mc47CQ4lnK8noBuriR8z+byUm\nHBtAScvMGyenE6EMwuqtJqbyfj4lL7kIocXRkxQBEJ753W9ZeNJJdC1f2ithaWSSVTVOAyWZKiop\nJyUrC1ZcdAlL3vFenvrB94ES1eRdSFpoi6ik5C4NA/79HQEJJUEccfwkpnz6k8z41mzGzdqfGGt4\njQTvKADfusKrIwoTjg3Qu9htLp5Ck8vcXrYRXFwbPXEpeqK+VZmaSpA6xd/+wRNH/idrHn4Eon9W\nIujaocDIEZJCHF1AowpoRPAxR73Do6ycO4d5//VuHvnAh2DFM5QSWZXViT721GENA7zCFlJSX3eW\nIVHwIVLmjny/17Dz9Vcx5fOforPWTizrtOOrxLfRhwnHRlCXcjZUJWV7upSj4TWFZKNmRM3TuL5V\nXePLOh2/+g2PHXgQ5fKnqZWxZzqvTxtb86kDjyi5pP0ZRAiqOB+RoJQi5KXSJo41s7/N3DccQsfP\nr2d8Z51QTZQ2ho4D/Y1VIzUiWYwUTsmo0RY9pZvAc048nu2OeCNZ9R2KUZqKbsKxERprQ0oCooqL\nqWhuip6kAv65NArMbEb0RLWnSpY2oid/+COLTziRziVPUNKYbAwj/sdyMa2qRUnp3wAhdZmRShik\nTK5/edttLPjQB5l/8mdwSxZRuli9rvKxNBK1WFv5rJWzPb5aveyFrNFs21Gls9fIfVuqN9KiMPtI\nZKQfiwNPdSzGaml47/BsT/RENi96IioQA3Wpp2GIFnTOfZSHDj+KjoceRFF8lbG47nVW1nN/uJOy\nRtd+n2fnnTRO/5TeXS5azFPnns0DB76BMOcvKYXfpSbUEUFiBgiiAQ2xz0S4zUHw6fd0KT3eC1VE\nJg1bI9oobJZaN7TmY0cUGz3SRWSaiNwkInNF5F4ROb7aPlFEbhCR+0XkehGZ0Os1Z4vIgyIyR0T2\nGMgvMNJRcWTBEcs6q667kXkHHEC2fAmo4mTtCteRFD1pFTkFPir1uffyyJveylNfP4viiUVITBGs\nKJGgBdEpKqPzBB4qNuUSWQIfVdUXAK8EjhOR3YGTgRtVdTfgJuAUABF5HTBTVZ8LfAC4cEAs32Jw\nlE5YfcttLDjhROpLFwHge4kGVFe8UYZWtVtFlfD4fJb9z6dZ8P5jWTXndqBM3oBmaWWrJO/DGBw2\nKhyqulhV51T3VwP3AdOAQ4HLq6ddXv1N9f8rquffAUwQkckttnuLIEVPSuI/HmLRkf9B57xHyGKe\n8kEaY/5qqBJaVFtjJKEoOTWyCN0CXRl0/OJa5r/2dTz14+9TFqnRo2qJYCnhg0lTg3IR2QnYA7gd\nmKyqSyCJCzCpetpUYEGvlz1ebTPWJQQ6bvoD817/erqWLaEWAmV1JW1MoegoFIwGqX9KnSA+1dao\n1yk8FGtW8uQR7+fJDx7HM/f+mRDT+qDRGd8YGjZZOERkPHANcELleazvV+rLXxzVv6iqEmJIha7K\nVFRYQxer7ryTJ074MGvmP9p4ZhX6XTunkYryjM5ZbK+pNQMa8DFFlfKQIliFdrP0u99h/pHvYsV3\nL4VyNdqrMlqsglVltRMbAqyqz7oZm8cmtf8UkYwkGt9R1WurzUtEZLKqLhGRKcCT1faFwPReL58G\nLOrrfU//7OkgSrFgATNjZM/N+gojAUFCpPQl6mu4WNJ1/6M8ePiR6OKFSFSyVGAQNCWT+Z5XGg0a\na1Qc0JE58qKgeOg+njj6ozx1671s/7lPM37SpJS0FxydXhkTtactQ0MoRGRUicZdMXLV7Nnk111P\nq9ZFyKbsQBG5Alimqh/tte0M4GlVPUNETga2UdWTReQQ4DhVfb2IvAL4hqq+oo/31BgUXKTj1tt4\neNZ+aNHdki/VSrQqRlMV/ccDOz3wAOOf+9xNfw9NXcx8hBgjz9z8e5Z84IN0PvwwVO/ZKPkHa1eX\nGn0TSO0dCwI5Veq65ox71SuY+omP0fba/fHjx4HUCdGTVWHVvo71piadFYrODp449liWX355dUz0\n/uWGBw4oq5VBHtC8jV1v/i1j93oliFZlEvtXr3FTwrGvAt4B7CsifxGRu0XkYOAM4AARuR/YD/gy\ngKr+EnhURB4CLgKO7Y+BWwqiqYBu551/YtHxJ9H1yCMg6/4Ajf6oxobwQOkiglAKdDqoSUHXH//A\no+/9AI+feir15Y8jRUZW5daMJg9jMNjoUEVV/8j6a1jtv57XfLg/Rm1pCOC0oPPv9zP/LW8hPvkU\nntRrpeE5prUb6Y8B7gE/4smAMkYKST5gjtClJe0I4aklPP3Nr9N91x1Mm/112ma+CJe32ZCvxZhH\nPAjEGFjx+1uY/5a3Ui5bTCkldQJE11P/qrHG0g7wjVMnCW1Nq9J+oSQT6BbFSY44R/3WP/LIq17P\nkrPPonvpEvM4WowJx0ZopJKrpF4bpQheHRojhIBqTPc1UCdQqkIsqWus8rfqdPx1LktPOp41Dz9E\no4GB71kjajRLY8FfrP4T0pyfUyi0QGJIHtzypSw9/Ys8+p9HUf/rnQSNEKEuZeqLrZLCL1W6eiS1\nu1zf71Ly7BYXAFnrirCMKEw4NkBKvEpl64KCdz4dbD6iokRSDw4VQQPksZoqc5DXBcqSNQ88wGNv\nPZSVf5+buqxHxWmq7UGv+hojaXn8cKARpl53zU61dGht2YE1a4i//j337XMIS799GeWaNdTqGR6l\nWwo0ptKGQQuC1nFBqoV0/4wnlY8sXZqLktxTH4Rl/sMRE44NIICrOhA5B2VIVanK+x6g7OwCV+Ji\niriUPi1/klBSqqd0gZW338aCw46iPn8eWUxXxMYBX1oewaChRFi5nMUnfJxFHz6eNffNodCIV0/d\np9wZFxSvHpX1lybUqLgAPgpeHLGItKqSwkjDhGMjOE1ehgBIpE7B3z92Co+/5/2s+O1vKWMXSlll\nLQpUZfTLe+ew5Ljj6f7HXKANVwmFqyInMDrXnwwVNTJYvZLl37ucJ/7jSFZcdgWuXlADIgHnc1KR\n2A0UEa3cGUWJGmkTRz5KT6HR+a03kZR0WDnEsXKBM2WbB+5lxQ+/x8L9DuTRvfel4/s/JixZRiwL\nggus+sdcHvqPd9D19zk4DZQx5acIpLkSIHPOPI5BoiPP6SaSuRo+KN33P8jCYz7C/I99nGLxYyma\nlYqhphmT9fwsAnT7QJGlE6dbA3WJo3KmapMyR0czUSKZxrSj1NGlSntVJ6MU0D/fwWPvfhf5C/8f\nWx20D2NmTmfxud+Chx7qEZ52JFXqVl27XB5T7cGiVhRkUiPGgiiKKjjpZOX5F9Jx95/Y7qQTmfjG\nQ9G2MWQa1+txBAlk85bCwqV0Z5CXjRqqow8Tjo0g2ruXRyQP0A2gqZpVBKSsU5/zJ566527cmDGE\nrq6eAUmq3ZWe3yMUmkTEBiqDgwOi1oFeI5HK24u338HiYz/MqttvYeqnPwkTpyAKhdRTeciYgRPq\nq1fxzHcu4PGLriJ78AEI6X2Das8gdTRhwrEBGgdDb1d0XS/hWX/HSOzoeFZvk8Y7jLYDa7jS+3dQ\nSrq8kC1/mo6vn8+839zFlAu/zFZ77oXXMSkwu+oZnr7tdyw77Qus+Nt95J0dqb+LT56GqIc4+nwO\nEw5j1CIIYwIUzhFcoLznTuYf+V88531HM/Htb2Hln++l8/vfY8WvfonUAxNUCQhBIu1aNZgb4PaV\nwxUTDmPUEvF01QK1ssRX9TzqCx5hwZe+xDNXXEr30mcIK1eDFKn/LRCcMgbIYiODdXR6kyYcxqhF\nvNJe19Stjwh5hpSRcV1rWPXQo/iqTaRqDhSptQPJ06gDZc3jitDCDn4jB5vYN0YtLqQCBpEqwa8s\ncCrU8YyjkZkaySSgCEFThz0QooAUI79lxeZiHocxamm0YgAgpr4pqZhSoN54QvWchlOhMT1fdHRf\ndU04jFFNX/MTro/HRuM8xoYYzaJpGMZmYsJhGEbTmHAYhtE0JhyGYTSNCYdhGE1jwmEYRtOYcBiG\n0TQmHIZhNI0Jh2EYTWPCYRhG05hwGIbRNCYchmE0jQmHYRhNsynd6qeJyE0iMldE7hWRj1TbTxOR\nhVX3+kYH+8ZrThGRB0XkPhE5cCC/gGEYg8+mLKsvgY+q6hwRGQ/8WUR+XT12lqqe1fvJIvJ84Ajg\n+cA04EYRea5aExHD2GLYqMehqotVdU51fzVwHzC1erivMgWHAj9Q1VJV5wEPAi9vjbmGYQwHmprj\nEJGdgD2AO6pNx4nIHBGZLSITqm1TgQW9XvY4a4XGMIwtgE0WjmqYcg1wQuV5nA/MVNU9gMXAmY2n\n9vFyG6YYxhbEJpUOFJGMJBrfUdVrAVR1aa+nXAL8vLq/EJje67FpwKK+3vf0z54OohQLFjAzRvZs\n0njDMDbOXTFy1ezZ5NddT6tKssumzFmKyBXAMlX9aK9tU1R1cXX/JOBlqnqUiLwAuBLYizRE+TXw\nT5OjIqIxKLhIx6238fCs/dCiuyVfyjBGMw4oUyMHPKB5G7ve/FvG7vVKEMWJQ1X7VUZ1ox6HiLwK\neAdwr4j8hTTs+CRwlIjsQWqfOg/4AICqzhWRq4G5QAEcaxEVw9iy2KhwqOofSU251+VXG3jNl4Av\n9cMuwzCGMZY5ahhG05hwGIbRNCYchmE0jQmHYRhNY8JhGEbTmHAYhtE0JhyGYTSNCYdhGE1jwmEY\nRtOYcBiG0TQmHIZhNI0Jh2EYTWPCYRhG05hwGIbRNCYchmE0jQmHYRhNY8JhGEbTmHAYhtE0JhyG\nYTSNCYdhGE1jwmEYRtOYcBiG0TQmHIZhNI0Jh2EYTbNJvWMHCkVRLcGVOAcBUzLD6C+lgBcI0aUT\nShxIXz3VNp8hFg6ImqGxjagO8JTW2N4w+oVToUBoI1KPkcwpQmjpZwypcAigIgQvlBppI1AfSoMM\nY4sgI9cAKGOAIiixxRfkIReOWoDOesDlNcqywGscSpMMY8SjlKgHF6ETcD7Dl62dBBhS4SBG1EXy\ntgjUyfM2ChMOw+gXDkeUSJk5xLvk2eet9eVFdWjmFEREb7zpN7x21iyK1c+gf59Dp45jjBZDYk+D\nm+++m9e89KVDasO6mE2bxnC0CQbfLokCLhIFfAlaK8mf/2KyrbYBUZw4VFX68xlD6nHc/Lvfs++s\nfcjGTkD3mkW7ksYvQ8jt19/Awa/ce2iNWAezadMYjjbBMLKrhU7C0EZVREEi6hwqigyxaAAgmm7D\nCbNp0xiONsGwsivSmqmAoZ3jCIGwphOvjuiAqEOfx1Ev0NUdQ23FszGbNo3haBMMG7si4GutOeWH\ndI5jSD7YMIx+z3EMmXAYhjFyGfKRgWEYIw8TDsMwmmZIhENEDhaRf4jIAyLyiaGwobJjnoj8VUT+\nIiJ3VtsmisgNInK/iFwvIhMGwY5LRWSJiNzTa9t67RCRs0XkQRGZIyJ7DKJNp4nIQhG5u7od3Oux\nUyqb7hORAwfIpmkicpOIzBWRe0Xk+Gr7kO2rPmz6SLV9SPfVgKOqg3ojidVDwI5ADswBdh9sOypb\nHgEmrrPtDOB/qvufAL48CHb8O7AHcM/G7ABeB/xfdX8v4PZBtOk04KN9PPf5wF9IUbqdqt9XBsCm\nKcAe1f3xwP3A7kO5rzZg05Duq4G+DYXH8XLgQVV9TFUL4AfAoUNgB6R0s3X3waHA5dX9y4E3D7QR\nqnoLsHwjdhzaa/sV1evuACaIyORBsgn6TtE7FPiBqpaqOg94kPQ7t9qmxao6p7q/GrgPmMYQ7qv1\n2DS1enjI9tVAMxTCMRVY0Ovvhazd0YONAteLyF0i8r5q22RVXQLpoAC2GyLbJq1jx6Rq+7r773EG\nd/8dV7n9s3sNCQbdJhHZieQR3c4//2ZDsq962XRHtWlY7KuBYCiEoy8VHqqY8N6quidwCOlHfvUQ\n2rKpDOX+Ox+Yqap7AIuBM4fCJhEZD1wDnFBd5df3WYNmVx82DYt9NVAMhXAsBGb0+nsasGgI7Ghc\nnVDVpcBPSS7jkoY7KyJTgCeHwrYN2LEQmN7reYO2/1R1qVYDdeAS1rrYg2aTiGSkE/Q7qnpttXlI\n91VfNg2HfTWQDIVw3AXsKiI7ikgNOBL42WAbISJjq6sEIjIOOBC4t7LlPdXT3g1c2+cbDIBJPPtq\n1NuO9/Sy42fAuwBE5BXAioabPtA2VSdlg7cAf+tl05EiUhORnYFdgTsHyKZvAXNV9Zu9tg31vvon\nm4bJvho4hmJGFjiYNPv8IHDyENmwMymi8xeSYJxcbd8WuLGy79fANoNgy/dIV51uYD7wXmDi+uwA\nziXNxv8VeOkg2nQFcE+1335KmltoPP+Uyqb7gAMHyKZXkUrTNn63u6tjab2/2UDvqw3YNKT7aqBv\nlnJuGEbTWOaoYRhNY8JhGEbTmHAYhtE0JhyGYTSNCYdhGE1jwmEYRtOYcBiG0TQmHIZhNM3/B7g9\nfy8nLy7LAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.imshow(result)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## Computing the Gradient\n", + "\n", + "- Gradients are free!" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0.5]\n" + ] + } + ], + "source": [ + "x = tf.placeholder(tf.float32)\n", + "y = tf.log(x) \n", + "var_grad = tf.gradients(y, x)\n", + "with tf.Session() as session:\n", + " var_grad_val = session.run(var_grad, feed_dict={x:2})\n", + " print(var_grad_val)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "# Warming up: Logistic Regression" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "skip" + } + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## MNIST Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "The MNIST dataset (available at http://yann.lecun.com/exdb/mnist/), is widely used for training and testing in the field of machine learning, and we will use it in the examples of this book. It contains black and white images of handwritten digits from 0 to 9.\n", + "\n", + "The data set is divided into two groups: 60,000 to train the model and an additional 10,000 to test it. The original images, in black and white, were normalized to fit into a box of size 28×28 pixels and centered by calculating the center of mass of the pixels." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "![](imgs/mnist.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "```\n", + "%load mnist_data.py\n", + "```\n", + "\n", + "(Adapted from [input_data.py](https://github.com/tensorflow/tensorflow/blob/r0.7/tensorflow/examples/tutorials/mnist/input_data.py))" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "skip" + } + }, + "outputs": [], + "source": [ + "# %load mnist_data.py" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import mnist_data" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting ./data/mnist/train-images-idx3-ubyte.gz\n", + "Extracting ./data/mnist/train-labels-idx1-ubyte.gz\n", + "Extracting ./data/mnist/t10k-images-idx3-ubyte.gz\n", + "Extracting ./data/mnist/t10k-labels-idx1-ubyte.gz\n" + ] + } + ], + "source": [ + "mnist = mnist_data.read_data_sets(\"./data/mnist/\", one_hot=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Dataset Statistics" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "55000\n", + "5000\n", + "10000\n" + ] + } + ], + "source": [ + "print(len(mnist.train.images))\n", + "print(len(mnist.validation.images))\n", + "print(len(mnist.test.images))" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvbuPZNue5/VZr/2KyMg6ec+pvlPT3LkeHsJqA4eWQAgD\naSwQ4ICw8Qes0XiAOx5ohEACgTBGYMGAcUcaC/6AaQaDZrp1uk/dqqzMiNiv9fphrB2ZkVl5Xrfq\n3Ft1cn9Lv1prR0RFrMjK7/791u+1lIiwYsWK5wX9h17AihUrfv9Yib9ixTPESvwVK54hVuKvWPEM\nsRJ/xYpniJX4K1Y8Q3wQ8ZVS/6ZS6s+UUv9MKfV3PtaiVqxY8dNC/a5xfKWUBv4Z8K8BXwP/F/Dv\nisifPXrdmiiwYsUfCCKinnrcfsB7/gnw/4jI/weglPofgL8N/Nn7L/27Z/PfAH/6AR/7U+M3rOv7\nEPyGT3d9v+HTXRt8/PX9vW995kNM/b8J/MXZ9V8uj61YseITx4cQ/ykTYjXrV6z4DPAhpv5fAr86\nu/5jyl7/CfzmbN58wEf+PvDrP/QCvge//kMv4Hvw6z/0Ar4Dv/5DL+B78OsP/Pd/vsj340Ocewb4\nvynOvb8C/k/g3xORf/rodfJwj79ixYrfD/7ex3fuiUhSSv3HwD+ibBn+wWPSr1ix4tPEh5j6iMj/\nCvyLH2ktK1as+D1hzdxbseIZYiX+ihXPECvxV6x4hliJv2LFM8RK/BUrniFW4q9Y8QyxEn/FimeI\nlfgrVjxDrMRfseIZYiX+ihXPECvxV6x4hliJv2LFM8RK/BUrniFW4q9Y8QyxEn/FimeIlfgrVjxD\nrMRfseIZYiX+ihXPECvxV6x4hliJv2LFM8RK/BUrniFW4q9Y8QyxEn/FimeIlfgrVjxDrMRfseIZ\nYiX+ihXPECvxV6x4hvigs/NWPFMoVQR1P7+7FrTKaJ3RSlAql+tFEEEEJAO5jCchw+92dvPZ0r5j\neWp5MiuFKIUoXeaU66w0IoCoB+sj318XyBPjh67894uV+Ct+HJQCbcCYJ0dtMpXxOBuWMVKZgDMe\nZz06Z5IXsoccIHnInrvHJH//Er5vecaAsct4tjxjQYwmaks0RcJprg3RWFLWZK8Qr8hB3c89ZK8W\n8ufvkM8DK/FX/DicmGXdvbj7uXaJqhpo3UhbRdpKaJ2nrUbaakCnSByEOFBkXOZAjEL+QO4YBc4u\ny6rul3aaZ2eZbc3sqmVUzNYwO81sHTEZ0qBJgyrjqEiDAjQSFZIFSGcSlxE+J82/En/Fj8NJ41sH\nVf2e6DriGmjrxEUzc9FkLprAthm5aA6Y5Al7we/BL2MAfBT8BPkDiWM01AbqqiypbqCuoWrKPNeO\noeoYqpahUgyVYawdQ6UZKsscHXGv70TtNYX0GiZNIXagED5QNhcsj2eeBfGVUn8O3FK+cRCRP/kY\ni1rxCeOk8d1C/KaFur0bdRuoNom2m7noNC864YtN4Itu4EV3wIaJ+Z0wNTBbYUaYIsyTMOtCpw+B\nU9AYaB20DTQdtF1ZXttBaisOTebYwKExHBrHsRFMo6BxaF/j32l0Y1BWAwaJmjwZlNYImUIbz/uk\n/3zwoRo/A38qIu8+xmJWfA44M/WrupC+20BbRG881cVMt+25uNB8sc18deH5cjvy5cUB5wfGRhit\nMAJjFMZJGI8wavkoxN9Y6CrYNNB1sNlCty1j6hpuOrjpDG1XUXUJ2wm0mtxZmCt0Y9DWUEhvyJMh\nHRdnARmYeZ/0+uyxTx8fSvyy+VnxfPDY1G/aQvrNBWwu0Bcz7rKn3VVcXBq+uBS+vAz88nLkl5cH\n3HxksEKPMEShn6A/CkMl9FoIH7i8WsHWwIWDbQ0XHWy3sN3BxQ7iRct2a2g3FfW2xWwiagt5qwkb\nR54qlDWAJUe7kN6iKoPSlrKfPyf9aZ//edHgQ4kvwP+mlBLgvxSR/+ojrGnFpwz1SOOfE397id5N\nVF/c0n1RcfGF5osvMl9+EfjlFyN/fHWgmvYcEY5ROEzC8Sgcb4RjJVTq4xD/0sKugl0Dlx3strDb\nweUXEHaBdldRXTTY3QwXkXwhhJ1murDEsQIsEi15sqSjJd44dGVBnYgPD0kfeG4a/18Rkb9WSn0F\n/O9KqX8qIv/k/Zf95mz+60VWfKpQ+lzk4XUlsMmoTYY2oeoILqBMAO3ZqpnNmXR6ZqMnOj3RqYlK\nT2QtpEWyEtKZ+A9cew20i3TqfjxJUJpOTbR6olUTrZ7LXE80esYbh7IJVSV0k9FtxmwEc5Fxl5lA\nJktEciTnRM4JyZmcPzwi8eH480W+Hx9EfBH562X8rVLqHwJ/AjxB/D/9kI9Z8ZNAPTFXKCMYB6YS\nbCWYqsxPol1EuwnlDLrKKBfQakKFI7q/YcvIlforLvJrmniN9bfI2BP6mf6Q8LMwfC1Mr4X5WvB7\nIQ18lBg+AAIpQgwwTzANxTjRiyUeo9D7yDTNhH4kHw+orcPcGOottGHAvrXUvaWNlmAtYWvxX1oC\njnCZ8d4TfCAET/D+/trnPzD5f81DpfqPv/WVvzPxlVIdoEXkqJTaAP8G8Pd+1/db8fuEOhsfzpUW\nbC24VnAduA6qTnCd4LqMtQEtEwZBS8AwoeWICTXa12ziyFX+LRfht7TzW+y4R44Dfj8z3CSsF4bX\nwvhamK8h7CEOJXmH/OHGsgjkBMGDn2C096TPGWLI9FNg6mdCN5A7B53BdlB1iTb31Mdi4qdkSc6Q\nt5aEJTUW38M4JKYhMQ6JccyMQ4npp4+Qh/D7wodo/D8C/uGyv7fAfyci/+jjLGvFTw/1pCgtmCrj\nOkWzE+qdLGOm3mUqIxgv2Dlg/ITxBuvN3dhMM7t4zcX8jma8xvZ7pOvxG0/fJUwQputC+vlaCHca\nXz5OREwgpUXjz49JD2kS+j4yNR7fjuTGoBowTaJuAkrXECwSDUQL1sCFhcYiLwx+1Bz2cFzE7Mv7\np1gsjM9ln/87E19E/l/gX/6Ia1nxe8Vj0hfnlNbFvK82Qr2D7graK6G7ErqrTKUFdwzYI7ijYI9g\nveAC2F6o80zj97TjgbbfYw57pBnwzczQJFQq5r2/hbCXRePfm/ofRePHovEfkD6C95BHYagiUzUT\nak2uQFUJWweqasK4Gm0Nxhq0MRhrS3jPlMemyXDzztI0BruE/FI0zJNBa8PPnvgrPmc8NvU1J+IX\njS+Lxld0V8LmpbB9mdm+zDQq4q4jlYs4IpWPOBVwPlL1ERtm7DRg3YCtRmw1INVAqGbEZSSf0nWX\nsX9o6n8w8r3Gh3tN7z1MI4jLzC4y25ngINkMzmPdRG1rpHG4rbmX5my+MUze0jQV1lZARYwV81TT\nHyu0/nxCeivxny3eJz3ossevMlWnqC+hvYLtS2H3Sti9ynQSqdxExUQdJqp+PptPmHFGrEeMR6wH\nExDr8cYz21S83/5UnCN3RTo/xR4/p6Lp7wp1LIgRoolEDdEksvEoYzHaURmL3hqaXxgaNHVjaKyh\n3mqaLw31LwxTrLC2xA1ibJmmlv4ouEotcX7zgd/g94OV+M8OT+/tTzeAovE1rks0i6lfiJ958avM\nJkcaJurQU/dHmndHatVThyNN38NhJuiMV5mgzkZdxiTyoBT3vCT3o+3x40J6DWrJrFV3XzUjKiIq\nISogSqGUwiqNUQp3qekwdI2me6HpnKHbarovDZs/1gypAbbEuGWaIsNR2N9oqsqh1eeRpw8r8Z8h\nZInNF9H64XW7y7QXiXaTaNtI2wTaKtDaQKsDLSONGqmlp8lH6nSkjgeacKD2B5gDivv0FkXhc6Jk\nt6fvWBn8MI3/XgsAteQYLP84L/X+IsXUF1nmd7zMy+eku887GelOFNWlphk03aTZeM02aTai2SqN\n04GDVmy0odUVtYo4lTEKlCq5/e/X6z+e/+GxEv+ZQRmwrnjubVXMelMpbKUwlaK9yOx+4bnYeDbW\n00RPdfSYNx6FhziRv+5Jr0fi9Yzee9QQUT7fxeFnCslPNWyJj1fColQx2c9r7I0BvYxQzPuUFokQ\nl1FSuQF8J7KgvKAGgb2g3gmqFbTLaAVaMuqvQP1Wod9p1MGgBgu+glxTUohOVXrnFXufVuXeSvxn\nBq3B1kK1xOmrDqpOLSO0m8RmO7PZznRmpg2e6jhjmVHTjPiJ/Hokvx6J1xNq72GI4Mv+HR6SPnJv\nxX+MX3ull4zhqtTYn2rt7VJvD8WRF0LZ5wcP6szR9/3EX77AIKiDoK9PpNfonNFZ0K8F/VuFeqdR\newOjA+8W4ldn3/hk6+SzN/80yL8S/5lBncJ1XYnPNztod2fzJtC6mcZOtHamiRPVYcJMM+pmQqYZ\nuZ5I1xPqeoa9hyEgPpMW4p+T/qTxP9avu1JFu7vqvsb+fBQp8XQ/lfEupJcg/pB9RKZo/FGh9oJy\nGaU1OmW0V2jJ6Leg3i7EPxjUaFGhOiP+eaMOtfwUPg3Cn7AS/5lBaYrXfiM0u8zmSthclbG7EtrK\nU8WJKo7UaaKKE24asWlCxRHGmbz3pFsPe4/sPTJEkk/EhfjnfWlO+u6jafxTO4CqkL3poNksNfdd\n+Yyxh8mVbY0AKUOI9z6A78Si8dUgKFd+Xjpn1KzQg0KLoG9B7xXqVsPBogZ3ZupX3N/yzqv4Tl6P\nTwMr8Z8ZSrhOqLpMu0tsrjIXL4vsXiYa67HHCXucMMcBO43Y44g5Dqh+RPqZPEQYIjJEpA/kIaJ9\nRi8W7bmB+zFJD/d7fLto/HZTau27iyIiYBcfgHAf0vPnJfTfhXNTX4FKoGYppD8Uja96QfWgeo3q\nTckLDg4lFWWPf16pd9rfn8IKn4bmX4n/zKB1KbqpukxzmdlcRXYvE5evEi9eJRo1o95OaEbUNKJj\njzoO6DcD6u2AHDzZJ8RnxCeyTyifUT6hFo0vvO/a+ljd6M4bAJ00flcqgtleFuJrvejXs9RdY4r2\n/t73z6DCYuovjr5CekHXi6k/g54Vetao2aAmi3rg3HtM+vMa/k8DK/GfGe4TdE4aP3HxMvLiVeTq\nV5FGZoQJmUbkZkDCgBx75E2P/GVfzPssyKkMNctD+cnXXzT+nam/uSf+7uq+wi/le03vxvJvfpSp\nnwTlQQ+gjKAtaKPQCDoJKipU0qhoUMlBdGfEh4ekP0+S+jSwEv9niPNaev2opv4+Th/ei9N3OlDn\niSwTKU5kP5GnidxP5MNEvp3J+6dbZTygvFagVdGwy/xOUEhWiCySH44PQ+Bydn1vTVQIlYJKCRUw\nK2FWMCvACMEKyQlSL6E4L5iQqZKAW97qQd/8M4E7b6TKD3sTaC1opDx+JwqyQmUNUnL374n+6RH+\nhJX4ny2+vZ6+xOkFW7HE6U+19UJ3EXnxC8/FJpzF6QPmjQc8Emfk6wFeT3A9wz7AkMCXPHv13ife\njwoQo8AZpNJIZeBuLI9lDClaUijFLXcSLCkaJKn7bJt8lnmzjJUIPmUmLwxTph+EgxNudWaLoHVi\nHhNzTsw2MXclS8/ViW4XqSa5SxHO4Wx+ShuWhegnDpsyvxMWayAJLKNKy0ZGUpEHm5tPY0//GCvx\nP0t8ez291oKt8xKnF6qu7OdP0m0CF9vAxRKnb4LHHWcMHqYZ8TPyeipy7ZF9KE48n+9M+ftavse1\nfYBW5FojrSN3FulOo4XOkXHE2RHmijC7ItNpXpGDgrSk3KXlM1Muj6mMy5kpZkaf6KfMwSY6ndmQ\n6HLG2ogkT04BMZ7cBag9bhewSUhzIg6lFDgOijSWIqEEyCnpYEnAU/YJQVCRRQTUYj7kbztg49O8\nAazE/2zxXfX0QtWxxObLXr7ZZZpdomsCnfNs7ExnZ+o44w4TeprhLk7vkesynmv8UxHN4/Kec0Er\ncmVInUXtavKugl1FXkRUTRwafN8wDzXzcDaahjRriKcSu1yEVGzynLGSGFKi95FmSrQ60pBoc6IJ\nkar2WDNj7YSzE7aesVbhjGBtcUqGPYuUsmBQSBTUxL0D/kR890goCUFKC0oJSpabk15sf/n0SQ8r\n8T9jPF1Pf+e82xTi38fpI91VoqsiTfTU0dOkiSZOuGnCpBHihIxz0fK3AfYB2YclM+9e459Ibs7G\n0xytSJVBbRxpV8FVg1w1sEhWLenQEQ4t06FjPHSMVcdoOkZaojIQEoRcYmks5rMqYiRSx0jlA7WO\nVETqHKhipJ4DTTvTdQNd6+hqQ9spTJtxXaJrAyoo5ncwNyURCIQcIU+lqOf0o1Sawg4HqlqkLgr+\nRPrT9kPljErfp+k/rRvASvzPEt9VT5+xVUnBbXecxekTu5eR1gbcsZj31XHGTRPuOGCOI/Qj0peE\nnBKnT9CXeD1PmPonbpyIbwHRqrSi7kqrW7lqyC871MsOXnZk1RFvtvibLXOzZXBber2lly193BLE\ngo4L6ZcE+xxLsj0JLQEXA9Z7HAGXAy547BxwY6DzIy+U47I2ZKswndBeJtyLQPdCY6LCNKCtUDR9\n6coTjwr08v1Opr5ZtHwFqgFdl5cotZj8uchpG4L6PMx8WIn/GePpevoSpy/Eby6Lxr+P00caFTBv\nAwaPmSZMHNHHEf1mgCVOX2L0JU4vPhfH3iNT/1zT2zMpxNfQOeSyJl816Jcd6dUGXm3J+oK43eHb\nHVO1YzCXHNlxSDsOfofPtmygiWXTnUO5EeiysVbZY5LHeI/JMzp6zOwxdsYYzzb2TLUhXyisEdou\nwRce99KyeamxSaHt4oiMQp4gHEFXC+FPFX/LF1MOOGn75vS0FM1/cvCFXEz990h/Tv5PCyvxPzt8\nXz09i8ZXtDu50/gvXqUlTh9Kld00w80MYUIdR3gzwF/25Ls4PXeedclLP7wnNP6J+G4Z0ap48DuL\n7CrSVYN62aFebZFf7RB9SWxe4N0LJvMFIy84phfs/Rfcji+YowMCSCikT6Fk4ailf73MqDij8oyK\nE0rNRXSZX+a6kD4LrU3sOo+8mLB/ZOn+WFNlUHeavpDe3ZSkJvXorna3x39M/OVnoaKgYgaTF1Pg\nu0j/aZF/Jf7nCK1Kepo2Z2OxTfUuYbYJu1G4FqpGqKtEYwOtnmnyhMhcwnZ+Ls68fkYOM/m2OPMU\nxZwVALX0lDk5vACDRqNZ0llIy20go8mmJpgLvNkS9JZgtnjdEXSH1w3TnbSMqmFULZNqGReZ724h\np9tJOJNlYyEa0tmG/OzmJ3VkF2qGXDHriugcuS2ddewLTZU1841gt2A6MDUoJ0XDn3ZQp7vZkoGr\nWlAdqA0okUXjLyZ+zGAz6LTcnE6HbjyVIPDpYCX+5wajwGlUZaCyUFmUs1A5VGXRu4j6MsMmglFI\nyMgxkN9MZAZyHJCvhyVctzjyhiUFd4nT66WphdZnuTfLXJRBcGSpyFQkypilQqhItiZIS/AtYWiI\nty2hbQmuJmIZNBzeCP3byPjWM78dCTcV6WCRQZXqmjmAj0th/WLuy6nwxS8yn83P6wETsvzJCLnQ\ntNyUMCQMmUy+ewUUKgPIvRnjuD+dYwtcLCKAWbR7TqXY30cw8d4qeVCX+Gnu9Vfif27QLHtog+oc\nqnPQVsu8Qm8CahtRmxmsgpiRQ0CYyVNP9sN7cXqWOL3kYu7qJR/e6tJd2uoixoAojZeamZYkHZ6O\nmQ6/zIOtieKI3hF7R7xxROuIOGIwTErR32SON5HxxjPfTPgbSzooZBCYbCmlC6mMpy4aeXH03Wl/\n/2gMnGvaQv2T0a1Ii2VSiK8W8rP8fVY8s1g2OKABOmBDIf2L08uWpKKYyzqnBCaUOB+an6428eNh\nJf7nBq2g0qjOonYWdVGhdnWRiwrVWrSdixVgFMRUNP40kW8G0tTDozi93GXmLR+xEL2ysCTdUVlw\nBpLS9FRE6RB2eNkxsKOXIrOtSaJIQZUEmVtNQhGjIo2qFL4dMuM+Mh4882EiHDTpIGUdkylkP4X0\n4tJKR05EOteqT1X+p0WTy91OO59p/Ix+pIdl0fgnr54U4p8K7U4afwdccp9JmIrTkymDO2n8E/HP\n6/E/TQffSvzPDXfEN6idQ31Roa5q9BcN6qpBVx6VhqV4hGLqTxGJMzkN5LEvZH8Up7/LzFt62NmF\n8I2FxpWxdhCVJlExy4YsO7xc0csVN1xxK1eMtiaTSD6R+0wikUMmjYl8m4qR3mfmITIPM3OvCIOQ\nhoQMAfxC/FPyTlpM6nxKjjmv9n9/lDMNeyL1ifj3Gv/cLtDLLWLBucavudf4O+41fpJ70g8JqvSI\n+E914FmJv+JDcG7q7yzqqkK/rNFftaiXLdoa9LFC9RaOCqZlj99P5H4g98NdLT3fEqc/1/iNg2U3\nQefAa80kFUfpENkxyxW9vORWXvJGXjKYiiye7D2ZmRw9eZiRvSdXvljHcyZMkTAr/AxhSqQ5INOS\nRXNK0T2l7Z7mTxLq/fF+j/9Q4xfSm/duDfcan0L8k1+x4eEe/0R8LzBK+fnV6ZHGNzw079c9/oqP\nAa2gXjT+hUNfVeivatSrBv03OrTWqLcVCgOTgsW5J29n8puBfOzv4/J+ScU9xeqX7a5+pPG7CjYV\nbCuYjeYoFUY6slzi8xWDvORWXvFGXnGkQqSU8+Y4IMOAqB5RClHldNkUMilGUhRiSKQYSGEux1Yl\n9XSBzkneq/R/fzw39U8aP51p/dMriovvnvSnKMZ3avwsMAkMGY4ZmgQuPSL+4yKdT4v0sBL/s4Na\nEmTKHt+hrirUywb9qkH/Cy1aFIq6OMluQGKGY0DeTOS/7Mn7fiEWS3xe7q7JgpgzjW/uNf62hl0N\no9E0uRBfZIfPRePfyCveyK/YpwriHgl7iHuIdnHIJyTOcDpWWiKS0xNluequBLfw5Wz+Xkz8qbFo\n/odaXz/S+Kfn9HvkvyP+U3v8F5Sf0yBwEOjymcY/yzV4sKbH808DK/E/MygEozJWJ6yJWBOw1mOt\nw7oSp9+qiS5PNHGmmj1m8KiDR26LF7+knHLfn/40d4Xs1mr04srPWpG0JiqNV4qZHZO6YGLLKBtG\ntWVgwyAdPS29VOWYnDCXPnTegbcwG/Aa8ilY/kNi20+VHrPEF1nkYe2/2qViDdUaZTQKVZpmzKq0\ny8qgBtCTYGawAWwCl4uS14BaegVEUfisGJPCpPI+Q645poohW6Zs8FkTRZVvIx+zkfhPi5X4nxm0\nZFwINNNI0wv1IdDcjDSbI3VT0+aR9u03tDdv6Q43tMOBdh5xIaByybnVGsyZnF87q6ls6VsdjKNX\nFSE7jtFRU9GrS76WL3ktl1xLy14Mg2S8TOR8gGQhHJZD8UZI8xKH/zG9dr8tO1EteQxLWnClUIuc\n5mZnsV9F7M5j6wkrBjcr3K3g6oQLkepNIr/L5H1G+oxMGeJi9YjCBEMaDdPRILeGuTYcreEdmiG3\nfP36gtdvN1zftuz7mmFy+KDJ8uk13Pg2rMT/zKCzUEVPO2e2Q2CzH9l2lk1j2ThDmyfc2ze4mzdU\nh1tcf6SaRlz0aCkGrVni884uYkqDSmdAG43SDnSL1y2eFpVbCC0qtRzVjtdytRC/Yy92If5MlkPJ\nqIs9hB7ShxD/YQ3CnWhVtHmrUd1DodPYrcXsPOZiwtYOi8FOYPeCI1H5RHqTyO8SeZ+QvuzZVRC0\nQMoKHQxpckxHh68dvXEoLDo6eml5fd3x+rrj+rZhf6wYRosPhpxX4q/4iaAlU4VMNwUuBrg8KC4b\nuHRwqRVNHtFvb9A3N6j9DXo4oOcRFR9q/CXZj3qR0xytCVR4afFcEGSLzxeEfIGXLUcuuJbtIi17\nDIOkovFlX0z5OBbSP9b433uaBbxffGQeznVxPpRwpkbtDHp3PzedxtQTth6wtcNhsCeNPyXcHMlv\nMvIuwyFDn1GToEOpswmiiMGQRoc/1kRTk6iJqSLNNUdpuL5t72R/PGl88/PS+EqpfwD8W8A3IvIv\nLY99AfyPwN8C/hz4d0Tk9idc54oFKmdcjLRz4qJPfFEnrlzkFzrxC0k0MpKvj+SbA/lwJA1H8jyR\nQyBLvkvFtUtcvqlLqK6poK0hK02fHCF1+HRBn18USS84pi845g17qbilZi81e+y9qX/KaEvzQ/md\nTX3zvmiDqixqU8KZ+sqgryzqyqCvDKbRWAYsNRZ3b+rPReO7ISLXglxn2GfUkNGzoKNgRJiyYgwG\nPzkmUzPSMqaWcW4Yh5ZeGvbHmttjzb6vz0z9n5/G/6+Bvw/8t2eP/SfA/yEi/4VS6u8A/+ny2Iqf\nGEXjB9pp5mKYeeFmvtKelzLzR3GmlolwO+JvRvxhxA8jfhrxMRCkJOgYs4TrXCF81xTZNCVBJ/iK\nPrQEf8Exv+Bd/pJ3ocghdQwoBlEMKHpRDGQ8E1nmpVNOeF8+SOOfinYMSltU5VCdRe8s+sqiX1r0\nS4d+aTGVxsxHzFxhZ4edDXZS2Flwc6LqE9xmuBXUXtC9FEdfEGwGsmJeNP5EzSG13PoN+6Fjf9hw\nlIZhcgyjY5gc/VjmPzvii8g/UUr9rUcP/23gX13m/w3wG1bi/16gcyF+N49c9ANf6J4vZeBvxJ5X\n80AtE8MxMB58GQfPMAckBmLOd0U4J1O/rQvpty1cdODR9NoBLT5t6eMXvMtf8k38Jd/Mf8Q+tngC\nnlhGiXfXmaWYRlLJtpMzyb/LHv9xxb8D7UphUudQlw515QrpXznMK1cO29i32H2N3Rfiu1nh9oK9\nTbhjhKOgjqD6QnwzCTaCFchZcfSGhGNKNfu55XrY8MZteeMuONLggynizf3852bqfwteisg3ACLy\n10qprz7imlZ8B7RkXPS008iFPvCF7Pkq7vnlvOePxz21zOzHzGHIHMaMGTIyZ0LITFL2+GYx9c81\n/kUHuw5GNBUV5BYfLjjyguv8Jd/EP+Iv5r/JbWxKlZ/0ZWQoGXqU6j/w94k25wk3d/Mf9C15X+O7\nIrpaNH6F2lXoK4d+WWFeVZhfVSU68dsWS42ZLVaKxne3gvttwt1G1ARqBD2BGQW7hPWcQMzFq5+T\nY5prDrrlrer4a33BX+lLDjTkrO5F1IPrzwW/J+feb87mv15kxe8CJYLJCRcDtZ9p9cRGDVzIkcu8\np5IZmUsM28TVAAAgAElEQVQPuTCXCtcpll4RSgANWWuy0SSniZUm1BrfaOZOM8uOMe0YwgW9veCo\ntxzYcJs33KSOfay4r43X3CfNBGDiPoHlO78FDw63fzBqNGYRtdD/VPEvKJvRNqONPBRdpFaeSjw2\nBswcUVNE+kQ6JNJNJh2EvBT1qVAqbEsadIkUVklhREM2JLH4aBml4ig1t7nhQPPR/08/Hv58ke/H\n70r8b5RSfyQi3yilfgm8/u6X/+nv+DErvhWnrfBTj78/vZtnDEE5Bl2BqYi2YnQVx6qirStGueBr\n/xWv3RXXdsveVAxG4XUgq57lYDlg5J7opyKZ71rsCfq+eYgy981ElrlR4JByaAZChaeS+2td6ZJp\nFB0yOtg7eOMQVyyCCzXQ/PYb7Ju3qDc3pOsjfj8yHAP7KZPCYnws/Tyk4s4oMRnMkn2rI+glGU89\nPvP7k8WveahU//G3vvKHEv/xr9n/AvyHwH8O/AfA//wjVrfiY+Jxjou8//T5XJTG6xoxLdF2jK6j\nqjpc3eGajlG2vPaXvJ4vuXZb9rZm0AqvwmLKawrpR+6bYZwY8diUf+LWo3Rpb2vc/WhcOeNKu6K1\n8bR4OvF0BEqlfxFTQVKWHC1psOS9IzlLxpKCpWOiuX6Le/cW9e6WfH3E304M/T3x7+43Z/ed09wm\nMBOYmXJG3gRqprTe/nQOu/1g/JBw3n9PUdm/UEr9c+DvAv8Z8D8ppf4j4J8D//ZPucgVPwJPWAHn\nD2U0XlVE3TGZHcru0G6HqnaoescoG66rjmvXcm27ovFPxFfD8i4zRdufE/+x8049PSpdmtnpqhx5\na+sHo9aJioFOMjtmdgRKxf/Ajh5bJYIyhGgJoyHeWgKGECxhMNQy0+5vcfuSx5D2R+b9yHD0HEYh\nJnCnBiN1OYPP1mU5pi7EtwOYHsywGCZQmv76j/I/9Engh3j1//1veepf/8hrWfGx8NgKOJtmpcm6\nIpkNye5I7opcXZHqK1JzxSQd+8lyW1n2zrK3hsFQTH16lrpU3m979V0a/9xgPGn8CmwD7iQtuAZt\nIhXCRmZ2wBWBKwauuOWKW6z1zMowR808GmYMc9DMg2G+NdgcaPoDrj+i+iO5P+L7ovHNlAlAXUFt\nysgG9NJPz3SlfZ7Zl+Vps7gYEyjPDzpt93PBmrn3c8FZA5knn1sgqmj8WXfMZoe3V8zuJXP1krl+\nySQtQ5UZXGawQm8yg86Lqe+5d+TFR3JO/Mcm/vkd6MzUtw24DuoOqiLaeCqZ6bDsEK7wvGTgJbe8\nlDc4MzEqzRA146gZg2YcNOOtZrAalSPtNGKnETVNpKnkMQxzQCYhGOgEsgHqQnp3CWoHZrcQvyq5\nDuekx/GkNfW5YiX+547H3hf1vsK/q75jMfV1xWA6BntJ767oq5f09Sv65hWTNPh6wlcz3s14O+HN\nvDj3Zgrpv60RxlMLO1/gEp+/0/h10fTVBuot1Fu0namkp8NySdH4Lxl4JXte8ZaKnqMo+qg4Bk0v\nqlyLwokip1LE5IJHhUCKgTkEJARiyIRm6TdiQNfgNpCLaYG5KsS3jzX9WHYnrBp/xSeBb9NA36GZ\nsirEH03H3uy4tVfcupfcVq+4rX/FJDW5OpDdgWwPZCNk7ckqLs69mfebS/yQZhP34TrUmalfdVBt\nod5Be4G2I5Xc0OEWjR94KQOvuOVXvKVKew5esQ+wn+HgFY2HyiuMh5DKkVY6l3Ptcs74nImSUTnj\nlz4Zetnj1xuQS1C/APNVSeIxLN59D3oE3SzEXzX+ik8S5179b/klFRRJDF5KSupAx5Ete3bc8IJZ\nqiXeFSkkN8u/PHm3vs/DtWh1pZ8ebY2yDmUMypSz/pTOKB1RKtDgadRMw0QjEy0jrRppGWjpqaUv\nvThDyVMIE/iTjEVLo3lwOIY6SwTMFaRKE53CW8VsNINRGK3QWjPkhkE3TKpmpiIoR8SS747Z+Xlg\nJf5zw6lL1Wmr7ikO+hHufHfTIiff3Y/qL7FUAenlXGltH8yVdRhXlXoBFTDSY2LA+COGmkvds+Fr\nal5juEZkT2BgxHMgM0c4zjB4mJb2+yGV1nzCvQtB2yVCaHkQPXSdprqw0FiCthyTJQyO4dZyg6VP\nDV+/2fL6Zsv1YcO+7xjmGh/sZ5WS+31Yif/ccCL+qUvUvMhIaTUl3IfoT1v6b3PaPwWlih1t3BIj\nc4u3rCrptlZjjcIZcMrjsqdKCufBZcVOHdnw+luJ71Ih/eBhjDAvrfdzLoaKUoXwtl5cCPX93NZg\nGlUO9GxqvK4JsWYY69KuzNf0oeH1Tcvrdy3Xh5b90DJMNT7azyol9/uwEv+54Vzjnw6mOWl8t7zm\nlJR3HqL/McRXphTT2GUfb+tlbFAajA445WmUp5ZQjuzOZbzgyIZraq4L8dkT5J74NhdNfxK/tN9P\ncq/xjSuEXwIF90GDFqg1wTqCrQm6I8SOMHR43xGOHcfQcL2vuD5UXO8r9kPFMFf44Mjy8/HurcR/\njjjX+CfiVyzH3fIwKe9c4/8gnDT+onbdErJzLbgOrTNGeioJ1OLppKfNPV3saeXIVo5s2FNzi2Ff\niH/S+FKIP0eYl5OrTho/LTemk8Z3dSF8cwHNdhkvIDtFnywhN/jc0ccLen9Bny/o85ajb9j3htve\nsB8M+94yzAYfP6+y2+/DSvznhqf2+CdT/9QZ+lzjn+/xf6ypb6tC+LorIbtqi1IRkwIu9jQp0OYj\n2/iOTXrHNr1jkw90DNQMGAaE/o74iozJDw/ZCYucNL7WC/Gb8rHtBXSX0L0oYzSaOFmGqSZMHX24\n4N30gnfTJe+mFxymmmGGYVIME/TL3Me7Ywd+FliJ/9zweI9/0vjnp8U+1vg/9jCYO+IvGr/qljj9\nDqU81velbX0uGn+b3nHhv2EXvqFL+1KYg8fgkaXaf8STyEtfvEL0dDpoRx7t8d29xm+3sHkBmyvY\nXoHXin5fmuZ733GMF7wbLvlm/wu+OVyxH2t8yPiYyhgSPpYxy6d3Is7vipX4zw2PNf4pYqfPnn+8\nx/8xpr5S9xttW5eNdbWB+gLaHYoRQ0WVoVGeNh/ZpGt24RteTH9BG29PB27fHX0RyKSl6l+xnK3x\neDzf49v7PX6zaPyLK9i9hFE077AoXxP6DX284Hp4wTe3V/zFm6+4HZpyElAO9+PZfCX+ij8czhPi\nzvtVnEz18/6U5+FnRenLnxM2BVycqcNIM/e06oDnFkON8ntUPEA+oijVKsqNpcjfzmfJeAp5lKAn\nVkHtoZqhnpFqBjeDnRE108hMJyOdDGVvn4908UAbb2nDDU3av/d1TxX/d7mBp376Rr3XW19tFLJV\n5A3kVpFqiJUi2KLt59wy5ZYhdfSh5Ti3HKaW26Hl5tiyH+uzLyRnn77G8Vf8IfG4D+WpOU21CDw8\nVPb8JkDp4GOzp04DXdyTwjXoBqMsFRBUhYoHtBzQ+oB2e5Q+oN0B3RxAPGJUqWc3CjEg+uxae0Rn\nREVEz4gaEd0jeY/4G+o4cTF/zda/5iJc08U9dR5wuezhH3/Nx/NSsK+hel+k0tBq4oVi2mpUUw4C\nmbzicNS0aIbY8fX1C17fbrg+NOxHxzBrfMxkOTUROT92+9M98fZDsBL/c8R5S7qzrlR3cfgT6S33\nxL+r0rsnfor7O9I7oJFI0q4k1dBjTI/W/f21HFE6kp1CLIhTd5KtQhwIFTkFJHpyHMmpR+KenDbk\nucOFmc6/LnIifhqw4tHLOd1P1hqcRIPUGloDnYXOQGeQ07w2xMowVWUcleHgDe5gsLNm8C2vby+X\nBJ2G/eAYZoUPJ+IL98Q/j2V+0h04fjRW4n+OeKod3bnGP8XnHxNflWadTjx1HiDtMaFo+lYi2zwh\nxmLMiNWLmIejcpFcKaRW5FqRK13m1XKdHXny5HEijz1pqsljQw4NaW4w3lP7a+pwTROuqU/EP9P4\n57uYx+PpmHDpLOwc7Cyyc8u8NOKMUoSTeAtzmQ9zw/Vhw/Vxw/WxXYiv8VEW4p88nydZNf6KTwHn\nrHhM/JPGP5WRnoh/5rFXi6nfpOHOvG8lEvJETEeUNbhqLmKW0ZWxcjOqTuRWkxpFbjX50ZiSIe1H\n0t6R9hWJiuQdSSqydzBGXNxj4y027sv8RHzJ75n3+pGciM/GIDsLVzVyVcFVhVxVZFsRJ3cnYa6I\nYZlPjnGs2Q81t0PDfqjvNX7M5OyXT0mP5OfjzT9hJf7niBMjzrpOfyfxz5x8mowVj0kDDhCJ5DQh\n6YjEG3SlqXSgdoHKBCoXqNpA3QaqJqA6IXeKtNGkTpPvRkXqNCkY4ltDcoaILafSDIaYDWm2yBRR\naUCnAZUGVOrReUCJRz+h8R+fp6O0Knv5btH0VxXysoGXNfKyIeuadFsx7StGakZfMfmK8VAz7ivG\n3jHMtshk6efHe/xSvPzQxF+Jv+IPjXNt/6jlPKfGkac9/1N7fMmLIw2URHSeUOqI1hVKVzjR1C5R\nS6LRkbpK1G2i2SbqbUJvIW11kQtN3GrS1pAuymNxVkSniChiUIRBEW05eTZ6RRoTIp6cPZKXUZb5\nE3v8c9Iblj1+pcu+/vJE/Bp51cKrlkhDrJtyGMbccDjUHHzD4VhzeNMwHi0+gg8KHxcJnBH/vMQ4\nPzH/eWAl/ueIx+R/rPFP6bdPEF8vzj0rEcuEUxqrNJYy1iiaRmgRGiO0ldC2QrPNtJeC3inizhAv\nNWlniDtN2mniZZnHURHIhCCEQQg3mWiEkDNhFsJUauMjmbCMUYqER179xxq/EP9+j180ftH08qpF\nfrUBaYmUI6+Oh5Yb1XLtG94dWt69bRn2mpwzWRI5p2W8v35/Ly+Pxp8HVuJ/ZhAUCUPAMUvNSENP\nx0Eit5JpZOIowkA5QMMv5MpnJqsmYSQVw0Du7xsOqCO0EdoETYY2QyPQAg1glCJqTVSGpDRRG6LW\nJFXGqCGoVMgvmSCJkDM2JWzM2Ch3CYPq7jsVw/oucq41ohVJl9NxT9dZa9SuQi46ZNMhbYc0LVK1\niG0R3TDmhoGGIbccY8PBN+znltux5V3fMPWae8fd6dPz2Sp+XgT/NqzE/8yQ0fhcMcSOfYRrb2mm\nBlttwb2gkYlx8oyzZ/SBMXrG5PHZLz3z8oPkvTOHPwBKKB1lQ8nXyRVEB1EXsuoMKUCchTQKqc+k\nA8RbSDcQJ4hfZ+LrTLzOxH3R/NFDzPeRxvOEwHMjWowmOUeulqOyqnKUr1quuajJv6iRTU22NRJr\n8rFB3tRkKsZoufnasH+t6a8V4x78kIk+ITku3/hxmO7n5bH/IViJ/5khy0L8BPtgaXyDnbcweqL1\nNDIRpoEwD4QwEMJIiAMhQ5ZCtfNd63mnPDn9lUAC5Lmcdh30kt2bQQUhzUIehdQL6SCkvZC6TNos\nB+S+LsRP10LcC2kQohdSlvcCZY+DZaI1uXZI25C7BukejZuKvHXkrSUbRw6WfHRkHHmyTN6xf204\nvFb014ppL/hBHhH/scf+57V//yFYif+Z4Z74ln1ssD7DnIk2M5li6su4R+Y94vdI3JdzK3NEmO58\n1k+1xzzlvecE2UOyhfSeQnoXyiETaYTcFtLnNpNblvAeJA/pOpOvhXSdy01hkPJ++Z5u3xYhF61I\nlSN1DXG3Ie22y7gh7jbkpiY5TbaaZBU5atJBkydNvtHMk+V4beivNf01dxo/PUn8VeOv+EyQMfhs\nGZLGBg1BE2fNZDRHpWkZMdM79NxgvMVEMCli8oQRfZfO/zgP7S5wlQvxUyj97CxL59kA1oOaIA9C\nriDXQq4hV7kk79RCjpD3Qr6VMu6lvN4LOT/dm/eBqa81qXKETYPfbQlXO8LVJX4ZY1WRIqQopCRl\nnLib+1Ez7Q3jrWbcnzT+ydS/O13gCXle5F+J/5nhTuNHB7Ei+opJVxxVxQ0VrYzUU0M9W2oPdYhU\naaKWI7Xou2Muz4kvFDokyiGSMYEJiyc9l7PkjC9HS+FAHGQH4jLZqpLC65YxgQyF7DII0gt5APEg\n+X2qPabdHfG7hnm3Yb66ZH55xfTyivnlFwRbE4+RdEzEYyJNp3kk9YnQgx8M86Dxg8L34Ach+bx4\n7Tn75J9fmO6HYiX+Z4YT8UkdMbRMuuOoOio6KuloZWTjLZsZNiGyiRObdETlCrdU6jz+tT938EVZ\nDowEVF7mnrszLpUF0UI2lAIdA2LkvmAnF5LjpZDdSzk52wvkb4+Sv6fxu5b5cst4dcn48orh1VeM\nr74qx3+99UQ8cfLE6IlHT3zjiW896ZCJXi8CyQvRZ6IHueukIU998k/7H/eJYSX+Z4aTVz+mjinu\n0OoCzQ4tO3Ta0TJy6eEyRC79RApHVLrB5QpZusSeft3V3XveQy3OPZXv21IrddauWrGU4gqiQJYR\ntZTo3qlxec+aVvn7I+R52eOfNP54taN/eUX/6iuOv/ols9QERuI0EW5GYhiJx5HwxhD/EtI+Ilkh\nWZXmHBkkZ3IWJOcnPvGp658/VuJ/ijhnmlIPrsXWSFWTXYVUVZlXFbqqUM5hiSVVViwpG1LS5KTJ\nSiFKPUm893CuCJ84IfYxYWX5+3T9VDntg8dONxP9aK5Atoq5U6jWQG1JdUVwFZOtGXXLmGuiCCEW\nT32cEqGPxEMk3hry/tu+1fPaw38fVuJ/arjrWWeeHLVz2KbB1QbbZGw94RqFrSOuHmnk/2fvXV4t\n37Y8r8+Yr99jPfY++9yMe/OQZuVfINW3YQk2BUHQhiK+EBs2BG2o1Sko7GgjQQU7hRaWICh2Sjsi\nNrJAwYaioGg3qYRLnrh1d8Rea/1e82lj/tZea++IcyIi77lZ5967vjAY87diR6zfXrG+vzHmHK+B\n3fgP2I7v2I0H+nGgGWdMjkj41UtLP076Dy258CHNBOoErfNk7Mt07OfrZVfI95nQJhaJmMUjh4Xy\ni5nISIyJ+POZ9HYhPXryIVDGRPH5ypW/4VO4Ef/HBpFKcmMvYi9raQy20zSdpu0yTbfQdJG2m2j6\nA10a6A7/gP74SGcP9GqgyTMmBESVH8TmfYr8Z9K/1rA2wzzPy7walHu+njcQ9oWljYwSMH4lPnUA\nZgi5kv7tQn4M5EMkr8T/LSuZ/7XiRvwfG84W31hwzQeiWoXZZNpNpu8zm02k32T6TWazybTxRNP+\nAufe4dSBJg+4MGPmgMhfvsW/1sCzxbftOjJvnZd5npvp2sLSZMYm0VAtvmKhLDPxMBKXRHoM5Ef/\ngcX/rWqD+2vGJ4kvIv858E8B35ZS/tH1tb8B/OvA2/XH/nop5X/8td3l7xLOFt+uxG87aLpnrXqw\nu4Vmu7DZLey2C/tV73YLXTih7SNavUPnAyYM6HlB2/ijsPgi66yNrhK93UN3B+0ddHtwTWEsmRMJ\nVwJmCciyUMpEKhNxyeRDID/Fqg/xytX/AX653xF8jsX/28B/CvydV6//cSnlj3/4W/pdx5Wr75pK\n+n4DXRXZFuz+SLuP9PvMfr9wf3fiq/2J+/2J1h9AHSA/IeEA8wCnGeyv3+JfP1S+k/xXrr7bVML3\nD1U2D/V5d5wz3Zxwc0QvHjUvlHkmziNhSpXoY6ou/pDIY4Sbxf8ifJL4pZT/RUT+ykf+6Len5eiP\nCa9d/barpN/sYLND7RLmPtLeT2zuM7u7ha/uT3x9/8hP7t/RLAdSPpHDQJpPpNNIameSDWRVfpBt\n8KdI/32WX165+t1K/O0b2P0eWFU4PGVaEm4Ol8O9p5n0fiKOuVr3Vbhe34j/2fhV9vj/poj8i8D/\nDvw7pZSnH+iefrchryz+NfG3d8hdwN5PNA+azVeZ/cPC/VdHfvLwjjdffUs7P+HDjJ9n/GnCP834\nbsbbSFC/Hov/Xa7+9foDi99dWfyvK+n3vw+GwoZMt0Sa61P9tzPx7Ug8raf3z3kC5eX1DZ+Fvyjx\n/zPgb5ZSioj8B8AfA//ad//4n1yt/2iV32Z8T59YKSjJV/LyGgelWarYhWIWil5ALRS10BJoZaJd\nK/E7jlXKE315j+OArHOxCrUUNxFQX1Brfp06sObmPKcU1AQeIa/6ep1F6s/nguSCyiDlap0LjYA7\niwIrVYxapay9Q7IgSSApCELxUgtxFrVWEhUuEzXOv9rvusX/01U+jb8Q8Uspv7i6/FvA//D9f+Ov\n/UXe5jcUwqVfzCpyWWuVsdrjzmJ87W23XoszpCaSWk+yM5mRlE6k+UDiiY7AVn5Bn/8BLv4C7d8h\n0xNpGPDHGRZP+Lknvo2kx0g6JPKYn1NmPwV1TiNQ1fHQeg3BrWu0kLR6KUY/r8mgQ0aHhPa5rq90\nQ2GToF3ADvU4orQQzXmMn7A8asLJEnxDKh3ZbMn9Du7vqhcUM6RUJb7S5XeZ/H/ES6P6977zJz+X\n+NdJWIjIz0opf75e/jPA//NF9/dbjTPxLci5+d1lrVSiMSOdg95Gepvpnae3I70bUUYIxhP0TDAj\nUU6EeCCwJcQNTYps8yNdeIdbHtHTOzgdyIcT/v1M8QvhbSS8DcTHSD4kypif49yfOpiRs/Vd0wes\nWderxkK0imAN0emqrSFYjTgDCcwUsVNadcRMCTtGTM40QHsm/gjqCNS/xhJhVgp/1PiTJS6V+Mlu\nKJsdJd9B04APEGLVPkBYB2GcB+jd8El8Tjjvv6aa7K9F5O8DfwP4J0Tkr1JtyJ8C/8av8R5/w7AS\nX84TLhqQi1Yq4iz0LrJvZ/ZNZt969u3IvjmgdWZhZmHElxMLHUvq8bFjocP4yDYe6JYDzXRAD0/Q\nH0ibgdDPlOCJj5HwGImP6ZXF/zQp1HrE4Cw4V6VpLppG8I3CN4altejG4lsLjaW0FongjgF3ClUf\npQ66yRkXBCsFl8D6lfimOugxwjLDooVl0QRfLX4sHdluqsXXd9A2MC+r+OqOQC32l/h9v9oNV/ic\nU/1//iMv/+1fw738luDK4tOAdFWoWqmAM5FNM7NvFQ995qH3PPQjD/0BrQJzGJliwxwbptBerRuU\nSmz8QDcNuGHAHAdoB1I74NuZlDzpkEhPlfRn4rO6+p9j8bWuFr5poG2ha6tuO6AVll4z9xrdWXTf\nIJ2DviF3DiLY9wvunaZ1QqugyYU2JJpJsGvFn15ADbUKsMTasquMMBvBF43PllhWi282FL2ndHfg\nGximeoN6JX3O9ckht0DT5+KWufeDQ9Y9va2WXjpgA1JFqQVnZnp3Yt8pHjaZN9uFN9uRN9sDRmbG\n2TJMlnF2DNEyRsswW8bZUkqmnWdaO+PcjHYz4maym/F2RudAHjNpzFUP+Yv3+GbNH2pcJX3fw6av\nmo0wbRV6a1Bbi2wcZduStx1x00Io2F7TOEWroMuFLiT6OdCZ2syjJGBZj+ISlBnSAPEAixMWYwjG\nEk1DtKvFNzuwdxAaMFekTyvpF38j/hfgRvwfHK9d/W4l/Q5kh1Izzg70jePuTPyd55v9yDd3BywD\np6PmiOEUNUc0p6RpZ01z0qRYsCZgdcCagNYRTCDpQDCBmBPFF4ovZJ9XXZ7j3J9t8W21+F1XSb/d\nVmEnmL1C7Q2yt5R9Q953xF2H3/eIL1inaBR0OdOHxGYKbI6a3gg6rGdxC7WTzgJRQ9JVz43ge43v\nLaFvSGbd4/c7Sn8HsXnp3oeV9FrXp9YNn4Ub8X9wXB/unV39M/HvUNrhzHt65y4Wf+f55n7kD+8P\nuHLggOIpCodZ2CA8RaFZBHsSwgJKvQwFIpmkCkUyUs5x7VqLXuPcfHac+9nim4vF32xgt4X9HrgD\nfa/g3lDuHfm+Id63+Psec7+BpeAUNCXThkQ/RTYnw65VbI2gAJ/qQZ6H5z5/53F/Syssd5qQLVE3\nxK4jm3WPf3df2/7Cxb1fPExzPZG8WfzPxo34fxGoNai9zmV/XouANEixqKIRVG1XTUZKRIqnZxVZ\n6Fjo5Swzvcw4ZkKhSqriAwQP3te9MXxY5772z3iOu59v6bmk/xxVvIq/X8u5Xj83QtpC2gipg9gI\nwYI3wqIExLLIporqV+lWaUGB1QWrClaDNYLVCmcU1miU9vhcWHLGp4zPmSXV9ZIzPhSSzdBmVJ+w\nKdIS2CjP3i6oDMUEsolkncg6k1UmS1n7B9/wObgR/0uhhPqNvhJzWQsWExtMFEz06DhgYsTEERPf\nc5dHdvHndMtb7PSIuANRj0ziOZaMzXB6gvEI81hPumP4/BD12VU/l/DrNSavruLwUWuS1kRtSOs6\nr+KtZnKCOEVywmIVI4qDF/qTomTLkFvG2DL4hmFqGYaW8dgyPDkkwPRtYfyl4vjUsBl7NmGhZ2Fj\nF3TjSdETlSdJqIeRxROLJ2VPLBnJHptG+njgzj9SlhY9G9oRhtLglxG/jAQ/4sOITxM+e3zJt+S9\nz8SN+F+KM/FbW6Wzl3VrUWjMLLhFaOaAWxJuGWlmwSVhnwb28S2df4udH8EciDIyF88xZUyB4Vhl\nHsHP1dqn8/SJT+A64/dcxv8ci7dQrBCswVuLt5ZgLcU6sJZsLVkZJtEkNAuaSRTHonG+SvaGKVom\nb5lmyzRappNlerKMW4tEoXunOD06uqdENya6EOlKojMJ4xZQI4SRwghMkMf6u0msY7TOxA8HSmgx\ni6GdYTdFhtIyzgvj4hmDZ4qeMXnIYZ0YdMPn4Eb8L8WZ+J2FTQPb5oUWwAyRZgh0q7REuhjolsAu\nn9iHR7rlEasfQQ7EMjInzzFkdIFpgGmsepkh+try+nMtvrqq6m3cqteS/tIIc6PRjUWaltI0xKaF\npiE1dRpNCgbvDSqY2qI7aNRi0MGQlGbxmnlWLKNiPqo1vKdYOgVZaI+O5gjNEdoRmiC0QGPBugUt\nBzSr5EPNDJSIZiavFt+lkU08YLyhXWA7RxY3M5aWw5w5+MTBZw4xQ8rEnJlvyTufjRvxvxRyZfG3\nDdx1sO+qvutQJWMOI66JdDqwYWATR7bLwEYGtulUE3D8E1YOSDkQ08jkPXrOqLImsqzi55qYluKX\nuX3CpZ4AACAASURBVPrnU/lz/L1bS/pLp9CdRjpH6Rpi16O6Drqe3PX43FAGSznZqrEUX4XBEovC\nz7VldWjAtwV/1i1IEexscLPBzRo3G2wwuGJwRuPcjOMdrrS4bHAaXIo4mXEoVLlYfB0NbYDkI3me\nyfbEUFrezYp2UZggEBUxKeaiUNfTQW/4XtyI/6W4dvU3K/EfNs8iJWKaSKPHmlefBvbLO3bje/a8\nY5OPtHGkXUZMGSENxDAyzx6m2oY2+Er24C/ypXv8c3Fft5bznyX3gmw1ZWOJmxa/6VGbDWy2pM2W\nEFvCu4ZoHYHat/9ZnxwxKKKNRJeINhFsIrpItIno6jQ+kxpMajGpwaamnnnQYGxDw0xHS1cMXYYu\nRTo106kTvSgsAV08Lo6oANpH1DKj7Qlt3jOUlnZ2mMWBd8RomZPjlB2qOGpE5YZP4Ub8L8W1q7+9\nIv7v7eDNDikeo0ccQpcCm2VgP7zj3rzlK/mWLh/QwWOKRyePeE9cPJP2BF13qDGuse5Vp/jlxD8n\n4LQr8be7GofPO6HsDHHn8LuGedeh91vY7cm7PWHpmV2dwjuHhnlomGmZfcN8aggTJO3J2pPMqq+u\nRWmU6tHSo9QGpTbPa202tDKxzYZthm2K7OJM0CeyuDUKktHZYxM0MdKGmWY50RpHoxwjHWbuwPfE\n0DOHnlPqcBlUOc8Fv+FTuBH/S6E+4uo/bODNDn7/DpVnDO9pEnRLYDsO7Nt3fGW+5Wv5M9r8RCmZ\nkjJFqkTJBLVec6k1OVee5rL2h/+LuPpXxN/vId0J8U7j7yzzXYu961F3W7jbk+7v8dOWiZaT7zid\nWgbbciodJ98ynDr8sVBkWmWmyARqooiiSAFjENchzQ5xe2j2iLtD3B5xezo9cZfhLkXuTSV9Uu8R\ncRgRXMm02WNTHQay9YqtVuy0YiuKkQ7mPXHZMfs9pxh5n8BlQz1JuOFzcCP+xyCvY/NX0jZIa1GN\nRpxCLIjJiI6I8nQsdDLTyUQnAx2ntWb+ia68py0HUnk5O+5cKf9cMS91aMXzbSjq/5ScS88VGfVC\nF6lrmtqwshbYFJwtLLqwKFhUIUnLIjXuPr8WaZlUyygdQ2k55Y5TajnGjqNvOS4dYT4Xv58nZZwH\nX6/W1mhozRrtcFDWAiXdQu6YRUB1KNthcoctLY1qWXRLcC2RheIK0hSUiRgBVwpNKPRSoCx0XmiD\nocktpiQ0BaU1Ylx9L861+uXD9Q3AjfgfQtbUNbMGvl9p2VrMpsFYqe76MmBOEfM4YnhPn0bu3/6c\n3eNb+qdHmuGAmUckeHLJz+QWLqOrNFcT3ATEXEr4xVy0aEhKE7B46t47SdUBh8dhrOC7zNwlRpMZ\nSuYYEk9TZksmJ8dT2vEUdjzNPU9jw9PJ8nRQHJ7gNBeGnyemt5HlMeAPijgK2bNW9xVgBhZqrt2r\nubel1GT8FGpebphrr61z9pBaoEx1CqcqlFZRsqOUjpx3ZApREkFlFknMkrCSMCWjfGLKwhA0U7bM\nNATVEe2WzI6i7iA1l8mfZ0lX6xv5gRvxP8Q5Z9XZj4rqNKYXnBWassbpTyMNgvNCHwd2j28vxD9d\nET/n5+w6YJ1k9zILT1TliHIgbuXM1TpoxSQNhY5IT5KehZ6Rnoke0YrZRiYbGUzkSKKPkc0Y6UMk\nB8MxbDjNPcex53hqOR0sx43muIVhyUxvK/HnR48/CHGsM+jKM3EWLsQPXIhP/fPzuN3oIUxXKYOl\n1uOaiaIDpckUoyjaUUxPMTtygRgjPgV8DMwxomNAxQApM0bFkDVTsiw0eN0R2ZD0WsSTmjXjKa73\nEEBCfT597n7pdwA34r/GeZPsbK397poXWlowLtLYQFcCnQ+0x0jna8y+Dyf6p8cqhyuLH6vFPw+p\n/E6RdbqMA9VeRK/aGwU4ovTM7EmyZ2HPIHuO7MloRgkMeFoJdMXThkAXPO0cyLNinFvGsWU4tYxd\ny9BZxl4xdjD5zPyYmB8jy6PgDxDGTPbpivj+Sj5i8fP5RHKBcEX6nKAE0BMYT+kydIrSOXLbU7o9\nuQhx8oR5YZk8elpQE0jI5CUxRWFEM2FZSoNXHVFtqsVnJX5YVm/D1ycpVNLneDP4K27Efw25svhd\nA5vuhSibMTLiJNKVemq/8SPbocbpO3+iOR1oT080w6Fa/OlDi3929c/n0Oa8Xp872lWy6x70ZtU9\nzFYRxDHLBmFPlgcWHhjkgSceCMXQxAUXl4sOl+uiYB4tc+OYGsfc2CqtZm5giRl/SPinSnp/KMQx\nk3yi5MjaNoNq6c/6lcUvCXKo5XZI3SKkvFrfSGkmignQFcpeU/aWsu/Iux0JRTpMhINhOdTIPCFT\nSiR5mIIwKM2kLLNqCHolvtpT9Ep8P4E3H5L+FuN/xo34r6E+QvzdBvYb2G0QEzEh0oSRLgS2fmAf\n3rEL79mHd3TLETOPmGnEziNmHl64+tfT2BWV9I7nBl0YqeXmxtXe83oDZgtmV8U6xSSOEz0ie5I8\nsPCGQd7wJG+Yo8VOM2aesPOEDTM2TPV6mikl4a3BW02wNTffO423Cm8hpEwcE3GEOBbCkIljJHsN\n+RwqO1v5a3lt8QPPpNe5egBxAZ1ge7H4Za8oXzvK1z3l60IumvhLTbBrOk7IlDGSSyB4YV6E0Rom\na1nUavHthmR3FHNXDxO1uZTulvW91a1e/xo34r/GtcVvV+LvN3C3g/sdojxmGHGj0PmrOP34lq+G\nb2nnAxL8s6ir9flwbz2gf7b458p9R+0+a8+59i2YHuwOzF3dwupWccJhpUfkrhJf3jDwDU/yDYN3\naDOiGdFxRDGgw4geR/RppIRA0lJr4I0Qtay18FWnnEkesi8kn0hekb0ieUXJZ+Lk75Az8dfT/gLI\n+hBQSz2htBnKQjHhJfF/Vsg/U5X49srSD5GkAzEr/CJMizBQLf5i1j2+3ZCbHaVZiX+29HklvfY1\nj/lG/GfciP8a5z1+88ri3+/g6ztUWeP0AbqyWvzTO756/y1fv/sz2vmpWvZVSskvrq8t/tnFt1TS\nd9SW086AdevQiU0lvr0D9wCqVbyXM/H3F+LLN7yXP+Q4NQgDEk/IfELKCQknZBqQwwnmhaIyWS55\nBEWurkvtW19yXmv6ZZ0xL1fEf91N/9U4jXNhQa6ZfJWIq24LpUQwaXX1FeUnjvz7ivKPWFI21SEP\nmTwm8ntPNIZQNMYL07y6+vb6VH9DanbQ30NZ6/XPlj76NbJwS+e9xo34ryGC6NrsXRpBOgUbhewV\ncqewRXC+4MZEoz1NmWnDQDsd6E7vaafDs+ObgSRXNlGt7n1RyFUcPqEI1FzzIlCk1PpyVZ7XZZVF\n2hcyS/dCJmnXPe1V2+mQYEkwR5iFD9308979jF8l5n2OnZ9P0M/HmevJRgSVE0oSWieMzdim4Dpw\nG8EVQXeCamoTo6yFKFKPCeLawCPXvnxRDFFZsnYU21BcC7hKdOOq26TNau3VjfdXuBH/FUQKSieU\niSgbUM2M6hyqt6iNYVNG2mGk6SZss6CtR0wElUlSR1Q99+ig7tlfzNNAo7BIcSQcc3Es1DxzwWEQ\nTE7YmLBzwowJaxNWJwyJod3xc+l5i+NRFAfJjOLxMpI51O6z7wc4jjCMME/gl+pul+u9+SsX/Yf/\nJLkQ/hK3EDK6RGxKNDHQhcBm8eyWwP0c6PJEWY7g15rkuN57zhRKHd6x3nGRizy/RblaX3/2N7zA\njfivIQWlMtpEjPOYZkG3E6Y36K2iLyPdaaBp55X4ATGJovNzJp6WKkqt4bmrdUGRqG2jU+lZSk8q\nfY3Jlx5EYXLAhDop1kwBowOGgEmB0XW8lY63YnnkiviMZDnAYuE4w2Gq3WinlfjRXxE/8+JA7ley\n8B98gK/kzEINa0cincGlTBMDfZjZ+IX9PHM3LXR5Is0DeRnJYSJFT06BnBOplOecwWfCCxR1Rf4z\n8V+T/kb+F7gR/xWqxc8YHbE2YJsF2xlsr7Bb6PNE14807YRtZrSrFr+oVC3+FdHtOhbqWicUc3Hk\n0uPLnjnvmcqeuVSdRaHzgo4evSwYvdSWGGlBh4XZOh7peRTHI8JBEiMLXoZKfG9gWC4yL9VyptfE\nf23xf+iU1g9JDxpVCjpzZfEXtn5kN0/cTyNtGQnLTPQTIcyEuBBTJOQEvCT92erz2uJfv3az+B/F\njfivIFJQKmFMxDpP02hcp3A9NJtCVybafsR1E6ZZMDagVlc/y2UarZZKdKeg0dCsOqIoxbHkDbHs\nmdUDx/zAoTxwyA8ENDpPqDCh1YRiQqcZ5Sf0MuG15iA9TzgOZ4vP2dU3dRbV5GEOVU+hNuuL4RXx\nCy9J/4N+inzc4uvq6mewKdOuxN8sE7vlxN18os0jy+xZ/MISFlT0kAKpJEopLx9V5wfAdQbUx1z9\nGz7AjfivIBT06upbG3CNqs0s+kK7TXTPFn+u3WTWPX7Rlz0+q8U3K9m7VXoNXhRLdqjck/KeqTxw\nkDf8Mr/hl7xhEYPkARUHFAMqD6gwoJYBGQeiEkbpGXGMKIYz8RnJQu1TvUTwryTFNYnlnH33Mfnh\nPsXXe/tn4peELmeLH+n8wsZP7OcT99OBJo9MS8AsERVqym3MAcmZsrr6BdZBnSvp5Xtcfbg9AD6C\nG/FfQ0Dpa4sPbVfo+kS/DbRn4p8P91xAdHX1z9aI9XDPqmrpew0bU2VB1aYRuSflO+b8wFHe8Cjf\n8C3fMGCRfETCEUlVK3VEVIsoR5KEp8Hj8KLWebgeD2RiDR2kVAdLxlWndV2e75DvD8n9ih/gsz6z\n8Tw01NRGJavFr3v8hc0yVos/HWjKiJkTyidKSKSY8Smh1nThZ1efqz3+x1z92+He9+JG/FeQ9XDP\nmLj2rSu0baLvA/3G0+UZ1w/rHv8jp/rrF+3s6p8t/tbAzsAkiiatxE97JnngwBt+yTf8OX/IMTvI\nT5CfkPIE9FBapDjAUIonr2HALIpMJuPJRDLzyorrQv7zmu8oTf11nei/tvj1ASBodC6v9vgT+3ng\nbj7Q5hG11IEgyRdCLJhUUKW8dPU/dqL/sb29vLqtG4Ab8T+AUJ7Jr3XCGMHYgnUF12RsXrAuoG1E\nm4TohKhc4+ysLqiGYoXipM7UcII4qRV34ijRkYIjxAYfW6bQMsSWY+k44KB4yN8hzzV9Z5wr+dNf\n+if1ca1QotYoRkZJqqFNVYd/7NuZrV3YqJkuzzRhxs4zZpjRTxNSFuQIMnCp/g01/b+U796kPHsA\n51v5mNzwjBvxfwC8cJiVkKwidkJYO89OvcJ0Ct3XDjLT7JhnzTIXwhxI80wuAyU81Ukx+Qh5gDJB\nWb/5lzYdPwJcx+dfar0O1HCm4EzEmbCuwZnCXTfx0907vmoPbNSATRNMnuUQOVIwCU6PMDzBdIRl\nqsV2KX7WsN8bPhM34v8KeJ3rBtXiZ6uIrcZvNcvOYHYavdOonWakYzxZ5pPCDxmvI7Es5DgAB8h2\nJf0V8cuPifhn83kJ0V1rJYXGRjoX6JtI7+Kq6/WunXjoDnzVHNmoEy5OlGlhIXL0BR1hfILx/Ur8\nca2yDS8t/g2/Gm7E/xXxQaKrEpJTxE4TtpblzqLvDeori9zbSvwnx+wUiy6EEolhJs8DhadK/DJd\nyUr88mMhPlw21ddFxVUrSTiT6ZvCvovsu7lKX/XWTWzMia0eVos/UybPEiKnodRJPEeYTzCdLsR/\n3V78x/JJ/Kbik8QXkT8A/g7wM6rZ+VullP9ERL4C/hvgrwB/CvxzpZSnX+O9/mjwfcdjF1ff4LcW\nfeeQrx38xJF/4phKx+Qss1IspRBCIE0zWQ+U4iCbleyr8GNz9a8t/joc9LnUyKBUde83Dey7wMN2\n5mF74mE78LAd2JgRWyZcnrB5wsaJ4j0+J465UHzN1l3GK3129fMavlvxY/g0flPxORY/Av92KeX/\nEpEt8H+IyP8E/CvA/1xK+Y9E5N8F/n3g3/s13uuPGs8HTOri6oetQd07+LqhvGlIP2uZSseoHVPR\n+FgIcySeZpIZgHPNe1itfLha/1iIDy8tvuHSUcChxOPMRO8K+z7ysJ15czfwZv/Em7snej1S5gUW\nT1kW8FUvS2SZS23ac26gM9f1a4v/sS3WDV+GTxK/lPLnwJ+v65OI/H/AHwD/NPCPrz/2XwJ/wu8Q\n8b9rr5mvLL5sLdw5ytct6act8fc7ptIyFcccVD3cOwVis1SLT1mJv57Sl1f6R/E1/5jFv3QUUAqc\n0fRN4W61+G/uTnzz1RPffPVLWhnxx4gnsoSITxE/RZZjxB8zcb60y0vx5fq8x4cb+X9VfNEeX0T+\nCPirwP8G/LSU8i3Uh4OI/N4Pfne/AfgwpHTZ47O1lHtH+rohvukI3/RMuWUMlnlSLKdCeB9JzUw2\nhXJOwPneoNWPAR+z+A5oUXIh/rPF3w9889UTf/iTR5oycCJzCoXjUCgxs0wF/5Q5Phb8xHNF78c0\nfJz8N3wZPpv4q5v/3wH/1mr5v+Bz/5Or9R+t8uNEQShFyFmRkyYlTYyGEAzBG3ROSLCoaNBJU7KG\nrJAil8ltV/kzqYCsQeZchCCQVKGYhLiIagXTQ7PJdLtEyXr9ksuz5CyUrJ6/+Oc7/bj+XHxXTqtc\nLeXqj86v17JiVTQKQRVqR4ESUcVzp2b2amYnE1uZ2KqRrRrZyEgvA44RX2ofDhUBD2muzXj9UMN3\ncv3WsmYuSG3gkww4XbBSMGR0SagckRiREGowP8TagyCmmrWY85rM9IUf0W8c/nSVT+OziC8ihkr6\n/6qU8nfXl78VkZ+WUr4VkZ8Bb7/7X/hrn3UzPwacSV/J7lgWi5ocanRwsuRiyONImVpkadDeUpJB\nskIXwZSC9hk1JtQhoB490mrEqLV9dsL8MtIMji4mtiYTtoX8k/qt7O4M0WuS18Qg61oRvSZ6vZL/\nusjm9fpTeB1/fyVyXUssH6y1gM2Cy3XqjcsLLntcHnBFuFMjP82/5Kv4nu1yxA0jxS0sOnCiYAqc\n3tWQ3XyqfTHjsjbALetdKNBqLW9WL6+dKwSbWXRkIjBGj10WtJoQRiixliPPMywefKgPgPN4ot9q\n/BEvjerf+86f/FyL/18A/28p5T++eu2/B/5l4D8E/iXg737k7/3mobBaekMIFr00yNzC2JCHhpw1\nZeyRucMsDSk4SlyJD+h8Jn5EHyKq9c+kJ2aUSZhTxJ0SXcoEW0jb+oVUrdANsIyCHxV+FPyk8WM9\nNU/RrO2vXnfQSZeb/yyz9jJ//rLWILqyzEitMnrWda0oNCnQpUifIn3y9Pm8juzUwEN5x1fxPZv5\ngB1HMDOeyCllVIbxUGU+1ZP76OsenlwtvFG17aHVa0nzujbraLDFZmaVGErAJY/1M5qp5j/kWHsQ\nTPUAkXBF/N9+k//Z+Jxw3j8G/AvA/y0i/yf10/vrVML/tyLyrwJ/H/hnf503+peFgpDOFt87ZGlh\nailjRz51lKyR8YieOuzS4LytxE8aVQSTQV1bfLNuAWJG5oRqEiYkmpjpYiGbAjvQrWDvhXES5oNi\nOmimg0If6gFaig41OzJCDbScRZ7v/PNwXTjz3Nv3aq3X0kJ9qSt2qjLPKRQRFyf6OLKPfpWJfRzZ\ny8RWDWzLgU08sFmOuGGgsOBj4OgLki8x+vlUXfsXFn+18k6zZvtBc611YTKZQdX25k3yWL+g04SE\nsRYoLVOdMT77WpkYY3X5b7x/xuec6v+vfPcI0n/yh72df/i4dvUlWFgaytyRxg3x1JOLRo8b3NwS\nloYcHCXqi6ufC7ISX0xYSV+QOcEpIl3CmIwzmW41uKoVrBYao+hmzfDOYFtQpp6ep2gJc4Oolkrc\nc87+Nenzev2pb/eZ+NdtPq9EzGVkmNO1yqhddaNRsuBCYeM9+1B4CAsP4cSDHHjgqWbjlREXBuwy\nYhkhLXgfKVOmlDU+P60x+qnOvTi37BfWAiddid7adQzfKlYXBsmcSHREmrhg0oyS1dVPqcYBfQ0V\n/m65+p+PW+beK5Qi1dWPBryjLA1p6ghDjxm2lKyxY08zd7RLQwr24uoXQa/ElzG+IL2cDPI+oDYR\ns8002wLbOjTDboV2q+g3ms4bbJtQptaX5qgJs2E5uSvin/fkcCH9lxTpnC3++UR+HWxJC2JBmZX4\nZh2AqaEz0GqUzDgf6M3A3mcetOeNnHjDO96UX9JzhLL2yptnSloofmaZIosp5HKJzQd/idmnlfhq\ndfXPxO8s9A42TdVWFY4506dImwMue2xe0HlC0rh21p1XWRuQPBP/h/mO/DbgRvzXuLL4OVjS0hCn\nDjX2qNOWUjTN2LNMHXFpSN5BNEjWlUoZxNdDtoulV4gLiFOofcR+Xd9KtYI1imariD/RxK8NbQwo\n0wCQoxBmzXyyaHe2+NeW/rmXL59fgvYxi9+u0oE40LZO9Tj7152FvjYUUDLg5oFeG+505kEW3jDw\nTXnPN/ktXTriS51751PE+4CXiJeAV6W2B4gfSo41XCfrQZ5Vl7feNHUi+bYFK3AImQ2JLq+ufpjR\nYULCUN36vNTxXdmvjTpXV//G/GfciP8K1dXX5GgQ75Dnw70NMmwhK7qxZ7O6+imsp/rp4urjc22A\nMdeT8NpWfl3fRwx1Dp69VxSryFtN/omh/IGhTQ7I5FgIszCfNM17i3EOkTPx4aWlj1evfwqv9/hn\ni98BPUgDyq4TPew6X8DCxsLWokTh9Ht6pdmrwgOeN+XEN+kdf5h+QVOeGHLhlDKntT//kjM+F045\n478rRr+eS35g8R1sHOxa2HdgS2HLavFLoInrHn+ekHlcG474Va7ajd0GZr7AjfivUdZEubCmyk/A\nADQFXKnzYUeF95qQLUE5gm2IfUfc9URTTZfkGsiXlJFQIKf6GsDeoO4NajKURdcZc1lT0GRl6dXM\nrCyzcswys4hjEYcXh0GBWhBZQM3rel4n1Swg8RO/X80VoOg1Z2B97/X6ck5QKJJBrppySqYVTysL\nrcy0THSMdDLSyUDHiaYMhAwmglqfSfnc2j+CL2uMfpVzm7Jz3N5ZME6hnULc+mC0imgUXit86QnS\nE2lJxZKyIsdCCXHtJkz9z3uW63ZjN5xxI/5rlLVNlQ+1Q+0wgb3MYit6IY0TMSa8FpbOMZUtg7un\n2U7kuUF8RPmIChHxoa59RPlQE0l8hjFRDgEefT08M6rSrYD+Obi3mf4xEQ4exgXtJ1we8Eohdkbs\nUrVZLmu7rFkx3/frWXJYyGEkh+4isSOFllIaSq5NO4u3sBiKNhRVtwY7ObFZfk4zv0XPj5TlQFhG\nJu85xsyS4JhgyDBlWAqENZGpwPOgIqXX4aBr9PB8ba3GWQvOEZxjMI6gHCccTXIMuefnac/btOcx\n7znkDWNx+KKoA8rOZH/dWPTHlPn4Dx834r9GKXWf6EONBV+RnpwpJpDTTEyxEr93TG7DuLvHxUha\nWvS4XGSaqwZ0XK2+zzBGyiFC659JT6wjrNTbTPM20j8GymFBjxPOD/S5JypBWY+0HtV6VLsgXV1L\nuyDm+w/5cjCkeSLODWluV90Qp5YYG3JxlKQp0VC8oWhNEUNBU7JhIyOb5S2Nf4v2K/H9yBg8h5ix\nEcb6XGPO667nI8Q/7ySMrR/x+VoZhegGTIfXPV73iOqh9EjuOaWOt6nnbe54zD2H0jPmlfjl/Lt/\nbHDIjfTXuBH/NXKpFn/x9RupL6QnRIqLJD0RdSQYxeIaJr3B6YjRQvId5jA+iz2omtEeE2X2qFwo\nq8XnENZRO1BihjlRSkI/JtxjoDwu6MNEM7b0vmWfW7ISlA3oNqC2AbXxqG1Ab+q12O8nfvKGcHKE\nYRXtCDhCdETviNlSsiZHTQmaojSZuhXISdPJxMY/0oRHVHgk+wM+jEzBcwwZsxJ+ylWfLf655Z+s\n1r32M4TGrbqpGqMIOLz0eNkTZF81e3zac4odj8lWyZZDtozF4JHV4p/PPV5N8b0R/wVuxH+NUmos\n2AeY5vpaqqRn9tBlcjcT+4R3wtw5bLfBdoLuHcl32HcnbOtwRpMAGzNlDqAUKsdnVx8T6r8fC2WN\n8xci+hBwTzP60OAOjn50JN8QswOR2u+vi+htRO8j+i6i7wJ6H5Hm+7/gaVEsTxb/ZFm0wWNZksUv\nFi+GiCEnRY6K7HVt6rnWLeSgaPBs4gEXn9DxQIkHQqzEP8SMTnUf73PVS4aQLxZfydmlr2RvW2g7\n6Lqqs1YM2RHKBp/3DPmBoTww5AdO+YFTajkkeMrCIcEhw1gEX1iJf52+/Hpi0A1n3Ij/GqVc9vhw\nIf3iwc2UTSERiO7i6ts7Qd875H5DCj2udTRG1a/caunlNCFK0OHi6gNXpDfwPgAeNVqa0cBoYaha\nvIFs1wP3hG4TZpPRdwnzUEU/JFT3/V/wOAqzM8xaM2OYk2ZeNPNoWJQmRE3KUomPuqyDIi+CkUiX\nRpo0otNISQM+jYzJw5qSG1crH0sNbpz3+PDS1XdNJXy/uUhUihAdQ+wJcc8pPvAuvuFdfsO79IZj\nbBlTYEyRMUeGEhhLxJdIJvDhhKDbHv9juBH/Nc6uPjy79yy+xrWNpgQhO4hbCFqxdK6S/g3wBmLa\n0K6WPsVMmT2cJsQZlJL6PfR53dMXWOP8uLCmxCqM11dydZ11zWO3GdNm7LZg7jLmIWPeZOzvFVT/\n/V/uMAijVkwopqSYFsU4KiarmEThiyIlIVGrAlMSUhCSFrIShIQrHpc9Onty9oTsmYon5oxaSZ7L\namuv1td7fGurm9+uxN/uYLsFrxXD4sD3+OWOQR54l9/wLd/wbfqGQ2rwacLnKykTnkQumXq492Mu\naf5x4Eb81zhb/DPpz6NvRUAUJWnS1hCTxWuL6h1ybyg/taQ/sMQ8fWjp359QzmBEXU71r+L8XMf5\nEfRa/dZmoc3QPK+lps1bsF3BbsDeFczXBft7YH+/oDff/+v5IwwIQxJGD8MoDEdhsMKoYClCSH7F\n0wAAErZJREFUypCKkKQmviWp6yRQKKhSO/tLyRQyoWTi+tqZX8/Fwq+u5bWrf0X8/R4mpXCTg6nu\n8U/lgcdYif9n+Q95io6cjuR0IOdj3YaUTC7L1an+9Tu+Xt8AN+J/HOXKN339R40he6lFPEWhxKJM\nC66ldC0Uh+omdDdg257sGsRZtNUYK1gFtQSQy3tcldSf8+kaXiTS0q3aUbPX7FmfRVWtP5HHYxQv\nhk7mq3WhPoPS+uufj8euz8ivCVwXl3J9YB1pJZT1Yfms13VuIG2F2EPsIDYQLHgNi4JFunN2AEPu\nOKWeY+p4ih3vQ8churWcb+1PmDXPs7M/uzT5hhvxvxQFSqo59Mkb4mzxo4NTA8cWXQrN2MDs0NFg\ni6FVmt4q9g24jhfbznK9Bc2gyqWDnV7fL1PLcgrVUTAe7AxmAPMEtlmL6QDVf//thxOMv4DpEaa1\nNHaZ11qW8mEQ7PWxmHA1/nvVSl0mBKOFZDTJ6qqNJllDMppsNMEJc1MHjeRGWGw9nDsu0CFMZcfP\n5x1vp5bHWXOYCuMc8MtE9of6lIhHiAOkqabn/uh6Ev74cSP+F+K5O0/QpKUSX0YHQwOHFkMhjQ0s\nDhUttmhaUWyMYt8KzXr+VFZ2PY+zS+seuIAuq/e/Zrkl1h0I9YRcBzAzmBMYtx6WsXa1ab///uME\n8y9hfoT5CeahEj+Edcze5XY+eiwma18Ooy+ptee10YAVQqPxjSO0ltDUtN/cWEprCUYziZBE8CKM\nAsciuEVwXpjShrfLlrdLy+NiOCxn4o8r8U0lfRzqL5MXyFejdm74LNyI/6UoUJIiBY3yBpktjI5y\nasjHFrsSvywOFeyzxd9YYddAdzZO5x6acdWsBLvqK3f+GqeV9FAzaHUAPVXia7220EigF1DN999+\nnME/wfK06rULzrXFz3xo8a+3Avpcnr8W8DXn2nkNpRXm3mB6y9w3sGnIfUvoG3LfkJQhR8EHQQVB\nB4UKgvKCDsIUOh79jkff8ugNh1AYfcD71eJHXS39s1wR/2bxPxs34n8hSpHV1VdEb2C2lNGRTw3p\n0NLIxeLraHHZ0IquFr8R+ljrRsqaVXq93+Zqb33eVz+731LXFFAB9FyrZ8+kVx70WIvrvg9pgTBU\nlz8Mq8z1HPPa4r8OhJ0h8rJ6rjWXWvnWQOkEs9OonYVdS951hF0H+76usfhBKIOijEIeFMULZamv\nzUvDIWx5ii2HYDhEGEPAx5EcFCS1kn256Bvxvxg34n8pipCTQoKGRVMmSx4daWhQxxYvhXi2+PFs\n8RW9FXYtbNaIUwn1IOx5uG2u66jWxJdS9/XnB4AvdbRGzpX4al63A7GSXo3VzZdP/I/mWA3luWT9\neR1qysKZPt8VDHvRKENf6uX7VedeobYa7iz5q4Zw36Hvt8j9hny/JRZHeKcI74SAInhFKIqwCOGo\nWCbLGFvG1DAmzZAKYwz4NJFTXkMLoZI9X80fuLn6X4Qb8b8Q5w64KWqKN+TZkiaHOjVI2+H1R/b4\n6mLxt/mK8FyRft33hwIj9c/Oh/7nw72JSk4Ja1V+rAV5alz7Zzg+WZ1b0oUzOazl6mcplzPxa7J/\nsMd/XS9/VTOfe4GdJt87wkPD8nWP/skGvt6Tv94TUsPkFDOKySvmQTEVxbwopqNiGTQ+a3w2qy74\nHPA5k/OyuiHp43Kz+J+NG/G/FFcWP3uDzAZGh5wacC3e5Oc9/vOpvqx7/Bb25SqUxhXxU30gLGso\nPAJ+DZGdLf5MTQGQAJJA/BpWW5vjynWPju+8/8uDhvzywXMVhr/+8Rc4H+7ZqwY9vYNdA/sWUi/8\n/+2dXYhs2VXHf2ufjzpV1d1z7wxzMyZjnKDPMuDHg5kHRZDgS0QQQ0RUJPgQVNAHZV4GxAf1YSAI\neTBGSEQRFTT6YhRERUEzakZHkhhB2jgMc28yd25XV506Z38tH8453dX39tcdb1fXndo/2NQ+p6r7\nrN5d/7P3PnuvteJuhrtR0D5Vsbw1xtzaQW7tEW89gXNjGgxzm3E4N8xzw1wz5q1hfmhoD4WoelxQ\nojqiWuJg8BC/nFPqiUuRhP+wKN2DOdeNx3WpnQ9qESEPuCziGu3ixDcG22a0PqcNJa2OaGXUrWmL\nEJH+6ypHpRXFGsUR8aIElCCRKEonAz0yYzUq/uorgxu90sUEWJ20D5/tfeGP1uDz41+iIsTeRsQc\n2ytCyMEX4MuuuBJcod06vEBkRCsVrVQ0QzFDGbM0FTUZCzXMo+EwZBw6w6E1HLYZtukb+MTC4mk7\nCRL/H5LwH5bBX9/1/vr1SX/9mDXYtqZuLbM2crcxVG1J3k6g3WOnzVCbodZ0xWWo73YEajR4UdrC\nYXNPWzjawhMLR5Y7RoUnN/FBoa/UJXYZZ8Uq4lYLXUCQ4dH8GWH1YyaELMdnGT7L8Vm3Bj+cG2Lr\nI4oHWlFqUWYKE6uE5YiDw11m5YSDrGImIw5izoE1zBo4dMr8jUh9B9q7YGfgawh2WNEYhH/WToLE\noyAJ/2FR7YLEDYE68pP++tG0WFtTW8vMRipryG2399zbPSa2RF0OLkdd3vm9hxyNfckisWjQcUus\nGnTcEMct2bihGrdo7k8V/FAXr8hSMY1ilvFkXbopwolQ+veF1Q+FwRY5bVFiywKKklCUxKLElyUe\ng3rFe6X1ysIpI98Xp3gpmZdT5tmUQxkzjyWHtmDeGOYLWHilvtOV5m7EzsDVEG0XpqtvSFIQjasl\nCf9hGdx2nYX2pOhxnmgs1vWBKVwkdwZciXcTGrfH2I9RX6C+RF0JQz2UoAUigbyoycYL8t0F2e6C\nfLcm2xXK3YiUD07jV4+NVcw8YuZKNpe+HskQTNDuOcD9ofRX6t3OugwzKqGqCKMKqcbEUYWvKkLM\nCEulXSr5MpItlayO5K5LJBJCwSIbU8uEOlbUbkTd5NSLjHoGS680dzvRN3el7/GVYLVPFgLn7yRI\nPAqS8B+WVbdd0/vrx9hH7bFE8VjfuanmPoI3eF/S+Alz7xmFgPoKwggNIwgV6vt6rMjxjIsZ1XjG\neHdEdTNnfNNQPhkZ3XTklZ76/G44ZxoluxfJDgLZCLIcMpQsCJkVxOsDofRXix0J2SRHJyVhXGEn\nU2Q8RSdT3HiCCwXtLHaO8LOI0Dvc1124Ha9ZN6+PIxpb0TQlzTynnRmat6GNSjuL2APBzgbhS9/j\nrwr/rJ0EiUdBEv7DEocev/cCG7z4rIVlQ5SAjcsj/3QfDE0omccJ9wIUAYhjNI4h9K8r9ZE49ooJ\ne+MRfjdHnhRGTyvmaUd1q6GcdMPhsx7emzqSTwN51Yk+F8iDklklr7u99UeBdSse8AZqxgI7OWGn\nxO2MyadTZGeXuLOLn+7R+oLwjYAvIoFIcIFQ93UbcM7gYoG1JbYpcIsCWxXYkcFV4KLi6m5e72rw\ni4CrhWClc7Y5cxdBEv+jJAn/YTmK0EO3qO49ZLZXWUYkYtVCdPgYadQwjyVlhFJzspiBTiBOUZ0e\n1dEpxAljsTxVjPDjHNkTypuR6S1H9k1LRu8tGO8cB9M8TfxmHilG0oUPAIqg5FbIa6Eo+2X+IZT+\n+MHXcmoIexl2r6TZq8j2psjeHnH3CfzeDRpbYouAJdC6gK0jbR6wGmhtwDVCcBmhyfCLnFBk+Dwj\nFIaQQ1AlWAhWCVa60PcWgpVTsgGftpMg8ShIwn9YhqF+iN22ObsSK1oMEcXS+ac3KEYNhhKjOUYr\nhBJ0B9jpX3dPHO+IxRcFMjaUu5Hpk47wdIN574Lq/SWTPXeuedksUuRQiFIEpbCGoo4Us84HPlNO\n9viTkyXfFeyNnOZGSXFjTHZjCjd2iTdu4G48SduMqPEsXaCuA/VBoM4DSw3ULuBq7Zb+zLErrvb1\naED7/OHd3gE53rw07CV4sMEf4T8vMZCE/04YMkCc8kVdXYXuWE13BSe72/F99QlBMyY6ZkrFDmMW\njKmlYtmviefm/GCamYlE44lyesmIx0l3TinDGnwrFa3pSrNSlmbEUgIL+qKBRfQsQqD2AeceRqhJ\n1NdFEv7aGR5aebr5Qkv3b+gG7jFanF2yrB3zWeTgrlBWOXk+AiZMd8/fmmfmkfy2p7gdyN/y5PcC\nxaEnXwYK6zFOH8y5OayeeViGirfjDvf8hLftiLeXBfcWGQeHcHgQqdtA/UaguRNp70bcLBJqJdqz\neuzEJpKEv3aGMcGQ6aVlNS1WjA5nG5pe+KNKyPIcGOH9hPH0/M34plbybwSytwL5W4H8XiCfB7Jl\nILehE/6QMGfVHN+Z07gRM7/DzI6ZNRWzumA2N8wOhPlOZGE70Td3AvZuxM0UX2v/VP4KmitxJVwo\nfBF5Fvgs8AzdV+S3VPU3ReQl4GPAnf6jL6rqX1yZpe8a7u/xV7tdT4wBaxuWtWUxC+Q5QE7wI2wz\nYTQ+/18mjZLdC92S3r1AdhDJDiNZHchdn85reHg+9PSuN8WCbQvmdsqimTCvRywOC+aTjMUE5uPI\n0nWCb+/GXvhDj9/N3ROPB5fp8T3wC6r6qojsAP8iIn/Vv/eyqr58dea9Wxm6WNsfD245Do0BZ5dd\nj59HoIvv1zYj6vmUYlSc+5vFKtmhHm/cmUfMYSRbRoyNyGqcv+H+Mwi/ATcqWDYVy3rMcjRiWRUs\nR4ZlBcsq0HrBzSLuQLvXox4/DfUfJy4Uvqq+CbzZ1+ci8mXgff3bl8nLnDjBquJYqReAJcZ4NNSH\niPdC2+TU84rDe5AXF0Ta8IpZDuW+LbtW+2SeHN977tu2G46265bda1Fgy4y2AFtGXPD4uhN7qBW/\nSEP9x5GHmuOLyHPA88A/AS8AHxeRHwf+GfhFVT141Aa++xgm1fDgWNsQo+Jsl/jR+0jbCMt5TlFC\nURpMdr66JJx0yjnhrGPp9yHQPVZwnOKkY0445/gjhx3weSTErnePtttmO9S7of6VNFjiCri08Pth\n/h8DP9/3/J8EfkVVVUR+FXgZ+OkrsvNdxOpWVKFT3/F6WozgbCT4iG0UMYIxOcZkGFMgcsE8etjr\nEjnq3btM13r83qp7n9x/fOySq2K6uhFUIEo88t/naO1dTxwnHg8uJXwRyelE/7uq+jkAVf36ykc+\nBfz52b/hb1bqz/VlmzlnJ5pCDF3pEPpA2xtAeni32ez35WIu2+P/DvAlVf3EcEJEnunn/wA/DPzH\n2T/+vZe8TCKReOc8x8lO9W/P/ORllvM+CPwY8JqIfJHutv8i8FEReZ5u3LoP/Mw7NTeRSKyXyzzV\n/wdOH2umNftE4jHlgpisiUTi3UgSfiKxhSThJxJbSBJ+IrGFJOEnEltIEn4isYUk4ScSW0gSfiKx\nhSThJxJbSBJ+IrGFJOEnEltIEn4isYVcg/D313/Jh2L/ug24gP3rNuAC9q/bgHPYv24DLmB/bVdK\nwn+A/es24AL2r9uAC9i/bgPOYf+6DbiA/bVdKQ31E4ktJAk/kdhCRPVq46jJhdEhE4nEVaGqp4bA\nv3LhJxKJzSMN9ROJLSQJP5HYQtYmfBH5kIh8RUS+KiK/tK7rXhYR2ReRfxORL4rIFzbAnk+LyG0R\n+feVczdF5C9F5D9F5PMi8sSG2feSiLwuIv/alw9do33Pishfi8iXROQ1Efm5/vxGtOEp9v1sf34t\nbbiWOb6IGOCrwPcDbwCvAB9R1a9c+cUviYj8N/Adqvr2ddsCICIvAHPgs6r67f25XwfeUtXf6G+e\nN1X1lzfIvpeAw01IpCoizwDPrCZ7BT4M/BQb0Ibn2PejrKEN19XjfzfwX6r6P6rqgD+g+yM3CWGD\npj6q+vfA/TehDwOf6eufAX5orUatcIZ9sCGJVFX1TVV9ta/PgS8Dz7IhbXiGfWtLRruuL/r7gP9d\nOX6d4z9yU1Dg8yLyioh87LqNOYNbqnobjrIYP33N9pzGx0XkVRH57euciqyykuz1H4H3bFob3peM\nFtbQhusS/ml3sE1bR/weVf1O4AfpGv6F6zboMeSTwLeq6vN0qdU3Ych/ItkrG/a9O8W+tbThuoT/\nOvD+leNn6eb6G8OQB7BPBvondNOTTeO2iLwHjuaId67ZnhOo6tf1+KHRp4Dvuk57Tkv2yga14VnJ\naNfRhusS/ivAt4nIt4hICXwE+LM1XftCRGTS33kRkSnwA5ybBHRtrCaxhq7NfrKv/wTwuft/YM2c\nsK8X0sAFiVTXwgPJXtmsNjw1Ge3K+1fWhmvbudcvS3yC7mbzaVX9tbVc+BKIyAfoenmlyyf4e9dt\nn4j8Pl2a4aeA28BLwJ8CfwR8M/A14EdU9d4G2fd9dHPVo0Sqw3z6Guz7IPB3wGsc5yV/EfgC8Idc\ncxueY99HWUMbpi27icQWsjHLV4lEYn0k4ScSW0gSfiKxhSThJxJbSBJ+IrGFJOEnEltIEn4isYUk\n4ScSW8j/AfkWSZR2cVBlAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "image = mnist.train.images[5, :]\n", + "image = np.reshape(image,[28,28])\n", + "plt.imshow(image)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Logistic Regression" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "This algorithm has nothing to do with the canonical linear regression, but it is an algorithm that allows us to solve problems of classification(supervised learning). \n", + "\n", + "In fact, to estimate the dependent variable, now we make use of the so-called **logistic function** or **sigmoid**. \n", + "\n", + "It is precisely because of this feature we call this algorithm logistic regression.\n", + "\n", + "![](imgs/sigmoid.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### Hands On - Logistic Regression" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# Parameters\n", + "learning_rate = 0.01\n", + "training_epochs = 25\n", + "batch_size = 100\n", + "display_step = 1" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# tf Graph Input\n", + "x = tf.placeholder(\"float\", [None, 784]) # mnist data image of shape 28*28=784\n", + "y = tf.placeholder(\"float\", [None, 10]) # 0-9 digits recognition => 10 classes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## The Model" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "# Set model weights\n", + "W = tf.Variable(tf.zeros([784, 10]))\n", + "b = tf.Variable(tf.zeros([10]))" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# Construct model\n", + "activation = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ + "# Minimize error using cross entropy\n", + "cross_entropy = y*tf.log(activation)\n", + "cost = tf.reduce_mean(-tf.reduce_sum(cross_entropy,reduction_indices=1))" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# Set the Optimizer\n", + "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# Initializing the variables\n", + "init = tf.global_variables_initializer()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Learning" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "def training_phase():\n", + " avg_set, epoch_set = [], []\n", + " # Training cycle\n", + " for epoch in range(training_epochs):\n", + " avg_cost = 0.\n", + " total_batch = int(mnist.train.num_examples/batch_size)\n", + " # Loop over all batches\n", + " for i in range(total_batch):\n", + " batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n", + " # Fit training using batch data\n", + " sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})\n", + " # Compute average loss\n", + " avg_cost += sess.run(cost, feed_dict={x: batch_xs, y: batch_ys}) / total_batch\n", + " \n", + " # Display logs per epoch step\n", + " if epoch % display_step == 0:\n", + " print(\"Epoch:\", '%04d' % (epoch+1), \"cost=\", \"{:.9f}\".format(avg_cost))\n", + " \n", + " avg_set.append(avg_cost)\n", + " epoch_set.append(epoch+1)\n", + " return avg_set, epoch_set" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Prediction" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "def testing_phase():\n", + " # Test model\n", + " correct_prediction = tf.equal(tf.argmax(activation, 1), tf.argmax(y, 1))\n", + " # Calculate accuracy\n", + " accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n", + " print(\"Model accuracy:\", accuracy.eval({x: mnist.test.images, y: mnist.test.labels}))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## TF Session" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch: 0001 cost= 1.174406662\n", + "Epoch: 0002 cost= 0.662001156\n", + "Epoch: 0003 cost= 0.550464896\n", + "Epoch: 0004 cost= 0.496610615\n", + "Epoch: 0005 cost= 0.463699823\n", + "Epoch: 0006 cost= 0.440842383\n", + "Epoch: 0007 cost= 0.423908034\n", + "Epoch: 0008 cost= 0.410584232\n", + "Epoch: 0009 cost= 0.399826936\n", + "Epoch: 0010 cost= 0.390904010\n", + "Epoch: 0011 cost= 0.383289029\n", + "Epoch: 0012 cost= 0.376714404\n", + "Epoch: 0013 cost= 0.370987393\n", + "Epoch: 0014 cost= 0.365903865\n", + "Epoch: 0015 cost= 0.361352632\n", + "Epoch: 0016 cost= 0.357227456\n", + "Epoch: 0017 cost= 0.353577220\n", + "Epoch: 0018 cost= 0.350052592\n", + "Epoch: 0019 cost= 0.347016569\n", + "Epoch: 0020 cost= 0.344181476\n", + "Epoch: 0021 cost= 0.341469029\n", + "Epoch: 0022 cost= 0.338967005\n", + "Epoch: 0023 cost= 0.336669330\n", + "Epoch: 0024 cost= 0.334486312\n", + "Epoch: 0025 cost= 0.332483579\n", + "Training phase finished\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt0VPXZ9vHvHU4aaAIhghBiCIFXLYpH0FbRIFV5oEu6\nFFo08AIKta1il9QWWxoTGpVXautqbfu0VRQpINanS6vVKojGQxWF+iAVxQOHCAlYDCAHC4Hkfv/I\nZMgOOU4yMzlcn7WyVmZmz557diZzze+0x9wdERGRKgnxLkBERFoXBYOIiAQoGEREJEDBICIiAQoG\nEREJUDCIiEhAVIPBzBaa2admtr6O268zs3fMbJ2ZvWZmZ0azHhERaVi0WwwPA1fWc/tm4BJ3Pxu4\nE3ggyvWIiEgDOkdz5+7+mpll1HP76moXVwNp0axHREQa1prGGGYAf493ESIiHV1UWwyNZWajgOnA\nxfGuRUSko4t7MJjZMOCPwBh331PPdjqpk4hIBNzdmrJ9LLqSLPRz/A1mpwB/Aaa4+6aGduTu+nEn\nLy8v7jW0lh8dCx0LHYv6fyIR1RaDmS0DsoHeZvYJkAd0Bdzd/wjkAinA78zMgCPuPiKaNYmISP2i\nPSvpugZunwnMjMZjb9lSRG7uIoqLK0hLS6CgYBqZmXVOkBIRkZC4jzFEw5YtRVx++f1s2jQP6A4c\nZPXqPFaunNUuwiE7OzveJbQaOhbH6Fgco2PRPBZpH1SsmZk3ttbJk+exdOltVIZClYPk5NzLkiV5\nUalPRKQ1MjO8iYPP7bLFUFxcQTAUALpTUlIRj3LiYuDAgRQVFcW7DBGJkYyMDLZu3doi+2qXwZCW\nlgAcpGaLoX//1rSeL7qKiooinpEgIm1P5fydltEu3ykLCqaRlZVHZTgAHCQrK4+Cgmlxq0lEpK1o\nl2MMcGxWUklJBf37d7xZSaF+xXiXISIxUtf/fCRjDO02GDo6BYNIx9KSwdAuu5Kk41m2bBljxoyJ\n6L5nnHEGr7zySgtX1PqNHTuWP/3pT/Euo0FXXnkljz76aItv25KmTJnCz372s5g/brSoxdBOteYW\nQ2ZmJgsXLuSyyy6L+WNPnz6d9PT0Zv8TFxUVkZmZSY8ePQBITU3lxhtvZM6cOS1RZps1duxYXn31\nVcyMQ4cOYWZ069YNgMmTJ/O73/0uzhVGx5QpUxgyZAh33HFH3GpoyRZDu5yVJHVriRXhWlVeycz4\n/PPPMTP++c9/cumll3L++eczevToFn2c8vJyOnXq1KL7jJZnn302/HtjQrgtPbeORF1JHUjVivCl\nS2+jsLByEeDll9/Pli2NX+/QEvuozwMPPMCQIUNITU3lG9/4Bjt27AjftmLFCk477TR69erFTTfd\nRHZ2Ng899BAAjzzyCCNHjgxve+utt9K3b1969uzJ2WefzXvvvccDDzzA0qVLWbBgAUlJSYwfPx6o\nbMG8+OKLAFRUVHD33XczePBgkpOTGT58OMXFxXXWW/UJ7bzzzmPo0KGsW7cufNuOHTuYMGECffr0\nISsri/vvvz9826FDh5g6dSopKSkMHTqUn//856Snp4dvz8zMZMGCBZx11ln06NGDioqKeve3Zs0a\nhg8fTnJyMv369eO2224D4PDhw0yZMoXU1FR69erFBRdcwK5duwAYNWpU+Pi5O3feeScDBw7k5JNP\nZtq0aezbtw+obB0lJCSwePFiMjIy6NOnD3fffXej/6bVrVq1iszMTObPn0+/fv349re/ze7duxk3\nbhx9+vShd+/eXHXVVZSUlITvM3LkSBYvXgzAwoULyc7OZvbs2fTq1YvBgwezcuXKiLbdvHkzI0eO\nJDk5mTFjxvC9732P6dOn11t3QUEBqampZGVl8dhjjwW2KS0tZezYsSQlJXHRRRcF1hHNmjWL9PR0\nevbsyQUXXMAbb7wRvu3NN9/kvPPOC//tqrc6//GPf/CVr3yFXr16ce655/Lqq69GctibLt5n/mvC\nGQJdGq+245WTk+9wwMGr/RzwnJz8Ru+3JfYxcOBAX7Vq1XHXr1q1ylNTU33dunVeVlbms2bN8ksu\nucTd3Xft2uVJSUn+5JNPenl5uf/qV7/yrl27+sKFC93dfdGiRT5y5Eh3d3/++ef9/PPP93379rm7\n+8aNG33nzp3u7j5t2jTPzc2ts54FCxb4sGHD/KOPPnJ39/Xr1/vu3buPq3Xr1q2ekJDgR48edXf3\nN954w7t37+5PPvmku7tXVFT4eeed53feeacfPXrUt2zZ4llZWb5ixQp3d58zZ45nZ2f7559/7sXF\nxT5s2DBPT08P1HTOOed4cXGxHzp0qMH9feUrX/ElS5a4u/vBgwf9zTffdHf3P/zhD37VVVeF9/H2\n22/7/v373d09Ozs7fPwWLlzoQ4YM8a1bt/rBgwf96quv9ilTpoSfq5n5t7/9bT98+LC/88473q1b\nN9+4cWO9f+fajvULL7zgnTt39p/+9Kd+5MgRP3TokO/atcuffPJJP3z4sO/fv9+vueYanzhxYvg+\nF198sT/yyCPu7v7ggw96165dfdGiRV5RUeH3339/4Lg1ZdsRI0b4j3/8Yz9y5Ii/8sor/qUvfcmn\nT59e63OpqnvOnDleVlbmL774oicmJvqmTZvc3X3y5Ml+0kkn+dtvv+1Hjx71b33rW+Hj5+6+ZMkS\n37t3r5eXl/uCBQs8LS3Ny8rK3N19+PDhvnz5cnd3P3DggL/11lvu7r5t2zbv3bu3r1y50t0rX9ep\nqam1vh7da/+fr3Z9k95v1WLoQFpiRXg0V5UvW7aMG264gbPOOosuXbowf/58Vq9ezSeffMLf//53\nzjjjDMaPH09CQgK33HILffv2rXU/Xbp0Yf/+/bz33nu4O6eeemqd29a0cOFC7rrrLgYPHgzAmWee\nSa9evWrd1t056aSTSExM5KKLLuJ73/teuBWyZs0aPvvsM+bOnUunTp0YOHAgM2bMYPny5QA8/vjj\nzJ07l6SkJPr3788tt9xy3P6///3v079/f7p169bg/rp06cLHH39MaWkpiYmJjBgxInx9aWkpH374\nIWbGOeecEx4XqXnsZ8+eTUZGBomJicyfP5/ly5dTUVH5dzUz8vPz6dq1K8OGDeOss87inXfeadQx\nralLly7k5+fTuXNnunXrRmpqKuPHj6dr16706NGD22+/nZdffrnO+2dlZTF16lTMjKlTp7J9+3Z2\n797dqG2Li4vZvXs3mzdvZv369eTl5dG5c2dGjhzJuHHj6q27U6dO5Ofn06VLF0aNGsWYMWN4/PHH\nw7dPmDCBc845h06dOpGTkxNoPebk5JCcnExCQgK33XYb+/bt4+OPPwaga9eufPTRR+zevZvu3bsz\nfPhwABYvXsz48eP52te+BsAVV1zBWWedxXPPPde4A90MCoYO5NiK8OqatiK8JfZRl5KSEjIyjo1V\ndO/enZSUFIqLiykpKQl0tQAMGDCg1v2MGjWKm2++mZtuuomTTz6Z73znOxw4cKBRNWzbto1BgwY1\nalszo7S0lIMHD3LvvfdSWFjI0aNHgcrul+LiYlJSUkhJSaFXr17Mnz+ff//73+HnWr3+ms+t5vNr\naH8PPfQQH3zwAaeddhoXXHABzzzzDFA5KHrllVcyadIkBgwYwJw5cygvLz/usWoe+4yMDI4ePcqn\nn34avq56uCYmJjb6mNbUt2/fwLjCF198wYwZM8jIyKBnz56MHj2azz77rM77n3zyyYE6gDprqbmt\nu3PgwAF27NhB7969wwPjUPvfoLrevXtzwgknhC9nZGQEurxqPlb1mhYsWMDpp59Or169SElJ4Ysv\nvgg/x4cffpgNGzZw6qmncuGFF/L3v1d+w3FRURHLli0L/M3ffPPNwGNGi4KhA2mJFeHRXFXev3//\nQL/swYMHKS0tJS0tjX79+rFt27bA9tu3b69zXzfffDNr165lw4YNfPDBB/z85z8HGj5tQHp6Ops2\nNfidUWHujplx66230q1bt/Csm/T0dAYNGsTu3bvZvXs3e/bs4fPPP+fpp58OP9fq9X/yySfH7bt6\nrQ3tLysri2XLlrFr1y5+9KMfMWHCBP7zn//QuXNncnNz2bBhA6+//jp/+9vfwn3w1dU89kVFRXTp\n0qXRLa2mqPk3WLBgAUVFRaxdu5a9e/eGx3uiqV+/fpSWllJWVha+rubrq6bS0lIOHz4cvvzJJ5/Q\nv3//Bh/rpZde4r777uOJJ55gz5497Nmzh+7du4fHp4YMGcKjjz7Krl27mD17Ntdccw1lZWWkp6dz\n/fXXB/7m+/fv5wc/+EGEz7rxFAwdSGZmBitXziIn515GjcojJ+feJp+KvCX2AVBWVsbhw4fDP+Xl\n5Vx33XU8/PDDrF+/nsOHD/OTn/yECy+8kFNOOYVx48bx7rvv8tRTT1FeXs5vfvObwKfZ6tauXctb\nb73F0aNHOfHEEznhhBPCn1D79u3L5s2b66xrxowZ5Obmhpv5//rXv9izp/ZvnK36x65y++23c889\n91BWVsaIESNISkpiwYIFHDp0iPLycjZs2MDatWsBmDhxIvPnz2fv3r0UFxfz29/+tt7j1dD+li5d\nGv4EmpycjJnRqVMnCgsLeffdd6moqKBHjx506dKFzp2Pn4x47bXXct9997F161YOHDjA3LlzmTRp\nEgkJCbU+15a0f/9+EhMTSU5OprS0lHnz5kXtsaoMGjSIM888k3nz5nHkyBFee+21cCurLuXl5eTn\n53PkyBEKCwt57rnnmDhxYoOPdeDAAbp06UJKSgplZWXk5eXxxRdfhG9fsmQJpaWlACQlJZGQkEBC\nQgJTpkzhiSee4IUXXqCiooJDhw5RWFjIzp07m/fkG0HB0MFkZmawZEkeL744jyVL8iKaZtoS+xg3\nbhyJiYmceOKJJCYmMm/ePC677DIKCgq4+uqrSUtLY8uWLeE+9N69e/P444/zwx/+kNTUVDZu3Mj5\n558f6Aqosm/fPmbOnElKSgqZmZmkpqaGZ+nccMMNbNiwgZSUFK6++mog+Al29uzZfPOb3+SKK64g\nOTmZGTNm8J///KfW51Dzk++4ceNISUnhgQceICEhgaeffpp169aRmZlJnz59mDlzZnimzx133EFa\nWhqZmZlcccUVTJw4MfBcau67of0999xzDB06lKSkJG699VYee+wxunbtys6dO5kwYQLJyckMHTqU\nUaNGkZOTc9xjXH/99UyZMoVLLrmErKwsEhMT+fWvf11nPY05YVtjT+o2e/Zs9u7dS+/evbn44ouP\n6+tvaD/Vb2/Kto8++igvv/wyqampFBQUMGnSpFpfT1XS09Pp3r07/fr1Y/r06SxcuDDc7Vjf444d\nO5bRo0czZMgQBg0aRM+ePenXr1/49meffZbTTz+d5ORkfvSjH/HnP/+Zzp07k5GRwRNPPEFBQQEn\nnXQSAwcO5Je//GV43CeatMCtnWrNC9xagrszYMAAli1bxqWXXhrvcprt97//PY899hgvvfRSvEvp\nsKoGj+fOnXvcbatWrWLmzJn1tjbjTafEkA5pxYoVfP755xw+fJi77roLgAsvvDDOVUVm586dvP76\n67g7H3zwAb/4xS/CLRiJjTVr1rB161bcnWeffZZnnnkmPKuso9PKZ2kz3njjDa677jqOHDnCl7/8\nZf7617/W2/RvzcrKyrjxxhvZunUrPXv25Nprr+W73/1uvMvqUEpKSrjmmmvYs2cPAwYM4MEHH+SM\nM86Id1mtgrqS2qn23pUkIkHqShIRkahRMIiISICCQUREAjT43E5lZGS06JeDi0jrVv2UJs2lwWcR\nkXZMg88iItJsCgYREQlQMIiISICCQUREAhQMIiISoGAQEZEABYOIiAQoGEREJEDBICIiAQoGEREJ\nUDCIiEiAgkFERAIUDCIiEqBgEBGRAAWDiIgERDUYzGyhmX1qZuvr2ebXZvaRma0zs7OjWY+IiDQs\n2i2Gh4Er67rRzP4LyHL3IcCNwO+jXI+IiDQgqsHg7q8Be+rZZDywOLTtm0CymfWNZk0iIlK/eI8x\npAHbql0uDl0nIiJx0jnOj1/b95DW+cXO+fn54d+zs7PJzs5u+YpERNqwwsJCCgsLm7UPc6/zfbhF\nmFkG8LS7D6vltt8DL7n7Y6HLG4FL3f3TWrb1aNcqItLemBnuXtuH8DrFoivJqL1lAPAU8H8BzOxC\nYG9toSAiIrET1a4kM1sGZAO9zewTIA/oCri7/9HdnzWzsWb2MXAQmB7NekREpGFR70pqKepKEhFp\nutbalSQiIm2IgkFERAIUDCIiEqBgEBGRAAWDiIgEKBhERCRAwSAiIgEKBhERCVAwiIhIgIJBREQC\nFAwiIhKgYBARkQAFg4iIBCgYREQkQMEgIiIBCgYREQlQMIiISICCQUREAhQMIiISoGAQEZEABYOI\niAQoGEREJEDBICIiAQoGEREJUDCIiEiAgkFERAIUDCIiEqBgEBGRAAWDiIgEKBhERCRAwSAiIgEK\nBhERCVAwiIhIgIJBREQCFAwiIhKgYBARkQAFg4iIBCgYREQkQMEgIiIBCgYREQmIejCY2Rgz22hm\nH5rZnFpuTzezF83sbTNbZ2b/Fe2aRESkbubu0du5WQLwITAaKAHWAJPcfWO1bf4AvO3ufzCz04Fn\n3T2zln15NGsVEWmPzAx3t6bcJ9othhHAR+5e5O5HgOXA+BrbVABJod97AsVRrklEROrROcr7TwO2\nVbu8ncqwqG4esMLMbgESga9FuSYREalHtFsMtTVfavYHXQs87O7pwDhgSZRrEhGRekS7xbAdOKXa\n5QFUjjVUdwNwJYC7rzazE8ws1d0/q7mz/Pz88O/Z2dlkZ2e3dL0iIm1aYWEhhYWFzdpHtAefOwEf\nUDn4vAN4C7jW3d+vts0zwJ/d/ZHQ4PNKdx9Qy740+Cwi0kStbvDZ3cuBm4EVwAZgubu/b2bzzOzr\noc1uA2aa2TpgKTA1mjWJiEj9otpiaElqMYiINF3UWgxmNrEx14mISNvXqBaDmb3t7uc2dF00qcUg\nItJ0kbQY6p2VFDo9xVggzcx+Xe2mJOBo00sUEZHWrqHpqiXAWuAq4J/Vrt8P3BqtokREJH4a25XU\nJXRKC8ysF5Du7uujXVyNGtSVJCLSRNGcrrrSzJLMLAV4G3jAzO5rcoUiItLqNTYYkt19H3A1sNjd\nL6By0ZqIiLQzjQ2GzmbWD/gm8Lco1iMiInHW2GD4GfA8sMnd15jZIOCj6JUlIiLxopXPIiLtWDRX\nPg8wsyfM7N9m9qmZ/cXMjjvRnYiItH2N7Up6GHgK6E/ll+88HbpORETamcauY1jn7mc3dF00qStJ\nRKTpormO4TMzm2xmnUI/k4HSppcoIiKtXWOD4Xoqp6rupPILdyYA06NVlIiIxE9jv9qzAJjq7nsA\nQiug76UyMEREpB1pbIthWFUoALj7buCc6JQkIiLx1NhgSAidPA8Itxga29oQEZE2pLFv7r8AXjez\n/wGcyvGGu6JWlYiIxE2jVz6b2ZeBywADVrn7e9EsrJbH13RVEZEmimS6qk6JISLSjkVzHYOIiHQQ\nCgYREQlQMIiISICCQUREAhQMIiISoGAQEZEArV6uZsuWInJzF1FcXEFaWgIFBdPIzMyId1kiIjGl\ndQwhW7YUcfnl97Np0zygO3CQrKw8Vq6cpXAQkTZL6xiaITd3UbVQAOjOpk3zyM1dFMeqRERiT8EQ\nUlxcwbFQqNKdkpKKeJQjIhI3CoaQtLQE4GCNaw/Sv78OkYh0LHrXCykomEZWVh7HwqFyjKGgYFrc\nahIRiQcNPldTNSuppKSC/v01K0lE2j6dXVVERAI0K0lERJpNwSAiIgEKBhERCVAwiIhIgIJBREQC\nFAwiIhIQ9WAwszFmttHMPjSzOXVs800z22Bm/zKzJdGuSURE6hbVdQxmlgB8CIwGSoA1wCR331ht\nm8HAY8Aod99nZqnu/lkt+9I6BhGRJmqN6xhGAB+5e5G7HwGWA+NrbDMT+K277wOoLRRERCR2oh0M\nacC2ape3h66r7v8Ap5rZa2b2upldGeWaRESkHtH+Brfami81+4M6A4OBS4BTgFfNbGhVC6K6/Pz8\n8O/Z2dlkZ2e3WKEiIu1BYWEhhYWFzdpHtMcYLgTy3X1M6PLtgLv7PdW2+W/gDXdfHLr8AjDH3f9Z\nY18aYxARaaLWOMawBhhsZhlm1hWYBDxVY5sngcsAzCwVGAJsjnJdIiJSh6gGg7uXAzcDK4ANwHJ3\nf9/M5pnZ10PbPA+UmtkGYBVwm7vviWZdIiJSN512W0SkHWuNXUkiItLGKBhERCRAwSAiIgEKBhER\nCVAwiIhIQLRXPrd7W7YUkZu7iOLiCtLSEigomEZmZka8yxIRiZimqzbDli1FXH75/WzaNA/oDhwk\nKyuPlStnKRxEpFXQdNUYy81dVC0UALqzadM8cnMXxbEqEZHmUTA0Q3FxBcdCoUp3Skoq4lGOiEiL\nUDA0Q1paAnCwxrUH6d9fh1VE2i69gzVDQcE0srLyOBYOlWMMBQXT4laTiEhzafC5mapmJZWUVNC/\nv2YliUjrEsngs4JBRKQd06wkERFpNgWDiIgEKBhERCRAwSAiIgEKBhERCdBJ9OJEJ98TkdZK01Xj\nQCffE5FY0XTVNkIn3xOR1kzBEAc6+Z6ItGYKhjjQyfdEpDXTO1Ec6OR7ItKaafA5TnTyPRGJBZ1E\nT0REAiIJBq1jaEO09kFEYkEthjZCax9EJBJax9COae2DiMSKgqGN0NoHEYkVBUMbobUPIhIreldp\nI7T2QURiRYPPbUikax80m0mk49I6BjmOZjOJdGyalSTH0WwmEWkqBUM7p9lMItJUWvnczh2bzVQ9\nHBqezaRxCZGOS2MM7VwkYwwalxBpPzT4LLVq6mymyZPnsXTpbdRsZeTk3MuSJXnRLldEWpBOoie1\nyszMaNIbeqTjEup+EmkfFAxynEjGJWrrflq9Wt1PIm1R1GclmdkYM9toZh+a2Zx6tptgZhVmdm60\na5L6RbLKWtNiRdqPqLYYzCwB+A0wGigB1pjZX919Y43tegCzgNXRrEcaJzMzg5UrZ5Gbe2+1cYn6\nP/k3Z1qsuqBEWpdodyWNAD5y9yIAM1sOjAc21tiuALgH+GGU65FGauq4RHOmxaoLSqR1iXZXUhqw\nrdrl7aHrwszsbGCAuz8b5VokiiI9yV+kXVBbthQxefI8Ro3KY/LkeWzZUtSc8kWkmmi3GGqbIhWe\nc2pmBtwHTG3gPgDk5+eHf8/OziY7O7vZBUrLiKT7CSLrglIrQ6RuhYWFFBYWNmsfUV3HYGYXAvnu\nPiZ0+XbA3f2e0OUk4GPgAJWBcDJQClzl7m/X2JfWMbRDkayZaM46C41nSEfTGtcxrAEGm1kGsAOY\nBFxbdaO77wP6VF02s5eA2e7+v1GuS1qJgoJprF6dd9wq64KCWXXepznrLNTSEGlYVMcY3L0cuBlY\nAWwAlrv7+2Y2z8y+XttdqKcrSdqfqi6onJx7GTUqj5ycext8o4702+w0niHSSO7eJn4qSxVx37x5\nq2dl/cDhgIM7HPCsrB/45s1b671fdvYdoe2DP6NG3dHij7V581bPycn37Ow7PCcnv8HtRaIl9N7Z\npPdbrXyWNifSge5IptTW3cqoezyjOV1WGgORVqGpSRKvH9RikGaK5NN/JK2MnJz8ao/h4cfKyclv\n8fpEGkIELQZ9UY90GLEaz4h0cFxjINJaqCtJOpSmruiOZNZUpKvAY7mmQ11WUq+mNjHi9YO6kiRO\nqgaSR41q3EBypF1CkXRBRXKf5nRZRTKoroH4+CKCrqS4v+E3ulAFg7QhTQ2Tqvu0tzGQWIeQHE/B\nINLGNTVQInmTjyRMIn2sWA/EK0yOF0kwaIxBpBVpb2MgLT8Q3/LThCMdb2nX4zRNTZJ4/aAWg0it\n2tsYiHvsusia0zKJZYumOS0h1JUkIo0RqzGQWIZQLMdb2lIIKRhEJKoiDZRYhFAsx1vaVgg1PRg0\nxiAijdbUMZDm3Keppz2J5XhLJPeLdLwlkvsdP0bTNAoGEWmVmhoosQqTSO8X/xBqvKh+UU9L0hf1\niEg0VM0uOhYmTZuV1Nj71TZrKiurcbOmmnq/4JdZNf2LehQMIiIxEp8Q6qFgEBGRY2GydGl+k4NB\nZ1dtg5r7Rd/tiY7FMToWx+hYRDboX0XB0AbpRX+MjsUxOhbH6Fg0j4JBREQCFAwiIhLQpgaf412D\niEhb1G5nJYmISGyoK0lERAIUDCIiEtAmgsHMxpjZRjP70MzmxLueeDKzrWb2jpn9r5m9Fe96YsnM\nFprZp2a2vtp1vcxshZl9YGbPm1lyPGuMlTqORZ6ZbTezt0M/Y+JZYyyY2QAze9HM3jOzf5nZLaHr\nO9zropZjMSt0fZNfF61+jMHMEoAPgdFACbAGmOTuG+NaWJyY2WbgPHffE+9aYs3MLgYOAIvdfVjo\nunuAUndfEPrQ0Mvdb49nnbFQx7HIA/a7+y/jWlwMmdnJwMnuvs7MegD/BMYD0+lgr4t6jsW3aOLr\noi20GEYAH7l7kbsfAZZT+WQ7KqNt/N1anLu/BtQMxPHAI6HfHwG+EdOi4qSOYwGVr48Ow913uvu6\n0O8HgPeBAXTA10UdxyItdHO7OyVGGrCt2uXtHHuyHZEDz5vZGjObGe9iWoE+7v4pVP5jACfFuZ54\nu8nM1pnZgx2h+6Q6MxsInA2sBvp25NdFtWPxZuiqJr0u2kIw1JZ0rbv/K7q+6u7nA2Op/GNfHO+C\npNX4HZDl7mcDO4GO1KXUA/gf4PuhT8sd9j2ilmPR5NdFWwiG7cAp1S4PoHKsoUMKffrB3XcBT1DZ\n1daRfWpmfSHcx/rvONcTN+6+q9opiB8Ahseznlgxs85UvhH+yd3/Grq6Q74uajsWkbwu2kIwrAEG\nm1mGmXUFJgFPxbmmuDCzxNCnAcysO3AF8G58q4o5I9iKfAqYFvp9KvDXmndoxwLHIvQGWOVqOs5r\n4yHgPXf/VbXrOurr4rhjEcnrotXPSoLK6arAr6gMsoXu/v/iXFJcmFkmla0Ep/JrWZd2pGNhZsuA\nbKA38CkiyQUkAAACLklEQVSQBzwJPA6kA58AE919b7xqjJU6jsUoKvuVK4CtwI1V/eztlZldBLwC\n/IvK/wsHfgK8BfyZDvS6qOdYXEcTXxdtIhhERCR22kJXkoiIxJCCQUREAhQMIiISoGAQEZEABYOI\niAQoGEREJEDBIBIDZnapmT0d7zpEGkPBIBI7WjQkbYKCQaQaM8sxszdDX2jy32aWYGb7zeyXZvau\nma00s96hbc82szdCZ638S9VZK80sK7TdOjNbG1qxDvAlM3vczN43sz/F7UmKNEDBIBJiZqdR+aUm\nX3X3c6k8hUAOkAi85e5nUHnKgbzQXR4Bfhg6a+W71a5fCtwfuv6rwI7Q9WcDtwBfBrLM7KvRf1Yi\nTdc53gWItCKjgXOBNWZmwAlUnoeogsrz7gAsAf5iZklAcugLc6AyJP4cOslhmrs/BeDuZQCVu+Mt\nd98RurwOGAi8HoPnJdIkCgaRYwx4xN3nBq40y62xnVfbvrZ91OVwtd/L0f+ftFLqShI5ZhUwwcxO\ngvAXyp8CdAImhLbJAV5z933A7tAZLQGmAC+7+35gm5mND+2jq5mdGNNnIdJM+sQiEuLu75vZT4EV\nZpYAlAE3AweBEaGWw6dUjkNA5Xn+/xB6499M5RfQQ2VI/NHMfhbax8TaHi56z0SkeXTabZEGmNl+\nd/9SvOsQiRV1JYk0TJ+epENRi0FERALUYhARkQAFg4iIBCgYREQkQMEgIiIBCgYREQlQMIiISMD/\nB7TXZhORdklGAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model accuracy: 0.9137\n" + ] + } + ], + "source": [ + "# Launch the graph\n", + "with tf.Session() as sess:\n", + " # Plug TensorBoard Visualisation\n", + " merged = tf.summary.merge_all() \n", + " writer = tf.summary.FileWriter(\"/tmp/logistic_logs\", session.graph)\n", + " \n", + " sess.run(init)\n", + " avg_set, epoch_set = training_phase()\n", + " print(\"Training phase finished\")\n", + " \n", + " #plotting\n", + " plt.plot(epoch_set,avg_set, 'o', label='Logistic Regression Training phase')\n", + " plt.ylabel('cost')\n", + " plt.xlabel('epoch')\n", + " plt.legend()\n", + " plt.show()\n", + " \n", + " #Testing\n", + " testing_phase()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "[Open TensorBoard](http://localhost:6006)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "# Why Tensorflow ?" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "On a typical system, there are multiple computing devices. \n", + "\n", + "In TensorFlow, the supported device types are **CPU** and **GPU**. \n", + "\n", + "They are represented as strings. For example:\n", + "\n", + "* `\"/cpu:0\"`: The CPU of your machine.\n", + "* `\"/gpu:0\"`: The GPU of your machine, if you have one.\n", + "* `\"/gpu:1\"`: The second GPU of your machine, etc." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "If a TensorFlow operation has both **CPU** and **GPU** implementations, the GPU devices will be given priority when the operation is assigned to a device. \n", + "\n", + "For example, `matmul` has both CPU and GPU kernels. On a system with devices `cpu:0` and `gpu:0`, `gpu:0` will be selected to run `matmul`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Example 1. Logging Device Placement\n", + "\n", + "`tf.Session(config=tf.ConfigProto(log_device_placement=True))`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "```python\n", + "# Creates a graph.\n", + "a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')\n", + "b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')\n", + "c = tf.matmul(a, b)\n", + "# Creates a session with log_device_placement set to True.\n", + "sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\n", + "# Runs the op.\n", + "print(sess.run(c))\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "```\n", + "Device mapping:\n", + "/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 760, pci bus\n", + "id: 0000:05:00.0\n", + "b: /job:localhost/replica:0/task:0/gpu:0\n", + "a: /job:localhost/replica:0/task:0/gpu:0\n", + "MatMul: /job:localhost/replica:0/task:0/gpu:0\n", + "[[ 22. 28.]\n", + " [ 49. 64.]]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "## Using Multiple GPUs" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "```python\n", + "# Creates a graph.\n", + "c = []\n", + "for d in ['/gpu:0', '/gpu:1']:\n", + " with tf.device(d):\n", + " a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])\n", + " b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])\n", + " c.append(tf.matmul(a, b))\n", + "with tf.device('/cpu:0'):\n", + " sum = tf.add_n(c)\n", + "# Creates a session with log_device_placement set to True.\n", + "sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\n", + "# Runs the op.\n", + "print sess.run(sum)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true, + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "```\n", + "Device mapping:\n", + "/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 760, pci bus\n", + "id: 0000:02:00.0\n", + "/job:localhost/replica:0/task:0/gpu:1 -> device: 1, name: GeForce GTX 760, pci bus\n", + "id: 0000:03:00.0\n", + "Const_3: /job:localhost/replica:0/task:0/gpu:0\n", + "Const_2: /job:localhost/replica:0/task:0/gpu:0\n", + "MatMul_1: /job:localhost/replica:0/task:0/gpu:0\n", + "Const_1: /job:localhost/replica:0/task:0/gpu:1\n", + "Const: /job:localhost/replica:0/task:0/gpu:1\n", + "MatMul: /job:localhost/replica:0/task:0/gpu:1\n", + "AddN: /job:localhost/replica:0/task:0/cpu:0\n", + "[[ 44. 56.]\n", + " [ 98. 128.]]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## More on Tensorflow\n", + "\n", + "[Official Documentation](https://www.tensorflow.org/versions/r0.10/get_started/)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.2" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/1.3 Introduction - Keras.ipynb b/1.3 Introduction - Keras.ipynb index 8428f8e..2431089 100644 --- a/1.3 Introduction - Keras.ipynb +++ b/1.3 Introduction - Keras.ipynb @@ -1,5 +1,33 @@ { "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "%matplotlib inline" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt" + ] + }, { "cell_type": "code", "execution_count": 3, @@ -8,12 +36,14 @@ }, "outputs": [], "source": [ - "%matplotlib inline" + "from sklearn.preprocessing import StandardScaler\n", + "from sklearn.preprocessing import LabelEncoder\n", + "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 4, "metadata": { "collapsed": false }, @@ -22,21 +52,12 @@ "name": "stderr", "output_type": "stream", "text": [ - "Using Theano backend.\n" + "Using TensorFlow backend.\n" ] } ], "source": [ - "import numpy as np\n", - "import pandas as pd\n", - "import theano\n", - "import theano.tensor as T\n", - "import matplotlib.pyplot as plt\n", - "import keras \n", - "from sklearn.preprocessing import StandardScaler\n", - "from sklearn.preprocessing import LabelEncoder \n", "from keras.utils import np_utils\n", - "from sklearn.cross_validation import train_test_split\n", "from keras.callbacks import EarlyStopping, ModelCheckpoint\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Activation" @@ -44,7 +65,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "For this section we will use the Kaggle otto challenge.\n", "If you want to follow, Get the data from Kaggle: https://www.kaggle.com/c/otto-group-product-classification-challenge/data" @@ -52,14 +76,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "#### About the data" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "The Otto Group is one of the world’s biggest e-commerce companies, A consistent analysis of the performance of products is crucial. However, due to diverse global infrastructure, many identical products get classified differently.\n", "For this competition, we have provided a dataset with 93 features for more than 200,000 products. The objective is to build a predictive model which is able to distinguish between our main product categories. \n", @@ -70,9 +100,11 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 5, "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -108,9 +140,11 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 6, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -137,9 +171,11 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 7, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { @@ -154,19 +190,19 @@ " 0. 0. 0. 0. 0. 0. 0. 3. 0. 0. 0. 0. 1. 1.\n", " 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", " 0. 11. 1. 20. 0. 0. 0. 0. 0.]]\n", - "(9L, 'classes')\n", - "(93L, 'dims')\n" + "9 classes\n", + "93 dims\n" ] } ], "source": [ "print(\"Loading data...\")\n", - "X, labels = load_data('train.csv', train=True)\n", + "X, labels = load_data('data/train.csv', train=True)\n", "X, scaler = preprocess_data(X)\n", "Y, encoder = preprocess_labels(labels)\n", "\n", "\n", - "X_test, ids = load_data('test.csv', train=False)\n", + "X_test, ids = load_data('data/test.csv', train=False)\n", "X_test, ids = X_test[:1000], ids[:1000]\n", "\n", "#Plotting the data\n", @@ -183,21 +219,30 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Now lets create and train a logistic regression model." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "---" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "# Keras\n", "\n", @@ -206,7 +251,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\n", "ref: https://keras.io/" @@ -214,7 +262,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## Why this name, Keras?\n", "\n", @@ -227,54 +278,59 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## Hands On - Keras Logistic Regression\n" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "(93L, 'dims')\n", + "93 dims\n", "Building model...\n", - "(9L, 'classes')\n", + "9 classes\n", "Epoch 1/10\n", - "61878/61878 [==============================] - 1s - loss: 1.0574 \n", + "61878/61878 [==============================] - 2s - loss: 1.0935 \n", "Epoch 2/10\n", - "61878/61878 [==============================] - 1s - loss: 0.7730 \n", + "61878/61878 [==============================] - 2s - loss: 0.7749 \n", "Epoch 3/10\n", - "61878/61878 [==============================] - 1s - loss: 0.7297 \n", + "61878/61878 [==============================] - 3s - loss: 0.7303 \n", "Epoch 4/10\n", - "61878/61878 [==============================] - 1s - loss: 0.7080 \n", + "61878/61878 [==============================] - 2s - loss: 0.7084 \n", "Epoch 5/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6948 \n", + "61878/61878 [==============================] - 3s - loss: 0.6949 \n", "Epoch 6/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6854 \n", + "61878/61878 [==============================] - 3s - loss: 0.6856 \n", "Epoch 7/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6787 \n", + "61878/61878 [==============================] - 3s - loss: 0.6788 \n", "Epoch 8/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6734 \n", + "61878/61878 [==============================] - 2s - loss: 0.6737 \n", "Epoch 9/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6691 \n", + "61878/61878 [==============================] - 3s - loss: 0.6693 \n", "Epoch 10/10\n", - "61878/61878 [==============================] - 1s - loss: 0.6657 \n" + "61878/61878 [==============================] - 2s - loss: 0.6660 \n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 6, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -296,14 +352,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Simplicity is pretty impressive right? :)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Now lets understand:\n", "
The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.
\n" @@ -311,14 +373,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "##### Dense" ] @@ -326,22 +394,99 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "source": [ "```python\n", "from keras.layers.core import Dense\n", "\n", - "Dense(output_dim, init='glorot_uniform', activation='linear', \n", - " weights=None, W_regularizer=None, b_regularizer=None,\n", - " activity_regularizer=None, W_constraint=None, \n", - " b_constraint=None, bias=True, input_dim=None)\n", + "Dense(units, activation=None, use_bias=True, \n", + " kernel_initializer='glorot_uniform', bias_initializer='zeros', \n", + " kernel_regularizer=None, bias_regularizer=None, \n", + " activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n", "```" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "* `units`: int > 0.\n", + "\n", + "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n", + "\n", + "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n", + "\n", + "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n", + "\n", + "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n", + "\n", + "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n", + "\n", + "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n", + "\n", + "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n", + "\n", + "* `bias_constraint`: instance of the constraints module, applied to the bias.\n", + "\n", + "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## (some) others `keras.core.layers`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "* `keras.layers.core.Flatten()`\n", + "* `keras.layers.core.Reshape(target_shape)`\n", + "* `keras.layers.core.Permute(dims)`\n", + "\n", + "```python\n", + "model = Sequential()\n", + "model.add(Permute((2, 1), input_shape=(10, 64)))\n", + "# now: model.output_shape == (None, 64, 10)\n", + "# note: `None` is the batch dimension\n", + "```\n", + "\n", + "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n", + "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "\n", + "\n", + "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "##### Activation" ] @@ -349,7 +494,9 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "deletable": true, + "editable": true }, "source": [ "```python\n", @@ -361,14 +508,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "**Supported Activations** : [https://keras.io/activations/]\n", + "\n", + "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "##### Optimizer" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n", "Here we used SGD (stochastic gradient descent) as an optimization algorithm for our trainable weights. " @@ -376,7 +541,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "\n", + "\n", + "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "\"Data Sciencing\" this example a little bit more\n", "=====" @@ -384,7 +564,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "What we did here is nice, however in the real world it is not useable because of overfitting.\n", "Lets try and solve it with cross validation." @@ -392,14 +575,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "##### Overfitting" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n", "\n", @@ -409,7 +598,9 @@ { "cell_type": "markdown", "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "source": [ "\n", @@ -418,7 +609,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "
To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n",
     "Next: we will use two of keras's callbacks EarlyStopping and ModelCheckpoint
" @@ -426,38 +620,40 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 10, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Train on 19835 samples, validate on 3501 samples\n", + "Train on 44706 samples, validate on 7890 samples\n", "Epoch 1/20\n", - "19835/19835 [==============================] - 0s - loss: 0.6391 - val_loss: 0.6680\n", + "44706/44706 [==============================] - 0s - loss: 0.6569 - val_loss: 0.6697\n", "Epoch 2/20\n", - "19835/19835 [==============================] - 0s - loss: 0.6386 - val_loss: 0.6689\n", + "44706/44706 [==============================] - 0s - loss: 0.6563 - val_loss: 0.6702\n", "Epoch 3/20\n", - "19835/19835 [==============================] - 0s - loss: 0.6384 - val_loss: 0.6695\n", + "44706/44706 [==============================] - 0s - loss: 0.6558 - val_loss: 0.6706\n", "Epoch 4/20\n", - "19835/19835 [==============================] - 0s - loss: 0.6381 - val_loss: 0.6702\n", + "44706/44706 [==============================] - 0s - loss: 0.6552 - val_loss: 0.6708\n", "Epoch 5/20\n", - "19835/19835 [==============================] - 0s - loss: 0.6378 - val_loss: 0.6709\n", + "44706/44706 [==============================] - 0s - loss: 0.6548 - val_loss: 0.6709\n", "Epoch 6/20\n", - "19328/19835 [============================>.] - ETA: 0s - loss: 0.6380Epoch 00005: early stopping\n", - "19835/19835 [==============================] - 0s - loss: 0.6375 - val_loss: 0.6716\n" + "44706/44706 [==============================] - 1s - loss: 0.6543 - val_loss: 0.6710\n", + "Epoch 00005: early stopping\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 13, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -468,21 +664,27 @@ "fBestModel = 'best_model.h5' \n", "early_stop = EarlyStopping(monitor='val_loss', patience=4, verbose=1) \n", "best_model = ModelCheckpoint(fBestModel, verbose=0, save_best_only=True)\n", - "model.fit(X, Y, validation_data = (X_test, Y_test), nb_epoch=20, \n", + "model.fit(X, Y, validation_data = (X_test, Y_test), epochs=20, \n", " batch_size=128, verbose=True, validation_split=0.15, \n", " callbacks=[best_model, early_stop]) " ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## Multi-Layer Perceptron and Fully Connected" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "So, how hard can it be to build a Multi-Layer percepton with keras?\n", "It is baiscly the same, just add more layers!" @@ -490,11 +692,50 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 11, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/10\n", + "44706/44706 [==============================] - 2s - loss: 0.8963 \n", + "Epoch 2/10\n", + "44706/44706 [==============================] - 2s - loss: 0.7089 \n", + "Epoch 3/10\n", + "44706/44706 [==============================] - 3s - loss: 0.6809 \n", + "Epoch 4/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6678 \n", + "Epoch 5/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6596 \n", + "Epoch 6/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6542 \n", + "Epoch 7/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6505 \n", + "Epoch 8/10\n", + "44706/44706 [==============================] - 3s - loss: 0.6472 \n", + "Epoch 9/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6451 \n", + "Epoch 10/10\n", + "44706/44706 [==============================] - 2s - loss: 0.6434 \n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "model = Sequential()\n", "model.add(Dense(100, input_shape=(dims,)))\n", @@ -506,21 +747,30 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Your Turn!" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "## Hands On - Keras Fully Connected\n" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Take couple of minutes and Try and optimize the number of layers and the number of parameters in the layers to get the best results. " ] @@ -529,7 +779,9 @@ "cell_type": "code", "execution_count": null, "metadata": { - "collapsed": false + "collapsed": false, + "deletable": true, + "editable": true }, "outputs": [], "source": [ @@ -548,56 +800,80 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "#### Theoretical Motivations for depth" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ ">Much has been studied about the depth of neural nets. Is has been proven mathematically[1] and empirically that convolutional neural network benifit from depth! " ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "[1] - On the Expressive Power of Deep Learning: A Tensor Analysis - Cohen, et al 2015" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "#### Theoretical Motivations for depth" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "One much quoted theorem about neural network states that:" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ ">Universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of $\\mathbb{R}^n$, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "deletable": true, + "editable": true + }, "source": [ "[1] - Approximation Capabilities of Multilayer Feedforward Networks - Kurt Hornik 1991" ] diff --git a/1.4 Keras Backend.ipynb b/1.4 Keras Backend.ipynb new file mode 100755 index 0000000..fc60bf1 --- /dev/null +++ b/1.4 Keras Backend.ipynb @@ -0,0 +1,1075 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "# Linear Regression\n", + "To get familiar with automatic differentiation, we start by learning a simple linear regression model using Stochastic Gradient Descent (SGD).\n", + "\n", + "Recall that given a dataset $\\{(x_i, y_i)\\}_{i=0}^N$, with $x_i, y_i \\in \\mathbb{R}$, the objective of linear regression is to find two scalars $w$ and $b$ such that $y = w\\cdot x + b$ fits the dataset. In this tutorial we will learn $w$ and $b$ using SGD and a Mean Square Error (MSE) loss:\n", + "\n", + "$$\\mathcal{l} = \\frac{1}{N} \\sum_{i=0}^N (w\\cdot x_i + b - y_i)^2$$\n", + "\n", + "Starting from random values, parameters $w$ and $b$ will be updated at each iteration via the following rule:\n", + "\n", + "$$w_t = w_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial w}$$\n", + "
\n", + "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n", + "\n", + "where $\\eta$ is the learning rate.\n", + "\n", + "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Placeholders and variables\n", + "To implement and run this simple model, we will use the [Keras backend module](http://keras.io/backend/), which provides an abstraction over Theano and Tensorflow, two popular tensor manipulation libraries that provide automatic differentiation.\n", + "\n", + "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n", + "\n", + "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Using TensorFlow backend.\n" + ] + } + ], + "source": [ + "import keras.backend as K\n", + "import numpy as np\n", + "\n", + "# Placeholders and variables\n", + "x = K.placeholder()\n", + "target = K.placeholder()\n", + "lr = K.variable(0.1)\n", + "w = K.variable(np.random.rand())\n", + "b = K.variable(np.random.rand())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "source": [ + "## Model definition\n", + "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "# Define model and loss\n", + "y = w * x + b\n", + "loss = K.mean(K.square(y-target))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": true, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "grads = K.gradients(loss, [w,b])\n", + "updates = [(w, w-lr*grads[0]), (b, b-lr*grads[1])]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [], + "source": [ + "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "## Training\n", + "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n", + "\n", + "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loss: 0.054, w, b: [0.65, 0.24]\n", + "Loss: 0.003, w, b: [0.77, 0.34]\n", + "Loss: 0.002, w, b: [0.82, 0.32]\n", + "Loss: 0.001, w, b: [0.85, 0.30]\n", + "Loss: 0.001, w, b: [0.87, 0.29]\n", + "Loss: 0.000, w, b: [0.89, 0.27]\n", + "Loss: 0.000, w, b: [0.91, 0.27]\n", + "Loss: 0.000, w, b: [0.92, 0.26]\n", + "Loss: 0.000, w, b: [0.93, 0.26]\n", + "Loss: 0.000, w, b: [0.94, 0.25]\n" + ] + } + ], + "source": [ + "# Generate data\n", + "np_x = np.random.rand(1000)\n", + "np_target = 0.96*np_x + 0.24\n", + "\n", + "# Training\n", + "loss_history = []\n", + "for epoch in range(200):\n", + " current_loss = train([np_x, np_target])[0]\n", + " loss_history.append(current_loss)\n", + " if epoch % 20 == 0:\n", + " print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "deletable": true, + "editable": true + }, + "source": [ + "We can also plot the loss history:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false, + "deletable": true, + "editable": true + }, + "outputs": [ + { + "data": { + "application/javascript": [ + "/* Put everything inside the global mpl namespace */\n", + "window.mpl = {};\n", + "\n", + "\n", + "mpl.get_websocket_type = function() {\n", + " if (typeof(WebSocket) !== 'undefined') {\n", + " return WebSocket;\n", + " } else if (typeof(MozWebSocket) !== 'undefined') {\n", + " return MozWebSocket;\n", + " } else {\n", + " alert('Your browser does not have WebSocket support.' +\n", + " 'Please try Chrome, Safari or Firefox ≥ 6. ' +\n", + " 'Firefox 4 and 5 are also supported but you ' +\n", + " 'have to enable WebSockets in about:config.');\n", + " };\n", + "}\n", + "\n", + "mpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n", + " this.id = figure_id;\n", + "\n", + " this.ws = websocket;\n", + "\n", + " this.supports_binary = (this.ws.binaryType != undefined);\n", + "\n", + " if (!this.supports_binary) {\n", + " var warnings = document.getElementById(\"mpl-warnings\");\n", + " if (warnings) {\n", + " warnings.style.display = 'block';\n", + " warnings.textContent = (\n", + " \"This browser does not support binary websocket messages. \" +\n", + " \"Performance may be slow.\");\n", + " }\n", + " }\n", + "\n", + " this.imageObj = new Image();\n", + "\n", + " this.context = undefined;\n", + " this.message = undefined;\n", + " this.canvas = undefined;\n", + " this.rubberband_canvas = undefined;\n", + " this.rubberband_context = undefined;\n", + " this.format_dropdown = undefined;\n", + "\n", + " this.image_mode = 'full';\n", + "\n", + " this.root = $('
');\n", + " this._root_extra_style(this.root)\n", + " this.root.attr('style', 'display: inline-block');\n", + "\n", + " $(parent_element).append(this.root);\n", + "\n", + " this._init_header(this);\n", + " this._init_canvas(this);\n", + " this._init_toolbar(this);\n", + "\n", + " var fig = this;\n", + "\n", + " this.waiting = false;\n", + "\n", + " this.ws.onopen = function () {\n", + " fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n", + " fig.send_message(\"send_image_mode\", {});\n", + " if (mpl.ratio != 1) {\n", + " fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n", + " }\n", + " fig.send_message(\"refresh\", {});\n", + " }\n", + "\n", + " this.imageObj.onload = function() {\n", + " if (fig.image_mode == 'full') {\n", + " // Full images could contain transparency (where diff images\n", + " // almost always do), so we need to clear the canvas so that\n", + " // there is no ghosting.\n", + " fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n", + " }\n", + " fig.context.drawImage(fig.imageObj, 0, 0);\n", + " };\n", + "\n", + " this.imageObj.onunload = function() {\n", + " this.ws.close();\n", + " }\n", + "\n", + " this.ws.onmessage = this._make_on_message_function(this);\n", + "\n", + " this.ondownload = ondownload;\n", + "}\n", + "\n", + "mpl.figure.prototype._init_header = function() {\n", + " var titlebar = $(\n", + " '
');\n", + " var titletext = $(\n", + " '
');\n", + " titlebar.append(titletext)\n", + " this.root.append(titlebar);\n", + " this.header = titletext[0];\n", + "}\n", + "\n", + "\n", + "\n", + "mpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n", + "\n", + "}\n", + "\n", + "\n", + "mpl.figure.prototype._root_extra_style = function(canvas_div) {\n", + "\n", + "}\n", + "\n", + "mpl.figure.prototype._init_canvas = function() {\n", + " var fig = this;\n", + "\n", + " var canvas_div = $('
');\n", + "\n", + " canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n", + "\n", + " function canvas_keyboard_event(event) {\n", + " return fig.key_event(event, event['data']);\n", + " }\n", + "\n", + " canvas_div.keydown('key_press', canvas_keyboard_event);\n", + " canvas_div.keyup('key_release', canvas_keyboard_event);\n", + " this.canvas_div = canvas_div\n", + " this._canvas_extra_style(canvas_div)\n", + " this.root.append(canvas_div);\n", + "\n", + " var canvas = $('');\n", + " canvas.addClass('mpl-canvas');\n", + " canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n", + "\n", + " this.canvas = canvas[0];\n", + " this.context = canvas[0].getContext(\"2d\");\n", + "\n", + " var backingStore = this.context.backingStorePixelRatio ||\n", + "\tthis.context.webkitBackingStorePixelRatio ||\n", + "\tthis.context.mozBackingStorePixelRatio ||\n", + "\tthis.context.msBackingStorePixelRatio ||\n", + "\tthis.context.oBackingStorePixelRatio ||\n", + "\tthis.context.backingStorePixelRatio || 1;\n", + "\n", + " mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n", + "\n", + " var rubberband = $('');\n", + " rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n", + "\n", + " var pass_mouse_events = true;\n", + "\n", + " canvas_div.resizable({\n", + " start: function(event, ui) {\n", + " pass_mouse_events = false;\n", + " },\n", + " resize: function(event, ui) {\n", + " fig.request_resize(ui.size.width, ui.size.height);\n", + " },\n", + " stop: function(event, ui) {\n", + " pass_mouse_events = true;\n", + " fig.request_resize(ui.size.width, ui.size.height);\n", + " },\n", + " });\n", + "\n", + " function mouse_event_fn(event) {\n", + " if (pass_mouse_events)\n", + " return fig.mouse_event(event, event['data']);\n", + " }\n", + "\n", + " rubberband.mousedown('button_press', mouse_event_fn);\n", + " rubberband.mouseup('button_release', mouse_event_fn);\n", + " // Throttle sequential mouse events to 1 every 20ms.\n", + " rubberband.mousemove('motion_notify', mouse_event_fn);\n", + "\n", + " rubberband.mouseenter('figure_enter', mouse_event_fn);\n", + " rubberband.mouseleave('figure_leave', mouse_event_fn);\n", + "\n", + " canvas_div.on(\"wheel\", function (event) {\n", + " event = event.originalEvent;\n", + " event['data'] = 'scroll'\n", + " if (event.deltaY < 0) {\n", + " event.step = 1;\n", + " } else {\n", + " event.step = -1;\n", + " }\n", + " mouse_event_fn(event);\n", + " });\n", + "\n", + " canvas_div.append(canvas);\n", + " canvas_div.append(rubberband);\n", + "\n", + " this.rubberband = rubberband;\n", + " this.rubberband_canvas = rubberband[0];\n", + " this.rubberband_context = rubberband[0].getContext(\"2d\");\n", + " this.rubberband_context.strokeStyle = \"#000000\";\n", + "\n", + " this._resize_canvas = function(width, height) {\n", + " // Keep the size of the canvas, canvas container, and rubber band\n", + " // canvas in synch.\n", + " canvas_div.css('width', width)\n", + " canvas_div.css('height', height)\n", + "\n", + " canvas.attr('width', width * mpl.ratio);\n", + " canvas.attr('height', height * mpl.ratio);\n", + " canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n", + "\n", + " rubberband.attr('width', width);\n", + " rubberband.attr('height', height);\n", + " }\n", + "\n", + " // Set the figure to an initial 600x600px, this will subsequently be updated\n", + " // upon first draw.\n", + " this._resize_canvas(600, 600);\n", + "\n", + " // Disable right mouse context menu.\n", + " $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n", + " return false;\n", + " });\n", + "\n", + " function set_focus () {\n", + " canvas.focus();\n", + " canvas_div.focus();\n", + " }\n", + "\n", + " window.setTimeout(set_focus, 100);\n", + "}\n", + "\n", + "mpl.figure.prototype._init_toolbar = function() {\n", + " var fig = this;\n", + "\n", + " var nav_element = $('
')\n", + " nav_element.attr('style', 'width: 100%');\n", + " this.root.append(nav_element);\n", + "\n", + " // Define a callback function for later on.\n", + " function toolbar_event(event) {\n", + " return fig.toolbar_button_onclick(event['data']);\n", + " }\n", + " function toolbar_mouse_event(event) {\n", + " return fig.toolbar_button_onmouseover(event['data']);\n", + " }\n", + "\n", + " for(var toolbar_ind in mpl.toolbar_items) {\n", + " var name = mpl.toolbar_items[toolbar_ind][0];\n", + " var tooltip = mpl.toolbar_items[toolbar_ind][1];\n", + " var image = mpl.toolbar_items[toolbar_ind][2];\n", + " var method_name = mpl.toolbar_items[toolbar_ind][3];\n", + "\n", + " if (!name) {\n", + " // put a spacer in here.\n", + " continue;\n", + " }\n", + " var button = $('