Part_4.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# Part 4",
      "metadata": {
        "tags": [],
        "cell_id": "00000-4943bd68-5973-410b-8b00-c12bceb3ce04",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Lecture 4\n\n<img src=\"./image/Capacities.png\" height=\"150\" />\n\nBigger size of training set decreases the apparent error on the training set. If the training set is small its more likely to overfit.\n- Can be done by\n    - Use more data (data augmentation)\n    - Reduce the number of features\n    - Reduce complexity/flexibility of the model ( how to quantify this? )\n\n**Overfitting**:\n- High bias, low variance\n- Similarly distributed data has different accuracy\n\n**Underfitting**:\n- Low bias, high variance\n\n**Bayes Error**:\n- Irreducible error",
      "metadata": {
        "tags": [],
        "cell_id": "00001-db296936-86d8-4b52-9822-0f6fdbce1dd1",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Assignment 4: Overfit & regularization\n\n- 1 epoch: one iteration over all examples\n- 1 batch: one iteration over the examples in the batch\n\n**MNIST-1D**:\n- Lightweight, quickly trained\n\n<img src=\"./image/MNIST-1D.png\" height=\"100\" />\n\n**Overfitting**:\n- Training accuracy is higher than the test accuracy.\n- Identify and alleviate this a little by splitting into train, test (and sometimes validation).\n- Low bias, High variance\n- Too little samples might overfit the model\n\n<img src=\"./image/Overfitting.png\" height=\"200\" />\n\n### Prevent overfitting\nReduce the number of parameters in the model could be used to try and prevent overfitting, but does not allow for much flexibility. However, more expressivity with simpler decision boundaries are preferred which lead to regularization methods.",
      "metadata": {
        "tags": [],
        "cell_id": "00002-ce5b7715-8f76-4170-b6f9-b6aaf519fa89",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "**Regularization**:\n_Discourage overly complex models_, hopes to improve generalization of model.\n- *L2 Regularization*: $$\\mathcal{L} = \\mathcal{L}_0 + \\frac{\\lambda}{2}\\sum_w w^2 $$\n    - Encourage network to learn small parameters. Small weights are not very susceptible to small changes in the the input, making the decision boundary relatively smooth.\n    - Also called **weight decay**\n- *L1 Regularization*:\n    - Reduce the weights to exactly 0, perform feature selection/neuron removal during training\n- *Early Stoppping*: Halt training when the network starts to overfit on the training data\n    - Continuously monitor performance via separate validation set and stop when accuracy no longer improves (with a certain partience taken into account).\n- *Dropout*: deactivate a random fraction of the neurons in a layer during a training iteration by setting them to zero.\n    - Will act as ensemble model when all neurons are active because during training many \"similar models\" are trained (by way of random deactivation).\n    - neurons become less reliant on the presence specific neurons and thus is forced to learn a robust feature (noise injection / peturbations), which makes it somewhat **similiar to L2 regularization**.\n- *Noise robustness* (Adversarial data)\n- *Weight sharing*\n- *Dataset augmentation* ( + feature reduction )\n- *Weight decay* \n    - discourage large weights\n- *Batch normalisation*",
      "metadata": {
        "tags": [],
        "cell_id": "00002-dca218d0-f673-42e5-8b06-e92bea882f0e",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "<a style='text-decoration:none;line-height:16px;display:flex;color:#5B5B62;padding:10px;justify-content:end;' href='https://deepnote.com?utm_source=created-in-deepnote-cell&projectId=de0be7a9-29e1-4ab6-9ce7-607fa646094e' target=\"_blank\">\n<img alt='Created in deepnote.com' style='display:inline;max-height:16px;margin:0px;margin-right:7.5px;' src='data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iODBweCIgaGVpZ2h0PSI4MHB4IiB2aWV3Qm94PSIwIDAgODAgODAiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICA8IS0tIEdlbmVyYXRvcjogU2tldGNoIDU0LjEgKDc2NDkwKSAtIGh0dHBzOi8vc2tldGNoYXBwLmNvbSAtLT4KICAgIDx0aXRsZT5Hcm91cCAzPC90aXRsZT4KICAgIDxkZXNjPkNyZWF0ZWQgd2l0aCBTa2V0Y2guPC9kZXNjPgogICAgPGcgaWQ9IkxhbmRpbmciIHN0cm9rZT0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIxIiBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPgogICAgICAgIDxnIGlkPSJBcnRib2FyZCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTEyMzUuMDAwMDAwLCAtNzkuMDAwMDAwKSI+CiAgICAgICAgICAgIDxnIGlkPSJHcm91cC0zIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjM1LjAwMDAwMCwgNzkuMDAwMDAwKSI+CiAgICAgICAgICAgICAgICA8cG9seWdvbiBpZD0iUGF0aC0yMCIgZmlsbD0iIzAyNjVCNCIgcG9pbnRzPSIyLjM3NjIzNzYyIDgwIDM4LjA0NzY2NjcgODAgNTcuODIxNzgyMiA3My44MDU3NTkyIDU3LjgyMTc4MjIgMzIuNzU5MjczOSAzOS4xNDAyMjc4IDMxLjY4MzE2ODMiPjwvcG9seWdvbj4KICAgICAgICAgICAgICAgIDxwYXRoIGQ9Ik0zNS4wMDc3MTgsODAgQzQyLjkwNjIwMDcsNzYuNDU0OTM1OCA0Ny41NjQ5MTY3LDcxLjU0MjI2NzEgNDguOTgzODY2LDY1LjI2MTk5MzkgQzUxLjExMjI4OTksNTUuODQxNTg0MiA0MS42NzcxNzk1LDQ5LjIxMjIyODQgMjUuNjIzOTg0Niw0OS4yMTIyMjg0IEMyNS40ODQ5Mjg5LDQ5LjEyNjg0NDggMjkuODI2MTI5Niw0My4yODM4MjQ4IDM4LjY0NzU4NjksMzEuNjgzMTY4MyBMNzIuODcxMjg3MSwzMi41NTQ0MjUgTDY1LjI4MDk3Myw2Ny42NzYzNDIxIEw1MS4xMTIyODk5LDc3LjM3NjE0NCBMMzUuMDA3NzE4LDgwIFoiIGlkPSJQYXRoLTIyIiBmaWxsPSIjMDAyODY4Ij48L3BhdGg+CiAgICAgICAgICAgICAgICA8cGF0aCBkPSJNMCwzNy43MzA0NDA1IEwyNy4xMTQ1MzcsMC4yNTcxMTE0MzYgQzYyLjM3MTUxMjMsLTEuOTkwNzE3MDEgODAsMTAuNTAwMzkyNyA4MCwzNy43MzA0NDA1IEM4MCw2NC45NjA0ODgyIDY0Ljc3NjUwMzgsNzkuMDUwMzQxNCAzNC4zMjk1MTEzLDgwIEM0Ny4wNTUzNDg5LDc3LjU2NzA4MDggNTMuNDE4MjY3Nyw3MC4zMTM2MTAzIDUzLjQxODI2NzcsNTguMjM5NTg4NSBDNTMuNDE4MjY3Nyw0MC4xMjg1NTU3IDM2LjMwMzk1NDQsMzcuNzMwNDQwNSAyNS4yMjc0MTcsMzcuNzMwNDQwNSBDMTcuODQzMDU4NiwzNy43MzA0NDA1IDkuNDMzOTE5NjYsMzcuNzMwNDQwNSAwLDM3LjczMDQ0MDUgWiIgaWQ9IlBhdGgtMTkiIGZpbGw9IiMzNzkzRUYiPjwvcGF0aD4KICAgICAgICAgICAgPC9nPgogICAgICAgIDwvZz4KICAgIDwvZz4KPC9zdmc+' > </img>\nCreated in <span style='font-weight:600;margin-left:4px;'>Deepnote</span></a>",
      "metadata": {
        "tags": [],
        "created_in_deepnote_cell": true,
        "deepnote_cell_type": "markdown"
      }
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {
    "orig_nbformat": 2,
    "deepnote": {
      "is_reactive": false
    },
    "deepnote_notebook_id": "bbda719a-04a8-4c6f-a163-1d2c175a55cd",
    "deepnote_execution_queue": []
  }
}