lab_day1/day1_code02_keras_basics_[EXERCISE].ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Assessing the available GPU\n",
        "\n",
        "The following commands will tell you what GPU is available. Remember that if you want to actually use the GPU you must enable hardware acceleration under \"Runtime/Change runtime type\" otherwise you'll be using just the CPU (in this latter case you'll get something along the lines of \"NVIDIA-SMI has failed\").\n",
        "\n",
        "Also keep in mind that the (free) GPU runtime is limited!"
      ],
      "metadata": {
        "id": "S4o1EYcAQYXT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!/usr/local/cuda/bin/nvcc --version\n",
        "!nvidia-smi -L"
      ],
      "metadata": {
        "id": "YtpEkSioQX6p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4DXwxt9r1bx_"
      },
      "source": [
        "## A first (actually second) encounter with Keras\n",
        "\n",
        "`Keras` is a high-level Python library for **deep learning**: Keras is an API wrapper for `TensorFlow`, which is the **backend engine** that performs low-level computations (e.g. tensor products, convolutions, etc.).\n",
        "\n",
        "[Don't worry if there are many terms that you don't fully understand now: we will go over these concepts repeatedly during the course, with increasing levels of detail]\n",
        "\n",
        "Keras allows easy and fast deployment of neural networks models: the code chunk below imports `Keras` functions that define a specific neural network architecture:\n",
        "\n",
        "- `Sequential()`: a network made up of a sequence of successive layers\n",
        "- `Dense()`: fully-connected (dense) layers\n",
        "- the type of **activation functions** to be used in each layer\n",
        "- the number of units in each layer\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-uUdGW-D1ayA"
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Activation,Conv2D,MaxPooling2D,Flatten,Dropout,Input"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iV5ShyTb98uU"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Input((50,)))\n",
        "model.add(Dense(64, activation='relu')) #input shape of 50\n",
        "model.add(Dense(28, activation='relu')) #input shape of 64\n",
        "model.add(Dense(10, activation='softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6wsTLk6GEqNR"
      },
      "source": [
        "## Model definition\n",
        "\n",
        "As we saw above, the first step is to choose the neural network architecture and define the deep learning model.\n",
        "\n",
        "We first use the `Sequential()` function to select the Keras API for the construction of deep learning models. Sequential models are a stack of layers, each with one input and one output tensor.\n",
        "\n",
        "The function `add()` will then allow us to add subsequent layers to our deep learning model, by specifying the type of layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6AkjMRyaNK8Y"
      },
      "source": [
        "model = Sequential()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Input layer\n",
        "\n",
        "This layer doesn't do any actual operation, but it's used to explicitly declare the shape of the input data. Count the parenthesis: we are passing a tuple."
      ],
      "metadata": {
        "id": "venWArEaRzwQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.add(Input((12,12,3))) ##input shape: tensor size e.g. 12 x 12 pixels x 3 channels (RGB)"
      ],
      "metadata": {
        "id": "JstI-jawSHNA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZuZ5apaYPhCM"
      },
      "source": [
        "### Dense layer\n",
        "\n",
        "The first very common type of layer is the **Fully Connected** (dense) layer: we can specify the number of units in the layer and the activation function to use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_81JIwrxPgQX"
      },
      "source": [
        "model.add(Dense(units = 32, activation='relu'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vjeG4RIPaK8"
      },
      "source": [
        "### Convolutional layer\n",
        "\n",
        "Another common type of layers is the **convolutional layer** where we specify the number of filters, the size of the filter (kernel) and the activation function.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CFQbEniLPYS7"
      },
      "source": [
        "model.add(Conv2D(filters = 8, kernel_size = (3, 3), activation='relu'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mraG7PM1S0Ea"
      },
      "source": [
        "### MaxPooling layer\n",
        "\n",
        "MaxPooling layers downsample the input representation by taking the maximum value over the window defined by pool size for each dimension along the features axis ($\\rightarrow$ dimensionality reduction)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pkJ0AGASSzHZ"
      },
      "source": [
        "model.add(MaxPooling2D(pool_size=(2, 2)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxsTd-EyVy24"
      },
      "source": [
        "### Dropout layer\n",
        "\n",
        "The **dropout layer** randomly sets input units to 0 with a specified frequency (`rate` argument) at each step during training, which helps prevent overfitting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d_Vtwu-3VcwA"
      },
      "source": [
        "model.add(Dropout(rate = 0.5))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hF2wo-KpeOhp"
      },
      "source": [
        "We can get an overview of the defined deep learning model by using the `model.summary()` function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JJrocEIkZBPU"
      },
      "source": [
        "print(model.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1jau0_BeY5B"
      },
      "source": [
        "We see that we have a large number of parameters to train in this model:\n",
        "\n",
        "- 4 parameters (3 channels + bias term) per 32 units in the Dense layer $\\rightarrow$ 128 parameters\n",
        "- for convolutional layer number of parameter is computed as:\n",
        "    - number_parameters = out_channels * (in_channels * kernel_h * kernel_w + 1)\n",
        "    - the \"+ 1\" is for bias term\n",
        "    - `out_channels` = number of units in current layers = 8\n",
        "    - `in_channels` = number of units in previous layer = 32\n",
        "    - `kernel_h` = `kernel_w` = 3\n",
        "    - 8 * (32 * 3 * 3 + 1) = 2312\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmhJQx3OXs6d"
      },
      "source": [
        "## Compiling the model\n",
        "\n",
        "Once the model architecture has been defined, you go on compiling the model by setting up relevant configurations: **loss function**, **optimizer**, etc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4h2t5ivJEI6J"
      },
      "source": [
        "model.compile(loss='mean_squared_error', optimizer='adam')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BoSoa-PphdjJ"
      },
      "source": [
        "The next steps would be to:\n",
        "\n",
        "- fit the built and compiled model\n",
        "- evaluate the model performance\n",
        "\n",
        "You can find the Keras code for these two steps below: since we currently haven't loaded any data to work on, we commented out these lines of code, for us to see and discuss the syntax"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ObGoYi48hqMa"
      },
      "source": [
        "# model.fit(x_train, y_train, batch_size=32, epochs=10, validation_data=(x_val, y_val))\n",
        "# score = model.evaluate(x_test, y_test, batch_size=32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3y4cx-02hm6T"
      },
      "source": [
        "## An example with simple linear regression\n",
        "\n",
        "We now use Keras to fit a simple linear regression model:\n",
        "\n",
        "$$\n",
        "y = \\mu + \\beta x + e\n",
        "$$\n",
        "\n",
        "we start by generating data for the feature *x* and the target variable *y* (continuous)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MF_wy4MciQHA"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "x = np.linspace(1,2,200) ## generate 200 samples with values between 1 and 2\n",
        "print(x.shape)\n",
        "y = x*4 + np.random.randn(*x.shape) * 0.3 ## *x.shape is the number of random numbers to generate (* because x.shape is a tuple)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(x,y)"
      ],
      "metadata": {
        "id": "dMK5fz8q8iC-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwVguveFkz4A"
      },
      "source": [
        "We now build the neural networks model by specifying one Dense layer with one single unit. The activation function is `linear` (identity function). the input dimension is 1 (one feature)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2IIPVkGMkuua"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Input((1,)))\n",
        "model.add(Dense(1, activation='linear'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8NTEzG-SlTyi"
      },
      "source": [
        "We then compile the model, selecting *stochastic gradient descent* as optimizer and *mean square error* as loss function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZarF5DOTkxKU"
      },
      "source": [
        "model.compile(optimizer='sgd', loss='mse', metrics=['mse'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0ZxIjWMdlY63"
      },
      "source": [
        "print(model.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_0LqhG-UlbsP"
      },
      "source": [
        "We here have two parameters to train: the intercept $\\mu$ and the slope $\\beta$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-08cpPgbmrla"
      },
      "source": [
        "We are now ready to fit our simple linear regression model with Keras. We define epochs=30 (30 iterations of optimization)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SBYLRGn3lsNX"
      },
      "source": [
        "history = model.fit(x,y, epochs=30, shuffle=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XjqiaLe0pbEB"
      },
      "source": [
        "Finally, we are ready to evaluate the fitted deep learning model.\n",
        "First, we look at predicted vs observed values: using the fitted model, we predict values for our 200 generated samples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9JCA-6JgoV8R"
      },
      "source": [
        "predict = model.predict(x)\n",
        "y_hat = predict.reshape(200,)\n",
        "print(y.shape)\n",
        "print(y_hat.shape)\n",
        "\n",
        "## correlation coefficient\n",
        "print(np.corrcoef(y,y_hat))\n",
        "print(\"The correlation between observed and predicted y's is: \", np.corrcoef(y,y_hat)[0,1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "twrvpVMRzgw_"
      },
      "source": [
        "### Question\n",
        "\n",
        "Why do you think that we have such high predictive accuracy?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K1JchT4zhlFE"
      },
      "source": [
        "plt.scatter(y_hat, y)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Seiszuwq3A5"
      },
      "source": [
        "We can plot the decay of the loss over epochs, using results saved in the `history` object"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GvtM5m56mX3n"
      },
      "source": [
        "plt.plot(history.history['loss'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDdf-AQ5q-cT"
      },
      "source": [
        "We can also retrieve the weights (coefficients) estimated in the final layer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MDY2BIpgoQdL"
      },
      "source": [
        "model.get_weights()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "huMD0ZmkoygY"
      },
      "source": [
        "## Exercise 2.1 [optional]: do-it-together\n",
        "\n",
        "Using the `scikitlearn` dataset `diabetes`, you should fit a simple linear regression model with **Keras**:\n",
        "\n",
        "- select one feature from the data (age, sex, bmi etc.)\n",
        "- build your Keras model\n",
        "- compile the model\n",
        "- fit the model\n",
        "- evaluate the model\n",
        "\n",
        "We have prepared the data loading step for you:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CypetAKGrUwq"
      },
      "source": [
        "import pandas as pd\n",
        "import sklearn.datasets\n",
        "\n",
        "diabetes = sklearn.datasets.load_diabetes()\n",
        "diabetes.data = pd.DataFrame(diabetes.data, columns=diabetes.feature_names) #converting numpy array -> pandas DataFrame\n",
        "diabetes.target = pd.Series(diabetes.target)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bnUXPvVUryGn"
      },
      "source": [
        "print(diabetes.DESCR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bi7iWJJTrqLd"
      },
      "source": [
        "diabetes.data ## features"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L2OfxaI6ru_5"
      },
      "source": [
        "diabetes.target ## quantitative measure of disease progression"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lYSoICBFswA7"
      },
      "source": [
        "Now it's our turn to play a bit with basic Keras: let's enjoy it!"
      ]
    }
  ]
}