C2W3_Lab_01_Model_Evaluation_and_Selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FvYDy9apGRlt"
   },
   "source": [
    "# Optional Lab: Model Evaluation and Selection\n",
    "\n",
    "Quantifying a learning algorithm's performance and comparing different models are some of the common tasks when applying machine learning to real world applications. In this lab, you will practice doing these using the tips shared in class. Specifically, you will:\n",
    "\n",
    "* split datasets into training, cross validation, and test sets\n",
    "* evaluate regression and classification models\n",
    "* add polynomial features to improve the performance of a linear regression model\n",
    "* compare several neural network architectures\n",
    "\n",
    "This lab will also help you become familiar with the code you'll see in this week's programming assignment. Let's begin!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jj1P1kTmGRlv"
   },
   "source": [
    "## Imports and Lab Setup\n",
    "\n",
    "First, you will import the packages needed for the tasks in this lab. We also included some commands to make the outputs later more readable by reducing verbosity and suppressing non-critical warnings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xuHD2EOxGRlv"
   },
   "outputs": [],
   "source": [
    "# for array computations and loading data\n",
    "import numpy as np\n",
    "\n",
    "# for building linear regression models and preparing data\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import StandardScaler, PolynomialFeatures\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# for building and training neural networks\n",
    "import tensorflow as tf\n",
    "\n",
    "# custom functions\n",
    "import utils\n",
    "\n",
    "# reduce display precision on numpy arrays\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "# suppress warnings\n",
    "tf.get_logger().setLevel('ERROR')\n",
    "tf.autograph.set_verbosity(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M_UnQ-tWGRlw"
   },
   "source": [
    "## Regression\n",
    "\n",
    "First, you will be tasked to develop a model for a regression problem. You are given the dataset below consisting of 50 examples of an input feature `x` and its corresponding target `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QLsSum08GRlx",
    "outputId": "0d843000-30bf-4a55-dcd8-cd5e42dc716e"
   },
   "outputs": [],
   "source": [
    "# Load the dataset from the text file\n",
    "data = np.loadtxt('./data/data_w3_ex1.csv', delimiter=',')\n",
    "\n",
    "# Split the inputs and outputs into separate arrays\n",
    "x = data[:,0]\n",
    "y = data[:,1]\n",
    "\n",
    "# Convert 1-D arrays into 2-D because the commands later will require it\n",
    "x = np.expand_dims(x, axis=1)\n",
    "y = np.expand_dims(y, axis=1)\n",
    "\n",
    "print(f\"the shape of the inputs x is: {x.shape}\")\n",
    "print(f\"the shape of the targets y is: {y.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ar2jLTqlpt4x"
   },
   "source": [
    "You can plot the dataset to get an idea of how the target behaves with respect to the input. In case you want to inspect the code, you can find the `plot_dataset()` function in the `utils.py` file outside this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "VxJqex92GRlx",
    "outputId": "b6d781d1-bff3-4912-abe1-0d7fe5738b08"
   },
   "outputs": [],
   "source": [
    "# Plot the entire dataset\n",
    "utils.plot_dataset(x=x, y=y, title=\"input vs. target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WbZdSoJHGRly"
   },
   "source": [
    "## Split the dataset into training, cross validation, and test sets\n",
    "\n",
    "In previous labs, you might have used the entire dataset to train your models. In practice however, it is best to hold out a portion of your data to measure how well your model generalizes to new examples. This will let you know if the model has overfit to your training set.\n",
    "\n",
    "As mentioned in the lecture, it is common to split your data into three parts:\n",
    "\n",
    "* ***training set*** - used to train the model\n",
    "* ***cross validation set (also called validation, development, or dev set)*** - used to evaluate the different model configurations you are choosing from. For example, you can use this to make a decision on what polynomial features to add to your dataset.\n",
    "* ***test set*** - used to give a fair estimate of your chosen model's performance against new examples. This should not be used to make decisions while you are still developing the models.\n",
    "\n",
    "Scikit-learn provides a [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function to split your data into the parts mentioned above. In the code cell below, you will split the entire dataset into 60% training, 20% cross validation, and 20% test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "22JKPjmjGRly",
    "outputId": "6f02748d-e8ef-4243-d7ff-a206be7e66f5"
   },
   "outputs": [],
   "source": [
    "# Get 60% of the dataset as the training set. Put the remaining 40% in temporary variables: x_ and y_.\n",
    "x_train, x_, y_train, y_ = train_test_split(x, y, test_size=0.40, random_state=1)\n",
    "\n",
    "# Split the 40% subset above into two: one half for cross validation and the other for the test set\n",
    "x_cv, x_test, y_cv, y_test = train_test_split(x_, y_, test_size=0.50, random_state=1)\n",
    "\n",
    "# Delete temporary variables\n",
    "del x_, y_\n",
    "\n",
    "print(f\"the shape of the training set (input) is: {x_train.shape}\")\n",
    "print(f\"the shape of the training set (target) is: {y_train.shape}\\n\")\n",
    "print(f\"the shape of the cross validation set (input) is: {x_cv.shape}\")\n",
    "print(f\"the shape of the cross validation set (target) is: {y_cv.shape}\\n\")\n",
    "print(f\"the shape of the test set (input) is: {x_test.shape}\")\n",
    "print(f\"the shape of the test set (target) is: {y_test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rrgciBNd1HYr"
   },
   "source": [
    "You can plot the dataset again below to see which points were used as training, cross validation, or test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "yGm8JbUUzoUW",
    "outputId": "2f7227ef-9e48-4df5-d913-a5b48e92fe81"
   },
   "outputs": [],
   "source": [
    "utils.plot_train_cv_test(x_train, y_train, x_cv, y_cv, x_test, y_test, title=\"input vs. target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ayr22PeEGRly"
   },
   "source": [
    "## Fit a linear model\n",
    "\n",
    "Now that you have split the data, one of the first things you can try is to fit a linear model. You will do that in the next sections below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z1UdF77SGRlz"
   },
   "source": [
    "### Feature scaling\n",
    "\n",
    "In the previous course of this specialization, you saw that it is usually a good idea to perform feature scaling to help your model converge faster. This is especially true if your input features have widely different ranges of values. Later in this lab, you will be adding polynomial terms so your input features will indeed have different ranges. For example, $x$ runs from around 1600 to 3600, while $x^2$ will run from 2.56 million to 12.96 million. \n",
    "\n",
    "You will only use $x$ for this first model but it's good to practice feature scaling now so you can apply it later. For that, you will use the [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) class from scikit-learn. This computes the z-score of your inputs. As a refresher, the z-score is given by the equation:\n",
    "\n",
    "$$ z = \\frac{x - \\mu}{\\sigma} $$\n",
    "\n",
    "where $\\mu$ is the mean of the feature values and $\\sigma$ is the standard deviation. The code below shows how to prepare the training set using the said class. You can plot the results again to inspect if it still follows the same pattern as before. The new graph should have a reduced range of values for `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "QlJ7eQ0TGRlz",
    "outputId": "ed899629-4c5e-468d-d5c8-a4b3349bd62a"
   },
   "outputs": [],
   "source": [
    "# Initialize the class\n",
    "scaler_linear = StandardScaler()\n",
    "\n",
    "# Compute the mean and standard deviation of the training set then transform it\n",
    "X_train_scaled = scaler_linear.fit_transform(x_train)\n",
    "\n",
    "print(f\"Computed mean of the training set: {scaler_linear.mean_.squeeze():.2f}\")\n",
    "print(f\"Computed standard deviation of the training set: {scaler_linear.scale_.squeeze():.2f}\")\n",
    "\n",
    "# Plot the results\n",
    "utils.plot_dataset(x=X_train_scaled, y=y_train, title=\"scaled input vs. target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vzDw7PcJGRlz"
   },
   "source": [
    "### Train the model\n",
    "\n",
    "Next, you will create and train a regression model. For this lab, you will use the [LinearRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) class but take note that there are other [linear regressors](https://scikit-learn.org/stable/modules/classes.html#classical-linear-regressors) which you can also use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DIol3uTkGRlz",
    "outputId": "283e5e1c-9aff-41bb-c2d3-6977861d63bd"
   },
   "outputs": [],
   "source": [
    "# Initialize the class\n",
    "linear_model = LinearRegression()\n",
    "\n",
    "# Train the model\n",
    "linear_model.fit(X_train_scaled, y_train )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x8LW5leOGRlz"
   },
   "source": [
    "### Evaluate the Model\n",
    "\n",
    "To evaluate the performance of your model, you will measure the error for the training and cross validation sets. For the training error, recall the equation for calculating the mean squared error (MSE):\n",
    "\n",
    "$$J_{train}(\\vec{w}, b) = \\frac{1}{2m_{train}}\\left[\\sum_{i=1}^{m_{train}}(f_{\\vec{w},b}(\\vec{x}_{train}^{(i)}) - y_{train}^{(i)})^2\\right]$$\n",
    "\n",
    "Scikit-learn also has a built-in [`mean_squared_error()`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html) function that you can use. Take note though that [as per the documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error), scikit-learn's implementation only divides by `m` and not `2*m`, where `m` is the number of examples. As mentioned in Course 1 of this Specialization (cost function lectures), dividing by `2m` is a convention we will follow but the calculations should still work whether or not you include it. Thus, to match the equation above, you can use the scikit-learn function then divide by 2 as shown below. We also included a for-loop implementation so you can check that it's equal. \n",
    "\n",
    "Another thing to take note: Since you trained the model on scaled values (i.e. using the z-score), you should also feed in the scaled training set instead of its raw values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xmuskcDhfy8z",
    "outputId": "10bb7e65-1723-4ab6-8bbf-f800e8777474"
   },
   "outputs": [],
   "source": [
    "# Feed the scaled training set and get the predictions\n",
    "yhat = linear_model.predict(X_train_scaled)\n",
    "\n",
    "# Use scikit-learn's utility function and divide by 2\n",
    "print(f\"training MSE (using sklearn function): {mean_squared_error(y_train, yhat) / 2}\")\n",
    "\n",
    "# for-loop implementation\n",
    "total_squared_error = 0\n",
    "\n",
    "for i in range(len(yhat)):\n",
    "    squared_error_i  = (yhat[i] - y_train[i])**2\n",
    "    total_squared_error += squared_error_i                                              \n",
    "\n",
    "mse = total_squared_error / (2*len(yhat))\n",
    "\n",
    "print(f\"training MSE (for-loop implementation): {mse.squeeze()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lujrQns9f-vu"
   },
   "source": [
    "You can then compute the MSE for the cross validation set with basically the same equation:\n",
    "\n",
    "$$J_{cv}(\\vec{w}, b) = \\frac{1}{2m_{cv}}\\left[\\sum_{i=1}^{m_{cv}}(f_{\\vec{w},b}(\\vec{x}_{cv}^{(i)}) - y_{cv}^{(i)})^2\\right]$$\n",
    "\n",
    "As with the training set, you will also want to scale the cross validation set. An *important* thing to note when using the z-score is you have to use the mean and standard deviation of the **training set** when scaling the cross validation set. This is to ensure that your input features are transformed as expected by the model. One way to gain intuition is with this scenario:\n",
    "\n",
    "* Say that your training set has an input feature equal to `500` which is scaled down to `0.5` using the z-score.\n",
    "* After training, your model is able to accurately map this scaled input `x=0.5` to the target output `y=300`. \n",
    "* Now let's say that you deployed this model and one of your users fed it a sample equal to `500`. \n",
    "* If you get this input sample's z-score using any other values of the mean and standard deviation, then it might not be scaled to `0.5` and your model will most likely make a wrong prediction (i.e. not equal to `y=300`). \n",
    "\n",
    "You will scale the cross validation set below by using the same `StandardScaler` you used earlier but only calling its [`transform()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler.transform) method instead of [`fit_transform()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler.fit_transform)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JJQbmq70GRl0",
    "outputId": "7b75d664-7cf0-4a27-c79c-3236857f444b"
   },
   "outputs": [],
   "source": [
    "# Scale the cross validation set using the mean and standard deviation of the training set\n",
    "X_cv_scaled = scaler_linear.transform(x_cv)\n",
    "\n",
    "print(f\"Mean used to scale the CV set: {scaler_linear.mean_.squeeze():.2f}\")\n",
    "print(f\"Standard deviation used to scale the CV set: {scaler_linear.scale_.squeeze():.2f}\")\n",
    "\n",
    "# Feed the scaled cross validation set\n",
    "yhat = linear_model.predict(X_cv_scaled)\n",
    "\n",
    "# Use scikit-learn's utility function and divide by 2\n",
    "print(f\"Cross validation MSE: {mean_squared_error(y_cv, yhat) / 2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BZtb_blcGRl0"
   },
   "source": [
    "## Adding Polynomial Features\n",
    "\n",
    "From the graphs earlier, you may have noticed that the target `y` rises more sharply at smaller values of `x` compared to higher ones. A straight line might not be the best choice because the target `y` seems to flatten out as `x` increases. Now that you have these values of the training and cross validation MSE from the linear model, you can try adding polynomial features to see if you can get a better performance. The code will mostly be the same but with a few extra preprocessing steps. Let's see that below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the additional features\n",
    "\n",
    "First, you will generate the polynomial features from your training set. The code below demonstrates how to do this using the [`PolynomialFeatures`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html) class. It will create a new input feature which has the squared values of the input `x` (i.e. degree=2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rt1FCgQOGRl0"
   },
   "outputs": [],
   "source": [
    "# Instantiate the class to make polynomial features\n",
    "poly = PolynomialFeatures(degree=2, include_bias=False)\n",
    "\n",
    "# Compute the number of features and transform the training set\n",
    "X_train_mapped = poly.fit_transform(x_train)\n",
    "\n",
    "# Preview the first 5 elements of the new training set. Left column is `x` and right column is `x^2`\n",
    "# Note: The `e+<number>` in the output denotes how many places the decimal point should \n",
    "# be moved. For example, `3.24e+03` is equal to `3240`\n",
    "print(X_train_mapped[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will then scale the inputs as before to narrow down the range of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dVJUmPQKGRl1"
   },
   "outputs": [],
   "source": [
    "# Instantiate the class\n",
    "scaler_poly = StandardScaler()\n",
    "\n",
    "# Compute the mean and standard deviation of the training set then transform it\n",
    "X_train_mapped_scaled = scaler_poly.fit_transform(X_train_mapped)\n",
    "\n",
    "# Preview the first 5 elements of the scaled training set.\n",
    "print(X_train_mapped_scaled[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then proceed to train the model. After that, you will measure the model's performance against the cross validation set. Like before, you should make sure to perform the same transformations as you did in the training set. You will add the same number of polynomial features then scale the range of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WO7rK-f9GRl1",
    "outputId": "12a61725-7eab-40c0-91c2-2acc9f1c5d2a"
   },
   "outputs": [],
   "source": [
    "# Initialize the class\n",
    "model = LinearRegression()\n",
    "\n",
    "# Train the model\n",
    "model.fit(X_train_mapped_scaled, y_train )\n",
    "\n",
    "# Compute the training MSE\n",
    "yhat = model.predict(X_train_mapped_scaled)\n",
    "print(f\"Training MSE: {mean_squared_error(y_train, yhat) / 2}\")\n",
    "\n",
    "# Add the polynomial features to the cross validation set\n",
    "X_cv_mapped = poly.transform(x_cv)\n",
    "\n",
    "# Scale the cross validation set using the mean and standard deviation of the training set\n",
    "X_cv_mapped_scaled = scaler_poly.transform(X_cv_mapped)\n",
    "\n",
    "# Compute the cross validation MSE\n",
    "yhat = model.predict(X_cv_mapped_scaled)\n",
    "print(f\"Cross validation MSE: {mean_squared_error(y_cv, yhat) / 2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the MSEs are significantly better for both the training and cross validation set when you added the 2nd order polynomial. You may want to introduce more polynomial terms and see which one gives the best performance. As shown in class, you can have 10 different models like this:\n",
    "\n",
    "<img src='images/C2_W3_poly.png' width=50%>\n",
    "\n",
    "You can create a loop that contains all the steps in the previous code cells. Here is one implementation that adds polynomial features up to degree=10. We'll plot it at the end to make it easier to compare the results for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize lists to save the errors, models, and feature transforms\n",
    "train_mses = []\n",
    "cv_mses = []\n",
    "models = []\n",
    "polys = []\n",
    "scalers = []\n",
    "\n",
    "# Loop over 10 times. Each adding one more degree of polynomial higher than the last.\n",
    "for degree in range(1,11):\n",
    "    \n",
    "    # Add polynomial features to the training set\n",
    "    poly = PolynomialFeatures(degree, include_bias=False)\n",
    "    X_train_mapped = poly.fit_transform(x_train)\n",
    "    polys.append(poly)\n",
    "    \n",
    "    # Scale the training set\n",
    "    scaler_poly = StandardScaler()\n",
    "    X_train_mapped_scaled = scaler_poly.fit_transform(X_train_mapped)\n",
    "    scalers.append(scaler_poly)\n",
    "    \n",
    "    # Create and train the model\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_train_mapped_scaled, y_train )\n",
    "    models.append(model)\n",
    "    \n",
    "    # Compute the training MSE\n",
    "    yhat = model.predict(X_train_mapped_scaled)\n",
    "    train_mse = mean_squared_error(y_train, yhat) / 2\n",
    "    train_mses.append(train_mse)\n",
    "    \n",
    "    # Add polynomial features and scale the cross validation set\n",
    "    X_cv_mapped = poly.transform(x_cv)\n",
    "    X_cv_mapped_scaled = scaler_poly.transform(X_cv_mapped)\n",
    "    \n",
    "    # Compute the cross validation MSE\n",
    "    yhat = model.predict(X_cv_mapped_scaled)\n",
    "    cv_mse = mean_squared_error(y_cv, yhat) / 2\n",
    "    cv_mses.append(cv_mse)\n",
    "    \n",
    "# Plot the results\n",
    "degrees=range(1,11)\n",
    "utils.plot_train_cv_mses(degrees, train_mses, cv_mses, title=\"degree of polynomial vs. train and CV MSEs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R83nSGpXGRl2"
   },
   "source": [
    "### Choosing the best model\n",
    "\n",
    "When selecting a model, you want to choose one that performs well both on the training and cross validation set. It implies that it is able to learn the patterns from your training set without overfitting. If you used the defaults in this lab, you will notice a sharp drop in cross validation error from the models with degree=1 to degree=2. This is followed by a relatively flat line up to degree=5. After that, however, the cross validation error is generally getting worse as you add more polynomial features. Given these, you can decide to use the model with the lowest `cv_mse` as the one best suited for your application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model with the lowest CV MSE (add 1 because list indices start at 0)\n",
    "# This also corresponds to the degree of the polynomial added\n",
    "degree = np.argmin(cv_mses) + 1\n",
    "print(f\"Lowest CV MSE is found in the model with degree={degree}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then publish the generalization error by computing the test set's MSE. As usual, you should transform this data the same way you did with the training and cross validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add polynomial features to the test set\n",
    "X_test_mapped = polys[degree-1].transform(x_test)\n",
    "\n",
    "# Scale the test set\n",
    "X_test_mapped_scaled = scalers[degree-1].transform(X_test_mapped)\n",
    "\n",
    "# Compute the test MSE\n",
    "yhat = models[degree-1].predict(X_test_mapped_scaled)\n",
    "test_mse = mean_squared_error(y_test, yhat) / 2\n",
    "\n",
    "print(f\"Training MSE: {train_mses[degree-1]:.2f}\")\n",
    "print(f\"Cross Validation MSE: {cv_mses[degree-1]:.2f}\")\n",
    "print(f\"Test MSE: {test_mse:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m5uStJF3GRl2"
   },
   "source": [
    "## Neural Networks\n",
    "\n",
    "The same model selection process can also be used when choosing between different neural network architectures. In this section, you will create the models shown below and apply it to the same regression task above.\n",
    "\n",
    "<img src='images/C2_W3_NN_Arch.png' width=40%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the Data\n",
    "\n",
    "You will use the same training, cross validation, and test sets you generated in the previous section. From earlier lectures in this course, you may have known that neural networks can learn non-linear relationships so you can opt to skip adding polynomial features. The code is still included below in case you want to try later and see what effect it will have on your results. The default `degree` is set to `1` to indicate that it will just use `x_train`, `x_cv`, and `x_test` as is (i.e. without any additional polynomial features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-armALWpGRl2"
   },
   "outputs": [],
   "source": [
    "# Add polynomial features\n",
    "degree = 1\n",
    "poly = PolynomialFeatures(degree, include_bias=False)\n",
    "X_train_mapped = poly.fit_transform(x_train)\n",
    "X_cv_mapped = poly.transform(x_cv)\n",
    "X_test_mapped = poly.transform(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will scale the input features to help gradient descent converge faster. Again, notice that you are using the mean and standard deviation computed from the training set by just using `transform()` in the cross validation and test sets instead of `fit_transform()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hBn-0m0IGRl2"
   },
   "outputs": [],
   "source": [
    "# Scale the features using the z-score\n",
    "scaler = StandardScaler()\n",
    "X_train_mapped_scaled = scaler.fit_transform(X_train_mapped)\n",
    "X_cv_mapped_scaled = scaler.transform(X_cv_mapped)\n",
    "X_test_mapped_scaled = scaler.transform(X_test_mapped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build and train the models\n",
    "\n",
    "You will then create the neural network architectures shown earlier. The code is provided in the `build_models()` function in the `utils.py` file in case you want to inspect or modify it. You will use that in the loop below then proceed to train the models. For each model, you will also record the training and cross validation errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize lists that will contain the errors for each model\n",
    "nn_train_mses = []\n",
    "nn_cv_mses = []\n",
    "\n",
    "# Build the models\n",
    "nn_models = utils.build_models()\n",
    "\n",
    "# Loop over the the models\n",
    "for model in nn_models:\n",
    "    \n",
    "    # Setup the loss and optimizer\n",
    "    model.compile(\n",
    "    loss='mse',\n",
    "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.1),\n",
    "    )\n",
    "\n",
    "    print(f\"Training {model.name}...\")\n",
    "    \n",
    "    # Train the model\n",
    "    model.fit(\n",
    "        X_train_mapped_scaled, y_train,\n",
    "        epochs=300,\n",
    "        verbose=0\n",
    "    )\n",
    "    \n",
    "    print(\"Done!\\n\")\n",
    "\n",
    "    \n",
    "    # Record the training MSEs\n",
    "    yhat = model.predict(X_train_mapped_scaled)\n",
    "    train_mse = mean_squared_error(y_train, yhat) / 2\n",
    "    nn_train_mses.append(train_mse)\n",
    "    \n",
    "    # Record the cross validation MSEs \n",
    "    yhat = model.predict(X_cv_mapped_scaled)\n",
    "    cv_mse = mean_squared_error(y_cv, yhat) / 2\n",
    "    nn_cv_mses.append(cv_mse)\n",
    "\n",
    "    \n",
    "# print results\n",
    "print(\"RESULTS:\")\n",
    "for model_num in range(len(nn_train_mses)):\n",
    "    print(\n",
    "        f\"Model {model_num+1}: Training MSE: {nn_train_mses[model_num]:.2f}, \" +\n",
    "        f\"CV MSE: {nn_cv_mses[model_num]:.2f}\"\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the recorded errors, you can decide which is the best model for your application. Look at the results above and see if you agree with the selected `model_num` below. Finally, you will compute the test error to estimate how well it generalizes to new examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the model with the lowest CV MSE\n",
    "model_num = 3\n",
    "\n",
    "# Compute the test MSE\n",
    "yhat = nn_models[model_num-1].predict(X_test_mapped_scaled)\n",
    "test_mse = mean_squared_error(y_test, yhat) / 2\n",
    "\n",
    "print(f\"Selected Model: {model_num}\")\n",
    "print(f\"Training MSE: {nn_train_mses[model_num-1]:.2f}\")\n",
    "print(f\"Cross Validation MSE: {nn_cv_mses[model_num-1]:.2f}\")\n",
    "print(f\"Test MSE: {test_mse:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "In this last part of the lab, you will practice model evaluation and selection on a classification task. The process will be similar, with the main difference being the computation of the errors. You will see that in the following sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Dataset\n",
    "\n",
    "First, you will load a dataset for a binary classification task. It has 200 examples of two input features (`x1` and `x2`), and a target `y` of either `0` or `1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset from a text file\n",
    "data = np.loadtxt('./data/data_w3_ex2.csv', delimiter=',')\n",
    "\n",
    "# Split the inputs and outputs into separate arrays\n",
    "x_bc = data[:,:-1]\n",
    "y_bc = data[:,-1]\n",
    "\n",
    "# Convert y into 2-D because the commands later will require it (x is already 2-D)\n",
    "y_bc = np.expand_dims(y_bc, axis=1)\n",
    "\n",
    "print(f\"the shape of the inputs x is: {x_bc.shape}\")\n",
    "print(f\"the shape of the targets y is: {y_bc.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can plot the dataset to examine how the examples are separated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.plot_bc_dataset(x=x_bc, y=y_bc, title=\"x1 vs. x2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split and prepare the dataset\n",
    "\n",
    "Next, you will generate the training, cross validation, and test sets. You will use the same 60/20/20 proportions as before. You will also scale the features as you did in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Get 60% of the dataset as the training set. Put the remaining 40% in temporary variables.\n",
    "x_bc_train, x_, y_bc_train, y_ = train_test_split(x_bc, y_bc, test_size=0.40, random_state=1)\n",
    "\n",
    "# Split the 40% subset above into two: one half for cross validation and the other for the test set\n",
    "x_bc_cv, x_bc_test, y_bc_cv, y_bc_test = train_test_split(x_, y_, test_size=0.50, random_state=1)\n",
    "\n",
    "# Delete temporary variables\n",
    "del x_, y_\n",
    "\n",
    "print(f\"the shape of the training set (input) is: {x_bc_train.shape}\")\n",
    "print(f\"the shape of the training set (target) is: {y_bc_train.shape}\\n\")\n",
    "print(f\"the shape of the cross validation set (input) is: {x_bc_cv.shape}\")\n",
    "print(f\"the shape of the cross validation set (target) is: {y_bc_cv.shape}\\n\")\n",
    "print(f\"the shape of the test set (input) is: {x_bc_test.shape}\")\n",
    "print(f\"the shape of the test set (target) is: {y_bc_test.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale the features\n",
    "\n",
    "# Initialize the class\n",
    "scaler_linear = StandardScaler()\n",
    "\n",
    "# Compute the mean and standard deviation of the training set then transform it\n",
    "x_bc_train_scaled = scaler_linear.fit_transform(x_bc_train)\n",
    "x_bc_cv_scaled = scaler_linear.transform(x_bc_cv)\n",
    "x_bc_test_scaled = scaler_linear.transform(x_bc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the error for classification models\n",
    "\n",
    "In the previous sections on regression models, you used the mean squared error to measure how well your model is doing. For classification, you can get a similar metric by getting the fraction of the data that the model has misclassified. For example, if your model made wrong predictions for 2 samples out of 5, then you will report an error of `40%` or `0.4`. The code below demonstrates this using a for-loop and also with Numpy's [`mean()`](https://numpy.org/doc/stable/reference/generated/numpy.mean.html) function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample model output\n",
    "probabilities = np.array([0.2, 0.6, 0.7, 0.3, 0.8])\n",
    "\n",
    "# Apply a threshold to the model output. If greater than 0.5, set to 1. Else 0.\n",
    "predictions = np.where(probabilities >= 0.5, 1, 0)\n",
    "\n",
    "# Ground truth labels\n",
    "ground_truth = np.array([1, 1, 1, 1, 1])\n",
    "\n",
    "# Initialize counter for misclassified data\n",
    "misclassified = 0\n",
    "\n",
    "# Get number of predictions\n",
    "num_predictions = len(predictions)\n",
    "\n",
    "# Loop over each prediction\n",
    "for i in range(num_predictions):\n",
    "    \n",
    "    # Check if it matches the ground truth\n",
    "    if predictions[i] != ground_truth[i]:\n",
    "        \n",
    "        # Add one to the counter if the prediction is wrong\n",
    "        misclassified += 1\n",
    "\n",
    "# Compute the fraction of the data that the model misclassified\n",
    "fraction_error = misclassified/num_predictions\n",
    "\n",
    "print(f\"probabilities: {probabilities}\")\n",
    "print(f\"predictions with threshold=0.5: {predictions}\")\n",
    "print(f\"targets: {ground_truth}\")\n",
    "print(f\"fraction of misclassified data (for-loop): {fraction_error}\")\n",
    "print(f\"fraction of misclassified data (with np.mean()): {np.mean(predictions != ground_truth)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build and train the model\n",
    "\n",
    "You will use the same neural network architectures in the previous section so you can call the `build_models()` function again to create new instances of these models. \n",
    "\n",
    "You will follow the recommended approach mentioned last week where you use a `linear` activation for the output layer (instead of `sigmoid`) then set `from_logits=True` when declaring the loss function of the model. You will use the [binary crossentropy loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy) because this is a binary classification problem.\n",
    "\n",
    "After training, you will use a [sigmoid function](https://www.tensorflow.org/api_docs/python/tf/math/sigmoid) to convert the model outputs into probabilities. From there, you can set a threshold and get the fraction of misclassified examples from the training and cross validation sets.\n",
    "\n",
    "You can see all these in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize lists that will contain the errors for each model\n",
    "nn_train_error = []\n",
    "nn_cv_error = []\n",
    "\n",
    "# Build the models\n",
    "models_bc = utils.build_models()\n",
    "\n",
    "# Loop over each model\n",
    "for model in models_bc:\n",
    "    \n",
    "    # Setup the loss and optimizer\n",
    "    model.compile(\n",
    "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),\n",
    "    )\n",
    "\n",
    "    print(f\"Training {model.name}...\")\n",
    "\n",
    "    # Train the model\n",
    "    model.fit(\n",
    "        x_bc_train_scaled, y_bc_train,\n",
    "        epochs=200,\n",
    "        verbose=0\n",
    "    )\n",
    "    \n",
    "    print(\"Done!\\n\")\n",
    "    \n",
    "    # Set the threshold for classification\n",
    "    threshold = 0.5\n",
    "    \n",
    "    # Record the fraction of misclassified examples for the training set\n",
    "    yhat = model.predict(x_bc_train_scaled)\n",
    "    yhat = tf.math.sigmoid(yhat)\n",
    "    yhat = np.where(yhat >= threshold, 1, 0)\n",
    "    train_error = np.mean(yhat != y_bc_train)\n",
    "    nn_train_error.append(train_error)\n",
    "\n",
    "    # Record the fraction of misclassified examples for the cross validation set\n",
    "    yhat = model.predict(x_bc_cv_scaled)\n",
    "    yhat = tf.math.sigmoid(yhat)\n",
    "    yhat = np.where(yhat >= threshold, 1, 0)\n",
    "    cv_error = np.mean(yhat != y_bc_cv)\n",
    "    nn_cv_error.append(cv_error)\n",
    "\n",
    "# Print the result\n",
    "for model_num in range(len(nn_train_error)):\n",
    "    print(\n",
    "        f\"Model {model_num+1}: Training Set Classification Error: {nn_train_error[model_num]:.5f}, \" +\n",
    "        f\"CV Set Classification Error: {nn_cv_error[model_num]:.5f}\"\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the output above, you can choose which one performed best. If there is a tie on the cross validation set error, then you can add another criteria to break it. For example, you can choose the one with a lower training error. A more common approach is to choose the smaller model because it saves computational resources. In our example, Model 1 is the smallest and Model 3 is the largest.\n",
    "\n",
    "Finally, you can compute the test error to report the model's generalization error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the model with the lowest error\n",
    "model_num = 3\n",
    "\n",
    "# Compute the test error\n",
    "yhat = models_bc[model_num-1].predict(x_bc_test_scaled)\n",
    "yhat = tf.math.sigmoid(yhat)\n",
    "yhat = np.where(yhat >= threshold, 1, 0)\n",
    "nn_test_error = np.mean(yhat != y_bc_test)\n",
    "\n",
    "print(f\"Selected Model: {model_num}\")\n",
    "print(f\"Training Set Classification Error: {nn_train_error[model_num-1]:.4f}\")\n",
    "print(f\"CV Set Classification Error: {nn_cv_error[model_num-1]:.4f}\")\n",
    "print(f\"Test Set Classification Error: {nn_test_error:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap Up\n",
    "\n",
    "In this lab, you practiced evaluating a model's performance and choosing between different model configurations. You split your datasets into training, cross validation, and test sets and saw how each of these are used in machine learning applications. In the next section of the course, you will see more tips on how to improve your models by diagnosing bias and variance. Keep it up!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}