04_linear_regression_ex4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![book](https://raw.githubusercontent.com/ageron/tensorflow-safari-course/master/images/intro_to_tf_course.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try not to peek at the solutions when you go through the exercises. ;-)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's make sure this notebook works well in both Python 2 and Python 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression with TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "data = np.loadtxt(\"data/life_satisfaction.csv\",\n",
    "                  dtype=np.float32,\n",
    "                  delimiter=\",\",\n",
    "                  skiprows=1,\n",
    "                  usecols=[1, 2])\n",
    "X_train = data[:, 0:1] / 10000 # feature scaling\n",
    "y_train = data[:, 1:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row in `X_train` represents a training instance, in this case a country. In this simple regression example, there is just one feature per instance (i.e., one column), in this case the country's GDP per capita (in tens of thousands of dollars)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_life_satisfaction(X_train, y_train):\n",
    "    plt.plot(X_train * 10000, y_train, \"bo\")\n",
    "    plt.axis([0, 60000, 0, 10])\n",
    "    plt.xlabel(\"GDP per capita ($)\")\n",
    "    plt.ylabel(\"Life Satisfaction\")\n",
    "    plt.grid()\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plot_life_satisfaction(X_train, y_train)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Linear Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Exercise](https://c1.staticflickr.com/9/8101/8553474140_c50cf08708_b.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In this exercise we will build a linear regression model using TensorFlow. If you are not familiar with the maths behind linear regression models, you can read the explanation below. If you already know this (or if you don't care much about the maths), you can just skip this explanation and simply follow the instructions given in questions 4.1 to 4.3 below.\n",
    "\n",
    "In a linear regression model, the predictions are a linear combination of the input features. In other words, the predicted value $\\hat{y}$ can be computed using the following equation:\n",
    "\n",
    "$\\hat{y} = w_1 x_1 + w_2 x_2 + \\dots + w_n x_n + b$\n",
    "\n",
    "where:\n",
    "* $x_1, x_2, \\dots, x_n $ are the input features,\n",
    "* $w_1, w_2, \\dots, w_n $, are their corresponding weights,\n",
    "* and $b$ is the bias term (also called the intercept term).\n",
    "\n",
    "This equation can be expressed in a more compact way using vectors:\n",
    "\n",
    "$\\hat{y} = \\langle \\mathbf{x}, \\mathbf{w} \\rangle + b$\n",
    "\n",
    "where:\n",
    "* $ \\mathbf{x} = \\begin{pmatrix}x_1 \\\\ x_2 \\\\ \\vdots \\\\ x_n \\end{pmatrix}$ is the input feature vector (by convention, vectors are written in bold font),\n",
    "* $ \\mathbf{w} = \\begin{pmatrix}w_1 \\\\ w_2 \\\\ \\vdots \\\\ w_n \\end{pmatrix}$ is the weight vector,\n",
    "* $\\langle \\mathbf{x}, \\mathbf{w} \\rangle$ is the inner product of vectors $\\mathbf{x}$ and $\\mathbf{w}$, equal to $w_1 x_1 + w_2 x_2 + \\dots + w_n x_n$.\n",
    "\n",
    "It is often more convenient to handle vectors as matrices with a single column (a \"column vector\"). The inner product $\\langle \\mathbf{x}, \\mathbf{w} \\rangle$ is then replaced with the matrix dot product: $\\mathbf{x}^T \\cdot \\mathbf{w}$, where $\\mathbf{x}^T$ is the transpose of the column vector $\\mathbf{x}$. Transposing a column vector gives you a \"row vector\" (i.e., a matrix with a single row): $\\mathbf{x}^T = \\begin{pmatrix} x_1 & x_2 & \\dots & x_n \\end{pmatrix}$. Once again $\\mathbf{x}^T \\cdot \\mathbf{w} = w_1 x_1 + w_2 x_2 + \\dots + w_n x_n$.\n",
    "\n",
    "Lastly, it is possible to compute predictions for many instances at a time by putting all their input features in a matrix $\\mathbf{X}$ (by convention, matrices are in capital letters with a bold font, except when they just represent column or row vectors). The vector containing the predictions for every instance can be computed using the following equation:\n",
    "\n",
    "$\\hat{\\mathbf{y}} = \\mathbf{X} \\cdot \\mathbf{w} + b$\n",
    "\n",
    "where:\n",
    "* $ \\hat{\\mathbf{y}} = \\begin{pmatrix}\\hat{y}^{(1)} \\\\ \\hat{y}^{(2)} \\\\ \\vdots \\\\ \\hat{y}^{(m)} \\end{pmatrix}$ is the prediction vector, containing the predictions for all $m$ instances.\n",
    "* $ \\mathbf{X} = \\begin{pmatrix}x_1^{(1)} & x_2^{(1)} & \\cdots & x_n^{(1)} \\\\\n",
    "                                x_1^{(2)} & x_2^{(2)} & \\cdots & x_n^{(2)} \\\\\n",
    "                                \\vdots    & \\vdots    & \\ddots & \\vdots    \\\\\n",
    "                                x_1^{(m)} & x_2^{(m)} & \\cdots & x_n^{(m)}\\end{pmatrix} =                  \\begin{pmatrix}(\\mathbf{x}^{(1)})^T \\\\\n",
    "                                (\\mathbf{x}^{(2)})^T  \\\\\n",
    "                                \\vdots \\\\\n",
    "                                (\\mathbf{x}^{(m)})^T\\end{pmatrix} $ is the input feature matrix. It contains the input features of all instances for which you want to make predictions. Each row represents an instance, each column represents a feature.\n",
    "* Note that the matrix dot product $\\mathbf{X} \\cdot \\mathbf{w}$ returns a column vector, so when we add the bias term $b$, we mean adding that value to each and every element in the column vector (this is called _broadcasting_)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "4.1) Create a graph containing:\n",
    "* a constant `X` initialized with `X_train`, which contains the input features of the training instances. In this particular example, there is just a single feature per instance (i.e., the GDP per capita).\n",
    "* a constant `y` initialized with `y_train`, which contains the labels of each instance (i.e., the life satisfaction).\n",
    "* a variable `b`, representing the bias term (initialized to 0.0).\n",
    "* a variable `w`, representing the weight vector (initialized to a column vector full of zeros, using `tf.zeros()`). Since there is just one input feature per instance in this example, this column vector contains a single row (it is a matrix with a single item).\n",
    "* an operation `y_pred` that computes the equation presented above: $\\hat{\\mathbf{y}} = \\mathbf{X} \\cdot \\mathbf{w} + b$. You will need to use `tf.matmul()`.\n",
    "* as always, don't forget to add an `init` operation, using `tf.global_variables_initializer()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.2) Start a session, run the `init` operation and evaluate the predictions `y_pred`. Since both variables `b` and `w` are initialized with zeros, you should get a vector full of zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.3) Let's measure how bad the model is using a cost function (also called a loss function). In regression tasks, it is common to use the Mean Square Error (MSE) as the cost function. It is given by the following equation:\n",
    "\n",
    "$\\text{MSE}(\\mathbf{w}, b) = \\dfrac{1}{m} \\sum\\limits_{i=1}^{m}{(\\hat{y}^{(i)}-y^{(i)})^2}$.\n",
    "\n",
    "Add an `mse` operation to your graph, to compute the Mean Square Error. Hint: use `tf.reduce_mean()` and `tf.square()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.4) Now start a session, initalize the variables and evaluate the MSE. As you can see, the result is quite high: this makes sense since we have not trained the model yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.5) To find the optimal values for the model parameters (i.e., the variables `w` and `b`), we will use Gradient Descent. For this, we first need to compute the gradient of the cost function with regards to the model parameters.\n",
    "\n",
    "The gradient of the MSE with regards to the weight vector $\\mathbf{w}$ is:\n",
    "\n",
    "$\\nabla_{\\mathbf{w}}\\, \\text{MSE}(\\mathbf{w}, b) =\n",
    "\\begin{pmatrix}\n",
    " \\frac{\\partial}{\\partial w_0} \\text{MSE}(\\mathbf{w}, b) \\\\\n",
    " \\frac{\\partial}{\\partial w_1} \\text{MSE}(\\mathbf{w}, b) \\\\\n",
    " \\vdots \\\\\n",
    " \\frac{\\partial}{\\partial w_n} \\text{MSE}(\\mathbf{w}, b)\n",
    "\\end{pmatrix}\n",
    " = \\dfrac{2}{m} \\mathbf{X}^T \\cdot (\\hat{\\mathbf{y}} - \\mathbf{y})\n",
    "$\n",
    "\n",
    "And the partial derivative with regards to the bias $b$ is:\n",
    "\n",
    "$\n",
    "\\dfrac{\\partial}{\\partial b} \\text{MSE}(\\mathbf{w}, b) = \\dfrac{2}{m} \\sum\\limits_{i=1}^{m}(\\hat{y}^{(i)}-y^{(i)})\n",
    "$\n",
    "\n",
    "Add the operations `gradients_w` and `gradients_b` to your graph, using the equations above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.6) To perform a Gradient Descent step, we need to subtract the gradients (multiplied by the learning rate $\\eta$) from the weight vector and the bias:\n",
    "\n",
    "$\n",
    "\\mathbf{w} \\gets \\mathbf{w} - \\eta \\nabla_{\\mathbf{w}}\\, \\text{MSE}(\\mathbf{w}, b)\n",
    "$\n",
    "\n",
    "$\n",
    "\\mathbf{b} \\gets \\mathbf{b} - \\eta \\dfrac{\\partial}{\\partial b} \\text{MSE}(\\mathbf{w}, b)\n",
    "$\n",
    "\n",
    "Add two assignment operations, `tweak_w_ops` and `tweak_b_ops` that perform the assigments above, using a small learning rate $\\eta = 0.01$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.7) That's it! We're ready to train the model. Start a session, initialize the variables, then write a loop that will repeatedly evaluate the assignment operations (e.g., 2000 times). Every 100 iterations, evaluate the MSE and print it out. Within a few hundred iterations the MSE should drop below 1.0, and eventually reach about 0.18. Congratulations! You built and trained your first Machine Learning model using TensorFlow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try not to peek at the solution below before you have done the exercise! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![thinking](https://upload.wikimedia.org/wikipedia/commons/0/06/Filos_segundo_logo_%28flipped%29.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4 - Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    X = tf.constant(X_train, name=\"X\")\n",
    "    y = tf.constant(y_train, name=\"y\")\n",
    "\n",
    "    b = tf.Variable(0.0, name=\"b\")\n",
    "    w = tf.Variable(tf.zeros([1, 1]), name=\"w\")\n",
    "    y_pred = tf.add(tf.matmul(X, w), b, name=\"y_pred\")  # X @ w + b\n",
    "    \n",
    "    init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session(graph=graph) as sess:\n",
    "    init.run()\n",
    "    print(y_pred.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with graph.as_default():\n",
    "    error = y_pred - y\n",
    "    square_error = tf.square(error)\n",
    "    mse = tf.reduce_mean(square_error, name=\"mse\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session(graph=graph) as sess:\n",
    "    init.run()\n",
    "    print(mse.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with graph.as_default():\n",
    "    m = len(X_train)\n",
    "    gradients_w = 2/m * tf.matmul(tf.transpose(X), error)\n",
    "    gradients_b = 2 * tf.reduce_mean(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "\n",
    "with graph.as_default():\n",
    "    tweak_w_op = tf.assign(w, w - learning_rate * gradients_w)\n",
    "    tweak_b_op = tf.assign(b, b - learning_rate * gradients_b)\n",
    "    training_op = tf.group(tweak_w_op, tweak_b_op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iterations = 2000\n",
    "\n",
    "with tf.Session(graph=graph) as sess:\n",
    "    init.run()\n",
    "    for iteration in range(n_iterations):\n",
    "        if iteration % 100 == 0:\n",
    "            print(\"Iteration {:5}, MSE: {:.4f}\".format(iteration, mse.eval()))\n",
    "        training_op.run()\n",
    "    w_val, b_val = sess.run([w, b])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_life_satisfaction_with_linear_model(X_train, y_train, w, b):\n",
    "    plot_life_satisfaction(X_train, y_train)\n",
    "    plt.plot([0, 60000], [b, w[0][0] * (60000 / 10000) + b])\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plot_life_satisfaction_with_linear_model(X_train, y_train, w_val, b_val)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}