02-supervised-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tutorial\n",
    "\n",
    "- Jupyter notebook tutorial : https://www.youtube.com/watch?v=HW29067qVWk\n",
    "- Pandas tutorial :https://www.youtube.com/watch?v=ZyhVh-qRZPA\n",
    "- Numpy : https://www.youtube.com/watch?v=GB9ByFAIAH4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recap : AI, ML, DL\n",
    "\n",
    "<img src=\"images/ml_dl_vs_ai_1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/MLvsDL3.webp\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/nueural_nets.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 2\n",
    "\n",
    "- Today\n",
    "  -  Supervised learning : Classification  and Regression\n",
    "  -  Toy SKlearn dataset\n",
    "  -  Train simple KNN algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised Learning\n",
    "\n",
    "- Supervised machine learning is one of the most commonly used and successful types of machine learning <br><br>\n",
    "\n",
    "\n",
    "- Supervised learning is used whenever we want to predict a certain outcome from a given input, and we have examples of input/output pairs <br><br>\n",
    "\n",
    "\n",
    "\n",
    "> Analogy : Imagine a computer is a child, we are its supervisor (e.g. parent, guardian, or teacher), and we want the child (computer) to learn what a pig looks like. We will show the child several different pictures, some of which are pigs and the rest could be pictures of anything (cats, dogs, etc).When we see a pig, we shout “pig!” When it’s not a pig, we shout “no, not pig!” After doing this several times with the child, we show them a picture and ask “pig?” and they will correctly (most of the time) say “pig!” or “no, not pig!” depending on what the picture is. That is supervised machine learning.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/alade.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Types of classification\n",
    "\n",
    "- There are two major types of supervised machine learning problems, called classi#cation and regression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"images/class_vs_reg.png\" style=\"width: 700px;/\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "Classification can be Binary classification or multi-class classification <br><br>\n",
    "\n",
    "- Binary Classification: Classification tasks with two classes.\n",
    "- Multi-class Classification: Classification tasks with more than two classes.\n",
    " \n",
    " <img src=\"images/binary_vs_multilabel.png\" style=\"width: 700px;/\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Some algorithms are designed for binary classification problems, not multi-classification problem. Examples include : Logistic Regression, Perceptron and Support Vector Machines.\n",
    "\n",
    "However, heuristic methods can be used to split a multi-class classification problem into multiple binary classification datasets and train a binary classification model each (One-vs-Rest (OvR), One-vs-One (OvO))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generalization\n",
    "\n",
    "> Generalization refers to how well the concepts learned by a machine learning model apply to specific examples not seen by the model when it was learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In supervised , the goal of a good model is to generalize well from the training data to any unseen data from the problem domain. This allows us to make predictions in the future on data the model has never seen. <br><br>\n",
    "\n",
    "- If a model is able to make accurate predictions on unseen data, we say it is able to **generalize** from the training set to the test set.  <br><br>\n",
    "  \n",
    "- We want to build a model that is able to generalize as accurately as possible. <br><br>\n",
    "- Two terminologies are used in machine learning when we talk about how well a machine learning model learns and generalizes to new data: **overfitting** and **underfitting** <br><br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ocerfitting vs Underfitting\n",
    "\n",
    "- Overfitting and underfitting are the two biggest causes for poor performance of machine learning algorithms.\n",
    "\n",
    "- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model_complexity](images/overfitting_underfitting_cartoon.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relation of Model Complexity to Dataset Size\n",
    "\n",
    "#### Never underestimate the power of more data\n",
    "> It’s important to note that model complexity is intimately tied to the variation of inputs contained in your training dataset: the larger variety of data points your dataset contains, the more complex a model you can use without overfitting. <br><br>\n",
    "\n",
    "> Usually, collecting more data points will yield more variety, so larger datasets allow building more complex models. However, simply duplicating the same data points or collecting very similar data will not help. <br><br>\n",
    "\n",
    "> Having more data and building appropriately more complex models can often work wonders for supervised learning tasks.\n",
    "\n",
    "> In the real world, you often have the ability to decide how much data to collect, which might be more beneficial than tweaking and tuning your model. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervised Machine Learning Algorithms\n",
    "- We will use top-down approach of learning  <br><br>\n",
    "- We will now review the most popular machine learning algorithms and explain how they learn from data and how they make predictions.<br><br>\n",
    "\n",
    "- We will examine the strengths and weaknesses of each algorithm, and what kind of data they can best be applied to.<br><br>\n",
    "  \n",
    "-  We will also explain the meaning of the most important parameters and options. <br><br>\n",
    "\n",
    "-  Many algorithms have a classification and a regression variant, and we will describe bot\n",
    "\n",
    "\n",
    "#### Some Sample or Toy Datasets\n",
    "\n",
    "- We will use several datasets to illustrate the different algorithms. Some of the datasets will be small and synthetic (meaning made-up). <br><br>\n",
    "- In Sklearn : https://scikit-learn.org/stable/datasets/toy_dataset.html\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">  An example of a synthetic two-class classification dataset is the forge dataset, which has two features\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "caption": "Forge dataset",
    "label": "forge_scatter"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'mglearn' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/l_/499_8k_5387b9tz3wp5g8lg00000gn/T/ipykernel_7509/3542948531.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Matplotlib session?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# generate dataset\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmglearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatasets\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_forge\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;31m# plot dataset\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmglearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiscrete_scatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'mglearn' is not defined"
     ]
    }
   ],
   "source": [
    "# Matplotlib session?\n",
    "# generate dataset\n",
    "X, y = mglearn.datasets.make_forge()\n",
    "# plot dataset\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "plt.legend([\"Class 0\", \"Class 1\"], loc=4)\n",
    "plt.xlabel(\"First feature\")\n",
    "plt.ylabel(\"Second feature\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/l_/499_8k_5387b9tz3wp5g8lg00000gn/T/ipykernel_6015/221398922.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# how many data points do we have?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"X.shape:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'X' is not defined"
     ]
    }
   ],
   "source": [
    "# how many data points do we have?\n",
    "print(\"X.shape:\", X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from X.shape, this dataset consists of 26 data points, with 2 features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> When do we use Scatter plot?\n",
    "\n",
    "The main purpose of a scatter plot is to show how strong the relationship, or correlation, between the two variables is. The tighter the data points fall along a straight line, the higher the correlation (positive, negative, none).\n",
    "\n",
    "Scatter plots' show relationships between two numeric variables. The dots in a scatter plot not only report the values of individual data points, but also patterns when the data are taken as a whole\n",
    "\n",
    "Read more here: https://jhudatascience.org/tidyversecourse/dataviz.html#:~:text=Scatterplots%20are%20helpful,sets%20of%20numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To illustrate example of regression algorithms, we will use the synthetic wave dataset\n",
    "\n",
    "The wave dataset has a single input feature and a continuous target variable (or response) that we want to model\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mglearn.datasets.make_wave(n_samples=40)\n",
    "plt.plot(X, y, 'o')\n",
    "plt.ylim(-3, 3)\n",
    "plt.xlabel(\"Feature\")\n",
    "plt.ylabel(\"Target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: How do we decide what we put on x and y axis in scatter plot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **independent variable** should be plotted on the x axis. The **dependent variable** (the one affected by the independent variable) should be plotted on the y axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above plot, we plot the single feature on the x-axis and the regression target (the output) on the y-axis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Low-dimension dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**low-dimensional datasets** is a dataset with few feautures. For example data with two feauture is easy and can be easily plot with scatter plot. When the dimensionality of the dataset is low, it is possible (and “easy”) for an analyst to look at the interplay of factors in detail, and understand the data before going on to build the model. Doing so can help the analyst identify patterns in the data that may not be that apparent to a machine. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hih-dimension data\n",
    "\n",
    "Low-dimensional datasets is a dataset with many feautures. High dimensional data cannot ploted with Scatterplot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Rwo real-world datasets that are included in scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The previous datasets are synthetic, meaning we created them. <br><br>\n",
    "- We will also used two real-word datasets  included in Scikit-learn <br><br>\n",
    "\n",
    "#### Wisconsis Dataset\n",
    "\n",
    "- Wisconsin Breast Cancer dataset (cancer, for short), which records clinical measurements of breast cancer tumors. Each tumor is labeled as “benign” (for harmless tumors) or “malignant” (for cancerous tumors), and the task is to learn to predict whether a tumor is malignant based on the measurements of the tissue <br><br>\n",
    "- See here : https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The data can be loaded using the load_breast_cancer function from scikit-learn:\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "cancer = load_breast_cancer()\n",
    "cancer.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Datasets that are included in scikit-learn are usually stored as Bunch objects, which contain some information about the dataset as well as the actual data. All you need to know about Bunch objects is that they behave like dictionaries, with the added benefit that you can access values using a dot (as in bunch.key instead of bunch['key'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many feautures and instances do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer.data.shape\n",
    "#The dataset consists of 569 data points, with 30 features each:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the targets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer['target_names']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer.target_names # use list to change to list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are they feautures?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many malignant and beingn do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Sample counts per class:\\n\",\n",
    "      {n: v for n, v in zip(cancer.target_names, np.bincount(cancer.target))})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boston Housing Dataset\n",
    "\n",
    "- We will also be using a real-world regression dataset, the Boston Housing dataset. <br><br>\n",
    "  \n",
    "-  The task associated with this dataset is to predict the median value of homes in several Boston neighborhoods in the 1970s, using information such as crime rate, proximity to the Charles River, highway accessibility, and so on.  <br><br>\n",
    "\n",
    "-  The dataset contains 506 data points, described by 13 features <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the shape of the dataset?, what shape tell us?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Data shape:\", boston.data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For our purposes here, we will actually expand this dataset by not only considering these 13 measurements as input features, but also looking at all products (also called interactions) between features. In other words, we will not only consider crime rate and highway accessibility as features, but also the product of crime rate and highway accessibility. Including derived feature like these is called feature engineering, which we will discuss in more detail in Chapter 4. This derived dataset can be loaded using the load_extended_boston function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mglearn.datasets.load_extended_boston()\n",
    "print(\"X.shape:\", X.shape) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The resulting 104 features are the 13 original features together with the 91 possible combinations of two features within those 13\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Nearest Neighbors : k-Nearest Neighbors\n",
    "\n",
    "- The k-NN algorithm is arguably the simplest machine learning algorithm. The algorithm assumes that similar things exist in close proximity. In other words, similar things are near to each other. \n",
    "   \n",
    "   - “Birds of a feather flock together.”  It just classifies a data point based on its few nearest neighbors. How many neighbors? That is what we decide. <br><br>\n",
    "\n",
    "   - <img src=\"images/knn.jpeg\"> <br><br> \n",
    "\n",
    "- It can be used to solve both classification and regression problems. <br><br>\n",
    "\n",
    "-  KNN for classification predicts a new sample using the idea of similarity or K-closest samples from the training set. “Closeness” is determined by a distance metric, like Euclidean and Minkowsk ( how to calculate distance : https://www.mathsisfun.com/algebra/distance-2-points.html) <br><br>\n",
    "\n",
    "- Building the model consists only of storing the training dataset. <br><br>\n",
    "\n",
    "- Class probability estimates for the new sample are calculated as the proportion of training set neighbors in each class. The new sample’s predicted class is the class with the highest probability estimate; if two or more classes are tied for the highest estimate, then the tie is broken at random or by looking ahead to the K + 1 closest neighbor\n",
    "\n",
    "\n",
    "\n",
    "> For any distance metric, it is important to recall that the original measurement scales of the predictors affect the resulting distance calculations. This implies that if predictors are on widely different scales, the distance value between samples will be biased towards predictors with larger scales. To allow each predictor to contribute equally to the distance calculation, we recommend centering and scaling all predictors prior to performing KNN\n",
    "\n",
    "\n",
    "#### k-Neighbors classification\n",
    "\n",
    "-  In its simplest version, the k-NN algorithm only considers exactly one nearest neighbor, which is the closest training data point to the point we want to make a prediction for.  <br><br>\n",
    "\n",
    "-  The prediction is then simply the known output for this training point. Figure 2-4 illustrates this for the case of classification on the forge dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "# n_neighbors is an example of hyperparameters or model hyperparameters\n",
    "mglearn.plots.plot_knn_classification(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_knn_classification(n_neighbors=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of considering only the closest neighbor, we can also consider an arbitrary number, k, of neighbors. This is where the name of the k-nearest neighbors algorithm comes from. <br><br>\n",
    "\n",
    "When considering more than one neighbor, we use voting to assign a label. This means that for each test point, we count how many neighbors belong to class 0 and how many neighbors belong to class 1. We then assign the class that is more frequent: in other words, the majority class among the k-nearest neighbor\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to apply KNN Algorithm?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# frame the problem\n",
    "# get the data, \n",
    "#prepare the data\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = mglearn.datasets.make_forge() # synthethic dataset with 2 features\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we import and instantiate the class. This is when we can set parameters, like the number of neighbors to use: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "clf = KNeighborsClassifier(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we fit the classifier using the training set. For KNeighborsClassifier this means storing the dataset, so we can compute neighbors during prediction:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make predictions on the test data, we call the predict method. For each data point in the test set, this computes its nearest neighbors in the training set and finds the most common class among these:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is our real y_test, that has been predicted? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test\n",
    "\n",
    "#Classification accuracy : we look at the accuracy of that model as the number of correct predictions from all predictions made.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, we cannot do this for large dataset and go and check manually were the model is correct.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate how well our model generalizes, we can call the score method with the test data together with the test labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.score(X_test, y_test) # returns the mean accuracy, which is a classification metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification metric is a number that measures the performance that your machine learning model when it comes to assigning observations to certain classes. <br><br>\n",
    "\n",
    "Typically the performance is presented on a range from 0 to 1 (though not always) where a score of 1 is reserved for the perfect model. \n",
    "\n",
    "More : https://towardsdatascience.com/the-5-classification-evaluation-metrics-you-must-know-aa97784ff226\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What score funtion does?\n",
    "\n",
    "It calculate the mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred == y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(y_pred == y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that our model is about 86% accurate, meaning the model predicted the class correctly for 86% of the samples in the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Analyzing KNeighborsClassifier\n",
    "\n",
    "- For two-dimensional datasets, we can also illustrate the prediction for all possible test points in the xy-plane.  <br><br>\n",
    "  \n",
    "- We color the plane according to the class that would be assigned to a point in this region.  <br><br>\n",
    "\n",
    "- This lets us view the decision boundary, which is the divide between where the algorithm assigns class 0 versus where it assigns class. <br><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(10, 3))\n",
    "\n",
    "for n_neighbors, ax in zip([1, 3, 9], axes):\n",
    "    # the fit method returns the object self, so we can instantiate\n",
    "    # and fit in one line\n",
    "    clf = KNeighborsClassifier(n_neighbors=n_neighbors).fit(X, y)\n",
    "    mglearn.plots.plot_2d_separator(clf, X, fill=True, eps=0.5, ax=ax, alpha=.4)\n",
    "    mglearn.discrete_scatter(X[:, 0], X[:, 1], y, ax=ax)\n",
    "    ax.set_title(\"{} neighbor(s)\".format(n_neighbors))\n",
    "    ax.set_xlabel(\"feature 0\")\n",
    "    ax.set_ylabel(\"feature 1\")\n",
    "axes[0].legend(loc=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- As you can see on the left in the figure, using a single neighbor results in a decision boundary that follows the training data closely <br><br>\n",
    "\n",
    "- Considering more and more neighbors leads to a smoother decision boundary.  <br><br>\n",
    "\n",
    "- A smoother boundary corresponds to a simpler model. In other words, using few neighbors corresponds to high model complexity (as shown on the right side of Figure 2-1), and <br><br>\n",
    "\n",
    "- using many neighbors corresponds to low model complexity (as shown on the left side of Figure 2-1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question : Question: What is the influence of the hyperparamter n_neighbour??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/train_test_val.png\" style=\"width: 700px;/\"> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let’s investigate whether we can confirm the connection between model complexity and generalization <br><br>\n",
    "\n",
    "- We use a real word breast_cancer dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, stratify=cancer.target, random_state=66)\n",
    "# both train_size and test_size are not available? tes_size will be 0.25 and train_size will be 0.75\n",
    "training_accuracy = []\n",
    "test_accuracy = []\n",
    "# try n_neighbors from 1 to 10\n",
    "neighbors_settings = range(1, 11)\n",
    "\n",
    "for n_neighbors in neighbors_settings:\n",
    "    # build the model\n",
    "    clf = KNeighborsClassifier(n_neighbors=n_neighbors)\n",
    "    clf.fit(X_train, y_train)\n",
    "    # record training set accuracy\n",
    "    training_accuracy.append(clf.score(X_train, y_train))\n",
    "    # record generalization accuracy\n",
    "    test_accuracy.append(clf.score(X_test, y_test))\n",
    "    \n",
    "plt.plot(neighbors_settings, training_accuracy, label=\"training accuracy\")\n",
    "plt.plot(neighbors_settings, test_accuracy, label=\"test accuracy\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.xlabel(\"n_neighbors\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Considering a single nearest neighbor, the prediction on the training set is perfect. But when more neighbors are considered, the model becomes simpler and the training accuracy drops.  <br><br> \n",
    "  \n",
    "- The test set accuracy for using a single neighbor is lower than when using more neighbors, indicating that using the single nearest neighbor leads to a model that is too complex. <br><br> \n",
    "\n",
    "- On the other hand, when considering 10 neighbors, the model is too simple and performance is even worse. <br><br> \n",
    "  \n",
    "- The best performance is somewhere in the middle, using around six neighbors. Still, it is good to keep the scale of the plot in mind. The worst performance is around 88% accuracy, which might still be acceptable <br><br> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### k-neighbors regression\n",
    "\n",
    "- There is also a regression variant of the k-nearest neighbors algorithm.  <br><br>\n",
    "\n",
    "- let’s start by using the single nearest neighbor using the wave dataset.It has a single input feature and a continuous target variable (or response) that we want to model <br><br>\n",
    "\n",
    "- If you have not install mglearn, install here : https://github.com/amueller/mglearn\n",
    " \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mglearn.datasets.make_wave(n_samples=40)\n",
    "plt.plot(X, y, 'o')\n",
    "plt.ylim(-3, 3)\n",
    "plt.xlabel(\"Feature\")\n",
    "plt.ylabel(\"Target\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_knn_regression(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note : we can use more than the single closest neighbor for regression. When using multiple nearest neighbors, the prediction is the average, or mean, of the relevant neighbors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_knn_regression(n_neighbors=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The k-nearest neighbors algorithm for regression \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "X, y = mglearn.datasets.make_wave(n_samples=40)\n",
    "\n",
    "# split the wave dataset into a training and a test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "\n",
    "# instantiate the model and set the number of neighbors to consider to 3\n",
    "reg = KNeighborsRegressor(n_neighbors=3)\n",
    "# fit the model using the training data and training targets\n",
    "reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can make predic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg.score(X_test, y_test)\n",
    "#print(r_squared)\n",
    "#print(f\"Test set R^2:  {r_squared:.2f}\")\n",
    "# more on decimal places using fstring:https://stackoverflow.com/questions/45310254/fixed-digits-after-decimal-with-f-strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The R2 score, also known as the coefficient of determination, is a measure of goodness of a prediction for a regression model, and yields a score between 0 and 1. A value of 1 corresponds to a perfect prediction, and a value of 0 corresponds to a constant model that just predicts the mean of the training set responses, y_train\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score is 0.83, which indicates a relatively good model fit.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analyzing KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
    "# create 1,000 data points, evenly spaced between -3 and 3\n",
    "line = np.linspace(-3, 3, 1000).reshape(-1, 1)\n",
    "for n_neighbors, ax in zip([1, 3, 9], axes):\n",
    "    # make predictions using 1, 3, or 9 neighbors\n",
    "    reg = KNeighborsRegressor(n_neighbors=n_neighbors)\n",
    "    reg.fit(X_train, y_train)\n",
    "    ax.plot(line, reg.predict(line))\n",
    "    ax.plot(X_train, y_train, '^', c=mglearn.cm2(0), markersize=8)\n",
    "    ax.plot(X_test, y_test, 'v', c=mglearn.cm2(1), markersize=8)\n",
    "\n",
    "    ax.set_title(\n",
    "        \"{} neighbor(s)\\n train score: {:.2f} test score: {:.2f}\".format(\n",
    "            n_neighbors, reg.score(X_train, y_train),\n",
    "            reg.score(X_test, y_test)))\n",
    "    ax.set_xlabel(\"Feature\")\n",
    "    ax.set_ylabel(\"Target\")\n",
    "axes[0].legend([\"Model predictions\", \"Training data/target\",\n",
    "                \"Test data/target\"], loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Strengths, weaknesses, and parameters\n",
    "\n",
    "- In principle, there are two important parameters to the KNeighbors classifier: the number of neighbors and how you measure distance between data points.<br><br>\n",
    "\n",
    "- In practice, using a small number of neighbors like three or five often works well, but you should certainly adjust this parameter. Choosing the right distance measure is somewhat beyond the scope of this book. By default, Euclidean distance is used, which works well in many settings. <br><br>\n",
    "  \n",
    "- One of the strengths of k-NN is that the model is very easy to understand, and often gives reasonable performance without a lot of adjustments. Using this algorithm is a good baseline method to try before considering more advanced techniques. <br><br>\n",
    "  \n",
    "- Building the nearest neighbors model is usually very fast, but when your training set is very large (either in number of features or in number of samples) prediction can be slow. <br><br>\n",
    "\n",
    "- When using the k-NN algorithm, it’s important to preprocess your data (see Chapter 3). This approach often does not perform well on datasets with many features (hundreds or more), and it does particularly badly with datasets where most features are 0 most of the time (so-called sparse datasets). <br><br>\n",
    "\n",
    "- So, while the nearest k-neighbors algorithm is easy to understand, it is not often used in practice, due to prediction being slow and its inability to handle many features. The method we discuss next has neither of these drawbacks<br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Models\n",
    "\n",
    "- Linear models are a class of models that are widely used in practice and  make a prediction using a linear function of the input features.   <br><br>\n",
    "- They can be use for both regression and classification  <br><br>\n",
    "  \n",
    "#### Linear models for regression\n",
    "\n",
    "- Linear regression is a linear model, e.g. a model that assumes a linear relationship between the input variables (x) and the single output variable (y). More specifically, that y can be calculated from a linear combination of the input variables (x). <br><br>\n",
    "\n",
    "\n",
    "- The general formula for linear models for regression is :  <br><br>\n",
    " \n",
    "<img src=\"images/linearmodels.png\" style=\"width: 700px;/\"> \n",
    "\n",
    "\n",
    "- Explanatory variables (features), B0...Bn are parameters of the model that are learned and y is the prediction <br><br>\n",
    "\n",
    "- Linear regression was developed in the field of statistics and is studied as a model for understanding the relationship between input and output numerical variables, but has been borrowed by machine learning. It is both a statistical algorithm and a machine learning algorithm <br><br>\n",
    " \n",
    "- When there is a single input variable (x), the method is referred to as simple linear regression. When there are multiple input variables, literature from statistics often refers to the method as multiple linear regression. <br><br>\n",
    "\n",
    "- Different techniques can be used to prepare or train the linear regression equation from data, the most common of which is called Ordinary Least Squares. It is common to therefore refer to a model prepared this way as Ordinary Least Squares Linear Regression or just Least Squares Regression. <br><br>\n",
    "\n",
    "> For datasets with many features, linear models can be very powerful.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "##### Linear models for regression\n",
    "\\begin{align*}\n",
    "\\end{align*}\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_linear_regression_wave()\n",
    "# y = mx + c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Type of Linear Models for Regression\n",
    "\n",
    "- There are many different linear models for regression. The book covers the following: \n",
    "  - Linear regression (aka ordinary least squares)\n",
    "  - Ridge regression\n",
    "  - Lasso <br><br>\n",
    "  \n",
    "-  The difference between these models lies in how :\n",
    "   -  the model parameters B0 (intercept) and Bn (Slope) are learned from the training data, and \n",
    "   -  how model complexity can be controlled.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear regression aka ordinary least squares\n",
    "\n",
    "- Linear regression, or ordinary least squares (OLS), is the simplest and most classic linear method for regression.  <br><br>\n",
    "- It finds the parameters bo and b1 (y = b0 + b1x + e) that minimize the mean squared error between predictions and the true regression targets, y, on the training set.  <br><br>\n",
    "- The mean squared error is the sum of the squared differences between the predictions and the true values. <br><br>\n",
    "- Linear regression has no parameters, which is a benefit, but it also has no way to control model complexity <br><br>\n",
    "- The mathematical formula of the linear regression can be written as y = b0 + b1*x + e, where:\n",
    "\n",
    "    - b0 is the intercept of the regression line; that is the predicted value when x = 0.\n",
    "    - b1 is the slope of the regression line.\n",
    "    - e is the error term (also known as the residual errors), the part of y that can be explained by the regression model\n",
    "The figure below illustrat\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/simple_linear_Reg.png\" style=\"width: 700px;/\"> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "X, y = mglearn.datasets.make_wave(n_samples=60)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
    "\n",
    "lr = LinearRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# slope and intercept are store at : coef_ and ntercept_ aattributes\n",
    "\n",
    "print(\"lr.coef_:\", lr.coef_)\n",
    "print(\"lr.intercept_:\", lr.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intercept_ attribute is always a single float number, while the coef_ attribute is a NumPy array with one entry per input feature. As we only have a single input feature in the wave dataset, lr.coef_ only has a single entry\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You might notice the strange-looking trailing underscore at the end of coef_ and intercept_. scikit-learn always stores anything that is derived from the training data in attributes that end with a trailing underscore. That is to separate them from parameters that are set by the user.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training set score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test set score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">  R squared is the measure of goodness of fit of the model. It is also known as the Coefficient of Determination, is a value between 0 and 1 that measures how well our regression line fits our data. The closer R-Squared is to 1 or 100% the better our model will be at predicting our dependent variable : https://medium.com/@erika.dauria/looking-at-r-squared-721252709098"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- An R2 of around 0.66 is not very good, but we can see that the scores on the training and test sets are very close together. This means we are likely underfitting, not overfitting. For this one-dimensional dataset, there is little danger of overfitting, as the model is very simple (or restricted). \n",
    "\n",
    "- However, with higher-dimensional datasets (meaning datasets with a large number of features), linear models become more powerful, and there is a higher chance of overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear  Regression performance on a more complex dataset, like the Boston Housing dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mglearn.datasets.load_extended_boston()\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "lr = LinearRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Training set score: {:.2f}\".format(lr.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(lr.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This discrepancy between performance on the training set and the test set is a clear sign of overfitting, and therefore we should try to find a model that allows us to control complexity. One of the most commonly used alternatives to standard linear regression is ridge regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An extension to linear regression invokes adding penalties to the loss function during training that encourages simpler models that have smaller coefficient values. These extensions are referred to as regularized linear regression or penalized linear regression. We will see some of them next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Ridge regression\n",
    "\n",
    "Ridge Regression is a popular type of regularized linear regression that includes an L2 penalty. This has the effect of shrinking the coefficients for those input variables that do not contribute much to the prediction task.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "ridge = Ridge().fit(X_train, y_train) # default alpha = 1 (L2)\n",
    "print(\"Training set score: {:.2f}\".format(ridge.score(X_train, y_train))) # remember we are using the same Boston Dataset\n",
    "print(\"Test set score: {:.2f}\".format(ridge.score(X_test, y_test)))\n",
    "\n",
    "#https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- As you can see, the training set score of Ridge is lower than for LinearRegression, while the test set score is higher. With linear regression, we were overfitting our data\n",
    "\n",
    "- But, Ridge is a more restricted model, so we are less likely to overfit\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> A less complex model means worse performance on the training set, but better generalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Increasing alpha decreases training set performance but might help generalization <br><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ridge10 = Ridge(alpha=10).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.2f}\".format(ridge10.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(ridge10.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For very small values of alpha, coefficients are barely restricted at all, and we end up with a model that resembles LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ridge01 = Ridge(alpha= 1).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.2f}\".format(ridge01.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(ridge01.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, how do we choose Alpha? Scikit-learn includes a RidgeCV method that allows us select the ideal value for alpha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(ridge.coef_, 's', label=\"Ridge alpha=1\")\n",
    "plt.plot(ridge10.coef_, '^', label=\"Ridge alpha=10\")\n",
    "plt.plot(ridge01.coef_, 'v', label=\"Ridge alpha=0.1\")\n",
    "\n",
    "plt.plot(lr.coef_, 'o', label=\"LinearRegression\")\n",
    "plt.xlabel(\"Coefficient index\")\n",
    "plt.ylabel(\"Coefficient magnitude\")\n",
    "xlims = plt.xlim()\n",
    "plt.hlines(0, xlims[0], xlims[1])\n",
    "plt.xlim(xlims)\n",
    "plt.ylim(-25, 25)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mglearn.plots.plot_ridge_n_samples()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The lesson here is that with enough training data, regularization becomes less important, and given enough data, ridge and linear regression will have the same performance. Also, If more data is added, it becomes harder for a model to overfit, or memorize the data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Lasso\n",
    "\n",
    "- Lasso Regression is a popular type of regularized linear regression that use L1 penalty   <br><br>\n",
    "- As with ridge regression, using the lasso also restricts coefficients to be close to zero, but in a slightly different way, called L1 regularization <br><br>\n",
    "- The consequence of L1 regularization is that when using the lasso, some coefficients are exactly zero. This means some features are entirely ignored by the model. This can be seen as a form of automatic feature selection. Having some coefficients be exactly zero often makes a model easier to interpret, and can reveal the most important features of your model  <br><br>\n",
    "  \n",
    "- An L1 penalty minimizes the size of all coefficients and allows some coefficients to be minimized to the value zero, which removes the predictor from the model. <br><br>\n",
    "-  L1 and L2 Intuitive diff : https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "lasso = Lasso(alpha=0.1).fit(X_train, y_train) # default alpha = 1.0, and alpha = 0 is equivalent to an ordinary least square\n",
    "print(\"Training set score: {:.2f}\".format(lasso.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(lasso.score(X_test, y_test)))\n",
    "print(\"Number of features used:\", np.sum(lasso.coef_ != 0))\n",
    "#https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As you can see, Lasso does quite badly, both on the training and the test set. This indicates that we are underfitting, and we find that it used only 4 of the 105 features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we change alpha = 0?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to Ridge, Lasso has a regularization parameter, alpha, that controls how strongly coefficients are pushed toward zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we increase the default setting of \"max_iter\",\n",
    "# otherwise the model would warn us that we should increase max_iter.\n",
    "lasso001 = Lasso(alpha=0.01, max_iter=100000).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.2f}\".format(lasso001.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(lasso001.score(X_test, y_test)))\n",
    "print(\"Number of features used:\", np.sum(lasso001.coef_ != 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> A lower alpha allowed us to fit a more complex model, which worked better on the training and test data. The performance is slightly better than using Ridge, and we are using only 33 of the 105 features. This makes this model potentially easier to understand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we set alpha too low, however, we again remove the effect of regularization and end up overfitting, with a result similar to LinearRegression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso00001 = Lasso(alpha=0.0001, max_iter=100000).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.2f}\".format(lasso00001.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(lasso00001.score(X_test, y_test)))\n",
    "print(\"Number of features used:\", np.sum(lasso00001.coef_ != 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In practice, ridge regression is usually the first choice between these two models. However, if you have a large amount of features and expect only a few of them to be important, Lasso might be a better choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(lasso.coef_, 's', label=\"Lasso alpha=1\")\n",
    "plt.plot(lasso001.coef_, '^', label=\"Lasso alpha=0.01\")\n",
    "plt.plot(lasso00001.coef_, 'v', label=\"Lasso alpha=0.0001\")\n",
    "\n",
    "plt.plot(ridge01.coef_, 'o', label=\"Ridge alpha=0.1\")\n",
    "plt.legend(ncol=2, loc=(0, 1.05))\n",
    "plt.ylim(-25, 25)\n",
    "plt.xlabel(\"Coefficient index\")\n",
    "plt.ylabel(\"Coefficient magnitude\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Linear models for classification\n",
    "\n",
    "- Linear models are also extensively used for classification <br><br>\n",
    "\n",
    "- For linear models for regression, the output, ŷ, is a linear function of the features: a line (𝑎𝑥+𝑏𝑦=𝑐), plane (𝑎𝑥+𝑏𝑦+𝑐𝑧=𝑑), or hyperplane (in higher dimensions) <br><br>\n",
    "- For linear models for classification, the decision boundary is a linear function of the input. a (binary) linear classifier is a classifier that separates two classes using a line, a plane, or a hyperplane. <br><br>\n",
    "- There are many algorithms for learning linear models. These algorithms all differ in the following two ways\n",
    "  - The way in which they measure how well a particular combination of coefficients and intercept fits the training data \n",
    "  - If and what kind of regularization they use <br><br>\n",
    "- The two most common linear classification algorithms are logistic regression, implemented in linear_model.LogisticRegression, and linear support vector machines (linear SVMs), implemented in svm.LinearSVC (SVC stands for support vector classifier).<br><br>\n",
    "- Despite its name, LogisticRegression is a classification algorithm and not a regression algorithm, and it should not be confused with LinearRegression <br><br>\n",
    "- More examples from Sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LogisticRegression and LinearSVC models to the forge dataset,\n",
    "\n",
    "- Let us visualize the decision boundary of the two models\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "X, y = mglearn.datasets.make_forge()\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 3))\n",
    "\n",
    "for model, ax in zip([LinearSVC(), LogisticRegression()], axes):\n",
    "    clf = model.fit(X, y)\n",
    "    mglearn.plots.plot_2d_separator(clf, X, fill=False, eps=0.5,\n",
    "                                    ax=ax, alpha=.7)\n",
    "    mglearn.discrete_scatter(X[:, 0], X[:, 1], y, ax=ax)\n",
    "    ax.set_title(clf.__class__.__name__)\n",
    "    ax.set_xlabel(\"Feature 0\")\n",
    "    ax.set_ylabel(\"Feature 1\")\n",
    "axes[0].legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- From above any new data point that lies above the black line will be classified as class 1 by the respective classifier, while any point that lies below the black line will be classified as class 0. <br><br>\n",
    "\n",
    "- The two models come up with similar decision boundaries. Note that both misclassify two of the points. By default, both models apply an L2 regularization, in the same way that Ridge does for regression  <br><br>\n",
    "\n",
    "- For LogisticRegression and LinearSVC the trade-off parameter that determines the strength of the regularization is called C, and higher values of C correspond to les regularization. <br><br>\n",
    "\n",
    "- So, when you use a high value for the parameter C, LogisticRegression and LinearSVC try to fit the training set as best as possible, while with low values of the parameter C, the models put more emphasis on finding a coefficient vector (w) that is close to zero. <br><br>\n",
    " \n",
    "#### Effect of different configuration of C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_linear_svc_regularization()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- On the lefthand side, we have a very small C corresponding to a lot of regularization.  The strongly regularized model chooses a relatively horizontal line, misclassifying two points. <br><br>\n",
    "- In the center plot, C is slightly higher, and the model focuses more on the two misclassified samples, tilting the decision boundary. <br><br>\n",
    "- on the righthand side, the very high value of C in the model tilts the decision boundary a lot, now correctly classifying all points in class 0. One of the points in class 1 is still misclassified, as it is not possible to correctly classify all points in this dataset using a straight line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The model on the righthand side tries hard to correctly classify all points, but might not capture the overall layout of the classes well. In other words, this model is likely overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Similar to Linear Regression, in high dimensions, linear models for classification become very powerful, and guarding against overfitting becomes increasingly important when considering more features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LinearLogistic on the Breast Cancer dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "cancer = load_breast_cancer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, stratify=cancer.target, random_state=42)\n",
    "logreg = LogisticRegression().fit(X_train, y_train)\n",
    "print(\"Training set score: {:.3f}\".format(logreg.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.3f}\".format(logreg.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default value of C=1 provides quite good performance. But as training and test set performance are very close, it is likely that we are underfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s try to increase C to fit a more flexible model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg100 = LogisticRegression(C=100).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.2f}\".format(logreg100.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.2f}\".format(logreg100.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"isa\"\n",
    "#'Hello, %s' % name\n",
    "#'Hello, {}'.format(name)\n",
    "f'Hello, {name}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg001 = LogisticRegression(C=0.001).fit(X_train, y_train)\n",
    "print(\"Training set score: {:.3f}\".format(logreg001.score(X_train, y_train)))\n",
    "print(\"Test set score: {:.3f}\".format(logreg001.score(X_test, y_test)))\n",
    "# Read about string formating: https://realpython.com/python-string-formatting/#3-string-interpolation-f-strings-python-36"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reducing the value of C from an already underfit model, both training and test set accuracy decrease relative to the default parameters.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Linear models for multiclass classification\n",
    "\\begin{align*}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_blobs\n",
    "\n",
    "X, y = make_blobs(random_state=42)\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")\n",
    "plt.legend([\"Class 0\", \"Class 1\", \"Class 2\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_svm = LinearSVC().fit(X, y)\n",
    "print(\"Coefficient shape: \", linear_svm.coef_.shape)\n",
    "print(\"Intercept shape: \", linear_svm.intercept_.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "line = np.linspace(-15, 15)\n",
    "for coef, intercept, color in zip(linear_svm.coef_, linear_svm.intercept_,\n",
    "                                  mglearn.cm3.colors):\n",
    "    plt.plot(line, -(line * coef[0] + intercept) / coef[1], c=color)\n",
    "plt.ylim(-10, 15)\n",
    "plt.xlim(-10, 8)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")\n",
    "plt.legend(['Class 0', 'Class 1', 'Class 2', 'Line class 0', 'Line class 1',\n",
    "            'Line class 2'], loc=(1.01, 0.3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mglearn.plots.plot_2d_classification(linear_svm, X, fill=True, alpha=.7)\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "line = np.linspace(-15, 15)\n",
    "for coef, intercept, color in zip(linear_svm.coef_, linear_svm.intercept_,\n",
    "                                  mglearn.cm3.colors):\n",
    "    plt.plot(line, -(line * coef[0] + intercept) / coef[1], c=color)\n",
    "plt.legend(['Class 0', 'Class 1', 'Class 2', 'Line class 0', 'Line class 1',\n",
    "            'Line class 2'], loc=(1.01, 0.3))\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summary, Strengths, weaknesses and parameters\n",
    "\n",
    "- The main parameter of linear models is the regularization parameter, called alpha in the regression models and C in LinearSVC and LogisticRegression. <br><br>\n",
    "- The other decision you have to make is whether you want to use L1 regularization or L2 regularization. If you assume that only a few of your features are actually important, you should use L1 Otherwise, you should default to L2. <br><br>\n",
    "- L1 can also be useful if interpretability of the model is important. As L1 will use only a few features, it is easier to explain which features are important to the model, and what the effects of these features are. <br><br>\n",
    "- Linear models are very fast to train, and also fast to predict. They scale to very large datasets and work well with sparse data <br><br>\n",
    "- Other options are the SGDClassifier class and the SGDRegressor class, which implement even more scalable versions of the linear models described here. <br><br>\n",
    "- Another strength of linear models is that they make it relatively easy to understand how a prediction is made, using the formulas we saw earlier for regression and classification. <br><br>\n",
    "- Linear models often perform well when the number of features is large compared to the number of samples <br><br>\n",
    "- They are also often used on very large datasets, simply because it’s not feasible to train other models <br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method Chaining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiating and fitting models using one liner below is called method chaining "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate model\n",
    "logreg = LogisticRegression()\n",
    "#fitting the model\n",
    "logreg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate model and fit it in one line\n",
    "logreg = LogisticRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another common application of method chaining in scikit-learn is to fit and predict in one line\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg = LogisticRegression()\n",
    "y_pred = logreg.fit(X_train, y_train).predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even do model instantiation, fitting, and predicting in one line\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = LogisticRegression().fit(X_train, y_train).predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This very short variant is not ideal, though. A lot is happening in a single line, which might make the code hard to read. Additionally, the fitted logistic regression model isn’t stored in any variable, so we can’t inspect it or use it to predict on any other data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes Classifiers\n",
    "\n",
    "- Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other. <br><br>\n",
    "\n",
    "- This algorithm is called “Naive” because it makes a naive assumption that each feature is independent of other features which is not true in real life. <br><br>\n",
    "- As for the “Bayes” part, it refers to the statistician and philosopher, Thomas Bayes and the theorem named after him, Bayes’ theorem, which is the base for Naive Bayes Algorithm. <br><br>\n",
    "- So, Naive Bayes classifiers are supervised classification algorithm which is based on Bayes theorem with an assumption of independence among features <br><br>\n",
    "- It can be use for  text classification e.g pam Filtering in Email, Sentiment analyis, Recommener System <br><br>\n",
    "- The reason that naive Bayes models are so efficient is that they learn parameters by looking at each feature individually and collect simple per-class statistics from each feature <br><br>\n",
    "- There are three kinds of naive Bayes classifiers implemented in scikit-\n",
    "GaussianNB, BernoulliNB, and MultinomialNB. GaussianNB can be applied to any continuous data, while BernoulliNB assumes binary data and MultinomialNB assumes count data (that is, that each feature represents an integer count of something, like how often a word appears in a sentence). BernoulliNB and MultinomialNB are mostly used in text data classification <br><br>\n",
    "\n",
    "- Example : https://medium.com/@srishtisawla/introduction-to-naive-bayes-for-classification-baefefb43a2d <br><br>\n",
    "- Sklearn has examples for each alg: https://scikit-learn.org/stable/modules/naive_bayes.html\n",
    "\n",
    "- See : https://scikit-learn.org/stable/auto_examples/index.html#classification\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "X, y = load_iris(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)\n",
    "gnb = GaussianNB()\n",
    "y_pred = gnb.fit(X_train, y_train).predict(X_test)\n",
    "print(\"Number of mislabeled points out of a total %d points : %d\"  % (X_test.shape[0], (y_test != y_pred).sum()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gnb.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strengths, weaknesses and parameters\n",
    "\n",
    "- The naive Bayes models share many of the strengths and weaknesses of the linear models. They are very fast to train and to predict, and the training procedure is easy to understand. The models work very well with high-dimensional sparse data and are relatively robust to the parameters.  <br><br>\n",
    "\n",
    "- Naive Bayes models are great baseline models and are often used on very large datasets, where training even a linear model might take too long. <br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Decision trees are a powerful prediction method and extremely popular. <br><br>\n",
    "\n",
    "- They are popular because the final model is so easy to understand by practitioners and domain experts alike. The final decision tree can explain exactly why a specific prediction was made, making it very attractive for operational use. <br><br>\n",
    "\n",
    "- Decision trees also provide the foundation for more advanced ensemble methods such as bagging, random forests and gradient boosting <br><br>\n",
    "- There are many algorithms that implement decision tree alg. Classification and Regression Trees or CART for short is an acronym introduced by Leo Breiman to refer to Decision Tree algorithms that can be used for classification or regression predictive modeling problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How they work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "import graphviz\n",
    "mglearn.plots.plot_animal_tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Building decision trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_tree_progressive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Controlling complexity of decision trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "6e5d7a76-9bba-42f7-b26e-907775d289b2"
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, stratify=cancer.target, random_state=42)\n",
    "tree = DecisionTreeClassifier(random_state=0)\n",
    "tree.fit(X_train, y_train)\n",
    "print(\"Accuracy on training set: {:.3f}\".format(tree.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(tree.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = DecisionTreeClassifier(max_depth=4, random_state=0)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(tree.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(tree.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analyzing Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "export_graphviz(tree, out_file=\"tree.dot\", class_names=[\"malignant\", \"benign\"],\n",
    "                feature_names=cancer.feature_names, impurity=False, filled=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import graphviz\n",
    "\n",
    "with open(\"tree.dot\") as f:\n",
    "    dot_graph = f.read()\n",
    "display(graphviz.Source(dot_graph))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature Importance in trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "dc2f68ee-0df0-47ed-b500-7ec99d5a0a5d"
   },
   "outputs": [],
   "source": [
    "print(\"Feature importances:\")\n",
    "print(tree.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_feature_importances_cancer(model):\n",
    "    n_features = cancer.data.shape[1]\n",
    "    plt.barh(np.arange(n_features), model.feature_importances_, align='center')\n",
    "    plt.yticks(np.arange(n_features), cancer.feature_names)\n",
    "    plt.xlabel(\"Feature importance\")\n",
    "    plt.ylabel(\"Feature\")\n",
    "    plt.ylim(-1, n_features)\n",
    "\n",
    "plot_feature_importances_cancer(tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "tree = mglearn.plots.plot_tree_not_monotone()\n",
    "display(tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "ram_prices = pd.read_csv(os.path.join(mglearn.datasets.DATA_PATH, \"ram_price.csv\"))\n",
    "\n",
    "plt.semilogy(ram_prices.date, ram_prices.price)\n",
    "plt.xlabel(\"Year\")\n",
    "plt.ylabel(\"Price in $/Mbyte\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "# use historical data to forecast prices after the year 2000\n",
    "data_train = ram_prices[ram_prices.date < 2000]\n",
    "data_test = ram_prices[ram_prices.date >= 2000]\n",
    "\n",
    "# predict prices based on date\n",
    "X_train = data_train.date[:, np.newaxis]\n",
    "# we use a log-transform to get a simpler relationship of data to target\n",
    "y_train = np.log(data_train.price)\n",
    "\n",
    "tree = DecisionTreeRegressor(max_depth=3).fit(X_train, y_train)\n",
    "linear_reg = LinearRegression().fit(X_train, y_train)\n",
    "\n",
    "# predict on all data\n",
    "X_all = ram_prices.date[:, np.newaxis]\n",
    "\n",
    "pred_tree = tree.predict(X_all)\n",
    "pred_lr = linear_reg.predict(X_all)\n",
    "\n",
    "# undo log-transform\n",
    "price_tree = np.exp(pred_tree)\n",
    "price_lr = np.exp(pred_lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.semilogy(data_train.date, data_train.price, label=\"Training data\")\n",
    "plt.semilogy(data_test.date, data_test.price, label=\"Test data\")\n",
    "plt.semilogy(ram_prices.date, price_tree, label=\"Tree prediction\")\n",
    "plt.semilogy(ram_prices.date, price_lr, label=\"Linear prediction\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strengths, weaknesses and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ensembles of Decision Trees\n",
    "##### Random forests\n",
    "###### Building random forests\n",
    "###### Analyzing random forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "b84dcdfe-994f-4a3d-842e-830153eefc59"
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X, y = make_moons(n_samples=100, noise=0.25, random_state=3)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,\n",
    "                                                    random_state=42)\n",
    "\n",
    "forest = RandomForestClassifier(n_estimators=5, random_state=2)\n",
    "forest.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "76ce4154-b441-475e-97e3-1b507964eb29"
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 3, figsize=(20, 10))\n",
    "for i, (ax, tree) in enumerate(zip(axes.ravel(), forest.estimators_)):\n",
    "    ax.set_title(\"Tree {}\".format(i))\n",
    "    mglearn.plots.plot_tree_partition(X_train, y_train, tree, ax=ax)\n",
    "    \n",
    "mglearn.plots.plot_2d_separator(forest, X_train, fill=True, ax=axes[-1, -1],\n",
    "                                alpha=.4)\n",
    "axes[-1, -1].set_title(\"Random Forest\")\n",
    "mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0)\n",
    "forest = RandomForestClassifier(n_estimators=100, random_state=0)\n",
    "forest.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(forest.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(forest.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_feature_importances_cancer(forest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Strengths, weaknesses, and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient Boosted Regression Trees (Gradient Boosting Machines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0)\n",
    "\n",
    "gbrt = GradientBoostingClassifier(random_state=0)\n",
    "gbrt.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(gbrt.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(gbrt.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbrt = GradientBoostingClassifier(random_state=0, max_depth=1)\n",
    "gbrt.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(gbrt.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(gbrt.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbrt = GradientBoostingClassifier(random_state=0, learning_rate=0.01)\n",
    "gbrt.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(gbrt.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(gbrt.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbrt = GradientBoostingClassifier(random_state=0, max_depth=1)\n",
    "gbrt.fit(X_train, y_train)\n",
    "\n",
    "plot_feature_importances_cancer(gbrt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Strengths, weaknesses and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Kernelized Support Vector Machines\n",
    "#### Linear Models and Non-linear Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_blobs(centers=4, random_state=8)\n",
    "y = y % 2\n",
    "\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "linear_svm = LinearSVC().fit(X, y)\n",
    "\n",
    "mglearn.plots.plot_2d_separator(linear_svm, X)\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the squared first feature\n",
    "X_new = np.hstack([X, X[:, 1:] ** 2])\n",
    "\n",
    "\n",
    "from mpl_toolkits.mplot3d import Axes3D, axes3d\n",
    "figure = plt.figure()\n",
    "# visualize in 3D\n",
    "ax = Axes3D(figure, elev=-152, azim=-26)\n",
    "# plot first all the points with y==0, then all with y == 1\n",
    "mask = y == 0\n",
    "ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',\n",
    "           cmap=mglearn.cm2, s=60, edgecolor='k')\n",
    "ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',\n",
    "           cmap=mglearn.cm2, s=60, edgecolor='k')\n",
    "ax.set_xlabel(\"feature0\")\n",
    "ax.set_ylabel(\"feature1\")\n",
    "ax.set_zlabel(\"feature1 ** 2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_svm_3d = LinearSVC().fit(X_new, y)\n",
    "coef, intercept = linear_svm_3d.coef_.ravel(), linear_svm_3d.intercept_\n",
    "\n",
    "# show linear decision boundary\n",
    "figure = plt.figure()\n",
    "ax = Axes3D(figure, elev=-152, azim=-26)\n",
    "xx = np.linspace(X_new[:, 0].min() - 2, X_new[:, 0].max() + 2, 50)\n",
    "yy = np.linspace(X_new[:, 1].min() - 2, X_new[:, 1].max() + 2, 50)\n",
    "\n",
    "XX, YY = np.meshgrid(xx, yy)\n",
    "ZZ = (coef[0] * XX + coef[1] * YY + intercept) / -coef[2]\n",
    "ax.plot_surface(XX, YY, ZZ, rstride=8, cstride=8, alpha=0.3)\n",
    "ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',\n",
    "           cmap=mglearn.cm2, s=60, edgecolor='k')\n",
    "ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',\n",
    "           cmap=mglearn.cm2, s=60, edgecolor='k')\n",
    "\n",
    "ax.set_xlabel(\"feature0\")\n",
    "ax.set_ylabel(\"feature1\")\n",
    "ax.set_zlabel(\"feature1 ** 2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ZZ = YY ** 2\n",
    "dec = linear_svm_3d.decision_function(np.c_[XX.ravel(), YY.ravel(), ZZ.ravel()])\n",
    "plt.contourf(XX, YY, dec.reshape(XX.shape), levels=[dec.min(), 0, dec.max()],\n",
    "             cmap=mglearn.cm2, alpha=0.5)\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Kernel Trick\n",
    "#### Understanding SVMs\n",
    "\\begin{align*}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "X, y = mglearn.tools.make_handcrafted_dataset()                                                                  \n",
    "svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)\n",
    "mglearn.plots.plot_2d_separator(svm, X, eps=.5)\n",
    "mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
    "# plot support vectors\n",
    "sv = svm.support_vectors_\n",
    "# class labels of support vectors are given by the sign of the dual coefficients\n",
    "sv_labels = svm.dual_coef_.ravel() > 0\n",
    "mglearn.discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tuning SVM parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(3, 3, figsize=(15, 10))\n",
    "\n",
    "for ax, C in zip(axes, [-1, 0, 3]):\n",
    "    for a, gamma in zip(ax, range(-1, 2)):\n",
    "        mglearn.plots.plot_svm(log_C=C, log_gamma=gamma, ax=a)\n",
    "        \n",
    "axes[0, 0].legend([\"class 0\", \"class 1\", \"sv class 0\", \"sv class 1\"],\n",
    "                  ncol=4, loc=(.9, 1.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "19dca39b-9061-4fc6-9aab-f759854ec208"
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0)\n",
    "\n",
    "svc = SVC()\n",
    "svc.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.2f}\".format(svc.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.2f}\".format(svc.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.boxplot(X_train, manage_ticks=False)\n",
    "plt.yscale(\"symlog\")\n",
    "plt.xlabel(\"Feature index\")\n",
    "plt.ylabel(\"Feature magnitude\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Preprocessing data for SVMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the minimum value per feature on the training set\n",
    "min_on_training = X_train.min(axis=0)\n",
    "# Compute the range of each feature (max - min) on the training set\n",
    "range_on_training = (X_train - min_on_training).max(axis=0)\n",
    "\n",
    "# subtract the min, divide by range\n",
    "# afterward, min=0 and max=1 for each feature\n",
    "X_train_scaled = (X_train - min_on_training) / range_on_training\n",
    "print(\"Minimum for each feature\\n\", X_train_scaled.min(axis=0))\n",
    "print(\"Maximum for each feature\\n\", X_train_scaled.max(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use THE SAME transformation on the test set,\n",
    "# using min and range of the training set. See Chapter 3 (unsupervised learning) for details.\n",
    "X_test_scaled = (X_test - min_on_training) / range_on_training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = SVC()\n",
    "svc.fit(X_train_scaled, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(\n",
    "        svc.score(X_train_scaled, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(svc.score(X_test_scaled, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = SVC(C=1000)\n",
    "svc.fit(X_train_scaled, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(\n",
    "    svc.score(X_train_scaled, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(svc.score(X_test_scaled, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strengths, weaknesses and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Networks (Deep Learning)\n",
    "#### The Neural Network Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(mglearn.plots.plot_logistic_regression_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(mglearn.plots.plot_single_hidden_layer_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = np.linspace(-3, 3, 100)\n",
    "plt.plot(line, np.tanh(line), label=\"tanh\")\n",
    "plt.plot(line, np.maximum(line, 0), label=\"relu\")\n",
    "plt.legend(loc=\"best\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"relu(x), tanh(x)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "mglearn.plots.plot_two_hidden_layer_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tuning Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X, y = make_moons(n_samples=100, noise=0.25, random_state=3)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,\n",
    "                                                    random_state=42)\n",
    "\n",
    "mlp = MLPClassifier(solver='lbfgs', random_state=0).fit(X_train, y_train)\n",
    "mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3)\n",
    "mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp = MLPClassifier(solver='lbfgs', random_state=0, hidden_layer_sizes=[10])\n",
    "mlp.fit(X_train, y_train)\n",
    "mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3)\n",
    "mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using two hidden layers, with 10 units each\n",
    "mlp = MLPClassifier(solver='lbfgs', random_state=0,\n",
    "                    hidden_layer_sizes=[10, 10])\n",
    "mlp.fit(X_train, y_train)\n",
    "mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3)\n",
    "mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using two hidden layers, with 10 units each, now with tanh nonlinearity.\n",
    "mlp = MLPClassifier(solver='lbfgs', activation='tanh',\n",
    "                    random_state=0, hidden_layer_sizes=[10, 10])\n",
    "mlp.fit(X_train, y_train)\n",
    "mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3)\n",
    "mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 4, figsize=(20, 8))\n",
    "for axx, n_hidden_nodes in zip(axes, [10, 100]):\n",
    "    for ax, alpha in zip(axx, [0.0001, 0.01, 0.1, 1]):\n",
    "        mlp = MLPClassifier(solver='lbfgs', random_state=0,\n",
    "                            hidden_layer_sizes=[n_hidden_nodes, n_hidden_nodes],\n",
    "                            alpha=alpha)\n",
    "        mlp.fit(X_train, y_train)\n",
    "        mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3, ax=ax)\n",
    "        mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train, ax=ax)\n",
    "        ax.set_title(\"n_hidden=[{}, {}]\\nalpha={:.4f}\".format(\n",
    "                      n_hidden_nodes, n_hidden_nodes, alpha))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 4, figsize=(20, 8))\n",
    "for i, ax in enumerate(axes.ravel()):\n",
    "    mlp = MLPClassifier(solver='lbfgs', random_state=i,\n",
    "                        hidden_layer_sizes=[100, 100])\n",
    "    mlp.fit(X_train, y_train)\n",
    "    mglearn.plots.plot_2d_separator(mlp, X_train, fill=True, alpha=.3, ax=ax)\n",
    "    mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train, ax=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Cancer data per-feature maxima:\\n{}\".format(cancer.data.max(axis=0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0)\n",
    "\n",
    "mlp = MLPClassifier(random_state=42)\n",
    "mlp.fit(X_train, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.2f}\".format(mlp.score(X_train, y_train)))\n",
    "print(\"Accuracy on test set: {:.2f}\".format(mlp.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute the mean value per feature on the training set\n",
    "mean_on_train = X_train.mean(axis=0)\n",
    "# compute the standard deviation of each feature on the training set\n",
    "std_on_train = X_train.std(axis=0)\n",
    "\n",
    "# subtract the mean, and scale by inverse standard deviation\n",
    "# afterward, mean=0 and std=1\n",
    "X_train_scaled = (X_train - mean_on_train) / std_on_train\n",
    "# use THE SAME transformation (using training mean and std) on the test set\n",
    "X_test_scaled = (X_test - mean_on_train) / std_on_train\n",
    "\n",
    "mlp = MLPClassifier(random_state=0)\n",
    "mlp.fit(X_train_scaled, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(\n",
    "    mlp.score(X_train_scaled, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(mlp.score(X_test_scaled, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp = MLPClassifier(max_iter=1000, random_state=0)\n",
    "mlp.fit(X_train_scaled, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(\n",
    "    mlp.score(X_train_scaled, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(mlp.score(X_test_scaled, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp = MLPClassifier(max_iter=1000, alpha=1, random_state=0)\n",
    "mlp.fit(X_train_scaled, y_train)\n",
    "\n",
    "print(\"Accuracy on training set: {:.3f}\".format(\n",
    "    mlp.score(X_train_scaled, y_train)))\n",
    "print(\"Accuracy on test set: {:.3f}\".format(mlp.score(X_test_scaled, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 5))\n",
    "plt.imshow(mlp.coefs_[0], interpolation='none', cmap='viridis')\n",
    "plt.yticks(range(30), cancer.feature_names)\n",
    "plt.xlabel(\"Columns in weight matrix\")\n",
    "plt.ylabel(\"Input feature\")\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strengths, weaknesses and parameters\n",
    "##### Estimating complexity in neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uncertainty estimates from classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.datasets import make_circles\n",
    "X, y = make_circles(noise=0.25, factor=0.5, random_state=1)\n",
    "\n",
    "# we rename the classes \"blue\" and \"red\" for illustration purposes:\n",
    "y_named = np.array([\"blue\", \"red\"])[y]\n",
    "\n",
    "# we can call train_test_split with arbitrarily many arrays;\n",
    "# all will be split in a consistent manner\n",
    "X_train, X_test, y_train_named, y_test_named, y_train, y_test = \\\n",
    "    train_test_split(X, y_named, y, random_state=0)\n",
    "\n",
    "# build the gradient boosting model\n",
    "gbrt = GradientBoostingClassifier(random_state=0)\n",
    "gbrt.fit(X_train, y_train_named)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Decision Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"X_test.shape:\", X_test.shape)\n",
    "print(\"Decision function shape:\",\n",
    "      gbrt.decision_function(X_test).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the first few entries of decision_function\n",
    "print(\"Decision function:\", gbrt.decision_function(X_test)[:6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Thresholded decision function:\\n\",\n",
    "      gbrt.decision_function(X_test) > 0)\n",
    "print(\"Predictions:\\n\", gbrt.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make the boolean True/False into 0 and 1\n",
    "greater_zero = (gbrt.decision_function(X_test) > 0).astype(int)\n",
    "# use 0 and 1 as indices into classes_\n",
    "pred = gbrt.classes_[greater_zero]\n",
    "# pred is the same as the output of gbrt.predict\n",
    "print(\"pred is equal to predictions:\",\n",
    "      np.all(pred == gbrt.predict(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decision_function = gbrt.decision_function(X_test)\n",
    "print(\"Decision function minimum: {:.2f} maximum: {:.2f}\".format(\n",
    "      np.min(decision_function), np.max(decision_function)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(13, 5))\n",
    "mglearn.tools.plot_2d_separator(gbrt, X, ax=axes[0], alpha=.4,\n",
    "                                fill=True, cm=mglearn.cm2)\n",
    "scores_image = mglearn.tools.plot_2d_scores(gbrt, X, ax=axes[1],\n",
    "                                            alpha=.4, cm=mglearn.ReBl)\n",
    "\n",
    "for ax in axes:\n",
    "    # plot training and test points\n",
    "    mglearn.discrete_scatter(X_test[:, 0], X_test[:, 1], y_test,\n",
    "                             markers='^', ax=ax)\n",
    "    mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train,\n",
    "                             markers='o', ax=ax)\n",
    "    ax.set_xlabel(\"Feature 0\")\n",
    "    ax.set_ylabel(\"Feature 1\")\n",
    "cbar = plt.colorbar(scores_image, ax=axes.tolist())\n",
    "cbar.set_alpha(1)\n",
    "cbar.draw_all()\n",
    "axes[0].legend([\"Test class 0\", \"Test class 1\", \"Train class 0\",\n",
    "                \"Train class 1\"], ncol=4, loc=(.1, 1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predicting Probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Shape of probabilities:\", gbrt.predict_proba(X_test).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the first few entries of predict_proba\n",
    "print(\"Predicted probabilities:\")\n",
    "print(gbrt.predict_proba(X_test[:6]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(13, 5))\n",
    "    \n",
    "mglearn.tools.plot_2d_separator(\n",
    "    gbrt, X, ax=axes[0], alpha=.4, fill=True, cm=mglearn.cm2)\n",
    "scores_image = mglearn.tools.plot_2d_scores(\n",
    "    gbrt, X, ax=axes[1], alpha=.5, cm=mglearn.ReBl, function='predict_proba')\n",
    "\n",
    "for ax in axes:\n",
    "    # plot training and test points\n",
    "    mglearn.discrete_scatter(X_test[:, 0], X_test[:, 1], y_test,\n",
    "                             markers='^', ax=ax)\n",
    "    mglearn.discrete_scatter(X_train[:, 0], X_train[:, 1], y_train,\n",
    "                             markers='o', ax=ax)\n",
    "    ax.set_xlabel(\"Feature 0\")\n",
    "    ax.set_ylabel(\"Feature 1\")\n",
    "# don't want a transparent colorbar\n",
    "cbar = plt.colorbar(scores_image, ax=axes.tolist())\n",
    "cbar.set_alpha(1)\n",
    "cbar.draw_all()\n",
    "axes[0].legend([\"Test class 0\", \"Test class 1\", \"Train class 0\",\n",
    "                \"Train class 1\"], ncol=4, loc=(.1, 1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![classifier_comparison](images/classifier_comparison.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Uncertainty in multiclass classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    iris.data, iris.target, random_state=42)\n",
    "\n",
    "gbrt = GradientBoostingClassifier(learning_rate=0.01, random_state=0)\n",
    "gbrt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Decision function shape:\", gbrt.decision_function(X_test).shape)\n",
    "# plot the first few entries of the decision function\n",
    "print(\"Decision function:\")\n",
    "print(gbrt.decision_function(X_test)[:6, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Argmax of decision function:\")\n",
    "print(np.argmax(gbrt.decision_function(X_test), axis=1))\n",
    "print(\"Predictions:\")\n",
    "print(gbrt.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the first few entries of predict_proba\n",
    "print(\"Predicted probabilities:\")\n",
    "print(gbrt.predict_proba(X_test)[:6])\n",
    "# show that sums across rows are one\n",
    "print(\"Sums:\", gbrt.predict_proba(X_test)[:6].sum(axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Argmax of predicted probabilities:\")\n",
    "print(np.argmax(gbrt.predict_proba(X_test), axis=1))\n",
    "print(\"Predictions:\")\n",
    "print(gbrt.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg = LogisticRegression()\n",
    "\n",
    "# represent each target by its class name in the iris dataset\n",
    "named_target = iris.target_names[y_train]\n",
    "logreg.fit(X_train, named_target)\n",
    "print(\"unique classes in training data:\", logreg.classes_)\n",
    "print(\"predictions:\", logreg.predict(X_test)[:10])\n",
    "argmax_dec_func = np.argmax(logreg.decision_function(X_test), axis=1)\n",
    "print(\"argmax of decision function:\", argmax_dec_func[:10])\n",
    "print(\"argmax combined with classes_:\",\n",
    "      logreg.classes_[argmax_dec_func][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary and Outlook"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "latex_metadata": {
   "author": "Andreas C. M\\\"ller",
   "title": "Machine Learning with Python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}