lectures/02_regression-metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/573_banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2: Regression metrics\n",
    "\n",
    "UBC Master of Data Science program, 2021-22\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.compose import (\n",
    "    ColumnTransformer,\n",
    "    TransformedTargetRegressor,\n",
    "    make_column_transformer,\n",
    ")\n",
    "from sklearn.dummy import DummyRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import Ridge, RidgeCV\n",
    "from sklearn.metrics import make_scorer, mean_squared_error, r2_score\n",
    "from sklearn.model_selection import (\n",
    "    GridSearchCV,\n",
    "    cross_val_score,\n",
    "    cross_validate,\n",
    "    train_test_split,\n",
    ")\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Learning outcomes \n",
    "\n",
    "From this lecture, students are expected to be able to:\n",
    "\n",
    "- Carry out feature transformations on somewhat complicated dataset. \n",
    "- Visualize transformed features as a dataframe. \n",
    "- Use `Ridge` and `RidgeCV`.\n",
    "- Explain how `alpha` hyperparameter of `Ridge` relates to the fundamental tradeoff. \n",
    "- Explain the effect of `alpha` on the magnitude of the learned coefficients. \n",
    "- Examine coefficients of transformed features.  \n",
    "- Appropriately select a scoring metric given a regression problem.\n",
    "- Interpret and communicate the meanings of different scoring metrics on regression problems.\n",
    "    - MSE, RMSE, $R^2$, MAPE\n",
    "- Apply log-transform on the target values in a regression problem with `TransformedTargetRegressor`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iClicker Exercise 2.1 \n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/C0P55**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) Average precision score at threshold 0.5 is the same as F1-score. \n",
    "- (B) Using `class_weight=\"balanced\"` is likely to decrease false negatives but increase false positives.\n",
    "- (C) In ROC curve, the threshold goes from lower left (1.0) to upper right (0.0).\n",
    "- (D) It's possible that automated tools such as `pandas-profiling` classify a categorical feature as a numeric feature.  \n",
    "\n",
    "**V's answers: B, C, D**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After carrying out preprocessing, why it's useful to get feature names for transformed features?   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More comments on tackling class imbalance\n",
    "\n",
    "- In lecture 1 we talked about a few rather simple approaches to deal with class imbalance. \n",
    "- If you have a problem such as fraud detection problem where you want to spot rare events, you can also think of this problem as anomaly detection problem and use different kinds of algorithms such as [isolation forests](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html).  \n",
    "- If you are interested in this area, it might be worth checking out this book on this topic.\n",
    "[Imbalanced Learning: Foundations, Algorithms, and Applications](https://www.amazon.com/dp/1118074629/ref=as_li_ss_tl?&linkCode=sl1&tag=inspiredalgor-20&linkId=615e87a9105582e292ad2b7e2c7ea339&language=en_US)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note} \n",
    "When you calculate precision, recall, f1 score, by default only the positive label is evaluated, assuming by default that the positive class is labeled 1. This is configurable through the `pos_label` parameter. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dataset [[video](https://youtu.be/lgGTKLwNgkQ)]\n",
    "\n",
    "In this lecture, we'll be using [Kaggle House Prices dataset](https://www.kaggle.com/c/home-data-for-ml-course/). As usual, to run this notebook you'll need to download the data. For this dataset, train and test have already been separated. We'll be working with the train portion in this lecture. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>303</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>118.0</td>\n",
       "      <td>13704</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>205000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>768</td>\n",
       "      <td>50</td>\n",
       "      <td>RL</td>\n",
       "      <td>75.0</td>\n",
       "      <td>12508</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Shed</td>\n",
       "      <td>1300</td>\n",
       "      <td>7</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>160000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>429</th>\n",
       "      <td>430</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>130.0</td>\n",
       "      <td>11457</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2009</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>175000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139</th>\n",
       "      <td>1140</td>\n",
       "      <td>30</td>\n",
       "      <td>RL</td>\n",
       "      <td>98.0</td>\n",
       "      <td>8731</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>144000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>558</th>\n",
       "      <td>559</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>57.0</td>\n",
       "      <td>21872</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR2</td>\n",
       "      <td>HLS</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>175000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "302    303          20       RL        118.0    13704   Pave   NaN      IR1   \n",
       "767    768          50       RL         75.0    12508   Pave   NaN      IR1   \n",
       "429    430          20       RL        130.0    11457   Pave   NaN      IR1   \n",
       "1139  1140          30       RL         98.0     8731   Pave   NaN      IR1   \n",
       "558    559          60       RL         57.0    21872   Pave   NaN      IR2   \n",
       "\n",
       "     LandContour Utilities  ... PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "302          Lvl    AllPub  ...        0    NaN   NaN         NaN       0   \n",
       "767          Lvl    AllPub  ...        0    NaN   NaN        Shed    1300   \n",
       "429          Lvl    AllPub  ...        0    NaN   NaN         NaN       0   \n",
       "1139         Lvl    AllPub  ...        0    NaN   NaN         NaN       0   \n",
       "558          HLS    AllPub  ...        0    NaN   NaN         NaN       0   \n",
       "\n",
       "     MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "302       1   2006        WD         Normal     205000  \n",
       "767       7   2008        WD         Normal     160000  \n",
       "429       3   2009        WD         Normal     175000  \n",
       "1139      5   2007        WD         Normal     144000  \n",
       "558       8   2008        WD         Normal     175000  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/housing-kaggle/train.csv\")\n",
    "train_df, test_df = train_test_split(df, test_size=0.10, random_state=123)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The supervised machine learning problem is predicting housing price given features associated with properties. \n",
    "- Here, the target is `SalePrice`, which is continuous. So it's a **regression problem** (as opposed to classification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1314, 81)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Let's separate `X` and `y`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train = train_df.drop(columns=[\"SalePrice\"])\n",
    "y_train = train_df[\"SalePrice\"]\n",
    "\n",
    "X_test = test_df.drop(columns=[\"SalePrice\"])\n",
    "y_test = test_df[\"SalePrice\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1089.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1307.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "      <td>1314.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>734.182648</td>\n",
       "      <td>56.472603</td>\n",
       "      <td>69.641873</td>\n",
       "      <td>10273.261035</td>\n",
       "      <td>6.076104</td>\n",
       "      <td>5.570015</td>\n",
       "      <td>1970.995434</td>\n",
       "      <td>1984.659056</td>\n",
       "      <td>102.514155</td>\n",
       "      <td>441.425419</td>\n",
       "      <td>...</td>\n",
       "      <td>94.281583</td>\n",
       "      <td>45.765601</td>\n",
       "      <td>21.726788</td>\n",
       "      <td>3.624049</td>\n",
       "      <td>13.987062</td>\n",
       "      <td>3.065449</td>\n",
       "      <td>46.951294</td>\n",
       "      <td>6.302131</td>\n",
       "      <td>2007.840183</td>\n",
       "      <td>179802.147641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>422.224662</td>\n",
       "      <td>42.036646</td>\n",
       "      <td>23.031794</td>\n",
       "      <td>8997.895541</td>\n",
       "      <td>1.392612</td>\n",
       "      <td>1.112848</td>\n",
       "      <td>30.198127</td>\n",
       "      <td>20.639754</td>\n",
       "      <td>178.301563</td>\n",
       "      <td>459.276687</td>\n",
       "      <td>...</td>\n",
       "      <td>125.436492</td>\n",
       "      <td>65.757545</td>\n",
       "      <td>60.766423</td>\n",
       "      <td>30.320430</td>\n",
       "      <td>53.854129</td>\n",
       "      <td>42.341109</td>\n",
       "      <td>522.283421</td>\n",
       "      <td>2.698206</td>\n",
       "      <td>1.332824</td>\n",
       "      <td>79041.260572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>1300.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1872.000000</td>\n",
       "      <td>1950.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2006.000000</td>\n",
       "      <td>34900.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>369.250000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>7500.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1953.000000</td>\n",
       "      <td>1966.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>2007.000000</td>\n",
       "      <td>129600.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>735.500000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>9391.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1972.000000</td>\n",
       "      <td>1993.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>376.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2008.000000</td>\n",
       "      <td>162000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1099.750000</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>11509.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2000.000000</td>\n",
       "      <td>2004.000000</td>\n",
       "      <td>165.500000</td>\n",
       "      <td>704.750000</td>\n",
       "      <td>...</td>\n",
       "      <td>168.000000</td>\n",
       "      <td>66.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2009.000000</td>\n",
       "      <td>212975.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1460.000000</td>\n",
       "      <td>190.000000</td>\n",
       "      <td>313.000000</td>\n",
       "      <td>215245.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>1378.000000</td>\n",
       "      <td>5644.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>857.000000</td>\n",
       "      <td>547.000000</td>\n",
       "      <td>552.000000</td>\n",
       "      <td>508.000000</td>\n",
       "      <td>480.000000</td>\n",
       "      <td>738.000000</td>\n",
       "      <td>15500.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>755000.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Id   MSSubClass  LotFrontage        LotArea  OverallQual  \\\n",
       "count  1314.000000  1314.000000  1089.000000    1314.000000  1314.000000   \n",
       "mean    734.182648    56.472603    69.641873   10273.261035     6.076104   \n",
       "std     422.224662    42.036646    23.031794    8997.895541     1.392612   \n",
       "min       1.000000    20.000000    21.000000    1300.000000     1.000000   \n",
       "25%     369.250000    20.000000    59.000000    7500.000000     5.000000   \n",
       "50%     735.500000    50.000000    69.000000    9391.000000     6.000000   \n",
       "75%    1099.750000    70.000000    80.000000   11509.000000     7.000000   \n",
       "max    1460.000000   190.000000   313.000000  215245.000000    10.000000   \n",
       "\n",
       "       OverallCond    YearBuilt  YearRemodAdd   MasVnrArea   BsmtFinSF1  ...  \\\n",
       "count  1314.000000  1314.000000   1314.000000  1307.000000  1314.000000  ...   \n",
       "mean      5.570015  1970.995434   1984.659056   102.514155   441.425419  ...   \n",
       "std       1.112848    30.198127     20.639754   178.301563   459.276687  ...   \n",
       "min       1.000000  1872.000000   1950.000000     0.000000     0.000000  ...   \n",
       "25%       5.000000  1953.000000   1966.250000     0.000000     0.000000  ...   \n",
       "50%       5.000000  1972.000000   1993.000000     0.000000   376.000000  ...   \n",
       "75%       6.000000  2000.000000   2004.000000   165.500000   704.750000  ...   \n",
       "max       9.000000  2010.000000   2010.000000  1378.000000  5644.000000  ...   \n",
       "\n",
       "        WoodDeckSF  OpenPorchSF  EnclosedPorch    3SsnPorch  ScreenPorch  \\\n",
       "count  1314.000000  1314.000000    1314.000000  1314.000000  1314.000000   \n",
       "mean     94.281583    45.765601      21.726788     3.624049    13.987062   \n",
       "std     125.436492    65.757545      60.766423    30.320430    53.854129   \n",
       "min       0.000000     0.000000       0.000000     0.000000     0.000000   \n",
       "25%       0.000000     0.000000       0.000000     0.000000     0.000000   \n",
       "50%       0.000000    24.000000       0.000000     0.000000     0.000000   \n",
       "75%     168.000000    66.750000       0.000000     0.000000     0.000000   \n",
       "max     857.000000   547.000000     552.000000   508.000000   480.000000   \n",
       "\n",
       "          PoolArea       MiscVal       MoSold       YrSold      SalePrice  \n",
       "count  1314.000000   1314.000000  1314.000000  1314.000000    1314.000000  \n",
       "mean      3.065449     46.951294     6.302131  2007.840183  179802.147641  \n",
       "std      42.341109    522.283421     2.698206     1.332824   79041.260572  \n",
       "min       0.000000      0.000000     1.000000  2006.000000   34900.000000  \n",
       "25%       0.000000      0.000000     5.000000  2007.000000  129600.000000  \n",
       "50%       0.000000      0.000000     6.000000  2008.000000  162000.000000  \n",
       "75%       0.000000      0.000000     8.000000  2009.000000  212975.000000  \n",
       "max     738.000000  15500.000000    12.000000  2010.000000  755000.000000  \n",
       "\n",
       "[8 rows x 38 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1314 entries, 302 to 1389\n",
      "Data columns (total 81 columns):\n",
      " #   Column         Non-Null Count  Dtype  \n",
      "---  ------         --------------  -----  \n",
      " 0   Id             1314 non-null   int64  \n",
      " 1   MSSubClass     1314 non-null   int64  \n",
      " 2   MSZoning       1314 non-null   object \n",
      " 3   LotFrontage    1089 non-null   float64\n",
      " 4   LotArea        1314 non-null   int64  \n",
      " 5   Street         1314 non-null   object \n",
      " 6   Alley          81 non-null     object \n",
      " 7   LotShape       1314 non-null   object \n",
      " 8   LandContour    1314 non-null   object \n",
      " 9   Utilities      1314 non-null   object \n",
      " 10  LotConfig      1314 non-null   object \n",
      " 11  LandSlope      1314 non-null   object \n",
      " 12  Neighborhood   1314 non-null   object \n",
      " 13  Condition1     1314 non-null   object \n",
      " 14  Condition2     1314 non-null   object \n",
      " 15  BldgType       1314 non-null   object \n",
      " 16  HouseStyle     1314 non-null   object \n",
      " 17  OverallQual    1314 non-null   int64  \n",
      " 18  OverallCond    1314 non-null   int64  \n",
      " 19  YearBuilt      1314 non-null   int64  \n",
      " 20  YearRemodAdd   1314 non-null   int64  \n",
      " 21  RoofStyle      1314 non-null   object \n",
      " 22  RoofMatl       1314 non-null   object \n",
      " 23  Exterior1st    1314 non-null   object \n",
      " 24  Exterior2nd    1314 non-null   object \n",
      " 25  MasVnrType     1307 non-null   object \n",
      " 26  MasVnrArea     1307 non-null   float64\n",
      " 27  ExterQual      1314 non-null   object \n",
      " 28  ExterCond      1314 non-null   object \n",
      " 29  Foundation     1314 non-null   object \n",
      " 30  BsmtQual       1280 non-null   object \n",
      " 31  BsmtCond       1280 non-null   object \n",
      " 32  BsmtExposure   1279 non-null   object \n",
      " 33  BsmtFinType1   1280 non-null   object \n",
      " 34  BsmtFinSF1     1314 non-null   int64  \n",
      " 35  BsmtFinType2   1280 non-null   object \n",
      " 36  BsmtFinSF2     1314 non-null   int64  \n",
      " 37  BsmtUnfSF      1314 non-null   int64  \n",
      " 38  TotalBsmtSF    1314 non-null   int64  \n",
      " 39  Heating        1314 non-null   object \n",
      " 40  HeatingQC      1314 non-null   object \n",
      " 41  CentralAir     1314 non-null   object \n",
      " 42  Electrical     1313 non-null   object \n",
      " 43  1stFlrSF       1314 non-null   int64  \n",
      " 44  2ndFlrSF       1314 non-null   int64  \n",
      " 45  LowQualFinSF   1314 non-null   int64  \n",
      " 46  GrLivArea      1314 non-null   int64  \n",
      " 47  BsmtFullBath   1314 non-null   int64  \n",
      " 48  BsmtHalfBath   1314 non-null   int64  \n",
      " 49  FullBath       1314 non-null   int64  \n",
      " 50  HalfBath       1314 non-null   int64  \n",
      " 51  BedroomAbvGr   1314 non-null   int64  \n",
      " 52  KitchenAbvGr   1314 non-null   int64  \n",
      " 53  KitchenQual    1314 non-null   object \n",
      " 54  TotRmsAbvGrd   1314 non-null   int64  \n",
      " 55  Functional     1314 non-null   object \n",
      " 56  Fireplaces     1314 non-null   int64  \n",
      " 57  FireplaceQu    687 non-null    object \n",
      " 58  GarageType     1241 non-null   object \n",
      " 59  GarageYrBlt    1241 non-null   float64\n",
      " 60  GarageFinish   1241 non-null   object \n",
      " 61  GarageCars     1314 non-null   int64  \n",
      " 62  GarageArea     1314 non-null   int64  \n",
      " 63  GarageQual     1241 non-null   object \n",
      " 64  GarageCond     1241 non-null   object \n",
      " 65  PavedDrive     1314 non-null   object \n",
      " 66  WoodDeckSF     1314 non-null   int64  \n",
      " 67  OpenPorchSF    1314 non-null   int64  \n",
      " 68  EnclosedPorch  1314 non-null   int64  \n",
      " 69  3SsnPorch      1314 non-null   int64  \n",
      " 70  ScreenPorch    1314 non-null   int64  \n",
      " 71  PoolArea       1314 non-null   int64  \n",
      " 72  PoolQC         7 non-null      object \n",
      " 73  Fence          259 non-null    object \n",
      " 74  MiscFeature    50 non-null     object \n",
      " 75  MiscVal        1314 non-null   int64  \n",
      " 76  MoSold         1314 non-null   int64  \n",
      " 77  YrSold         1314 non-null   int64  \n",
      " 78  SaleType       1314 non-null   object \n",
      " 79  SaleCondition  1314 non-null   object \n",
      " 80  SalePrice      1314 non-null   int64  \n",
      "dtypes: float64(3), int64(35), object(43)\n",
      "memory usage: 841.8+ KB\n"
     ]
    }
   ],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `pandas_profiler`\n",
    "\n",
    "We do not have `pandas_profiling` in our course environment. You will  have to install it in the environment on your own if you want to run the code below. \n",
    "\n",
    "```conda install -c conda-forge pandas-profiling```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from pandas_profiling import ProfileReport\n",
    "\n",
    "# profile = ProfileReport(train_df, title=\"Pandas Profiling Report\")  # , minimal=True)\n",
    "# profile.to_notebook_iframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Feature types \n",
    "\n",
    "- Do not blindly trust all the info given to you by automated tools. \n",
    "- How does pandas profiling figure out the data type?\n",
    "    - You can look at the Python data type and say floats are numeric, strings are categorical.\n",
    "    - However, in doing so you would miss out on various subtleties such as some of the string features being ordinal rather than truly categorical.\n",
    "    - Also, it will think free text is categorical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In addition to tools such as above, it's important to go through data description to understand the data.\n",
    "- The data description for our dataset is available [here](https://www.kaggle.com/c/home-data-for-ml-course/data?select=data_description.txt).     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Feature types \n",
    "\n",
    "- We have mixed feature types and a bunch of missing values. \n",
    "- Now, let's identify feature types and transformations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's get the numeric-looking columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Id', 'MSSubClass', 'LotFrontage', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces', 'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold']\n"
     ]
    }
   ],
   "source": [
    "numeric_looking_columns = X_train.select_dtypes(include=np.number).columns.tolist()\n",
    "print(numeric_looking_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Not all numeric looking columns are necessarily numeric. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 20,  50,  30,  60, 160,  85,  90, 120, 180,  80,  70,  75, 190,\n",
       "        45,  40])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[\"MSSubClass\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MSSubClass: Identifies the type of dwelling involved in the sale.\t\n",
    "\n",
    "        20\t1-STORY 1946 & NEWER ALL STYLES\n",
    "        30\t1-STORY 1945 & OLDER\n",
    "        40\t1-STORY W/FINISHED ATTIC ALL AGES\n",
    "        45\t1-1/2 STORY - UNFINISHED ALL AGES\n",
    "        50\t1-1/2 STORY FINISHED ALL AGES\n",
    "        60\t2-STORY 1946 & NEWER\n",
    "        70\t2-STORY 1945 & OLDER\n",
    "        75\t2-1/2 STORY ALL AGES\n",
    "        80\tSPLIT OR MULTI-LEVEL\n",
    "        85\tSPLIT FOYER\n",
    "        90\tDUPLEX - ALL STYLES AND AGES\n",
    "       120\t1-STORY PUD (Planned Unit Development) - 1946 & NEWER\n",
    "       150\t1-1/2 STORY PUD - ALL AGES\n",
    "       160\t2-STORY PUD - 1946 & NEWER\n",
    "       180\tPUD - MULTILEVEL - INCL SPLIT LEV/FOYER\n",
    "       190\t2 FAMILY CONVERSION - ALL STYLES AND AGES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also, month sold is more of a categorical feature than a numeric feature. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1,  7,  3,  5,  8, 10,  6,  9, 12,  2,  4, 11])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[\"MoSold\"].unique()  # Month Sold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "drop_features = [\"Id\"]\n",
    "numeric_features = [\n",
    "    \"BedroomAbvGr\",\n",
    "    \"KitchenAbvGr\",\n",
    "    \"LotFrontage\",\n",
    "    \"LotArea\",\n",
    "    \"OverallQual\",\n",
    "    \"OverallCond\",\n",
    "    \"YearBuilt\",\n",
    "    \"YearRemodAdd\",\n",
    "    \"MasVnrArea\",\n",
    "    \"BsmtFinSF1\",\n",
    "    \"BsmtFinSF2\",\n",
    "    \"BsmtUnfSF\",\n",
    "    \"TotalBsmtSF\",\n",
    "    \"1stFlrSF\",\n",
    "    \"2ndFlrSF\",\n",
    "    \"LowQualFinSF\",\n",
    "    \"GrLivArea\",\n",
    "    \"BsmtFullBath\",\n",
    "    \"BsmtHalfBath\",\n",
    "    \"FullBath\",\n",
    "    \"HalfBath\",\n",
    "    \"TotRmsAbvGrd\",\n",
    "    \"Fireplaces\",\n",
    "    \"GarageYrBlt\",\n",
    "    \"GarageCars\",\n",
    "    \"GarageArea\",\n",
    "    \"WoodDeckSF\",\n",
    "    \"OpenPorchSF\",\n",
    "    \"EnclosedPorch\",\n",
    "    \"3SsnPorch\",\n",
    "    \"ScreenPorch\",\n",
    "    \"PoolArea\",\n",
    "    \"MiscVal\",\n",
    "    \"YrSold\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "I've not looked at all the features carefully. It might be appropriate to apply some other encoding on some of the numeric features above. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MSSubClass', 'MoSold'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(numeric_looking_columns) - set(numeric_features) - set(drop_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll treat the above numeric-looking features as categorical features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- There are a bunch of ordinal features in this dataset. \n",
    "- Ordinal features with the same scale \n",
    "    - Poor (Po), Fair (Fa), Typical (TA), Good (Gd), Excellent (Ex)\n",
    "    - These we'll be calling `ordinal_features_reg`.\n",
    "- Ordinal features with different scales\n",
    "    - These we'll be calling `ordinal_features_oth`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       " ['Po', 'Fa', 'TA', 'Gd', 'Ex']]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ordinal_features_reg = [\n",
    "    \"ExterQual\",\n",
    "    \"ExterCond\",\n",
    "    \"BsmtQual\",\n",
    "    \"BsmtCond\",\n",
    "    \"HeatingQC\",\n",
    "    \"KitchenQual\",\n",
    "    \"FireplaceQu\",\n",
    "    \"GarageQual\",\n",
    "    \"GarageCond\",\n",
    "    \"PoolQC\",\n",
    "]\n",
    "ordering = [\n",
    "    \"Po\",\n",
    "    \"Fa\",\n",
    "    \"TA\",\n",
    "    \"Gd\",\n",
    "    \"Ex\",\n",
    "]  # if N/A it will just impute something, per below\n",
    "ordering_ordinal_reg = [ordering] * len(ordinal_features_reg)\n",
    "ordering_ordinal_reg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll pass the above as categories in our `OrdinalEncoder`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- There are a bunch more ordinal features using different scales.\n",
    "  - These we'll be calling `ordinal_features_oth`. \n",
    "  - We are encoding them separately. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal_features_oth = [\n",
    "    \"BsmtExposure\",\n",
    "    \"BsmtFinType1\",\n",
    "    \"BsmtFinType2\",\n",
    "    \"Functional\",\n",
    "    \"Fence\",\n",
    "]\n",
    "ordering_ordinal_oth = [\n",
    "    [\"NA\", \"No\", \"Mn\", \"Av\", \"Gd\"],\n",
    "    [\"NA\", \"Unf\", \"LwQ\", \"Rec\", \"BLQ\", \"ALQ\", \"GLQ\"],\n",
    "    [\"NA\", \"Unf\", \"LwQ\", \"Rec\", \"BLQ\", \"ALQ\", \"GLQ\"],\n",
    "    [\"Sal\", \"Sev\", \"Maj2\", \"Maj1\", \"Mod\", \"Min2\", \"Min1\", \"Typ\"],\n",
    "    [\"NA\", \"MnWw\", \"GdWo\", \"MnPrv\", \"GdPrv\"],\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The remaining features are categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SaleType',\n",
       " 'Neighborhood',\n",
       " 'Foundation',\n",
       " 'Street',\n",
       " 'Exterior2nd',\n",
       " 'RoofStyle',\n",
       " 'MSZoning',\n",
       " 'PavedDrive',\n",
       " 'Utilities',\n",
       " 'LotConfig',\n",
       " 'SaleCondition',\n",
       " 'LandSlope',\n",
       " 'LotShape',\n",
       " 'GarageFinish',\n",
       " 'Condition2',\n",
       " 'MasVnrType',\n",
       " 'Exterior1st',\n",
       " 'GarageType',\n",
       " 'LandContour',\n",
       " 'BldgType',\n",
       " 'Electrical',\n",
       " 'MSSubClass',\n",
       " 'CentralAir',\n",
       " 'Condition1',\n",
       " 'RoofMatl',\n",
       " 'Alley',\n",
       " 'MiscFeature',\n",
       " 'HouseStyle',\n",
       " 'Heating',\n",
       " 'MoSold']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_features = list(\n",
    "    set(X_train.columns)\n",
    "    - set(numeric_features)\n",
    "    - set(ordinal_features_reg)\n",
    "    - set(ordinal_features_oth)\n",
    "    - set(drop_features)\n",
    ")\n",
    "categorical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We are not doing it here but we can engineer our own features too. \n",
    "- Would price per square foot be a good feature to add in here?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying feature transformations\n",
    "\n",
    "- Since we have mixed feature types, let's use `ColumnTransformer` to apply different transformations on different features types.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "\n",
    "numeric_transformer = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n",
    "ordinal_transformer_reg = make_pipeline(\n",
    "    SimpleImputer(strategy=\"most_frequent\"),\n",
    "    OrdinalEncoder(categories=ordering_ordinal_reg),\n",
    ")\n",
    "\n",
    "ordinal_transformer_oth = make_pipeline(\n",
    "    SimpleImputer(strategy=\"most_frequent\"),\n",
    "    OrdinalEncoder(categories=ordering_ordinal_oth),\n",
    ")\n",
    "\n",
    "categorical_transformer = make_pipeline(\n",
    "    SimpleImputer(strategy=\"constant\", fill_value=\"missing\"),\n",
    "    OneHotEncoder(handle_unknown=\"ignore\", sparse=False),\n",
    ")\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (\"drop\", drop_features),\n",
    "    (numeric_transformer, numeric_features),\n",
    "    (ordinal_transformer_reg, ordinal_features_reg),\n",
    "    (ordinal_transformer_oth, ordinal_features_oth),\n",
    "    (categorical_transformer, categorical_features),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examining the preprocessed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'drop': 'drop',\n",
       " 'pipeline-1': Pipeline(steps=[('simpleimputer', SimpleImputer(strategy='median')),\n",
       "                 ('standardscaler', StandardScaler())]),\n",
       " 'pipeline-2': Pipeline(steps=[('simpleimputer', SimpleImputer(strategy='most_frequent')),\n",
       "                 ('ordinalencoder',\n",
       "                  OrdinalEncoder(categories=[['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex'],\n",
       "                                             ['Po', 'Fa', 'TA', 'Gd', 'Ex']]))]),\n",
       " 'pipeline-3': Pipeline(steps=[('simpleimputer', SimpleImputer(strategy='most_frequent')),\n",
       "                 ('ordinalencoder',\n",
       "                  OrdinalEncoder(categories=[['NA', 'No', 'Mn', 'Av', 'Gd'],\n",
       "                                             ['NA', 'Unf', 'LwQ', 'Rec', 'BLQ',\n",
       "                                              'ALQ', 'GLQ'],\n",
       "                                             ['NA', 'Unf', 'LwQ', 'Rec', 'BLQ',\n",
       "                                              'ALQ', 'GLQ'],\n",
       "                                             ['Sal', 'Sev', 'Maj2', 'Maj1',\n",
       "                                              'Mod', 'Min2', 'Min1', 'Typ'],\n",
       "                                             ['NA', 'MnWw', 'GdWo', 'MnPrv',\n",
       "                                              'GdPrv']]))]),\n",
       " 'pipeline-4': Pipeline(steps=[('simpleimputer',\n",
       "                  SimpleImputer(fill_value='missing', strategy='constant')),\n",
       "                 ('onehotencoder',\n",
       "                  OneHotEncoder(handle_unknown='ignore', sparse=False))])}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor.fit(X_train)  # Calling fit to examine all the transformers.\n",
    "preprocessor.named_transformers_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ohe_columns = list(\n",
    "    preprocessor.named_transformers_[\"pipeline-4\"]\n",
    "    .named_steps[\"onehotencoder\"]\n",
    "    .get_feature_names_out(categorical_features)\n",
    ")\n",
    "new_columns = (\n",
    "    numeric_features + ordinal_features_reg + ordinal_features_oth + ohe_columns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>MoSold_3</th>\n",
       "      <th>MoSold_4</th>\n",
       "      <th>MoSold_5</th>\n",
       "      <th>MoSold_6</th>\n",
       "      <th>MoSold_7</th>\n",
       "      <th>MoSold_8</th>\n",
       "      <th>MoSold_9</th>\n",
       "      <th>MoSold_10</th>\n",
       "      <th>MoSold_11</th>\n",
       "      <th>MoSold_12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>0.154795</td>\n",
       "      <td>-0.222647</td>\n",
       "      <td>2.312501</td>\n",
       "      <td>0.381428</td>\n",
       "      <td>0.663680</td>\n",
       "      <td>-0.512408</td>\n",
       "      <td>0.993969</td>\n",
       "      <td>0.840492</td>\n",
       "      <td>0.269972</td>\n",
       "      <td>-0.961498</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1.372763</td>\n",
       "      <td>-0.222647</td>\n",
       "      <td>0.260890</td>\n",
       "      <td>0.248457</td>\n",
       "      <td>-0.054669</td>\n",
       "      <td>1.285467</td>\n",
       "      <td>-1.026793</td>\n",
       "      <td>0.016525</td>\n",
       "      <td>-0.573129</td>\n",
       "      <td>0.476092</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>429</th>\n",
       "      <td>0.154795</td>\n",
       "      <td>-0.222647</td>\n",
       "      <td>2.885044</td>\n",
       "      <td>0.131607</td>\n",
       "      <td>-0.054669</td>\n",
       "      <td>-0.512408</td>\n",
       "      <td>0.563314</td>\n",
       "      <td>0.161931</td>\n",
       "      <td>-0.573129</td>\n",
       "      <td>1.227559</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139</th>\n",
       "      <td>0.154795</td>\n",
       "      <td>-0.222647</td>\n",
       "      <td>1.358264</td>\n",
       "      <td>-0.171468</td>\n",
       "      <td>-0.773017</td>\n",
       "      <td>-0.512408</td>\n",
       "      <td>-1.689338</td>\n",
       "      <td>-1.679877</td>\n",
       "      <td>-0.573129</td>\n",
       "      <td>0.443419</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>558</th>\n",
       "      <td>0.154795</td>\n",
       "      <td>-0.222647</td>\n",
       "      <td>-0.597924</td>\n",
       "      <td>1.289541</td>\n",
       "      <td>0.663680</td>\n",
       "      <td>-0.512408</td>\n",
       "      <td>0.828332</td>\n",
       "      <td>0.598149</td>\n",
       "      <td>-0.573129</td>\n",
       "      <td>0.354114</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 263 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      BedroomAbvGr  KitchenAbvGr  LotFrontage   LotArea  OverallQual  \\\n",
       "302       0.154795     -0.222647     2.312501  0.381428     0.663680   \n",
       "767       1.372763     -0.222647     0.260890  0.248457    -0.054669   \n",
       "429       0.154795     -0.222647     2.885044  0.131607    -0.054669   \n",
       "1139      0.154795     -0.222647     1.358264 -0.171468    -0.773017   \n",
       "558       0.154795     -0.222647    -0.597924  1.289541     0.663680   \n",
       "\n",
       "      OverallCond  YearBuilt  YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  \\\n",
       "302     -0.512408   0.993969      0.840492    0.269972   -0.961498  ...   \n",
       "767      1.285467  -1.026793      0.016525   -0.573129    0.476092  ...   \n",
       "429     -0.512408   0.563314      0.161931   -0.573129    1.227559  ...   \n",
       "1139    -0.512408  -1.689338     -1.679877   -0.573129    0.443419  ...   \n",
       "558     -0.512408   0.828332      0.598149   -0.573129    0.354114  ...   \n",
       "\n",
       "      MoSold_3  MoSold_4  MoSold_5  MoSold_6  MoSold_7  MoSold_8  MoSold_9  \\\n",
       "302        0.0       0.0       0.0       0.0       0.0       0.0       0.0   \n",
       "767        0.0       0.0       0.0       0.0       1.0       0.0       0.0   \n",
       "429        1.0       0.0       0.0       0.0       0.0       0.0       0.0   \n",
       "1139       0.0       0.0       1.0       0.0       0.0       0.0       0.0   \n",
       "558        0.0       0.0       0.0       0.0       0.0       1.0       0.0   \n",
       "\n",
       "      MoSold_10  MoSold_11  MoSold_12  \n",
       "302         0.0        0.0        0.0  \n",
       "767         0.0        0.0        0.0  \n",
       "429         0.0        0.0        0.0  \n",
       "1139        0.0        0.0        0.0  \n",
       "558         0.0        0.0        0.0  \n",
       "\n",
       "[5 rows x 263 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_enc = pd.DataFrame(\n",
    "    preprocessor.transform(X_train), index=X_train.index, columns=new_columns\n",
    ")\n",
    "X_train_enc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1314, 80)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1314, 263)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_enc.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We went from 80 features to 263 features!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other possible preprocessing?  \n",
    "\n",
    "- There is a lot of room for improvement ...\n",
    "- We're just using `SimpleImputer`.\n",
    "    - In reality we'd want to go through this more carefully.\n",
    "    - We may also want to drop some columns that are almost entirely missing.    \n",
    "- We could also check for outliers, and do other exploratory data analysis (EDA).\n",
    "- But for now this is good enough ...    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model building "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `DummyRegressor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000892</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>-0.003547</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000425</td>\n",
       "      <td>0.000095</td>\n",
       "      <td>-0.001266</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000552</td>\n",
       "      <td>0.000102</td>\n",
       "      <td>-0.011767</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000515</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>-0.006744</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000511</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>-0.076533</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.000424</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>-0.003133</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.000395</td>\n",
       "      <td>0.000090</td>\n",
       "      <td>-0.000397</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.000407</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>-0.003785</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.000390</td>\n",
       "      <td>0.000086</td>\n",
       "      <td>-0.001740</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.000387</td>\n",
       "      <td>0.000085</td>\n",
       "      <td>-0.000117</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.000892    0.000188   -0.003547          0.0\n",
       "1  0.000425    0.000095   -0.001266          0.0\n",
       "2  0.000552    0.000102   -0.011767          0.0\n",
       "3  0.000515    0.000100   -0.006744          0.0\n",
       "4  0.000511    0.000119   -0.076533          0.0\n",
       "5  0.000424    0.000094   -0.003133          0.0\n",
       "6  0.000395    0.000090   -0.000397          0.0\n",
       "7  0.000407    0.000089   -0.003785          0.0\n",
       "8  0.000390    0.000086   -0.001740          0.0\n",
       "9  0.000387    0.000085   -0.000117          0.0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy = DummyRegressor()\n",
    "pd.DataFrame(cross_validate(dummy, X_train, y_train, cv=10, return_train_score=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's try a linear model: `Ridge`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Recall that we are going to use `Ridge()` instead of `LinearRegression()` in this course. \n",
    "- Similar to linear regression, ridge regression is also a linear model for regression. \n",
    "- So the formula it uses to make predictions is the same one used for ordinary least squares (which you'll be learning in DSCI 561). \n",
    "- But it has a hyperparameter `alpha` which controls the fundamental tradeoff.     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = make_pipeline(preprocessor, Ridge())\n",
    "lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([224795.63596803,  65063.79363463, 133408.78346075, 250878.88144979,\n",
       "       128875.99852629, 205597.41832137, 334578.36315754, 159241.04183629,\n",
       "       148367.15710315, 129278.59247656])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_preds = lr.predict(X_test)\n",
    "lr_preds[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(411432.16086516465, 38827.24302274236)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_preds.max(), lr_preds.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Smallest coefficient:  -191129.77431368965\n",
      "Largest coefficient: 83646.71100760289\n"
     ]
    }
   ],
   "source": [
    "print(\"Smallest coefficient: \", lr.named_steps[\"ridge\"].coef_.min())\n",
    "print(\"Largest coefficient:\", lr.named_steps[\"ridge\"].coef_.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's carry out cross-validation with `Ridge`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.024186</td>\n",
       "      <td>0.005183</td>\n",
       "      <td>0.861355</td>\n",
       "      <td>0.911906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.019860</td>\n",
       "      <td>0.004168</td>\n",
       "      <td>0.812301</td>\n",
       "      <td>0.913861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.018951</td>\n",
       "      <td>0.003956</td>\n",
       "      <td>0.775283</td>\n",
       "      <td>0.915963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.019101</td>\n",
       "      <td>0.003930</td>\n",
       "      <td>0.874519</td>\n",
       "      <td>0.910849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.018593</td>\n",
       "      <td>0.004010</td>\n",
       "      <td>0.851969</td>\n",
       "      <td>0.911622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.018503</td>\n",
       "      <td>0.004136</td>\n",
       "      <td>0.826198</td>\n",
       "      <td>0.910176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.018924</td>\n",
       "      <td>0.004096</td>\n",
       "      <td>0.825533</td>\n",
       "      <td>0.913781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.018370</td>\n",
       "      <td>0.003908</td>\n",
       "      <td>0.872238</td>\n",
       "      <td>0.910071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.018573</td>\n",
       "      <td>0.003908</td>\n",
       "      <td>0.196663</td>\n",
       "      <td>0.921448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.018850</td>\n",
       "      <td>0.003883</td>\n",
       "      <td>0.890474</td>\n",
       "      <td>0.908221</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.024186    0.005183    0.861355     0.911906\n",
       "1  0.019860    0.004168    0.812301     0.913861\n",
       "2  0.018951    0.003956    0.775283     0.915963\n",
       "3  0.019101    0.003930    0.874519     0.910849\n",
       "4  0.018593    0.004010    0.851969     0.911622\n",
       "5  0.018503    0.004136    0.826198     0.910176\n",
       "6  0.018924    0.004096    0.825533     0.913781\n",
       "7  0.018370    0.003908    0.872238     0.910071\n",
       "8  0.018573    0.003908    0.196663     0.921448\n",
       "9  0.018850    0.003883    0.890474     0.908221"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_pipe = make_pipeline(preprocessor, Ridge())\n",
    "pd.DataFrame(cross_validate(lr_pipe, X_train, y_train, cv=10, return_train_score=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Quite a bit of variation in the test scores. \n",
    "- Performing poorly in fold 8. Not sure why. \n",
    "    - Probably it contains the outliers in the data which we kind of ignored. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Feature names of transformed data\n",
    "\n",
    "- If you want to get the column names of newly created columns, you need to fit the transformer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;Id&#x27;]),\n",
       "                                (&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;BedroomAbvGr&#x27;, &#x27;KitchenAbvGr&#x27;, &#x27;LotFrontage&#x27;,\n",
       "                                  &#x27;LotArea&#x27;, &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;,\n",
       "                                  &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;, &#x27;MasVnrArea&#x27;,\n",
       "                                  &#x27;BsmtFinSF1&#x27;, &#x27;BsmtFinSF2&#x27;, &#x27;BsmtUnfSF&#x27;,\n",
       "                                  &#x27;TotalBsmtSF&#x27;, &#x27;...\n",
       "                                 [&#x27;SaleType&#x27;, &#x27;Neighborhood&#x27;, &#x27;Foundation&#x27;,\n",
       "                                  &#x27;Street&#x27;, &#x27;Exterior2nd&#x27;, &#x27;RoofStyle&#x27;,\n",
       "                                  &#x27;MSZoning&#x27;, &#x27;PavedDrive&#x27;, &#x27;Utilities&#x27;,\n",
       "                                  &#x27;LotConfig&#x27;, &#x27;SaleCondition&#x27;, &#x27;LandSlope&#x27;,\n",
       "                                  &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;, &#x27;Condition2&#x27;,\n",
       "                                  &#x27;MasVnrType&#x27;, &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;,\n",
       "                                  &#x27;LandContour&#x27;, &#x27;BldgType&#x27;, &#x27;Electrical&#x27;,\n",
       "                                  &#x27;MSSubClass&#x27;, &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;,\n",
       "                                  &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;, &#x27;MiscFeature&#x27;,\n",
       "                                  &#x27;HouseStyle&#x27;, &#x27;Heating&#x27;, &#x27;MoSold&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;Id&#x27;]),\n",
       "                                (&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;BedroomAbvGr&#x27;, &#x27;KitchenAbvGr&#x27;, &#x27;LotFrontage&#x27;,\n",
       "                                  &#x27;LotArea&#x27;, &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;,\n",
       "                                  &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;, &#x27;MasVnrArea&#x27;,\n",
       "                                  &#x27;BsmtFinSF1&#x27;, &#x27;BsmtFinSF2&#x27;, &#x27;BsmtUnfSF&#x27;,\n",
       "                                  &#x27;TotalBsmtSF&#x27;, &#x27;...\n",
       "                                 [&#x27;SaleType&#x27;, &#x27;Neighborhood&#x27;, &#x27;Foundation&#x27;,\n",
       "                                  &#x27;Street&#x27;, &#x27;Exterior2nd&#x27;, &#x27;RoofStyle&#x27;,\n",
       "                                  &#x27;MSZoning&#x27;, &#x27;PavedDrive&#x27;, &#x27;Utilities&#x27;,\n",
       "                                  &#x27;LotConfig&#x27;, &#x27;SaleCondition&#x27;, &#x27;LandSlope&#x27;,\n",
       "                                  &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;, &#x27;Condition2&#x27;,\n",
       "                                  &#x27;MasVnrType&#x27;, &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;,\n",
       "                                  &#x27;LandContour&#x27;, &#x27;BldgType&#x27;, &#x27;Electrical&#x27;,\n",
       "                                  &#x27;MSSubClass&#x27;, &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;,\n",
       "                                  &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;, &#x27;MiscFeature&#x27;,\n",
       "                                  &#x27;HouseStyle&#x27;, &#x27;Heating&#x27;, &#x27;MoSold&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Id&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;BedroomAbvGr&#x27;, &#x27;KitchenAbvGr&#x27;, &#x27;LotFrontage&#x27;, &#x27;LotArea&#x27;, &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;, &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;, &#x27;MasVnrArea&#x27;, &#x27;BsmtFinSF1&#x27;, &#x27;BsmtFinSF2&#x27;, &#x27;BsmtUnfSF&#x27;, &#x27;TotalBsmtSF&#x27;, &#x27;1stFlrSF&#x27;, &#x27;2ndFlrSF&#x27;, &#x27;LowQualFinSF&#x27;, &#x27;GrLivArea&#x27;, &#x27;BsmtFullBath&#x27;, &#x27;BsmtHalfBath&#x27;, &#x27;FullBath&#x27;, &#x27;HalfBath&#x27;, &#x27;TotRmsAbvGrd&#x27;, &#x27;Fireplaces&#x27;, &#x27;GarageYrBlt&#x27;, &#x27;GarageCars&#x27;, &#x27;GarageArea&#x27;, &#x27;WoodDeckSF&#x27;, &#x27;OpenPorchSF&#x27;, &#x27;EnclosedPorch&#x27;, &#x27;3SsnPorch&#x27;, &#x27;ScreenPorch&#x27;, &#x27;PoolArea&#x27;, &#x27;MiscVal&#x27;, &#x27;YrSold&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ExterQual&#x27;, &#x27;ExterCond&#x27;, &#x27;BsmtQual&#x27;, &#x27;BsmtCond&#x27;, &#x27;HeatingQC&#x27;, &#x27;KitchenQual&#x27;, &#x27;FireplaceQu&#x27;, &#x27;GarageQual&#x27;, &#x27;GarageCond&#x27;, &#x27;PoolQC&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;]])</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;BsmtExposure&#x27;, &#x27;BsmtFinType1&#x27;, &#x27;BsmtFinType2&#x27;, &#x27;Functional&#x27;, &#x27;Fence&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;NA&#x27;, &#x27;No&#x27;, &#x27;Mn&#x27;, &#x27;Av&#x27;, &#x27;Gd&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;Unf&#x27;, &#x27;LwQ&#x27;, &#x27;Rec&#x27;, &#x27;BLQ&#x27;, &#x27;ALQ&#x27;, &#x27;GLQ&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;Unf&#x27;, &#x27;LwQ&#x27;, &#x27;Rec&#x27;, &#x27;BLQ&#x27;, &#x27;ALQ&#x27;, &#x27;GLQ&#x27;],\n",
       "                           [&#x27;Sal&#x27;, &#x27;Sev&#x27;, &#x27;Maj2&#x27;, &#x27;Maj1&#x27;, &#x27;Mod&#x27;, &#x27;Min2&#x27;, &#x27;Min1&#x27;,\n",
       "                            &#x27;Typ&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;MnWw&#x27;, &#x27;GdWo&#x27;, &#x27;MnPrv&#x27;, &#x27;GdPrv&#x27;]])</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;SaleType&#x27;, &#x27;Neighborhood&#x27;, &#x27;Foundation&#x27;, &#x27;Street&#x27;, &#x27;Exterior2nd&#x27;, &#x27;RoofStyle&#x27;, &#x27;MSZoning&#x27;, &#x27;PavedDrive&#x27;, &#x27;Utilities&#x27;, &#x27;LotConfig&#x27;, &#x27;SaleCondition&#x27;, &#x27;LandSlope&#x27;, &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;, &#x27;Condition2&#x27;, &#x27;MasVnrType&#x27;, &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;, &#x27;LandContour&#x27;, &#x27;BldgType&#x27;, &#x27;Electrical&#x27;, &#x27;MSSubClass&#x27;, &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;, &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;, &#x27;MiscFeature&#x27;, &#x27;HouseStyle&#x27;, &#x27;Heating&#x27;, &#x27;MoSold&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('drop', 'drop', ['Id']),\n",
       "                                ('pipeline-1',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['BedroomAbvGr', 'KitchenAbvGr', 'LotFrontage',\n",
       "                                  'LotArea', 'OverallQual', 'OverallCond',\n",
       "                                  'YearBuilt', 'YearRemodAdd', 'MasVnrArea',\n",
       "                                  'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF',\n",
       "                                  'TotalBsmtSF', '...\n",
       "                                 ['SaleType', 'Neighborhood', 'Foundation',\n",
       "                                  'Street', 'Exterior2nd', 'RoofStyle',\n",
       "                                  'MSZoning', 'PavedDrive', 'Utilities',\n",
       "                                  'LotConfig', 'SaleCondition', 'LandSlope',\n",
       "                                  'LotShape', 'GarageFinish', 'Condition2',\n",
       "                                  'MasVnrType', 'Exterior1st', 'GarageType',\n",
       "                                  'LandContour', 'BldgType', 'Electrical',\n",
       "                                  'MSSubClass', 'CentralAir', 'Condition1',\n",
       "                                  'RoofMatl', 'Alley', 'MiscFeature',\n",
       "                                  'HouseStyle', 'Heating', 'MoSold'])])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SaleType_COD',\n",
       " 'SaleType_CWD',\n",
       " 'SaleType_Con',\n",
       " 'SaleType_ConLD',\n",
       " 'SaleType_ConLI',\n",
       " 'SaleType_ConLw',\n",
       " 'SaleType_New',\n",
       " 'SaleType_Oth',\n",
       " 'SaleType_WD',\n",
       " 'Neighborhood_Blmngtn',\n",
       " 'Neighborhood_Blueste',\n",
       " 'Neighborhood_BrDale',\n",
       " 'Neighborhood_BrkSide',\n",
       " 'Neighborhood_ClearCr',\n",
       " 'Neighborhood_CollgCr',\n",
       " 'Neighborhood_Crawfor',\n",
       " 'Neighborhood_Edwards',\n",
       " 'Neighborhood_Gilbert',\n",
       " 'Neighborhood_IDOTRR',\n",
       " 'Neighborhood_MeadowV',\n",
       " 'Neighborhood_Mitchel',\n",
       " 'Neighborhood_NAmes',\n",
       " 'Neighborhood_NPkVill',\n",
       " 'Neighborhood_NWAmes',\n",
       " 'Neighborhood_NoRidge',\n",
       " 'Neighborhood_NridgHt',\n",
       " 'Neighborhood_OldTown',\n",
       " 'Neighborhood_SWISU',\n",
       " 'Neighborhood_Sawyer',\n",
       " 'Neighborhood_SawyerW',\n",
       " 'Neighborhood_Somerst',\n",
       " 'Neighborhood_StoneBr',\n",
       " 'Neighborhood_Timber',\n",
       " 'Neighborhood_Veenker',\n",
       " 'Foundation_BrkTil',\n",
       " 'Foundation_CBlock',\n",
       " 'Foundation_PConc',\n",
       " 'Foundation_Slab',\n",
       " 'Foundation_Stone',\n",
       " 'Foundation_Wood',\n",
       " 'Street_Grvl',\n",
       " 'Street_Pave',\n",
       " 'Exterior2nd_AsbShng',\n",
       " 'Exterior2nd_AsphShn',\n",
       " 'Exterior2nd_Brk Cmn',\n",
       " 'Exterior2nd_BrkFace',\n",
       " 'Exterior2nd_CBlock',\n",
       " 'Exterior2nd_CmentBd',\n",
       " 'Exterior2nd_HdBoard',\n",
       " 'Exterior2nd_ImStucc',\n",
       " 'Exterior2nd_MetalSd',\n",
       " 'Exterior2nd_Other',\n",
       " 'Exterior2nd_Plywood',\n",
       " 'Exterior2nd_Stone',\n",
       " 'Exterior2nd_Stucco',\n",
       " 'Exterior2nd_VinylSd',\n",
       " 'Exterior2nd_Wd Sdng',\n",
       " 'Exterior2nd_Wd Shng',\n",
       " 'RoofStyle_Flat',\n",
       " 'RoofStyle_Gable',\n",
       " 'RoofStyle_Gambrel',\n",
       " 'RoofStyle_Hip',\n",
       " 'RoofStyle_Mansard',\n",
       " 'RoofStyle_Shed',\n",
       " 'MSZoning_C (all)',\n",
       " 'MSZoning_FV',\n",
       " 'MSZoning_RH',\n",
       " 'MSZoning_RL',\n",
       " 'MSZoning_RM',\n",
       " 'PavedDrive_N',\n",
       " 'PavedDrive_P',\n",
       " 'PavedDrive_Y',\n",
       " 'Utilities_AllPub',\n",
       " 'Utilities_NoSeWa',\n",
       " 'LotConfig_Corner',\n",
       " 'LotConfig_CulDSac',\n",
       " 'LotConfig_FR2',\n",
       " 'LotConfig_FR3',\n",
       " 'LotConfig_Inside',\n",
       " 'SaleCondition_Abnorml',\n",
       " 'SaleCondition_AdjLand',\n",
       " 'SaleCondition_Alloca',\n",
       " 'SaleCondition_Family',\n",
       " 'SaleCondition_Normal',\n",
       " 'SaleCondition_Partial',\n",
       " 'LandSlope_Gtl',\n",
       " 'LandSlope_Mod',\n",
       " 'LandSlope_Sev',\n",
       " 'LotShape_IR1',\n",
       " 'LotShape_IR2',\n",
       " 'LotShape_IR3',\n",
       " 'LotShape_Reg',\n",
       " 'GarageFinish_Fin',\n",
       " 'GarageFinish_RFn',\n",
       " 'GarageFinish_Unf',\n",
       " 'GarageFinish_missing',\n",
       " 'Condition2_Artery',\n",
       " 'Condition2_Feedr',\n",
       " 'Condition2_Norm',\n",
       " 'Condition2_PosA',\n",
       " 'Condition2_PosN',\n",
       " 'Condition2_RRAe',\n",
       " 'Condition2_RRAn',\n",
       " 'Condition2_RRNn',\n",
       " 'MasVnrType_BrkCmn',\n",
       " 'MasVnrType_BrkFace',\n",
       " 'MasVnrType_None',\n",
       " 'MasVnrType_Stone',\n",
       " 'MasVnrType_missing',\n",
       " 'Exterior1st_AsbShng',\n",
       " 'Exterior1st_AsphShn',\n",
       " 'Exterior1st_BrkComm',\n",
       " 'Exterior1st_BrkFace',\n",
       " 'Exterior1st_CBlock',\n",
       " 'Exterior1st_CemntBd',\n",
       " 'Exterior1st_HdBoard',\n",
       " 'Exterior1st_ImStucc',\n",
       " 'Exterior1st_MetalSd',\n",
       " 'Exterior1st_Plywood',\n",
       " 'Exterior1st_Stone',\n",
       " 'Exterior1st_Stucco',\n",
       " 'Exterior1st_VinylSd',\n",
       " 'Exterior1st_Wd Sdng',\n",
       " 'Exterior1st_WdShing',\n",
       " 'GarageType_2Types',\n",
       " 'GarageType_Attchd',\n",
       " 'GarageType_Basment',\n",
       " 'GarageType_BuiltIn',\n",
       " 'GarageType_CarPort',\n",
       " 'GarageType_Detchd',\n",
       " 'GarageType_missing',\n",
       " 'LandContour_Bnk',\n",
       " 'LandContour_HLS',\n",
       " 'LandContour_Low',\n",
       " 'LandContour_Lvl',\n",
       " 'BldgType_1Fam',\n",
       " 'BldgType_2fmCon',\n",
       " 'BldgType_Duplex',\n",
       " 'BldgType_Twnhs',\n",
       " 'BldgType_TwnhsE',\n",
       " 'Electrical_FuseA',\n",
       " 'Electrical_FuseF',\n",
       " 'Electrical_FuseP',\n",
       " 'Electrical_Mix',\n",
       " 'Electrical_SBrkr',\n",
       " 'Electrical_missing',\n",
       " 'MSSubClass_20',\n",
       " 'MSSubClass_30',\n",
       " 'MSSubClass_40',\n",
       " 'MSSubClass_45',\n",
       " 'MSSubClass_50',\n",
       " 'MSSubClass_60',\n",
       " 'MSSubClass_70',\n",
       " 'MSSubClass_75',\n",
       " 'MSSubClass_80',\n",
       " 'MSSubClass_85',\n",
       " 'MSSubClass_90',\n",
       " 'MSSubClass_120',\n",
       " 'MSSubClass_160',\n",
       " 'MSSubClass_180',\n",
       " 'MSSubClass_190',\n",
       " 'CentralAir_N',\n",
       " 'CentralAir_Y',\n",
       " 'Condition1_Artery',\n",
       " 'Condition1_Feedr',\n",
       " 'Condition1_Norm',\n",
       " 'Condition1_PosA',\n",
       " 'Condition1_PosN',\n",
       " 'Condition1_RRAe',\n",
       " 'Condition1_RRAn',\n",
       " 'Condition1_RRNe',\n",
       " 'Condition1_RRNn',\n",
       " 'RoofMatl_ClyTile',\n",
       " 'RoofMatl_CompShg',\n",
       " 'RoofMatl_Membran',\n",
       " 'RoofMatl_Metal',\n",
       " 'RoofMatl_Roll',\n",
       " 'RoofMatl_Tar&Grv',\n",
       " 'RoofMatl_WdShake',\n",
       " 'RoofMatl_WdShngl',\n",
       " 'Alley_Grvl',\n",
       " 'Alley_Pave',\n",
       " 'Alley_missing',\n",
       " 'MiscFeature_Gar2',\n",
       " 'MiscFeature_Othr',\n",
       " 'MiscFeature_Shed',\n",
       " 'MiscFeature_TenC',\n",
       " 'MiscFeature_missing',\n",
       " 'HouseStyle_1.5Fin',\n",
       " 'HouseStyle_1.5Unf',\n",
       " 'HouseStyle_1Story',\n",
       " 'HouseStyle_2.5Fin',\n",
       " 'HouseStyle_2.5Unf',\n",
       " 'HouseStyle_2Story',\n",
       " 'HouseStyle_SFoyer',\n",
       " 'HouseStyle_SLvl',\n",
       " 'Heating_Floor',\n",
       " 'Heating_GasA',\n",
       " 'Heating_GasW',\n",
       " 'Heating_Grav',\n",
       " 'Heating_OthW',\n",
       " 'Heating_Wall',\n",
       " 'MoSold_1',\n",
       " 'MoSold_2',\n",
       " 'MoSold_3',\n",
       " 'MoSold_4',\n",
       " 'MoSold_5',\n",
       " 'MoSold_6',\n",
       " 'MoSold_7',\n",
       " 'MoSold_8',\n",
       " 'MoSold_9',\n",
       " 'MoSold_10',\n",
       " 'MoSold_11',\n",
       " 'MoSold_12']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_columns = list(\n",
    "    preprocessor.named_transformers_[\"pipeline-4\"]\n",
    "    .named_steps[\"onehotencoder\"]\n",
    "    .get_feature_names_out(categorical_features)\n",
    ")\n",
    "ohe_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "new_columns = (\n",
    "    numeric_features + ordinal_features_reg + ordinal_features_oth + ohe_columns\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tuning `alpha` hyperparameter of `Ridge`\n",
    "- Recall that `Ridge` has a hyperparameter `alpha` that controls the fundamental tradeoff.\n",
    "- This is like `C` in `LogisticRegression` but, annoyingly, `alpha` is the inverse of `C`.\n",
    "- That is, large `C` is like small `alpha` and vice versa.\n",
    "- Smaller `alpha`: lower training error (overfitting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\"ridge__alpha\": 10.0 ** np.arange(-5, 5, 1)}\n",
    "\n",
    "pipe_ridge = make_pipeline(preprocessor, Ridge())\n",
    "\n",
    "search = GridSearchCV(pipe_ridge, param_grid, return_train_score=True, n_jobs=-1)\n",
    "search.fit(X_train, y_train)\n",
    "train_scores = search.cv_results_[\"mean_train_score\"]\n",
    "cv_scores = search.cv_results_[\"mean_test_score\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAG1CAYAAAARLUsBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWVElEQVR4nO3deXxU1f3/8dfMZIesBJKQHRAhoCwJWzBaUUFwwy5gVSyKWr/fWkWqrfxQW/lqcaEWK0K1QtGWIlbr0spiXCog+6rsazKBJIRANsiemd8fk0wICZj9TjLv5+NxH0nu3LnzuRnjvDnn3HNMdrvdjoiIiIgbMRtdgIiIiEh7UwASERERt6MAJCIiIm5HAUhERETcjgKQiIiIuB0FIBEREXE7CkAiIiLidhSARERExO14GF2AK7LZbGRmZuLv74/JZDK6HBEREWkEu91OUVERPXv2xGy+dBuPAlADMjMziY6ONroMERERaYaMjAyioqIueYwCUAP8/f0Bxy8wICDA4GpERESkMQoLC4mOjnZ+jl+KAlADarq9AgICFIBEREQ6mMYMX9EgaBEREXE7CkAiIiLidtQFJiIi0s6qqqqoqKgwuowOycvL63vv8GoMBSAREZF2Yrfbyc7OJj8/3+hSOiyz2Ux8fDxeXl4tOo8CkIiISDupCT89evTAz89Pc801Uc08fVlZWcTExLTo96cAJCIi0g6qqqqc4adbt25Gl9Nhde/enczMTCorK/H09Gz2eTQIWkREpB3UjPnx8/MzuJKOrabrq6qqqkXnUQASERFpR+r2apnW+v0pAImIiIjbUQASERERt6MAJCIiIu0mLi6OefPmGV2G7gITERGRS/vBD37A4MGDWyW4bNmyhS5durS8qBZSAGpHh3OKmP2ffUaXIa2gOUPwmjpur6mvYTaZ8LCY8DCbnV89LeftM5vwsFTvcx5ziX3VX537nOdznMvTYsZiNjn31Rzv3Fd9vAZ8inR+drudqqoqPDy+P1Z07969HSr6fgpA7aiwtJI1B08ZXYZIu7KYTY5QdEHYOj881QQqD4uJAB9PBkUFMjQ2mCHRwQT6NX+eDxFXZrfbKalo2a3czeXraWn0P06mTp3K119/zddff82rr74KwF//+lfuvfdeVq1axaxZs/j2229ZvXo1MTExzJgxg40bN3Lu3Dn69+/PnDlzuP76653ni4uLY/r06UyfPh1w3NX1l7/8hU8//ZTVq1cTGRnJH/7wB2699dZWv+7zKQC1o9gQP16ZNMjoMqSF7PZmPKfJr9H0F7HZ7VRU2amy2amoslFps1NZZaOiyk6lrebn6n3Vjzn32aqPa2ifzVb98wXnq3K8TpXN7jyfrYGyq2yOmsodPzXqWr4+7x8KfXp0JTEmmKGxQSTGBtMrtCtms1qVpOMrqagi4ZnVhrz23tnj8PNqXAR49dVXOXjwIAMHDmT27NkA7NmzB4Bf//rXzJ07l169ehEUFMTx48eZMGECzz33HD4+Prz99tvccsstHDhwgJiYmIu+xrPPPstLL73Eyy+/zGuvvcZdd91Feno6ISEhLb/Yi1AAakfdunrzw6FRRpch0mZsNjsVFwSmyppAVrPvvPBU81hVdeiqqLJxsqiMHel5bLfmkXa6mMM5Zzmcc5blWzMACPDxYEhMMImxwQyNCWZQdCD+PmolEmkrgYGBeHl54efnR3h4OAD79+8HYPbs2dxwww3OY7t168agQbX/0H/uuef48MMP+eSTT3j44Ycv+hpTp07lpz/9KQC///3vee2119i8eTM33nhjW1wSoAAkIq3IbDbhbbbg3cL/s0wZGQtA7tkydljz2VYdiL49nk9haSVfHzzlbCUym6BvmD9DY4OrW4qCieumNZbE9fl6Wtg7e5xhr90akpKS6vx87tw5nn32Wf7zn/84l6soKSnBarVe8jxXXnml8/suXbrg7+9PTk5Oq9R4MQpAIuKyQrt6c0NCGDckhAFQUWVjX1Yh29Pz2F4djE7kl7A/u4j92UX8Y5Pjf7IhXbwYGhPkbCm6Miqw0c39Iu3FZDJ1+P8uL7yb64knnmD16tXMnTuXPn364Ovry49//GPKy8sveZ4L1/QymUzYbLZWr/d8Hfs3LyJuxdNi5sqoIK6MCmLqaMe+nMJStlvzqluJ8vnuRAFnzpXz+b4cPt/n+BekxWyif4S/s4VoaEwwUcG+aiUSaSQvL69Grb21du1apk6dyu233w7A2bNnSUtLa+PqmkcBSEQ6tB4BPtw4MIIbB0YAUFZZxZ7MmlaiPLan55NdWMruE4XsPlHI2xvSAeju783QmCDnWKKBkYH4tFK3gEhnExcXx6ZNm0hLS6Nr164XbZ3p06cP//rXv7jlllswmUw8/fTTbd6S01wKQCLSqXh7WBga4wg1NTLzS5zjiLZb89lzooBTRWWs3nOS1XtOAuBpMTGgZyBDawZYxwYREehr1GWIuJTHH3+cn/3sZyQkJFBSUsJf//rXBo/74x//yH333UdycjKhoaH85je/obCwsJ2rbRyTvTn323ZyhYWFBAYGUlBQQEBAgNHliEgrK62o4rsTBWxPr+06yz1bVu+4iEAfZ5dZYmwwCREBeHloBSFpntLSUo4dO0Z8fDw+Pj5Gl9NhXer32JTPb7UAiYjb8fG0MCwuhGFxjjlG7HY7GWdKqluIHKFof3YRWQWlfPptFp9+mwWAt4eZKyIDSYwNZkj13EQ9/PVBJtIRKQCJiNszmUzEdPMjppsfE4dEAlBcXsmujILqcUSOYJRXXMHW9Dy2puc5nxsd4lvbbRYTTL9wfzwsaiUScXUKQCIiDfDz8mBU726M6t0NcLQSHcs9x3ZrvjMUHThZRMaZEjLOlPDxzkzAMb/KoOhAhsWF8MOhUcSHGr/oo4jUpwAkItIIJpOJXt270qt7V36c6JjRvai0gp0Z+WxPz3d2nxWVVrLx6Bk2Hj3Da18eZnSfbtw1IpYbEsLwVMuQiMtQABIRaSZ/H09SLutOymWO1a1tNjtHTp1luzWP1XtO8tWBHL45fJpvDp+mu783k5Oi+emIGCKDdHeZiNEUgEREWonZbOKyMH8uC/Nn8rAYjucVs3xLBu9uyeBUURnzvzrMgv8e5trLe3DXyBiu6dsDixZ2FTGEApCISBuJCvbjV2Mv55HrLiN170mWbkrnm8On+WJ/Dl/szyEyyJefDo9mUlI0PQJ0N5lIe1IAEhFpY54WMxOuiGDCFREcPXWWZZut/HPbcU7klzD3s4PM+/wQYweEcdeIWEb16oZZrUIibU4BSESkHfXq3pVZNyXwq7GXs3J3Fks3WtmanseK77JZ8V028aFduHN4DD9OjCK4i5fR5Yp0WgpAIiIG8PG0cPuQKG4fEsX+7EL+scnKv7af4FjuOZ5fsY+XPzvATVdEcNeIGBJjg7Vwq0gr01IYDdBSGCJihHNllXyyK5O/b0xnT2bt+kn9wv25a0QME4dE4u/jaWCF0hJaCqN1tNZSGJqUQkTERXTx9uCnw2P4zy+v4uNfjGZSUhQ+nmb2Zxfx9Md7GPH7L5j5r2/ZfaLA6FLFzdhsNl588UX69OmDt7c3MTExPP/884waNYonn3yyzrGnTp3C09OTr776yqBqG0ctQA1QC5CIuIqCkgo+3H6cpZusHMo569w/KCqQu0bEcsugnvh6WQysUBqrXsuF3Q4VxcYU4+kHTehW/c1vfsNf/vIX/vjHP3LVVVeRlZXF/v37KS0t5eWXXyYtLc3ZTTt//nxeeOEFrFYrZnPrt7O0VguQAlADFIBExNXY7Xa2pOWxdFM6K7/LprzKBoC/jwc/GhrFnSNi6Bvmb3CVcin1PrjLz8HvexpTzP/LBK/GLdNSVFRE9+7dmT9/Pvfff3+dx06dOkXPnj358ssvSUlJASA5OZmrrrqKl156qdXLBnWBiYi4FZPJxPD4EF69YwgbZo7hyfH9iAnxo6i0kiXr0xj7xzVM+vMGPt55grLKKqPLlU5k3759lJWVcd1119V7rHv37txwww0sXboUgGPHjrFhwwbuuuuu9i6zyQy/C2zBggW8/PLLZGVlMWDAAObNm+dMkQ15/fXXmT9/PmlpacTExDBr1izuueeeOsd88MEHPP300xw5coTevXvz/PPPc/vtt7f1pYiItItuXb156JrePJjSi3WHc1m6KZ3P9+WwOe0Mm9POENLFi58kRXHn8Bhiu2kxVpfl6edoiTHqtRvJ1/fSS7fcddddPProo7z22mv84x//YMCAAQwaNKilFbY5Q1uAli9fzvTp05k1axY7duwgJSWF8ePHY7VaGzx+4cKFzJw5k9/97nfs2bOHZ599ll/84hf8+9//dh6zYcMGJk+ezJQpU9i1axdTpkxh0qRJbNq0qb0uS0SkXZjNJq7u2503piTxzW/G8Nj1fYkI9OHMuXLe+Poo17z8X6Ys2sSq3dlUVneZiQsxmRzdUEZsTRj/c9lll+Hr68sXX3zR4OMTJ06ktLSUVatW8Y9//IO77767tX5DbcrQMUAjRoxg6NChLFy40Lmvf//+TJw4kTlz5tQ7Pjk5mdGjR/Pyyy87902fPp2tW7eybt06ACZPnkxhYSErV650HnPjjTcSHBzMsmXLGlWXxgCJSEdVWWXjqwOnWLopna8PnqLm//BhAd5MHhbDHcOi6anFWA3RkW+Df/bZZ3n11VeZN28eo0eP5tSpU+zZs4dp06YBjlagPXv28O233zp7aNpKa40BMqwLrLy8nG3bttW7fW7s2LGsX7++weeUlZXVu1hfX182b95MRUUFnp6ebNiwgccee6zOMePGjWPevHkXraWsrIyysjLnz4WFhRc9VkTElXlYzNyQEMYNCWFknClm2WYr723N4GRhGX/64hDzvzzEmH5hjsVYL+uuZTekUZ5++mk8PDx45plnyMzMJCIigoceesj5+F133cVNN93E1Vdf3abhpzUZFoByc3OpqqoiLCyszv6wsDCys7MbfM64ceN46623mDhxIkOHDmXbtm0sXryYiooKcnNziYiIIDs7u0nnBJgzZw7PPvtsyy9KRMSFRIf48esb+zH9+r6s3pPN0k3pbDx6hs/3neTzfSeJCvblzhEx/CQxmu7+3kaXKy7MbDYza9YsZs2a1eDjEyZMoKPdVG74XWAXTu9ut9svOuX7008/zfjx4xk5ciSenp7cdtttTJ06FQCLpXYejKacE2DmzJkUFBQ4t4yMjGZejYiI6/HyMHPLoJ68++AoPp9xDfeNjifAx4PjeSW8tOoAyS98wcP/2M6GI6c73IeYSHMZFoBCQ0OxWCz1WmZycnLqteDU8PX1ZfHixRQXF5OWlobVaiUuLg5/f39CQ0MBCA8Pb9I5Aby9vQkICKiziYh0Rn16dOWZWxLYPOt65v5kEENigqiosvOfb7P46V82ct0rX7No3THyi8uNLlWkTRkWgLy8vEhMTCQ1NbXO/tTUVJKTky/5XE9PT6KiorBYLLz77rvcfPPNztkmR40aVe+cn3322feeU0TEnfh4WvhxYhQf/u9oPn3kKu4aEUMXLwtHT53j//6zlxG//4JfvbeL7dY8o0sVaROGzgM0Y8YMpkyZQlJSEqNGjeLNN9/EarU6B1bNnDmTEydO8M477wBw8OBBNm/ezIgRI8jLy+OVV15h9+7dvP32285zPvroo1x99dW8+OKL3HbbbXz88cd8/vnnzrvERESkrgE9A3n+9iuYOaE/H+04wdJNVvZlFfLB9uN8sP04Y/r14OmbE4gP1ZxC0nkYGoAmT57M6dOnmT17NllZWQwcOJAVK1YQGxsLQFZWVp05gaqqqvjDH/7AgQMH8PT05Nprr2X9+vXExcU5j0lOTubdd9/lqaee4umnn6Z3794sX76cESNGtPfliYh0KF29Pbh7ZCx3jYhhR0Y+f9+Yzr93ZfLl/hzWHjrFtKt68csxfejibfgcuiItprXAGqB5gEREHI6cOsuz/97LmoOnAMd8QjPH9+e2wT0veXOJ1Fczf01cXNz3zq4sF1dSUkJaWprWAhMRkbbTu3tX3r53GG/dk0RMiB8nC8uYvnwnP/7zBnafKDC6vA7F09MTgOJig1aA7yTKyx0D9M+/+7s51ALUALUAiYjUV1pRxaJ1x5j/5WFKKqowmeCOYTE8PrYv3bpqHqHGyMrKIj8/nx49euDn56dWtCay2WxkZmbi6elJTExMvd9fUz6/FYAaoAAkInJxWQUlvLByPx/vdCzkGeDjwYwb+nL3yFg8LOpYuBS73U52djb5+flGl9Jhmc1m4uPj8fLyqveYAlALKQCJiHy/zcfO8LtP9rA3y7F80OVh/vz2lgSS+4QaXJnrq6qqoqKiwugyOiQvLy/n1DcXUgBqIQUgEZHGqbLZeXeLlbmrD5BX7PhAn3BFOP9vQn+igv0Mrk7cjQJQCykAiYg0TX5xOa+kHuTvG9Ox2cHbw8z//KA3D13TGx/Plg1WFWksBaAWUgASEWmefVmF/O6TPWw6dgaAyCBfnrqpPzcODNeAX2lzCkAtpAAkItJ8drudT7/L4vef7iOzoBSA5N7d+N2tA+gb5m9wddKZKQC1kAKQiEjLFZdX8uf/HuHPa45SXmnDYjYxZWQsj93Ql0BfT6PLk05IAaiFFIBERFpPxplinvt0L6v3nAQgpIsXvx53OT9JisZiVreYtB4FoBZSABIRaX1rD53i2X/v5XDOWQCuiAzkd7cmkBgbYnBl0lkoALWQApCISNuoqLLxzoZ05qUepKisEoAfDonkyfH96BHg8z3PFrk0BaAWUgASEWlbp4rKeHn1fv657Th2O3TxsvDL6y7j3tFxeHvotnlpHgWgFlIAEhFpH7sy8vntJ3vYmZEPQHxoF565OYFr+/UwtjDpkBSAWkgBSESk/dhsdv614wQvrNxP7tkyAMb068EzNycQF9rF4OqkI1EAaiEFIBGR9ldUWsFrXx5m8bpjVNrseFnMTEuJ5+Fr+9DF28Po8qQDUABqIQUgERHjHM45y+z/7GXNwVMAhAV4M3N8f24b3FOzScslKQC1kAKQiIix7HY7X+zLYfZ/9mI9UwxAYmwwz946gIGRgQZXJ65KAaiFFIBERFxDaUUVi9YdY/6XhympqMJkgjuGxfD42L506+ptdHniYhSAWkgBSETEtWQVlPDCyv18vDMTgAAfD2bc0Je7R8biYTEbXJ24CgWgFlIAEhFxTZuPneG3n+xhX1YhAJeH+fPbWxJI7hNqcGXiChSAWkgBSETEdVXZ7CzbbGXuZwfIL64AYMIV4fy/Cf2JCvYzuDoxkgJQCykAiYi4vvzicl5JPcjfN6Zjs4OPp5n/uaYPP7+mFz6emk3aHSkAtZACkIhIx7Evq5DffbKHTcfOABAZ5MtTN/XnxoHhum3ezSgAtZACkIhIx2K32/n0uyye/3QfWQWlAIzu043f3jKAvmH+Blcn7UUBqIUUgEREOqbi8kr+/N8j/HnNUcorbVjMJu4ZFcv06/sS6OtpdHnSxpry+a17B0VEpNPw8/JgxtjL+WLGNYxNCKPKZuev36Rx7dz/8t8DOUaXJy5EAUhERDqd6BA/3rwniXfuG07v7l04c66cB97Zymd7so0uTVyEApCIiHRaV/ftzspHr+amKyKoqLLzv0u3s/K7LKPLEhegACQiIp2al4eZV+8YzG2De1Jps/Pwsh18sivT6LLEYApAIiLS6XlYzLwyaTA/HBpJlc3O9Hd38K/tx40uSwykACQiIm7BYjYx98eDuGNYNDY7/Oqfu3hvS4bRZYlBFIBERMRtmM0mfn/7Fdw9Mga7HX79wbcs3ZRudFliAAUgERFxK2azif+7bSD3jo4DYNaHu1nyzTFji5J2pwAkIiJux2Qy8czNCfz86l4A/O7fe3lr7VGDq5L2pAAkIiJuyWQy8eT4fjx8bR8Anvt0H69/ddjgqqS9GB6AFixYQHx8PD4+PiQmJrJ27dpLHr906VIGDRqEn58fERER3HvvvZw+fdr5+JIlSzCZTPW20tLStr4UERHpYEwmE4+Pu5zHru8LwMurDzDv84NolajOz9AAtHz5cqZPn86sWbPYsWMHKSkpjB8/HqvV2uDx69at45577mHatGns2bOHf/7zn2zZsoX777+/znEBAQFkZWXV2Xx8fNrjkkREpAN69PrLeGLc5QDM+/wQf/hMIaizMzQAvfLKK0ybNo3777+f/v37M2/ePKKjo1m4cGGDx2/cuJG4uDgeeeQR4uPjueqqq/j5z3/O1q1b6xxnMpkIDw+vs4mIiFzKL67tw6wJ/QGY/9VhXli5XyGoEzMsAJWXl7Nt2zbGjh1bZ//YsWNZv359g89JTk7m+PHjrFixArvdzsmTJ3n//fe56aab6hx39uxZYmNjiYqK4uabb2bHjh2XrKWsrIzCwsI6m4iIuJ8Hru7F725JAOCNNUeZ/Z+9CkGdlGEBKDc3l6qqKsLCwursDwsLIzu74cXqkpOTWbp0KZMnT8bLy4vw8HCCgoJ47bXXnMf069ePJUuW8Mknn7Bs2TJ8fHwYPXo0hw4dumgtc+bMITAw0LlFR0e3zkWKiEiHM3V0PM9NHAjAX79J4+mPd2OzKQR1NoYPgjaZTHV+ttvt9fbV2Lt3L4888gjPPPMM27ZtY9WqVRw7doyHHnrIeczIkSO5++67GTRoECkpKbz33nv07du3Tki60MyZMykoKHBuGRmaGVRExJ3dPTKWl350JSYT/H2jlf/34XcKQZ2Mh1EvHBoaisViqdfak5OTU69VqMacOXMYPXo0TzzxBABXXnklXbp0ISUlheeee46IiIh6zzGbzQwbNuySLUDe3t54e3u34GpERKSzmTQsGg+Licf/uYt3t2RQUWXnpR9ficXc8D/SpWMxrAXIy8uLxMREUlNT6+xPTU0lOTm5wecUFxdjNtct2WKxAFy0j9Zut7Nz584Gw5GIiMil/HBoFPPuGILFbOKD7ceZ8d5OKqtsRpclrcCwFiCAGTNmMGXKFJKSkhg1ahRvvvkmVqvV2aU1c+ZMTpw4wTvvvAPALbfcwgMPPMDChQsZN24cWVlZTJ8+neHDh9OzZ08Ann32WUaOHMlll11GYWEhf/rTn9i5cyevv/66YdcpIiId162DeuJpNvHLZTv4eGcmlVV25t0xGE+L4aNIpAUMDUCTJ0/m9OnTzJ49m6ysLAYOHMiKFSuIjY0FICsrq86cQFOnTqWoqIj58+fzq1/9iqCgIMaMGcOLL77oPCY/P58HH3yQ7OxsAgMDGTJkCGvWrGH48OHtfn0iItI5jL8igoUWM/+7dBuffpdFeZWN+XcOwdvDYnRp0kwmu+7vq6ewsJDAwEAKCgoICAgwuhwREXERX+3P4ed/30Z5pY0x/Xqw4K6h+HgqBLmKpnx+q/1ORESkka7t14NFP0vC28PMl/tzeOCdrZRWVBldljSDApCIiEgTpFzWnb/eOwxfTwtrD+Vy35ItFJdXGl2WNJECkIiISBMl9w7l7fuG08XLwvojp5m6eAtnyxSCOhIFIBERkWYYHh/CO9NG4O/twea0M9yzaBOFpRVGlyWNpAAkIiLSTImxwfz9/hEE+Hiw3ZrPlEWbKShWCOoIFIBERERaYFB0EP94YCTBfp7sysjnrkUbyTtXbnRZ8j0UgERERFpoYGQgyx4cSbcuXuw+UchP/7KR3LNlRpcll6AAJCIi0gr6hQfw7oMj6e7vzf7sIn765kZyikqNLksuQgFIRESklVwW5s/yB0cSHuDDoZyz3PHGRrILFIJckQKQiIhIK+rVvSvLfz6SyCBfjuaeY/KbGziRX2J0WXIBBSAREZFWFtutC+8+OJKoYF/STxcz+Y0NZJwpNrosOY8CkIiISBuIDvHjvZ+PIrabH8fzSpj8xgbST58zuiyppgAkIiLSRnoG+bL8wVH06t6FzIJSJr2xgSOnzhpdlqAAJCIi0qbCA31498GRXNajKycLy7jjzY0cOllkdFluTwFIRESkjfXwd4SgfuH+nCpyhKD92YVGl+XWFIBERETaQbeu3ix7YCQDIwM4fa6cn765kd0nCowuy20pAImIiLST4C5eLL1/JIOig8grruDOv2xkV0a+0WW5JQUgERGRdhTo68nfpg0nMTaYwtJK7n5rE9vS84wuy+0oAImIiLSzAB9P3r5vOMPjQygqq+SeRZvYfOyM0WW5FQUgERERA3T19mDJvcNI7t2Nc+VV/GzxZtYfyTW6LLehACQiImIQPy8PFk8dRsploZRUVHHvX7ew5uApo8tyCwpAIiIiBvLxtPCXe5IY068HZZU27n9nK1/tzzG6rE5PAUhERMRgPp4W/nx3ImMTwiivtPHg37by2Z5so8vq1BSAREREXICXh5nX7xrKTVdEUFFl53+Xbmfld1lGl9VpKQCJiIi4CE+LmVfvGMxtg3tSabPz8LIdfLIr0+iyOiUFIBERERfiYTHzyqTB/GhoFFU2O9Pf3cEH244bXVanowAkIiLiYixmEy//+EruGBaNzQ6Pv7+L5VusRpfVqSgAiYiIuCCz2cTvb7+CKSNjsdvhNx98x983phtdVqehACQiIuKizGYTs28bwL2j4wB46qPdfLTjhLFFdRIKQCIiIi7MZDLxzM0JTE2OA+CdDWmG1tNZKACJiIi4OJPJ5AxAu08UUlpRZWxBnYACkIiISAcQ282P0K5elFfZ2H2iwOhyOjwFIBERkQ7AZDKRGBsMwNb0PIOr6fgUgERERDoIZwBKUwBqKQUgERGRDiIxNgSA7dY87Ha7wdV0bIYHoAULFhAfH4+Pjw+JiYmsXbv2kscvXbqUQYMG4efnR0REBPfeey+nT5+uc8wHH3xAQkIC3t7eJCQk8OGHH7blJYiIiLSLgZEBeHmYOXOunGO554wup0MzNAAtX76c6dOnM2vWLHbs2EFKSgrjx4/Ham14tst169Zxzz33MG3aNPbs2cM///lPtmzZwv333+88ZsOGDUyePJkpU6awa9cupkyZwqRJk9i0aVN7XZaIiEib8PawcGVkIKBxQC1lshvYhjZixAiGDh3KwoULnfv69+/PxIkTmTNnTr3j586dy8KFCzly5Ihz32uvvcZLL71ERkYGAJMnT6awsJCVK1c6j7nxxhsJDg5m2bJljaqrsLCQwMBACgoKCAgIaO7liYiItLo5K/fxxtdHuWNYNC/86Eqjy3EpTfn8NqwFqLy8nG3btjF27Ng6+8eOHcv69esbfE5ycjLHjx9nxYoV2O12Tp48yfvvv89NN93kPGbDhg31zjlu3LiLnhOgrKyMwsLCOpuIiIgrSqoeB6QWoJYxLADl5uZSVVVFWFhYnf1hYWFkZ2c3+Jzk5GSWLl3K5MmT8fLyIjw8nKCgIF577TXnMdnZ2U06J8CcOXMIDAx0btHR0S24MhERkbYzNCYIgMM5Z8kvLje2mA7M8EHQJpOpzs92u73evhp79+7lkUce4ZlnnmHbtm2sWrWKY8eO8dBDDzX7nAAzZ86koKDAudV0p4mIiLiabl296RXaBXDcDSbN42HUC4eGhmKxWOq1zOTk5NRrwakxZ84cRo8ezRNPPAHAlVdeSZcuXUhJSeG5554jIiKC8PDwJp0TwNvbG29v7xZekYiISPtIjA3maO45tqblMabfxT/f5OIMawHy8vIiMTGR1NTUOvtTU1NJTk5u8DnFxcWYzXVLtlgsAM75EEaNGlXvnJ999tlFzykiIp2U3Q5lRZCfAbmHoKrS6IpajWaEbjnDWoAAZsyYwZQpU0hKSmLUqFG8+eabWK1WZ5fWzJkzOXHiBO+88w4At9xyCw888AALFy5k3LhxZGVlMX36dIYPH07Pnj0BePTRR7n66qt58cUXue222/j444/5/PPPWbdunWHXKSIizVQTYkrzoSS/aV9LC8B2XujxDoReV0PvMdD7OgiObddLaU1JcY4AtCsjn4oqG54Ww0e0dDiGBqDJkydz+vRpZs+eTVZWFgMHDmTFihXExjr+o8zKyqozJ9DUqVMpKipi/vz5/OpXvyIoKIgxY8bw4osvOo9JTk7m3Xff5amnnuLpp5+md+/eLF++nBEjRrT79YmICNUhprDpAaYk3xFi7C1c+dzsCRZPKCuAff92bAAhvR1hqM91EHcVePu37HXaUa/QrgT5eZJfXMGezEIGRwcZXVKHY+g8QK5K8wCJiFzAZnOEmOa2xNhtLXt9ixf4BIFvUNO/evo5Xj9zJxz5Eo58ARmb6wYrsydEj4De1zpCUcRgMLt2q8p9S7bw5f4cnrqpP/en9DK6HJfQlM9vQ1uARETEBVVVwLE1sPcjSFsHxWccIYYW/nvZw6cFIca3Za9tskBUomO75gkoLYS0tXD4C0coyjsG6esc25f/B74h1WHoOsfXgJ4te/02kBgbzJf7c3QnWDMpAImIiCP0HP3aEXr2/wdKLvKh6uHbvADjEwSePm18EU3gEwD9bnJsAGeOVrcOfeX4PZScgd0fODaA7v0dXWW9r4XY0S0PZK0g6byV4b9vuhepTwFIRMRdVZbDsa9hz0eO0FOaX/uYXyj0vwX63QxB0bVBxqOTThkS0suxDbvfEQaPb3V0lR35Ek5sh1P7HNuG+WDxhtjk2vFDPRLAgPBxZVQQHmYTOUVlHM8rITrEr91r6Mg0BqgBGgMkIp1WZTkc/W9tS09pQe1jXbpD/1sh4TZHK4dF/0YGHF2AR/9b3UL0JRSeqPt41/DqO8vGQK8fQNfu7Vbaba9/w66MfOZNHszEIZHt9rquSmOARESkVmWZo2tn78dw4NMLQk8PSLgVEiY6WjXMFsPKdFl+ITDwh47Nbofcg44gdPgLxxips9mw6x+ODSBiUO2t9tEjwMOrzUpLig1mV0Y+W9PPKAA1kQKQiEhnVFnm+JDe8xEcWOm4BbxG1zBHS8+AiRAzSqGnKUwm6H65Yxv5P47fs3VDbetQ9neQtcuxrfsjeHZx3GLf5zpHKOrWp1W7yxJjg1m07hhb0zQQuqkUgEREOouKUseH8N6PqkNPYe1jXcNrW3piRir0tBYPb0e3V68fwA2zoehk3e6yczlwaLVjAwiMqb3Vvtc14BvcopevGQh94GQRRaUV+Pt4tuh87kQBSESkI6sohcOfV3dvrYTyotrH/CMc43kSJjq6Ylx8XptOwT8MBk12bDYb5OypvdXeugEKrLD9bcdmMkNkYm13WWRik8dd9QjwITrEl4wzJeyw5nN13/Ybf9TRKQCJiHQ0FSWO0LPnIzi4CsrP1j7m39MRegZMhKjhCj1GMpsh/ArHdtV0KC+G9G9qxw/lHoDjWxzb1y9esFTHGAiOa9TLJMYEk3GmhK3peQpATaAAJCLSEVSUwKFUR/fWwdV1Q09AVG3oiUxS6HFVXn5w2Q2ODaDguGNw+pEvHN1mJXnNWqojMS6Ej3Zmsl0LozaJApCIiKsqL4bDqdUtPauh4lztY4HRtd1bkYkKPR1RYBQMneLYbFXnLdXxJRzfDGeOOLYtfwGzR/VSHWMg8V7o0s15mppxQDuseVRW2fDQwqiNogAkIuJKys/Boc8cY3oOroaK4trHAmMcA5kH3O4IPZr5t/MwN2apjm8cW84++PEi51P7hvnj7+1BUVkl+7OLGBgZaOCFdBwKQCIiRis/5wg7ez9ydHOdH3qCYhytPAMmQs+hCj3uoqGlOr57H7563hGIbDZnq5/FbGJwTBBrD+Wy3ZqnANRICkAiIkYoO+u4NXrPR47QU1lS+1hQrCPwJEyEnkMUesSxTMfo6Y65hUrOQM5eCB/ofDgpNoS1h3LZmpbHPaPiDCuzI1EAEhFpL2VFdVt6KktrHwuOrw09EYMUeqQ+Dy/HxJVHvnB0j50XgBKrxwFt00DoRlMAEhFpS2VFcGCVI/Qc/rxu6AnpVdu9FX6lQo98v/gURwA6ttYxE3W1wTFBmE1wIr+ErIISIgKNX63e1SkAiYi0tuIztfP0HP4cqspqHwvpXdvSE36FQo80TdzVjq9p6xx3jlXP6N3V24P+EQHsySxkW3oeN1+pAPR9FIBERFrqXK7j7py0dZD2jWP23/N1u6w29IQNUOiR5osYBF7+jrXdsr91jBGrlhQbzJ7MQram5XHzlT0NLLJjUAASEWmqs6cgvTrspK2DU/vqH9MjAfrd7Ag+PRIUeqR1WDwgNtkxgP7Y2joBaGhsMG9vSNc4oEZSABIR+T5FJ+sGntwD9Y/pMQDiRjtm7I0dDV1C279OcQ/xVzsCUNpaGP2Ic3dSXAgAe7MKKS6vxM9LH/GXot+OiMiFCrPO69JaB6cP1T8m7IrawBOTXGdmXpE2FZ/i+Jq+HqoqwOJYAT4yyJeIQB+yCkrZmZFPcm+F8EtRABIRKcysDTtp6xzLD9RhctxyHJdSHXhGgV+IIaWKEHYF+ARBab5j+YzoYc6HhsYG8+m3WWxLy1MA+h4KQCLifgqOV3dnrXW09Jw5WvdxU/Uq3nEpju6s2FHgG2xMrSIXMpsdQXz/fyBtTZ0AlFQTgKwaB/R9FIBEpPPLt9aO30lfB3lpdR83mR1318RdBbFXQcxI8A0yolKRxolLcQSgY2sh5VfO3UmxjpbJ7el52Gx2zGYNvr+YZgeg/Px83n//fY4cOcITTzxBSEgI27dvJywsjMjIyNasUUSk8ex2yE+vG3jyrXWPMVmg52BH605cCsSMAB+tnyQdSHz1fEDWjVBZ7pglGugX4Y+vp4XC0koO5Zzl8nB/A4t0bc0KQN9++y3XX389gYGBpKWl8cADDxASEsKHH35Ieno677zzTmvXKSLSMLvd0aJTM34n/RsoyKh7jMkCkUPrBh5vfTBIB9ajP/iFQnEunNjquDUe8LSYGRwdxIajp9mWnqcAdAnNCkAzZsxg6tSpvPTSS/j71/5yx48fz5133tlqxYmI1GO3O8bs1ISdtHVQeKLuMWYPiEysDjxXQfQI8O5qTL0ibcFkcvy3vfcjRzdYdQACSIoLZsPR02xNP8OdI2KMq9HFNSsAbdmyhTfeeKPe/sjISLKzs1tclIiIk90Op4/UDlhOWwdFWXWPMXtCVNJ5gWc4eHUxpl6R9hKf4ghAaWuB3zh3D9XCqI3SrADk4+NDYWFhvf0HDhyge/fuLS5KRNyY3Q65h+oGnrMn6x5j8YLIJEfYiRsNUcPBy8+YekWMEn+N42vGZqgoBU8fAIbGBGMyQfrpYk4VldHd39vAIl1XswLQbbfdxuzZs3nvvfcAMJlMWK1WnnzySX70ox+1aoEi0slUljkCTWGWoyWnKBuKMqu/ZkHOfjiXU/c5Fm+IGnZe4BkGnlrsUdxctz7QNRzOZsPxzc6B0YG+nvTt4c+Bk0VsS8/jxoHhBhfqmpoVgObOncuECRPo0aMHJSUlXHPNNWRnZzNq1Cief/751q5RRDoCW5VjUdDzw0ydkFP9ffHp7z+Xh895gecqR2tP9b9uRaSayeToBvvun3BsTe2dYTi6wRwB6IwC0EU0KwAFBASwbt06vvzyS7Zv347NZmPo0KFcf/31rV2fiBjNbnfMONtQmDk/6Jw9Cfaqxp3T4gX+4eAfcd5W/XNwrGOBRw8124t8r7iaALS2zu6k2GCWbbZqHNAlNDkAVVZW4uPjw86dOxkzZgxjxoxpi7pEpD2UFzccZi7smqosbdz5TGbo0gMCLgg154ecgJ6OWZW1OrpIy9WsC3ZiG5Sfcw7+T4pzDITefaKQ0ooqfDwtRlXospocgDw8PIiNjaWqqpH/0hOR9ldVcelxNkXZjsfKChp/Tt9g8O9ZG2oCzg844Y7HunQHiyaYF2k3wfEQGO2Y+8q6EfpcB0BMiB+hXb3IPVvOdycKGBantesu1Kz/Uz311FPMnDmTv//974SE6JfaaLYqqCgxuopWYr/gR3vDj9kvOO6Sj7X2+S722IW12xzvjb0KbDbH1zr7qn+ut6/meNsF+y5xvN1W+xp19l3sNasuOP6C78/fV3a2tjXnXG7967wYzy4XabE5L+h0DdcYHBFXZDI5usF2/cNx52R1ADKZTCTGBrN6z0m2pecpADWgWQHoT3/6E4cPH6Znz57ExsbSpUvd+Ta2b9/eKsV1Oie2wyKNk5J2YvY8L8hUdz011CXl7a/uKJGOLL46AB1bU2d3UmwIq/ecZGtaHlxjUG0urFkBaOLEia1WwIIFC3j55ZfJyspiwIABzJs3j5SUlAaPnTp1Km+//Xa9/QkJCezZsweAJUuWcO+999Y7pqSkBB8f/Qu2c7vgQ7zOh/r3PGa2OJZLMFsc41hqvjr3WRwrMJvOf/y8ffWOP++Y9jre069uyPENcTxfRDq3uOrPzMydUFoIPgFA7YSI26152O12TPqHTh3NCkC//e1vW+XFly9fzvTp01mwYAGjR4/mjTfeYPz48ezdu5eYmPrTd7/66qu88MILzp8rKysZNGgQP/nJT+ocFxAQwIEDB+rsc4nwEzkU/l/W9x/XGtrlP/RLhYoLHm+tx/QHLCJSV1A0BMc51sSzboC+4wAYGBmAl4eZM+fKOZZ7jl7dtRzM+Vo0WnHbtm3s27cPk8lEQkICQ4YMadLzX3nlFaZNm8b9998PwLx581i9ejULFy5kzpw59Y4PDAwkMLB2xeaPPvqIvLy8ei0+JpOJ8HAXnPfAbNFstSIi0vrir3YEoGNrnAHI28PCoKhAtqTlsTU9TwHoAs1qH8/JyWHMmDEMGzaMRx55hIcffpjExESuu+46Tp061ahzlJeXs23bNsaOHVtn/9ixY1m/fn2jzrFo0SKuv/56YmNj6+w/e/YssbGxREVFcfPNN7Njx45LnqesrIzCwsI6m4iISIcRVz0JYlrd+YCc64KlaT6gCzUrAP3yl7+ksLCQPXv2cObMGfLy8ti9ezeFhYU88sgjjTpHbm4uVVVVhIWF1dkfFhbWqAVVs7KyWLlypbP1qEa/fv1YsmQJn3zyCcuWLcPHx4fRo0dz6NChi55rzpw5ztalwMBAoqOjG3UNIiIiLqFmPqCsb6H4jHN3Uqzj7q9tVgWgCzUrAK1atYqFCxfSv39/576EhARef/11Vq5c2aRzXTgoq7EDtZYsWUJQUFC9AdkjR47k7rvvZtCgQaSkpPDee+/Rt29fXnvttYuea+bMmRQUFDi3jIyMJl2DiIiIofzDodtlgB3Sa3tREqtbgA7nnCW/uNyg4lxTswKQzWbD09Oz3n5PT09sNlujzhEaGorFYqnX2pOTk1OvVehCdrudxYsXM2XKFLy8vC55rNlsZtiwYZdsAfL29iYgIKDOJiIi0qHE1+8GC+niRa9Qx1Q1WhajrmYFoDFjxvDoo4+SmZnp3HfixAkee+wxrrvuukadw8vLi8TERFJTU+vsT01NJTk5+ZLP/frrrzl8+DDTpk373tex2+3s3LmTiIiIRtUlIiLSIdV0g12wLlhNK5ACUF3NCkDz58+nqKiIuLg4evfuTZ8+fYiPj6eoqOiSXU0XmjFjBm+99RaLFy9m3759PPbYY1itVh566CHA0TV1zz331HveokWLGDFiBAMHDqz32LPPPsvq1as5evQoO3fuZNq0aezcudN5ThERkU6pZj6gnD3Vs8E71KwLtlUBqI5m3QYfHR3N9u3bSU1NZf/+/djtdhISEpq8GvzkyZM5ffo0s2fPJisri4EDB7JixQrnXV1ZWVlYrdY6zykoKOCDDz7g1VdfbfCc+fn5PPjgg2RnZxMYGMiQIUNYs2YNw4cPb86lioiIdAxdQqFHAuTsdXSDDbgdqG0B2pWRT3mlDS8PTZAKYLLbL7Ugk3sqLCwkMDCQgoICjQcSEZGOY8WvYfMbkDQNbn4FAJvNztDnUskvruCjX4xmcHSQsTW2oaZ8fjcrBj7yyCP86U9/qrd//vz5TJ8+vTmnFBERkZZqYCC02WwiMaa6GyztTEPPckvNCkAffPABo0ePrrc/OTmZ999/v8VFiYiISDPEjQZMkHsQimrvsh6qgdD1NCsAnT59us6SFDUCAgLIzc1t4BkiIiLS5nyDIfwKx/fn3Q2WFFs7EFojXxyaFYD69OnDqlWr6u1fuXIlvXr1anFRIiIi0kzObrA1zl2DooPwMJs4VVTG8bwSgwpzLc26C2zGjBk8/PDDnDp1ijFjxgDwxRdfMHfu3IvenSUiIiLtIC4FNsyv0wLk42lhQGQguzLy2Zp+hugQLczdrAB03333UVZWxvPPP8///d//ARAfH8+f//znBuftERERkXYSmwwmC+Qdg4LjEBgFOLrBdmXksy09j9uHRBlcpPGa1QVWUlLCz372M44fP87Jkyf59ttvefjhh793CQsRERFpYz4B0HOw4/uGxgFpZXigmQHotttu45133gEc639df/31vPLKK0ycOJGFCxe2aoEiIiLSRDWzQh+rHQdUMyHigZNFFJZWGFGVS2lWANq+fTspKY5f7vvvv09YWBjp6em88847Dc4PJCIiIu2oZl2wtLVQfddXjwAfokN8sdthpzXfuNpcRLMCUHFxMf7+/gB89tln/PCHP8RsNjNy5EjS09NbtUARERFpophRYPaAggzIS3PuTooNAbQuGLTgNviPPvqIjIwMVq9ezdixYwHIycnR0hEiIiJG8+oCkUmO78+bFbp2ZXjNCN2sAPTMM8/w+OOPExcXx4gRIxg1ahTgaA0aMmRIqxYoIiIizRB/8XFAO6z5VFbZjKjKZTQrAP34xz/GarWydevWOhMiXnfddfzxj39steJERESkmZwDoWvHAfUN88ff24Pi8ir2ZxcZWJzxmhWAAMLDwxkyZAhmc+0phg8fTr9+/VqlMBEREWmB6OFg8YKz2XD6MAAWs4khWhcMaEEAEhERERfm6QvRIxzfn98NFlO7Lpg7UwASERHprOLOux2+WlKcIwBtVwASERGRTim+/jigwdFBmE1wIr+ErAL3XRhVAUhERKSzikwED18ozoWcfQB08fagf4Rjyhp3XhZDAUhERKSz8vCGmOpxQGn11wVz54HQCkAiIiKdWfzVjq/nD4SOc8wIrQAkIiIinVNcdQBKWwc2x+SHNRMi7s0q5FxZpVGVGUoBSEREpDPrORi8ukJpPpz8DoDIIF8iAn2ostnZdTzfyOoMowAkIiLSmVk8HYujguNusGrOdcHcdCC0ApCIiEhnVzMOqIGFUd11QkQFIBERkc6uZj6g9PVQ5RjzkxTrGAi93ZqHzWY3qjLDKACJiIh0duFXgk8glBVC1i4A+kf44+tpoai0kkM5Zw0usP0pAImIiHR2ZgvEjnZ8n+a4Hd7DYmZwdBAAW9PPGFSYcRSARERE3EHcectiVKtZF8wd5wNSABIREXEHNQOhrRuhqgI4704wBSARERHplHokgG8IVJyDE9sBGBITjMkE6aeLOVVUZnCB7UsBSERExB2YzRB3leP76mUxAn096dvDH3C/ViAFIBEREXfhnA/o/HXBarrB3GsgtAKQiIiIu6gZCJ2xGSodXV6JMe45IaICkIiIiLvofjl06QGVpXB8C1B7J9juEwWUVlQZWV27UgASERFxFyZT7azQ1bfDx4T4EdrVm4oqO9+dKDCwuPalACQiIuJOnPMBOcYBmUwmEmODANjqRgujGh6AFixYQHx8PD4+PiQmJrJ27dqLHjt16lRMJlO9bcCAAXWO++CDD0hISMDb25uEhAQ+/PDDtr4MERGRjqFmIPTxLVBeDNSuC+ZOd4IZGoCWL1/O9OnTmTVrFjt27CAlJYXx48djtVobPP7VV18lKyvLuWVkZBASEsJPfvIT5zEbNmxg8uTJTJkyhV27djFlyhQmTZrEpk2b2uuyREREXFdILwiIBFsFZDg+G2vuBNtuzcNud4+FUU12A690xIgRDB06lIULFzr39e/fn4kTJzJnzpzvff5HH33ED3/4Q44dO0ZsbCwAkydPprCwkJUrVzqPu/HGGwkODmbZsmWNqquwsJDAwEAKCgoICAho4lWJiIi4uH/9HL59F1J+Bdc9Q1llFVf87jPKK2188atr6N29q9EVNktTPr8NawEqLy9n27ZtjB07ts7+sWPHsn79+kadY9GiRVx//fXO8AOOFqALzzlu3LhLnrOsrIzCwsI6m4iISKd1wUBobw8Lg6ICAffpBjMsAOXm5lJVVUVYWFid/WFhYWRnZ3/v87Oysli5ciX3339/nf3Z2dlNPuecOXMIDAx0btHR0U24EhERkQ6mZiD0iW1QVgRAYs04IDcZCG34IGiTyVTnZ7vdXm9fQ5YsWUJQUBATJ05s8TlnzpxJQUGBc8vIyGhc8SIiIh1RcCwExYC9yrE4KrULo251kxmhDQtAoaGhWCyWei0zOTk59VpwLmS321m8eDFTpkzBy8urzmPh4eFNPqe3tzcBAQF1NhERkU6t5m6w6tvhawLQkVPnyDtXblRV7cawAOTl5UViYiKpqal19qemppKcnHzJ53799dccPnyYadOm1Xts1KhR9c752Weffe85RURE3EpczbpgjnFAIV286NW9C+C4G6yz8zDyxWfMmMGUKVNISkpi1KhRvPnmm1itVh566CHA0TV14sQJ3nnnnTrPW7RoESNGjGDgwIH1zvnoo49y9dVX8+KLL3Lbbbfx8ccf8/nnn7Nu3bp2uSYREZEOoWYgdNYuKMkH3yASY4I5euocW9PzuK7/pXtjOjpDxwBNnjyZefPmMXv2bAYPHsyaNWtYsWKF866urKysenMCFRQU8MEHHzTY+gOQnJzMu+++y1//+leuvPJKlixZwvLlyxkxYkSbX4+IiEiHEdATQnqD3Qbpjjulk5wrw3f+FiBD5wFyVZoHSERE3MK/H4VtS2Dk/8KNczicc5brX/kabw8z3/1uHF4eht8r1SQdYh4gERERMZhzILRjHFDv7l0I8vOkrNLGnszOvTCqApCIiIi7qpkP6OR3UHzGsTBqjHt0gykAiYiIuKuuPaB7P8f31XeDJbrJOCAFIBEREXcWV3dZjJqV4bemd+6FURWARERE3Fl83fmArowKxNNi4lRRGcfzSgwsrG0pAImIiLizuKsAE5zaD2dz8PG0MKCnY2HUzrwshgKQiIiIO/MLgbDqiYXTarrBqtcF68QLoyoAiYiIuLuaWaEvWBesMw+EVgASERFxdxcMhK65E+zAySIKSyuMqqpNKQCJiIi4u9hkMJnhzBEozKSHvw8xIX7Y7bDDmm90dW1CAUhERMTd+QZBxCDH9zWtQDXdYGmdcyC0ApCIiIic1w12wTgga+ccB6QAJCIiIufNB+QIQDUrw++w5lNZZTOqqjajACQiIiIQMxJMFsi3Ql46l/Xwx9/bg+LyKvZnFxldXatTABIRERHw9ofIRMf3aWuxmE0M6cS3wysAiYiIiEP8heuCVU+IqAAkIiIindb5A6Ht9k59J5gCkIiIiDhEjwCzJxRlwpmjDI4OwmI2kVlQSmZ+51oYVQFIREREHLz8IHq44/tja+ji7UH/CH+g840DUgASERGRWjXdYNULoybGdM6B0ApAIiIiUuv8gdB2O4lxIYACkIiIiHRmUcPAwwfO5cCpA847wfZmFXKurNLg4lqPApCIiIjU8vCuHQeUtpaeQb5EBPpQZbOzKyPf0NJakwKQiIiI1FWzLMaF64J1om4wBSARERGpK65mXbB1YLN1ygkRFYBERESkrsih4NkFSs5Azh4SYx0Dobdb87DZ7AYX1zoUgERERKQui6djcVSAY2vpH+GPn5eFotJKDuWcNba2VqIAJCIiIvXF184H5GExMzg6CICt6Z1jWQwFIBEREamvZiB02jdgqzpvXbDOMQ5IAUhERETqCx8E3gFQVgDZ39YGIKsCkIiIiHRWFg+ITXZ8f2wNQ2KCMZkg/XQxp4rKjK2tFSgAiYiISMPiapfFCPT1pG+PmoVRO/44IAUgERERaVjNOCDrBqiqIDGu80yIqAAkIiIiDQsbCL7BUH4WMnd2qgkRFYBERESkYWYzxI52fJ+2xjkQeveJAkorqgwsrOUMD0ALFiwgPj4eHx8fEhMTWbt27SWPLysrY9asWcTGxuLt7U3v3r1ZvHix8/ElS5ZgMpnqbaWlpW19KSIiIp3PeeuCxYT4EdrVm4oqO9+dKDC2rhbyMPLFly9fzvTp01mwYAGjR4/mjTfeYPz48ezdu5eYmJgGnzNp0iROnjzJokWL6NOnDzk5OVRWVtY5JiAggAMHDtTZ5+Pj02bXISIi0mnVDIS2bsJUVU5SbDCr9mSzNS2PYXEhxtbWAoYGoFdeeYVp06Zx//33AzBv3jxWr17NwoULmTNnTr3jV61axddff83Ro0cJCXH80uPi4uodZzKZCA8Pb9PaRURE3EKP/uAXCsW5cGIbSXHhrNqTXX0nWG+jq2s2w7rAysvL2bZtG2PHjq2zf+zYsaxfv77B53zyySckJSXx0ksvERkZSd++fXn88ccpKSmpc9zZs2eJjY0lKiqKm2++mR07dlyylrKyMgoLC+tsIiIiAphMtctiHFvL0NjaO8Hs9o67MKphASg3N5eqqirCwsLq7A8LCyM7O7vB5xw9epR169axe/duPvzwQ+bNm8f777/PL37xC+cx/fr1Y8mSJXzyyScsW7YMHx8fRo8ezaFDhy5ay5w5cwgMDHRu0dHRrXORIiIinUFc7bpgA3sG4u1hJq+4gqO554ytqwUMHwRtMpnq/Gy32+vtq2Gz2TCZTCxdupThw4czYcIEXnnlFZYsWeJsBRo5ciR33303gwYNIiUlhffee4++ffvy2muvXbSGmTNnUlBQ4NwyMjJa7wJFREQ6upqB0Bmb8LKXMSgqCOjY64IZFoBCQ0OxWCz1WntycnLqtQrViIiIIDIyksDAQOe+/v37Y7fbOX78eIPPMZvNDBs27JItQN7e3gQEBNTZREREpFq3PtA1HKrKIWNznW6wjsqwAOTl5UViYiKpqal19qemppKcnNzgc0aPHk1mZiZnz5517jt48CBms5moqKgGn2O329m5cycRERGtV7yIiIg7MZnOWx1+7XkTInbcJTEM7QKbMWMGb731FosXL2bfvn089thjWK1WHnroIcDRNXXPPfc4j7/zzjvp1q0b9957L3v37mXNmjU88cQT3Hffffj6+gLw7LPPsnr1ao4ePcrOnTuZNm0aO3fudJ5TREREmqGBgdBHTp0j71y5gUU1n6G3wU+ePJnTp08ze/ZssrKyGDhwICtWrCA2NhaArKwsrFar8/iuXbuSmprKL3/5S5KSkujWrRuTJk3iueeecx6Tn5/Pgw8+SHZ2NoGBgQwZMoQ1a9YwfPjwdr8+ERGRTqNmIPSJrYR4lNOrexeOnjrHdmse1/VveOiKKzPZO/I9bG2ksLCQwMBACgoKNB5IREQEwG6HeVdAQQbc/QG/3tmd97Ye539+0Jvf3NjP6OqApn1+G34XmIiIiHQAJlNtK9CxtSTFOiYk7qh3gikAiYiISOOcNxC6ZhzQruP5lFfaDCyqeRSAREREpHFqBkJn7qC3fyXBfp6UVdrYk9nxFkZVABIREZHGCYyC4Hiw2zBZN5LYgecDUgASERGRxouvXRajphtsawccB6QAJCIiIo0Xf43j67E1tQOhrR1vYVQFIBEREWm8uKscX7O/48puNjwtJk4VlZFxpsTYuppIAUhEREQazz8cQvsCdnxObGRAT8f6nB1tWQwFIBEREWka53xAa5zrgnW0gdAKQCIiItI05w2ETopTABIRERF3UNMClLOXxO5VABw4WURBSYWBRTWNApCIiIg0TZdQ6DEAgO65W4gJ8cNuh50Z+cbW1QQKQCIiItJ08Q2MA0rrOAOhFYBERESk6c5bGNU5IWIHGgekACQiIiJNFzcaMMHpQ4zsXgY4usAqqzrGwqgKQCIiItJ0vsEQcSUAvc7uxN/bg+LyKvZnFxlcWOMoAImIiEjzVHeDmdPXMsS5LljHGAekACQiIiLNE3+14+v5A6Gt+cbV0wQKQCIiItI8MaPAZIG8NJJDHWuBdZQ7wRSAREREpHl8AqDnEACuKN+FxWwis6CUzHzXXxhVAUhERESar3o+IO/j6+kf4Q90jGUxFIBERESk+eLOWxcspuOsC6YAJCIiIs0XMxLMnlCQwVWhZwHYmu7644AUgERERKT5vLpAZCIASfbdAOzLKuJcWaWRVX0vBSARERFpmerb4YNObqRnoA9VNju7XHxhVAUgERERaZn489YFiwkCXH9dMAUgERERaZmo4WDxhrPZXNe9AHD9gdAKQCIiItIynj4QPRyAEaa9AGy35mGz2Y2s6pIUgERERKTlqm+HDz+zBT8vC0WllRzMcd2FURWAREREpOWqB0Kb09cxOCoQcO1uMAUgERERabnIRPD0g+JcxnV3BJ9taQpAIiIi0pl5eEH0CABGezjGAbnynWAKQCIiItI6qm+Hjy3chskE1jPF5BSVGlxUwxSAREREpHXEXwOAZ8Z6+vXoAsB2F20FUgASERGR1hExGLz8oTSfm3qcBmCri44DUgASERGR1mHxgNhRAFzttQ+AbVYFoAYtWLCA+Ph4fHx8SExMZO3atZc8vqysjFmzZhEbG4u3tze9e/dm8eLFdY754IMPSEhIwNvbm4SEBD788MO2vAQRERGpUT0f0GXndgCw+0QBpRVVRlbUIEMD0PLly5k+fTqzZs1ix44dpKSkMH78eKxW60WfM2nSJL744gsWLVrEgQMHWLZsGf369XM+vmHDBiZPnsyUKVPYtWsXU6ZMYdKkSWzatKk9LklERMS9VQ+E9sncRHhXDyqq7Hx7vMDgouoz2e12w+apHjFiBEOHDmXhwoXOff3792fixInMmTOn3vGrVq3ijjvu4OjRo4SEhDR4zsmTJ1NYWMjKlSud+2688UaCg4NZtmxZo+oqLCwkMDCQgoICAgICmnhVIiIibsxWBS/FQ2kBL0S+zp+PBPObG/vxPz/o3eYv3ZTPb8NagMrLy9m2bRtjx46ts3/s2LGsX7++wed88sknJCUl8dJLLxEZGUnfvn15/PHHKSkpcR6zYcOGeuccN27cRc8Jjm61wsLCOpuIiIg0g9kCsVcBMMbnAADb0s8YWVGDDAtAubm5VFVVERYWVmd/WFgY2dnZDT7n6NGjrFu3jt27d/Phhx8yb9483n//fX7xi184j8nOzm7SOQHmzJlDYGCgc4uOjm7BlYmIiLi56m6wfqU7AceSGAZ2ODXI8EHQJpOpzs92u73evho2mw2TycTSpUsZPnw4EyZM4JVXXmHJkiV1WoGack6AmTNnUlBQ4NwyMjJacEUiIiJurnogtH/OVrp62MgrruBo7jmDi6rLsAAUGhqKxWKp1zKTk5NTrwWnRkREBJGRkQQGBjr39e/fH7vdzvHjxwEIDw9v0jkBvL29CQgIqLOJiIhIM/VIAN8QTBXF3N7jJOB664IZFoC8vLxITEwkNTW1zv7U1FSSk5MbfM7o0aPJzMzk7Nmzzn0HDx7EbDYTFRUFwKhRo+qd87PPPrvoOUVERKSVmc3ObrAb/A4CsNXFxgEZ2gU2Y8YM3nrrLRYvXsy+fft47LHHsFqtPPTQQ4Cja+qee+5xHn/nnXfSrVs37r33Xvbu3cuaNWt44oknuO+++/D19QXg0Ucf5bPPPuPFF19k//79vPjii3z++edMnz7diEsUERFxT9XdYAPLdwGOcUCuxMPIF588eTKnT59m9uzZZGVlMXDgQFasWEFsbCwAWVlZdeYE6tq1K6mpqfzyl78kKSmJbt26MWnSJJ577jnnMcnJybz77rs89dRTPP300/Tu3Zvly5czYsSIdr8+ERERtxV/NQDBp3fgTTlHTkHeuXKCu3gZXJiDofMAuSrNAyQiItJCdjv84XI4e5IZvs/xr7xevHVPEtcnXHxMbkt1iHmAREREpBMzmZzdYOO7HgZgqwt1gykAiYiISNuoHgg9uOo7ALYrAImIiEinV90CFJr/LT6Uset4PuWVNoOLclAAEhERkbYR0gsCIjHZKviB7xHKKm3sznSNhVEVgERERKRtnDcO6JaAI4DrdIMpAImIiEjbqb4dPtG+G4CtLjIjtAKQiIiItJ3qgdBhRXvpQglbXWRhVAUgERERaTtBMRAUi8lexUiPg+SeLSPjTMn3P6+NKQCJiIhI24qvGQdUMx+Q8euCKQCJiIhI24pzjAMazh7ANSZEVAASERGRtlXdAhRRfJAAzrnEnWAKQCIiItK2AnpCtz6YsDHcvJ8DJ4soKKkwtCQFIBEREWl71fMBjfM7gN0OO6zGtgIpAImIiEjbq+4GG23ZBxg/IaICkIiIiLS96hagnmVHCKbQ8IHQCkAiIiLS9rr2gO79ARjX9TA9g3wNLcfD0FcXERER9xGfAqf2MWdIAaabBhlailqAREREpH1Ud4OZjq0xuBAFIBEREWkvcVcBJsg9AEUnDS1FAUhERETah18IhA90fJ+21tBSFIBERESk/VQvi6EAJCIiIu6jej4gjhkbgHQXmIiIiLSf2GS4YbZzQLRRFIBERESk/fgEwuhHja5CXWAiIiLifhSARERExO0oAImIiIjbUQASERERt6MAJCIiIm5HAUhERETcjgKQiIiIuB0FIBEREXE7CkAiIiLidhSARERExO0oAImIiIjbUQASERERt6MAJCIiIm5Hq8E3wG63A1BYWGhwJSIiItJYNZ/bNZ/jl6IA1ICioiIAoqOjDa5EREREmqqoqIjAwMBLHmOyNyYmuRmbzUZmZib+/v6YTCajy3FJhYWFREdHk5GRQUBAgNHluD29H65F74fr0XviWtrq/bDb7RQVFdGzZ0/M5kuP8lELUAPMZjNRUVFGl9EhBAQE6H8mLkTvh2vR++F69J64lrZ4P76v5aeGBkGLiIiI21EAEhEREbejACTN4u3tzW9/+1u8vb2NLkXQ++Fq9H64Hr0nrsUV3g8NghYRERG3oxYgERERcTsKQCIiIuJ2FIBERETE7SgAiYiIiNtRABIRERG3owAk7aK4uJjY2Fgef/xxo0txexkZGfzgBz8gISGBK6+8kn/+859Gl+R2/vOf/3D55Zdz2WWX8dZbbxldjlvT34Nrao/PDN0GL+1i1qxZHDp0iJiYGObOnWt0OW4tKyuLkydPMnjwYHJychg6dCgHDhygS5cuRpfmFiorK0lISOCrr74iICCAoUOHsmnTJkJCQowuzS3p78E1tcdnhlqApM0dOnSI/fv3M2HCBKNLESAiIoLBgwcD0KNHD0JCQjhz5oyxRbmRzZs3M2DAACIjI/H392fChAmsXr3a6LLclv4eXE97fWYoALm5NWvWcMstt9CzZ09MJhMfffRRvWMWLFhAfHw8Pj4+JCYmsnbt2ia9xuOPP86cOXNaqeLOrz3ekxpbt27FZrMRHR3dwqrdR0vfn8zMTCIjI50/R0VFceLEifYovVNqzb8X/T20XGu8H+31maEA5ObOnTvHoEGDmD9/foOPL1++nOnTpzNr1ix27NhBSkoK48ePx2q1Oo9JTExk4MCB9bbMzEw+/vhj+vbtS9++fdvrkjq8tn5Papw+fZp77rmHN998s82vqTNp6fvT0KgDk8nUpjV3Zq3x9wL6e2gtLX0/2vUzwy5SDbB/+OGHdfYNHz7c/tBDD9XZ169fP/uTTz7ZqHM++eST9qioKHtsbKy9W7du9oCAAPuzzz7bWiV3em3xntjtdntpaak9JSXF/s4777RGmW6rOe/PN998Y584caLzsUceecS+dOnSNq/VHTT370V/D22jOe9He35mqAVILqq8vJxt27YxduzYOvvHjh3L+vXrG3WOOXPmkJGRQVpaGnPnzuWBBx7gmWeeaYty3UJrvCd2u52pU6cyZswYpkyZ0hZluq3GvD/Dhw9n9+7dnDhxgqKiIlasWMG4ceOMKLfTa8z7ob+H9tOY96M9PzMUgOSicnNzqaqqIiwsrM7+sLAwsrOzDarKvbXGe/LNN9+wfPlyPvroIwYPHszgwYP57rvv2qJct9OY98fDw4M//OEPXHvttQwZMoQnnniCbt26GVFup9eY90N/D+3H1T5TPNr9FaXDuXB8gt1ub9aYhalTp7ZSRdKS9+Sqq67CZrO1RVlS7fven1tvvZVbb721vctyW5d6P/T30P4a+/+vtv7MUAuQXFRoaCgWi6VeMs/JyamX4KV96D1xbXp/XIveD9fiau+HApBclJeXF4mJiaSmptbZn5qaSnJyskFVuTe9J65N749r0fvhWlzt/VAXmJs7e/Yshw8fdv587Ngxdu7cSUhICDExMcyYMYMpU6aQlJTEqFGjePPNN7FarTz00EMGVt256T1xbXp/XIveD9fSod6PNrm3TDqMr776yg7U2372s585j3n99dftsbGxdi8vL/vQoUPtX3/9tXEFuwG9J65N749r0fvhWjrS+6G1wERERMTtaAyQiIiIuB0FIBEREXE7CkAiIiLidhSARERExO0oAImIiIjbUQASERERt6MAJCIiIm5HAUhERETcjgKQiHQaaWlpmEwmdu7c2ejnLFmyhKCgoDarSURckwKQiIiIuB0FIBEREXE7CkAi0qGsWrWKq666iqCgILp168bNN9/MkSNHGjz2v//9LyaTiU8//ZRBgwbh4+PDiBEj+O677+odu3r1avr370/Xrl258cYbycrKcj62ZcsWbrjhBkJDQwkMDOSaa65h+/btbXaNItL2FIBEpEM5d+4cM2bMYMuWLXzxxReYzWZuv/12bDbbRZ/zxBNPMHfuXLZs2UKPHj249dZbqaiocD5eXFzM3Llz+dvf/saaNWuwWq08/vjjzseLior42c9+xtq1a9m4cSOXXXYZEyZMoKioqE2vVUTajofRBYiINMWPfvSjOj8vWrSIHj16sHfvXrp27drgc377299yww03APD2228TFRXFhx9+yKRJkwCoqKjgz3/+M7179wbg4YcfZvbs2c7njxkzps753njjDYKDg/n666+5+eabW+3aRKT9qAVIRDqUI0eOcOedd9KrVy8CAgKIj48HwGq1XvQ5o0aNcn4fEhLC5Zdfzr59+5z7/Pz8nOEHICIigpycHOfPOTk5PPTQQ/Tt25fAwEACAwM5e/bsJV9TRFybWoBEpEO55ZZbiI6O5i9/+Qs9e/bEZrMxcOBAysvLm3Qek8nk/N7T07PeY3a73fnz1KlTOXXqFPPmzSM2NhZvb29GjRrV5NcUEdehACQiHcbp06fZt28fb7zxBikpKQCsW7fue5+3ceNGYmJiAMjLy+PgwYP069ev0a+7du1aFixYwIQJEwDIyMggNze3GVcgIq5CAUhEOozg4GC6devGm2++SUREBFarlSeffPJ7nzd79my6detGWFgYs2bNIjQ0lIkTJzb6dfv06cPf/vY3kpKSKCws5IknnsDX17cFVyIiRtMYIBHpMMxmM++++y7btm1j4MCBPPbYY7z88svf+7wXXniBRx99lMTERLKysvjkk0/w8vJq9OsuXryYvLw8hgwZwpQpU3jkkUfo0aNHSy5FRAxmsp/f0S0i0on897//5dprryUvL0/LXYhIHWoBEhEREbejACQiIiJuR11gIiIi4nbUAiQiIiJuRwFIRERE3I4CkIiIiLgdBSARERFxOwpAIiIi4nYUgERERMTtKACJiIiI21EAEhEREbejACQiIiJu5/8DIxeb8fl1jAsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(param_grid[\"ridge__alpha\"], train_scores.tolist(), label=\"train\")\n",
    "plt.semilogx(param_grid[\"ridge__alpha\"], cv_scores.tolist(), label=\"cv\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"alpha\")\n",
    "plt.ylabel(\"score\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ridge__alpha': 100.0}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_alpha = search.best_params_\n",
    "best_alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7950686603241023"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It seems alpha=100 is the best choice here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- General intuition: larger `alpha` leads to smaller coefficients.\n",
    "- Smaller coefficients mean the predictions are less sensitive to changes in the data. Hence less chance of overfitting.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>OverallQual</th>\n",
       "      <td>9689.632266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GrLivArea</th>\n",
       "      <td>7816.816972</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1stFlrSF</th>\n",
       "      <td>5930.575341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <td>5208.547961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageCars</th>\n",
       "      <td>5055.756475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SaleType_WD</th>\n",
       "      <td>-1214.377135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageFinish_Unf</th>\n",
       "      <td>-1280.655867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foundation_CBlock</th>\n",
       "      <td>-1767.116100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofStyle_Gable</th>\n",
       "      <td>-1984.434183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <td>-2623.381280</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>263 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Coefficients\n",
       "OverallQual         9689.632266\n",
       "GrLivArea           7816.816972\n",
       "1stFlrSF            5930.575341\n",
       "TotRmsAbvGrd        5208.547961\n",
       "GarageCars          5055.756475\n",
       "...                         ...\n",
       "SaleType_WD        -1214.377135\n",
       "GarageFinish_Unf   -1280.655867\n",
       "Foundation_CBlock  -1767.116100\n",
       "RoofStyle_Gable    -1984.434183\n",
       "KitchenAbvGr       -2623.381280\n",
       "\n",
       "[263 rows x 1 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_bigalpha = make_pipeline(preprocessor, Ridge(alpha=1000))\n",
    "pipe_bigalpha.fit(X_train, y_train)\n",
    "bigalpha_coeffs = pipe_bigalpha.named_steps[\"ridge\"].coef_\n",
    "pd.DataFrame(\n",
    "    data=bigalpha_coeffs, index=new_columns, columns=[\"Coefficients\"]\n",
    ").sort_values(by=\"Coefficients\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Smaller `alpha` leads to bigger coefficients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>RoofMatl_WdShngl</th>\n",
       "      <td>128419.408757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_Membran</th>\n",
       "      <td>127334.605534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_Metal</th>\n",
       "      <td>106033.814922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Condition2_PosA</th>\n",
       "      <td>83475.834626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_CompShg</th>\n",
       "      <td>70577.031608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Exterior1st_ImStucc</th>\n",
       "      <td>-34430.377551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Heating_OthW</th>\n",
       "      <td>-34801.030771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Condition2_RRAe</th>\n",
       "      <td>-63399.065969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Condition2_PosN</th>\n",
       "      <td>-194897.224244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_ClyTile</th>\n",
       "      <td>-582393.604620</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>263 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      Coefficients\n",
       "RoofMatl_WdShngl     128419.408757\n",
       "RoofMatl_Membran     127334.605534\n",
       "RoofMatl_Metal       106033.814922\n",
       "Condition2_PosA       83475.834626\n",
       "RoofMatl_CompShg      70577.031608\n",
       "...                            ...\n",
       "Exterior1st_ImStucc  -34430.377551\n",
       "Heating_OthW         -34801.030771\n",
       "Condition2_RRAe      -63399.065969\n",
       "Condition2_PosN     -194897.224244\n",
       "RoofMatl_ClyTile    -582393.604620\n",
       "\n",
       "[263 rows x 1 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_smallalpha = make_pipeline(preprocessor, Ridge(alpha=0.01))\n",
    "pipe_smallalpha.fit(X_train, y_train)\n",
    "smallalpha_coeffs = pipe_smallalpha.named_steps[\"ridge\"].coef_\n",
    "pd.DataFrame(\n",
    "    data=smallalpha_coeffs, index=new_columns, columns=[\"Coefficients\"]\n",
    ").sort_values(by=\"Coefficients\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the best alpha found by the grid search, the coefficients are somewhere in between. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>OverallQual</th>\n",
       "      <td>14484.902165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GrLivArea</th>\n",
       "      <td>11704.053037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_NridgHt</th>\n",
       "      <td>9662.969631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_NoRidge</th>\n",
       "      <td>9497.598615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtQual</th>\n",
       "      <td>8073.088562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_ClyTile</th>\n",
       "      <td>-3992.399179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LandContour_Bnk</th>\n",
       "      <td>-5001.996997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_Gilbert</th>\n",
       "      <td>-5197.585536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_CollgCr</th>\n",
       "      <td>-5467.463086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_Edwards</th>\n",
       "      <td>-5796.508529</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>263 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      Coefficients\n",
       "OverallQual           14484.902165\n",
       "GrLivArea             11704.053037\n",
       "Neighborhood_NridgHt   9662.969631\n",
       "Neighborhood_NoRidge   9497.598615\n",
       "BsmtQual               8073.088562\n",
       "...                            ...\n",
       "RoofMatl_ClyTile      -3992.399179\n",
       "LandContour_Bnk       -5001.996997\n",
       "Neighborhood_Gilbert  -5197.585536\n",
       "Neighborhood_CollgCr  -5467.463086\n",
       "Neighborhood_Edwards  -5796.508529\n",
       "\n",
       "[263 rows x 1 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_bestalpha = make_pipeline(\n",
    "    preprocessor, Ridge(alpha=search.best_params_[\"ridge__alpha\"])\n",
    ")\n",
    "pipe_bestalpha.fit(X_train, y_train)\n",
    "bestalpha_coeffs = pipe_bestalpha.named_steps[\"ridge\"].coef_\n",
    "pd.DataFrame(\n",
    "    data=bestalpha_coeffs, index=new_columns, columns=[\"Coefficients\"]\n",
    ").sort_values(by=\"Coefficients\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize: \n",
    "- Higher values of `alpha` means a more restricted model.  \n",
    "- The values of coefficients are likely to be smaller for higher values of `alpha` compared to lower values of alpha. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `RidgeCV`\n",
    "\n",
    "Because it's so common to want to tune `alpha` with `Ridge`, sklearn provides a class called `RidgeCV`, which automatically tunes `alpha` based on cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "alphas = 10.0 ** np.arange(-6, 6, 1)\n",
    "ridgecv_pipe = make_pipeline(preprocessor, RidgeCV(alphas=alphas, cv=10))\n",
    "ridgecv_pipe.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100.0"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_alpha = ridgecv_pipe.named_steps[\"ridgecv\"].alpha_\n",
    "\n",
    "best_alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([228728.1963872 , 104718.39905565, 155778.96723311, 246316.71119031,\n",
       "       127633.10676873, 243207.19441128, 304930.24461291, 145374.59435295,\n",
       "       157059.38983893, 128487.51979632])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_tuned = make_pipeline(preprocessor, Ridge(alpha=best_alpha))\n",
    "ridge_tuned.fit(X_train, y_train)\n",
    "ridge_preds = ridge_tuned.predict(X_test)\n",
    "ridge_preds[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    data={\"coefficients\": ridge_tuned.named_steps[\"ridge\"].coef_}, index=new_columns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>OverallQual</th>\n",
       "      <td>14484.902165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GrLivArea</th>\n",
       "      <td>11704.053037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_NridgHt</th>\n",
       "      <td>9662.969631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_NoRidge</th>\n",
       "      <td>9497.598615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtQual</th>\n",
       "      <td>8073.088562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RoofMatl_ClyTile</th>\n",
       "      <td>-3992.399179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LandContour_Bnk</th>\n",
       "      <td>-5001.996997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_Gilbert</th>\n",
       "      <td>-5197.585536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_CollgCr</th>\n",
       "      <td>-5467.463086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhood_Edwards</th>\n",
       "      <td>-5796.508529</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>263 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      coefficients\n",
       "OverallQual           14484.902165\n",
       "GrLivArea             11704.053037\n",
       "Neighborhood_NridgHt   9662.969631\n",
       "Neighborhood_NoRidge   9497.598615\n",
       "BsmtQual               8073.088562\n",
       "...                            ...\n",
       "RoofMatl_ClyTile      -3992.399179\n",
       "LandContour_Bnk       -5001.996997\n",
       "Neighborhood_Gilbert  -5197.585536\n",
       "Neighborhood_CollgCr  -5467.463086\n",
       "Neighborhood_Edwards  -5796.508529\n",
       "\n",
       "[263 rows x 1 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(\"coefficients\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So according to this model:\n",
    "\n",
    "- As `OverallQual` feature gets bigger the housing price will get bigger.\n",
    "- `Neighborhood_Edwards` is associated with reducing the housing price. \n",
    "    - We'll talk more about interpretation of different kinds of features next week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(390726.10647423257, 30791.092505420762)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_preds.max(), ridge_preds.min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### iClicker Exercise 2.2\n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/C0P55**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) Price per square foot would be a good feature to add in our `X`. \n",
    "- (B) The `alpha` hyperparameter of `Ridge` has similar interpretation of `C` hyperparameter of `LogisticRegression`; higher `alpha` means more complex model. \n",
    "- (C) In `Ridge`, smaller alpha means bigger coefficients whereas bigger alpha means smaller coefficients.  \n",
    "\n",
    "**V's answers:  C**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we use the metrics we looked at in Lecture 1 for classification problems? Why or why not? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression scoring functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "- We aren't doing classification anymore, so we can't just check for equality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "302     False\n",
       "767     False\n",
       "429     False\n",
       "1139    False\n",
       "558     False\n",
       "        ...  \n",
       "1041    False\n",
       "1122    False\n",
       "1346    False\n",
       "1406    False\n",
       "1389    False\n",
       "Name: SalePrice, Length: 1314, dtype: bool"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_tuned.predict(X_train) == y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([205000, 160000, 175000, ..., 262500, 133000, 131000])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([212894.62756285, 178502.78223444, 189937.18327372, ...,\n",
       "       245233.6751565 , 129863.13373552, 135439.89186716])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_tuned.predict(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need a score that reflects how right/wrong each prediction is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "A number of popular scoring functions for regression. We are going to look at some common metrics: \n",
    "\n",
    "- mean squared error (MSE)\n",
    "- $R^2$\n",
    "- root mean squared error (RMSE)\n",
    "- MAPE\n",
    "\n",
    "See [sklearn documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics) for more details. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mean squared error (MSE)\n",
    "\n",
    "- A common metric is mean squared error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "preds = ridge_tuned.predict(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "873230473.3636105"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean((y_train - preds) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Perfect predictions would have MSE=0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean((y_train - y_train) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This is also implemented in sklearn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "873230473.3636105"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "mean_squared_error(y_train, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- MSE looks huge and unreasonable. There is an error of ~\\$1 Billion!\n",
    "- Is this score good or bad?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Unlike classification, with regression **our target has units**. \n",
    "- The target is in dollars, the mean squared error is in $dollars^2$ \n",
    "- The score also depends on the scale of the targets. \n",
    "- If we were working in cents instead of dollars, our MSE would be $10,000 \\times (100^2$) higher!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8732304733636.104"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean((y_train * 100 - preds * 100) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Root mean squared error or RMSE\n",
    "\n",
    "- The MSE above is in $dollars^2$.\n",
    "- A more relatable metric would be the root mean squared error, or RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29550.473318774617"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(mean_squared_error(y_train, ridge_tuned.predict(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Error of \\$30,000 makes more sense.\n",
    "- Let's dig deeper. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAGwCAYAAABrUCsdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC8aklEQVR4nOzdeXzU1bn48c/3O8t3lsxM9oSwg4gLoIAV0CpuuBSk2629YmnxttjWBalarbZWa61brbZq1Wtrq9Uqt72t/vSqiK1LS0WhoAIiCMgespDMllm/y/n9MWQkbCYYSIDn/XrNS5nvycyZgWSePOc5z9GUUgohhBBCCPGp6T09ASGEEEKIQ4UEVkIIIYQQ3UQCKyGEEEKIbiKBlRBCCCFEN5HASgghhBCim0hgJYQQQgjRTSSwEkIIIYToJu6ensDhxnEc6uvrCYVCaJrW09MRQgghRCcopUgmk9TV1aHre85LSWB1gNXX19O/f/+enoYQQggh9sGmTZvo16/fHq9LYHWAhUIhoPAXEw6He3g2QgghhOiMRCJB//79i5/jeyKB1QHWvvwXDoclsBJCCCEOMp9UxiPF60IIIYQQ3UQCKyGEEEKIbiKBlRBCCCFEN5HASgghhBCim0hgJYQQQgjRTSSwEkIIIYToJhJYCSGEEEJ0EwmshBBCCCG6iQRWQgghhBDdRAIrIYQQQohuIoGVEEIIIUQ3kcBKCCGEEKKbSGAlhBBCCNFNJLASQgghxEHv+eef55JLLkEp1aPzkMBKCCGEEAetbDbL5ZdfztSpU/nNb37Dn/70px6djwRWQgghhDho6brOggULALj66qv5whe+0KPzcffoswshhBBCdJFSCsdxcLlceL1enn76adatW8c555zT01OTjJUQQgghDh7Nzc2cf/753HzzzcX7jjzyyF4RVIEEVkIIIYQ4SMybN49Ro0bxwgsvcM8999DY2NjTU9qFBFZCCCGE6NVyuRxXX30155xzDg0NDRx77LG89dZb1NTU9PTUdiE1VkIIIYTotVauXMmFF17Iu+++C8Bll13Gz3/+c/x+f89ObA8ksBJCCCFEr5ROpzn11FNpbm6moqKC3/3ud0ydOrWnp7VXshQohBBCiF4pEAhw++23c+aZZ7J06dJeH1QBaKqnW5QeZhKJBJFIhHg8Tjgc7unpCCGEEL3K66+/jsfj4eSTTwYKrRWUUuh6z+aCOvv5LRkrIYQQQvQ40zS54YYbOOOMM7jwwguJRqMAaJrW40FVV0iNlRBCCCF61Nq1a5k2bRoLFy4EYNKkSXg8nh6e1b45eEJAIYQQQhxSlFI88cQTHH/88SxcuJDS0lL+9Kc/8eijj1JSUtLT09snkrESQgghxAGXy+X4r//6L5566ikATjnlFJ588kkGDBjQwzP7dCRjJYQQQogDzuv1ksvlcLlc/PSnP+W111476IMqkF2BB5zsChRCCHG4sm2bTCZTXOZrbW3lww8/ZPz48T08s08muwKFEEII0Wts3LiR008/nW9+85u053TKy8sPiqCqK6TGSgghhBD71Z///GcuueQSYrEYoVCIdevWMWTIkJ6e1n4hGSshhBBC7BdtbW1885vf5IILLiAWizFu3DjeeeedQzaoAgmshBBCCLEfLF68mLFjx/K73/0OTdP44Q9/yD//+U+GDh3a01Pbr2QpUAghhBDdyrIsLrjgAj766CP69evHk08+ycSJE3t6WgdEj2asBg0ahKZpu9wuu+wyoNA47Oabb6aurg6/389pp53G+++/3+ExcrkcV1xxBZWVlQSDQaZOncrmzZs7jIlGo0yfPp1IJEIkEmH69OnEYrEOYzZu3Mj5559PMBiksrKSWbNmkc/nO4xZtmwZEydOxO/307dvX2655RZkU6UQQgjRkdvt5tFHH+UrX/kK77333mETVEEPB1aLFi1i69atxdsrr7wCwFe+8hUA7rrrLu655x4eeOABFi1aRG1tLZMmTSKZTBYfY/bs2TzzzDPMmTOH+fPn09bWxpQpU7Btuzhm2rRpvPvuu8ydO5e5c+fy7rvvMn369OJ127aZPHkyqVSK+fPnM2fOHP7yl79w9dVXF8ckEgkmTZpEXV0dixYt4v777+fuu+/mnnvu2d9vkxBCCNHrPf/88zz99NPFP5922mn86U9/ory8vAdn1QNUL3LllVeqoUOHKsdxlOM4qra2Vt1xxx3F69lsVkUiEfXwww8rpZSKxWLK4/GoOXPmFMds2bJF6bqu5s6dq5RSasWKFQpQb731VnHMggULFKBWrlyplFLqxRdfVLquqy1bthTHPP3008owDBWPx5VSSj344IMqEomobDZbHHP77beruro65ThOp19jPB5XQPFxhRBCiINZOp1Wl156qQJUMBhUa9as6ekp7Red/fzuNcXr+XyeJ598kv/6r/9C0zTWrVtHQ0MDZ599dnGMYRhMnDiRN998EygUxpmm2WFMXV0dI0aMKI5ZsGABkUiEcePGFceMHz+eSCTSYcyIESOoq6srjjnnnHPI5XIsXry4OGbixIkYhtFhTH19PevXr9/j68rlciQSiQ43IYQQ4lCwdOlSTjjhBB588EEAvvOd79CvX78enlXP6jWB1bPPPkssFmPGjBkANDQ0AFBTU9NhXE1NTfFaQ0MDXq+XsrKyvY6prq7e5fmqq6s7jNn5ecrKyvB6vXsd0/7n9jG7c/vttxdruyKRCP3799/zmyCEEEIcBJRS3HfffZx44omsWLGCmpoaXn75Ze6+++4OCYjDUa8JrB599FHOO++8DlkjAE3TOvxZKbXLfTvbeczuxnfHGLW9cH1v87n++uuJx+PF26ZNm/Y6dyGEEKI3cxyHz3/+81x55ZXkcjkmT57M0qVLO6weHc56RWC1YcMG/va3v/Gtb32reF9tbS2wazaoqampmCmqra0ln88TjUb3OqaxsXGX52xubu4wZufniUajmKa51zFNTU3Arlm1HRmGQTgc7nATQgghDla6rnPsscdiGAb3338/zz///G5Xhg5XvSKw+v3vf091dTWTJ08u3jd48GBqa2uLOwWhUIf1xhtvcNJJJwEwduxYPB5PhzFbt25l+fLlxTETJkwgHo+zcOHC4pi3336beDzeYczy5cvZunVrccy8efMwDIOxY8cWx/zjH//o0IJh3rx51NXVMWjQoG58N4QQQojeJZfLdfiMvOWWW3jnnXe4/PLLP3EV6bCz38voP4Ft22rAgAHquuuu2+XaHXfcoSKRiPrrX/+qli1bpi688ELVp08flUgkimO+853vqH79+qm//e1vasmSJeqMM85Qxx13nLIsqzjm3HPPVaNGjVILFixQCxYsUCNHjlRTpkwpXrcsS40YMUKdeeaZasmSJepvf/ub6tevn7r88suLY2KxmKqpqVEXXnihWrZsmfrrX/+qwuGwuvvuu7v0emVXoBBCiIPJBx98oI4//nh1wgknqFwu19PT6TGd/fzu8cDq5ZdfVoBatWrVLtccx1E33XSTqq2tVYZhqFNPPVUtW7asw5hMJqMuv/xyVV5ervx+v5oyZYrauHFjhzEtLS3qoosuUqFQSIVCIXXRRRepaDTaYcyGDRvU5MmTld/vV+Xl5eryyy/v0FpBKaWWLl2qTjnlFGUYhqqtrVU333xzl1otKCWBlRBCiIOD4zjqkUceUX6/XwGqoqJil8/gw0lnP781paR1+IGUSCSIRCLE43GptxJCCNErtba2MnPmTP76178CcNZZZ/H444/vssHscNLZz+9eUWMlhBBCiN7h9ddfZ9SoUfz1r3/F4/Hw85//nJdffvmwDqq6Qg5hFkIIIQRQaCP0gx/8gC1btnDkkUfy1FNPFTdxic6RjJUQQgghgEJfxieffJJLL72UxYsXS1C1D6TG6gCTGishhBC9hVKKJ598kg0bNvCjH/2op6fTq3X281uWAoUQQojDUDwe59JLL+Wpp55C0zTOPvtsTjzxxJ6e1kFPAishhBDiMPPmm29y0UUXsX79elwuFz/5yU9k2a+bSGAlhBBCHCYsy+K2227jlltuwbZtBg8ezFNPPcX48eN7emqHDAmshBBCiMOAUoqpU6fy0ksvAXDRRRfx4IMPSr1vN5NdgUIIIcRhQNM0vvKVrxAKhXjiiSd48sknJajaD2RX4AEmuwKFEEIcKG1tbaxfv54RI0YAhaxVY2MjtbW1PTyzg490XhdCCCEOY4sXL2bMmDGce+65tLa2AoWslQRV+5cEVkIIIcQhxHEc7rrrLiZMmMDq1avRNI2NGzf29LQOG1K8LoQQQhwi6uvr+frXv87f//53AL785S/zyCOPUF5e3sMzO3xIxkoIIYQ4BPy///f/GDVqFH//+98JBAL89re/5c9//rMEVQeYZKyEEEKIQ8BTTz1FS0sLo0eP5umnn2b48OE9PaXDkgRWQgghxEFKKYWmaQA8/PDDjBgxgmuvvRbDMHp4ZocvWQoUQgghDjJKKe677z6+9rWv0d41qaysjBtvvFGCqh4mGSshhBDiINLU1MTFF1/Miy++CBQ6qH/uc5/r4VmJdpKxEkIIIQ4SL7/8MqNGjeLFF1/EMAweeOABzjvvvJ6eltiBZKyEEEKIXi6Xy3HDDTdwzz33ADBixAiefvrpYkd10XtIYCWEEEL0chdccAHPPfccAJdffjl33XUXfr+/h2cldkeWAoUQQohe7pprrqGmpobnn3+e+++/X4KqXkwyVkIIIUQv09raypIlSzjrrLMAOOWUU/joo48IBAI9PDPxSSRjJYQQQvQir732GqNGjeLzn/88q1atKt4vQdXBQQIrIYQQohcwTZMbbriBM888ky1bttCvXz9yuVxPT0t0kSwFCiGEED1szZo1XHTRRSxcuBCAb33rW/zyl78kGAz28MxEV0nGSgghhOhBTzzxBKNHj2bhwoWUlpby5z//md/85jcSVB2kJGMlhBBC9KBVq1bR1tbGqaeeypNPPkn//v17ekriU5DASgghhDjALMvC7S58BN90000MGjSIiy++GJfL1cMzE5+WLAUKIYQQB4hlWdxyyy189rOfJZ/PA+DxePjWt74lQdUhQjJWQgghxAGwYcMGvva1rzF//nwA/vd//5dp06b18KxEd5OMlRBCCLGf/elPf+K4445j/vz5hEIhnnjiCQmqDlGSsRJCCCH2k7a2NmbNmsXvf/97AMaNG8dTTz3FkCFDenhmYn+RjJUQQgixn1xyySX8/ve/R9M0fvSjH/HPf/5TgqpDXI8HVlu2bOFrX/saFRUVBAIBjj/+eBYvXly8rpTi5ptvpq6uDr/fz2mnncb777/f4TFyuRxXXHEFlZWVBINBpk6dyubNmzuMiUajTJ8+nUgkQiQSYfr06cRisQ5jNm7cyPnnn08wGKSyspJZs2YViwvbLVu2jIkTJ+L3++nbty+33HILSqnufVOEEEIcEm699VaOPfZYXn/9dX7605/i8Xh6ekpiP+vRwCoajXLyySfj8Xh46aWXWLFiBb/4xS8oLS0tjrnrrru45557eOCBB1i0aBG1tbVMmjSJZDJZHDN79myeeeYZ5syZw/z582lra2PKlCnYtl0cM23aNN59913mzp3L3Llzeffdd5k+fXrxum3bTJ48mVQqxfz585kzZw5/+ctfuPrqq4tjEokEkyZNoq6ujkWLFnH//fdz9913c8899+zfN0oIIcRBYcuWLfzud78r/nnIkCEsXbqUU089tQdnJQ4o1YOuu+469dnPfnaP1x3HUbW1teqOO+4o3pfNZlUkElEPP/ywUkqpWCymPB6PmjNnTnHMli1blK7rau7cuUoppVasWKEA9dZbbxXHLFiwQAFq5cqVSimlXnzxRaXrutqyZUtxzNNPP60Mw1DxeFwppdSDDz6oIpGIymazxTG33367qqurU47jdOo1x+NxBRQfUwghxKHh2WefVRUVFQpQr7zySk9PR3Szzn5+92jG6rnnnuOEE07gK1/5CtXV1YwePZrf/OY3xevr1q2joaGBs88+u3ifYRhMnDiRN998E4DFixdjmmaHMXV1dYwYMaI4ZsGCBUQiEcaNG1ccM378eCKRSIcxI0aMoK6urjjmnHPOIZfLFZcmFyxYwMSJEzEMo8OY+vp61q9fv9vXmMvlSCQSHW5CCCEOHel0mksvvZQvfOELtLS0MGbMGOmefhjr0cDqo48+4qGHHmLYsGG8/PLLfOc732HWrFn84Q9/AKChoQGAmpqaDl9XU1NTvNbQ0IDX66WsrGyvY6qrq3d5/urq6g5jdn6esrIyvF7vXse0/7l9zM5uv/32Yl1XJBKRbzYhhDiELF26lM985jM89NBDAHz/+99nwYIFDB8+vIdnJnpKjwZWjuMwZswYbrvtNkaPHs23v/1tZs6cWfwH2k7TtA5/Vkrtct/Odh6zu/HdMUZtL1zf03yuv/564vF48bZp06a9zlsIIcTB4ZFHHuHEE09kxYoV1NbWMm/ePO666y68Xm9PT030oB4NrPr06cMxxxzT4b6jjz6ajRs3AlBbWwvsmg1qamoqZopqa2vJ5/NEo9G9jmlsbNzl+ZubmzuM2fl5otEopmnudUxTUxOwa1atnWEYhMPhDjchhBAHP7/fTy6XY8qUKSxdupRJkyb19JREL9CjgdXJJ5/MqlWrOtz34YcfMnDgQAAGDx5MbW0tr7zySvF6Pp/njTfe4KSTTgJg7NixeDyeDmO2bt3K8uXLi2MmTJhAPB5n4cKFxTFvv/028Xi8w5jly5ezdevW4ph58+ZhGAZjx44tjvnHP/7RoQXDvHnzqKurY9CgQd3xlgghhOjF4vF48f+/9rWvMXfuXJ577jmqqqp6cFaiV9n/dfR7tnDhQuV2u9XPfvYztXr1avXHP/5RBQIB9eSTTxbH3HHHHSoSiai//vWvatmyZerCCy9Uffr0UYlEojjmO9/5jurXr5/629/+ppYsWaLOOOMMddxxxynLsopjzj33XDVq1Ci1YMECtWDBAjVy5Eg1ZcqU4nXLstSIESPUmWeeqZYsWaL+9re/qX79+qnLL7+8OCYWi6mamhp14YUXqmXLlqm//vWvKhwOq7vvvrvTr1l2BQohxMEnm82q733ve6pv376qubm5p6cjekBnP797NLBSSqnnn39ejRgxQhmGoY466ij1yCOPdLjuOI666aabVG1trTIMQ5166qlq2bJlHcZkMhl1+eWXq/LycuX3+9WUKVPUxo0bO4xpaWlRF110kQqFQioUCqmLLrpIRaPRDmM2bNigJk+erPx+vyovL1eXX355h9YKSim1dOlSdcoppyjDMFRtba26+eabO91qQSkJrIQQ4mCzYsUKddxxxylAAerRRx/t6SmJHtDZz29NKWkbfiAlEgkikQjxeFzqrYQQohdTSvGb3/yG2bNnk8lkqKys5Pe//z1Tpkzp6amJHtDZz285hFkIIYTYSUtLCzNnzuSZZ54BYNKkSTz++OP06dOnh2cmersePytQCCGE6G1uuukmnnnmGTweD3fffTdz586VoEp0imSshBBCiJ3ceuutrF69mttvv50xY8b09HTEQUQyVkIIIQ57a9as4Uc/+lGx6XNpaSkvv/yyBFWiyyRjJYQQ4rCllOIPf/gDl19+OW1tbQwcOJCZM2f29LTEQUwCKyGEEIelWCzGd7/7XebMmQPAqaeeyrnnntvDsxIHO1kKFIctpRTRVJ6GeJZoKo90HhHi8PHmm29y/PHHM2fOHFwuF7feeiuvvvoq/fv37+mpiYOcZKzEYakpkWX5lgRbYmnytoPXpdO3NMCIvmGqw76enp4QYj/69a9/zaxZs3AchyFDhvDHP/6R8ePH9/S0xCFCMlbisNOUyPL6qmbWNicJ+zz0Kw0Q9nlY25zk9VXNNCWyPT1FIcR+dPzxxwMwffp03nnnHQmqRLeSjJU4rCilWL4lQTyTZ1BFEE3TAAgabgZ5g6xvSbF8S4LTQ0bxmhDi4Ld+/XoGDRoEwMknn8x7773HiBEjenZS4pAkGStxWImlTbbE0lSHfLsETpqmUR3ysSWWJpY2e2iGQojulEwmufjiiznmmGP44IMPivdLUCX2FwmsxGElZznkbQefx7Xb6z6Pi7ztkLOcAzwzIUR3W7RoEWPGjOGxxx4jl8vxz3/+s6enJA4DEliJw4rh1vG6dLKmvdvrWdPG69Ix3PKtIcTBynEc7rrrLk466STWrFlD//79ee2117jkkkt6emriMCCfHuKwUhrw0Lc0QFMyu0t7BaUUTcksfUsDlAY8PTRDIcSnsWXLFiZNmsR1112HZVn8x3/8B++99x6nnnpqT09NHCYksBKHFU3TGNE3TMTvZX1LilTOwnYUqZzF+pYUkYCXEX3DUrguxEHq8ccf59VXXyUQCPDoo4/ypz/9ibKysp6eljiMyK5AcdipDvs4bXhVsY/VtlQOr0tnaFVI+lgJcZC79tpr2bhxI9/73vcYPnx4T09HHIY0Je2mD6hEIkEkEiEejxMOh3t6Ooc1pRSxtEnOcjDcOqUBj2SqhDjILF26lNtvv53HHnsMwzB6ejriENbZz2/JWInDlqZplAW9PT0NIcQ+UEpx3333ce2115LP5xk2bBi33HJLT09LCAmshBBCHFwaGxu5+OKLeemllwCYMmUKV1xxRQ/PSogCKV4XQghx0Jg7dy6jRo3ipZdewufz8cADD/Dcc89RVVXV01MTApCMlRBCiIPEr3/9ay6//HKg0Dn96aeflg7qoteRjJUQQoiDwuc+9znC4TBXXHEFCxculKBK9EqSsRJCCNErKaV4++23GT9+PACDBw9m1apV1NbW9vDMhNgzyVgJIYTodVpaWvjSl77EhAkTeOWVV4r3S1AlejvJWAkhhOhVXn31VaZPn059fT0ej4f169f39JSE6DTJWAkhhOgVTNPk+uuv56yzzqK+vp7hw4fz9ttvM3PmzJ6emhCdJhkrIYQQPW7NmjVMmzaNRYsWATBz5kzuvfdegsFgD89MiK6RwEoIIUSPW7hwIYsWLaKsrIzf/OY3fPnLX+7pKQmxTySwEkII0SOUUsXzOadNm8bmzZu58MIL6d+/fw/PTIh9JzVWQgghDrh//etfnHTSSTQ3Nxfvu/baayWoEgc9CayEEEIcMJZl8ZOf/IRTTz2Vt956ixtvvLGnpyREt5KlQCGEEAfEhg0buOiii/jXv/4FwNe//nXuuuuuHp6VEN1LMlZCCCH2uzlz5nDcccfxr3/9i3A4zB//+Ecef/xxwuFwT09NiG4lGSshhBD71W9/+9tiL6rx48fz1FNPMXjw4B6elTjUKKWIpU1yloPh1ikNeIqbIw6kHs1Y3XzzzWia1uG243EFSiluvvlm6urq8Pv9nHbaabz//vsdHiOXy3HFFVdQWVlJMBhk6tSpbN68ucOYaDTK9OnTiUQiRCIRpk+fTiwW6zBm48aNnH/++QSDQSorK5k1axb5fL7DmGXLljFx4kT8fj99+/bllltuQSnVvW+KEEIcYr7yla9wxBFHcOONN/LPf/5TgirR7ZoSWV5b2cz/La3nhWX1/N/Sel5b2UxTInvA59LjS4HHHnssW7duLd6WLVtWvHbXXXdxzz338MADD7Bo0SJqa2uZNGkSyWSyOGb27Nk888wzzJkzh/nz59PW1saUKVOwbbs4Ztq0abz77rvMnTuXuXPn8u677zJ9+vTiddu2mTx5MqlUivnz5zNnzhz+8pe/cPXVVxfHJBIJJk2aRF1dHYsWLeL+++/n7rvv5p577tnP75DorZRSRFN5GuJZoqm8BNlCbOc4Dn/+85+L3xORSISlS5dyyy234HbLQonoXk2JLK+vamZtc5Kwz0O/0gBhn4e1zUleX9UDwZXqQTfddJM67rjjdnvNcRxVW1ur7rjjjuJ92WxWRSIR9fDDDyullIrFYsrj8ag5c+YUx2zZskXpuq7mzp2rlFJqxYoVClBvvfVWccyCBQsUoFauXKmUUurFF19Uuq6rLVu2FMc8/fTTyjAMFY/HlVJKPfjggyoSiahsNlscc/vtt6u6ujrlOE6nX3M8HldA8XHFwakxnlF/X9Go/vDmOvXbf65Vf3hznfr7ikbVGM/09NSE6FGbN29WZ5xxhgLUgw8+2NPTEYc4x3HU31c0qkfeWKPmLd+qXnm/oXibt3yreuSNNervKxq79Dm9J539/O7xjNXq1aupq6tj8ODB/Od//icfffQRAOvWraOhoYGzzz67ONYwDCZOnMibb74JwOLFizFNs8OYuro6RowYURyzYMECIpEI48aNK44ZP348kUikw5gRI0ZQV1dXHHPOOeeQy+VYvHhxcczEiRMxDKPDmPr6+r0eEJrL5UgkEh1u4uDW6347EqKXePbZZxk1ahSvvvoqgUCAQCDQ01MSh7hY2mRLLE11yLdLPZWmaVSHfGyJpYmlzQM2px4NrMaNG8cf/vAHXn75ZX7zm9/Q0NDASSedREtLCw0NDQDU1NR0+JqampritYaGBrxeL2VlZXsdU11dvctzV1dXdxiz8/OUlZXh9Xr3Oqb9z+1jduf2228v1nZFIhFpfneQU0qxfEuCeCbPoIogQcONS9cIGm4GVQSJZ/Is35KQZUFxWEmn03znO9/hi1/8Iq2trYwZM4YlS5bwjW98o6enJg5xOcshbzv4PK7dXvd5XORth5zlHLA59Whgdd555/HlL3+ZkSNHctZZZ/HCCy8A8PjjjxfH7ByBqh2OQNiTncfsbnx3jGn/8NzbfK6//nri8XjxtmnTpr3OXfRuvfG3IyF60tKlSznhhBP47//+b6DQPX3BggUMHz68h2cmDgeGW8fr0sma9m6vZ00br0vHcB+4cKfHlwJ3FAwGGTlyJKtXry7uDtw5G9TU1FTMFNXW1pLP54lGo3sd09jYuMtzNTc3dxiz8/NEo1FM09zrmKamJmDXrNqODMMgHA53uImDV2/87UiInpRKpfjwww/p06cPr7zyCnfeeSder7enpyUOE6UBD31LAzQls7usFCilaEpm6VsaoDTgOWBz6lWBVS6X44MPPqBPnz4MHjyY2tpaXnnlleL1fD7PG2+8wUknnQTA2LFj8Xg8HcZs3bqV5cuXF8dMmDCBeDzOwoULi2Pefvtt4vF4hzHLly9n69atxTHz5s3DMAzGjh1bHPOPf/yjQwuGefPmUVdXx6BBg7r/zRC9Um/87UiIA800P87ITpgwgaeffpqlS5dy1lln9eCsxOFI0zRG9A0T8XtZ35IilbOwHUUqZ7G+JUUk4GVE3/AB7WfVoz/9r7nmGt544w3WrVvH22+/zX/8x3+QSCT4xje+gaZpzJ49m9tuu41nnnmG5cuXM2PGDAKBANOmTQMKW3i/+c1vcvXVV/P3v/+dd955h6997WvFpUWAo48+mnPPPZeZM2fy1ltv8dZbbzFz5kymTJlSTFWfffbZHHPMMUyfPp133nmHv//971xzzTXMnDmzmGGaNm0ahmEwY8YMli9fzjPPPMNtt93GVVdd1SMNyETP6I2/HQlxIL300ksMGzaMFStWFO/7yle+QmVlZQ/OShzOqsM+ThtexdCqEImsyeZYmkTWZGhViNOOrKI67DuwE/rU+w8/ha9+9auqT58+yuPxqLq6OvWlL31Jvf/++8XrjuOom266SdXW1irDMNSpp56qli1b1uExMpmMuvzyy1V5ebny+/1qypQpauPGjR3GtLS0qIsuukiFQiEVCoXURRddpKLRaIcxGzZsUJMnT1Z+v1+Vl5eryy+/vENrBaWUWrp0qTrllFOUYRiqtrZW3XzzzV3ewintFg5+jfGM+p+FG9Ujb6xRzy7ZrOYu26qeXbJZPfLGGvU/izZKywVxSMpkMurKK69UgALUhRde2NNTEqIDx3FUa1tObY1lVGtbrltaLOyos5/fmlKyfelASiQSRCIR4vG41FsdxJoSWZZvSbAlliZvO3hdOn1LA4zoGz7wvx0JsZ+tWLGCadOm8d577wEwa9Ys7rzzTnw++bcuDh+d/fzepxa4sViM//3f/2Xt2rV8//vfp7y8nCVLllBTU0Pfvn33edJCHCyqwz5ODxm94lwqIfYXpRSPPPII3/ve98hkMlRVVfH73/+eyZMn9/TUhOi1uhxYtRcoRiIR1q9fz8yZMykvL+eZZ55hw4YN/OEPf9gf8xSi19E0jbKg7H4Sh67/+Z//4Tvf+Q5QqEV9/PHHO5znKoTYVZeL16+66ipmzJjB6tWrO6SBzzvvPP7xj3906+SEEEL0nP/4j//gzDPP5J577uGll16SoEqITuhyxmrRokXFRnA76tu37147kAshhOjd8vk8v/71r/nud7+Lz+fD7XYzb948dF3ah+xvSikpLThEdDmw8vl8uz3vbtWqVVRVVXXLpIQQQhxYq1evZtq0afz73/9m48aN3HvvvQASVB0Ashnm0NLl75jPf/7z3HLLLcUGcZqmsXHjRn7wgx/w5S9/udsnKIQQYv9RSvH4448zevRo/v3vf1NWVsZnP/vZnp7WYUMOde86pRTRVJ6GeJZoKt/rzmbtcruFRCLB5z73Od5//32SySR1dXU0NDQwYcIEXnzxRYLB4P6a6yFB2i0IIXqLWCzGd7/7XebMmQPAxIkTeeKJJ+Sw+ANEKcVrKwtB1aCKYIelP6UU61tSDK0KcfpRVbIsuF1PZvf2W7uFcDjM/PnzefXVV1myZAmO4zBmzBg5ykAIIQ4iS5Ys4Utf+hIbNmzA5XJxyy23cN111+Fy7f4cTNH9unKou+xA/ji7F8/kqQ758HlcZE2btc1JtrXlOG14D3RZ34196mMFcMYZZ3DGGWd051yEEEIcIBUVFcRiMYYMGcJTTz3FuHHjenpKh53OHOq+LZWTQ90pZPCWb0kQz+Q7ZPeChptB3iDrW1Is35Lg9JDR49m9LtdYzZo1i/vuu2+X+x944AFmz57dHXMSQgixH8Tj8eL/Dxw4kBdffJF33nlHgqoeIoe6d15Xsns9rct/W3/5y184+eSTd7n/pJNO4n//93+7ZVJCCCG615w5cxg0aBBz584t3nfSSSdJrWcPkkPdO68z2b287fSK7F6XA6uWlhYikcgu94fDYbZt29YtkxJCCNE9kskkM2bM4MILLyQWi+22D6HoGZqmMaJvmIjfy/qWFKmche0oUjmL9S0pIgEvI/qGe3xpqzc4mLJ7XZ7BEUcc0eE3nnYvvfQSQ4YM6ZZJCSGE+PQWLlzI6NGjefzxx9F1nRtvvJE///nPPT0tsYPqsI/ThlcxtCpEImuyOZYmkTUZWhXitCN7RzF2b3AwZfe6XLx+1VVXcfnll9Pc3FwsXv/73//OL37xC375y1929/yEEEJ0kW3b/PznP+fGG2/Esiz69+/PH//4R0455ZSenprYDTnU/ZO1Z/e2teVY35LqsCuwKZntVdm9LvexAnjooYf42c9+Rn19PQCDBg3i5ptv5utf/3q3T/BQI32shBD72yuvvMLZZ58NwAUXXMDDDz9MWVlZD89KiE/vYOhjtU+BVbvm5mb8fj8lJSX7+hCHHQmshBAHwmWXXcYJJ5zAjBkzesVv8UJ0l546V/GABFai6ySwEkJ0t3Q6zU033cT3v/99qqure3o6QhySurXz+pgxY/j73/9OWVkZo0eP3mtkuGTJkq7PVgghxD559913ufDCC1m5ciUrV67k+eef7+kp7Tc9lakQois6FVh9/vOfxzAMAL7whS/sz/kIIYToBMdxuO+++7juuuvI5/P06dPnkG7S3JO1NUJ0RZeWAm3bZv78+YwaNUoKIfeRLAUKIT6txsZGZsyYUWx9M3XqVB599FEqKyt7eGb7x57OiGtKZon4vb3mjDhxaOvs53eX+li5XC7OOeccYrHYp52fEEKIffDOO+8watQo5s6di8/n48EHH+TZZ589ZIOqnc+ICxpuXLpWOCOuIkg8k2f5lsQuvY2E6Cld7mM1cuRIPvroIwYPHrw/5iOEEGIvjjjiCEpKSqipqeHpp5/m2GOP7ekp7VddOSOuLOjtoVkK8bEud17/2c9+xjXXXMP//d//sXXrVhKJRIebEEKI7rV+/fpiRiYUCvHyyy+zcOHCQz6ogoPrjDghYB8Cq3PPPZf33nuPqVOn0q9fP8rKyigrK6O0tFTqroQQohsppXj44Yc5+uij+fWvf128/4gjjsDnOzxqig6mM+L2RClFNJWnIZ4lmsrLsuUhrstLga+99tr+mIcQQogdbNu2jW9961v8v//3/wD429/+xmWXXXbYtRdoPyNubXOSQd5gh9fffkbc0KpQrzgjbndkN+Php8uB1cSJE/fHPIQQQmz36quvMn36dOrr6/F4PNx5551ceeWVxaDicOrndDCdEbezPe1mXNucZFtbTnYzHqK6HFgBRKNRHn30UT744AM0TePoo4/m4osvpry8vLvnJ4QQh418Ps+Pf/xj7rrrLpRSHHXUUTz11FOMHj26OKa7MiAHU3BWHfZx2vCq4uvelsrhdekMrQr12szPzrsZ29/boOFmkDfI+pYUy7ckOD1k9Nr3XeybLh9p88YbbzB16lQikQgnnHACAIsXLyYWi/Hcc89JRusTSB8rIcSevPPOO3zmM5/Btm0uueQS7rnnHoLBYPF6d/VzOliXpw6mYDCayvN/S+sJ+zwEjV1zGKmcRSJrMmVUnexmPEh065E2O7rsssv46le/ykMPPYTLVdilYds2l156KZdddhnLly/f91kLIcRhbPTo0fziF7+gf//+fOlLX+pwrbsyIPu6PNUbghpN0w6aIKQzuxm3pXKym/EQ1OXAau3atfzlL38pBlVQaBx61VVX8Yc//KFbJyeEEIeyWCzGrFmzuPbaaxkxYgQAV1555e7HdkM/p30Nzg7WDFdP2nE34+4yVgfDbkaxb7r8NzpmzBg++OCDXe7/4IMPOP7447tjTkIIccibP38+xx13HE888QRf//rXcZy9Zy7aMyCGR6ctZxFN52nLWSgK1Ryd6efUleCsXXuGa21zkrDPQ7/SAGGfh7XNSV5f1UxTIvsp3oVDV/tuxqZkdpf2Cu27GfuWBnrtbkax77qcsZo1axZXXnkla9asYfz48QC89dZb/PrXv+aOO+5g6dKlxbGjRo3qvpkKIcQhwLIsbr31Vn7605/iOA5Dhw7l4YcfRtf3/nuu4dbJmQ5LNsRI5S0sx8Gt61QEDQZW+HHr+idmQLq6PCUF2PvuYN7NKD6dLgdWF154IQDXXnvtbq9pmoZSCk3TsO3dN3QTQojD0fr167nooot48803Afj617/OAw88QCgU+sSvzVs2zckcG1vTDK4I4PV4MC2HrfE08XSe0qCX4/qV7jUD0tXlqYP5OJneUBN2MO5mFJ9elwOrdevW7Y95CCHEIW358uWcfPLJJBIJwuEwDz/8cPEX1U+ilOL9+iQhn5t+ZX7iOYuIpuF16UR8Hta1pNF1jWPrQnsNHrrabPNgLcDuTTVh1WEfp4eMHg/yxIHT5RqrgQMHdvrWFbfffjuapjF79uzifUopbr75Zurq6vD7/Zx22mm8//77Hb4ul8txxRVXUFlZSTAYZOrUqWzevLnDmGg0yvTp04lEIkQiEaZPn04sFuswZuPGjZx//vkEg0EqKyuZNWsW+Xy+w5hly5YxceJE/H4/ffv25ZZbbpGjCYQQnXL00Udz/PHHc9JJJ/Hee+91OqiCjzNHQ6tKGNk3Qm3YR8a0aUnlyFgOQ6uCVIW8eN27D4DatS9PRfxe1rekSOUsbEeRylmsb0ntsjy1L8fJ9PTxLb2xJqx9N2NtxEdZ0CtB1SFunxqEdrdFixbxyCOP7FKTddddd3HPPffw2GOPceSRR3LrrbcyadIkVq1aVUydz549m+eff545c+ZQUVHB1VdfzZQpU1i8eHFx5+K0adPYvHkzc+fOBeCSSy5h+vTpPP/880ChXcTkyZOpqqpi/vz5tLS08I1vfAOlFPfffz9Q6F8xadIkTj/9dBYtWsSHH37IjBkzCAaDXH311QfqrRJC9LCuLDEtWbKEo48+Gr/fj8vl4q9//SuRSAS3u2s/enfMHAUNNxG/h1TOxnQcPLqOz6OzJZ7pVOaoK8tTXc1w9XSmSGrCRG/Q5Qah3a2trY0xY8bw4IMPcuutt3L88cfzy1/+EqUUdXV1zJ49m+uuuw4oZKdqamq48847+fa3v008HqeqqoonnniCr371qwDU19fTv39/XnzxRc455xw++OADjjnmGN566y3GjRsHFIrtJ0yYwMqVKxk+fDgvvfQSU6ZMYdOmTdTV1QEwZ84cZsyYQVNTE+FwmIceeojrr7+exsZGDMMA4I477uD+++9n8+bNnf4mlQahQhy8Ohs42LbNXXfdxY9//GO++93vct99932q590fzSY7GyDutSlpwMtpRxb6XnVX89JPQ5pyiv2ps5/fPd5A47LLLmPy5MmcddZZHe5ft24dDQ0NnH322cX7DMNg4sSJxcLPxYsXY5pmhzF1dXWMGDGiOGbBggVEIpFiUAUwfvx4IpFIhzEjRowoBlUA55xzDrlcjsWLFxfHTJw4sRhUtY+pr69n/fr1e3x9uVyORCLR4SaEOPh0dolp8+bNTJo0iRtuuAHLsmhqavrUG3k6s3W/rtSPUqrTS3CdXZ5qz3ANrQqRyJpsjqVJZE2GVoWKQdXOmaKg4cala4VMUUWQeCbP8i2J/b4suHNNmEJ1aE1hePRPbEkhxKfVo0uBc+bMYcmSJSxatGiXaw0NDQDU1NR0uL+mpoYNGzYUx3i9XsrKynYZ0/71DQ0NVFdX7/L41dXVHcbs/DxlZWV4vd4OYwYNGrTL87RfGzx48G5f4+23385PfvKT3V4TQhwcOrvEFPvbi8ycOZPW1laCwSD3338/M2bM+NTLTp+0dV/XNOJpkxeWbd0vS3CfVIDdW3YP7lgTZjkOG1oytKRyxdYUQa+b0oBHmnKK/Wqf/nXFYjF++9vfcv3119Pa2goUagm2bNnS6cfYtGkTV155JU8++SQ+356/8Xf+Jm1v5bA3O4/Z3fjuGNP+29fe5nP99dcTj8eLt02bNu117kKI3ueTAoew2+HW62fz5S9/mdbWVsaOHcuSJUu4+OKLu62WZ0+Zo8oSA6WgJZXbr8Xae8twdWb34IHIFLVn9j7almLppjhb42kCHhcVQQO/W2dtcxvNyRx5S1oBif2ny4HV0qVLOfLII7nzzju5++67i7vrnnnmGa6//vpOP87ixYtpampi7NixuN1u3G43b7zxBvfddx9ut7tDNmhHTU1NxWu1tbXk83mi0ehexzQ2Nu7y/M3NzR3G7Pw80WgU0zT3OqapqQnYNau2I8MwCIfDHW5CiIPLJwUOZjrB239/ASj0+HvzzTc58sgju30e1WEfpx9VxZRRdUweWcfkkX0I+zwoVIcluIDhorLEYHM0w1sftXxiV/dPa192D+4PmlZoOZHImGyOZYj4PHhcOnnLIZ6z6FfmJ+Rz8359UnZ0i/2my//Kr7rqKmbMmMHq1as7ZJrOO+88/vGPf3T6cc4880yWLVvGu+++W7ydcMIJXHTRRbz77rsMGTKE2tpaXnnlleLX5PN53njjDU466SQAxo4di8fj6TBm69atLF++vDhmwoQJxONxFi5cWBzz9ttvE4/HO4xZvnw5W7duLY6ZN28ehmEwduzY4ph//OMfHVowzJs3j7q6ul2WCIUQh5bdBQ47fjCHKmr59o9/wV+ff5E777wTr3f/LXftmDnSNI36eKZDJi2eybNsc4J/b4iyKZpi3opG/u+9hv3aZqA3Hd/idbuoCnkZWhUkYzmFlhSmTW3Yx8i+EYZWlexybI8Q3anLNVaLFi3iv//7v3e5v2/fvrtkdPYmFAoVDx1tFwwGqaioKN4/e/ZsbrvtNoYNG8awYcO47bbbCAQCTJs2DYBIJMI3v/lNrr76aioqKigvL+eaa65h5MiRxWL4o48+mnPPPZeZM2cW533JJZcwZcoUhg8fDsDZZ5/NMcccw/Tp0/n5z39Oa2sr11xzDTNnzixmmKZNm8ZPfvITZsyYwQ033MDq1au57bbb+PGPfyzbdoU4xLUHDmuak1Qpg21NjTz002v48vRvceIpZ9KUzDJlylROP6rqgM5r50xae1CVypuU+r2EDDeNiSyrmxNkLXu/7czrTce35CwHw+NizIASsqZTbEkRNFyFE0Ec1SubmopDR5cDK5/Pt9udbatWraKqqnt/qFx77bVkMhkuvfRSotEo48aNY968eR2Of7j33ntxu91ccMEFZDIZzjzzTB577LFiDyuAP/7xj8yaNau4e3Dq1Kk88MADxesul4sXXniBSy+9lJNPPhm/38+0adO4++67i2MikQivvPIKl112GSeccAJlZWVcddVVXHXVVd36moUQvY+madRGDP65uplnn/s/3vjtLaTjraxbvYofPv4KA6rCnQocuvuYlR0zaQHDxYaWDKm8SU3IB5q2/X43QypL2NaW2689nHrL8S3t70nOcijxffKxPUJ0ty73sbrkkktobm7mT3/6E+Xl5SxduhSXy8UXvvAFTj31VH75y1/up6keGqSPlRAHn6ZElnlLN/HIL27ln8/+AYDK/kdw2ndvZcSxI/jKCf04pi7yiY/R2eaZnQ3AlFK8trLQAqKyxODfG6IEPC4Mj6uwBNeWozbsY1TfCOm8fUB6OPX0GX07vic77uBsv7a+JcXQqhCnH1Ulqw2iSzr7+d3ljNXdd9/N5z73Oaqrq8lkMkycOJGGhgYmTJjAz372s081aSGE6G2UUvzfG4u46XuXsHntSgAm/+d/ceHlPyDg99OczNIQz3F0nz3vWN5T88y1zUm2teU6LNF1JQDbcQlu3bY06bxFyHCTNW3iWZOgx83A8gCaph2wc/3aa8B6Sm9alhSHpy4HVuFwmPnz5/Pqq6+yZMkSHMdhzJgxuzT4FEKIQ8F7Kz7kO/8xCTOfo7S8gmt+9ivGnfrxzztd0/bao6krx6w0J3OdDsDatS/BvfVRCx9ta6MxkSVguKkN+xhYHqA0UJjT4bQE1luWJcXhqcuB1R/+8Ae++tWvcsYZZ3DGGWcU78/n88yZM4evf/3r3TpBIYToSbX9BnLiWeeTjW/j2p/dR3lVx4bDn5QJ6mzzzGgqv8/n3FWHfUwZ1QeUxurmBEMqSygx3MVxuzvX71D3SU1Nhdhfuvyry8UXX0w8Ht/l/mQyycUXX9wtkxJCiK5SShFN5Tt9pMvevPrqq8X+d4Zb5+Lv38oPf/WHXYIq+ORMUGebZzYn853uXr47uq4zfmg5/cuCbGvLkc7b2I4ilbNY35I6LJfAOntsjxDdqcsZqz11Pt+8eTORyN6LN4UQYn/oSl3S3uTzeX784x9z1113cc455/DCCy9QGvAwuKaMtc1JSnyeXYqh2zNBEb+baCpfzI5E/G7iGYt4Oo9lO2TyFiW+XbNF7YEZqE8MwD6pRqqzS2A9XWAuxKGs04HV6NGj0TQNTdM488wzcbs//lLbtlm3bh3nnnvufpmkEELsSVcKw/dm9erVTJs2jX//+98ADBgwANM08Xq99Cvzs7a5jRVbEwwsD+D3ujsUQ9dGDF5fta0Y2OVMe3vQ4sJw62xqzfBRc4oTBpZ3qMPaMTCrChnF1glBY9/bBHzSElh3BaFCiN3rdGD1hS98AYB3332Xc845h5KSkuI1r9fLoEGD+PKXv9ztExRCiD3pSmH4njIySikee+wxrrjiClKpFOXl5fz2t7/li1/8YiEI+aiZLbHCjrvmZI7mZI7KkJeKoMHQqhC1EaM4h+qQj5zpsKaxjaZkjuqQwegBpQyvLeGtj6K88WETJw6uoCbs22WXWlnQS9/SQKFNgHfXNgFdqZHa08687gpChRB71unA6qabbgJg0KBB/Od//ieGYey3SQkhRGd0tjB8Tzv24vE4l1xyCX/6058AOO2003jiiSfo169fhyCkqsQg4HUT8XtojGcJeNyMG1zBEdVBXl+1rRjYAaxpbsNSiqNqSmhqy7E5mmVkvzCnD3excH2MDxuT5G0bw+3aZYlu5zYBhlunNZVnayJDedDg2LrQPi/ZdUcQKoT4ZF2usTrmmGN49913GTduXIf73377bVwuFyeccEK3TU4IIfamM4Xhe6tLcrlcvPPOO7jdbn7605/y/e9/H5fL1SEIKfV7WbstRWuqUCvl0nVa03kqSwwqS7xsiaWpKjFI5WximTxbYxlK/R40XafU76UllSOVsykNGJw0tJzGRI5TjqimOmzsUttUHfYx8chK3l7XygcNSRpiGUxbUeJz4/cUDg/WNG2fskqfNggVQnROl3cFXnbZZWzatGmX+7ds2cJll13WLZMSQojO2N3hyDvaXV2SZVk4TiHQKikpYc6cOfzrX//iBz/4QfEorPYgxHC7eL8+QUMii9/joiJoEPC6yFoO/1zdzMqGJC2pHGuaUiza0MriDVE+2paiPpEllbPwuHUsx8G0C8/n97hxuzQiAQ+lAQ+xtNlhF2NTIsv79Uk2taZY25wkms5THvQyun8pfUv9rG1O8vqq5n06ULmzuxPlDD0hPp0uZ6xWrFjBmDFjdrl/9OjRrFixolsmJYQQn0QphVKKgNfNhpYUR/UJoWt6h+s71yWtW7eOr33ta1xwwQVceeWVALv9eZazHHKWTSxlkTItqks+Xh7z6S7qIj5WN7XxzsYo65tTKE2jqsTAcOm0tOVoacuRyzvURHy4dR2PqzCv9kAvkTFZUd+xgDzodRNN57Edh3jGImx4KC31EM+afLA1ych+YQZV7PuS3Y5B6KcpjhdC7F2Xv4MMwyj2d9nR1q1bO+wUFEKI/aUpkeW1lc28sGwrm6Ip1m1L8bcPmtkaT++xd9PTTz/N8ccfz5tvvsltt91GKpXa4+Mbbh3bUTQmMkR8u7YisGxFqd/L+m2FoMqjg8+t4/e6KAsauDWNjGmysTVFecBL0HAVA70Sw807G6OsbU4S9nnoVxogZLh5e10LSzbGMNw6qbxNacCLz+umJuQjlTfZ0JIBjU/sZ7UnpQEPfUsDNCWzu/T4ap9b39LAYdNAVIj9pcuB1aRJk7j++us7NAmNxWLccMMNTJo0qVsnJ4QQO2svKm8PTI6ujTB6QBkaiiUbYqxsSJDImgytCnHakVX4NZNvfOMbTJs2jUQiwUknncTbb79NMBjc7eO3Z8I8uovGZA7Pzj8llSKWyRMJuEnmLI6sDlJieGlMZslbDpUlXlwunVjGJJk1cesazckc61pSRPwelIJE1mRQRZCg4cala4CGS9dx67ChNYu5PYsFgKZ1qNXa1yW79jP0In4v61tSpHLWYd9AVIj9ocsppl/84heceuqpDBw4kNGjRwOFFgw1NTU88cQT3T5BIcShq6uNKve0s62u1E9t2OCDhiT9SgOcflQVZUEvixYtYtq0aaxduxZd17nxxhv50Y9+tMfs+o49npqSWeLpPO9sijFk+3KiaTnEMnmChoeygJf6WIbaSIDaiGJDS4aWVA7LcQgabjI5D/FsnsUbo4T9HoZWBRnZN8KaprZdCshNx8F2HCpLDOKZPEChHkov1EN53DpWzsS0HTTY5yU7OUNPiP2vy4FV3759Wbp0KX/84x9577338Pv9XHzxxVx44YV4PJJCFkJ0zr40qtzbzjZd1xlUESSRLSyRrdlQz8TTTiObyTBgwAD++Mc/8tnPfnav89mxx1NVf4O2nMmyLQnWNiWpifgI+Tz0iQQYUO6jPpalImjg0qDE52VkPw+pnE1LW47VTW14XBq1+BjdvwyfW6ctZ7J8S4JU3qQ61PH1eXQdt0tHAboOJUahtspw62iahmk5uHUdt6596jP/5Aw9IfavfSqKCgaDXHLJJd09FyHEYaI9iIll8pQYboIuN45SrNmhUWXVbj78O7OzbV1LG6+taiadt5j8jSvYtPp9fvizezny6P7FcTtnyiJ+924zYcf3L0PXNTZsSxHyuRndvxS3q7C0Vxvx0a8sQHNbjuD2A4+DXhdrMya24+B1adSVBulf7kdDo0oZfLA1wbZknkx5x+NtgoaL8qCXja1pgl4XAyuCrGtO0dSWI2K4iWbyVAR9NLflKO2GJbs9NRAVQnx6nQqsnnvuOc477zw8Hg/PPffcXsdOnTq1WyYmhDg0tS/nbYllcBzFhpY0llPIyJQHvKRyNvNXbyMS8FAfy3TIZvUr8+91Z9sLzz1Lq6sMfexoBlYE+dals8maNo1tOV5f1cxpw6sAdsmURXxeNsfS9C31dwhYIgEPQ6tCWLZiWzLP1niG0oC3uHQG8PqqZta1pCgx3GTyNhtaUmhAWdBgYEUhqIJCMDOgIkBTMseG1jTH9Pk4ONI0jQFlAdY1t2E5LiJ+N8f0CbOmOcmG1jQeVyGwPEKW7ITo9TTViSPgdV2noaGB6upqdH3P6/qapmHbu+8nIwoSiQSRSIR4PE44HO7p6QhxwEVTeZ5euJFNrYWAqtTvxePWi/VLpqXIWjbH9AkzsCJYPHalKZkl7Pfg0XVaUrkOmaVMOsXDd97Ei//7JBX9hvC7Z+YRCHx87JZSivUtKSqCBqbjkMiYHY50WdOcZN22NKcOq6Q8WDhVIp7JF+umcpZFtM3k5COq+OywSobVlBSfe0V9nJffb2Btc4p4Jk9LW57BlUFOGFhG//KOBfK2o1jZkCDgdQOqwxyakoXeVH6Pi3jGxEERNtyUBXwMqQ4Ud+zJkp0QPaOzn9+dyli1N9Pb+f+FEKKrMnmL1Y1JcrZDn4gPw6MDGobHRbXL4J3NUdJ5h9OHVxezUjseu1JZYhAy3KzYmqDU76Fp/Sp+9aMr2LJ+DWga4yeehcfdcZlL295n6t8bolSHDI7eIVsUNNwMqSzhw8Y21jSl+MxgL4mMybLNhXqoUr+3UOukNJI5k3c3xSgNeKgO+4p1YkGvi5OGVJAxbd7bFEMDNrRkCPs9RPwfzyVr2pQHvYwbXMHmaKZDAXlF0EDToC1r4ajCL7RlQYNxg8uoifgP1F+PEOJTksZTQogDpimR5Y1V21jd1IbPo5PKWYR8HqpDhbP4srbCtkEDrJ1+ifv42JUMYZ+bpniG//n9w7z9pwdwLJNIRTUXXXsXUz93Dvr2AMV0HNyaBhpE0yZb4xmOqA7ukvUpMdwMLA+wsTXFUbUlbGjJkMqb1IR8KKCpLUdtqZ+jakpY2djGPz/cxmlHVbJsc5x4Js/gykIGSylFLGPSEM/QlitkvEb286ChdWhYOqymhGE1JcU6r0TG5J1N0WImrXr7Ic2NiSxvfLhtt4cjd3VHpRDiwOhUYHXfffd1+gFnzZq1z5MRQhy62gvWt8RTlBguNDQ8ukZzonCky8CKIB4XZEybsN+Nx7VrgXrOslmxNUEfv8NLv7iGdxa8AcCICWcw84Y7cQcjNCayNLflaE3lSWZMElkTRSFYa0rm2NSapcTwUBr4OJOkaRpHVIWoj2f5YGvhKBnDXVhyjKVNSgw3hktn2ZYEG1vTvLspyodNCVJZmyNrQx1qpQaWB0hmLKKZPPXxDAMrAsXdfDv3iioLelFKsaI+QSJjdvpw5H3ZUSmEODA6FVjde++9Hf7c3NxMOp2mtLQUKDQIDQQCVFdXS2AlhNjFjv2nhlaFqI/l2BJLsyWeRTmKjGnTmsoR8nnQgFK/F+9OfZoUijVNKUzbYXi/KhzLxGv4+M51P2HyV6azvjVNUzzH+1viBAw3HpdOImuRMR1QDlnTIegtHMq8fItiRN9wh+DK8Ogc0yeEUrBkY5S8ZZMyHXQNQl43HzQksZUi6HFhOg5rmtpoSeXJWTYBr6v4WO279tZvS7N2W5KNrWkqS7x77BXV1cORd24L0V6jtXaHHZUSXAnRczoVWK1bt674/0899RQPPvggjz76KMOHDwdg1apVzJw5k29/+9v7Z5ZCiIPajsFDwOuiLOBhTZODchw0dLxunZzpEDQc0DR0XSfg7RhYtcbb+KihhSG15UQCBtfd8QDptiQDhh5JKlc4564hmSFnKXweh5aMSd6yCRtuUnkLtwvKAgYencIRMa1pIn5PcQmvMZklZHjY1pbFpWt43DoVXhd+t4v6WJZoJo/P48JxFKUBDxVBLy2pPKub2wj7PYwfUlEMjEoDXo6o1gj5XZx2ZDXVYd8el+o600JiWypHznL22CB1b9ktIcSB1eXWvTfeeCP3339/MagCGD58OPfeey8/+tGPunVyQojeSylFNJWnIV5YytvbBuMOwYMGoOF1a5QHvNREfPSN+CkLegqNOUu8+D0661vSxWNXPljxPt+7aDIL59zHEVWFpbeq2joidYNZuiXOog2tLNkY5aPmFGG/C5/HQ2u6sLMu7ziUlxgc1SdEic+F2+Uibzk0xDIkshapnMXy+gRbo1ne2RjlnU0xYukczck8JR4dNI20aaGUwnBr5EwbR0FVyKBfaYC8pVjT1EZbzurw3jS35TiiKkRVyCBnOcTS5m7fox0PR96dHQ9H7kp2SwjRM7pcvL5161ZMc9dvWtu2d3s4sxDi0NPVGp8dgwcFZC2b4TUhYhmLtqxJ1nLQdZ2+kQAVIS/xjEmJ10V9LM3f//okzz1yJ2Y+R1ushVwqDiVVxNJ5lm2JE0ubBA0XLh3cLg3L1shphd13tWEfHpeOz6PjKLCcHMNqSoi2maxtTrKxNYXhdpHImOQdu3Den+2QySvacibvbjbxG24SGROXrtGaMikPGrh1jZylqAoVjqBpSGRpSGQLBfjbWydomkYia/LCsq17fY/aD0de25xkkLdjYf2OBe+lAQ+NiVyns1tCiJ7R5cDqzDPPZObMmTz66KOMHTsWTdP497//zbe//W3OOuus/TFHIUQvsi81PjsGDxG/B8txqAgalAa8hd1vyRxhnxvDo7OxJcPmWIqPNtXzyiO3sG7xPwD4zGdP59s/upusy4/jOKzYmmBNcxsuDVraCgcbO0oR9OqkLJusaePRtWIQYlo2br3Q1qAi6CXkdzPxyCo+bGgjk7doiJu0ZS383kKQ5nXrtOVsrMz24vft8Y5LK/Sjsh1F0HDTryxAOm+TytlsjqWLrROi6Tzb2nKf+B61H468rS3H+pbULr2tdix43zFA3V2D1B2zW0KIntHl777f/e539O3blxNPPBGfz4dhGIwbN44+ffrw29/+dn/MUQjxKXVl2e6THmfHGp+g4cala4Uan4og8Uye5VsSKKVQStHalmPV1gSrGpKEjMJZeOu2pbBtRda0SGRNGhM53NvPw2tMZknmTFYtWcDj3/8q6xb/A93lYfy07zHxil+Q1EvQ0Fi4vpUl61tJZQrBlOU4lAY8BLwe1jS3EXS70IDmtu2vVSlimTwVQYOAV6c5mWNYdYjqkEEsU8jwxLMWluPQksoTy5hkTIegR8etFQ499rp0Sgw3joJ03kLXC+9H2rQ5qjbEl0bXMXlkHZNH9iES8OAo9YnvUbv2w5GHVoVIZE02x9IksiZDq0KcduTHQVh7gNqUzO7yd9ie3WpvJCqE6BldzlhVVVXx4osv8uGHH7Jy5UqUUhx99NEceeSR+2N+QohP6dNszd+5V5JSqlM1Pqsb23i/Ps7iDVHWb0vRks7jQqMq5KEs6KUhnuODrSZ+rxvDBTlbYdqKsOHio60tLH7sZux0An9Vf478zx9ROmQ49bEM0bTFqH5hNrSkqE9kCXhdpEyb0oCXulI/fo+r0BIhmibkc+N2aWyOZUApIgEvVSEPG1rSxSxQ3lbEsybNyRyZvIWDhu3YuHUNy1FYgA0YuobpKAr7/gpBY86ySWRMbNth7MAqBlcVellFU3nqY5lO7/Jr15nDkbuS3RJC9Ix9bhA6aNAglFIMHToUt1v6jArRG32arfm7C8gCHjetqcJj7U77IcgvLKvn/S0JGhJZtiUzZC0HpaAhoVMbNvB6XDgKvC4Nt66xKZomkbWwlQPKTf/zZ5H+6B0GnPcd3IaPrOVgOop4PE1TMkfY76JfqX97NkjHtG2aEjkGVgQZXlvC+m0plILKkJdUtlAUXhH0AlqHtgfRVB5NQUMii6ZBXdhHfSwL2Lh1sByF4yhsHYLeQuG9QkOhEU0V+lsdP6CMzw6rBArH9WyKFoKmqpCxx/doT3VQnTkcuT271f530965fU/tHIQQB1aXI6J0Os0VV1zB448/DsCHH37IkCFDmDVrFnV1dfzgBz/o9kkKIbpub1vzB3oDrNya5J+rt3H68CrKgt4OWY6mRJbXVjbRkMhS6vdQ6vfi0mBDa5r121KUBz30iQR2ec5M3mJbIs+2VI7GRIaWZA5LaZRuP9YlmbPY0JqmPOhhQHkJiYyF6dhsevsF3MEIoWHjaMvZBIZNoHLEqYWltrxFxrQp8eqkcjapvEnEH6SixEfaNAl5XUDhfL2mZI7+ZT78HjcnH1HJpGNqi/VGeVvtkgUqtE3wEcuY1IQMfF43VWGDlGlhmg6WY6PrGrqmMaSqhLDPQ952MC2HI6pDjOobYWS/CACvrWwuZKIyeVY1tpHMWgyvLelwpA10Tx1UZ7JbQoie0eXA6vrrr+e9997j9ddf59xzzy3ef9ZZZ3HTTTdJYCVEL7GnrfnthwvXxzN80JBgW1uWYdXhYrZDKcX81dtYsjGKS9epj2Vwu3TKg14qAh4sR7F4Q4zPjTDQ9Y93pyml2NCaLi6RpfM2DlBiuHBvP7w96HGTNR2SWZtY2sTJtbH6L/ey/u1XcAfCHHP5I1haCbYDjrIIGi48Lh3LUUTTFsmshe04hWJ3vyJvOphWjojfS8DjIprK4ShFyOdm/JBK+pTu/Yw9TdMY0S9EybtuWlJ5PLpOiddFecCgNZUrdIB3u/B7XZw+vJq6Uh8rG9roVxrg9KMKAWlzMtchK1hVYpDMWKxtbiNn2ozqHykGVzvv8vs0OpPdEkIceF0OrJ599ln+53/+h/Hjx3f4YX3MMcewdu3abp2cEKJrdqyJiqdNcpaNz/Px0lA8ky8eLhzxedAU+NyuDkuDsbTJP1c34wBVJR68Lg/xbJ53NkYxbUXAq9PSlufZdx2OqQvTJ+LHpUNzMkfAcGN4XCSzhZYsLl3HpX2cmdE00HVwFGxcsZjVf7qDTLQRTXdRc9J/4CsJk0wVlshs5ZAzweXS0bTCrr6cVah/qi4xMB3wujWypoNLN9E1yFoOFbrGKcOqGFZT0qn3rF9ZkM8MKmf5ljjbUnkMj4bPo1Hi8wCFIK025Mdwu9jYmqFPqZ9TjqykvMTYY1bwyJoQOcthczSD4XExZmApOdOROighDgNdDqyam5uprq7e5f5UKiU/KIToQTvXRFm2YlNrGq/LRV2pH4XqcLhw1nLwenQifi9Bw8X6lhTLNseJpvMkcxbDqktw6TrpvEVDLIfjOGwvMsKlwcbW9PbjWjyEfV6GVAU5rl8psbYcllIoBW5dw1EK1/afDQoNHJutbzxF0z/ngHLwV/al3xevJdB3OFkbNM1BAxSF5TsPDm6XTt5RuHQNv9dFTdhHLG2RtSx0zaHE8BD2uYmm84weUMpnh1Xu9ufR7g4uLg14OGFgOW5dI5O32daWQwF1kcLzt+VMDK+G5Ti71DHtKStYGvAysm8Ew61TH88QanRTGvBIHZQQh4EuB1af+cxneOGFF7jiiisAij9MfvOb3zBhwoTunZ0QolN2V6SeMS3WNqdYuK6FiUdW49l+qHCp34sC4lmT2rCPoOEq7lZb05wkk3coDXixbIVLVzQmsiRzeUKGh7Rps6E1TdDr5rj+YTa2pNmWzLOxNcsHDYnC8qEG2bxN3rLxuF3kbYVPK4RK2Wyatb//PpnNHwBQfvwkhn3+cjz+EvKWTVvOKvxMUQrlgO7WsBXoShFwa3h0F4bHVQik/F6a23JE23Jsa8tiK4OR/SJMHtlnt4HL3nZHtu+0i6VzDK4MousajqNI5kwMt4sTBpXTt9S/Sx3T3o6jKQ14GTOgjFBzkolHVtG/LCB1UEIcBrpcPXn77bfzwx/+kO9+97tYlsWvfvUrJk2axGOPPcbPfvazLj3WQw89xKhRowiHw4TDYSZMmMBLL71UvK6U4uabb6aurg6/389pp53G+++/3+ExcrkcV1xxBZWVlQSDQaZOncrmzZs7jIlGo0yfPp1IJEIkEmH69OnEYrEOYzZu3Mj5559PMBiksrKSWbNmkc/nO4xZtmwZEydOxO/307dvX2655ZZ97gckRHfZU2+pEsPDiYNKUWj8e0MrsXSOvG1jK0VTW46gx83A8kDxrDzLUTQlcqRNm5oSg1gmT0sqx4aWNMmMTX08y5bWNNFUjoDXRSJjsaE1Q308C0qRyVmsaWpjRUMb0bRJPGsRz5jYtk06bxHPWmQdF76qgehGgH5fvI6jLrgW2+XDsm1MW6EBOgpNB5cOunJw6xp+t4tIwKBfWYCqkI9YttAJfUCpn0FVQYI+N8f3K2XaiQOoiexaV9UeeK5tThL2eehXGiDs87C2Ocnrq5oBOG14FUdUh7GVIpW3sJViWHWYz43sw4i+kV0K/OGTj6PJWQ6lfi/9ywK7/XohxKGnyxmrk046iTfffJOf//znDB06lHnz5jFmzBgWLFjAyJEju/RY/fr144477uCII44A4PHHH+fzn/8877zzDsceeyx33XUX99xzD4899hhHHnkkt956K5MmTWLVqlWEQiEAZs+ezfPPP8+cOXOoqKjg6quvZsqUKSxevBiXq/Bb5LRp09i8eTNz584F4JJLLmH69Ok8//zzQOE4nsmTJ1NVVcX8+fNpaWnhG9/4Bkop7r//fgASiQSTJk3i9NNPZ9GiRXz44YfMmDGDYDDI1Vdf3dW3UYhus7fz4yLbl6Q+bEyyNZ4jmbHAgdpSPwPLA5QGvMTSeTa0pmmIZWhN5XG7dKrCXrKmw4aGQqPK0oAH23ZI5i0sB7ZE06zdlqQtW2hLYDsubAU5u9BU09TBq+nkTYdsKoUbE90fAQ2Gnv9d9LOn02/AQGLpPKmcRcZUOKoQWLVnrEr8nuKfy4JeBlQE8Lpd1IR8BAwXrak8SdvBcRSDyoOcO6KWmoh/l+W+iN/duYOLj6ri9KOqurTTrjTgoS7iZ3l9nD4RH16Xq5gB7M5CdSHEwUNTXUi5mKbJJZdcwo033siQIUP2y4TKy8v5+c9/zn/9139RV1fH7Nmzue6664BCdqqmpoY777yTb3/728TjcaqqqnjiiSf46le/CkB9fT39+/fnxRdf5JxzzuGDDz7gmGOO4a233mLcuHEAvPXWW0yYMIGVK1cyfPhwXnrpJaZMmcKmTZuoq6sDYM6cOcyYMYOmpibC4TAPPfQQ119/PY2NjRhGoT/NHXfcwf3338/mzZs7/ZtoIpEgEokQj8cJh8Pd/faJw1BDPMsLy+rpVxrApe+6+6+5Lcu2ZI7hNSVkTAdd1/jMwDJ0XSeWLnQBT+VN8pZTOFTYcVjb1EbetknlHJLZPC5dJ29ZNCXzeHQNTYe27MdZGg3QNQ00DZcOylFoukYwsY5lT9yKv6Ivx33zdipCPgaUB7EcRb9yP1ujWRZtaKU1XegnFfa70XSNtqxF2OsmEvTg87gI+9xomkb/8iCj+kUI+zykcjY522J9c5qh1SWccVQ1pu3wfn2yw3JfxOdlcyxN3+19r3aWyhU6wE8ZVdflXXZNiSz/WrONf3zYTDJnUeb3UBP2Ux02yFk2kYC3Q+d0IcTBq7Of311aCvR4PDzzzDOfenK7Y9s2c+bMIZVKMWHCBNatW0dDQwNnn312cYxhGEycOJE333wTgMWLF2OaZocxdXV1jBgxojhmwYIFRCKRYlAFMH78eCKRSIcxI0aMKAZVAOeccw65XI7FixcXx0ycOLEYVLWPqa+vZ/369Xt8XblcjkQi0eEmRHfa3XJU++6/rfE0Hl2jqsRHXWmAEp+HWCrP8voEbVmT9dvSRDOFY1/KggZDqoMMrwlRFTJoTuYp9bsZXFmCUg7RlEXOckhvb5dgOtDe49JywHQUtlPIICnHpumf/8O/759FtnUr+dbNHBmy+M8TB3LKkVWE/B7yloOpFBUlBiGvG6/HRcjvpSLopTrkQ3e5sJzCQccul4vacCHD5tYLByonc3kWb4jTkMiyOZZmzqKNPPzGR7y3OdZhuW91c4IVWxPkrN0v1/k8LvK20+WDi9uXF7e15RgzsJSjasIoTWNlY4J3NkapCBoSVAlxGOpyjdUXv/hFnn322W6bwLJlyygpKcEwDL7zne/wzDPPcMwxx9DQ0ABATU1Nh/E1NTXFaw0NDXi9XsrKyvY6Zne7GKurqzuM2fl5ysrK8Hq9ex3T/uf2Mbtz++23F2u7IpEI/fv33/sbIkQX7Xx+3I67/6pLDDKWQ0XIS1XIYERdmLqyAJoGDYkMa7cl8bh06kqDjOwbJuL3UhrwcmRNiPKgl2TOIpW32NaWJ2taeHUKheXb89yKQuuE9v8CkGph4x9/yLbXH0c5NkeMm8SUm54gWFGL26VR4nNTETRoTuZIZEwifg9Bn5uQ4UY5DvG0hUvTGFzpZ2hVCScOquS4fqVcOK4/x/UrJZE1WdmQYMmGGBqK0QPKOKomTCxtsrE1TSyVx3Kc4vl8QypLMG2HNU0pFLsm6PelYefOdW19IgGOGxDhs0dUcvYxNQyuDBIJePbYfV0Icejqco3VEUccwU9/+lPefPNNxo4dSzAY7HB91qxZXXq84cOH8+677xKLxfjLX/7CN77xDd54443i9Z2X2JRSn7jstvOYPW27/rRj2ldR9zaf66+/nquuuqr450QiIcGV6FY7nx8XNNw0t2Xxe1w0pfIditQBhlQGSWRMhteGyJiKAeWBwhIcH/87rggaDCj3szWWIZG18bldOI4iYzlYTsfwxN7hD6lVb9L04n3Y2TY0j48TL7qaIeM/R0WJga7BqoYkgyuDVJR42RrXiWdMwj43lm3j0nRypoOmafgNFw6QzFqkTZPSgJd+ZQFG9I0QTeV5bWWh4Pzo2hC6rtOWtUjlbQZXBIhnTTa0ZBjZz4OGRolReP0bW1Mc3SdEyPdxvdO+1kHtrq6t/bkw3BhuF/WxzC7nAQohDn1dDqx++9vfUlpayuLFi4vLZO00TetyYOX1eovF6yeccAKLFi3iV7/6VbGuqqGhgT59+hTHNzU1FTNFtbW15PN5otFoh6xVU1MTJ510UnFMY2PjLs/b3Nzc4XHefvvtDtej0SimaXYYs3NmqqmpCdg1q7YjwzA6LB8KsT/seH5coZt6jqqgj9qwr1ik3q79rLqI30tliReXDqmcjWk7eFw6QcNFayrL6sYUG1tTZE0bpRSOKtRSGW6wHTAdigGWBji2ybbX/4CdbcPXZxjDvnoDR408imjaoq/hJmfafLQtxeqmNkoDXjwuHVCs25YiljExbYVb1ykxXOTyFprjkLEV/1q9jfNG9iHiL9RZaZpG2rQYVBFE397R3XQcLNvB6/NQ6tdoSeVI5WxKjMLXHFEVoj6eZd22NEOrgp/64OK9tVnY8T3u6vKiEOLg1+XAat26dftjHkVKKXK5HIMHD6a2tpZXXnmF0aNHA5DP53njjTe48847ARg7diwej4dXXnmFCy64AICtW7eyfPly7rrrLgAmTJhAPB5n4cKFnHjiiQC8/fbbxOPxYvA1YcIEfvazn7F169ZiEDdv3jwMw2Ds2LHFMTfccAP5fB6v11scU1dXx6BBg/breyIOP7trZPlJH/zt58cNrgyAUpQFCst/O39dJm9h2Q5KFRpx/mtNCx5XoQGmbatCr6qWNMmsid/jIpO30TQN23awtnc7NzwuXNvrktpDB83lofL8a8ivfpPBk2YQCflJZG1cukYqa2IrGN2/lLasTWMiw5ZohtaUSd52iPg8ZEwb21GkchaxtIlSCrdLo9XwMHd5AyWGm88Oq8JR7BLUeHS90ETUdvC6dayciWl/HNQYHp1j+oToVxYgnjE/9cHFO9a17a4gvjvOAxRCHJy6HFjtqDNLYXtzww03cN5559G/f3+SySRz5szh9ddfZ+7cuWiaxuzZs7ntttsYNmwYw4YN47bbbiMQCDBt2jQAIpEI3/zmN7n66qupqKigvLyca665hpEjR3LWWWcBcPTRR3Puuecyc+ZM/vu//xsotFuYMmUKw4cPB+Dss8/mmGOOYfr06fz85z+ntbWVa665hpkzZxYr/6dNm8ZPfvITZsyYwQ033MDq1au57bbb+PGPfyy9aXqpfQlOeoO9NbL8pABA0zQGVQY5uk+Etc1Jqvg4W6qUYks0w+KNUXxunXTOZlVjkpa2HEHDjeMoGhJZtkQz5B2HsOEmEvAWlvq2f687QN5S+DwKHWh561lsdCpOnIquaXjrjiA8ZDgejwe3Dn6PTsDjJmc5HNMnwtCqwjEzbbkg722K0ZDI4tbAtB3cuo7t2FiOImcrXBoEXDpBr4toxmTu8gaiaZNThlXtEtQEDRflQS8NiSwRw41b17dnxD5e7juqNsJpwyuJZ6xP/W+iva5tbXOSQd5gh8eQNgtCHN72KbB69NFHuffee1m9ejUAw4YNY/bs2XzrW9/q0uM0NjYyffp0tm7dSiQSYdSoUcydO5dJkyYBcO2115LJZLj00kuJRqOMGzeOefPmFXtYAdx777243W4uuOACMpkMZ555Jo899lixhxXAH//4R2bNmlXcPTh16lQeeOCB4nWXy8ULL7zApZdeysknn4zf72fatGncfffdxTGRSIRXXnmFyy67jBNOOIGysjKuuuqqDvVTovf4NMFJT9pdB/WsaXc4y68zwdWONVfVIR850+G9zTHe2xTFBoZWBgvF3EoR8XvZ0JIilbexbQdbKVxA1nRIxdKYFnhcGiU+F6mcg+koktEWGp+/h9RHS0B3UzdiPEZFXyzHQdN02rs+OApS+cKy3YByf2HJ0XHIWw7JrEmp30NUmUTTJqbj4DhgOQqvDl63TtDnxu0qPJjHrbGqsY2B5X7Choc1zUmGVJYUl/sGlgdIZEzWtaQZWlWCz6OTylkdlvt0Xe+WmqfdvcefdnlRCHFo6FIfK4Abb7yRe++9lyuuuKJ4hM2CBQt44IEHuPLKK7n11lv3y0QPFdLHav/bU3DSlMwS8Xs7FZz0BKUUr60sdAffsZFl+7X1LSmGVoU4/agqgE/MxjUlsizbEufdjTFWbI0XDkn2uhheG8JRsHxLghLDhUeH1U0pAByliKbyeFw6GctBKUX7iprXreHWNVpXvk3TC7/ETsfR3Ab9z/s2deOn0Jaz0TQIGW7CAS+l/kK2xnIUJw2pwFaKhniWnOVgO4otsTTxtEksY5I1bTwuHdtxUICu6SilKDE8lAc92Eoxun8p6byN3+uiNuxnXUsK03YYWB7giKoQhkdnbXMbyZxFVYmB4dH3e0B9sAbwQoiu6+znd5czVg899BC/+c1vuPDCC4v3TZ06lVGjRnHFFVdIYCV61M7b4PfYZXs3tUc9bW8d1NvP8tsSS7O6sY3N0UynPsyVo2hIZGjL2egahH1eXLqOriBg6CRzJtuSeTKmjWUrNK0QCOVtGwdwUThaRimwzBwNr/6e+L8LJxb4awcz4qIf0XfwMDRV6GM1oDyIUg6Gx82IvmEMt4tXVjTw1rpW3C5QTmH3XMq02diaImcW6qh8Hh3QMO1CQbxLUziA5TiFZUKXRipnsTmWJeRzM6IuQv9yP2uaCgX29fEsx/QJcXz/Mo6tC+F1uw7IEnB7XdvBuOQshNg/uhxY2bbNCSecsMv9Y8eOxbKsbpmUEPuqs8FJb9wG35mdZutbUrzxYTOg9rpU2J612xrPgIJBlQE2bEuTzJls2KaoifhwHEhmLNpyFh4X24+UUdiK4g5Al0srtFdQNvV/uJZswxoAjjjjK5Se8g2CJQH6RPyk8zZ1pX7KgwYoRWMySzxjc2ydHw1Yv62NARUBygIGLh3ScQsdjbxt43a58Htc5KxCexPHVuSVg0fXsGyHTN7G69ZZ1dhGOm9T4i28P+VBg88M9nJ0nxDrtqXpVxbgtOGVxZ2CB0r7kTtCCAH70CD0a1/7Gg899NAu9z/yyCNcdNFF3TIpIfZVZ4KTfemyfSB80oG+mbzJ5tYMjckMlSUGAa+r2ARzUEWQeKZwPI3jOMWsXW3YwNzewVPXIOgtZKka4xmypk3OsnHrGsrRsJVC13QMdyEgVYBpF3pWKVz4jz4FV7CU4V//GUecfzlew2BAqZ/SgJeQ4S7u7GvLF/pebWvL0pzMk7UKzTpV4YEwbUXGdAh43Xh0Hdt2cJSGS9fw6BoOHy8/apqGAkp8HizHIeB1gQ4fNaeIpfNoaIR8HoZWBYlnTOIZ+eVOCNGz9rl4fd68eYwfPx4onL23adMmvv71r3co5r7nnnu6Z5ZCdNLBvA1+bzvNYukc89e0sCWWoa/t49/5KOVBLwPLA0T8hXPzvC6d1Y0JygJuPmhI4NJhdXOKLbEMLiCaMdkYtfG5NJoS2ULHdKWhoWE6No6jCmt/mobhUmSScfKZJEZFP1w6VI7/IgPGnYcvVEbA66IsGGBIdQmtKZPNsSyJrAm0B2iFQ4iDhotkxqQi6KXE8JC1LDJmoXi9Imjg0jWSOQu/x0Vb3kLXNfweF9b2nlpoUOIr/D3ajqJvxGBYVYiUabOhNU3EX1h2k75RQojeosuB1fLlyxkzZgwAa9euBaCqqoqqqiqWL19eHCc1BqInHMzb4Pe006wpmeGtj6JkTZvKEi+1ET/W9tYIjfEsJb5CO4N4Os+maJoFH23b3j8KvC6doFenLWejHId03iSLjtcNGjqm7aA0hdetk7UUWdMCpZH86F0a/+8eNG+A/t/4JR6/D7fLRThYXti159apCflw6zplfjfvZ02i6TzDqoK43W6yeZvmVJ7FG2I0JnOUB70oVcgkVgQMPLpGwOtCaYXarOP7R0CD1pRJImPSnMyQzNn43IUgOOLzEPK5GVQZpMTnwe3SaU3lC01Afe5eHTALIQ4vXQ6sXnvttf0xDyG6xcG+DX7HDupbYmm2teVYty2FW4fj+kXY2JrGsh18XjeW7bB8axzD7aK6xMtHzW3EMhbpvE0sZRI0XFSGvICLdN4mYzkEPW6i6TzJrAMUdvkFvS4CRiE4S2VMml9/gsTCvwLgqQhiZWLkjFpytsLrsRlQHqA25KOyxKAhkSGds/Fvf4/jOYuIppHIFhp8tu8w9HtcGG4XqVyhSD5guEnnLHQKWS0HjeoSgz5hH+tb0lSWGFiOw3H9SikLeon43XzUnKYhkQGl8Lp0ktlCi4aeDpgP1n5pQoj941M1CBWiN9olOPmUXbYPtB13mn2wNcHKrQk8usbmaJqt8QwbWtMMrQqQzBbO17Mshw8b24hmTGrCfsr9LuJpi2TOosRwFRpubq/byloO9vZDkw2Xhq2gLWeRzNsQq6f+L3eS2VroT1dy/HlUnvFNggE/DpCzFDnLoaLES5tpQQq2RDM0JnMEtu/qc2nQ3JajLWfhc7s4oqqE1nQe2yk8T9DrJpW38Lo1sqZN1nIY0aeUPqWFgvtoxiRkuDlpaBnJnEnf0gABbyEgKwt6aE7maEhkCHjduDQd03JY35LqsYBZ2i0IIXYmgZU4JB3s2+A1TcO0Hd6vTxDNmFSVeIlnLXKmTX08y6bWNAGPiz6lBg3xPMmsSSTgIexzg1Zo5pnJK1ozFgGXTjxnUuJ1bT8tQeHRdbxunXTeJms6tC17hZa//TfKzKH7QlScN4vAkRPQAUspfB43Lr1wAPN7W+IYuk5p0EvQKOzoUygypk3eUhxRWULKtKgsMQj53GyOZlAKommTlGljK4dYFmrDfiJ+NwMrAnjdOmG/m8qgj5H9IhxRHeT1Vdt4d1MUpaA1nceyC60X8pZiW1ua2ogP07apCfkZUh3A49I7dUh7d+mOZq5CiEOPBFbikHUwb4Nv78eVyVuEDDdbYlkyeQvb0Sgx3DQn8yQyJtF0DluB26VhWoqmRBaf24VS4CgNzVFsy+ZImw55s9A13XYUjqaRMwuBkoOibcUbKDOHMWAUlVOuwh2qBMAGNKewmzBraphWoQaqJmRQE/KSyDloGvg9bgZWGAS9bkIBD3pOI2R4MC2HkM/DkTUhtrXl2BrLkDELwdxZR1dzyrDKPfacqo0YbF2WoTVt0rfURyjgJZ232JLJEvZ7OGloYY7xdJ6312XxumIHLFt0MPdLE0LsXxJYCdELtffjGlDuZ3VTG02JLBqF7JHhdlEXMdiayJHOWTgK/B6NpDJJ5QrLfHnbQtdc9In4iOsaOTuP5TjkTYWug4ZDylQoTUPTdComf4/0B/8gdMLn0fSOrSosB5qSedwuHUuBshSmBbG0hdulkc0X6qZG9y/F63YRTecByFs28axJn0iAulIfdaU+hlaVEM/kyZoOZxxVTXmJseuLpxC4NMRz1JUFqI0oWtN5opk8bl3nmD5hMqbNOxtj9IkY1IT9BzxbdDD3SxNC7F8SWAnRC2XyFltjWba5NJoT2e3BiMLn0ci5HDStUDdlK4XlgJ1XgIOuaTjbT6kqMRzylk0mb+FxaYVzAAHLson/62mcbJLySd8FwB2qJHzil/Y4H9MpdEH36qB0DQ3FxmgGUIX2CI5iZWOSIZVB0BRel4t1LWn6lfkZWOFHY3tGx+uiOekwrDq014CjPXAZUhkkYBRqrMztLRiCXp1F62NsiqcZ1S+yw0HMBy5b1Jl+adL+QYjDkwRWQnTBvuwA6+rXrKiP8//eqeeND5uIp/NEMyaOUvjcOqbtwlE2edMmZzqgFTJU9vZO6e1BlQZk8g6NiRyWrVAoLAVmrIGW5+8mV78SgODISRi1R3T69esuHbeuYSmF363jcrkp9bu3z0GxurGNkN/DiLoAhttF2O/BrevYjurSzswdAxeNwvJnu7acRTJnEtge0O3oQGWLDuZ+aUKI/UsCKyE6aV92gHX1a1bUx/n9/HXUxzM4SpG1HDSlCmf1OQqvpgGKvO1gQSGq2s61/TPccrYvBzpgmw464ABt779O88sPovJpNCNIxdmXdimogkLg5tF1cpZDyPAQ9LnImA5lAQ9H1ZSwZlshyzTj5IFYtuL9+uQ+7czcW+Bi2g5Z08bvKXRu39mByBYdzP3ShBD7lwRWQnTCvuwA6+zXtGe0MnmL//dOPS1tWXRNKxyM7Dg4FM7uy5kKRxVaF9gdEzVoFA5K9rjAcQqBFNvvs3Jptv3tIdqWF3rQGX2PofL8q3FHarr0HmiAz60XzsZxFLGMidcNbVmbsN9FczpP34iPihIvuqZTE/FSHfbt087MvQUubl0jk7epKjEIGrsuxR2IbNHB3i9NCLH/SGAlxCfYlx1gSimWbYnTEM9QG/GhAF3f9WuOVR9ndeqjGV5d2Yhl27SmLRylyNsKZ4dAyt7DUXiKQvCVtz4OqqCQYWr4nx+R2/ohaDqlJ/0n4ZO+ukuBemd4XIVjZbxuDY9bI2c5NCTyKBSJjIXPY1EZ8HU4i3Ffd2buLXBpbsvRJ+LfbX3TgcwWHez90oQQ+4cEVkJ8gn3ZAba6sY3XVjaRtRy2xDO4NI2g4aE2bODzuPC4NJZsjLK+pQ3bKez02xRL0xDPkrNsbKcQiBXO8+vcPLefcdyRphGecAHRvz1C9dRr8PU7ZpdsV2d4XRDyuXFUob+WS9cwLZuI38uAigBH9wnj0jSaU3limUIriNrIpwss9hS4HFEV4rNHVLJ8S6LHs0UHe780IUT3k8BKiE/QmR1gzW1ZmhKFup5ExuT1D5toSGQZWB7AchRbYxne2RgjY9oEvS4iATfJrM3AiiBj+kdY29zGhm2p7Ue0FLJOanvqqauVQlaiGSvWgG/ASAACw8bjHzwGze3dp6DKAwS8btwuF6V+N/GMSTpv43brDKwMcFTt9p15SuHRQdddbIqmGVZT8qkDjL0FLpUlRq/IFh3M/dKEEN1PAishPsEn7QBrTGRZvy2N7TTh0mH9tjTJnEXYcJExbTa0pGlKZklmTbJ5h7zpJpO3ydo2FQEP/1rTiuXYxNImCgq9othN9qkTUqv+Revc+wGNPv91P+5QZaHRgXvfP/hdLvC6dXxunbxdKI7XtMJOvbpIAL/XRc60iWXylPi8DK0KUB/LdNuuvD0FLpItEkL0RhJYCfEJ9lZIHU3lWbiuhYDhoSZsYDuwykyiHIdY1ia6LU0iZ5E1bXRNozTgwXQUGdMiZytSOYtkLkvWtImm82iofQqonHyW6Ku/oe29lwHw1g5D2RZdCTF07eObZYMOoEHQ5wYUqbyNz62oCfnQdehX5sfw6LSkcrh1nT6RAAMr/JQYHjbH0gekh5Nki4QQvY0EVkJ8gh0Lqddta6PE8KDrhYab722OodA4cVApJYaHaDqPrmv0Kw2wujlJU1sOAMd2cLtdOBR2rbl1Dct2WN2cwrYdMvlC807YvsOvC/PLN66l+bmfY7VuBjTC479M6WcvQnN5PvFxtB3+q1HYUahpheVHlw4hw01F0ItLd5E1LdKmg8cFAyqDjB1QRtjv/bhxp1HoOZXKWdLDSQhx2JLASohOqA77GNE3zMvvN7B0S4KcZaNTCJJG9Yug0EhmTXJ5m7zlkMxbhHxurO2F3m15B91ysG0H04GAR0cDkllrt60TOkMpRXLRs0TfeBwcC1dJORVTrsY/8Lguvbb2ZUetfefh9majIcPN4KoS0CCVs1CAR9fweVycMKCMrOUQ9LrQtI9/jEgPJyHE4U4CKyG227lDesTvJp6xigXpy7bECRpuju8XwVGFPlWLNkRZuC7GqoY2crYDSpE1HfK2wqMrMnmbkM+N36PjKIeMqVCOAnS8rl37UUHns1WapmFGt4Bj4R82norzZuHyh7v2mnfz/7oGtSEvQ6pKcLkKXdMNl07AcBPyucnmbYZWh9gczfT4rjwhhOhtJLASh5x9OXZm5w7pOdMhZ9mFwnW3zvptaRJZi8qgh5ytSGZNtkQzJDMmWZdFazqHx6Xh1nQCXh1NU6xtTpM2bRwUluWQ277W59KhLVs4PHmfXp9tobkK37plZ3wLo9+xBI857VMFMjrg92q4NA23y8UJg8oZUB4kaznYjsKla/jcOmnTZquZoSJocER1Sa/YlSeEEL2JBFbikLKvx87s2CE9Z9msaYzRlMxRHTIYVlNCUyLH5liGjS6No/uEsGyF21U48HhzNIujKHYBN20HpRSJXKF4uy3XsYh7X2u6lZUn+trvMFs2U/3VW9A0Hd3jo+TY07v8WDvWcXk08LgK/biGVZWwNZGjMZmnf1mgQ4sJpRQtqTwVQYOqkJfyEkN25QkhxE4ksBKHjH05dqa9q3osnaMq5CNn26xuSmHaNkfVlLApmuHNNS181NyG7ThkXTortybQ9cKRM7GMSdZy0DUw7cLBx/n9sBku37yebc/9HHPbBgCyG5biH3R8lx7DrRcyU+1nCbo08Ooafp8btwZH1YY546hq3tsc551NUTbFslSHDLwunbztEM/ksW2HMQPKijvxZFeeEEJ0JIGVOCTsy7EzUOiqvrIhTixtsqE1QypnsSWWoTzgwe3KE0ubNKdy2I6ixO/Fth3WbUuRMR0sRxU6o1OolUpknX1qlfBJr6vtnReJvvYoysqjB0qp/NzsLgVV7Tv+dA3cug5a4agcDfB6dBxHURb2MW5IBWVBg1H9IjQksjjKIZ7O057f0nWN4weU8dlhlZKVEkKIPZDAShwS9uXYGYAtsQwrtibxuXVKA17cukZDIkMya7E1nkPD2d5NHLI5m7ZcnrRpF7I+quOSWncHVXY6TstLvyKzZiEAvsFjqZw8G1ewrFNf3/4ueHTQNQ1LgWU7OKqQuSrxuQkZbhxg7IAy+pb5ATDcLsYMKKNvxE9DMkM6bxPwujiiKsTIfhGpnxJCiL2QwEocEjpz7My2VK5D00qlFB81t2HZDqUho/i1hrvQSbwxmcUFeFw6jlKk8jmSWavQeZx9747eWdueu5PshqXgclN22sWExp6PpnW+N5RGYbkv5HcDGpm8jeUodE3b3jZBx6XDwPIgYwaWoqEV2yUcVRvmtOGVxV2RUj8lhBCdI4GVOCR80rEzGdPCshXxtFkMEmJpk3jaZEB5kGg6h8/jwucpPM6GbWks28bRdKpCHpI5i3j8455T+zOgald2+jfZ9uKvqJw8G2/1kE5/nRuwKTT59Ht0ygNeAl43adMmm7cxbZvSgAEUslYnDS0n5POSylkd2iXoui71U0II0UUSWIlDwt6OnYmlcyxcH8Pr0vjnmiYMt4u+pQGqw17yjsMR1UFW1Ds0JrNE/B4cVOHmKHRXoYiqNZXbpwOMu8Js3UJu64fFXX7emqH0mfGrLmWJXBq4dA0chb196S+ZtdA0DdtWeNwu+pUHOL5/KXURPwHDRTpvszmWlnYJQgjRDSSwEoeEHY+d2bFpZVMyw1sfRdFQjBhcQU3YV9wpuKFFJ2c6VAZdjOwXZn1Lmg3bUjQlcpRvr7fK5C0akxlS7U2o9gOlFKllr9D6t/9G2Taeiv4YtUcUX1dn6YBb19B1DU1TKBt0TadPxEffsgBohVqrwZVBJo+sY1hNCYC0SxBCiG4kgZU4ZFSHfZw2vOrjppVtOdZtS1FiuDhhYHlxWStguKhUBh81p8hbNm6XRnnAAwpsR2E7iqDPhcflJeUpLAva++k8YTvbRuvcB0ivmg+AMWAUrmDpPj2WA9hK4QaUpuPRFRG/m5OGVjKoqgSPq9C8dENLms3RDMNqSqRdghBCdDMJrMQhpTrsKzatbEpksRyHmpCPEl/h3Lp4Js+GlgwtqRzpnEU0lcdoTZHN25T6vZQGPETSHkzLwbQVmZxNznTYH3FVdtNytj3/C+xkM+guSk+ZTvjEL6Lpuy/A3xMdMDw6Lq1Q++V16ViOoizgpTzopSrsoyzwcfC0px2SQgghPj0JrMQhpz0Lk7Mc3C4dv7fwzzyeybNsc4JU3sTr1smaDq1pk0zcImfa9C1VOIDXraPrYDiKDa0m+2MRMPbmHOLznwLl4C7rQ+X538foc+Q+PZauQ2XQQyTgpS1r4XXr5C2H0qCHUr+XiL/jYci72yEphBCie3R+7/Z+cPvtt/OZz3yGUChEdXU1X/jCF1i1alWHMUopbr75Zurq6vD7/Zx22mm8//77HcbkcjmuuOIKKisrCQaDTJ06lc2bN3cYE41GmT59OpFIhEgkwvTp04nFYh3GbNy4kfPPP59gMEhlZSWzZs0in893GLNs2TImTpyI3++nb9++3HLLLSh1IPaIid1RShFN5WmIZ4mm8h3+LnbcKahQbGjJkMqblBhuGmI5WlNZfB6diqCXqhIDG4XXBUdUlWBZivfrE2Ry1n7ZAah7fKAcgiPPos+M+z4xqNL5uC9V+//rgMcFjgPJnEUya+F2aVSFfZT43KA0hlaFCv+/g6xp43XpGO4e/fYXQohDUo9mrN544w0uu+wyPvOZz2BZFj/84Q85++yzWbFiBcFgEIC77rqLe+65h8cee4wjjzySW2+9lUmTJrFq1SpCoRAAs2fP5vnnn2fOnDlUVFRw9dVXM2XKFBYvXozLVVhWmTZtGps3b2bu3LkAXHLJJUyfPp3nn38eANu2mTx5MlVVVcyfP5+Wlha+8Y1voJTi/vvvByCRSDBp0iROP/10Fi1axIcffsiMGTMIBoNcffXVB/rtO+x90rmAO+4UrFQGLakcEb+HhkSOjGni1gvZrIxlETK8NCZyLNkUx7Ydouk8bbnu7aRuZ9tw+QoF46ETpuKpGtSpDup+N/i8bkzLwbJVoeM7heDK69KxUZi2Ip236FcWpDbkRQe8bhdH9ylB4+Ni9PY+VUOrQpQGPHt6SiGEEPtIU70o3dLc3Ex1dTVvvPEGp556Kkop6urqmD17Ntdddx1QyE7V1NRw55138u1vf5t4PE5VVRVPPPEEX/3qVwGor6+nf//+vPjii5xzzjl88MEHHHPMMbz11luMGzcOgLfeeosJEyawcuVKhg8fzksvvcSUKVPYtGkTdXV1AMyZM4cZM2bQ1NREOBzmoYce4vrrr6exsRHDKPQBuuOOO7j//vvZvHlzp3ZTJRIJIpEI8XiccDi8P97Gw8KezgVsSmaJ+L3FcwHbx22OZtgUTRHxe/iwMYntQNDrojTgZX1Ling6T0M8A4BlK0yHbqurcnJpWl95iFz9KvrM+BW61/+JX9OeS/J7NEI+N8mcjdetoxxFW95GKfC6NCJ+D7qmkTULRfhH9wkzYWgFVSU+YhkTR6ld35+Al9OO3PXcRCGEEHvW2c/vXrUWEI/HASgvLwdg3bp1NDQ0cPbZZxfHGIbBxIkTefPNNwFYvHgxpml2GFNXV8eIESOKYxYsWEAkEikGVQDjx48nEol0GDNixIhiUAVwzjnnkMvlWLx4cXHMxIkTi0FV+5j6+nrWr1+/29eUy+VIJBIdbuLT2fHg5MoSg7ztkMnb+D06lSUGm6Ip3lrbivP/27vzOKnKO9H/n7PWXtV7N80uAoogOmAUNeKSSBzFeGd+ZtEYmWzjuBJ1zOidGxOTiNHEJJOMxuTm6nWSifPLxBi3RDBRcEUHZARURGRtaHqtpWs7p8557h9FVygakKUB6f6+X69+QZ3zVNXz7W6tL8/yfXy/slPw2KYIrqfYlipQcH2itoGmaWxN5tnWm2NLMo/jlWtXOYOYVBW3rmHbQ9eTXf0cpWQ7hU1v7rW9qZcTpsa4RW3YpORDulgublp0SmRdb8cUoIZplmOIBk2aa0JMbI5RHwty/pQW/mbGKOZOb2VCY4x0wWVLMke64DKhMSZJlRBCHEIfmsXrSiluvPFGzjzzTKZOnQpAe3s7AM3NzVVtm5ub2bhxY6WNbdvU1tYOaNP//Pb2dpqamga8Z1NTU1WbXd+ntrYW27ar2owbN27A+/TfGz9+/ID3WLBgAd/85jc/+Bsg9tmuByeXvPIOPsfzsXUNBazvyoGmOO2YepriQS46cQQojTe39OKVfHqyDp7yKew4n8YrKTzKFcsHYwhX+R7pV/+T5Iu/AuVjxJtomHszwVFT9vgcHWiM2ph6ufZUW6qAp8DzfJQGJaXwvHLyFQ1aNMcD1EZsTE1HKcXI2hC9OQfX23E+4k47JKVOlRBCHB4fmsTq2muv5c033+TFF18ccG/XDwKl1Ad+OOzaZnftB6NN/0zqnvpz6623cuONN1Yep9NpRo8evde+i73b9eBkV9PYmsqSyjvUBG3GNUTIFF3e68hScP3KtOCxzRFeeq+DVdtSZAolDE3DKfloGqgdh/8NxkhVKd1J15Pfp7h5FQDh48+i/vyr0Xesr9oTXQPX89GBtlSBkudz2jF1uK7i/e4svdkieTxiQYvmWICxdWFAI5V3qQnbBE19R1b4l9RQ6lQJIcTh9aFIrK677joef/xxlixZwqhRoyrXW1pagPJo0IgRIyrXOzo6KiNFLS0tOI5Db29v1ahVR0cHp59+eqXN9u3bB7xvZ2dn1essXbq06n5vby+u61a16R+92vl9YOCoWr9AIFA1dSgOjlKKdR0Zck6JmB3A9322pfJkCi7xgEmh5NOeKtAQsxnfEKarr8iqtjQnKMUr67ppTxfQ0NAUOL6Ph0JTDOpxNb1//gXFzavQ7BB1H7+KyAnnfuA/BHTK5/plHR9la0QMjeaaICMSYSIBg+aaIGs7Mry3vQ9DL38fco6P6/kETYOmqE131qE+GqAxJr9vQghxpBzRNVZKKa699loeffRR/vznPw+YShs/fjwtLS0sWrSocs1xHBYvXlxJmmbMmIFlWVVttm3bxqpVqyptZs2aRSqV4rXXXqu0Wbp0KalUqqrNqlWr2LZtW6XNwoULCQQCzJgxo9JmyZIlVSUYFi5cSGtr64ApwuFgb6UODpW12/t4fUMvRdfjv9tSLFnbxesbe9jck2NtR5a2ZI53tqfRKK89aooFaevN8er7PazZnsEydEKWQSRoErR0NMrlCgZT7ce+QuiYmYyY9yOiU8/bp6k3UyvXo9JQ1IVsGmI20aBVPvcPjbpIgBNGJBhZE6ZYUnRnHfoKLomQRUsiQKboUvJhxthaGaESQogj6IjuCrz66qv593//d37/+98zefLkyvVEIkEoVN459d3vfpcFCxbw4IMPMnHiRO68806ef/75qnIL//AP/8CTTz7JQw89RF1dHTfffDPd3d1V5RYuuOACtm7dygMPPACUyy2MHTu2qtzCSSedRHNzM/fccw89PT3MmzePSy65pFJuIZVKMXnyZM4991xuu+021q5dy7x58/j617++z+UWhsquwA8qdXCo3vP3K7aydH030YDOyi0ZurMFiiVF0NIJ2yYF18P3FTPH1fOJqc1EA+VdgN1Zh9VtSbqzRbb2FnD9v/zaH+xoVbH9PfLrXqfmjM8e0PM1wNixaN0yDSY1RsiXfOqjQY5tihAyDdiRnKXzDks39ICC8fURoiED0NE1jcnNUeZOb5WF6UIIcQjs6+f3EZ0KvP/++wE4++yzq64/+OCDzJs3D4BbbrmFfD7P1VdfTW9vL6eeeioLFy6sJFUAP/jBDzBNk0996lPk83nOO+88HnrooUpSBfCrX/2K66+/vrJ78OKLL+YnP/lJ5b5hGDz11FNcffXVnHHGGYRCIS677DK+973vVdokEgkWLVrENddcw8yZM6mtreXGG2+sWkM1HOyp1MG6zgxdfcXKmqbB1L8TMOeUaIoG6MoW0XUIWibg4fmKYsknHjTxfEgXXDZ055jQGCHnlHinPcW2VAHP98u7/walTz6Z1x+jd/HD4JewG8cRnjRrn54bNEDTdDy/vHhe+eXcKWxptKcL2KZJZ18BpXzioXIB00jAxDZ0RsSDTGqOURexybs+Ydvg2KYo00YmJKkSQogj7ENVx2o4ONpHrJRSPPdOJ+s6M4yrj1RNcyml2NCdZUJjjHOOaxzU3We9WYcn39xKLGCyemuaV9d347geQdsgnffIOSUMTREJWURtk6BlUB+xGZEIUXQ9lm/qpiPlknZcCqWD/5Uv9fXQ/dQPKGx4A4DQxNOov+B6jNC+/UxtQyMaMLBNnbzjkXM9DK18TaHRFA9QG7J3jKYpwpbJiESQ7ZkiY+rC/P1Z4wlYpuz2E0KIw+SoGLESR59kzqUtmaMpFhzwQa5p2iE74LdY8nE8n5Bt0pIIoqNRLCkCNsSCBq7nkXM8Ql55rdL2TIHt6SI9WYe8U2J7xiFZcHEHYagq995rdD/9Q/x8Gs0MUHvel4lOn7PPa6kMvbxQ3VfliuklHyKWQSRgUh8NYOjg+dBX9KiLmHhKo6OvQKrgcuKoBJfOHEVLTfjgAxFCCDHoJLES+6U/wQlaxm7vD+YBv0qpSg2mvFPC0jXyTomgaVAftcm5JVzXB13D1DV0TSPvlujtcnAcj1DAIGhqdGSKpPODk1QlX/glqZcfAcBqGk/j3FuwGvZePqN/h4hlaCgU5cXoNmgaJc8nYpu0Jmy6syViQRPT0GlNhNjcm0MpqI/YxIM2YUvns6eMYfKIo2+kUwghhgtJrMR+2flg40hg4K/PYB3wO2BxvK6zqSfH9nSRaMDEKXmUSgpMiJsG2Xz5iBrTNMgVC5SUTybvkyt6lHZUUx8M9oiJAMRmfpLa2VeimXsfldMor50KWjpBy8D1fAquR1/RI2QbhGyD5niQnONhGuVRrFjQoj5iEwkYJHMu00YmiAYtkjmHRFh2/AkhxIeZJFZiv+x8sPE4e+Aaq8E44Hd3i+O3pwts6M7RkS7Qkggxui6M4yk29uTY2ls+4y9g6vQ5BRyvPEoUDel4JQ6qDIRSilKyHau2XEctfOypjPjifdgNY/bp+f0HJddGbGIBk+6sg6Zp1EdtfAUhU8ctKSxDx9B9LN2gMRoATcM2DQyjRDhgYuoatnnwCasQQohDSxIrsV80TWPqyDhdfUU2dGd3e8Dv1JHxA15I3b/7L5V3KovjlVJ09hWpDVsETR1N09A0jZqwxdakhuP5GLqG6/m4Xvl1fCCdP7hhKi+XovvpH1Jse5sRf/cTzHgDwB6TKm3HV/+7GoBpQEPUJhqwKHk7FpqHLOrCNp5SZAolgihGxEPUR8vFSsN2eZrVLfmYuo6pa4OSsAohhDj0JLES+63/YOP+qbqubBHb0JnQGDvoOla7Lo5XKDoyRbb05IgEDAKmTrboMa4+hG3obE3myRZdoJxYDdYW1/z6N+h+6l68bC8YJs62dyuJ1Z4oyuf4mYCha0QDFkW3RKbgknU8dE0RsEzG1oY5fWIDM8bWsrotQ1uqvJOy5ClWb03T0VckETDpzTvUR4J09hWpOciEVQghxOEhiZU4IIfqgN+dF8en8g4bu/Ns7s3ybkdmxxF45UP9iiWfzr4CyZyLruvlNVeDUEJdeS7JJf9G+rVHAbDqR9Nw8T9iNx2zT8/XNaiL2NRGbFJ5F103QUGx5OFroHuKoq+Y2BzjxNG1tCRCPL+mk66+Ik2xIFNGxHmvM8PGnhyWUf6eHjsICasQQojDQxIrccAOxQG//YvjOzJ51nXkyDouOuXDkst5kyof86JgS2+BTN7B0DUKrs/BbkR0u7fQ9cQ9ONvXARA9+a+pPecL6Na+JzTxgElNyAI0EiEbQ4OSUgRNk9qwRU3YIpUv8ae3O5jYFKU5Eaoa/XM8n1G1Iaa21nBMU5iRNWGpUSWEEEcRSazEEbdzWQXb0BiRCLLwrQ5836M5FmRDdw5D1zHYUZ5cK9d1KjglSr6PUjqez0FPA2beeBpn+zr0YIz6v76B8MTT9uv5pgZB2yIcMCmWFA1Rm/iO9VSmoRO0dECjLlxiczLP0vd7mXtS8JCN/gkhhDj8JLESR9TuzhxEQSbvYFsGqUKJTLFEbdiis69IyS0nHus7+8i7Hp4qTx8OxtqqmtlXojyHxOmfwYztfT3VrmwDWmIBZo6roc/x2ZbM0ZN1GF0XJhqsXnBumwZhy2Bzb7ZSSPVQjP4JIYQ4/GTvtjhiOtIFnlvTwaq2JL5S1IRsYkGTzj6Hku9TF7LJOiXSeQeUojZk4QOdGYe+QgnfV3jegY9UFTavouvJ76P88lZC3QpQP+fa/U6qyiUVDMY0RGitDTO6NoSvIJl32d2Yk+P5hCwTf8daMSGEEEOHjFiJI0IpxUvvdbFsYxJTh7ZUHlPXqY8EqItYGLqOaWhMbIzSniqQK7p0ZBzSBRdfgQegOKDDlJVXIvXSr0m9+htQPvaIScRnzD3gWEwdEmGLMXURQCNoGdRGbLp2HKcTD/1lJEopRargUhuySAQtqUslhBBDjCRW4ohYu72PJe924itFIhbEMnXcks+2VI5U3qQharO2I0NTLEDJ9+nKOvQVSyilKLrqgBIqADfZXl6gvnUNAJFpHyM67WMHFUs0YDChMUrdjqroGhAPmsSDJlvTBRIhG9s0cDyfVMElbBkELYNRtRGpSyWEEEOMJFbisFNK8eaWJD1Zh7H1YRTlMgUBy6DZDLI9UyBiG6TyLj1Zh2LJI1sokXPKa6oOVN/q5+hZeB/KyaMFItTPuYbI8Wft9+uUCz6URSyN+miAUbUhfMB1PZL58tEzp0+oxykpNifzhC2DkGVSG7LKSVVdWOpSCSHEECSJlTjs1m7vY+n73aSKJd7dnkbXdMIBi6aYTV3YxjIM3t6aJplzcT2fdLGEUnAQJ9NUHZ4cGDmFhrk3YyaaDui1+rsRtXTGNoSZNrIGTdPozhYxdZ2WeAhd15g+qoYpI6K8tj7J5t4sPopE0GJUbUTqUgkhxBAliZU4rDrSBRa/20lPziFm67Sni7iuh+Mr3rdMRtYEyLs+21MFFGDpYGmQLXkHPP0HEJ58BunXHyN+6t+SmPUpNN3Yr+drgKGV/0SHpmiAi6e3Uh8N4APRgImuafhK0VcsVSqlN8WDzD0pJKUUhBBimJDEShw2/ecA5pwStmGwvrOPzj4XH4WhQTrv0J7KoWkalqHjej6GBo6ncLz9fC/fo7j1XYKjjgfAbhrPyKt+gRFO7Nfr6EDI0vApl0mI2AbTRsa58vTxzJrQQGemWCkXkfc8bEMfUCldSikIIcTwIYmVOGz6zwFMBE029vTRnXUJW3p5bZLnky8pXB80FKbu4/kHtuuvlO6k68nvU2x7m5bP3UNgxCSA/UqqbB0sU8c2NOIBk1xJ0RwLMLk5yldmH8vkEXHg0B3tI4QQ4ugkiZU4bIoln2LJY2tvgb6CR8A0ME0NU9PoU6qyeEkBpQOspJ5950V6/vhj/GIWzQ7hZbpgR2K1PxQQNA1qwxbRkMXYkEVD2GZsY4REuHr0SUakhBBC9JPEShxy/UfWpHIOmUKJzT05dE2jIWqTdz3SBZdkzq3a8be/SZXvFOj908/oe3MhAPaIiTTM/Ues2tYD7nfINvDRiAcsxtdHSOddDE2T2lNCCCH2SBIrMah2PvcvYOo4JY/VWzPlI2tKPpt7cmzqyeL6PpmiT9Yp0VcoHVQZhWL7e3Q9cQ+lnjZAIz7rUmrOuAzNOLBfb0MrfzklnxGJIMc0RjB1jZzrMVpqTwkhhNgLSazEoNn13L+i69OZKRILmkxojBK0DJySx+vv99DVVwDKZyr7B3nQX3HLako9bRjRehrm3kRwzIkH9Do65TP/NE3DNnXCtsExjVFMXWN9d44x9WFOPaZW1k8JIYTYI0msxEHpH6FqS+Z4fX0vrufRHA8RsHSWb0yyqSfHqNoQrufTky2yamuafKlEcT93+e3uffsTnNiMi1FukehJn8AIxQ/o9QwgZGsoNHQ0GqNBApZWnsLMu4ypC3PpjFE0J0IH13EhhBBDmiRW4oBVRqh6c6zelqY35zCpKUZdxEcBWafEuLoQnTmHF9Z2srU3R9bxMHUdgx3n/R2A3HuvkX7l/6fp099Ct0NomkZi1qcOOA4NsEwImiaWqROyDcIBHU3TaU2EOHlsDaeOr5OkSgghxAeSxEockI50gefXdJLKO0QCJpoGzbEA7ZkCmUKJ0XUhMgWXXk/Rm3NY19mHAhrCFjm3VH0uzD7y3SLJ5/8PmeVPAZBe+ig1H738oGMJWxpB2yQRshhZE2JcfZiOPgdbNxhTH5KkSgghxD6TxErst/5Cn6m8w7j6CMm8i68U0aBNDOjoK7KuM8OW3jxOyaPo+mTyLpGAyfaMQ9bx93uxutO5ga7H78bt2gRA7JRLDmqUSgcCpoahgW0bxAImE5uj1EcC5Es+LYkQU1tj9OZcVm/N0BQPytoqIYQQH0gSK7Hf+tdUNcWClSrppqaTzrv4PmTzLu9395EteGQdD18pfKWwDA3PV3j7UfVTKUVm+ZP0Pvd/wHPRIzU0/PVXCR0z44D6rlNeMF8Xtjh1fB2dWYe849GcCBIPWaBpjEiEGVsfIhEqn1vYlsyRzLlSq0oIIcQHksRK7LdiycfxfIJW+by9kufTkSmyrrOPku+TzjvkXZ+wbaDDjsQKkrkS+zv/l3rlP0i98EsAQsfMpP6v52NEag647yFLR9c04iGLvOsztbWGaMBgTH1kR/KnEwkYaOVTAQlaBl3ZIsXSwZxUKIQQYriQxErst4CpYxs6Bdej5Pu8tr6HjnQet+SRKZTIOR4lH5KlEpYOpqGjaeVja/Z3CjA2fQ59K54h/pH/QWzG3IOejquPWNSGbUIBg+mja5hzQguvbejB1DUigYH1qQpu+fw/KQoqhBBiX8inhdhvNWGLkTVhtmcKvN/Rx8aeLOwoU1BwSyjA0Mu/XCWfymhPSX3weJUquWTffqHy2IjUMvIrDxCfefFBJ1WmBqZpEAtZTGiMMeeEFhpjAcK2ycbuLL6qHpVSStGRKTCyJixFQYUQQuwTGbEapnatkJ4ImaTypX06SFjTNKaOjLOqLckL67pI5Vwcz6cvX8L3IWhraGj0Ff3yIcoKvH2oreB2b6HriXtwtq8DIHL8R8vvZx782iYdCFg6xzRFGF8f5bjmGFuSeV7b0EN3tsjGrhxbkgWmjYzRFAtRcD06MgUSYZupI+OycF0IIcQ+kcRqGNpdhfRiySNg6gQsA9vQGVkTZurIOE3x4B5fx1NQ8hWer3Bcj5Lv4QNFV+Gj2NdVSUop+t5cSO+ffoZyi+ihOJoVGJRY+5k6jIgHGF8XoTkeJJl36ck5NMWCNMWC1IUDrN6aYvnGJOMaXOoiNhMaYx/4PRBCCCF2JonVMLNz/ammWJBiyeO97Uk6MkWaYgFOHl1LwNJZ15mhq6/I2ZMbByQW/eUWNBQjE0EyOZdssVxNXcF+rU/38hl6/vhjcu++DEBw7HTqL7wRM1Y/aDFbOjTEApw4uoZTx9eTKZbo6isyrj5SGYlqrQnREg/wdnuGUTVhzjmukdqILSNVQggh9oskVsPIrvWn0GBdZ5aS73Ncc5SOrMPmZI4TRyYYVx9hQ3eWVW1pzokFqhKM3my5kvr7nRneac/QmXH2eXRqZ4Utq+l6/B68TBfoBjVnfZ74R/4HmjZ4S/9CFkwdUcPpExu5cNoIGqI2T63cVikVsTNd1xlXHyFdcNE0TZIqIYQQ++2ILl5fsmQJc+fOpbW1FU3TeOyxx6ruK6X4xje+QWtrK6FQiLPPPpvVq1dXtSkWi1x33XU0NDQQiUS4+OKL2bJlS1Wb3t5errjiChKJBIlEgiuuuIJkMlnVZtOmTcydO5dIJEJDQwPXX389juNUtVm5ciWzZ88mFAoxcuRI7rjjDpQ6yBOED6Nd609lix7d2SI1IRtN10kELXqyDtmih6ZpNMWClRpO/ban8vxs8TqeenMrb2xO0t13YEkVgF/M4WW6MGtH0PK575E49W8POqnSgIAOsYDOsQ1h/mpMLZ/+yBguP3UMk1piOJ6qKhWxq6BllKdHpbyCEEKIA3BER6yy2SzTp0/n7/7u7/jbv/3bAffvvvtu7r33Xh566CEmTZrEt7/9bT7+8Y+zZs0aYrEYAPPnz+eJJ57gkUceob6+nptuuomLLrqIZcuWYRjlD8/LLruMLVu28Mc//hGAr3zlK1xxxRU88cQTAHiex4UXXkhjYyMvvvgi3d3dXHnllSil+PGPfwxAOp3m4x//OOeccw6vv/467777LvPmzSMSiXDTTTcdjm/XQdu1/pTr+ZR8H8ss73izDZ1MwcX1y0nFrjWc3tqa4qEX1/Pcu50kcw6ez34nVcpz0Yzy+4UnnELD3H8kdOxH0O2DPzImaulMaokxsTnK2LowYdui6Hmce1wTddHymq2dS0VEAgN//aW8ghBCiIOhqQ/JkIumafzud7/jkksuAcqjVa2trcyfP5+vfe1rQHl0qrm5me9+97v8/d//PalUisbGRv7t3/6NT3/60wBs3bqV0aNH8/TTTzNnzhzefvttpkyZwquvvsqpp54KwKuvvsqsWbN45513mDx5Mn/4wx+46KKL2Lx5M62trQA88sgjzJs3j46ODuLxOPfffz+33nor27dvJxAof0jfdddd/PjHP2bLli37PG2UTqdJJBKkUini8fhgfgs/UG/W4ck3txIPWkQCJn3FEq9v6CFsGQQsg4LrkXc9ThlbRzRoki2WSBdcLjqxFafkce+id3llXQ/d2QKO6+9T+YSd9a1+juSSh2m5/LuY8aZBja0+YjK+PsLY+iitNSHG1IVJ5h0mNMY457jGys9HKcVz73SyrjNTtcaq/96G7uyA5wghhBD7+vn9of1n+fr162lvb+f888+vXAsEAsyePZuXXy4vdF62bBmu61a1aW1tZerUqZU2r7zyColEopJUAZx22mkkEomqNlOnTq0kVQBz5syhWCyybNmySpvZs2dXkqr+Nlu3bmXDhg17jKNYLJJOp6u+jpT++lMdmQJKKSIBg/pIgGTeIee4tKcKhG2DcECvquGUCJm8+n43b21L4/rlFeqG0V+b/IP5xRxdT3yP7ie/j5fuJP367wctJh0IWzp1kSDHNsdJhEzWdWb44+pt5F2PE1pjVQlSf6mIRMhmQ3eWbLGE5yuyxRIburNSXkEIIcRB+dAmVu3t7QA0NzdXXW9ubq7ca29vx7Ztamtr99qmqWng6EhTU1NVm13fp7a2Ftu299qm/3F/m91ZsGBBZW1XIpFg9OjRew/8ENo1qcgVPRJBg+3pIi+9183WVJ7OTJH/2pBkZVsK09Bpitts7M7x7vYMmXwJ29AwTA2lFPuSexTb3mHbg9eRfet50HQSZ15O7TlfOOhYDMoFP6MBg4lNUT4yrhan5LG2s4+OTJHurMPG7hyr2tJ0pAtVz22KBzl7ciMTGmOkCy5bkjnSBZcJjTHOnjRwF6QQQgixrz70uwJ3HTkof6Dv/RN91za7az8YbfpnUffWn1tvvZUbb7yx8jidTh/R5Ko/qVjVluad9jRvbUujazC+PkzIsgDFu9vTmLrOpJYoS9f3UHA83tmWoVAqEQ+a4Cscb+/TgMr3SL36G1Iv/jsoHyPRTMNFNxMcdfxB9d/QIBrQUQp8BWPrI3x0UiNj68Is25QkEbIZV29hGRqZfIlVW1N0Z50BZSOa4kHOiQWqiqTurSiqEEIIsS8+tIlVS0sLUB4NGjFiROV6R0dHZaSopaUFx3Ho7e2tGrXq6Ojg9NNPr7TZvn37gNfv7Oysep2lS5dW3e/t7cV13ao2u45MdXR0AANH1XYWCASqpg8PtV0rqu8uWWiKBzk7atNXKJF3SxzTECViG+Qcn66+AqvaUmxN5elIFzmmIUJwx2LvQsnHybiUPiCpAsgsf7JyeHL4+NnUz7kaPRA56PgMDSxdw/HBNnXGN0RoiJbrT+WdEmPrQmiaju8rsrrHiESQVN7ZbdkITdOojRx8VXchhBCi34d2KnD8+PG0tLSwaNGiyjXHcVi8eHElaZoxYwaWZVW12bZtG6tWraq0mTVrFqlUitdee63SZunSpaRSqao2q1atYtu2bZU2CxcuJBAIMGPGjEqbJUuWVJVgWLhwIa2trYwbN27wvwEHoCNd4Ll3Onnyza08tXIrT765lefe6RwwFQaQypdIFRyObYwRC1rouo7reby6voc3t6TpSBdZur6H373RxrquPsbUh/BKHqm8u0+lCGInXUBg5PHUX3gjDXNvHpSkCsA0NXTDIGQbjK4NEwuZrGpLsWJzks4+h409eXJOCcfzMQ0d2zB2WzZCCCGEOBSO6IhVX18f7733XuXx+vXrWbFiBXV1dYwZM4b58+dz5513MnHiRCZOnMidd95JOBzmsssuAyCRSPDFL36Rm266ifr6eurq6rj55puZNm0aH/vYxwA4/vjj+cQnPsGXv/xlHnjgAaBcbuGiiy5i8uTJAJx//vlMmTKFK664gnvuuYeenh5uvvlmvvzlL1dW/l922WV885vfZN68edx2222sXbuWO++8k69//esfiumjXSuqB3fs8ttTBfVdSy8kcw4vru3m3e0ZTF0nHrIoOB45x2P5xl5KPjieYk9H/vlOgcwbTxI/5X+g6QaaadN8+d2D8r3RANuAuojFCa01ZIolmuMBgqZBe6pIU9QmZBqEbYPebJFcoUQ0aDK2IUIkYOArqspGCCGEEIfKEU2s/uu//otzzjmn8rh/LdKVV17JQw89xC233EI+n+fqq6+mt7eXU089lYULF1ZqWAH84Ac/wDRNPvWpT5HP5znvvPN46KGHKjWsAH71q19x/fXXV3YPXnzxxfzkJz+p3DcMg6eeeoqrr76aM844g1AoxGWXXcb3vve9SptEIsGiRYu45pprmDlzJrW1tdx4441V66eOlF0rqvcnM5GAyTh79xXUd67nFLYNNnRn2ZrOY6CRCJkopWGZiljQYmN3lnTB3eP0X7H9PbqeuIdSTxvKK1Fz+meAva8921f9U3+2qdMYC2HoGlNaEoxtCNOTdQjZJsmci9IUvoKIbdKeLqDpGmPqwmiaRsEpSW0qIYQQh8WHpo7VcHEo6ljtWp9qVzvXo+pfU7RzPaeGaICX3utiY0+OTN4laBkUSx7RgIVCsTWZp+D6pHJu1YiVUj7p1x4jueRh8EsY0Xoa5t5EcMyJgxKXqUHA0jF1jZqwzVmTmzhlbC2Tm2O8tqGHeNDC9Xw2dud4sy1JMueSCFmELJ1IwOKsSY1EdiSNUptKCCHEwdjXz+8P7eJ1se92ndbb1a4V1OEvpRe6+oq839VHX6GEqWsELZ1kziEesgjZBtszhfK5eaiqKuulvh66n7yXwsYVAIQnnU7dJ67DCMU4WBqgaxC2DRJhi8ZogOmja/n8rLGMa4iwPV2sxBsJmCRCFg1Rm1VtaYq+R33Yps8pkco7dGZ8qU0lhBDisJHEagg40GNa+ksvvLrOYE17H7kdU2bxkE3QKrf1fJ+S51PyFLoGnoL8xv+m6/ffxc+n0awAted+mej0OYOWuARMjVjQxDYMGqIBjm2OcubEBsY1lKc5d41X0zRG1oaJBEw29uRoT+bpK5YouD4Tm2JMHRmX2lRCCCEOC0mshoBEyCQRtFnbmeaYhijRHckGUKmgPqExRk3YGvDcpniQi6a3oPB59u0ONE2jLmzRlXXo6SvilsAt+WgG5YKgCoxwAt/JYzUdQ+PF/4hVv+91ufTyS1T+9CmPUPXPR5s61EdtLF3H8RQR22Ryc5xpIxOVmPoryK/rzDDOjux03SYeNDF1jVE1Yc45rpHaiC0jVUIIIQ4bSayOch3pAqva0mxJ5ljflePd7X2MrQtzbGOMgKXTkSl84FSYruvMmtBAZ8ZhxeZeevMuTdFykpIquCSzimI2hW9FAbAbx9H8mW8TaJmEZg5M1vZEAwydSnFPTQNDQX3UIut4mLpOOGCgazoKjaa4zSemtnD+CS1VI047T2Nu6M5W7YLsyBQYURPio5MaKgcvCyGEEIeLJFZHsZ1LLIysCdEQtXmvI8umnixbUwWmjIhxXEtit1Nh/YVEC65HwfUIWgZnTWqkNmKxfGMvG3ty5B0Pz/Ppfv1xtv7pIUZ8+ttYreUSFcFRJ+xXX/tTOl3TsC2Nouvj+mAbGqahMaU1zlkTG4kFLHJOid6cy9SRceZOb0XXB+7m27mCfFsyR1e2iG3oTGiUqT8hhBBHjiRWR6ndlVgIBwyOH6HTmgjSlswzsibE2ZMbqhITpRRrt/exsi3F+q4snZkCWccjGjAYUxemORZkckuM2rDNsjUbeP3n36Tn7VcByK56ltrWyR9Ydb2fSbmgp+8rSgp0HXyl8HwNy9CIh0ya40EaYwHOmdxESyJEwfVwfZ+JLTFmTWjYbVLVT46lEUII8WEjidVRKplzaUvmaIoF0TSNVN5hY3ee7myRku/j+fD6hl6OH5FgUkt5p15HusCLa7t4YW0n3dkiuaJHwNIZkQhT8hTvbu9j+cZeAqaBt/m/eeyuWyiku9EMi/F//fc0nnYxyaxLwfvg1Cqgl4+c8YHGaIC8U6Lg+Tiuj65BSyLEGcc2MH1ULZoGfcUSW5K5/R51kmNphBBCfJhIYnWU2rnEQirvsHJLmqzjUhOysUyLouuxsSfH4nc7K4vWn3ung+WbevGBqG1RdMsFFHqzDqNrg6ztylLIF3nriQd4Z9GvAQg2jmXc//c14iMnommg9jJepfOXxei1ERtd0/CUorU2RGPUprPPpTke4LzjGpk5to5wwKr0TUadhBBCDAWSWB2l+ksO5N0SG7vzZB2X5lhwx9a98khOUyxIzimxcksKNGhPFzB0nXhQZ1MuRyJkYxoaqYJLW7pI0SnxzssLK0lVy2lzGTXnK3iGTcEtoWlUalr1U5QTqf6F6QGtnFwZhk5d2KIpFuTU8bVkHI8Z4wL89dQWmhOhAfHIqJMQQoihQBKro1R/yYFVbUm6+grUhOxKUqWUIlVwaYkHGVsX5r3ODKBRE7LYmsxjGQaeUpiGVl6bZZmk8y6pQomGk89lwqYVtEw/i+Axp5IuuEQsg3TeIe/42KaG4ymUKhfxNLTycTMaCtcvJ1eaopz0OR7JvMvGnjwzxtZx5sQGWVQuhBBiSJPE6ijVX3JgXWcfHZkiY+oMfF/heD6pgkvEMhlbFyZkm+QcD9BoiAYwDR1fKQxNo+QpvGKG1//zp4z7xBcpuDpBy+DML9yOj4/rKdIFl7zjYps6Tklh6hqov9SiigYNNE3DLfkYWnltVyJsc8KIGJGgxei6EL4Pri8HIAshhBj6JLE6ijXFg8ye1MiGrj66MkUU5SnCEYkQY+vD1IRtssUSYdsANAwN6iI27ek80YDJmjeW8uqDd5Dt2U6mr4/Rl3wVA41MsYiuGxi6RtAy6Ml6+MoHfCKBAJZh0Jtz0LXyiJela/gGFEvlBetTWmKMb4oxtq7cB6XUbg+CFkIIIYYaSayOcomQSX3UZmuqQMQu/zj7F5j3V10/tjEGGrzf2ceY2jDJTJ7nf3M///X4/wGliDaNYsLsvyEcsii4Pt19JeoiGvFggLH1YUxdozfvUPI1dE3n+BERAqZGW7JIMu/geT6GptMYszhnciMzxtYTCRiVBKp/vVdbMkcy58p6KiGEEEOWJFZHsY50gcXvdqFpOi3xII7nEbYN2lMFuvscGmMBRtWFmTYqAUB3n8O7763l19++iXVvrQBgzGkXMvVvr2PCyEZMTWNLMo9SipBtUix5GJrG8SNiFEserq+RCJqMbwgzviGKbWi815llSzJH2DaIBy1OaK3B0AeOSO3uIGghhBBiqJHE6ii1c4HQqa1x0nWhSh0rXYfenENTPMDsnRaM69tW8Z0vXUo+lyUUjXHpDXdw1pyLOaYhzOSWOFuTBX65dAO+r0iEbDRdQ/mKvFuiMR5iQmOYTMFjVE2YVMHB8XxqIzZTR9YwqjbE0vXd+30QtBBCCDGUSGJ1lNq1QGgiZDNtlEW26OF6Pq7n43kK2zQqzznrtBnEY1FOmDadL/7zvWixehSKLckCCp2muM34hihuSZWn/krl8/taayKMrQ8RDVg4Xo5TxtcStIyqulMAW3rzAw5Ghg8+CFoIIYQYKiSxOkrtXCC0n4ZGdMdokecrtiRzvPPuWmadXD7Xr6mpicf+8CwbilEyjld1ePG6zgwbu3UsQ2dsfRCI4no+lqGX10uhkS2WsI3yzsHdrZPa28HIH3QQtBBCCDEUyLzMUaq/QGjB9XZ7P1tw+MPD/8pHT5nOr39dLviplCIXbCLjeIyrjxAJmBi6RiRgMq4+gut5FEseHZkiEdugNmwTDZhoaJVRp5E14T2OOvUfjDyhMUa64LIlmSNdcJnQGOPsSY1Sw0oIIcSQJyNWR6n+AqG7m3rbvnUz37zpata++ToAS5Ys4bOf/eyA6cOdaZpGczxEWzKHpev7POqklKo6jqYxFuCc4xrliBohhBDDkiRWR6n+AqG7Tr396enf86/fuoVcX5pINMpP77+fz33uc8Dupw93FrQMApbBKePr6EgXaUvm6MoW93gwcke6wKq2NG3JHI7nYxs6I2vC+3yAshBCCDHUSGJ1FOufelvVlmbd1k4e/sE3efGp3wBw8oyZ/OY/HmHChAmV9jtPH+68c08pRbbokco7lDyf1kSQE1rjex116kgXeH5NJ6m8M2CtVldfkbMny9SfEEKI4UfWWB3lmuJBzjmukcb8Zl586jdomsZtt93G0lderkqq4C/Thx2ZAkqVi4gmcw5vtqV4bUMPL7zXxeaePG9sTtKZKVIbsWlJBKmN2AOm//pLPexurVYq77CqLV15DyGEEGK4kBGrIUDTNC656BN85zvf4YwzzmD27Nl7bLfz9GHANHivo49UwQWlaK0JMbEpwvudfXT3OXscdfqgtVpSZV0IIcRwJYnVEHLbbbd9YJv+6cOVW1I8t6aD9nSBxqhNNGDTUhMgFrRpjuts7M7t8Wy/fVmrJVXWhRBCDEeSWA1DTfEgJ4/ReK+zj5ZEkFTeJVcs8e52B1PPUh8J0Biz9jjqtKe1Wv2kyroQQojhSj75hinHU+UyCukiyayDrkHANNCBbckca7dn6ck6ux112t1arX77Uu9KCCGEGKpkxGqYsg2NzkyRjky5nEJHXxHPVxi6RixgkSoU8JXCNgbWn9pTqQepsi6EEGK4k8RqGMu7JToyBcK2QcS2sCwN11f05IrkHI9YcPdrqKC61MMH1bsSQgghhgtJrIapYsmn5CkMTUNToGmgtPKfmgJD0yh5aq8L0JviQc6JBaTKuhBCCLGDJFZDzK5HzOwp0Sm4Hp6C8Q0R8q5PX8HFU+VEqyYSIGTpFEr+Hs8i7KdpmpRUEEIIIXaQxGoI2Z8jZoKWQTRg4noe4+pCFLxgZY1V0NBozxSIBsw9llQQQgghxECyK3CI6D9iZl1nhnjQYlRNmHjQYl1nhufXdNKRLlS1D1oGY+rC2IZBR18RHQjb5V2BHX1FbLN8XxIrIYQQYt/JiNUQsOsRM/1Tf5GAyTg7wobu7IBinzVhi+Na4hRLPr6v6Mk5lIoupq7TEg+h6xrHtcSlZIIQQgixHySxGgIO5IiZnUsmJPMOI2qC6JqGrxR9xRI1UjJBCCGE2G8yFXgA7rvvPsaPH08wGGTGjBm88MILR7Q/+3LEjOP5A3b49ZdMOLYxhucrsk4Jz1cc2xjj7Em7PydQCCGEEHsmI1b76T/+4z+YP38+9913H2eccQYPPPAAF1xwAW+99RZjxow5In06mCNmpGSCEEIIMXhkxGo/3XvvvXzxi1/kS1/6Escffzw//OEPGT16NPfff/8R69PBHjHTXzKhJRGkNmJLUiWEEEIcIEms9oPjOCxbtozzzz+/6vr555/Pyy+/vNvnFItF0ul01ddg618vlQjZbOjOki2Wp/SyxRIburNyxIwQQghxmEhitR+6urrwPI/m5uaq683NzbS3t+/2OQsWLCCRSFS+Ro8efUj61r9eakJjjHTBZUsyR7rgMkHWSwkhhBCHjayxOgC7jvwopfY4GnTrrbdy4403Vh6n0+lDmlzJeikhhBDiyJHEaj80NDRgGMaA0amOjo4Bo1j9AoEAgUDgcHQPkCNmhBBCiCNJpgL3g23bzJgxg0WLFlVdX7RoEaeffvoR6pUQQgghPixkxGo/3XjjjVxxxRXMnDmTWbNm8bOf/YxNmzZx1VVXHemuCSGEEOIIk8RqP33605+mu7ubO+64g23btjF16lSefvppxo4de6S7JoQQQogjTFO7Fj4Sh1Q6nSaRSJBKpYjH40e6O0IIIYTYB/v6+S1rrIQQQgghBokkVkIIIYQQg0QSKyGEEEKIQSKJlRBCCCHEIJHESgghhBBikEi5hcOsfxPmoTiMWQghhBCHRv/n9gcVU5DE6jDr7u4GOGTnBQohhBDi0MlkMiQSiT3el8TqMKurqwNg06ZNe/3BDEX9B1Bv3rx5WNXwGq5xg8Q+HGMfrnGDxD7UY1dKkclkaG1t3Ws7SawOM10vL2tLJBJD9pfvg8Tj8WEZ+3CNGyT24Rj7cI0bJPahHPu+DIjI4nUhhBBCiEEiiZUQQgghxCCRxOowCwQC3H777QQCgSPdlcNuuMY+XOMGiX04xj5c4waJfbjGvis5hFkIIYQQYpDIiJUQQgghxCCRxEoIIYQQYpBIYiWEEEIIMUgksRJCCCGEGCSSWB1m9913H+PHjycYDDJjxgxeeOGFI92liiVLljB37lxaW1vRNI3HHnus6r5Sim984xu0trYSCoU4++yzWb16dVWbYrHIddddR0NDA5FIhIsvvpgtW7ZUtent7eWKK64gkUiQSCS44oorSCaTVW02bdrE3LlziUQiNDQ0cP311+M4TlWblStXMnv2bEKhECNHjuSOO+74wDOcdmfBggWccsopxGIxmpqauOSSS1izZs2wiP3+++/nxBNPrBT1mzVrFn/4wx+GfNy7WrBgAZqmMX/+/CEf+ze+8Q00Tav6amlpGfJxA7S1tfG5z32O+vp6wuEwJ510EsuWLRvysY8bN27Az1zTNK655pohHfcRo8Rh88gjjyjLstTPf/5z9dZbb6kbbrhBRSIRtXHjxiPdNaWUUk8//bT6n//zf6rf/va3ClC/+93vqu7fddddKhaLqd/+9rdq5cqV6tOf/rQaMWKESqfTlTZXXXWVGjlypFq0aJFavny5Ouecc9T06dNVqVSqtPnEJz6hpk6dql5++WX18ssvq6lTp6qLLrqocr9UKqmpU6eqc845Ry1fvlwtWrRItba2qmuvvbbSJpVKqebmZvWZz3xGrVy5Uv32t79VsVhMfe9739vvuOfMmaMefPBBtWrVKrVixQp14YUXqjFjxqi+vr4hH/vjjz+unnrqKbVmzRq1Zs0addtttynLstSqVauGdNw7e+2119S4cePUiSeeqG644YbK9aEa++23365OOOEEtW3btspXR0fHkI+7p6dHjR07Vs2bN08tXbpUrV+/Xj377LPqvffeG/Kxd3R0VP28Fy1apAD13HPPDem4jxRJrA6jj3zkI+qqq66qunbcccepf/qnfzpCPdqzXRMr3/dVS0uLuuuuuyrXCoWCSiQS6qc//alSSqlkMqksy1KPPPJIpU1bW5vSdV398Y9/VEop9dZbbylAvfrqq5U2r7zyigLUO++8o5QqJ3i6rqu2trZKm1//+tcqEAioVCqllFLqvvvuU4lEQhUKhUqbBQsWqNbWVuX7/kHF3tHRoQC1ePHiYRe7UkrV1taq//2///ewiDuTyaiJEyeqRYsWqdmzZ1cSq6Ec++23366mT5++23tDOe6vfe1r6swzz9zj/aEc+65uuOEGNWHCBOX7/rCK+3CRqcDDxHEcli1bxvnnn191/fzzz+fll18+Qr3ad+vXr6e9vb2q/4FAgNmzZ1f6v2zZMlzXrWrT2trK1KlTK21eeeUVEokEp556aqXNaaedRiKRqGozderUqoMu58yZQ7FYrAzbv/LKK8yePbuqGN2cOXPYunUrGzZsOKhYU6kU8JcDs4dL7J7n8cgjj5DNZpk1a9awiPuaa67hwgsv5GMf+1jV9aEe+9q1a2ltbWX8+PF85jOf4f333x/ycT/++OPMnDmTSy+9lKamJk4++WR+/vOfV+4P5dh35jgOv/zlL/nCF76ApmnDJu7DSRKrw6SrqwvP82hubq663tzcTHt7+xHq1b7r7+Pe+t/e3o5t29TW1u61TVNT04DXb2pqqmqz6/vU1tZi2/Ze2/Q/Ppjvp1KKG2+8kTPPPJOpU6dWvd5QjX3lypVEo1ECgQBXXXUVv/vd75gyZcqQj/uRRx5h+fLlLFiwYMC9oRz7qaeeysMPP8wzzzzDz3/+c9rb2zn99NPp7u4e0nG///773H///UycOJFnnnmGq666iuuvv56HH3646vWGYuw7e+yxx0gmk8ybN6/qtYZ63IeTeaQ7MNxomlb1WCk14NqH2YH0f9c2u2s/GG3UjsWNB/P9vPbaa3nzzTd58cUXB9wbqrFPnjyZFStWkEwm+e1vf8uVV17J4sWL9/peR3vcmzdv5oYbbmDhwoUEg8E9thuKsV9wwQWVv0+bNo1Zs2YxYcIE/u///b+cdtppe3yvoz1u3/eZOXMmd955JwAnn3wyq1ev5v777+fzn//8Xt/vaI99Z7/4xS+44IILqkaN9vReQynuw0lGrA6ThoYGDMMYkHF3dHQMyM4/jPp3De2t/y0tLTiOQ29v717bbN++fcDrd3Z2VrXZ9X16e3txXXevbTo6OoCB//LaV9dddx2PP/44zz33HKNGjapcH+qx27bNsccey8yZM1mwYAHTp0/nRz/60ZCOe9myZXR0dDBjxgxM08Q0TRYvXsy//Mu/YJrmHv+FPBRi31UkEmHatGmsXbt2SP/MR4wYwZQpU6quHX/88WzatKnyXjA0Y++3ceNGnn32Wb70pS9Vrg2HuA83SawOE9u2mTFjBosWLaq6vmjRIk4//fQj1Kt9N378eFpaWqr67zgOixcvrvR/xowZWJZV1Wbbtm2sWrWq0mbWrFmkUilee+21SpulS5eSSqWq2qxatYpt27ZV2ixcuJBAIMCMGTMqbZYsWVK1TXfhwoW0trYybty4/YpNKcW1117Lo48+yp///GfGjx8/bGLf0/ejWCwO6bjPO+88Vq5cyYoVKypfM2fO5PLLL2fFihUcc8wxQzb2XRWLRd5++21GjBgxpH/mZ5xxxoAyKu+++y5jx44Fhsd/5w8++CBNTU1ceOGFlWvDIe7D7hAujBe76C+38Itf/EK99dZbav78+SoSiagNGzYc6a4ppco7pN544w31xhtvKEDde++96o033qiUg7jrrrtUIpFQjz76qFq5cqX67Gc/u9stuaNGjVLPPvusWr58uTr33HN3uyX3xBNPVK+88op65ZVX1LRp03a7Jfe8885Ty5cvV88++6waNWpU1ZbcZDKpmpub1Wc/+1m1cuVK9eijj6p4PH5AW3L/4R/+QSUSCfX8889XbUnO5XKVNkM19ltvvVUtWbJErV+/Xr355pvqtttuU7quq4ULFw7puHdn512BQzn2m266ST3//PPq/fffV6+++qq66KKLVCwWq/x/aKjG/dprrynTNNV3vvMdtXbtWvWrX/1KhcNh9ctf/rLSZqjGrpRSnuepMWPGqK997WsD7g3luI8ESawOs3/9139VY8eOVbZtq7/6q7+qbOn/MHjuuecUMODryiuvVEqVtyPffvvtqqWlRQUCAXXWWWeplStXVr1GPp9X1157raqrq1OhUEhddNFFatOmTVVturu71eWXX65isZiKxWLq8ssvV729vVVtNm7cqC688EIVCoVUXV2duvbaa6u23yql1Jtvvqk++tGPqkAgoFpaWtQ3vvGNA9qOu7uYAfXggw9W2gzV2L/whS9Ufh8bGxvVeeedV0mqhnLcu7NrYjVUY++vUWRZlmptbVV/8zd/o1avXj3k41ZKqSeeeEJNnTpVBQIBddxxx6mf/exnVfeHcuzPPPOMAtSaNWsG3BvKcR8JmlJHUzlTIYQQQogPL1ljJYQQQggxSCSxEkIIIYQYJJJYCSGEEEIMEkmshBBCCCEGiSRWQgghhBCDRBIrIYQQQohBIomVEEIIIcQgkcRKCCGEEGKQSGIlhBAfAs8//zyappFMJo90V4QQB0Eqrwshhoyzzz6bk046iR/+8IdHuiv7zXEcenp6aG5uRtO0I90dIcQBkhErIcSwopSiVCod6W5UcV0X27ZpaWmRpEqIo5wkVkKIIWHevHksXryYH/3oR2iahqZpbNiwoTLF9swzzzBz5kwCgQAvvPAC8+bN45JLLql6jfnz53P22WdXHiuluPvuuznmmGMIhUJMnz6d//zP/9xrP8aNG8e3vvUtLrvsMqLRKK2trfz4xz+uaqNpGj/96U/55Cc/SSQS4dvf/vZupwJfeuklZs+eTTgcpra2ljlz5tDb23vAfRNCHHqSWAkhhoQf/ehHzJo1iy9/+cts27aNbdu2MXr06Mr9W265hQULFvD2229z4okn7tNr/vM//zMPPvgg999/P6tXr+arX/0qn/vc51i8ePFen3fPPfdw4oknsnz5cm699Va++tWvsmjRoqo2t99+O5/85CdZuXIlX/jCFwa8xooVKzjvvPM44YQTeOWVV3jxxReZO3cunucdVN+EEIeWeaQ7IIQQgyGRSGDbNuFwmJaWlgH377jjDj7+8Y/v8+tls1nuvfde/vznPzNr1iwAjjnmGF588UUeeOABZs+evcfnnnHGGfzTP/0TAJMmTeKll17iBz/4QdX7X3bZZVUJ1fr166te4+6772bmzJncd999lWsnnHDCQfdNCHFoSWIlhBgWZs6cuV/t33rrLQqFwoBkzHEcTj755L0+tz/Z2fnxrgvqP6g/K1as4NJLLx30vgkhDi1JrIQQw0IkEql6rOs6u26Kdl238nff9wF46qmnGDlyZFW7QCCw3++/66L0Xfuzq1AotMd7g903IcTgkcRKCDFk2LZdWYP0QRobG1m1alXVtRUrVmBZFgBTpkwhEAiwadOm/Z5ae/XVVwc8Pu644/brNU488UT+9Kc/8c1vfnPAvYPpmxDi0JLESggxZIwbN46lS5eyYcMGotEodXV1e2x77rnncs899/Dwww8za9YsfvnLX7Jq1arKVFosFuPmm2/mq1/9Kr7vc+aZZ5JOp3n55ZeJRqNceeWVe3ztl156ibvvvptLLrmERYsW8Zvf/Iannnpqv2K59dZbmTZtGldffTVXXXUVtm3z3HPPcemll9LQ0HDAfRNCHFqyK1AIMWTcfPPNGIbBlClTaGxsZNOmTXtsO2fOHP7X//pf3HLLLZxyyilkMhk+//nPV7X51re+xde//nUWLFjA8ccfz5w5c3jiiScYP378Xvtx0003sWzZMk4++WS+9a1v8f3vf585c+bsVyyTJk1i4cKF/Pd//zcf+chHmDVrFr///e8xTfOg+iaEOLSk8roQQgyicePGMX/+fObPn3+kuyKEOAJkxEoIIYQQYpBIYiWEEEIIMUhkKlAIIYQQYpDIiJUQQgghxCCRxEoIIYQQYpBIYiWEEEIIMUgksRJCCCGEGCSSWAkhhBBCDBJJrIQQQgghBokkVkIIIYQQg0QSKyGEEEKIQfL/AHCd6qD29Km6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(y_train, ridge_tuned.predict(X_train), alpha=0.3)\n",
    "grid = np.linspace(y_train.min(), y_train.max(), 1000)\n",
    "plt.plot(grid, grid, \"--k\")\n",
    "plt.xlabel(\"true price\")\n",
    "plt.ylabel(\"predicte price\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Here we can see a few cases where our prediction is way off.\n",
    "- Is there something weird about those houses, perhaps? Outliers? \n",
    "- Under the line means we're under-predicting, over the line means we're over-predicting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### $R^2$ (not in detail)\n",
    "\n",
    "A common score is the $R^2$\n",
    "\n",
    "- This is the score that `sklearn` uses by default when you call score()\n",
    "- You can [read about it](https://en.wikipedia.org/wiki/Coefficient_of_determination) if interested.\n",
    "- $R^2$ measures the proportion of variability in $y$ that can be explained using $X$. \n",
    "- You will learn about it in detail in DSCI 561.  \n",
    "- Independent of the scale of $y$. So the max is 1.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$R^2(y, \\hat{y}) = 1 - \\frac{\\sum_{i=1}^n (y_i - \\hat{y_i})^2}{\\sum_{i=1}^n (y_i - \\bar{y})^2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The denominator measures the total variance in $y$.  \n",
    "- The amount of variability that is left unexplained after performing regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Key points:\n",
    "- The maximum is 1 for perfect predictions\n",
    "- Negative values are very bad: \"worse than DummyRegressor\" (very bad)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "(Optional) Warning: MSE is \"reversible\" but $R^2$ is not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "873230473.3636105"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(y_train, preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "873230473.3636105"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(preds, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8601212294857902"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2_score(y_train, preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8279622258827066"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2_score(preds, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- When you call `fit` it minimizes MSE / maximizes $R^2$ (or something like that) by default.\n",
    "- Just like in classification, this isn't always what you want!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### MAPE\n",
    "\n",
    "- We got an RMSE of ~$30,000 before. \n",
    "\n",
    "Question: Is an error of \\$30,000 acceptable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29550.473318774617"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(mean_squared_error(y_train, ridge_tuned.predict(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- For a house worth \\$600k, it seems reasonable! That's 5% error.\n",
    "- For a house worth \\$60k, that is terrible. It's 50% error.\n",
    "\n",
    "We have both of these cases in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmxUlEQVR4nO3df3TU1Z3/8ddgYExwMisqM5kSSLSxKgGl4EaiNrGauDSw9bDrVqEWl+0eKKBk2V0gZrcET5kgdnPSNpoeWA8bD5vSPwSXXVpN2GrYbqBEfqwxuEgPIaTImK0bJlFoUsn9/sHy+TIkIBOSm8zk+Tjnc07nfu7MvN8Byat37mc+LmOMEQAAgCWjhroAAAAwshA+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFiVMNQFXKqnp0cffvihPB6PXC7XUJcDAACugjFGnZ2dCgQCGjXqymsbwy58fPjhh0pNTR3qMgAAQD+0trZqwoQJV5wz7MKHx+ORdL745OTkIa4GAABcjY6ODqWmpjq/x69k2IWPCx+1JCcnEz4AAIgxV7Nlgg2nAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwKmGoC8DgSVu9M+Lx8fUFQ1QJAAD/HysfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKuiCh+fffaZ/u7v/k7p6elKTEzUrbfequeff149PT3OHGOMSkpKFAgElJiYqNzcXDU1NQ144RgYaat3RhwAAAy2qMLHCy+8oB//+MeqqKjQ+++/rw0bNujFF1/Uj370I2fOhg0bVFZWpoqKCjU0NMjv9ysvL0+dnZ0DXjwAAIg9UYWPPXv26Otf/7oKCgqUlpamP/3TP1V+fr7eeecdSedXPcrLy1VcXKy5c+cqMzNTVVVVOnPmjKqrqwelAQAAEFuiCh8PPPCA/v3f/10ffPCBJOm//uu/9Mtf/lJf+9rXJEnNzc0KhULKz893nuN2u5WTk6P6+vo+X7Orq0sdHR0RBwAAiF9R3Vhu1apVCofDuuOOO3Tdddfp3LlzWrdunZ588klJUigUkiT5fL6I5/l8PrW0tPT5mqWlpVq7dm1/agcAADEoqpWPn/70p9qyZYuqq6t14MABVVVV6fvf/76qqqoi5rlcrojHxpheYxcUFRUpHA47R2tra5QtAACAWBLVysff/u3favXq1XriiSckSVOmTFFLS4tKS0u1YMEC+f1+SedXQFJSUpzntbW19VoNucDtdsvtdve3fgAAEGOiWvk4c+aMRo2KfMp1113nXGqbnp4uv9+v2tpa53x3d7fq6uqUnZ09AOUCAIBYF9XKx5w5c7Ru3TpNnDhRkydP1sGDB1VWVqaFCxdKOv9xS2FhoYLBoDIyMpSRkaFgMKikpCTNmzdvUBrA1eN7PAAAw0FU4eNHP/qR/v7v/15LlixRW1ubAoGAFi1apO9+97vOnJUrV+rs2bNasmSJ2tvblZWVpZqaGnk8ngEvHgAAxB6XMcYMdREX6+jokNfrVTgcVnJy8lCXE9P6s9JxfH3BIFQCAIh30fz+5t4uAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKyKKnykpaXJ5XL1OpYuXSpJMsaopKREgUBAiYmJys3NVVNT06AUDgAAYlNU4aOhoUGnTp1yjtraWknS448/LknasGGDysrKVFFRoYaGBvn9fuXl5amzs3PgKwcAADEpqvBxyy23yO/3O8e//du/6bbbblNOTo6MMSovL1dxcbHmzp2rzMxMVVVV6cyZM6qurh6s+gEAQIzp956P7u5ubdmyRQsXLpTL5VJzc7NCoZDy8/OdOW63Wzk5Oaqvr7/s63R1damjoyPiAAAA8avf4eP111/X6dOn9fTTT0uSQqGQJMnn80XM8/l8zrm+lJaWyuv1Okdqamp/SwIAADGg3+HjlVde0axZsxQIBCLGXS5XxGNjTK+xixUVFSkcDjtHa2trf0sCAAAxIKE/T2ppadGuXbu0bds2Z8zv90s6vwKSkpLijLe1tfVaDbmY2+2W2+3uTxkAACAG9WvlY/PmzRo/frwKCgqcsfT0dPn9fucKGOn8vpC6ujplZ2dfe6UAACAuRL3y0dPTo82bN2vBggVKSPj/T3e5XCosLFQwGFRGRoYyMjIUDAaVlJSkefPmDWjRAAAgdkUdPnbt2qUTJ05o4cKFvc6tXLlSZ8+e1ZIlS9Te3q6srCzV1NTI4/EMSLEAACD2uYwxZqiLuFhHR4e8Xq/C4bCSk5OHupyYlrZ6Z9TPOb6+4PMnAQBwiWh+f3NvFwAAYBXhAwAAWNWvS20x/PTnIxYAAIYCKx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsShroAxIe01TsjHh9fXzBElQAAhjtWPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFZFHT5Onjypb37zm7rpppuUlJSke+65R/v373fOG2NUUlKiQCCgxMRE5ebmqqmpaUCLxvCXtnpnrwMAACnK8NHe3q77779fo0eP1s9//nMdPnxY//AP/6A/+IM/cOZs2LBBZWVlqqioUENDg/x+v/Ly8tTZ2TnQtQMAgBiUEM3kF154Qampqdq8ebMzlpaW5vxvY4zKy8tVXFysuXPnSpKqqqrk8/lUXV2tRYsWDUzVAAAgZkW18rFjxw7NmDFDjz/+uMaPH69p06Zp06ZNzvnm5maFQiHl5+c7Y263Wzk5Oaqvr+/zNbu6utTR0RFxAACA+BXVysexY8dUWVmpFStW6LnnntO+ffv07LPPyu1261vf+pZCoZAkyefzRTzP5/OppaWlz9csLS3V2rVr+1n+yMUeCgBArIpq5aOnp0df/vKXFQwGNW3aNC1atEh/+Zd/qcrKyoh5Lpcr4rExptfYBUVFRQqHw87R2toaZQsAACCWRBU+UlJSdNddd0WM3XnnnTpx4oQkye/3S5KzAnJBW1tbr9WQC9xut5KTkyMOAAAQv6IKH/fff7+OHDkSMfbBBx9o0qRJkqT09HT5/X7V1tY657u7u1VXV6fs7OwBKBcAAMS6qPZ8/NVf/ZWys7MVDAb1Z3/2Z9q3b582btyojRs3Sjr/cUthYaGCwaAyMjKUkZGhYDCopKQkzZs3b1AaAAAAsSWq8HHvvfdq+/btKioq0vPPP6/09HSVl5dr/vz5zpyVK1fq7NmzWrJkidrb25WVlaWamhp5PJ4BLx4AAMSeqMKHJM2ePVuzZ8++7HmXy6WSkhKVlJRcS10AACBOcW8XAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFgV9aW2GHkuvYnd8fUFQ1QJACAesPIBAACsInwAAACrCB8AAMAq9nwgwqX7OwAAGGisfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArEoY6gLw+dJW7xzqEgAAGDCsfAAAAKsIHwAAwCrCBwAAsCqq8FFSUiKXyxVx+P1+57wxRiUlJQoEAkpMTFRubq6ampoGvGjEprTVOyMOAMDIFPXKx+TJk3Xq1CnnaGxsdM5t2LBBZWVlqqioUENDg/x+v/Ly8tTZ2TmgRQMAgNgVdfhISEiQ3+93jltuuUXS+VWP8vJyFRcXa+7cucrMzFRVVZXOnDmj6urqAS8cAADEpqjDx9GjRxUIBJSenq4nnnhCx44dkyQ1NzcrFAopPz/fmet2u5WTk6P6+vrLvl5XV5c6OjoiDgAAEL+i+p6PrKwsvfrqq7r99tv10Ucf6Xvf+56ys7PV1NSkUCgkSfL5fBHP8fl8amlpuexrlpaWau3atf0oHUOF/RoAgGsR1crHrFmz9Cd/8ieaMmWKHnnkEe3cef6XUFVVlTPH5XJFPMcY02vsYkVFRQqHw87R2toaTUkAACDGXNOltmPHjtWUKVN09OhR56qXCysgF7S1tfVaDbmY2+1WcnJyxAEAAOLXNYWPrq4uvf/++0pJSVF6err8fr9qa2ud893d3aqrq1N2dvY1FwoAAOJDVHs+/uZv/kZz5szRxIkT1dbWpu9973vq6OjQggUL5HK5VFhYqGAwqIyMDGVkZCgYDCopKUnz5s0brPoBAECMiSp8/OY3v9GTTz6p3/72t7rlllt03333ae/evZo0aZIkaeXKlTp79qyWLFmi9vZ2ZWVlqaamRh6PZ1CKBwAAscdljDFDXcTFOjo65PV6FQ6H2f/xf+L16pLj6wuGugQAwACJ5vc393YBAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYNU1hY/S0lK5XC4VFhY6Y8YYlZSUKBAIKDExUbm5uWpqarrWOhGH0lbv7HUAAOJfv8NHQ0ODNm7cqKlTp0aMb9iwQWVlZaqoqFBDQ4P8fr/y8vLU2dl5zcUCAIDY16/w8cknn2j+/PnatGmTbrzxRmfcGKPy8nIVFxdr7ty5yszMVFVVlc6cOaPq6uoBKxoAAMSufoWPpUuXqqCgQI888kjEeHNzs0KhkPLz850xt9utnJwc1dfXX1ulAAAgLiRE+4StW7fqwIEDamho6HUuFApJknw+X8S4z+dTS0tLn6/X1dWlrq4u53FHR0e0JQEAgBgS1cpHa2urli9fri1btuj666+/7DyXyxXx2BjTa+yC0tJSeb1e50hNTY2mJAAAEGOiCh/79+9XW1ubpk+froSEBCUkJKiurk4//OEPlZCQ4Kx4XFgBuaCtra3XasgFRUVFCofDztHa2trPVgAAQCyI6mOXhx9+WI2NjRFjf/7nf6477rhDq1at0q233iq/36/a2lpNmzZNktTd3a26ujq98MILfb6m2+2W2+3uZ/kAACDWRBU+PB6PMjMzI8bGjh2rm266yRkvLCxUMBhURkaGMjIyFAwGlZSUpHnz5g1c1QAAIGZFveH086xcuVJnz57VkiVL1N7erqysLNXU1Mjj8Qz0WwEAgBjkMsaYoS7iYh0dHfJ6vQqHw0pOTh7qcoaFkfTNn8fXFwx1CQCAfojm9zf3dgEAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYlRDN5MrKSlVWVur48eOSpMmTJ+u73/2uZs2aJUkyxmjt2rXauHGj2tvblZWVpZdeekmTJ08e8MJjQdrqnb3Gjq8v6NfzRopLe7+an9dgvg4AYOBFtfIxYcIErV+/Xu+8847eeecdffWrX9XXv/51NTU1SZI2bNigsrIyVVRUqKGhQX6/X3l5eers7ByU4gEAQOyJKnzMmTNHX/va13T77bfr9ttv17p163TDDTdo7969MsaovLxcxcXFmjt3rjIzM1VVVaUzZ86ourp6sOoHAAAxpt97Ps6dO6etW7fq008/1cyZM9Xc3KxQKKT8/HxnjtvtVk5Ojurr6y/7Ol1dXero6Ig4AABA/Io6fDQ2NuqGG26Q2+3W4sWLtX37dt11110KhUKSJJ/PFzHf5/M55/pSWloqr9frHKmpqdGWBAAAYkjU4eNLX/qSDh06pL179+o73/mOFixYoMOHDzvnXS5XxHxjTK+xixUVFSkcDjtHa2trtCUBAIAYEtXVLpI0ZswYffGLX5QkzZgxQw0NDfrBD36gVatWSZJCoZBSUlKc+W1tbb1WQy7mdrvldrujLQMAAMSoa/6eD2OMurq6lJ6eLr/fr9raWudcd3e36urqlJ2dfa1vAwAA4kRUKx/PPfecZs2apdTUVHV2dmrr1q16++239cYbb8jlcqmwsFDBYFAZGRnKyMhQMBhUUlKS5s2bN1j1AwCAGBNV+Pjoo4/01FNP6dSpU/J6vZo6dareeOMN5eXlSZJWrlyps2fPasmSJc6XjNXU1Mjj8QxK8QAAIPZEFT5eeeWVK553uVwqKSlRSUnJtdQEAADiGPd2AQAAVhE+AACAVVFfagsMNyP5RnwAEItY+QAAAFYRPgAAgFWEDwAAYBV7PjCssZ8DAOIPKx8AAMAqwgcAALCK8AEAAKxiz4dll+5hOL6+YIgqAQBgaLDyAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwKmGoCxjp0lbvHOoSRoS+fs7H1xcMQSUAAFY+AACAVYQPAABgFeEDAABYFVX4KC0t1b333iuPx6Px48frscce05EjRyLmGGNUUlKiQCCgxMRE5ebmqqmpaUCLBgAAsSuq8FFXV6elS5dq7969qq2t1Weffab8/Hx9+umnzpwNGzaorKxMFRUVamhokN/vV15enjo7Owe8eAAAEHuiutrljTfeiHi8efNmjR8/Xvv379dXvvIVGWNUXl6u4uJizZ07V5JUVVUln8+n6upqLVq0aOAqBwAAMema9nyEw2FJ0rhx4yRJzc3NCoVCys/Pd+a43W7l5OSovr6+z9fo6upSR0dHxAEAAOJXv8OHMUYrVqzQAw88oMzMTElSKBSSJPl8voi5Pp/POXep0tJSeb1e50hNTe1vSQAAIAb0O3wsW7ZM7777rn7yk5/0OudyuSIeG2N6jV1QVFSkcDjsHK2trf0tCQAAxIB+fcPpM888ox07dmj37t2aMGGCM+73+yWdXwFJSUlxxtva2nqthlzgdrvldrv7UwYAAIhBUa18GGO0bNkybdu2Tb/4xS+Unp4ecT49PV1+v1+1tbXOWHd3t+rq6pSdnT0wFQMAgJgW1crH0qVLVV1drX/5l3+Rx+Nx9nF4vV4lJibK5XKpsLBQwWBQGRkZysjIUDAYVFJSkubNmzcoDQAAgNgSVfiorKyUJOXm5kaMb968WU8//bQkaeXKlTp79qyWLFmi9vZ2ZWVlqaamRh6PZ0AKBgAAsS2q8GGM+dw5LpdLJSUlKikp6W9NAAAgjnFvFwAAYBXhAwAAWNWvS22BeJC2emfE4+PrC6J+ztU+DwDw/7HyAQAArCJ8AAAAqwgfAADAKvZ8AFfQ1x4PAMC1YeUDAABYRfgAAABWET4AAIBV7PkA/g/7OwDADlY+AACAVYQPAABgFeEDAABYRfgAAABWseEUuEb9uUEdAIxkrHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKxKiPYJu3fv1osvvqj9+/fr1KlT2r59ux577DHnvDFGa9eu1caNG9Xe3q6srCy99NJLmjx58kDWDQxbaat3fu6c4+sLLFQCAMNT1Csfn376qe6++25VVFT0eX7Dhg0qKytTRUWFGhoa5Pf7lZeXp87OzmsuFgAAxL6oVz5mzZqlWbNm9XnOGKPy8nIVFxdr7ty5kqSqqir5fD5VV1dr0aJF11YtAACIeQO656O5uVmhUEj5+fnOmNvtVk5Ojurr6/t8TldXlzo6OiIOAAAQv6Je+biSUCgkSfL5fBHjPp9PLS0tfT6ntLRUa9euHcgygLh16X4S9o4AiEWDcrWLy+WKeGyM6TV2QVFRkcLhsHO0trYORkkAAGCYGNCVD7/fL+n8CkhKSooz3tbW1ms15AK32y232z2QZQAAgGFsQFc+0tPT5ff7VVtb64x1d3errq5O2dnZA/lWAAAgRkW98vHJJ5/o17/+tfO4ublZhw4d0rhx4zRx4kQVFhYqGAwqIyNDGRkZCgaDSkpK0rx58wa0cCDeXc33hfQ1h30gAIa7qMPHO++8o4ceesh5vGLFCknSggUL9E//9E9auXKlzp49qyVLljhfMlZTUyOPxzNwVQMAgJgVdfjIzc2VMeay510ul0pKSlRSUnItdQEAgDjFvV0AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYNWAfr36SHc1XwoFXA5/fwCMFKx8AAAAqwgfAADAKsIHAACwij0fV4nP44HeLv3vgpvaAbgarHwAAACrCB8AAMAqwgcAALBqxO356GvvBp9Twzabe4hs/p3nvy8AV4OVDwAAYBXhAwAAWEX4AAAAVo24PR9Xg+/0QCy7mr+/Nr+fg+8CAXApVj4AAIBVhA8AAGAV4QMAAFhF+AAAAFax4RTAVRmojdgD+UVkbGYFYhMrHwAAwCrCBwAAsIrwAQAArGLPB4A+DfXN7y41UPtCrua1R9IN8tg3g6HAygcAALCK8AEAAKwifAAAAKsGbc/Hyy+/rBdffFGnTp3S5MmTVV5ergcffHCw3g7ANYiFmyn254Z5tl3N/onB2mMRL/tUBnP/z0gRCz/DQVn5+OlPf6rCwkIVFxfr4MGDevDBBzVr1iydOHFiMN4OAADEkEEJH2VlZfqLv/gLffvb39add96p8vJypaamqrKycjDeDgAAxJAB/9ilu7tb+/fv1+rVqyPG8/PzVV9f32t+V1eXurq6nMfhcFiS1NHRMdClSZJ6us70Grv0vfqaAyB+Xc2/AVfzb9Klz+vrOVczpz8Gs2abrubf36Gucbgbqp/hhdc0xnz+ZDPATp48aSSZ//zP/4wYX7dunbn99tt7zV+zZo2RxMHBwcHBwREHR2tr6+dmhUHbcOpyuSIeG2N6jUlSUVGRVqxY4Tzu6elRS0uL7rnnHrW2tio5OXmwShyWOjo6lJqaOuJ6H6l9S/Q+EnsfqX1LI7f3kdC3MUadnZ0KBAKfO3fAw8fNN9+s6667TqFQKGK8ra1NPp+v13y32y232x0xNmrU+a0oycnJcfuH9HlGau8jtW+J3kdi7yO1b2nk9h7vfXu93quaN+AbTseMGaPp06ertrY2Yry2tlbZ2dkD/XYAACDGDMrHLitWrNBTTz2lGTNmaObMmdq4caNOnDihxYsXD8bbAQCAGDIo4eMb3/iGPv74Yz3//PM6deqUMjMz9bOf/UyTJk26que73W6tWbOm18cxI8FI7X2k9i3R+0jsfaT2LY3c3kdq35fjMuZqrokBAAAYGNzbBQAAWEX4AAAAVhE+AACAVYQPAABg1bAMHy+//LLS09N1/fXXa/r06fqP//iPoS7JsXv3bs2ZM0eBQEAul0uvv/56xHljjEpKShQIBJSYmKjc3Fw1NTVFzOnq6tIzzzyjm2++WWPHjtUf//Ef6ze/+U3EnPb2dj311FPyer3yer166qmndPr06Yg5J06c0Jw5czR27FjdfPPNevbZZ9Xd3R0xp7GxUTk5OUpMTNQXvvAFPf/881f3vfuXKC0t1b333iuPx6Px48frscce05EjR+K+98rKSk2dOtX5YqCZM2fq5z//eVz3fDmlpaVyuVwqLCyM+/5LSkrkcrkiDr/fH/d9S9LJkyf1zW9+UzfddJOSkpJ0zz33aP/+/XHfe1paWq8/c5fLpaVLl8Z130Omn7dwGTRbt241o0ePNps2bTKHDx82y5cvN2PHjjUtLS1DXZoxxpif/exnpri42Lz22mtGktm+fXvE+fXr1xuPx2Nee+0109jYaL7xjW+YlJQU09HR4cxZvHix+cIXvmBqa2vNgQMHzEMPPWTuvvtu89lnnzlz/uiP/shkZmaa+vp6U19fbzIzM83s2bOd85999pnJzMw0Dz30kDlw4ICpra01gUDALFu2zJkTDoeNz+czTzzxhGlsbDSvvfaa8Xg85vvf/37UfT/66KNm8+bN5r333jOHDh0yBQUFZuLEieaTTz6J69537Nhhdu7caY4cOWKOHDlinnvuOTN69Gjz3nvvxW3Pfdm3b59JS0szU6dONcuXL3fG47X/NWvWmMmTJ5tTp045R1tbW9z3/b//+79m0qRJ5umnnza/+tWvTHNzs9m1a5f59a9/Hfe9t7W1Rfx519bWGknmrbfeiuu+h8qwCx9/+Id/aBYvXhwxdscdd5jVq1cPUUWXd2n46OnpMX6/36xfv94Z+93vfme8Xq/58Y9/bIwx5vTp02b06NFm69atzpyTJ0+aUaNGmTfeeMMYY8zhw4eNJLN3715nzp49e4wk89///d/GmPMhaNSoUebkyZPOnJ/85CfG7XabcDhsjDHm5ZdfNl6v1/zud79z5pSWlppAIGB6enquqfe2tjYjydTV1Y243m+88Ubzj//4jyOm587OTpORkWFqa2tNTk6OEz7iuf81a9aYu+++u89z8dz3qlWrzAMPPHDZ8/Hc+6WWL19ubrvtNtPT0zOi+rZlWH3s0t3drf379ys/Pz9iPD8/X/X19UNU1dVrbm5WKBSKqN/tdisnJ8epf//+/fr9738fMScQCCgzM9OZs2fPHnm9XmVlZTlz7rvvPnm93og5mZmZETfwefTRR9XV1eUske7Zs0c5OTkRX2rz6KOP6sMPP9Tx48evqddwOCxJGjdu3Ijp/dy5c9q6das+/fRTzZw5c0T0LElLly5VQUGBHnnkkYjxeO//6NGjCgQCSk9P1xNPPKFjx47Ffd87duzQjBkz9Pjjj2v8+PGaNm2aNm3a5JyP594v1t3drS1btmjhwoVyuVwjpm+bhlX4+O1vf6tz5871ugGdz+frdaO64ehCjVeqPxQKacyYMbrxxhuvOGf8+PG9Xn/8+PERcy59nxtvvFFjxoy54pwLj6/l52mM0YoVK/TAAw8oMzMz4vXisffGxkbdcMMNcrvdWrx4sbZv36677rorrnu+YOvWrTpw4IBKS0t7nYvn/rOysvTqq6/qzTff1KZNmxQKhZSdna2PP/44rvs+duyYKisrlZGRoTfffFOLFy/Ws88+q1dffTXi9eKx94u9/vrrOn36tJ5++umI14r3vm0alK9Xv1YulyvisTGm19hw1p/6L53T1/yBmGP+b0PStfw8ly1bpnfffVe//OUve52Lx96/9KUv6dChQzp9+rRee+01LViwQHV1dVd8n1jvWZJaW1u1fPly1dTU6Prrr7/svHjsf9asWc7/njJlimbOnKnbbrtNVVVVuu+++y77XrHed09Pj2bMmKFgMChJmjZtmpqamlRZWalvfetbV3y/WO/9Yq+88opmzZrV69bw8d63TcNq5ePmm2/Wdddd1yu5tbW19Up5w9GF3fBXqt/v96u7u1vt7e1XnPPRRx/1ev3/+Z//iZhz6fu0t7fr97///RXntLW1Seqd4K/WM888ox07duitt97ShAkTnPF47n3MmDH64he/qBkzZqi0tFR33323fvCDH8R1z9L5ZeS2tjZNnz5dCQkJSkhIUF1dnX74wx8qISHhsv9PK176v9jYsWM1ZcoUHT16NK7/3FNSUnTXXXdFjN155506ceKE815SfPZ+QUtLi3bt2qVvf/vbzthI6Nu2YRU+xowZo+nTp6u2tjZivLa2VtnZ2UNU1dVLT0+X3++PqL+7u1t1dXVO/dOnT9fo0aMj5pw6dUrvvfeeM2fmzJkKh8Pat2+fM+dXv/qVwuFwxJz33ntPp06dcubU1NTI7XZr+vTpzpzdu3dHXKJVU1OjQCCgtLS0qHozxmjZsmXatm2bfvGLXyg9PX3E9N7Xz6Krqyvue3744YfV2NioQ4cOOceMGTM0f/58HTp0SLfeemtc93+xrq4uvf/++0pJSYnrP/f777+/1yX0H3zwgXNT0Hju/YLNmzdr/PjxKigocMZGQt/WDeJm1n65cKntK6+8Yg4fPmwKCwvN2LFjzfHjx4e6NGPM+Z3/Bw8eNAcPHjSSTFlZmTl48KBzKfD69euN1+s127ZtM42NjebJJ5/s83KsCRMmmF27dpkDBw6Yr371q31ejjV16lSzZ88es2fPHjNlypQ+L8d6+OGHzYEDB8yuXbvMhAkTIi7HOn36tPH5fObJJ580jY2NZtu2bSY5Oblfl2N95zvfMV6v17z99tsRl6OdOXPGmROPvRcVFZndu3eb5uZm8+6775rnnnvOjBo1ytTU1MRtz1dy8dUu8dz/X//1X5u3337bHDt2zOzdu9fMnj3beDwe59+heO173759JiEhwaxbt84cPXrU/PM//7NJSkoyW7ZscebEa+/GGHPu3DkzceJEs2rVql7n4rnvoTDswocxxrz00ktm0qRJZsyYMebLX/6ycznncPDWW28ZSb2OBQsWGGPOX4q2Zs0a4/f7jdvtNl/5yldMY2NjxGucPXvWLFu2zIwbN84kJiaa2bNnmxMnTkTM+fjjj838+fONx+MxHo/HzJ8/37S3t0fMaWlpMQUFBSYxMdGMGzfOLFu2LOLSK2OMeffdd82DDz5o3G638fv9pqSkpF+XYvXVsySzefNmZ0489r5w4ULn7+Itt9xiHn74YSd4xGvPV3Jp+IjX/i98h8Po0aNNIBAwc+fONU1NTXHftzHG/Ou//qvJzMw0brfb3HHHHWbjxo0R5+O59zfffNNIMkeOHOl1Lp77HgouY2LpK9EAAECsG1Z7PgAAQPwjfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALDq/wH3ipdEUuUoVQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(y_train, bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How about looking at percent error? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "302      3.851038\n",
       "767     11.564239\n",
       "429      8.535533\n",
       "1139   -16.371069\n",
       "558     17.177968\n",
       "          ...    \n",
       "1041    -0.496571\n",
       "1122   -28.696351\n",
       "1346    -6.577648\n",
       "1406    -2.358546\n",
       "1389     3.389230\n",
       "Name: SalePrice, Length: 1314, dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_train = ridge_tuned.predict(X_train)\n",
    "percent_errors = (pred_train - y_train) / y_train * 100.0\n",
    "percent_errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are both positive (predict too high) and negative (predict too low)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can look at the absolute percent error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "302      3.851038\n",
       "767     11.564239\n",
       "429      8.535533\n",
       "1139    16.371069\n",
       "558     17.177968\n",
       "          ...    \n",
       "1041     0.496571\n",
       "1122    28.696351\n",
       "1346     6.577648\n",
       "1406     2.358546\n",
       "1389     3.389230\n",
       "Name: SalePrice, Length: 1314, dtype: float64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.abs(percent_errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "And, like MSE, we can take the average over examples. This is called mean absolute percent error (MAPE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_mape(true, pred):\n",
    "    return np.mean(np.abs((pred - true) / true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10093121294225234"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_mape(y_train, pred_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `sklearn` to calculate MAPE. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_percentage_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10093121294225234"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_percentage_error(y_train, pred_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ok, this is quite interpretable.\n",
    "- On average, we have around 10% error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Transforming the targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- When you have prices or count data, the target values are skewed. \n",
    "- Let's look at our target column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmxUlEQVR4nO3df3TU1Z3/8ddgYExwMisqM5kSSLSxKgGl4EaiNrGauDSw9bDrVqEWl+0eKKBk2V0gZrcET5kgdnPSNpoeWA8bD5vSPwSXXVpN2GrYbqBEfqwxuEgPIaTImK0bJlFoUsn9/sHy+TIkIBOSm8zk+Tjnc07nfu7MvN8Byat37mc+LmOMEQAAgCWjhroAAAAwshA+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFiVMNQFXKqnp0cffvihPB6PXC7XUJcDAACugjFGnZ2dCgQCGjXqymsbwy58fPjhh0pNTR3qMgAAQD+0trZqwoQJV5wz7MKHx+ORdL745OTkIa4GAABcjY6ODqWmpjq/x69k2IWPCx+1JCcnEz4AAIgxV7Nlgg2nAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwKmGoC8DgSVu9M+Lx8fUFQ1QJAAD/HysfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKuiCh+fffaZ/u7v/k7p6elKTEzUrbfequeff149PT3OHGOMSkpKFAgElJiYqNzcXDU1NQ144RgYaat3RhwAAAy2qMLHCy+8oB//+MeqqKjQ+++/rw0bNujFF1/Uj370I2fOhg0bVFZWpoqKCjU0NMjv9ysvL0+dnZ0DXjwAAIg9UYWPPXv26Otf/7oKCgqUlpamP/3TP1V+fr7eeecdSedXPcrLy1VcXKy5c+cqMzNTVVVVOnPmjKqrqwelAQAAEFuiCh8PPPCA/v3f/10ffPCBJOm//uu/9Mtf/lJf+9rXJEnNzc0KhULKz893nuN2u5WTk6P6+vo+X7Orq0sdHR0RBwAAiF9R3Vhu1apVCofDuuOOO3Tdddfp3LlzWrdunZ588klJUigUkiT5fL6I5/l8PrW0tPT5mqWlpVq7dm1/agcAADEoqpWPn/70p9qyZYuqq6t14MABVVVV6fvf/76qqqoi5rlcrojHxpheYxcUFRUpHA47R2tra5QtAACAWBLVysff/u3favXq1XriiSckSVOmTFFLS4tKS0u1YMEC+f1+SedXQFJSUpzntbW19VoNucDtdsvtdve3fgAAEGOiWvk4c+aMRo2KfMp1113nXGqbnp4uv9+v2tpa53x3d7fq6uqUnZ09AOUCAIBYF9XKx5w5c7Ru3TpNnDhRkydP1sGDB1VWVqaFCxdKOv9xS2FhoYLBoDIyMpSRkaFgMKikpCTNmzdvUBrA1eN7PAAAw0FU4eNHP/qR/v7v/15LlixRW1ubAoGAFi1apO9+97vOnJUrV+rs2bNasmSJ2tvblZWVpZqaGnk8ngEvHgAAxB6XMcYMdREX6+jokNfrVTgcVnJy8lCXE9P6s9JxfH3BIFQCAIh30fz+5t4uAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKyKKnykpaXJ5XL1OpYuXSpJMsaopKREgUBAiYmJys3NVVNT06AUDgAAYlNU4aOhoUGnTp1yjtraWknS448/LknasGGDysrKVFFRoYaGBvn9fuXl5amzs3PgKwcAADEpqvBxyy23yO/3O8e//du/6bbbblNOTo6MMSovL1dxcbHmzp2rzMxMVVVV6cyZM6qurh6s+gEAQIzp956P7u5ubdmyRQsXLpTL5VJzc7NCoZDy8/OdOW63Wzk5Oaqvr7/s63R1damjoyPiAAAA8avf4eP111/X6dOn9fTTT0uSQqGQJMnn80XM8/l8zrm+lJaWyuv1Okdqamp/SwIAADGg3+HjlVde0axZsxQIBCLGXS5XxGNjTK+xixUVFSkcDjtHa2trf0sCAAAxIKE/T2ppadGuXbu0bds2Z8zv90s6vwKSkpLijLe1tfVaDbmY2+2W2+3uTxkAACAG9WvlY/PmzRo/frwKCgqcsfT0dPn9fucKGOn8vpC6ujplZ2dfe6UAACAuRL3y0dPTo82bN2vBggVKSPj/T3e5XCosLFQwGFRGRoYyMjIUDAaVlJSkefPmDWjRAAAgdkUdPnbt2qUTJ05o4cKFvc6tXLlSZ8+e1ZIlS9Te3q6srCzV1NTI4/EMSLEAACD2uYwxZqiLuFhHR4e8Xq/C4bCSk5OHupyYlrZ6Z9TPOb6+4PMnAQBwiWh+f3NvFwAAYBXhAwAAWNWvS20x/PTnIxYAAIYCKx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsShroAxIe01TsjHh9fXzBElQAAhjtWPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFZFHT5Onjypb37zm7rpppuUlJSke+65R/v373fOG2NUUlKiQCCgxMRE5ebmqqmpaUCLxvCXtnpnrwMAACnK8NHe3q77779fo0eP1s9//nMdPnxY//AP/6A/+IM/cOZs2LBBZWVlqqioUENDg/x+v/Ly8tTZ2TnQtQMAgBiUEM3kF154Qampqdq8ebMzlpaW5vxvY4zKy8tVXFysuXPnSpKqqqrk8/lUXV2tRYsWDUzVAAAgZkW18rFjxw7NmDFDjz/+uMaPH69p06Zp06ZNzvnm5maFQiHl5+c7Y263Wzk5Oaqvr+/zNbu6utTR0RFxAACA+BXVysexY8dUWVmpFStW6LnnntO+ffv07LPPyu1261vf+pZCoZAkyefzRTzP5/OppaWlz9csLS3V2rVr+1n+yMUeCgBArIpq5aOnp0df/vKXFQwGNW3aNC1atEh/+Zd/qcrKyoh5Lpcr4rExptfYBUVFRQqHw87R2toaZQsAACCWRBU+UlJSdNddd0WM3XnnnTpx4oQkye/3S5KzAnJBW1tbr9WQC9xut5KTkyMOAAAQv6IKH/fff7+OHDkSMfbBBx9o0qRJkqT09HT5/X7V1tY657u7u1VXV6fs7OwBKBcAAMS6qPZ8/NVf/ZWys7MVDAb1Z3/2Z9q3b582btyojRs3Sjr/cUthYaGCwaAyMjKUkZGhYDCopKQkzZs3b1AaAAAAsSWq8HHvvfdq+/btKioq0vPPP6/09HSVl5dr/vz5zpyVK1fq7NmzWrJkidrb25WVlaWamhp5PJ4BLx4AAMSeqMKHJM2ePVuzZ8++7HmXy6WSkhKVlJRcS10AACBOcW8XAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFgV9aW2GHkuvYnd8fUFQ1QJACAesPIBAACsInwAAACrCB8AAMAq9nwgwqX7OwAAGGisfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArEoY6gLw+dJW7xzqEgAAGDCsfAAAAKsIHwAAwCrCBwAAsCqq8FFSUiKXyxVx+P1+57wxRiUlJQoEAkpMTFRubq6ampoGvGjEprTVOyMOAMDIFPXKx+TJk3Xq1CnnaGxsdM5t2LBBZWVlqqioUENDg/x+v/Ly8tTZ2TmgRQMAgNgVdfhISEiQ3+93jltuuUXS+VWP8vJyFRcXa+7cucrMzFRVVZXOnDmj6urqAS8cAADEpqjDx9GjRxUIBJSenq4nnnhCx44dkyQ1NzcrFAopPz/fmet2u5WTk6P6+vrLvl5XV5c6OjoiDgAAEL+i+p6PrKwsvfrqq7r99tv10Ucf6Xvf+56ys7PV1NSkUCgkSfL5fBHP8fl8amlpuexrlpaWau3atf0oHUOF/RoAgGsR1crHrFmz9Cd/8ieaMmWKHnnkEe3cef6XUFVVlTPH5XJFPMcY02vsYkVFRQqHw87R2toaTUkAACDGXNOltmPHjtWUKVN09OhR56qXCysgF7S1tfVaDbmY2+1WcnJyxAEAAOLXNYWPrq4uvf/++0pJSVF6err8fr9qa2ud893d3aqrq1N2dvY1FwoAAOJDVHs+/uZv/kZz5szRxIkT1dbWpu9973vq6OjQggUL5HK5VFhYqGAwqIyMDGVkZCgYDCopKUnz5s0brPoBAECMiSp8/OY3v9GTTz6p3/72t7rlllt03333ae/evZo0aZIkaeXKlTp79qyWLFmi9vZ2ZWVlqaamRh6PZ1CKBwAAscdljDFDXcTFOjo65PV6FQ6H2f/xf+L16pLj6wuGugQAwACJ5vc393YBAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYNU1hY/S0lK5XC4VFhY6Y8YYlZSUKBAIKDExUbm5uWpqarrWOhGH0lbv7HUAAOJfv8NHQ0ODNm7cqKlTp0aMb9iwQWVlZaqoqFBDQ4P8fr/y8vLU2dl5zcUCAIDY16/w8cknn2j+/PnatGmTbrzxRmfcGKPy8nIVFxdr7ty5yszMVFVVlc6cOaPq6uoBKxoAAMSufoWPpUuXqqCgQI888kjEeHNzs0KhkPLz850xt9utnJwc1dfXX1ulAAAgLiRE+4StW7fqwIEDamho6HUuFApJknw+X8S4z+dTS0tLn6/X1dWlrq4u53FHR0e0JQEAgBgS1cpHa2urli9fri1btuj666+/7DyXyxXx2BjTa+yC0tJSeb1e50hNTY2mJAAAEGOiCh/79+9XW1ubpk+froSEBCUkJKiurk4//OEPlZCQ4Kx4XFgBuaCtra3XasgFRUVFCofDztHa2trPVgAAQCyI6mOXhx9+WI2NjRFjf/7nf6477rhDq1at0q233iq/36/a2lpNmzZNktTd3a26ujq98MILfb6m2+2W2+3uZ/kAACDWRBU+PB6PMjMzI8bGjh2rm266yRkvLCxUMBhURkaGMjIyFAwGlZSUpHnz5g1c1QAAIGZFveH086xcuVJnz57VkiVL1N7erqysLNXU1Mjj8Qz0WwEAgBjkMsaYoS7iYh0dHfJ6vQqHw0pOTh7qcoaFkfTNn8fXFwx1CQCAfojm9zf3dgEAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYlRDN5MrKSlVWVur48eOSpMmTJ+u73/2uZs2aJUkyxmjt2rXauHGj2tvblZWVpZdeekmTJ08e8MJjQdrqnb3Gjq8v6NfzRopLe7+an9dgvg4AYOBFtfIxYcIErV+/Xu+8847eeecdffWrX9XXv/51NTU1SZI2bNigsrIyVVRUqKGhQX6/X3l5eers7ByU4gEAQOyJKnzMmTNHX/va13T77bfr9ttv17p163TDDTdo7969MsaovLxcxcXFmjt3rjIzM1VVVaUzZ86ourp6sOoHAAAxpt97Ps6dO6etW7fq008/1cyZM9Xc3KxQKKT8/HxnjtvtVk5Ojurr6y/7Ol1dXero6Ig4AABA/Io6fDQ2NuqGG26Q2+3W4sWLtX37dt11110KhUKSJJ/PFzHf5/M55/pSWloqr9frHKmpqdGWBAAAYkjU4eNLX/qSDh06pL179+o73/mOFixYoMOHDzvnXS5XxHxjTK+xixUVFSkcDjtHa2trtCUBAIAYEtXVLpI0ZswYffGLX5QkzZgxQw0NDfrBD36gVatWSZJCoZBSUlKc+W1tbb1WQy7mdrvldrujLQMAAMSoa/6eD2OMurq6lJ6eLr/fr9raWudcd3e36urqlJ2dfa1vAwAA4kRUKx/PPfecZs2apdTUVHV2dmrr1q16++239cYbb8jlcqmwsFDBYFAZGRnKyMhQMBhUUlKS5s2bN1j1AwCAGBNV+Pjoo4/01FNP6dSpU/J6vZo6dareeOMN5eXlSZJWrlyps2fPasmSJc6XjNXU1Mjj8QxK8QAAIPZEFT5eeeWVK553uVwqKSlRSUnJtdQEAADiGPd2AQAAVhE+AACAVVFfagsMNyP5RnwAEItY+QAAAFYRPgAAgFWEDwAAYBV7PjCssZ8DAOIPKx8AAMAqwgcAALCK8AEAAKxiz4dll+5hOL6+YIgqAQBgaLDyAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwKmGoCxjp0lbvHOoSRoS+fs7H1xcMQSUAAFY+AACAVYQPAABgFeEDAABYFVX4KC0t1b333iuPx6Px48frscce05EjRyLmGGNUUlKiQCCgxMRE5ebmqqmpaUCLBgAAsSuq8FFXV6elS5dq7969qq2t1Weffab8/Hx9+umnzpwNGzaorKxMFRUVamhokN/vV15enjo7Owe8eAAAEHuiutrljTfeiHi8efNmjR8/Xvv379dXvvIVGWNUXl6u4uJizZ07V5JUVVUln8+n6upqLVq0aOAqBwAAMema9nyEw2FJ0rhx4yRJzc3NCoVCys/Pd+a43W7l5OSovr6+z9fo6upSR0dHxAEAAOJXv8OHMUYrVqzQAw88oMzMTElSKBSSJPl8voi5Pp/POXep0tJSeb1e50hNTe1vSQAAIAb0O3wsW7ZM7777rn7yk5/0OudyuSIeG2N6jV1QVFSkcDjsHK2trf0tCQAAxIB+fcPpM888ox07dmj37t2aMGGCM+73+yWdXwFJSUlxxtva2nqthlzgdrvldrv7UwYAAIhBUa18GGO0bNkybdu2Tb/4xS+Unp4ecT49PV1+v1+1tbXOWHd3t+rq6pSdnT0wFQMAgJgW1crH0qVLVV1drX/5l3+Rx+Nx9nF4vV4lJibK5XKpsLBQwWBQGRkZysjIUDAYVFJSkubNmzcoDQAAgNgSVfiorKyUJOXm5kaMb968WU8//bQkaeXKlTp79qyWLFmi9vZ2ZWVlqaamRh6PZ0AKBgAAsS2q8GGM+dw5LpdLJSUlKikp6W9NAAAgjnFvFwAAYBXhAwAAWNWvS22BeJC2emfE4+PrC6J+ztU+DwDw/7HyAQAArCJ8AAAAqwgfAADAKvZ8AFfQ1x4PAMC1YeUDAABYRfgAAABWET4AAIBV7PkA/g/7OwDADlY+AACAVYQPAABgFeEDAABYRfgAAABWseEUuEb9uUEdAIxkrHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKxKiPYJu3fv1osvvqj9+/fr1KlT2r59ux577DHnvDFGa9eu1caNG9Xe3q6srCy99NJLmjx58kDWDQxbaat3fu6c4+sLLFQCAMNT1Csfn376qe6++25VVFT0eX7Dhg0qKytTRUWFGhoa5Pf7lZeXp87OzmsuFgAAxL6oVz5mzZqlWbNm9XnOGKPy8nIVFxdr7ty5kqSqqir5fD5VV1dr0aJF11YtAACIeQO656O5uVmhUEj5+fnOmNvtVk5Ojurr6/t8TldXlzo6OiIOAAAQv6Je+biSUCgkSfL5fBHjPp9PLS0tfT6ntLRUa9euHcgygLh16X4S9o4AiEWDcrWLy+WKeGyM6TV2QVFRkcLhsHO0trYORkkAAGCYGNCVD7/fL+n8CkhKSooz3tbW1ms15AK32y232z2QZQAAgGFsQFc+0tPT5ff7VVtb64x1d3errq5O2dnZA/lWAAAgRkW98vHJJ5/o17/+tfO4ublZhw4d0rhx4zRx4kQVFhYqGAwqIyNDGRkZCgaDSkpK0rx58wa0cCDeXc33hfQ1h30gAIa7qMPHO++8o4ceesh5vGLFCknSggUL9E//9E9auXKlzp49qyVLljhfMlZTUyOPxzNwVQMAgJgVdfjIzc2VMeay510ul0pKSlRSUnItdQEAgDjFvV0AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYNWAfr36SHc1XwoFXA5/fwCMFKx8AAAAqwgfAADAKsIHAACwij0fV4nP44HeLv3vgpvaAbgarHwAAACrCB8AAMAqwgcAALBqxO356GvvBp9Twzabe4hs/p3nvy8AV4OVDwAAYBXhAwAAWEX4AAAAVo24PR9Xg+/0QCy7mr+/Nr+fg+8CAXApVj4AAIBVhA8AAGAV4QMAAFhF+AAAAFax4RTAVRmojdgD+UVkbGYFYhMrHwAAwCrCBwAAsIrwAQAArGLPB4A+DfXN7y41UPtCrua1R9IN8tg3g6HAygcAALCK8AEAAKwifAAAAKsGbc/Hyy+/rBdffFGnTp3S5MmTVV5ergcffHCw3g7ANYiFmyn254Z5tl3N/onB2mMRL/tUBnP/z0gRCz/DQVn5+OlPf6rCwkIVFxfr4MGDevDBBzVr1iydOHFiMN4OAADEkEEJH2VlZfqLv/gLffvb39add96p8vJypaamqrKycjDeDgAAxJAB/9ilu7tb+/fv1+rVqyPG8/PzVV9f32t+V1eXurq6nMfhcFiS1NHRMdClSZJ6us70Grv0vfqaAyB+Xc2/AVfzb9Klz+vrOVczpz8Gs2abrubf36Gucbgbqp/hhdc0xnz+ZDPATp48aSSZ//zP/4wYX7dunbn99tt7zV+zZo2RxMHBwcHBwREHR2tr6+dmhUHbcOpyuSIeG2N6jUlSUVGRVqxY4Tzu6elRS0uL7rnnHrW2tio5OXmwShyWOjo6lJqaOuJ6H6l9S/Q+EnsfqX1LI7f3kdC3MUadnZ0KBAKfO3fAw8fNN9+s6667TqFQKGK8ra1NPp+v13y32y232x0xNmrU+a0oycnJcfuH9HlGau8jtW+J3kdi7yO1b2nk9h7vfXu93quaN+AbTseMGaPp06ertrY2Yry2tlbZ2dkD/XYAACDGDMrHLitWrNBTTz2lGTNmaObMmdq4caNOnDihxYsXD8bbAQCAGDIo4eMb3/iGPv74Yz3//PM6deqUMjMz9bOf/UyTJk26que73W6tWbOm18cxI8FI7X2k9i3R+0jsfaT2LY3c3kdq35fjMuZqrokBAAAYGNzbBQAAWEX4AAAAVhE+AACAVYQPAABg1bAMHy+//LLS09N1/fXXa/r06fqP//iPoS7JsXv3bs2ZM0eBQEAul0uvv/56xHljjEpKShQIBJSYmKjc3Fw1NTVFzOnq6tIzzzyjm2++WWPHjtUf//Ef6ze/+U3EnPb2dj311FPyer3yer166qmndPr06Yg5J06c0Jw5czR27FjdfPPNevbZZ9Xd3R0xp7GxUTk5OUpMTNQXvvAFPf/881f3vfuXKC0t1b333iuPx6Px48frscce05EjR+K+98rKSk2dOtX5YqCZM2fq5z//eVz3fDmlpaVyuVwqLCyM+/5LSkrkcrkiDr/fH/d9S9LJkyf1zW9+UzfddJOSkpJ0zz33aP/+/XHfe1paWq8/c5fLpaVLl8Z130Omn7dwGTRbt241o0ePNps2bTKHDx82y5cvN2PHjjUtLS1DXZoxxpif/exnpri42Lz22mtGktm+fXvE+fXr1xuPx2Nee+0109jYaL7xjW+YlJQU09HR4cxZvHix+cIXvmBqa2vNgQMHzEMPPWTuvvtu89lnnzlz/uiP/shkZmaa+vp6U19fbzIzM83s2bOd85999pnJzMw0Dz30kDlw4ICpra01gUDALFu2zJkTDoeNz+czTzzxhGlsbDSvvfaa8Xg85vvf/37UfT/66KNm8+bN5r333jOHDh0yBQUFZuLEieaTTz6J69537Nhhdu7caY4cOWKOHDlinnvuOTN69Gjz3nvvxW3Pfdm3b59JS0szU6dONcuXL3fG47X/NWvWmMmTJ5tTp045R1tbW9z3/b//+79m0qRJ5umnnza/+tWvTHNzs9m1a5f59a9/Hfe9t7W1Rfx519bWGknmrbfeiuu+h8qwCx9/+Id/aBYvXhwxdscdd5jVq1cPUUWXd2n46OnpMX6/36xfv94Z+93vfme8Xq/58Y9/bIwx5vTp02b06NFm69atzpyTJ0+aUaNGmTfeeMMYY8zhw4eNJLN3715nzp49e4wk89///d/GmPMhaNSoUebkyZPOnJ/85CfG7XabcDhsjDHm5ZdfNl6v1/zud79z5pSWlppAIGB6enquqfe2tjYjydTV1Y243m+88Ubzj//4jyOm587OTpORkWFqa2tNTk6OEz7iuf81a9aYu+++u89z8dz3qlWrzAMPPHDZ8/Hc+6WWL19ubrvtNtPT0zOi+rZlWH3s0t3drf379ys/Pz9iPD8/X/X19UNU1dVrbm5WKBSKqN/tdisnJ8epf//+/fr9738fMScQCCgzM9OZs2fPHnm9XmVlZTlz7rvvPnm93og5mZmZETfwefTRR9XV1eUske7Zs0c5OTkRX2rz6KOP6sMPP9Tx48evqddwOCxJGjdu3Ijp/dy5c9q6das+/fRTzZw5c0T0LElLly5VQUGBHnnkkYjxeO//6NGjCgQCSk9P1xNPPKFjx47Ffd87duzQjBkz9Pjjj2v8+PGaNm2aNm3a5JyP594v1t3drS1btmjhwoVyuVwjpm+bhlX4+O1vf6tz5871ugGdz+frdaO64ehCjVeqPxQKacyYMbrxxhuvOGf8+PG9Xn/8+PERcy59nxtvvFFjxoy54pwLj6/l52mM0YoVK/TAAw8oMzMz4vXisffGxkbdcMMNcrvdWrx4sbZv36677rorrnu+YOvWrTpw4IBKS0t7nYvn/rOysvTqq6/qzTff1KZNmxQKhZSdna2PP/44rvs+duyYKisrlZGRoTfffFOLFy/Ws88+q1dffTXi9eKx94u9/vrrOn36tJ5++umI14r3vm0alK9Xv1YulyvisTGm19hw1p/6L53T1/yBmGP+b0PStfw8ly1bpnfffVe//OUve52Lx96/9KUv6dChQzp9+rRee+01LViwQHV1dVd8n1jvWZJaW1u1fPly1dTU6Prrr7/svHjsf9asWc7/njJlimbOnKnbbrtNVVVVuu+++y77XrHed09Pj2bMmKFgMChJmjZtmpqamlRZWalvfetbV3y/WO/9Yq+88opmzZrV69bw8d63TcNq5ePmm2/Wdddd1yu5tbW19Up5w9GF3fBXqt/v96u7u1vt7e1XnPPRRx/1ev3/+Z//iZhz6fu0t7fr97///RXntLW1Seqd4K/WM888ox07duitt97ShAkTnPF47n3MmDH64he/qBkzZqi0tFR33323fvCDH8R1z9L5ZeS2tjZNnz5dCQkJSkhIUF1dnX74wx8qISHhsv9PK176v9jYsWM1ZcoUHT16NK7/3FNSUnTXXXdFjN155506ceKE815SfPZ+QUtLi3bt2qVvf/vbzthI6Nu2YRU+xowZo+nTp6u2tjZivLa2VtnZ2UNU1dVLT0+X3++PqL+7u1t1dXVO/dOnT9fo0aMj5pw6dUrvvfeeM2fmzJkKh8Pat2+fM+dXv/qVwuFwxJz33ntPp06dcubU1NTI7XZr+vTpzpzdu3dHXKJVU1OjQCCgtLS0qHozxmjZsmXatm2bfvGLXyg9PX3E9N7Xz6Krqyvue3744YfV2NioQ4cOOceMGTM0f/58HTp0SLfeemtc93+xrq4uvf/++0pJSYnrP/f777+/1yX0H3zwgXNT0Hju/YLNmzdr/PjxKigocMZGQt/WDeJm1n65cKntK6+8Yg4fPmwKCwvN2LFjzfHjx4e6NGPM+Z3/Bw8eNAcPHjSSTFlZmTl48KBzKfD69euN1+s127ZtM42NjebJJ5/s83KsCRMmmF27dpkDBw6Yr371q31ejjV16lSzZ88es2fPHjNlypQ+L8d6+OGHzYEDB8yuXbvMhAkTIi7HOn36tPH5fObJJ580jY2NZtu2bSY5Oblfl2N95zvfMV6v17z99tsRl6OdOXPGmROPvRcVFZndu3eb5uZm8+6775rnnnvOjBo1ytTU1MRtz1dy8dUu8dz/X//1X5u3337bHDt2zOzdu9fMnj3beDwe59+heO173759JiEhwaxbt84cPXrU/PM//7NJSkoyW7ZscebEa+/GGHPu3DkzceJEs2rVql7n4rnvoTDswocxxrz00ktm0qRJZsyYMebLX/6ycznncPDWW28ZSb2OBQsWGGPOX4q2Zs0a4/f7jdvtNl/5yldMY2NjxGucPXvWLFu2zIwbN84kJiaa2bNnmxMnTkTM+fjjj838+fONx+MxHo/HzJ8/37S3t0fMaWlpMQUFBSYxMdGMGzfOLFu2LOLSK2OMeffdd82DDz5o3G638fv9pqSkpF+XYvXVsySzefNmZ0489r5w4ULn7+Itt9xiHn74YSd4xGvPV3Jp+IjX/i98h8Po0aNNIBAwc+fONU1NTXHftzHG/Ou//qvJzMw0brfb3HHHHWbjxo0R5+O59zfffNNIMkeOHOl1Lp77HgouY2LpK9EAAECsG1Z7PgAAQPwjfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALDq/wH3ipdEUuUoVQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(y_train, bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A common trick in such cases is applying a log transform on the target column to make it more normal and less skewed.  \n",
    "- That is, transform $y\\rightarrow \\log(y)$.\n",
    "- Linear regression will usually work better on something that looks more normal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAfvUlEQVR4nO3df3BU1f3/8ddKdEk0gUphNykRoi6OiqglNhLrJ6mSdJDSdtCiBhXb6kCDPyI6lDR/sDi4wUxN0zaKxbEYRyOdsaK2KiZtxyCmaQM2LRMcpRIgCksGTZPwo0kj5/uHX1Y3CZAbds+yy/Mxc2e6597dfefd5PLy7D17XcYYIwAAAEvOiHUBAADg9EL4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGBVUqwLGOjIkSPas2ePUlNT5XK5Yl0OAAAYBmOMenp6lJGRoTPOOP7cxikXPvbs2aPMzMxYlwEAAEagvb1dEydOPO4xp1z4SE1NlfR58WlpaTGuBgAADEd3d7cyMzND/44fzykXPo5+1JKWlkb4AAAgzgznkgkuOAUAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFVJsS4AQOxNXvbaoLGdq2bHoBIApwNmPgAAgFWOw8fHH3+s2267TePGjVNKSoquuOIKbdmyJbTfGCO/36+MjAwlJycrPz9fra2tES0aAADEL0fho7OzU9dcc43OPPNMvfHGG9q2bZsee+wxjR07NnRMRUWFKisrVV1drebmZnm9XhUUFKinpyfStQMAgDjk6JqPRx99VJmZmVq7dm1obPLkyaH/bYxRVVWVysrKNHfuXElSTU2NPB6PamtrtXDhwshUDQAA4pajmY9XX31V2dnZ+sEPfqAJEyboyiuv1FNPPRXa39bWpmAwqMLCwtCY2+1WXl6eGhsbh3zN3t5edXd3h20AACBxOQofO3bs0OrVq+Xz+fTmm29q0aJFuu+++/Tss89KkoLBoCTJ4/GEPc/j8YT2DVReXq4xY8aEtszMzJH8HAAAIE44Ch9HjhzR17/+dQUCAV155ZVauHCh7r77bq1evTrsOJfLFfbYGDNo7KjS0lJ1dXWFtvb2doc/AgAAiCeOwkd6erouueSSsLGLL75Yu3fvliR5vV5JGjTL0dHRMWg25Ci32620tLSwDQAAJC5H4eOaa67R+++/Hzb2wQcfaNKkSZKkrKwseb1e1dfXh/b39fWpoaFBubm5ESgXAADEO0erXR544AHl5uYqEAho3rx5+vvf/641a9ZozZo1kj7/uKWkpESBQEA+n08+n0+BQEApKSkqKiqKyg8AAADii6PwcdVVV2n9+vUqLS3Vww8/rKysLFVVVWn+/PmhY5YuXarDhw+ruLhYnZ2dysnJUV1dnVJTUyNePAAAiD8uY4yJdRFf1t3drTFjxqirq4vrPwBLuLcLgJPl5N9v7u0CAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArEqKdQEAMNDkZa8NGtu5anYMKgEQDcx8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrWO0CJDhWjgA41TDzAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqR+HD7/fL5XKFbV6vN7TfGCO/36+MjAwlJycrPz9fra2tES8aAADEL8czH5deeqn27t0b2rZu3RraV1FRocrKSlVXV6u5uVler1cFBQXq6emJaNEAACB+OQ4fSUlJ8nq9oW38+PGSPp/1qKqqUllZmebOnaupU6eqpqZGhw4dUm1tbcQLBwAA8clx+Ni+fbsyMjKUlZWlW265RTt27JAktbW1KRgMqrCwMHSs2+1WXl6eGhsbj/l6vb296u7uDtsAAEDicnRvl5ycHD377LOaMmWK9u3bp5UrVyo3N1etra0KBoOSJI/HE/Ycj8ejXbt2HfM1y8vLtWLFihGUDsAm7hEDIFIczXzMmjVLN954oy677DLNnDlTr732+cmopqYmdIzL5Qp7jjFm0NiXlZaWqqurK7S1t7c7KQkAAMSZk1pqe/bZZ+uyyy7T9u3bQ6tejs6AHNXR0TFoNuTL3G630tLSwjYAAJC4Tip89Pb26r333lN6erqysrLk9XpVX18f2t/X16eGhgbl5uaedKEAACAxOLrm46GHHtKcOXN03nnnqaOjQytXrlR3d7cWLFggl8ulkpISBQIB+Xw++Xw+BQIBpaSkqKioKFr1AwCAOOMofHz00Ue69dZbtX//fo0fP15XX321mpqaNGnSJEnS0qVLdfjwYRUXF6uzs1M5OTmqq6tTampqVIoHAADxx1H4WLdu3XH3u1wu+f1++f3+k6kJAAAkMO7tAgAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKscfb06gNPH5GWvxboEAAmKmQ8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWcW8XAHFr4P1ndq6aHaNKADjBzAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACs4uvVAURVpL4CfeDrAIhfzHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKtY7QIgYliRAmA4mPkAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFUnFT7Ky8vlcrlUUlISGjPGyO/3KyMjQ8nJycrPz1dra+vJ1gkAABLEiMNHc3Oz1qxZo2nTpoWNV1RUqLKyUtXV1WpubpbX61VBQYF6enpOulgAABD/RhQ+Dhw4oPnz5+upp57SV77yldC4MUZVVVUqKyvT3LlzNXXqVNXU1OjQoUOqra2NWNEAACB+jSh8LF68WLNnz9bMmTPDxtva2hQMBlVYWBgac7vdysvLU2Nj45Cv1dvbq+7u7rANAAAkLsdfr75u3Tq9++67am5uHrQvGAxKkjweT9i4x+PRrl27hny98vJyrVixwmkZADDIUF/vvnPV7BhUAuB4HM18tLe36/7779dzzz2n0aNHH/M4l8sV9tgYM2jsqNLSUnV1dYW29vZ2JyUBAIA442jmY8uWLero6ND06dNDY5999pk2btyo6upqvf/++5I+nwFJT08PHdPR0TFoNuQot9stt9s9ktoBAEAccjTzcf3112vr1q1qaWkJbdnZ2Zo/f75aWlp0/vnny+v1qr6+PvScvr4+NTQ0KDc3N+LFAwCA+ONo5iM1NVVTp04NGzv77LM1bty40HhJSYkCgYB8Pp98Pp8CgYBSUlJUVFQUuaoBAEDccnzB6YksXbpUhw8fVnFxsTo7O5WTk6O6ujqlpqZG+q0AAEAcOunw8dZbb4U9drlc8vv98vv9J/vSAAAgAXFvFwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgVcSX2gI49Q11DxQAsIWZDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFatdAMQcq2+A0wszHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCKr1cHYBVfpQ6AmQ8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBWrXQCMGCtXAIwEMx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCpWuwBAhAxc/bNz1ewYVQKc2pj5AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWsdoFOEWwUiI66Ctw6mHmAwAAWEX4AAAAVhE+AACAVYQPAABgFRecAnFkOBdPDjwGAE41zHwAAACrHIWP1atXa9q0aUpLS1NaWppmzJihN954I7TfGCO/36+MjAwlJycrPz9fra2tES8aAADEL0fhY+LEiVq1apU2b96szZs367rrrtP3vve9UMCoqKhQZWWlqqur1dzcLK/Xq4KCAvX09ESleAAAEH8chY85c+bohhtu0JQpUzRlyhQ98sgjOuecc9TU1CRjjKqqqlRWVqa5c+dq6tSpqqmp0aFDh1RbWxut+gEAQJwZ8TUfn332mdatW6eDBw9qxowZamtrUzAYVGFhYegYt9utvLw8NTY2HvN1ent71d3dHbYBAIDE5Th8bN26Veecc47cbrcWLVqk9evX65JLLlEwGJQkeTyesOM9Hk9o31DKy8s1ZsyY0JaZmem0JAAAEEcch4+LLrpILS0tampq0k9+8hMtWLBA27ZtC+13uVxhxxtjBo19WWlpqbq6ukJbe3u705IAAEAccfw9H2eddZYuvPBCSVJ2draam5v1y1/+Uj/96U8lScFgUOnp6aHjOzo6Bs2GfJnb7Zbb7XZaBgAAiFMn/T0fxhj19vYqKytLXq9X9fX1oX19fX1qaGhQbm7uyb4NAABIEI5mPn72s59p1qxZyszMVE9Pj9atW6e33npLGzZskMvlUklJiQKBgHw+n3w+nwKBgFJSUlRUVBSt+gEAQJxxFD727dun22+/XXv37tWYMWM0bdo0bdiwQQUFBZKkpUuX6vDhwyouLlZnZ6dycnJUV1en1NTUqBQPAADij6Pw8fTTTx93v8vlkt/vl9/vP5maAAwT93EBEI+4twsAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACscvT16gAQ74b6Svqdq2Y7ft5wngNgaMx8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrWO0CACMw1KoZAMPDzAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrVLgBOe9y3BbCLmQ8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWcW8XABhg4L1eovna3EcGpyNmPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFY5Ch/l5eW66qqrlJqaqgkTJuj73/++3n///bBjjDHy+/3KyMhQcnKy8vPz1draGtGiAQBA/HIUPhoaGrR48WI1NTWpvr5e/f39Kiws1MGDB0PHVFRUqLKyUtXV1WpubpbX61VBQYF6enoiXjwAAIg/SU4O3rBhQ9jjtWvXasKECdqyZYv+7//+T8YYVVVVqaysTHPnzpUk1dTUyOPxqLa2VgsXLoxc5QAAIC6d1DUfXV1dkqRzzz1XktTW1qZgMKjCwsLQMW63W3l5eWpsbBzyNXp7e9Xd3R22AQCAxOVo5uPLjDFasmSJvvnNb2rq1KmSpGAwKEnyeDxhx3o8Hu3atWvI1ykvL9eKFStGWgYQlyYvey3WJSCODPX7snPV7BhUAkTGiGc+7rnnHv3rX//SCy+8MGify+UKe2yMGTR2VGlpqbq6ukJbe3v7SEsCAABxYEQzH/fee69effVVbdy4URMnTgyNe71eSZ/PgKSnp4fGOzo6Bs2GHOV2u+V2u0dSBgAAiEOOZj6MMbrnnnv00ksv6S9/+YuysrLC9mdlZcnr9aq+vj401tfXp4aGBuXm5kamYgAAENcczXwsXrxYtbW1euWVV5Samhq6xmPMmDFKTk6Wy+VSSUmJAoGAfD6ffD6fAoGAUlJSVFRUFJUfAAAAxBdH4WP16tWSpPz8/LDxtWvX6s4775QkLV26VIcPH1ZxcbE6OzuVk5Ojuro6paamRqRgAAAQ3xyFD2PMCY9xuVzy+/3y+/0jrQmAWBFzumAlC05H3NsFAABYRfgAAABWET4AAIBVhA8AAGDViL9eHQAQHVxsjETHzAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsCop1gUAiWbystcGje1cNTsGlSCRDfw943cM8YSZDwAAYBXhAwAAWEX4AAAAVhE+AACAVVxwCgCniaEuhh6IC1dhAzMfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArOLeLjgtDHVPC+5hAQCxwcwHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK1S7A/zfUipiBWCEDACfP8czHxo0bNWfOHGVkZMjlcunll18O22+Mkd/vV0ZGhpKTk5Wfn6/W1tZI1QsAAOKc4/Bx8OBBXX755aqurh5yf0VFhSorK1VdXa3m5mZ5vV4VFBSop6fnpIsFAADxz/HHLrNmzdKsWbOG3GeMUVVVlcrKyjR37lxJUk1NjTwej2pra7Vw4cKTqxYAAMS9iF5w2tbWpmAwqMLCwtCY2+1WXl6eGhsbI/lWAAAgTkX0gtNgMChJ8ng8YeMej0e7du0a8jm9vb3q7e0NPe7u7o5kSQAA4BQTldUuLpcr7LExZtDYUeXl5VqxYkU0ysBpYjirVGL9/rGuEbHB/+/hBvaD1WOnr4h+7OL1eiV9MQNyVEdHx6DZkKNKS0vV1dUV2trb2yNZEgAAOMVENHxkZWXJ6/Wqvr4+NNbX16eGhgbl5uYO+Ry32620tLSwDQAAJC7HH7scOHBA//73v0OP29ra1NLSonPPPVfnnXeeSkpKFAgE5PP55PP5FAgElJKSoqKioogWDgAA4pPj8LF582Z961vfCj1esmSJJGnBggV65plntHTpUh0+fFjFxcXq7OxUTk6O6urqlJqaGrmqAQBA3HIcPvLz82WMOeZ+l8slv98vv99/MnUBAOIYF9vieLixHAAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwKipfrw7EA67GR6KL1O84X4uOSGPmAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYxWoXwAFWyADAyWPmAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYxWoXAEgArMRCPGHmAwAAWEX4AAAAVhE+AACAVYQPAABgFRecAgBCuHAVNjDzAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsYrULrBl4Ff3OVbNjVAmAeDGc1TecS+IPMx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCpWu+CETrVVKtx7AkgMQ/0tx/r8AjuY+QAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVp12q11O56urI/Wz2+whK1uAU080/y4j9dqn2iq9gWJ9Ho11P5j5AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWnXarXYZjpFdbx/Lq4ZFezRytK8tj/ToATh/DOW8M55ihzpkjOSfFeiVJPIjazMcTTzyhrKwsjR49WtOnT9fbb78drbcCAABxJCrh43e/+51KSkpUVlamf/zjH7r22ms1a9Ys7d69OxpvBwAA4khUwkdlZaV+/OMf66677tLFF1+sqqoqZWZmavXq1dF4OwAAEEcifs1HX1+ftmzZomXLloWNFxYWqrGxcdDxvb296u3tDT3u6uqSJHV3d0e6NEnSkd5Dg8YGvtdQxwxHtGoejmj+XNEyVL9OtRoBnD4idU4azr8FwzlnR4qt9zr6msaYEx9sIuzjjz82ksw777wTNv7II4+YKVOmDDp++fLlRhIbGxsbGxtbAmzt7e0nzApRW+3icrnCHhtjBo1JUmlpqZYsWRJ6fOTIEX366acaN27ckMefKrq7u5WZman29nalpaXFupyYox9foBfh6Ec4+vEFehEu3vthjFFPT48yMjJOeGzEw8dXv/pVjRo1SsFgMGy8o6NDHo9n0PFut1tutztsbOzYsZEuK2rS0tLi8pckWujHF+hFOPoRjn58gV6Ei+d+jBkzZljHRfyC07POOkvTp09XfX192Hh9fb1yc3Mj/XYAACDOROVjlyVLluj2229Xdna2ZsyYoTVr1mj37t1atGhRNN4OAADEkaiEj5tvvlmffPKJHn74Ye3du1dTp07V66+/rkmTJkXj7WLC7XZr+fLlgz4yOl3Rjy/Qi3D0Ixz9+AK9CHc69cNlzHDWxAAAAEQGN5YDAABWET4AAIBVhA8AAGAV4QMAAFhF+BiG8vJyuVwulZSUHPe43t5elZWVadKkSXK73brgggv029/+1k6Rlgy3F88//7wuv/xypaSkKD09XT/84Q/1ySef2Ckyivx+v1wuV9jm9XqP+5yGhgZNnz5do0eP1vnnn68nn3zSUrXR57QfL730kgoKCjR+/HilpaVpxowZevPNNy1WHF0j+f046p133lFSUpKuuOKK6BZpyUh6kcjn0JH0I1HPo1KUltomkubmZq1Zs0bTpk074bHz5s3Tvn379PTTT+vCCy9UR0eH+vv7LVRpx3B7sWnTJt1xxx36xS9+oTlz5ujjjz/WokWLdNddd2n9+vWWqo2eSy+9VH/6059Cj0eNGnXMY9va2nTDDTfo7rvv1nPPPad33nlHxcXFGj9+vG688UYb5Uadk35s3LhRBQUFCgQCGjt2rNauXas5c+bob3/7m6688kob5Uadk34c1dXVpTvuuEPXX3+99u3bF83yrHLai0Q/hzrpR6KfRwkfx3HgwAHNnz9fTz31lFauXHncYzds2KCGhgbt2LFD5557riRp8uTJFqq0w0kvmpqaNHnyZN13332SpKysLC1cuFAVFRU2So26pKSkYf/X7JNPPqnzzjtPVVVVkqSLL75Ymzdv1s9//vOECR9O+nG0D0cFAgG98sor+sMf/pAw4cNJP45auHChioqKNGrUKL388svRKSwGnPQi0c+hkrN+JPp5lI9djmPx4sWaPXu2Zs6cecJjX331VWVnZ6uiokJf+9rXNGXKFD300EM6fPiwhUqjz0kvcnNz9dFHH+n111+XMUb79u3Tiy++qNmzZ1uoNPq2b9+ujIwMZWVl6ZZbbtGOHTuOeexf//pXFRYWho19+9vf1ubNm/W///0v2qVa4aQfAx05ckQ9PT2hf2wSgdN+rF27Vh9++KGWL19uqUJ7nPQi0c+hkrN+JPp5lJmPY1i3bp3effddNTc3D+v4HTt2aNOmTRo9erTWr1+v/fv3q7i4WJ9++mncf2bptBe5ubl6/vnndfPNN+u///2v+vv79d3vfle//vWvo1xp9OXk5OjZZ5/VlClTtG/fPq1cuVK5ublqbW3VuHHjBh0fDAYH3VDR4/Gov79f+/fvV3p6uq3So8JpPwZ67LHHdPDgQc2bN89CtdHntB/bt2/XsmXL9PbbbyspKbFOx057kcjnUMl5PxL5PCpJMhhk9+7dZsKECaalpSU0lpeXZ+6///5jPqegoMCMHj3a/Oc//wmN/f73vzcul8scOnQomuVG1Uh60draatLT001FRYX55z//aTZs2GAuu+wy86Mf/chCxXYdOHDAeDwe89hjjw253+fzmUAgEDa2adMmI8ns3bvXRolWnagfX1ZbW2tSUlJMfX29hcpi43j96O/vN9nZ2Wb16tWhseXLl5vLL7/cYoX2nOh3I1HPocdyon4k+nmU8DGE9evXG0lm1KhRoU2ScblcZtSoUaa/v3/Qc+644w5zwQUXhI1t27bNSDIffPCBrdIjbiS9uO2228xNN90UNvb2228bSWbPnj22Srdm5syZZtGiRUPuu/baa819990XNvbSSy+ZpKQk09fXZ6M8647Xj6PWrVtnkpOTzR//+EdLVcXOsfrR2dk56G/L5XKFxv785z/HoNroOt7vRqKeQ4/neP1I9PMo13wM4frrr9fWrVvV0tIS2rKzszV//ny1tLQMeYXyNddcoz179ujAgQOhsQ8++EBnnHGGJk6caLP8iBpJLw4dOqQzzgj/1Tp6nEmwWwn19vbqvffeO+bHJzNmzFB9fX3YWF1dnbKzs3XmmWfaKNGqE/VDkl544QXdeeedqq2tTZjPr4/leP1IS0sb9Le1aNEiXXTRRWppaVFOTk4MKo6eE/1uJOo59FhO1I+EP4/GOv3Ei4EfNSxbtszcfvvtocc9PT1m4sSJ5qabbjKtra2moaHB+Hw+c9ddd8Wg2ug6US/Wrl1rkpKSzBNPPGE+/PBDs2nTJpOdnW2+8Y1vxKDayHrwwQfNW2+9ZXbs2GGamprMd77zHZOammp27txpjBncix07dpiUlBTzwAMPmG3btpmnn37anHnmmebFF1+M1Y8QUU77UVtba5KSkszjjz9u9u7dG9q+PNUez5z2Y6BE+tjFaS8S/RzqtB+JfB41xpjEusLJor1792r37t2hx+ecc47q6+t17733Kjs7W+PGjdO8efNOuCw1EQzsxZ133qmenh5VV1frwQcf1NixY3Xdddfp0UcfjWGVkfHRRx/p1ltv1f79+zV+/HhdffXVampq0qRJkyQN7kVWVpZef/11PfDAA3r88ceVkZGhX/3qVwmzzNZpP37zm9+ov79fixcv1uLFi0PjCxYs0DPPPGO7/Ihz2o9E5rQXiX4OddqPRD6PSpLLmESYvwEAAPGCaz4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABW/T/+/UoFrBsKIAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(np.log10(y_train), bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can incorporate this in our pipeline using `sklearn`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import TransformedTargetRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "ttr = TransformedTargetRegressor(\n",
    "    Ridge(alpha=best_alpha), func=np.log1p, inverse_func=np.expm1\n",
    ") # transformer for log transforming the target\n",
    "ttr_pipe = make_pipeline(preprocessor, ttr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;Id&#x27;]),\n",
       "                                                 (&#x27;pipeline-1&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;BedroomAbvGr&#x27;,\n",
       "                                                   &#x27;KitchenAbvGr&#x27;,\n",
       "                                                   &#x27;LotFrontage&#x27;, &#x27;LotArea&#x27;,\n",
       "                                                   &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;,\n",
       "                                                   &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;,\n",
       "                                                   &#x27;MasVnrArea&#x27;, &#x27;BsmtFinSF1&#x27;,\n",
       "                                                   &#x27;Bs...\n",
       "                                                   &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;,\n",
       "                                                   &#x27;Condition2&#x27;, &#x27;MasVnrType&#x27;,\n",
       "                                                   &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;,\n",
       "                                                   &#x27;LandContour&#x27;, &#x27;BldgType&#x27;,\n",
       "                                                   &#x27;Electrical&#x27;, &#x27;MSSubClass&#x27;,\n",
       "                                                   &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;,\n",
       "                                                   &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;,\n",
       "                                                   &#x27;MiscFeature&#x27;, &#x27;HouseStyle&#x27;,\n",
       "                                                   &#x27;Heating&#x27;, &#x27;MoSold&#x27;])])),\n",
       "                (&#x27;transformedtargetregressor&#x27;,\n",
       "                 TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;,\n",
       "                                            inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                                            regressor=Ridge(alpha=100.0)))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;Id&#x27;]),\n",
       "                                                 (&#x27;pipeline-1&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;BedroomAbvGr&#x27;,\n",
       "                                                   &#x27;KitchenAbvGr&#x27;,\n",
       "                                                   &#x27;LotFrontage&#x27;, &#x27;LotArea&#x27;,\n",
       "                                                   &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;,\n",
       "                                                   &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;,\n",
       "                                                   &#x27;MasVnrArea&#x27;, &#x27;BsmtFinSF1&#x27;,\n",
       "                                                   &#x27;Bs...\n",
       "                                                   &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;,\n",
       "                                                   &#x27;Condition2&#x27;, &#x27;MasVnrType&#x27;,\n",
       "                                                   &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;,\n",
       "                                                   &#x27;LandContour&#x27;, &#x27;BldgType&#x27;,\n",
       "                                                   &#x27;Electrical&#x27;, &#x27;MSSubClass&#x27;,\n",
       "                                                   &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;,\n",
       "                                                   &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;,\n",
       "                                                   &#x27;MiscFeature&#x27;, &#x27;HouseStyle&#x27;,\n",
       "                                                   &#x27;Heating&#x27;, &#x27;MoSold&#x27;])])),\n",
       "                (&#x27;transformedtargetregressor&#x27;,\n",
       "                 TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;,\n",
       "                                            inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                                            regressor=Ridge(alpha=100.0)))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;Id&#x27;]),\n",
       "                                (&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;BedroomAbvGr&#x27;, &#x27;KitchenAbvGr&#x27;, &#x27;LotFrontage&#x27;,\n",
       "                                  &#x27;LotArea&#x27;, &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;,\n",
       "                                  &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;, &#x27;MasVnrArea&#x27;,\n",
       "                                  &#x27;BsmtFinSF1&#x27;, &#x27;BsmtFinSF2&#x27;, &#x27;BsmtUnfSF&#x27;,\n",
       "                                  &#x27;TotalBsmtSF&#x27;, &#x27;...\n",
       "                                 [&#x27;SaleType&#x27;, &#x27;Neighborhood&#x27;, &#x27;Foundation&#x27;,\n",
       "                                  &#x27;Street&#x27;, &#x27;Exterior2nd&#x27;, &#x27;RoofStyle&#x27;,\n",
       "                                  &#x27;MSZoning&#x27;, &#x27;PavedDrive&#x27;, &#x27;Utilities&#x27;,\n",
       "                                  &#x27;LotConfig&#x27;, &#x27;SaleCondition&#x27;, &#x27;LandSlope&#x27;,\n",
       "                                  &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;, &#x27;Condition2&#x27;,\n",
       "                                  &#x27;MasVnrType&#x27;, &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;,\n",
       "                                  &#x27;LandContour&#x27;, &#x27;BldgType&#x27;, &#x27;Electrical&#x27;,\n",
       "                                  &#x27;MSSubClass&#x27;, &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;,\n",
       "                                  &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;, &#x27;MiscFeature&#x27;,\n",
       "                                  &#x27;HouseStyle&#x27;, &#x27;Heating&#x27;, &#x27;MoSold&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Id&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;BedroomAbvGr&#x27;, &#x27;KitchenAbvGr&#x27;, &#x27;LotFrontage&#x27;, &#x27;LotArea&#x27;, &#x27;OverallQual&#x27;, &#x27;OverallCond&#x27;, &#x27;YearBuilt&#x27;, &#x27;YearRemodAdd&#x27;, &#x27;MasVnrArea&#x27;, &#x27;BsmtFinSF1&#x27;, &#x27;BsmtFinSF2&#x27;, &#x27;BsmtUnfSF&#x27;, &#x27;TotalBsmtSF&#x27;, &#x27;1stFlrSF&#x27;, &#x27;2ndFlrSF&#x27;, &#x27;LowQualFinSF&#x27;, &#x27;GrLivArea&#x27;, &#x27;BsmtFullBath&#x27;, &#x27;BsmtHalfBath&#x27;, &#x27;FullBath&#x27;, &#x27;HalfBath&#x27;, &#x27;TotRmsAbvGrd&#x27;, &#x27;Fireplaces&#x27;, &#x27;GarageYrBlt&#x27;, &#x27;GarageCars&#x27;, &#x27;GarageArea&#x27;, &#x27;WoodDeckSF&#x27;, &#x27;OpenPorchSF&#x27;, &#x27;EnclosedPorch&#x27;, &#x27;3SsnPorch&#x27;, &#x27;ScreenPorch&#x27;, &#x27;PoolArea&#x27;, &#x27;MiscVal&#x27;, &#x27;YrSold&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ExterQual&#x27;, &#x27;ExterCond&#x27;, &#x27;BsmtQual&#x27;, &#x27;BsmtCond&#x27;, &#x27;HeatingQC&#x27;, &#x27;KitchenQual&#x27;, &#x27;FireplaceQu&#x27;, &#x27;GarageQual&#x27;, &#x27;GarageCond&#x27;, &#x27;PoolQC&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-25\" type=\"checkbox\" ><label for=\"sk-estimator-id-25\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;],\n",
       "                           [&#x27;Po&#x27;, &#x27;Fa&#x27;, &#x27;TA&#x27;, &#x27;Gd&#x27;, &#x27;Ex&#x27;]])</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-26\" type=\"checkbox\" ><label for=\"sk-estimator-id-26\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;BsmtExposure&#x27;, &#x27;BsmtFinType1&#x27;, &#x27;BsmtFinType2&#x27;, &#x27;Functional&#x27;, &#x27;Fence&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-27\" type=\"checkbox\" ><label for=\"sk-estimator-id-27\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-28\" type=\"checkbox\" ><label for=\"sk-estimator-id-28\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;NA&#x27;, &#x27;No&#x27;, &#x27;Mn&#x27;, &#x27;Av&#x27;, &#x27;Gd&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;Unf&#x27;, &#x27;LwQ&#x27;, &#x27;Rec&#x27;, &#x27;BLQ&#x27;, &#x27;ALQ&#x27;, &#x27;GLQ&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;Unf&#x27;, &#x27;LwQ&#x27;, &#x27;Rec&#x27;, &#x27;BLQ&#x27;, &#x27;ALQ&#x27;, &#x27;GLQ&#x27;],\n",
       "                           [&#x27;Sal&#x27;, &#x27;Sev&#x27;, &#x27;Maj2&#x27;, &#x27;Maj1&#x27;, &#x27;Mod&#x27;, &#x27;Min2&#x27;, &#x27;Min1&#x27;,\n",
       "                            &#x27;Typ&#x27;],\n",
       "                           [&#x27;NA&#x27;, &#x27;MnWw&#x27;, &#x27;GdWo&#x27;, &#x27;MnPrv&#x27;, &#x27;GdPrv&#x27;]])</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;SaleType&#x27;, &#x27;Neighborhood&#x27;, &#x27;Foundation&#x27;, &#x27;Street&#x27;, &#x27;Exterior2nd&#x27;, &#x27;RoofStyle&#x27;, &#x27;MSZoning&#x27;, &#x27;PavedDrive&#x27;, &#x27;Utilities&#x27;, &#x27;LotConfig&#x27;, &#x27;SaleCondition&#x27;, &#x27;LandSlope&#x27;, &#x27;LotShape&#x27;, &#x27;GarageFinish&#x27;, &#x27;Condition2&#x27;, &#x27;MasVnrType&#x27;, &#x27;Exterior1st&#x27;, &#x27;GarageType&#x27;, &#x27;LandContour&#x27;, &#x27;BldgType&#x27;, &#x27;Electrical&#x27;, &#x27;MSSubClass&#x27;, &#x27;CentralAir&#x27;, &#x27;Condition1&#x27;, &#x27;RoofMatl&#x27;, &#x27;Alley&#x27;, &#x27;MiscFeature&#x27;, &#x27;HouseStyle&#x27;, &#x27;Heating&#x27;, &#x27;MoSold&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-32\" type=\"checkbox\" ><label for=\"sk-estimator-id-32\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">transformedtargetregressor: TransformedTargetRegressor</label><div class=\"sk-toggleable__content\"><pre>TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;, inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                           regressor=Ridge(alpha=100.0))</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-33\" type=\"checkbox\" ><label for=\"sk-estimator-id-33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">regressor: Ridge</label><div class=\"sk-toggleable__content\"><pre>Ridge(alpha=100.0)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-34\" type=\"checkbox\" ><label for=\"sk-estimator-id-34\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Ridge</label><div class=\"sk-toggleable__content\"><pre>Ridge(alpha=100.0)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('drop', 'drop', ['Id']),\n",
       "                                                 ('pipeline-1',\n",
       "                                                  Pipeline(steps=[('simpleimputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('standardscaler',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  ['BedroomAbvGr',\n",
       "                                                   'KitchenAbvGr',\n",
       "                                                   'LotFrontage', 'LotArea',\n",
       "                                                   'OverallQual', 'OverallCond',\n",
       "                                                   'YearBuilt', 'YearRemodAdd',\n",
       "                                                   'MasVnrArea', 'BsmtFinSF1',\n",
       "                                                   'Bs...\n",
       "                                                   'LotShape', 'GarageFinish',\n",
       "                                                   'Condition2', 'MasVnrType',\n",
       "                                                   'Exterior1st', 'GarageType',\n",
       "                                                   'LandContour', 'BldgType',\n",
       "                                                   'Electrical', 'MSSubClass',\n",
       "                                                   'CentralAir', 'Condition1',\n",
       "                                                   'RoofMatl', 'Alley',\n",
       "                                                   'MiscFeature', 'HouseStyle',\n",
       "                                                   'Heating', 'MoSold'])])),\n",
       "                ('transformedtargetregressor',\n",
       "                 TransformedTargetRegressor(func=<ufunc 'log1p'>,\n",
       "                                            inverse_func=<ufunc 'expm1'>,\n",
       "                                            regressor=Ridge(alpha=100.0)))])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ttr_pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why can't we incorporate preprocessing targets in our column transformer? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "ttr_pipe.fit(X_train, y_train); # y_train automatically transformed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([221355.29528077, 170663.43286226, 182608.09768702, ...,\n",
       "       248575.94877669, 132148.9047652 , 133262.17638244])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ttr_pipe.predict(X_train)  # predictions automatically un-transformed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07808600924240865"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_percentage_error(y_test, ttr_pipe.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We reduced MAPE from ~10% to ~8% with this trick! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Does `.fit()` know we care about MAPE?\n",
    "- No, it doesn't. Why are we minimizing MSE (or something similar) if we care about MAPE??\n",
    "- When minimizing MSE, the expensive houses will dominate because they have the biggest error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different scoring functions with `cross_validate`\n",
    "\n",
    "- Let's try using MSE instead of the default $R^2$ score. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.020544</td>\n",
       "      <td>0.005940</td>\n",
       "      <td>-7.060346e+08</td>\n",
       "      <td>-9.383069e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.023720</td>\n",
       "      <td>0.005778</td>\n",
       "      <td>-1.239851e+09</td>\n",
       "      <td>-8.267971e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.017837</td>\n",
       "      <td>0.004770</td>\n",
       "      <td>-1.125125e+09</td>\n",
       "      <td>-8.763019e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.018021</td>\n",
       "      <td>0.004723</td>\n",
       "      <td>-9.819320e+08</td>\n",
       "      <td>-8.847908e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.018081</td>\n",
       "      <td>0.004630</td>\n",
       "      <td>-2.268434e+09</td>\n",
       "      <td>-7.397199e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time    test_score   train_score\n",
       "0  0.020544    0.005940 -7.060346e+08 -9.383069e+08\n",
       "1  0.023720    0.005778 -1.239851e+09 -8.267971e+08\n",
       "2  0.017837    0.004770 -1.125125e+09 -8.763019e+08\n",
       "3  0.018021    0.004723 -9.819320e+08 -8.847908e+08\n",
       "4  0.018081    0.004630 -2.268434e+09 -7.397199e+08"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(\n",
    "    cross_validate(\n",
    "        ridge_tuned,\n",
    "        X_train,\n",
    "        y_train,\n",
    "        return_train_score=True,\n",
    "        scoring=\"neg_mean_squared_error\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.025276</td>\n",
       "      <td>0.005595</td>\n",
       "      <td>-0.096993</td>\n",
       "      <td>-0.104071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.018330</td>\n",
       "      <td>0.005233</td>\n",
       "      <td>-0.108030</td>\n",
       "      <td>-0.099662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.017748</td>\n",
       "      <td>0.004993</td>\n",
       "      <td>-0.118362</td>\n",
       "      <td>-0.101807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.017986</td>\n",
       "      <td>0.004697</td>\n",
       "      <td>-0.107847</td>\n",
       "      <td>-0.102472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.018004</td>\n",
       "      <td>0.005020</td>\n",
       "      <td>-0.121967</td>\n",
       "      <td>-0.098286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.025276    0.005595   -0.096993    -0.104071\n",
       "1  0.018330    0.005233   -0.108030    -0.099662\n",
       "2  0.017748    0.004993   -0.118362    -0.101807\n",
       "3  0.017986    0.004697   -0.107847    -0.102472\n",
       "4  0.018004    0.005020   -0.121967    -0.098286"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a scorer function that we can pass into cross-validation\n",
    "mape_scorer = make_scorer(my_mape, greater_is_better=False)\n",
    "\n",
    "pd.DataFrame(\n",
    "    cross_validate(\n",
    "        ridge_tuned, X_train, y_train, return_train_score=True, scoring=mape_scorer\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are finding `greater_is_better=False` argument confusing, here is the documentation: \n",
    "\n",
    "> greater_is_better(bool), default=True\n",
    "Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the score_func.\n",
    "\n",
    "Since our custom scorer `mape` gives an error and not a score, I'm passing `False` to it and it'll sign flip so that we can interpret bigger numbers as better performance.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ?make_scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>fit_time</th>\n",
       "      <td>2.498984e-02</td>\n",
       "      <td>1.821399e-02</td>\n",
       "      <td>1.756096e-02</td>\n",
       "      <td>1.761317e-02</td>\n",
       "      <td>1.713085e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>score_time</th>\n",
       "      <td>5.598068e-03</td>\n",
       "      <td>5.260229e-03</td>\n",
       "      <td>5.133867e-03</td>\n",
       "      <td>4.958868e-03</td>\n",
       "      <td>5.253315e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_r2</th>\n",
       "      <td>8.668969e-01</td>\n",
       "      <td>8.200460e-01</td>\n",
       "      <td>8.262644e-01</td>\n",
       "      <td>8.511854e-01</td>\n",
       "      <td>6.109505e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_r2</th>\n",
       "      <td>8.551369e-01</td>\n",
       "      <td>8.636241e-01</td>\n",
       "      <td>8.579735e-01</td>\n",
       "      <td>8.561893e-01</td>\n",
       "      <td>8.834054e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_mape_scorer</th>\n",
       "      <td>-9.699277e-02</td>\n",
       "      <td>-1.080304e-01</td>\n",
       "      <td>-1.183620e-01</td>\n",
       "      <td>-1.078469e-01</td>\n",
       "      <td>-1.219672e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_mape_scorer</th>\n",
       "      <td>-1.040712e-01</td>\n",
       "      <td>-9.966190e-02</td>\n",
       "      <td>-1.018073e-01</td>\n",
       "      <td>-1.024720e-01</td>\n",
       "      <td>-9.828607e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_sklearn MAPE</th>\n",
       "      <td>-9.699277e-02</td>\n",
       "      <td>-1.080304e-01</td>\n",
       "      <td>-1.183620e-01</td>\n",
       "      <td>-1.078469e-01</td>\n",
       "      <td>-1.219672e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_sklearn MAPE</th>\n",
       "      <td>-1.040712e-01</td>\n",
       "      <td>-9.966190e-02</td>\n",
       "      <td>-1.018073e-01</td>\n",
       "      <td>-1.024720e-01</td>\n",
       "      <td>-9.828607e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_neg_root_mean_square_error</th>\n",
       "      <td>-2.657131e+04</td>\n",
       "      <td>-3.521152e+04</td>\n",
       "      <td>-3.354288e+04</td>\n",
       "      <td>-3.133579e+04</td>\n",
       "      <td>-4.762808e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_neg_root_mean_square_error</th>\n",
       "      <td>-3.063179e+04</td>\n",
       "      <td>-2.875408e+04</td>\n",
       "      <td>-2.960240e+04</td>\n",
       "      <td>-2.974543e+04</td>\n",
       "      <td>-2.719779e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_neg_mean_squared_error</th>\n",
       "      <td>-7.060346e+08</td>\n",
       "      <td>-1.239851e+09</td>\n",
       "      <td>-1.125125e+09</td>\n",
       "      <td>-9.819320e+08</td>\n",
       "      <td>-2.268434e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_neg_mean_squared_error</th>\n",
       "      <td>-9.383069e+08</td>\n",
       "      <td>-8.267971e+08</td>\n",
       "      <td>-8.763019e+08</td>\n",
       "      <td>-8.847908e+08</td>\n",
       "      <td>-7.397199e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             0             1             2  \\\n",
       "fit_time                          2.498984e-02  1.821399e-02  1.756096e-02   \n",
       "score_time                        5.598068e-03  5.260229e-03  5.133867e-03   \n",
       "test_r2                           8.668969e-01  8.200460e-01  8.262644e-01   \n",
       "train_r2                          8.551369e-01  8.636241e-01  8.579735e-01   \n",
       "test_mape_scorer                 -9.699277e-02 -1.080304e-01 -1.183620e-01   \n",
       "train_mape_scorer                -1.040712e-01 -9.966190e-02 -1.018073e-01   \n",
       "test_sklearn MAPE                -9.699277e-02 -1.080304e-01 -1.183620e-01   \n",
       "train_sklearn MAPE               -1.040712e-01 -9.966190e-02 -1.018073e-01   \n",
       "test_neg_root_mean_square_error  -2.657131e+04 -3.521152e+04 -3.354288e+04   \n",
       "train_neg_root_mean_square_error -3.063179e+04 -2.875408e+04 -2.960240e+04   \n",
       "test_neg_mean_squared_error      -7.060346e+08 -1.239851e+09 -1.125125e+09   \n",
       "train_neg_mean_squared_error     -9.383069e+08 -8.267971e+08 -8.763019e+08   \n",
       "\n",
       "                                             3             4  \n",
       "fit_time                          1.761317e-02  1.713085e-02  \n",
       "score_time                        4.958868e-03  5.253315e-03  \n",
       "test_r2                           8.511854e-01  6.109505e-01  \n",
       "train_r2                          8.561893e-01  8.834054e-01  \n",
       "test_mape_scorer                 -1.078469e-01 -1.219672e-01  \n",
       "train_mape_scorer                -1.024720e-01 -9.828607e-02  \n",
       "test_sklearn MAPE                -1.078469e-01 -1.219672e-01  \n",
       "train_sklearn MAPE               -1.024720e-01 -9.828607e-02  \n",
       "test_neg_root_mean_square_error  -3.133579e+04 -4.762808e+04  \n",
       "train_neg_root_mean_square_error -2.974543e+04 -2.719779e+04  \n",
       "test_neg_mean_squared_error      -9.819320e+08 -2.268434e+09  \n",
       "train_neg_mean_squared_error     -8.847908e+08 -7.397199e+08  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scoring = {\n",
    "    \"r2\": \"r2\",\n",
    "    \"mape_scorer\": mape_scorer, # just for demonstration for a custom scorer\n",
    "    \"sklearn MAPE\": \"neg_mean_absolute_percentage_error\",\n",
    "    \"neg_root_mean_square_error\": \"neg_root_mean_squared_error\",\n",
    "    \"neg_mean_squared_error\": \"neg_mean_squared_error\",\n",
    "}\n",
    "\n",
    "pd.DataFrame(\n",
    "    cross_validate(\n",
    "        ridge_tuned, X_train, y_train, return_train_score=True, scoring=scoring\n",
    "    )\n",
    ").T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are we getting the same `alpha` with mape? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\"ridge__alpha\": 10.0 ** np.arange(-6, 6, 1)}\n",
    "\n",
    "pipe_ridge = make_pipeline(preprocessor, Ridge())\n",
    "\n",
    "search = GridSearchCV(\n",
    "    pipe_ridge, param_grid, return_train_score=True, n_jobs=-1, scoring=mape_scorer\n",
    ")\n",
    "search.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameter values:  {'ridge__alpha': 100.0}\n",
      "Best score: -0.111\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_train_score</th>\n",
       "      <td>-0.10126</td>\n",
       "      <td>-0.111549</td>\n",
       "      <td>-0.096454</td>\n",
       "      <td>-0.090825</td>\n",
       "      <td>-0.085733</td>\n",
       "      <td>-0.085533</td>\n",
       "      <td>-0.085569</td>\n",
       "      <td>-0.085573</td>\n",
       "      <td>-0.085574</td>\n",
       "      <td>-0.085574</td>\n",
       "      <td>-0.203402</td>\n",
       "      <td>-0.332121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>-0.11064</td>\n",
       "      <td>-0.115412</td>\n",
       "      <td>-0.116506</td>\n",
       "      <td>-0.122385</td>\n",
       "      <td>-0.12576</td>\n",
       "      <td>-0.12737</td>\n",
       "      <td>-0.127556</td>\n",
       "      <td>-0.127572</td>\n",
       "      <td>-0.127574</td>\n",
       "      <td>-0.127574</td>\n",
       "      <td>-0.204672</td>\n",
       "      <td>-0.332702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_ridge__alpha</th>\n",
       "      <td>100.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>100000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.030413</td>\n",
       "      <td>0.031483</td>\n",
       "      <td>0.030522</td>\n",
       "      <td>0.025756</td>\n",
       "      <td>0.031604</td>\n",
       "      <td>0.034754</td>\n",
       "      <td>0.031737</td>\n",
       "      <td>0.025349</td>\n",
       "      <td>0.024416</td>\n",
       "      <td>0.021097</td>\n",
       "      <td>0.028194</td>\n",
       "      <td>0.026882</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score           1         2         3         4         5   \\\n",
       "mean_train_score    -0.10126 -0.111549 -0.096454 -0.090825 -0.085733   \n",
       "mean_test_score     -0.11064 -0.115412 -0.116506 -0.122385  -0.12576   \n",
       "param_ridge__alpha     100.0    1000.0      10.0       1.0       0.1   \n",
       "mean_fit_time       0.030413  0.031483  0.030522  0.025756  0.031604   \n",
       "\n",
       "rank_test_score           6         7         8         9         10  \\\n",
       "mean_train_score   -0.085533 -0.085569 -0.085573 -0.085574 -0.085574   \n",
       "mean_test_score     -0.12737 -0.127556 -0.127572 -0.127574 -0.127574   \n",
       "param_ridge__alpha      0.01     0.001    0.0001   0.00001  0.000001   \n",
       "mean_fit_time       0.034754  0.031737  0.025349  0.024416  0.021097   \n",
       "\n",
       "rank_test_score           11        12  \n",
       "mean_train_score   -0.203402 -0.332121  \n",
       "mean_test_score    -0.204672 -0.332702  \n",
       "param_ridge__alpha   10000.0  100000.0  \n",
       "mean_fit_time       0.028194  0.026882  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Best hyperparameter values: \", search.best_params_)\n",
    "print(\"Best score: %0.3f\" % (search.best_score_))\n",
    "pd.DataFrame(search.cv_results_)[\n",
    "    [\n",
    "        \"mean_train_score\",\n",
    "        \"mean_test_score\",\n",
    "        \"param_ridge__alpha\",\n",
    "        \"mean_fit_time\",\n",
    "        \"rank_test_score\",\n",
    "    ]\n",
    "].set_index(\"rank_test_score\").sort_index().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using multiple metrics in `GridSearchCV` or `RandomizedSearchCV` \n",
    "\n",
    "- We could use multiple metrics with `GridSearchCV` or `RandomizedSearchCV`. \n",
    "- But if you do so, you need to set `refit` to the metric (string) for which the `best_params_` will be found and used to build the `best_estimator_` on the whole dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_multi = GridSearchCV(\n",
    "    pipe_ridge,\n",
    "    param_grid,\n",
    "    return_train_score=True,\n",
    "    n_jobs=-1,\n",
    "    scoring=scoring,\n",
    "    refit=\"mape_scorer\",\n",
    ")\n",
    "search_multi.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameter values:  {'ridge__alpha': 100.0}\n",
      "Best score: -0.111\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_ridge__alpha</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_r2</th>\n",
       "      <th>split1_test_r2</th>\n",
       "      <th>split2_test_r2</th>\n",
       "      <th>split3_test_r2</th>\n",
       "      <th>...</th>\n",
       "      <th>mean_test_neg_mean_squared_error</th>\n",
       "      <th>std_test_neg_mean_squared_error</th>\n",
       "      <th>rank_test_neg_mean_squared_error</th>\n",
       "      <th>split0_train_neg_mean_squared_error</th>\n",
       "      <th>split1_train_neg_mean_squared_error</th>\n",
       "      <th>split2_train_neg_mean_squared_error</th>\n",
       "      <th>split3_train_neg_mean_squared_error</th>\n",
       "      <th>split4_train_neg_mean_squared_error</th>\n",
       "      <th>mean_train_neg_mean_squared_error</th>\n",
       "      <th>std_train_neg_mean_squared_error</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rank_test_mape_scorer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.026089</td>\n",
       "      <td>0.007016</td>\n",
       "      <td>0.008769</td>\n",
       "      <td>0.003642</td>\n",
       "      <td>100.0</td>\n",
       "      <td>{'ridge__alpha': 100.0}</td>\n",
       "      <td>0.866897</td>\n",
       "      <td>0.820046</td>\n",
       "      <td>0.826264</td>\n",
       "      <td>0.851185</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.264275e+09</td>\n",
       "      <td>5.328496e+08</td>\n",
       "      <td>1</td>\n",
       "      <td>-9.383069e+08</td>\n",
       "      <td>-8.267971e+08</td>\n",
       "      <td>-8.763019e+08</td>\n",
       "      <td>-8.847908e+08</td>\n",
       "      <td>-7.397199e+08</td>\n",
       "      <td>-8.531833e+08</td>\n",
       "      <td>6.685761e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.032000</td>\n",
       "      <td>0.007684</td>\n",
       "      <td>0.007678</td>\n",
       "      <td>0.002079</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>{'ridge__alpha': 1000.0}</td>\n",
       "      <td>0.827194</td>\n",
       "      <td>0.786532</td>\n",
       "      <td>0.802856</td>\n",
       "      <td>0.812304</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.362306e+09</td>\n",
       "      <td>3.261402e+08</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.282106e+09</td>\n",
       "      <td>-1.156442e+09</td>\n",
       "      <td>-1.207920e+09</td>\n",
       "      <td>-1.225280e+09</td>\n",
       "      <td>-1.030069e+09</td>\n",
       "      <td>-1.180363e+09</td>\n",
       "      <td>8.519271e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.028621</td>\n",
       "      <td>0.004971</td>\n",
       "      <td>0.010537</td>\n",
       "      <td>0.004730</td>\n",
       "      <td>10.0</td>\n",
       "      <td>{'ridge__alpha': 10.0}</td>\n",
       "      <td>0.858957</td>\n",
       "      <td>0.820552</td>\n",
       "      <td>0.832899</td>\n",
       "      <td>0.846307</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.284019e+09</td>\n",
       "      <td>5.507493e+08</td>\n",
       "      <td>2</td>\n",
       "      <td>-7.031807e+08</td>\n",
       "      <td>-6.276881e+08</td>\n",
       "      <td>-6.910603e+08</td>\n",
       "      <td>-6.699413e+08</td>\n",
       "      <td>-5.916073e+08</td>\n",
       "      <td>-6.566956e+08</td>\n",
       "      <td>4.146121e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.027580</td>\n",
       "      <td>0.004528</td>\n",
       "      <td>0.009929</td>\n",
       "      <td>0.003815</td>\n",
       "      <td>1.0</td>\n",
       "      <td>{'ridge__alpha': 1.0}</td>\n",
       "      <td>0.834450</td>\n",
       "      <td>0.810031</td>\n",
       "      <td>0.831653</td>\n",
       "      <td>0.843537</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.388085e+09</td>\n",
       "      <td>6.365356e+08</td>\n",
       "      <td>4</td>\n",
       "      <td>-5.387270e+08</td>\n",
       "      <td>-4.897545e+08</td>\n",
       "      <td>-5.405067e+08</td>\n",
       "      <td>-5.288482e+08</td>\n",
       "      <td>-5.045273e+08</td>\n",
       "      <td>-5.204727e+08</td>\n",
       "      <td>2.001295e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.025182</td>\n",
       "      <td>0.005201</td>\n",
       "      <td>0.006671</td>\n",
       "      <td>0.001477</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'ridge__alpha': 0.1}</td>\n",
       "      <td>0.819830</td>\n",
       "      <td>0.792735</td>\n",
       "      <td>0.805207</td>\n",
       "      <td>0.863338</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.468319e+09</td>\n",
       "      <td>6.909866e+08</td>\n",
       "      <td>5</td>\n",
       "      <td>-4.573403e+08</td>\n",
       "      <td>-4.008696e+08</td>\n",
       "      <td>-4.345403e+08</td>\n",
       "      <td>-4.511181e+08</td>\n",
       "      <td>-4.845183e+08</td>\n",
       "      <td>-4.456773e+08</td>\n",
       "      <td>2.758988e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.026737</td>\n",
       "      <td>0.005319</td>\n",
       "      <td>0.009814</td>\n",
       "      <td>0.005536</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'ridge__alpha': 0.01}</td>\n",
       "      <td>0.822784</td>\n",
       "      <td>0.782073</td>\n",
       "      <td>0.787900</td>\n",
       "      <td>0.867310</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.504687e+09</td>\n",
       "      <td>7.065130e+08</td>\n",
       "      <td>6</td>\n",
       "      <td>-4.516497e+08</td>\n",
       "      <td>-3.925934e+08</td>\n",
       "      <td>-4.251121e+08</td>\n",
       "      <td>-4.441832e+08</td>\n",
       "      <td>-4.835932e+08</td>\n",
       "      <td>-4.394263e+08</td>\n",
       "      <td>3.006982e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.028774</td>\n",
       "      <td>0.004413</td>\n",
       "      <td>0.010363</td>\n",
       "      <td>0.004246</td>\n",
       "      <td>0.001</td>\n",
       "      <td>{'ridge__alpha': 0.001}</td>\n",
       "      <td>0.824481</td>\n",
       "      <td>0.780386</td>\n",
       "      <td>0.785360</td>\n",
       "      <td>0.867572</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.508977e+09</td>\n",
       "      <td>7.092363e+08</td>\n",
       "      <td>7</td>\n",
       "      <td>-4.515516e+08</td>\n",
       "      <td>-3.924490e+08</td>\n",
       "      <td>-4.249579e+08</td>\n",
       "      <td>-4.440666e+08</td>\n",
       "      <td>-4.835790e+08</td>\n",
       "      <td>-4.393208e+08</td>\n",
       "      <td>3.011365e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.029260</td>\n",
       "      <td>0.004155</td>\n",
       "      <td>0.009032</td>\n",
       "      <td>0.003749</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>{'ridge__alpha': 0.0001}</td>\n",
       "      <td>0.824671</td>\n",
       "      <td>0.780205</td>\n",
       "      <td>0.785095</td>\n",
       "      <td>0.867595</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.509418e+09</td>\n",
       "      <td>7.095221e+08</td>\n",
       "      <td>8</td>\n",
       "      <td>-4.515504e+08</td>\n",
       "      <td>-3.924474e+08</td>\n",
       "      <td>-4.249563e+08</td>\n",
       "      <td>-4.440653e+08</td>\n",
       "      <td>-4.835789e+08</td>\n",
       "      <td>-4.393197e+08</td>\n",
       "      <td>3.011413e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.027493</td>\n",
       "      <td>0.002594</td>\n",
       "      <td>0.010081</td>\n",
       "      <td>0.002627</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>{'ridge__alpha': 1e-05}</td>\n",
       "      <td>0.824691</td>\n",
       "      <td>0.780187</td>\n",
       "      <td>0.785068</td>\n",
       "      <td>0.867597</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.509463e+09</td>\n",
       "      <td>7.095507e+08</td>\n",
       "      <td>9</td>\n",
       "      <td>-4.515504e+08</td>\n",
       "      <td>-3.924474e+08</td>\n",
       "      <td>-4.249562e+08</td>\n",
       "      <td>-4.440653e+08</td>\n",
       "      <td>-4.835789e+08</td>\n",
       "      <td>-4.393197e+08</td>\n",
       "      <td>3.011413e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.032921</td>\n",
       "      <td>0.006119</td>\n",
       "      <td>0.006292</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>{'ridge__alpha': 1e-06}</td>\n",
       "      <td>0.824693</td>\n",
       "      <td>0.780185</td>\n",
       "      <td>0.785065</td>\n",
       "      <td>0.867597</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.509467e+09</td>\n",
       "      <td>7.095536e+08</td>\n",
       "      <td>10</td>\n",
       "      <td>-4.515504e+08</td>\n",
       "      <td>-3.924474e+08</td>\n",
       "      <td>-4.249562e+08</td>\n",
       "      <td>-4.440653e+08</td>\n",
       "      <td>-4.835789e+08</td>\n",
       "      <td>-4.393197e+08</td>\n",
       "      <td>3.011413e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.032045</td>\n",
       "      <td>0.006053</td>\n",
       "      <td>0.008120</td>\n",
       "      <td>0.004116</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>{'ridge__alpha': 10000.0}</td>\n",
       "      <td>0.593402</td>\n",
       "      <td>0.551970</td>\n",
       "      <td>0.568663</td>\n",
       "      <td>0.557891</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.686107e+09</td>\n",
       "      <td>3.315718e+08</td>\n",
       "      <td>11</td>\n",
       "      <td>-2.761798e+09</td>\n",
       "      <td>-2.571596e+09</td>\n",
       "      <td>-2.662233e+09</td>\n",
       "      <td>-2.624434e+09</td>\n",
       "      <td>-2.595316e+09</td>\n",
       "      <td>-2.643075e+09</td>\n",
       "      <td>6.663313e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.026692</td>\n",
       "      <td>0.004300</td>\n",
       "      <td>0.007573</td>\n",
       "      <td>0.003461</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>{'ridge__alpha': 100000.0}</td>\n",
       "      <td>0.135414</td>\n",
       "      <td>0.113850</td>\n",
       "      <td>0.118889</td>\n",
       "      <td>0.123419</td>\n",
       "      <td>...</td>\n",
       "      <td>-5.442218e+09</td>\n",
       "      <td>5.531025e+08</td>\n",
       "      <td>12</td>\n",
       "      <td>-5.629636e+09</td>\n",
       "      <td>-5.271544e+09</td>\n",
       "      <td>-5.379227e+09</td>\n",
       "      <td>-5.342554e+09</td>\n",
       "      <td>-5.497948e+09</td>\n",
       "      <td>-5.424182e+09</td>\n",
       "      <td>1.262024e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 80 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       mean_fit_time  std_fit_time  mean_score_time  \\\n",
       "rank_test_mape_scorer                                                 \n",
       "1                           0.026089      0.007016         0.008769   \n",
       "2                           0.032000      0.007684         0.007678   \n",
       "3                           0.028621      0.004971         0.010537   \n",
       "4                           0.027580      0.004528         0.009929   \n",
       "5                           0.025182      0.005201         0.006671   \n",
       "6                           0.026737      0.005319         0.009814   \n",
       "7                           0.028774      0.004413         0.010363   \n",
       "8                           0.029260      0.004155         0.009032   \n",
       "9                           0.027493      0.002594         0.010081   \n",
       "10                          0.032921      0.006119         0.006292   \n",
       "11                          0.032045      0.006053         0.008120   \n",
       "12                          0.026692      0.004300         0.007573   \n",
       "\n",
       "                       std_score_time param_ridge__alpha  \\\n",
       "rank_test_mape_scorer                                      \n",
       "1                            0.003642              100.0   \n",
       "2                            0.002079             1000.0   \n",
       "3                            0.004730               10.0   \n",
       "4                            0.003815                1.0   \n",
       "5                            0.001477                0.1   \n",
       "6                            0.005536               0.01   \n",
       "7                            0.004246              0.001   \n",
       "8                            0.003749             0.0001   \n",
       "9                            0.002627            0.00001   \n",
       "10                           0.000163           0.000001   \n",
       "11                           0.004116            10000.0   \n",
       "12                           0.003461           100000.0   \n",
       "\n",
       "                                           params  split0_test_r2  \\\n",
       "rank_test_mape_scorer                                               \n",
       "1                         {'ridge__alpha': 100.0}        0.866897   \n",
       "2                        {'ridge__alpha': 1000.0}        0.827194   \n",
       "3                          {'ridge__alpha': 10.0}        0.858957   \n",
       "4                           {'ridge__alpha': 1.0}        0.834450   \n",
       "5                           {'ridge__alpha': 0.1}        0.819830   \n",
       "6                          {'ridge__alpha': 0.01}        0.822784   \n",
       "7                         {'ridge__alpha': 0.001}        0.824481   \n",
       "8                        {'ridge__alpha': 0.0001}        0.824671   \n",
       "9                         {'ridge__alpha': 1e-05}        0.824691   \n",
       "10                        {'ridge__alpha': 1e-06}        0.824693   \n",
       "11                      {'ridge__alpha': 10000.0}        0.593402   \n",
       "12                     {'ridge__alpha': 100000.0}        0.135414   \n",
       "\n",
       "                       split1_test_r2  split2_test_r2  split3_test_r2  ...  \\\n",
       "rank_test_mape_scorer                                                  ...   \n",
       "1                            0.820046        0.826264        0.851185  ...   \n",
       "2                            0.786532        0.802856        0.812304  ...   \n",
       "3                            0.820552        0.832899        0.846307  ...   \n",
       "4                            0.810031        0.831653        0.843537  ...   \n",
       "5                            0.792735        0.805207        0.863338  ...   \n",
       "6                            0.782073        0.787900        0.867310  ...   \n",
       "7                            0.780386        0.785360        0.867572  ...   \n",
       "8                            0.780205        0.785095        0.867595  ...   \n",
       "9                            0.780187        0.785068        0.867597  ...   \n",
       "10                           0.780185        0.785065        0.867597  ...   \n",
       "11                           0.551970        0.568663        0.557891  ...   \n",
       "12                           0.113850        0.118889        0.123419  ...   \n",
       "\n",
       "                       mean_test_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                     \n",
       "1                                         -1.264275e+09   \n",
       "2                                         -1.362306e+09   \n",
       "3                                         -1.284019e+09   \n",
       "4                                         -1.388085e+09   \n",
       "5                                         -1.468319e+09   \n",
       "6                                         -1.504687e+09   \n",
       "7                                         -1.508977e+09   \n",
       "8                                         -1.509418e+09   \n",
       "9                                         -1.509463e+09   \n",
       "10                                        -1.509467e+09   \n",
       "11                                        -2.686107e+09   \n",
       "12                                        -5.442218e+09   \n",
       "\n",
       "                       std_test_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                    \n",
       "1                                         5.328496e+08   \n",
       "2                                         3.261402e+08   \n",
       "3                                         5.507493e+08   \n",
       "4                                         6.365356e+08   \n",
       "5                                         6.909866e+08   \n",
       "6                                         7.065130e+08   \n",
       "7                                         7.092363e+08   \n",
       "8                                         7.095221e+08   \n",
       "9                                         7.095507e+08   \n",
       "10                                        7.095536e+08   \n",
       "11                                        3.315718e+08   \n",
       "12                                        5.531025e+08   \n",
       "\n",
       "                       rank_test_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                     \n",
       "1                                                     1   \n",
       "2                                                     3   \n",
       "3                                                     2   \n",
       "4                                                     4   \n",
       "5                                                     5   \n",
       "6                                                     6   \n",
       "7                                                     7   \n",
       "8                                                     8   \n",
       "9                                                     9   \n",
       "10                                                   10   \n",
       "11                                                   11   \n",
       "12                                                   12   \n",
       "\n",
       "                       split0_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                        \n",
       "1                                            -9.383069e+08   \n",
       "2                                            -1.282106e+09   \n",
       "3                                            -7.031807e+08   \n",
       "4                                            -5.387270e+08   \n",
       "5                                            -4.573403e+08   \n",
       "6                                            -4.516497e+08   \n",
       "7                                            -4.515516e+08   \n",
       "8                                            -4.515504e+08   \n",
       "9                                            -4.515504e+08   \n",
       "10                                           -4.515504e+08   \n",
       "11                                           -2.761798e+09   \n",
       "12                                           -5.629636e+09   \n",
       "\n",
       "                       split1_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                        \n",
       "1                                            -8.267971e+08   \n",
       "2                                            -1.156442e+09   \n",
       "3                                            -6.276881e+08   \n",
       "4                                            -4.897545e+08   \n",
       "5                                            -4.008696e+08   \n",
       "6                                            -3.925934e+08   \n",
       "7                                            -3.924490e+08   \n",
       "8                                            -3.924474e+08   \n",
       "9                                            -3.924474e+08   \n",
       "10                                           -3.924474e+08   \n",
       "11                                           -2.571596e+09   \n",
       "12                                           -5.271544e+09   \n",
       "\n",
       "                       split2_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                        \n",
       "1                                            -8.763019e+08   \n",
       "2                                            -1.207920e+09   \n",
       "3                                            -6.910603e+08   \n",
       "4                                            -5.405067e+08   \n",
       "5                                            -4.345403e+08   \n",
       "6                                            -4.251121e+08   \n",
       "7                                            -4.249579e+08   \n",
       "8                                            -4.249563e+08   \n",
       "9                                            -4.249562e+08   \n",
       "10                                           -4.249562e+08   \n",
       "11                                           -2.662233e+09   \n",
       "12                                           -5.379227e+09   \n",
       "\n",
       "                       split3_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                        \n",
       "1                                            -8.847908e+08   \n",
       "2                                            -1.225280e+09   \n",
       "3                                            -6.699413e+08   \n",
       "4                                            -5.288482e+08   \n",
       "5                                            -4.511181e+08   \n",
       "6                                            -4.441832e+08   \n",
       "7                                            -4.440666e+08   \n",
       "8                                            -4.440653e+08   \n",
       "9                                            -4.440653e+08   \n",
       "10                                           -4.440653e+08   \n",
       "11                                           -2.624434e+09   \n",
       "12                                           -5.342554e+09   \n",
       "\n",
       "                       split4_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                        \n",
       "1                                            -7.397199e+08   \n",
       "2                                            -1.030069e+09   \n",
       "3                                            -5.916073e+08   \n",
       "4                                            -5.045273e+08   \n",
       "5                                            -4.845183e+08   \n",
       "6                                            -4.835932e+08   \n",
       "7                                            -4.835790e+08   \n",
       "8                                            -4.835789e+08   \n",
       "9                                            -4.835789e+08   \n",
       "10                                           -4.835789e+08   \n",
       "11                                           -2.595316e+09   \n",
       "12                                           -5.497948e+09   \n",
       "\n",
       "                       mean_train_neg_mean_squared_error  \\\n",
       "rank_test_mape_scorer                                      \n",
       "1                                          -8.531833e+08   \n",
       "2                                          -1.180363e+09   \n",
       "3                                          -6.566956e+08   \n",
       "4                                          -5.204727e+08   \n",
       "5                                          -4.456773e+08   \n",
       "6                                          -4.394263e+08   \n",
       "7                                          -4.393208e+08   \n",
       "8                                          -4.393197e+08   \n",
       "9                                          -4.393197e+08   \n",
       "10                                         -4.393197e+08   \n",
       "11                                         -2.643075e+09   \n",
       "12                                         -5.424182e+09   \n",
       "\n",
       "                       std_train_neg_mean_squared_error  \n",
       "rank_test_mape_scorer                                    \n",
       "1                                          6.685761e+07  \n",
       "2                                          8.519271e+07  \n",
       "3                                          4.146121e+07  \n",
       "4                                          2.001295e+07  \n",
       "5                                          2.758988e+07  \n",
       "6                                          3.006982e+07  \n",
       "7                                          3.011365e+07  \n",
       "8                                          3.011413e+07  \n",
       "9                                          3.011413e+07  \n",
       "10                                         3.011413e+07  \n",
       "11                                         6.663313e+07  \n",
       "12                                         1.262024e+08  \n",
       "\n",
       "[12 rows x 80 columns]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Best hyperparameter values: \", search_multi.best_params_)\n",
    "print(\"Best score: %0.3f\" % (search_multi.best_score_))\n",
    "pd.DataFrame(search_multi.cv_results_).set_index(\"rank_test_mape_scorer\").sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the test score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.09496387589495985"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_multi.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09496387589495985"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_mape(y_test, ridge_tuned.predict(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Using regression metrics with `scikit-learn`\n",
    "\n",
    "- In `sklearn`, you will notice that it has negative version of the metrics above (e.g., `neg_mean_squared_error`, `neg_root_mean_squared_error`). \n",
    "- The reason for this is that scores return a value to maximize, the higher the better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iClicker Exercise 2.3\n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/C0P55**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) We can use still use precision and recall for regression problems but now we have other metrics we can use as well.\n",
    "- (B) In `sklearn` for regression problems, using `r2_score()` and `.score()` (with default values) will produce the same results.\n",
    "- (C) RMSE is always going to be non-negative.\n",
    "- (D) MSE does not directly provide the information about whether the model is underpredicting or overpredicting.\n",
    "- (E) We can pass multiple scoring metrics to `GridSearchCV` or `RandomizedSearchCV` for regression as well as classification problems. \n",
    "\n",
    "**V's answers: B, C, D, E**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What did we learn today? \n",
    "\n",
    "- House prices dataset target is price, which is numeric -> regression rather than classification\n",
    "- There are corresponding versions of all the tools we used:\n",
    "    - `DummyClassifier` -> `DummyRegressor`\n",
    "    - `LogisticRegression` -> `Ridge`\n",
    "- `Ridge` hyperparameter `alpha` is like `LogisticRegression` hyperparameter `C`, but opposite meaning\n",
    "- We'll avoid `LinearRegression` in this course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Scoring metrics\n",
    "- $R^2$ is the default .score(), it is unitless, 0 is bad, 1 is best\n",
    "- MSE (mean squared error) is in units of target squared, hard to interpret; 0 is best\n",
    "- RMSE (root mean squared error) is in the same units as the target; 0 is best\n",
    "- MAPE (mean absolute percent error) is unitless; 0 is best, 1 is bad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/eva-seeyou.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:573]",
   "language": "python",
   "name": "conda-env-573-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}