03-03_Random_Forest.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "title: \"ClassifyAnything Part 3-3: Random Forest\"\n",
    "author: \"Joseph Chao-Chung Kuo (ckuo@ukaachen.de)\"\n",
    "date: today\n",
    "output:\n",
    "  general:\n",
    "    output_stdout: true\n",
    "    output_error: false\n",
    "    output: true\n",
    "format:\n",
    "  html:\n",
    "    toc: true\n",
    "    toc-depth: 3\n",
    "    html-math-method: katex\n",
    "    code-fold: true\n",
    "    code-tools: true\n",
    "    number_sections: true\n",
    "    embed-resources: true\n",
    "    page-layout: full\n",
    "execute:\n",
    "    echo: true\n",
    "    warning: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Supervised learning for classification_\n",
    "\n",
    "Random Forest is an ensemble learning method used for both classification and regression tasks. It is based on the concept of decision trees and combines the predictions of multiple individual decision trees to make more accurate and robust predictions.\n",
    "\n",
    "Here's a short explanation of how Random Forest works:\n",
    "\n",
    "1. *Random Sampling:* It randomly selects subsets of the original training data with replacement (bootstrap samples). Each subset is used to train a separate decision tree. This process is known as \"bagging.\"\n",
    "2. *Random Feature Selection:* At each node of the decision tree, only a random subset of features is considered for splitting. This helps to introduce diversity among the individual trees.\n",
    "3. *Voting (Classification) / Averaging (Regression):* For classification tasks, each decision tree in the Random Forest predicts the class of the input data point. The final prediction is determined by majority voting. For regression tasks, the predictions from all trees are averaged to obtain the final output.\n",
    "4. *Ensemble Effect:* By combining the predictions of multiple decision trees, Random Forest reduces overfitting and improves the model's generalization performance. It is less susceptible to noise and outliers compared to a single decision tree.\n",
    "\n",
    "Random Forest is a powerful and widely used algorithm due to its simplicity, robustness, and ability to handle high-dimensional data. It is effective in various applications, including classification, regression, and feature selection tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-3.1 Import data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import plot_tree\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "with open(\"outputs/03_Variables.pkl\", 'rb') as file:\n",
    "    (X_train, X_test, y_train, y_test, kfold) = pickle.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.2 Train the model with cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameters to search over in the grid search\n",
    "param_grid = {\n",
    "    'n_estimators': [10, 50, 100, 150],\n",
    "    'max_depth': [None, 1, 5, 10, 20],\n",
    "    'min_samples_split': [2, 5, 10],\n",
    "}\n",
    "\n",
    "# Create the Random Forest classifier\n",
    "rf_classifier = RandomForestClassifier()\n",
    "# Perform Grid Search to find the best hyperparameters\n",
    "grid_search = GridSearchCV(rf_classifier, param_grid, cv=kfold, n_jobs=4)\n",
    "grid_search.fit(X_train, y_train)\n",
    "# Get the best hyperparameters from the grid search\n",
    "best_params = grid_search.best_params_\n",
    "# Create the Random Forest classifier with the best hyperparameters\n",
    "rf_classifier = RandomForestClassifier(**best_params)\n",
    "# Fit the model to the data\n",
    "rf_classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.3 Visualize fine-tuning of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the grid search results\n",
    "results = pd.DataFrame(grid_search.cv_results_)\n",
    "\n",
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_n_estimators', \n",
    "                                  columns='param_max_depth', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure()\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: N estimators vs Max Depth')\n",
    "plt.xlabel('Max Depth')\n",
    "plt.ylabel('N estimators')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_max_depth', \n",
    "                                  columns='param_min_samples_split', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure()\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: Max Depth vs Min Sample Split')\n",
    "plt.xlabel('Min Sample Split')\n",
    "plt.ylabel('Max Depth')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.4 Visualize random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the individual decision trees from the Random Forest\n",
    "trees = rf_classifier.estimators_\n",
    "# Visualize the random forest\n",
    "for i in range(5):\n",
    "    plt.figure(figsize=(15, 10))\n",
    "    plot_tree(trees[i], \n",
    "            feature_names=X_train.columns.to_list(),\n",
    "            class_names=np.unique(y_train.to_list()).tolist())\n",
    "    plt.title(\"Tree \"+str(i+1))\n",
    "    # Use the line below to verify the label names\n",
    "    # np.unique(y_train.to_list(), return_counts=True)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.5 Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "# Save the final trained model to a file\n",
    "joblib.dump(rf_classifier, 'outputs/03-3_Random_Forest_final_model.joblib')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}