03-04_SVM.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "title: \"ClassifyAnything Part 3-4: SVM\"\n",
    "author: \"Joseph Chao-Chung Kuo (ckuo@ukaachen.de)\"\n",
    "date: today\n",
    "output:\n",
    "  general:\n",
    "    output_stdout: true\n",
    "    output_error: false\n",
    "    output: true\n",
    "format:\n",
    "  html:\n",
    "    toc: true\n",
    "    toc-depth: 3\n",
    "    html-math-method: katex\n",
    "    code-fold: true\n",
    "    code-tools: true\n",
    "    number_sections: true\n",
    "    embed-resources: true\n",
    "    page-layout: full\n",
    "execute:\n",
    "    echo: true\n",
    "    warning: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Supervised learning for classification_\n",
    "\n",
    "SVM (Support Vector Machine) is a popular supervised learning algorithm used for classification and regression tasks. It finds the best hyperplane that maximizes the margin between classes in the feature space, making it robust and suitable for high-dimensional data. SVM uses kernel functions to handle non-linearly separable problems, making it versatile and powerful for various applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4.1 Import data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "with open(\"outputs/03_Variables.pkl\", 'rb') as file:\n",
    "    (X_train, X_test, y_train, y_test, kfold) = pickle.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4.2 Train the model with cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the parameter grid for grid search\n",
    "param_grid = {\n",
    "    'C': [0.1, 1, 10, 100, 200],\n",
    "    'kernel': ['linear', 'rbf', 'poly'],\n",
    "    'gamma': ['scale', 'auto']\n",
    "}\n",
    "\n",
    "# Create the SVM classifier\n",
    "svm_model = SVC(probability=True, random_state=42)\n",
    "\n",
    "# Perform grid search with cross-validation\n",
    "grid_search = GridSearchCV(svm_model, param_grid, cv=kfold, n_jobs=4)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best hyperparameters from grid search\n",
    "print(\"Best hyperparameters:\")\n",
    "print(grid_search.best_params_)\n",
    "\n",
    "# Get the best model from grid search\n",
    "best_model = grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4.3 Visualize fine-tuning of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the grid search results\n",
    "results = pd.DataFrame(grid_search.cv_results_)\n",
    "\n",
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_C', \n",
    "                                  columns='param_kernel', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure(figsize=(10, 6))\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: C vs Kernel')\n",
    "plt.xlabel('Kernel')\n",
    "plt.ylabel('C')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_gamma', \n",
    "                                  columns='param_kernel', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure(figsize=(10, 6))\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: Gamma vs Kernel')\n",
    "plt.xlabel('Kernel')\n",
    "plt.ylabel('Gamma')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4.4 Visualize SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4.5 Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "# Save the final trained model to a file\n",
    "joblib.dump(best_model, 'outputs/03-4_SVM_final_model.joblib')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}