1-basic-use-xgboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses the diamond dataset in seaborn demonstrating how to use the xgboost. It is assumed that you have your environment setup for the xgboost. Some recommended use cases for this notebook:\n",
    "  - Testing if your environment is ready to run the xgboost\n",
    "  - Learning some basic setup for the xgboost\n",
    "  - Testing CPU and GPU running with xgboost\n",
    "\n",
    "Note: this notebook will show running the xgboost on both CPU and GPU. You can skip the GPU part if unavailable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "import xgboost as xgb\n",
    "seed = 123"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('0.13.2', '1.26.4', '1.5.2', '2.1.1')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sns.__version__, np.__version__, sklearn.__version__, xgb.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(   carat      cut color clarity  depth  table  price     x     y     z\n",
       " 0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43\n",
       " 1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31\n",
       " 2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31\n",
       " 3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63\n",
       " 4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75,\n",
       " carat       float64\n",
       " cut        category\n",
       " color      category\n",
       " clarity    category\n",
       " depth       float64\n",
       " table       float64\n",
       " price         int64\n",
       " x           float64\n",
       " y           float64\n",
       " z           float64\n",
       " dtype: object,\n",
       " (53940, 10))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use diamonds dataset in seaborn\n",
    "diamonds = sns.load_dataset('diamonds')\n",
    "diamonds.head(), diamonds.dtypes, diamonds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>53940</td>\n",
       "      <td>53940</td>\n",
       "      <td>53940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>SI1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>21551</td>\n",
       "      <td>11292</td>\n",
       "      <td>13065</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          cut  color clarity\n",
       "count   53940  53940   53940\n",
       "unique      5      7       8\n",
       "top     Ideal      G     SI1\n",
       "freq    21551  11292   13065"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we have three columns for non-numerical features: cut, color, clarity\n",
    "# their dtypes are already as category\n",
    "diamonds.describe(exclude=np.number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CategoricalDtype(categories=['Ideal', 'Premium', 'Very Good', 'Good', 'Fair'], ordered=False, categories_dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# note that category dtype can be either ordinal or nominal\n",
    "# cut is currently nominal\n",
    "diamonds['cut'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split X, Y\n",
    "Y = diamonds['price']\n",
    "X = diamonds.drop('price', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((43152,), (10788,))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split train and test\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=seed, train_size=0.8)\n",
    "Y_train.shape, Y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make DMatrix for xgboost\n",
    "# note the argument enable_categorical=True\n",
    "dtrain = xgb.DMatrix(X_train, Y_train, enable_categorical=True)\n",
    "dtest = xgb.DMatrix(X_test, Y_test, enable_categorical=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-rmse:2861.85524\ttest-rmse:2848.74215\n",
      "[50]\ttrain-rmse:439.96632\ttest-rmse:545.91459\n",
      "[100]\ttrain-rmse:378.66674\ttest-rmse:545.62390\n",
      "[126]\ttrain-rmse:360.34374\ttest-rmse:548.42863\n"
     ]
    }
   ],
   "source": [
    "# xgb.train\n",
    "# using cpu\n",
    "params_cpu = {\n",
    "    \"objective\": \"reg:squarederror\", \n",
    "    'eval_metric': 'rmse',\n",
    "    \"tree_method\": \"hist\",\n",
    "    'device': 'cpu'\n",
    "    }\n",
    "n = 5000\n",
    "evals = [(dtrain, \"train\"), (dtest, \"test\")]\n",
    "\n",
    "model = xgb.train(\n",
    "   params=params_cpu,\n",
    "   dtrain=dtrain,\n",
    "   num_boost_round=n,\n",
    "   evals=evals,\n",
    "   verbose_eval=50,\n",
    "   early_stopping_rounds=50,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-rmse:2861.57928\ttest-rmse:2848.91516\n",
      "[50]\ttrain-rmse:430.67545\ttest-rmse:545.64382\n",
      "[100]\ttrain-rmse:378.38766\ttest-rmse:543.54183\n",
      "[139]\ttrain-rmse:346.25134\ttest-rmse:545.59116\n"
     ]
    }
   ],
   "source": [
    "# using gpu\n",
    "params_gpu = {\n",
    "    \"objective\": \"reg:squarederror\", \n",
    "    'eval_metric': 'rmse',\n",
    "    \"tree_method\": \"hist\",\n",
    "    'device': 'gpu'\n",
    "    }\n",
    "\n",
    "model = xgb.train(\n",
    "   params=params_gpu,\n",
    "   dtrain=dtrain,\n",
    "   num_boost_round=n,\n",
    "   evals=evals,\n",
    "   verbose_eval=50,\n",
    "   early_stopping_rounds=50,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cputime 16.291748762130737 \n",
      " gputime 17.354901552200317\n"
     ]
    }
   ],
   "source": [
    "# compare running time\n",
    "tic = time.time()\n",
    "model_cpu = xgb.train(\n",
    "   params=params_cpu,\n",
    "   dtrain=dtrain,\n",
    "   num_boost_round=n,\n",
    "   evals=evals,\n",
    "   verbose_eval=0,\n",
    "#    early_stopping_rounds=50,\n",
    ")\n",
    "toc = time.time()\n",
    "cputime = toc - tic\n",
    "\n",
    "tic = time.time()\n",
    "model_cpu = xgb.train(\n",
    "   params=params_gpu,\n",
    "   dtrain=dtrain,\n",
    "   num_boost_round=n,\n",
    "   evals=evals,\n",
    "   verbose_eval=0,\n",
    "#    early_stopping_rounds=50,\n",
    ")\n",
    "toc = time.time()\n",
    "gputime = toc - tic\n",
    "\n",
    "print(f'cputime {cputime} \\n gputime {gputime}')\n",
    "# cputime > gputime\n",
    "# note: ran on 1 gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates the minimal xgboost setup. The work flow can be summarized:\n",
    "  - Data preprocessing: here the diamond dataset from seaborn was used for the demonstration. Since this notebook aims for the minimal setup, the preprocessing wasn't done thorougly. One thing to be mindful about is how each feature's dtype is declared, especially when there are categorical features either ordinal and nominal. Also, it is important to note that if there is any categorical feature, enable_categorical=True must be set when making the DMatrix.\n",
    "  - Making DMatrix: DMatrix is a specific xgboost's class for the dataset wrapper. The DMatrix object is the one being passed to the xgboost during training.\n",
    "  - Parameter setup: this step is the key to tell the xgboost how to learn. There are several parameters, and only some important ones are demonstrated here. Since the demo's task is price prediction, the model was told to perform regression with minimizing squred-error objective (reg:squarederror).\n",
    "  - Train: in this demonstration, the model is trained through xgb.train() API. It is important to note that there are other regression APIs in the xgboost such as xgb.XGBRegressor.\n",
    "\n",
    "Additionally, the runtimes between using CPU and 1 GPU are compared, showing that using GPU is faster as expected. It is important to note that a proper environment setup is needed to use GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}