xgboost.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CV0qgnlSqkbp"
      },
      "source": [
        " # Hand-crafted features + XGBoost"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-SxyZchM-Az"
      },
      "source": [
        "Install and import required packages:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sk2tcrewhaBo"
      },
      "outputs": [],
      "source": [
        "! pip install pandas numpy tqdm  scikit-learn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GjuDQHi9hNkO"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import torch\n",
        "from tqdm import tqdm\n",
        "from datetime import timedelta, datetime\n",
        "from sklearn.preprocessing import normalize"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QaWW2c5uNkBL"
      },
      "source": [
        "Configure notebook to use GPU:\n",
        "1. Go to Runtime --> Change runtime type\n",
        "2. Select GPU\n",
        "3. Check if the device was properly selected"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nmQ-Kt4xYlmw"
      },
      "outputs": [],
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print(f\"Device: '{device}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AM7BNZZ6hDv4"
      },
      "source": [
        "## Data\n",
        "Dataset is based on [eCommerce behavior data from multi category store](https://www.kaggle.com/datasets/mkechinov/ecommerce-behavior-data-from-multi-category-store).  \n",
        "To meet the limitations of the workshop environment we prepared a toy version of this dataset that can be downloaded from our GitHub repository.  \n",
        "To access data:\n",
        "1. Clone GitHub repository"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6_WhCTKDHllE"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Synerise/predicting-user-behavior-workshop.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_6s2JCcBvhW"
      },
      "source": [
        "2. Set the data directory to match the data folder location"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5kqUhjHBz6gC"
      },
      "outputs": [],
      "source": [
        "DATA_DIR = \"./predicting-user-behavior-workshop/data\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BbFBoUr5CA19"
      },
      "source": [
        "3. Check if you can load data properly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mmXWE5_WqzJD"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(f\"{DATA_DIR}/2019-Oct_small.csv\")\n",
        "target_brands = np.load(f\"{DATA_DIR}/target_brands.npy\", allow_pickle=True)\n",
        "train_target = np.load(f\"{DATA_DIR}/train_target.npy\")\n",
        "test_target = np.load(f\"{DATA_DIR}/test_target.npy\")\n",
        "user_ids = np.load(f\"{DATA_DIR}/user_ids.npy\")\n",
        "df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b07iiJjqWUMT"
      },
      "source": [
        "## Task\n",
        "We want to predict if a user will buy products of selected brands.\n",
        "Here, we use the 20 most popular brands:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Wn3rti4eqfm"
      },
      "outputs": [],
      "source": [
        "target_brands"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z2aOFHSKwKFu"
      },
      "source": [
        "### Number of events per brand for the top 20 brands"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UWtMSTmnsKPg"
      },
      "outputs": [],
      "source": [
        "plt.figure().set_figwidth(18)\n",
        "plt.bar(target_brands, df['brand'].value_counts()[:20].to_numpy())\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mj8tSwv5vYi2"
      },
      "source": [
        "### Other applications\n",
        "When working with real-life data, this task can be easily modified to better fit business requierments. For example:\n",
        "\n",
        "* selecting an arbitrary subset of brands for the targeted campaigns\n",
        "* predicting product categories user will buy\n",
        "* predicting offers a user will be interested in\n",
        "* predicting what kind of subscription plan is user willing to choose\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_mu7wmEq0WJu"
      },
      "source": [
        "## Target\n",
        "In this workshop, we use all events from October 2019 to create the model's input representing user interactions from this period. To create training targets (ground truth labels), we use the first week of November.\n",
        "\n",
        "**For each of the selected brands, we check if a user bought any product of this brand in the first week of November.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fH7Zx7WGnAAl"
      },
      "outputs": [],
      "source": [
        "train_target.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BsjgZX8X1Nmk"
      },
      "outputs": [],
      "source": [
        "example_ind = 871\n",
        "print(f\"User id: {user_ids[example_ind]}\")\n",
        "print(f\"Target: {train_target[example_ind]}\")\n",
        "print(f\"Brands in target: {target_brands[np.where(train_target[example_ind] == 1)]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cX091GfREnBG"
      },
      "source": [
        "Next, we use the following 7 days — the second week of November, to measure model performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EEfcBQ5GmWc2"
      },
      "outputs": [],
      "source": [
        "test_target.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHZYWadrHjlW"
      },
      "source": [
        "To create the train and the test sets, we use the same set of users but two consecutive time windows — the first week of November for training and the second for testing. This mimics a real-life setting where we predict the future actions of our users based on historical data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "88cJfCqMJJQJ"
      },
      "source": [
        "## Hand-crafted features\n",
        "\n",
        "To create a user representation that we can use as a model input, we must aggregate information from user history and represent it in a numerical form.  \n",
        "\n",
        "The presented method is based on features that are hand-picked and tested by a data scientist contrary to learned representations, e.g. embeddings.\n",
        "\n",
        "**Example of hand-crafted features in e-commerce**:\n",
        "\n",
        "* number of products bought in the last day, week or month\n",
        "* products bought/viewed/added-to-cart in the given category, brand\n",
        "* average cart value\n",
        "\n",
        "**Example of hand-crafted features in banking**:\n",
        "\n",
        "* number of transactions per day\n",
        "* number of transactions per category\n",
        "* average transaction value\n",
        "* account type\n",
        "* special offers\n",
        "\n",
        "**Example of hand-crafted features in telecom**:\n",
        "\n",
        "* type of subscription\n",
        "* internet usage\n",
        "* number of phone calls per day/week/month\n",
        "\n",
        "These are only a few examples of such features. In practice, feature creation stems from domain knowledge and information available in data. What should be considered are:\n",
        "* **available types of events** (e.g., page view, likes, transaction, subscription, product buy, add-to-cart)  \n",
        "* **time windows** relevant for given data characteristics and domain requirements (lastmonths, last days, last hours)\n",
        "* event **object metadata** (e.g., product brand, transaction value, offer popularity, page category)\n",
        "* **user metadata** (e.g., age, location, account type)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nisR8yoVQ50e"
      },
      "source": [
        "### Example features\n",
        "In the toy dataset, we have only information about user, brand, and time. What we can consider when creating a user feature vector:\n",
        "\n",
        "* total number of events per user\n",
        "* number of events per brand (or selected subset of brands)\n",
        "* number of events in selected time windows\n",
        "* number of events per brand in selected time windows\n",
        "\n",
        "In the following code, we want to consider for each user:\n",
        "\n",
        "* total number of events\n",
        "* number of events per brand for 30 most popular brands"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jQa8pzyBn3A1"
      },
      "outputs": [],
      "source": [
        "#@title Task: select top 30 brands that will be used to create features\n",
        "BRANDS = ..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Y79jK7f4P98V"
      },
      "outputs": [],
      "source": [
        "#@title Solution\n",
        "BRANDS = df[\"brand\"].value_counts()[:30].index\n",
        "BRANDS\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hRzUFw7GpCd1"
      },
      "source": [
        "To create features we consider events from October.  \n",
        "We group events by users and for each user, we check:\n",
        "* number of events in each of the selected 30 brands  \n",
        "* total number of events\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W9zP5mlbDhbq"
      },
      "outputs": [],
      "source": [
        "def compute_features(events, brands):\n",
        "    features = np.zeros(len(brands))\n",
        "    # number of events per brand in brands\n",
        "    for val in events[\"brand\"].unique():\n",
        "        features[np.where(brands == val)] += np.sum(\n",
        "            events[\"brand\"] == val\n",
        "        )\n",
        "      # total number of events\n",
        "    features = np.hstack([events.shape[0], features])\n",
        "    return features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b3ea_AvpDyyF"
      },
      "outputs": [],
      "source": [
        "user_events = df.groupby(\"user_id\")\n",
        "feature_size = len(BRANDS) + 1\n",
        "features = np.zeros((len(user_ids), feature_size))\n",
        "for user, events in tqdm(user_events):\n",
        "    user_features = compute_features(events, BRANDS)\n",
        "    features[np.where(user_ids == user), :] = user_features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9yK1lFJs6zWp"
      },
      "outputs": [],
      "source": [
        "features.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DXVcBS6z6_J_"
      },
      "outputs": [],
      "source": [
        "print(f\"User id: {user_ids[example_ind]}\")\n",
        "print(f\"Features: {features[example_ind]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uSZVpRe2wBV8"
      },
      "source": [
        "## Training XGBoost model\n",
        "\n",
        "First, we set model parameters:  \n",
        "**n_estimators** — number of trees  \n",
        "**max_depth** —  maximum depth of a tree  \n",
        "**learning_rate** — shrinkage factor used in model updates to prevent overfitting  \n",
        "**objective** — foss function; here we use loss function for binary classification task  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JdPo_Wm4qqL0"
      },
      "outputs": [],
      "source": [
        "N_ESTIMATORS = 10\n",
        "MAX_DEPTH = 4\n",
        "LEARNING_RATE = 0.1\n",
        "OBJECTIVE = 'binary:logistic'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wNJgfF2kw5A2"
      },
      "source": [
        "## Imbalanced data\n",
        "Positive examples are rare in the presented dataset. This means that the number of products bought for the selected brands is low relative to the number of users.\n",
        "\n",
        "**Total number of users**:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eD56Sh9GyXxk"
      },
      "outputs": [],
      "source": [
        "user_ids.size"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oE63-ds0yNno"
      },
      "source": [
        "### Number of users that bought products of a given brands in train target"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZWpQ9BdLxlGz"
      },
      "outputs": [],
      "source": [
        "plt.figure().set_figwidth(18)\n",
        "plt.bar(target_brands, np.sum(train_target, axis=0))\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KwwEzX9ziai"
      },
      "source": [
        "### Number of users that bought products of a given brands in test target"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "POrhXGXpzhTM"
      },
      "outputs": [],
      "source": [
        "plt.figure().set_figwidth(18)\n",
        "plt.bar(target_brands, np.sum(test_target, axis=0))\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mw_G5oWPxonV"
      },
      "source": [
        "### Working with imbalanced data\n",
        "One way to handle imbalanced data working with XGBoost library is to set `scale_pos_weight`. It is a multiplier of a loss function on positive training examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FyIws6E4CXvf"
      },
      "outputs": [],
      "source": [
        "SCALE_POS_WEIGHT = 100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vg_6Xdb891HF"
      },
      "source": [
        "\n",
        "## Model\n",
        "Next, we create an XGBoost model for the classification task and fit it on our data. Generated `features` are what we input to the model and `train_target` is what the model predicts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "43uYQoHVqxt7"
      },
      "outputs": [],
      "source": [
        "from xgboost import XGBClassifier\n",
        "from xgboost import XGBClassifier\n",
        "# create model instance\n",
        "bst = XGBClassifier(n_estimators=N_ESTIMATORS, max_depth=MAX_DEPTH, learning_rate=LEARNING_RATE, scale_pos_weight=SCALE_POS_WEIGHT, n_jobs=10, objective=OBJECTIVE)\n",
        "# fit model\n",
        "bst.fit(features, train_target)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GgsKmV0i0fVC"
      },
      "source": [
        "Finally, we compute the AUC score on the test target:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SFFUquHHq5Ko"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_auc_score\n",
        "# make predictions\n",
        "preds = bst.predict(features)\n",
        "# compute test score\n",
        "test_score = roc_auc_score(test_target, preds)\n",
        "test_score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Mdb4ecv2-xl"
      },
      "source": [
        "## EXERCISE\n",
        "\n",
        "(Advanced) Compute features for different time windows.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "N5h1y66r2_-1"
      },
      "outputs": [],
      "source": [
        "#@title Solution\n",
        "def compute_timewindows_features(events, num_days, max_date, brands):\n",
        "    features = np.array([])\n",
        "    events = events.sort_values(\"event_time\")\n",
        "    for days in num_days:\n",
        "        start_date = max_date - timedelta(days=days)\n",
        "        idx = np.searchsorted(events[\"event_time\"], start_date)\n",
        "        features_per_days = np.zeros(len(brands))\n",
        "        values = events[\"brand\"].to_numpy(na_value=\"nan\")[idx:]\n",
        "        for val in np.unique(values):\n",
        "            features_per_days[np.where(brands == val)] += np.sum(\n",
        "                values == val\n",
        "            )\n",
        "        features_per_days = np.hstack([len(values), features_per_days])\n",
        "        features = np.hstack([features, features_per_days])\n",
        "    return features\n",
        "\n",
        "TIME_WINDOWS = [1, 7, 30]\n",
        "\n",
        "df[\"event_time\"] = pd.to_datetime(df.event_time)\n",
        "user_events = df.groupby(\"user_id\")\n",
        "max_date = df[\"event_time\"].max()\n",
        "feature_size = len(TIME_WINDOWS) * len(BRANDS) + len(TIME_WINDOWS)\n",
        "features = np.zeros((len(user_ids), feature_size))\n",
        "for user, events in tqdm(user_events):\n",
        "    user_features = compute_timewindows_features(events, TIME_WINDOWS, max_date, BRANDS)\n",
        "    features[np.where(user_ids == user), :] = user_features"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}