documentation/tutorials/beginner_colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "36EdAGhThQov"
      },
      "source": [
        "# Build, train and evaluate models with TensorFlow Decision Forests\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/decision_forests/tutorials/beginner_colab\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/decision-forests/blob/main/documentation/tutorials/beginner_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/decision-forests/blob/main/documentation/tutorials/beginner_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/decision-forests/documentation/tutorials/beginner_colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvvDY0LVhuaW"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "Decision Forests (DF) are a large family of Machine Learning algorithms for\n",
        "supervised classification, regression and ranking. As the name suggests, DFs use\n",
        "decision trees as a building block. Today, the two most popular DF training\n",
        "algorithms are [Random Forests](https://en.wikipedia.org/wiki/Random_forest) and\n",
        "[Gradient Boosted Decision Trees](https://en.wikipedia.org/wiki/Gradient_boosting). Both algorithms are ensemble techniques that use multiple decision trees, but differ on how they do it.\n",
        "\n",
        "TensorFlow Decision Forests (TF-DF) is a library for the training,\n",
        "evaluation, interpretation and inference of Decision Forest models.\n",
        "\n",
        "In this tutorial, you will learn how to:\n",
        "\n",
        "1.  Train a binary classification Random Forest on a dataset containing numerical, categorical and missing features.\n",
        "1.  Evaluate the model on a test dataset.\n",
        "1.  Prepare the model for\n",
        "    [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving).\n",
        "1.  Examine the overall structure of the model and the importance of each feature.\n",
        "1.  Re-train the model with a different learning algorithm (Gradient Boosted Decision Trees).\n",
        "1.  Use a different set of input features.\n",
        "1.  Change the hyperparameters of the model.\n",
        "1.  Preprocess the features.\n",
        "1.  Train a model for regression.\n",
        "1.  Train a model for ranking.\n",
        "\n",
        "Detailed documentation is available in the [user manual](https://github.com/tensorflow/decision-forests/documentation).\n",
        "The [example directory](https://github.com/tensorflow/decision-forests/examples) contains other end-to-end examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jK9tCTcwqq4k"
      },
      "source": [
        "## Installing TensorFlow Decision Forests\n",
        "\n",
        "Install TF-DF by running the following cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pa1Pf37RhEYN"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow_decision_forests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vZGda2dOe-hH"
      },
      "source": [
        "Install [Wurlitzer](https://pypi.org/project/wurlitzer/) to display\n",
        "the detailed training logs. This is only needed in colabs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lk26uBSCe8Du"
      },
      "outputs": [],
      "source": [
        "!pip install wurlitzer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3oinwbhXlggd"
      },
      "source": [
        "## Importing libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "52W45tmDjD64"
      },
      "outputs": [],
      "source": [
        "import tensorflow_decision_forests as tfdf\n",
        "\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import math\n",
        "\n",
        "try:\n",
        "  from wurlitzer import sys_pipes\n",
        "except:\n",
        "  from colabtools.googlelog import CaptureLog as sys_pipes\n",
        "\n",
        "from IPython.core.magic import register_line_magic\n",
        "from IPython.display import Javascript"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0LPPwWxYxtDM"
      },
      "source": [
        "The hidden code cell limits the output height in colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2AhqJz3VmQM-"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "\n",
        "# Some of the model training logs can cover the full\n",
        "# screen if not compressed to a smaller viewport.\n",
        "# This magic allows setting a max height for a cell.\n",
        "@register_line_magic\n",
        "def set_cell_height(size):\n",
        "  display(\n",
        "      Javascript(\"google.colab.output.setIframeHeight(0, true, {maxHeight: \" +\n",
        "                 str(size) + \"})\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8gVQ-txtjFU4"
      },
      "outputs": [],
      "source": [
        "# Check the version of TensorFlow Decision Forests\n",
        "print(\"Found TensorFlow Decision Forests v\" + tfdf.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QGRtRECujKeu"
      },
      "source": [
        "## Training a Random Forest model\n",
        "\n",
        "In this section, we train, evaluate, analyse and export a binary classification Random Forest trained on the [Palmer's Penguins](https://allisonhorst.github.io/palmerpenguins/articles/intro.html) dataset.\n",
        "\n",
        "\u003ccenter\u003e\n",
        "\u003cimg src=\"https://allisonhorst.github.io/palmerpenguins/man/figures/palmerpenguins.png\" width=\"150\"/\u003e\u003c/center\u003e\n",
        "\n",
        "**Note:** The dataset was exported to a csv file without pre-processing: `library(palmerpenguins); write.csv(penguins, file=\"penguins.csv\", quote=F, row.names=F)`. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3qsSU1RfmNiP"
      },
      "source": [
        "### Load the dataset and convert it in a tf.Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nJ5igfElg2I"
      },
      "source": [
        "This dataset is very small (300 examples) and stored as a .csv-like file. Therefore, use Pandas to load it.\n",
        "\n",
        "**Note:** Pandas is practical as you don't have to type in name of the input features to load them. For larger datasets (\u003e1M examples), using the\n",
        "[TensorFlow Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) to read the files may be better suited.\n",
        "\n",
        "Let's assemble the dataset into a csv file (i.e. add the header), and load it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "44Jq6g_mJFmj"
      },
      "outputs": [],
      "source": [
        "# Download the dataset\n",
        "!wget -q https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins.csv -O /tmp/penguins.csv\n",
        "\n",
        "# Load a dataset into a Pandas Dataframe.\n",
        "dataset_df = pd.read_csv(\"/tmp/penguins.csv\")\n",
        "\n",
        "# Display the first 3 examples.\n",
        "dataset_df.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23AewWT1lkIK"
      },
      "source": [
        "The dataset contains a mix of numerical (e.g. `bill_depth_mm`), categorical\n",
        "(e.g. `island`) and missing features. TF-DF supports all these feature types natively (differently than NN based models), therefore there is no need for preprocessing in the form of one-hot encoding, normalization or extra `is_present` feature.\n",
        "\n",
        "Labels are a bit different: Keras metrics expect integers. The label (`species`) is stored as a string, so let's convert it into an integer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uO_jz2sj0IBZ"
      },
      "outputs": [],
      "source": [
        "# Encode the categorical label into an integer.\n",
        "#\n",
        "# Details:\n",
        "# This stage is necessary if your classification label is represented as a\n",
        "# string. Note: Keras expected classification labels to be integers.\n",
        "\n",
        "# Name of the label column.\n",
        "label = \"species\"\n",
        "\n",
        "classes = dataset_df[label].unique().tolist()\n",
        "print(f\"Label classes: {classes}\")\n",
        "\n",
        "dataset_df[label] = dataset_df[label].map(classes.index)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vwJjLFhbtozI"
      },
      "source": [
        "Next split the dataset into training and testing:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u7DEIxn2oB3U"
      },
      "outputs": [],
      "source": [
        "# Split the dataset into a training and a testing dataset.\n",
        "\n",
        "def split_dataset(dataset, test_ratio=0.30):\n",
        "  \"\"\"Splits a panda dataframe in two.\"\"\"\n",
        "  test_indices = np.random.rand(len(dataset)) \u003c test_ratio\n",
        "  return dataset[~test_indices], dataset[test_indices]\n",
        "\n",
        "\n",
        "train_ds_pd, test_ds_pd = split_dataset(dataset_df)\n",
        "print(\"{} examples in training, {} examples for testing.\".format(\n",
        "    len(train_ds_pd), len(test_ds_pd)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uWq7uQcCuBzO"
      },
      "source": [
        "And finally, convert the pandas dataframe (`pd.Dataframe`) into tensorflow datasets (`tf.data.Dataset`):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qtXgUBKluTX0"
      },
      "outputs": [],
      "source": [
        "train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label)\n",
        "test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_ds_pd, label=label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BRKLWIWNuOZ1"
      },
      "source": [
        "**Notes:** `pd_dataframe_to_tf_dataset` could have converted the label to integer for you.\n",
        "\n",
        "And, if you wanted to create the `tf.data.Dataset` yourself, there is a couple of things to remember:\n",
        "\n",
        "- The learning algorithms work with a one-epoch dataset and without shuffling.\n",
        "- The batch size does not impact the training algorithm, but a small value might slow down reading the dataset.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mYAoyfYtqHG4"
      },
      "source": [
        "### Train the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xete-FbuqJCV"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "\n",
        "# Specify the model.\n",
        "model_1 = tfdf.keras.RandomForestModel()\n",
        "\n",
        "# Optionally, add evaluation metrics.\n",
        "model_1.compile(\n",
        "    metrics=[\"accuracy\"])\n",
        "\n",
        "# Train the model.\n",
        "# \"sys_pipes\" is optional. It enables the display of the training logs.\n",
        "with sys_pipes():\n",
        "  model_1.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBnjxdip-MC0"
      },
      "source": [
        "### Remarks\n",
        "\n",
        "-   No input features are specified. Therefore, all the columns will be used as\n",
        "    input features except for the label. The feature used by the model are shown\n",
        "    in the training logs and in the `model.summary()`.\n",
        "-   DFs consume natively numerical, categorical, categorical-set features and\n",
        "    missing-values. Numerical features do not need to be normalized. Categorical\n",
        "    string values do not need to be encoded in a dictionary.\n",
        "-   No training hyper-parameters are specified. Therefore the default\n",
        "    hyper-parameters will be used. Default hyper-parameters provide\n",
        "    reasonable results in most situations.\n",
        "-   Calling `compile` on the model before the `fit` is optional. Compile can be\n",
        "    used to provide extra evaluation metrics.\n",
        "-   Training algorithms do not need validation datasets. If a validation dataset\n",
        "    is provided, it will only be used to show metrics.\n",
        "\n",
        "**Note:** A *Categorical-Set* feature is composed of a set of categorical values (while a *Categorical* is only one value). More details and examples are given later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tSdtNJUArBpl"
      },
      "source": [
        "## Evaluate the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Udtu_uS1paSu"
      },
      "source": [
        "Let's evaluate our model on the test dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xUy4ULEMtDXB"
      },
      "outputs": [],
      "source": [
        "evaluation = model_1.evaluate(test_ds, return_dict=True)\n",
        "print()\n",
        "\n",
        "for name, value in evaluation.items():\n",
        "  print(f\"{name}: {value:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tlhfzZ34pfO4"
      },
      "source": [
        "**Remark:** The test accuracy (0.86514) is close to the Out-of-bag accuracy\n",
        "(0.8672) shown in the training logs.\n",
        "\n",
        "See the **Model Self Evaluation** section below for more evaluation methods."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mHBFtUeElRYz"
      },
      "source": [
        "## Prepare this model for TensorFlow Serving."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JbC4lmgfr5Sm"
      },
      "source": [
        "Export the model to the SavedModel format for later re-use e.g.\n",
        "[TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "08YWGr9U2fza"
      },
      "outputs": [],
      "source": [
        "model_1.save(\"/tmp/my_saved_model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6-8R02_SXpbq"
      },
      "source": [
        "## Plot the model\n",
        "\n",
        "Plotting a decision tree and following the first branches helps learning about decision forests. In some cases, plotting a model can even be used for debugging.\n",
        "\n",
        "Because of the difference in the way they are trained, some models are more interresting to plan than others. Because of the noise injected during training and the depth of the trees, plotting Random Forest is less informative than plotting a CART or the first tree of a Gradient Boosted Tree.\n",
        "\n",
        "Never the less, let's plot the first tree of our Random Forest model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KUIxf8N6Yjl0"
      },
      "outputs": [],
      "source": [
        "tfdf.model_plotter.plot_model_in_colab(model_1, tree_idx=0, max_depth=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cPcL_hDnY7Zy"
      },
      "source": [
        "The root node on the left contains the first condition (`bill_depth_mm \u003e= 16.55`), number of examples (240) and label distribution (the red-blue-green bar).\n",
        "\n",
        "Examples that evaluates true to `bill_depth_mm \u003e= 16.55` are branched to the green path. The other ones are branched to the red path.\n",
        "\n",
        "The deeper the node, the more `pure` they become i.e. the label distribution is biased toward a subset of classes. \n",
        "\n",
        "**Note:** Over the mouse on top of the plot for details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ob3ovQ2seVY"
      },
      "source": [
        "## Model tructure and feature importance\n",
        "\n",
        "The overall structure of the model is show with `.summary()`. You will see:\n",
        "\n",
        "-   **Type**: The learning algorithm used to train the model (`Random Forest` in\n",
        "    our case).\n",
        "-   **Task**: The problem solved by the model (`Classification` in our case).\n",
        "-   **Input Features**: The input features of the model.\n",
        "-   **Variable Importance**: Different measures of the importance of each\n",
        "    feature for the model.\n",
        "-   **Out-of-bag evaluation**: The out-of-bag evaluation of the model. This is a\n",
        "    cheap and efficient alternative to cross-validation.\n",
        "-   **Number of {trees, nodes} and other metrics**: Statistics about the\n",
        "    structure of the decisions forests.\n",
        "\n",
        "**Remark:** The summary's content depends on the learning algorithm (e.g.\n",
        "Out-of-bag is only available for Random Forest) and the hyper-parameters (e.g.\n",
        "the *mean-decrease-in-accuracy* variable importance can be disabled in the\n",
        "hyper-parameters)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kzXME28Lq7Il"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "model_1.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4ApRpUm02zU"
      },
      "source": [
        "The information in ``summary`` are all available programatically using the model inspector:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G3xuB3jN1Cww"
      },
      "outputs": [],
      "source": [
        "# The input features\n",
        "model_1.make_inspector().features()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BZ2RBbU51L6s"
      },
      "outputs": [],
      "source": [
        "# The feature importances\n",
        "model_1.make_inspector().variable_importances()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zvyRJVk1aEk"
      },
      "source": [
        "The content of the summary and the inspector depends on the learning algorithm (`tfdf.keras.RandomForestModel` in this case) and its hyper-parameters (e.g. `compute_oob_variable_importances=True` will trigger the computation of Out-of-bag variable importances for the Random Forest learner)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tFVmrHtWXYKY"
      },
      "source": [
        "## Model Self Evaluation\n",
        "\n",
        "During training TFDF models can self evaluate even if no validation dataset is provided to the `fit()` method. The exact logic depends on the model. For example, Random Forest will use Out-of-bag evaluation while Gradient Boosted Trees will use internal train-validation.\n",
        "\n",
        "**Note:** While this evaluation is  computed during training, it is NOT computed on the training dataset and can be used as a low quality evaluation.\n",
        "\n",
        "The model self evaluation is available with the inspector's `evaluation()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BZPzyIMmYmsI"
      },
      "outputs": [],
      "source": [
        "model_1.make_inspector().evaluation()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vBSz-jE0Qss_"
      },
      "source": [
        "## Plotting the training logs\n",
        "\n",
        "The training logs show the quality of the model (e.g. accuracy evaluated on the out-of-bag or validation dataset) according to the number of trees in the model. These logs are helpful to study the balance between model size and model quality.\n",
        "\n",
        "The logs are available in multiple ways:\n",
        "\n",
        "1. Displayed in during training if `fit()` is wrapped in `with sys_pipes():` (see example above).\n",
        "1. At the end of the model summary i.e. `model.summary()` (see example above).\n",
        "1. Programmatically, using the model inspector i.e. `model.make_inspector().training_logs()`.\n",
        "1. Using [TensorBoard](https://www.tensorflow.org/tensorboard)\n",
        "\n",
        "Let's try the options 2 and 3:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZbRk7xvpTKQG"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 150\n",
        "model_1.make_inspector().training_logs()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WynFJCEbhuF_"
      },
      "source": [
        "Let's plot it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xzPH7Gggh0g1"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "logs = model_1.make_inspector().training_logs()\n",
        "\n",
        "plt.figure(figsize=(12, 4))\n",
        "\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.plot([log.num_trees for log in logs], [log.evaluation.accuracy for log in logs])\n",
        "plt.xlabel(\"Number of trees\")\n",
        "plt.ylabel(\"Accuracy (out-of-bag)\")\n",
        "\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.plot([log.num_trees for log in logs], [log.evaluation.loss for log in logs])\n",
        "plt.xlabel(\"Number of trees\")\n",
        "plt.ylabel(\"Logloss (out-of-bag)\")\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w1xzugBRhwuN"
      },
      "source": [
        "This dataset is small. You can see the model converging almost immediately.\n",
        "\n",
        "Let's use TensorBoard:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5R_m-JmvU9tu"
      },
      "outputs": [],
      "source": [
        "# This cell start TensorBoard that can be slow.\n",
        "# Load the TensorBoard notebook extension\n",
        "%load_ext tensorboard\n",
        "# Google internal version\n",
        "# %load_ext google3.learning.brain.tensorboard.notebook.extension"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j6mp7K6HWwqQ"
      },
      "outputs": [],
      "source": [
        "# Clear existing results (if any)\n",
        "!rm -fr \"/tmp/tensorboard_logs\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "16NbLILYo124"
      },
      "outputs": [],
      "source": [
        "# Export the meta-data to tensorboard.\n",
        "model_1.make_inspector().export_to_tensorboard(\"/tmp/tensorboard_logs\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TSsN6aTXW0LJ"
      },
      "outputs": [],
      "source": [
        "# docs_infra: no_execute\n",
        "# Start a tensorboard instance.\n",
        "%tensorboard --logdir \"/tmp/tensorboard_logs\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_tlSccjZ8kE"
      },
      "source": [
        "\u003c!-- \u003cimg class=\"tfo-display-only-on-site\" src=\"images/beginner_tensorboard.png\"/\u003e --\u003e\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "phTUr6F1t-_E"
      },
      "source": [
        "## Re-train the model with a different learning algorithm\n",
        "\n",
        "The learning algorithm is defined by the model class. For\n",
        "example, `tfdf.keras.RandomForestModel()` trains a Random Forest, while\n",
        "`tfdf.keras.GradientBoostedTreesModel()` trains a Gradient Boosted Decision\n",
        "Trees.\n",
        "\n",
        "The learning algorithms are listed by calling `tfdf.keras.get_all_models()` or in the\n",
        "[learner list](https://github.com/google/yggdrasil-decision-forests/manual/learners)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OwEAAzUZq2m8"
      },
      "outputs": [],
      "source": [
        "tfdf.keras.get_all_models()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmzvuI78voD4"
      },
      "source": [
        "The description of the learning algorithms and their hyper-parameters are also available in the [API reference](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf) and builtin help:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2hONToBav4DE"
      },
      "outputs": [],
      "source": [
        "# help works anywhere.\n",
        "help(tfdf.keras.RandomForestModel)\n",
        "\n",
        "# ? only works in ipython or notebooks, it usually opens on a separate panel.\n",
        "tfdf.keras.RandomForestModel?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuWEYvXaiwhk"
      },
      "source": [
        "## Using a subset of features\n",
        "\n",
        "The previous example did not specify the features, so all the columns were used\n",
        "as input feature (except for the label). The following example shows how to\n",
        "specify input features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sgn_LnRz3M7z"
      },
      "outputs": [],
      "source": [
        "feature_1 = tfdf.keras.FeatureUsage(name=\"bill_length_mm\")\n",
        "feature_2 = tfdf.keras.FeatureUsage(name=\"island\")\n",
        "\n",
        "all_features = [feature_1, feature_2]\n",
        "\n",
        "# Note: This model is only trained with two features. It will not be as good as\n",
        "# the one trained on all features.\n",
        "\n",
        "model_2 = tfdf.keras.GradientBoostedTreesModel(\n",
        "    features=all_features, exclude_non_specified_features=True)\n",
        "\n",
        "model_2.compile(metrics=[\"accuracy\"])\n",
        "model_2.fit(x=train_ds, validation_data=test_ds)\n",
        "\n",
        "print(model_2.evaluate(test_ds, return_dict=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zvM84cgCmbUR"
      },
      "source": [
        "**Note:** As expected, the accuracy is lower than previously."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFmqpivc7x7p"
      },
      "source": [
        "**TF-DF** attaches a **semantics** to each feature. This semantics controls how\n",
        "the feature is used by the model. The following semantics are currently supported:\n",
        "\n",
        "-   **Numerical**: Generally for quantities or counts with full ordering. For\n",
        "    example, the age of a person, or the number of items in a bag. Can be a\n",
        "    float or an integer. Missing values are represented with float(Nan) or with\n",
        "    an empty sparse tensor.\n",
        "-   **Categorical**: Generally for a type/class in finite set of possible values\n",
        "    without ordering. For example, the color RED in the set {RED, BLUE, GREEN}.\n",
        "    Can be a string or an integer. Missing values are represented as \"\" (empty\n",
        "    sting), value -2 or with an empty sparse tensor.\n",
        "-   **Categorical-Set**: A set of categorical values. Great to represent\n",
        "    tokenized text. Can be a string or an integer in a sparse tensor or a\n",
        "    ragged tensor (recommended). The order/index of each item doesn't matter.\n",
        "\n",
        "If not specified, the semantics is inferred from the representation type and shown in the training logs:\n",
        "\n",
        "- int, float (dense or sparse) → Numerical semantics.\n",
        "- str (dense or sparse) → Categorical semantics\n",
        "- int, str (ragged) → Categorical-Set semantics\n",
        "\n",
        "In some cases, the inferred semantics is incorrect. For example: An Enum stored as an integer is semantically categorical, but it will be detected as numerical. In this case, you should specify the semantic argument in the input. The `education_num` field of the Adult dataset is classical example.\n",
        "\n",
        "This dataset doesn't contain such a feature. However, for the demonstration, we will make the model treat the `year` as a categorical feature:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RNRIwLYC8zrp"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "\n",
        "feature_1 = tfdf.keras.FeatureUsage(name=\"year\", semantic=tfdf.keras.FeatureSemantic.CATEGORICAL)\n",
        "feature_2 = tfdf.keras.FeatureUsage(name=\"bill_length_mm\")\n",
        "feature_3 = tfdf.keras.FeatureUsage(name=\"sex\")\n",
        "all_features = [feature_1, feature_2, feature_3]\n",
        "\n",
        "model_3 = tfdf.keras.GradientBoostedTreesModel(features=all_features, exclude_non_specified_features=True)\n",
        "model_3.compile( metrics=[\"accuracy\"])\n",
        "\n",
        "with sys_pipes():\n",
        "  model_3.fit(x=train_ds, validation_data=test_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2AQaNwihcpP7"
      },
      "source": [
        "Note that `year` is in the list of CATEGORICAL features (unlike the first run)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GYrw7nKN40Vm"
      },
      "source": [
        "## Hyper-parameters\n",
        "\n",
        "**Hyper-parameters** are parameters of the training algorithm that impact\n",
        "the quality of the final model. They are specified in the model class\n",
        "constructor. The list of hyper-parameters is visible with the *question mark* colab command (e.g. `?tfdf.keras.GradientBoostedTreesModel`).\n",
        "\n",
        "Alternatively, you can find them on the [TensorFlow Decision Forest Github](https://github.com/tensorflow/decision-forests/keras/wrappers_pre_generated.py) or the [Yggdrasil Decision Forest documentation](https://github.com/google/yggdrasil_decision_forests/documentation/learners).\n",
        "\n",
        "The default hyper-parameters of each algorithm matches approximatively the initial publication paper. To ensure consistancy, new features and their matching hyper-parameters are always disable by default. That's why it is a good idea to tune your hyper-parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vHgPr4Pt43hv"
      },
      "outputs": [],
      "source": [
        "# A classical but slighly more complex model.\n",
        "model_6 = tfdf.keras.GradientBoostedTreesModel(\n",
        "    num_trees=500, growing_strategy=\"BEST_FIRST_GLOBAL\", max_depth=8)\n",
        "model_6.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uECgPGDc2P4p"
      },
      "outputs": [],
      "source": [
        "# A more complex, but possibly, more accurate model.\n",
        "model_7 = tfdf.keras.GradientBoostedTreesModel(\n",
        "    num_trees=500,\n",
        "    growing_strategy=\"BEST_FIRST_GLOBAL\",\n",
        "    max_depth=8,\n",
        "    split_axis=\"SPARSE_OBLIQUE\",\n",
        "    categorical_algorithm=\"RANDOM\",\n",
        "    )\n",
        "model_7.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xk7wEmUZu3V0"
      },
      "source": [
        "As new training methods are published and implemented, combinaisons of hyper-parameters can emerge as good or almost-always-better than the default parameters. To avoid changing the default hyper-parameter values these good combinaisons are indexed and available as hyper-parameter templates.\n",
        "\n",
        "For example, the `benchmark_rank1` template is the best combinaison on our internal benchmarks. Those templates are versioned to allow training configuration stability e.g. `benchmark_rank1@v1`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LtrRhMhj3hSu"
      },
      "outputs": [],
      "source": [
        "# A good template of hyper-parameters.\n",
        "model_8 = tfdf.keras.GradientBoostedTreesModel(hyperparameter_template=\"benchmark_rank1\")\n",
        "model_8.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSDXcKXB3u6M"
      },
      "source": [
        "The available tempaltes are available with `predefined_hyperparameters`. Note that different learning algorithms have different templates, even if the name is similar."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MQrWI2iv37Bo"
      },
      "outputs": [],
      "source": [
        "# The hyper-parameter templates of the Gradient Boosted Tree model.\n",
        "print(tfdf.keras.GradientBoostedTreesModel.predefined_hyperparameters())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcX4tov1_lwp"
      },
      "source": [
        "## Feature Preprocessing\n",
        "\n",
        "Pre-processing features is sometimes necessary to consume signals with complex\n",
        "structures, to regularize the model or to apply transfer learning.\n",
        "Pre-processing can be done in one of three ways:\n",
        "\n",
        "1.  Preprocessing on the Pandas dataframe. This solution is easy to implement\n",
        "    and generally suitable for experimentation. However, the\n",
        "    pre-processing logic will not be exported in the model by `model.save()`.\n",
        "\n",
        "2.  [Keras Preprocessing](https://keras.io/guides/preprocessing_layers/): While\n",
        "    more complex than the previous solution, Keras Preprocessing is packaged in\n",
        "    the model.\n",
        "\n",
        "3.  [TensorFlow Feature Columns](https://www.tensorflow.org/tutorials/structured_data/feature_columns):\n",
        "    This API is part of the TF Estimator library (!= Keras) and planned for\n",
        "    deprecation. This solution is interesting when using existing preprocessing\n",
        "    code.\n",
        "\n",
        "Note: Using [TensorFlow Hub](https://www.tensorflow.org/hub)\n",
        "pre-trained embedding is often, a great way to consume text and image with\n",
        "TF-DF. For example, `hub.KerasLayer(\"https://tfhub.dev/google/nnlm-en-dim128/2\")`. See the [Intermediate tutorial](intermediate_colab.ipynb) for more details.\n",
        "\n",
        "In the next example, pre-process the `body_mass_g` feature into `body_mass_kg = body_mass_g / 1000`. The `bill_length_mm` is consumed without pre-processing. Note that such\n",
        "monotonic transformations have generally no impact on decision forest models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tGcIvTeKAApp"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "\n",
        "body_mass_g = tf.keras.layers.Input(shape=(1,), name=\"body_mass_g\")\n",
        "body_mass_kg = body_mass_g / 1000.0\n",
        "\n",
        "bill_length_mm = tf.keras.layers.Input(shape=(1,), name=\"bill_length_mm\")\n",
        "\n",
        "raw_inputs = {\"body_mass_g\": body_mass_g, \"bill_length_mm\": bill_length_mm}\n",
        "processed_inputs = {\"body_mass_kg\": body_mass_kg, \"bill_length_mm\": bill_length_mm}\n",
        "\n",
        "# \"preprocessor\" contains the preprocessing logic.\n",
        "preprocessor = tf.keras.Model(inputs=raw_inputs, outputs=processed_inputs)\n",
        "\n",
        "# \"model_4\" contains both the pre-processing logic and the decision forest.\n",
        "model_4 = tfdf.keras.RandomForestModel(preprocessing=preprocessor)\n",
        "model_4.fit(x=train_ds)\n",
        "\n",
        "model_4.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1Bx3Feyjb2o"
      },
      "source": [
        "The following example re-implements the same logic using TensorFlow Feature\n",
        "Columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fnwe3sBt-yJk"
      },
      "outputs": [],
      "source": [
        "def g_to_kg(x):\n",
        "  return x / 1000\n",
        "\n",
        "feature_columns = [\n",
        "    tf.feature_column.numeric_column(\"body_mass_g\", normalizer_fn=g_to_kg),\n",
        "    tf.feature_column.numeric_column(\"bill_length_mm\"),\n",
        "]\n",
        "\n",
        "preprocessing = tf.keras.layers.DenseFeatures(feature_columns)\n",
        "\n",
        "model_5 = tfdf.keras.RandomForestModel(preprocessing=preprocessing)\n",
        "model_5.compile(metrics=[\"accuracy\"])\n",
        "model_5.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vif6gsAjfzv"
      },
      "source": [
        "## Training a regression model\n",
        "\n",
        "The previous example trains a classification model (TF-DF does not differentiate\n",
        "between binary classification and multi-class classification). In the next\n",
        "example, train a regression model on the\n",
        "[Abalone dataset](https://archive.ics.uci.edu/ml/datasets/abalone). The\n",
        "objective of this dataset is to predict the number of shell's rings of an\n",
        "abalone.\n",
        "\n",
        "**Note:** The csv file is assembled by appending UCI's header and data files. No preprocessing was applied.\n",
        "\n",
        "\u003ccenter\u003e\n",
        "\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/LivingAbalone.JPG/800px-LivingAbalone.JPG\" width=\"200\"/\u003e\u003c/center\u003e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0uKI_Uy7RyWN"
      },
      "outputs": [],
      "source": [
        "# Download the dataset.\n",
        "!wget -q https://storage.googleapis.com/download.tensorflow.org/data/abalone_raw.csv -O /tmp/abalone.csv\n",
        "\n",
        "dataset_df = pd.read_csv(\"/tmp/abalone.csv\")\n",
        "print(dataset_df.head(3))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_gjrquQySU7Q"
      },
      "outputs": [],
      "source": [
        "# Split the dataset into a training and testing dataset.\n",
        "train_ds_pd, test_ds_pd = split_dataset(dataset_df)\n",
        "print(\"{} examples in training, {} examples for testing.\".format(\n",
        "    len(train_ds_pd), len(test_ds_pd)))\n",
        "\n",
        "# Name of the label column.\n",
        "label = \"Rings\"\n",
        "\n",
        "train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)\n",
        "test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t8fUhQKISqYT"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 300\n",
        "\n",
        "# Configure the model.\n",
        "model_7 = tfdf.keras.RandomForestModel(task = tfdf.keras.Task.REGRESSION)\n",
        "\n",
        "# Optional.\n",
        "model_7.compile(metrics=[\"mse\"])\n",
        "\n",
        "# Train the model.\n",
        "with sys_pipes():\n",
        "  model_7.fit(x=train_ds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aSriIAaMSzwA"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model on the test dataset.\n",
        "evaluation = model_7.evaluate(test_ds, return_dict=True)\n",
        "\n",
        "print(evaluation)\n",
        "print()\n",
        "print(f\"MSE: {evaluation['mse']}\")\n",
        "print(f\"RMSE: {math.sqrt(evaluation['mse'])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S54mR6i9jkhp"
      },
      "source": [
        "## Training a ranking model\n",
        "\n",
        "Finaly, after having trained a classification and a regression models, train a [ranking](https://en.wikipedia.org/wiki/Learning_to_rank) model.\n",
        "\n",
        "The goal of a ranking is to **order** items by importance. The \"value\" of\n",
        "relevance does not matter directly. Ranking a set of *documents* with regard to\n",
        "a user *query* is an example of ranking problem: It is only important to get the right order, where the top documents matter more.\n",
        "\n",
        "TF-DF expects for ranking datasets to be presented in a \"flat\" format. A\n",
        "document+query dataset might look like that:\n",
        "\n",
        "query | document_id | feature_1 | feature_2 | relevance/label\n",
        "----- | ----------- | --------- | --------- | ---------------\n",
        "cat   | 1           | 0.1       | blue      | 4\n",
        "cat   | 2           | 0.5       | green     | 1\n",
        "cat   | 3           | 0.2       | red       | 2\n",
        "dog   | 4           | NA        | red       | 0\n",
        "dog   | 5           | 0.2       | red       | 1\n",
        "dog   | 6           | 0.6       | green     | 1\n",
        "\n",
        "The *relevance/label* is a floating point numerical value between 0 and 5\n",
        "(generally between 0 and 4) where 0 means \"completely unrelated\", 4 means \"very\n",
        "relevant\" and 5 means \"the same as the query\".\n",
        "\n",
        "Interestingly, decision forests are often good rankers, and many\n",
        "state-of-the-art ranking models are decision forests.\n",
        "\n",
        "In this example, use a sample of the\n",
        "[LETOR3](https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/#!letor-3-0)\n",
        "dataset. More precisely, we want to download the `OHSUMED.zip` from [the LETOR3 repo](https://onedrive.live.com/?authkey=%21ACnoZZSZVfHPJd0\u0026id=8FEADC23D838BDA8%21107\u0026cid=8FEADC23D838BDA8). This dataset is stored in the\n",
        "libsvm format, so we will need to convert it to csv."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "axD6x1ZivHCS"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 200\n",
        "\n",
        "archive_path = tf.keras.utils.get_file(\"letor.zip\",\n",
        "  \"https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip\",\n",
        "  extract=True)\n",
        "\n",
        "# Path to the train and test dataset using libsvm format.\n",
        "raw_dataset_path = os.path.join(os.path.dirname(archive_path),\"OHSUMED/Data/All/OHSUMED.txt\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rcManr98ZGID"
      },
      "source": [
        "The dataset is stored as a .txt file in a specific format, so first convert it into a csv file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mkiM9HJox-e8"
      },
      "outputs": [],
      "source": [
        "def convert_libsvm_to_csv(src_path, dst_path):\n",
        "  \"\"\"Converts a libsvm ranking dataset into a flat csv file.\n",
        "  \n",
        "  Note: This code is specific to the LETOR3 dataset.\n",
        "  \"\"\"\n",
        "  dst_handle = open(dst_path, \"w\")\n",
        "  first_line = True\n",
        "  for src_line in open(src_path,\"r\"):\n",
        "    # Note: The last 3 items are comments.\n",
        "    items = src_line.split(\" \")[:-3]\n",
        "    relevance = items[0]\n",
        "    group = items[1].split(\":\")[1]\n",
        "    features = [ item.split(\":\") for item in items[2:]]\n",
        "\n",
        "    if first_line:\n",
        "      # Csv header\n",
        "      dst_handle.write(\"relevance,group,\" + \",\".join([\"f_\" + feature[0] for feature in features]) + \"\\n\")\n",
        "      first_line = False\n",
        "    dst_handle.write(relevance + \",g_\" + group + \",\" + (\",\".join([feature[1] for feature in features])) + \"\\n\")\n",
        "  dst_handle.close()\n",
        "\n",
        "# Convert the dataset.\n",
        "csv_dataset_path=\"/tmp/ohsumed.csv\"\n",
        "convert_libsvm_to_csv(raw_dataset_path, csv_dataset_path)\n",
        "\n",
        "# Load a dataset into a Pandas Dataframe.\n",
        "dataset_df = pd.read_csv(csv_dataset_path)\n",
        "\n",
        "# Display the first 3 examples.\n",
        "dataset_df.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wB7bWAja1G-o"
      },
      "outputs": [],
      "source": [
        "train_ds_pd, test_ds_pd = split_dataset(dataset_df)\n",
        "print(\"{} examples in training, {} examples for testing.\".format(\n",
        "    len(train_ds_pd), len(test_ds_pd)))\n",
        "\n",
        "# Display the first 3 examples of the training dataset.\n",
        "train_ds_pd.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YQKqN9zN4L00"
      },
      "source": [
        "In this dataset, the `relevance` defines the ground-truth rank among rows of the same `group`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5QMbBkCEXxu_"
      },
      "outputs": [],
      "source": [
        "# Name of the relevance and grouping columns.\n",
        "relevance = \"relevance\"\n",
        "\n",
        "ranking_train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)\n",
        "ranking_test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ba1gb75SX1rr"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 400\n",
        "\n",
        "model_8 = tfdf.keras.GradientBoostedTreesModel(\n",
        "    task=tfdf.keras.Task.RANKING,\n",
        "    ranking_group=\"group\",\n",
        "    num_trees=50)\n",
        "\n",
        "with sys_pipes():\n",
        "  model_8.fit(x=ranking_train_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "spZCfxfR3VK0"
      },
      "source": [
        "At this point, keras does not propose any ranking metrics. Instead, the training and validation (a GBDT uses a validation dataset) are shown in the training\n",
        "logs. In this case the loss is `LAMBDA_MART_NDCG5`, and the final (i.e. at\n",
        "the end of the training) NDCG (normalized discounted cumulative gain) is `0.510136` (see line `Final model valid-loss: -0.510136`).\n",
        "\n",
        "Note that the NDCG is a value between 0 and 1. The larget the NDCG, the better\n",
        "the model. For this reason, the loss to be -NDCG.\n",
        "\n",
        "As before, the model can be analysed:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L4N1R8fM4jFh"
      },
      "outputs": [],
      "source": [
        "%set_cell_height 400\n",
        "\n",
        "model_8.summary()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "beginner_colab.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}