weak_recsys_steam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weakly Supervised Recommendation Systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiments steps:\n",
    " 1. **User's Preferences Model**: Leverage the most *explicit* ratings to build a *rate/rank prediction model*. This is a simple *Explicit Matrix Factorization* model. \n",
    " 2. **Generate Weak DataSet**: Use the above model to *predict* for all user/item pairs $(u,i)$ in *implicit feedback dataset* to build a new *weak explicit dataset* $(u, i, r^*)$.\n",
    " 3. **Evaluate**: Use the intact test split in the most explicit feedback, in order to evaluate the performance of any model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explicit Model Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains all the experiments based on the explicit matrix factorization model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explicit Review/Recommend Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explicit dataset (TEST) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (VALID) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (TRAIN) contains 11,324 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PLAY) contains 217,860 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PURCHASE) contains 344,292 interactions of 1,781 users and 9,535 items\n",
      "--------------------\n",
      "RMSE: 0.4613\n",
      "MRR: 0.0391\n",
      "nDCG: 0.0698\n",
      "nDCG@10: 0.0530\n",
      "nDCG@5: 0.0331\n",
      "MAP: 0.0343\n",
      "success@10: 0.1475\n",
      "success@5: 0.0758\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "from spotlight.evaluation import rmse_score\n",
    "\n",
    "dataset_recommend_train, dataset_recommend_test, dataset_recommend_dev, dataset_play, dataset_purchase = utils.parse_steam()\n",
    "\n",
    "print('Explicit dataset (TEST) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_test.ratings), ','),\n",
    "          format(dataset_recommend_test.num_users, ','),\n",
    "          format(dataset_recommend_test.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (VALID) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_dev.ratings), ','),\n",
    "          format(dataset_recommend_dev.num_users, ','),\n",
    "          format(dataset_recommend_dev.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (TRAIN) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_train.ratings), ','),\n",
    "          format(dataset_recommend_train.num_users, ','),\n",
    "          format(dataset_recommend_train.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PLAY) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_play.ratings), ','),\n",
    "          format(dataset_play.num_users, ','),\n",
    "          format(dataset_play.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PURCHASE) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_purchase.ratings), ','),\n",
    "          format(dataset_purchase.num_users, ','),\n",
    "          format(dataset_purchase.num_items, ',')))\n",
    "\n",
    "# train the explicit model based on recommend feedback\n",
    "model = utils.train_explicit(train_interactions=dataset_recommend_train, \n",
    "                             valid_interactions=dataset_recommend_dev,\n",
    "                             run_name='model_steam_explicit_recommend')\n",
    "\n",
    "mrr, ndcg, ndcg10, ndcg_5, mmap, success_10, success_5 = utils.evaluate(interactions=dataset_recommend_test,\n",
    "                                                                        model=model,\n",
    "                                                                        topk=20)\n",
    "rmse = rmse_score(model=model, test=dataset_recommend_test)\n",
    "print('-'*20)\n",
    "print('RMSE: {:.4f}'.format(rmse))\n",
    "print('MRR: {:.4f}'.format(mrr))\n",
    "print('nDCG: {:.4f}'.format(ndcg))\n",
    "print('nDCG@10: {:.4f}'.format(ndcg10))\n",
    "print('nDCG@5: {:.4f}'.format(ndcg_5))\n",
    "print('MAP: {:.4f}'.format(mmap))\n",
    "print('success@10: {:.4f}'.format(success_10))\n",
    "print('success@5: {:.4f}'.format(success_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove all valid/test rating values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_interact = set()\n",
    "for (uid, iid) in zip(dataset_recommend_test.user_ids, dataset_recommend_test.item_ids):\n",
    "    test_interact.add((uid, iid))\n",
    "\n",
    "for (uid, iid) in zip(dataset_recommend_dev.user_ids, dataset_recommend_dev.item_ids):\n",
    "    test_interact.add((uid, iid))\n",
    "\n",
    "# clean implicit dataset from test/dev rating\n",
    "for idx, (uid, iid, r) in enumerate(zip(dataset_play.user_ids, dataset_play.item_ids, dataset_play.ratings)):\n",
    "    if (uid, iid) in test_interact:\n",
    "        dataset_play.ratings[idx] = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explicit Play Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leverage the **explicit review/recommend model** trained at the previous section to annotate **missing values** in the **play** dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1 start at:  Tue Apr 23 09:05:08 2019\n",
      "epoch 1 end at:  Tue Apr 23 09:05:09 2019\n",
      "RMSE: 0.4413\n",
      "epoch 2 start at:  Tue Apr 23 09:05:09 2019\n",
      "epoch 2 end at:  Tue Apr 23 09:05:10 2019\n",
      "RMSE: 0.4353\n",
      "epoch 3 start at:  Tue Apr 23 09:05:11 2019\n",
      "epoch 3 end at:  Tue Apr 23 09:05:11 2019\n",
      "RMSE: 0.4419\n",
      "--------------------\n",
      "RMSE: 0.4590\n",
      "MRR: 0.0460\n",
      "nDCG: 0.0661\n",
      "nDCG@10: 0.0534\n",
      "nDCG@5: 0.0418\n",
      "MAP: 0.0382\n",
      "success@10: 0.1242\n",
      "success@5: 0.0899\n"
     ]
    }
   ],
   "source": [
    "# annotate the missing values in the play dataset based on the explicit recommend model\n",
    "dataset_play = utils.annotate(interactions=dataset_play, \n",
    "                              model=model, \n",
    "                              run_name='dataset_steam_play_explicit_annotated')\n",
    "\n",
    "# train the explicit model based on recommend feedback\n",
    "model = utils.train_explicit(train_interactions=dataset_play, \n",
    "                             valid_interactions=dataset_recommend_dev,\n",
    "                             run_name='model_steam_explicit_play')\n",
    "\n",
    "# evaluate the new model\n",
    "mrr, ndcg, ndcg10, ndcg_5, mmap, success_10, success_5 = utils.evaluate(interactions=dataset_recommend_test,\n",
    "                                                                        model=model,\n",
    "                                                                        topk=20)\n",
    "rmse = rmse_score(model=model, test=dataset_recommend_test)\n",
    "print('-'*20)\n",
    "print('RMSE: {:.4f}'.format(rmse))\n",
    "print('MRR: {:.4f}'.format(mrr))\n",
    "print('nDCG: {:.4f}'.format(ndcg))\n",
    "print('nDCG@10: {:.4f}'.format(ndcg10))\n",
    "print('nDCG@5: {:.4f}'.format(ndcg_5))\n",
    "print('MAP: {:.4f}'.format(mmap))\n",
    "print('success@10: {:.4f}'.format(success_10))\n",
    "print('success@5: {:.4f}'.format(success_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implicit Model Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains all the experiments based on the implicit matrix factorization model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implicit Review/Recommend Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explicit dataset (TEST) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (VALID) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (TRAIN) contains 11,324 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PLAY) contains 217,860 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PURCHASE) contains 344,292 interactions of 1,781 users and 9,535 items\n",
      "epoch 1 start at:  Sat Apr 20 11:06:27 2019\n",
      "epoch 1 end at:  Sat Apr 20 11:06:28 2019\n",
      "MRR: 0.0741\n",
      "epoch 2 start at:  Sat Apr 20 11:06:31 2019\n",
      "epoch 2 end at:  Sat Apr 20 11:06:32 2019\n",
      "MRR: 0.0800\n",
      "epoch 3 start at:  Sat Apr 20 11:06:35 2019\n",
      "epoch 3 end at:  Sat Apr 20 11:06:36 2019\n",
      "MRR: 0.0857\n",
      "epoch 4 start at:  Sat Apr 20 11:06:39 2019\n",
      "epoch 4 end at:  Sat Apr 20 11:06:40 2019\n",
      "MRR: 0.0856\n",
      "--------------------\n",
      "RMSE: 7.5434\n",
      "MRR: 0.0793\n",
      "nDCG: 0.0936\n",
      "nDCG@10: 0.0818\n",
      "nDCG@5: 0.0681\n",
      "MAP: 0.0581\n",
      "success@10: 0.1951\n",
      "success@5: 0.1416\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "from spotlight.evaluation import rmse_score\n",
    "\n",
    "dataset_recommend_train, dataset_recommend_test, dataset_recommend_dev, dataset_play, dataset_purchase = utils.parse_steam()\n",
    "\n",
    "print('Explicit dataset (TEST) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_test.ratings), ','),\n",
    "          format(dataset_recommend_test.num_users, ','),\n",
    "          format(dataset_recommend_test.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (VALID) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_dev.ratings), ','),\n",
    "          format(dataset_recommend_dev.num_users, ','),\n",
    "          format(dataset_recommend_dev.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (TRAIN) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_train.ratings), ','),\n",
    "          format(dataset_recommend_train.num_users, ','),\n",
    "          format(dataset_recommend_train.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PLAY) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_play.ratings), ','),\n",
    "          format(dataset_play.num_users, ','),\n",
    "          format(dataset_play.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PURCHASE) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_purchase.ratings), ','),\n",
    "          format(dataset_purchase.num_users, ','),\n",
    "          format(dataset_purchase.num_items, ',')))\n",
    "\n",
    "# train the explicit model based on recommend feedback\n",
    "model = utils.train_implicit_negative_sampling(train_interactions=dataset_play,\n",
    "                                               valid_interactions=dataset_recommend_dev,\n",
    "                                               run_name='model_steam_implicit')\n",
    "\n",
    "mrr, ndcg, ndcg10, ndcg_5, mmap, success_10, success_5 = utils.evaluate(interactions=dataset_recommend_test,\n",
    "                                                                        model=model,\n",
    "                                                                        topk=20)\n",
    "rmse = rmse_score(model=model, test=dataset_recommend_test)\n",
    "print('-'*20)\n",
    "print('RMSE: {:.4f}'.format(rmse))\n",
    "print('MRR: {:.4f}'.format(mrr))\n",
    "print('nDCG: {:.4f}'.format(ndcg))\n",
    "print('nDCG@10: {:.4f}'.format(ndcg10))\n",
    "print('nDCG@5: {:.4f}'.format(ndcg_5))\n",
    "print('MAP: {:.4f}'.format(mmap))\n",
    "print('success@10: {:.4f}'.format(success_10))\n",
    "print('success@5: {:.4f}'.format(success_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explicit dataset (TEST) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (VALID) contains 1,416 interactions of 1,781 users and 9,535 items\n",
      "Explicit dataset (TRAIN) contains 11,324 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PLAY) contains 217,860 interactions of 1,781 users and 9,535 items\n",
      "Implicit dataset (PURCHASE) contains 344,292 interactions of 1,781 users and 9,535 items\n",
      "fit the model\n",
      "evaluate the model\n",
      "MRR: 0.1126\n",
      "nDCG: 0.1296\n",
      "nDCG@10: 0.1150\n",
      "nDCG@5: 0.0946\n",
      "MAP: 0.0871\n",
      "success@10: 0.2437\n",
      "success@5: 0.1709\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "from popularity import PopularityModel\n",
    "from spotlight.evaluation import rmse_score\n",
    "\n",
    "dataset_recommend_train, dataset_recommend_test, dataset_recommend_dev, dataset_play, dataset_purchase = utils.parse_steam()\n",
    "\n",
    "print('Explicit dataset (TEST) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_test.ratings), ','),\n",
    "          format(dataset_recommend_test.num_users, ','),\n",
    "          format(dataset_recommend_test.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (VALID) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_dev.ratings), ','),\n",
    "          format(dataset_recommend_dev.num_users, ','),\n",
    "          format(dataset_recommend_dev.num_items, ',')))\n",
    "\n",
    "print('Explicit dataset (TRAIN) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_recommend_train.ratings), ','),\n",
    "          format(dataset_recommend_train.num_users, ','),\n",
    "          format(dataset_recommend_train.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PLAY) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_play.ratings), ','),\n",
    "          format(dataset_play.num_users, ','),\n",
    "          format(dataset_play.num_items, ',')))\n",
    "\n",
    "print('Implicit dataset (PURCHASE) contains %s interactions of %s users and %s items'%(\n",
    "          format(len(dataset_purchase.ratings), ','),\n",
    "          format(dataset_purchase.num_users, ','),\n",
    "          format(dataset_purchase.num_items, ',')))\n",
    "\n",
    "# train the explicit model based on recommend feedback\n",
    "model = PopularityModel()\n",
    "print('fit the model')\n",
    "model.fit(interactions=dataset_recommend_train)\n",
    "\n",
    "# evaluate the new model\n",
    "print('evaluate the model')\n",
    "mrr, ndcg, ndcg10, ndcg_5, mmap, success_10, success_5 = utils.evaluate(interactions=dataset_recommend_test,\n",
    "                                                                        model=model,\n",
    "                                                                        topk=20)\n",
    "# rmse = rmse_score(model=model, test=dataset_rate_test, batch_size=512)\n",
    "# print('-'*20)\n",
    "# print('RMSE: {:.4f}'.format(rmse))\n",
    "print('MRR: {:.4f}'.format(mrr))\n",
    "print('nDCG: {:.4f}'.format(ndcg))\n",
    "print('nDCG@10: {:.4f}'.format(ndcg10))\n",
    "print('nDCG@5: {:.4f}'.format(ndcg_5))\n",
    "print('MAP: {:.4f}'.format(mmap))\n",
    "print('success@10: {:.4f}'.format(success_10))\n",
    "print('success@5: {:.4f}'.format(success_5))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}