5. Model Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 5: Model Training\n",
    "While a Novelty Score can be calculated using a white-box method detailed in the previous Jupyter Notebook, we would like to predict whether a paper is novel in as close to real-time as possible. \n",
    "\n",
    "This notebook contains code used to train various machine learning models to achieve the task mentioned above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from src import preprocessing\n",
    "\n",
    "from tqdm import tqdm, tqdm_notebook\n",
    "tqdm.pandas()\n",
    "\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load training and validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>tags</th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>publish_date</th>\n",
       "      <th>title+abstract</th>\n",
       "      <th>top_topics</th>\n",
       "      <th>topic_score</th>\n",
       "      <th>author_score</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Learned feature representations and sub-phonem...</td>\n",
       "      <td>[Fred Richardson, Douglas Reynolds, Najim Dehak]</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>[cs.CL, cs.CV, cs.LG, cs.NE, stat.ML]</td>\n",
       "      <td>A Unified Deep Neural Network for Speaker and ...</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015-03-04</td>\n",
       "      <td>A Unified Deep Neural Network for Speaker and ...</td>\n",
       "      <td>(3, 12, 19)</td>\n",
       "      <td>0.864535</td>\n",
       "      <td>0.301551</td>\n",
       "      <td>0.583043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>We propose a simple neural network model to de...</td>\n",
       "      <td>[Muhammad Ghifary, W. Kleijn, Mengjie Zhang]</td>\n",
       "      <td>21</td>\n",
       "      <td>9</td>\n",
       "      <td>[cs.CV, cs.AI, cs.LG, cs.NE, stat.ML]</td>\n",
       "      <td>Domain Adaptive Neural Networks for Object Rec...</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-09-21</td>\n",
       "      <td>Domain Adaptive Neural Networks for Object Rec...</td>\n",
       "      <td>(3, 4, 9)</td>\n",
       "      <td>1.021247</td>\n",
       "      <td>0.338184</td>\n",
       "      <td>0.679716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Recent studies have demonstrated the power of ...</td>\n",
       "      <td>[Lionel Pigou, Aäron Oord, Sander Dieleman, Mi...</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>[cs.CV, cs.AI, cs.LG, cs.NE, stat.ML]</td>\n",
       "      <td>Beyond Temporal Pooling: Recurrence and Tempor...</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015-05-06</td>\n",
       "      <td>Beyond Temporal Pooling: Recurrence and Tempor...</td>\n",
       "      <td>(3, 9, 12)</td>\n",
       "      <td>0.556297</td>\n",
       "      <td>0.245878</td>\n",
       "      <td>0.401087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>In this paper, we address the task of Optical ...</td>\n",
       "      <td>[Rakesh Achanta, Trevor Hastie]</td>\n",
       "      <td>20</td>\n",
       "      <td>9</td>\n",
       "      <td>[stat.ML, cs.AI, cs.CV, cs.LG, cs.NE]</td>\n",
       "      <td>Telugu OCR Framework using Deep Learning</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015-09-20</td>\n",
       "      <td>Telugu OCR Framework using Deep Learning.  In ...</td>\n",
       "      <td>(1, 3, 19)</td>\n",
       "      <td>0.760365</td>\n",
       "      <td>0.739321</td>\n",
       "      <td>0.749843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Recent progress in using recurrent neural netw...</td>\n",
       "      <td>[Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas...</td>\n",
       "      <td>27</td>\n",
       "      <td>2</td>\n",
       "      <td>[stat.ML, cs.AI, cs.CL, cs.CV, cs.LG]</td>\n",
       "      <td>Describing Videos by Exploiting Temporal Struc...</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015-02-27</td>\n",
       "      <td>Describing Videos by Exploiting Temporal Struc...</td>\n",
       "      <td>(3, 9, 19)</td>\n",
       "      <td>0.477121</td>\n",
       "      <td>0.449995</td>\n",
       "      <td>0.463558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            abstract  \\\n",
       "0  Learned feature representations and sub-phonem...   \n",
       "1  We propose a simple neural network model to de...   \n",
       "2  Recent studies have demonstrated the power of ...   \n",
       "3  In this paper, we address the task of Optical ...   \n",
       "4  Recent progress in using recurrent neural netw...   \n",
       "\n",
       "                                             authors  day  month  \\\n",
       "0   [Fred Richardson, Douglas Reynolds, Najim Dehak]    3      4   \n",
       "1       [Muhammad Ghifary, W. Kleijn, Mengjie Zhang]   21      9   \n",
       "2  [Lionel Pigou, Aäron Oord, Sander Dieleman, Mi...    5      6   \n",
       "3                    [Rakesh Achanta, Trevor Hastie]   20      9   \n",
       "4  [Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas...   27      2   \n",
       "\n",
       "                                    tags  \\\n",
       "0  [cs.CL, cs.CV, cs.LG, cs.NE, stat.ML]   \n",
       "1  [cs.CV, cs.AI, cs.LG, cs.NE, stat.ML]   \n",
       "2  [cs.CV, cs.AI, cs.LG, cs.NE, stat.ML]   \n",
       "3  [stat.ML, cs.AI, cs.CV, cs.LG, cs.NE]   \n",
       "4  [stat.ML, cs.AI, cs.CL, cs.CV, cs.LG]   \n",
       "\n",
       "                                               title  year publish_date  \\\n",
       "0  A Unified Deep Neural Network for Speaker and ...  2015   2015-03-04   \n",
       "1  Domain Adaptive Neural Networks for Object Rec...  2014   2014-09-21   \n",
       "2  Beyond Temporal Pooling: Recurrence and Tempor...  2015   2015-05-06   \n",
       "3           Telugu OCR Framework using Deep Learning  2015   2015-09-20   \n",
       "4  Describing Videos by Exploiting Temporal Struc...  2015   2015-02-27   \n",
       "\n",
       "                                      title+abstract   top_topics  \\\n",
       "0  A Unified Deep Neural Network for Speaker and ...  (3, 12, 19)   \n",
       "1  Domain Adaptive Neural Networks for Object Rec...    (3, 4, 9)   \n",
       "2  Beyond Temporal Pooling: Recurrence and Tempor...   (3, 9, 12)   \n",
       "3  Telugu OCR Framework using Deep Learning.  In ...   (1, 3, 19)   \n",
       "4  Describing Videos by Exploiting Temporal Struc...   (3, 9, 19)   \n",
       "\n",
       "   topic_score  author_score     score  \n",
       "0     0.864535      0.301551  0.583043  \n",
       "1     1.021247      0.338184  0.679716  \n",
       "2     0.556297      0.245878  0.401087  \n",
       "3     0.760365      0.739321  0.749843  \n",
       "4     0.477121      0.449995  0.463558  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>tags</th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>publish_date</th>\n",
       "      <th>title+abstract</th>\n",
       "      <th>top_topics</th>\n",
       "      <th>author_score</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Real-time, accurate, and robust pupil detectio...</td>\n",
       "      <td>[Thiago Santini, Wolfgang Fuhl, Enkelejda Kasn...</td>\n",
       "      <td>24</td>\n",
       "      <td>12</td>\n",
       "      <td>[cs.CV, cs.HC]</td>\n",
       "      <td>PuRe: Robust pupil detection for real-time per...</td>\n",
       "      <td>2017</td>\n",
       "      <td>2017-12-24</td>\n",
       "      <td>PuRe: Robust pupil detection for real-time per...</td>\n",
       "      <td>(0, 9, 16)</td>\n",
       "      <td>0.289215</td>\n",
       "      <td>0.252405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The Stixel World is a medium-level, compact re...</td>\n",
       "      <td>[Daniel Hernandez-Juarez, Antonio Espinosa, Da...</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>[cs.CV]</td>\n",
       "      <td>GPU-accelerated real-time stixel computation</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-10-13</td>\n",
       "      <td>GPU-accelerated real-time stixel computation. ...</td>\n",
       "      <td>(1, 9, 16)</td>\n",
       "      <td>0.390351</td>\n",
       "      <td>0.736584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Endowing a chatbot with personality or an iden...</td>\n",
       "      <td>[Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfa...</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>[cs.CL]</td>\n",
       "      <td>Assigning personality/identity to a chatting m...</td>\n",
       "      <td>2017</td>\n",
       "      <td>2017-09-06</td>\n",
       "      <td>Assigning personality/identity to a chatting m...</td>\n",
       "      <td>(4, 6, 15)</td>\n",
       "      <td>0.868820</td>\n",
       "      <td>0.334940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>We present a matrix factorization algorithm th...</td>\n",
       "      <td>[Arthur Mensch, Julien Mairal, Gaël Varoquaux,...</td>\n",
       "      <td>30</td>\n",
       "      <td>11</td>\n",
       "      <td>[math.OC, cs.LG, stat.ML]</td>\n",
       "      <td>Subsampled online matrix factorization with co...</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-11-30</td>\n",
       "      <td>Subsampled online matrix factorization with co...</td>\n",
       "      <td>(0, 10, 11)</td>\n",
       "      <td>0.705347</td>\n",
       "      <td>0.695043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>We present an approximation algorithm that tak...</td>\n",
       "      <td>[Matthew Streeter]</td>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>[cs.LG, cs.AI, cs.NE]</td>\n",
       "      <td>Approximation Algorithms for Cascading Predict...</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018-02-21</td>\n",
       "      <td>Approximation Algorithms for Cascading Predict...</td>\n",
       "      <td>(1, 4, 16)</td>\n",
       "      <td>0.517207</td>\n",
       "      <td>0.459156</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            abstract  \\\n",
       "0  Real-time, accurate, and robust pupil detectio...   \n",
       "2  The Stixel World is a medium-level, compact re...   \n",
       "3  Endowing a chatbot with personality or an iden...   \n",
       "4  We present a matrix factorization algorithm th...   \n",
       "5  We present an approximation algorithm that tak...   \n",
       "\n",
       "                                             authors  day  month  \\\n",
       "0  [Thiago Santini, Wolfgang Fuhl, Enkelejda Kasn...   24     12   \n",
       "2  [Daniel Hernandez-Juarez, Antonio Espinosa, Da...   13     10   \n",
       "3  [Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfa...    9      6   \n",
       "4  [Arthur Mensch, Julien Mairal, Gaël Varoquaux,...   30     11   \n",
       "5                                 [Matthew Streeter]   21      2   \n",
       "\n",
       "                        tags  \\\n",
       "0             [cs.CV, cs.HC]   \n",
       "2                    [cs.CV]   \n",
       "3                    [cs.CL]   \n",
       "4  [math.OC, cs.LG, stat.ML]   \n",
       "5      [cs.LG, cs.AI, cs.NE]   \n",
       "\n",
       "                                               title  year publish_date  \\\n",
       "0  PuRe: Robust pupil detection for real-time per...  2017   2017-12-24   \n",
       "2       GPU-accelerated real-time stixel computation  2016   2016-10-13   \n",
       "3  Assigning personality/identity to a chatting m...  2017   2017-09-06   \n",
       "4  Subsampled online matrix factorization with co...  2016   2016-11-30   \n",
       "5  Approximation Algorithms for Cascading Predict...  2018   2018-02-21   \n",
       "\n",
       "                                      title+abstract   top_topics  \\\n",
       "0  PuRe: Robust pupil detection for real-time per...   (0, 9, 16)   \n",
       "2  GPU-accelerated real-time stixel computation. ...   (1, 9, 16)   \n",
       "3  Assigning personality/identity to a chatting m...   (4, 6, 15)   \n",
       "4  Subsampled online matrix factorization with co...  (0, 10, 11)   \n",
       "5  Approximation Algorithms for Cascading Predict...   (1, 4, 16)   \n",
       "\n",
       "   author_score     score  \n",
       "0      0.289215  0.252405  \n",
       "2      0.390351  0.736584  \n",
       "3      0.868820  0.334940  \n",
       "4      0.705347  0.695043  \n",
       "5      0.517207  0.459156  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train = pd.read_pickle('data/dataframes/main_df_scored.pkl')\n",
    "valid = (\n",
    "    pd\n",
    "    .read_pickle('data/dataframes/valid_scored.pkl')\n",
    "    .pipe(lambda d: d[d.notna().all(axis=1)])\n",
    ")\n",
    "display(train.head(), valid.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Model Training\n",
    "I have chosen to train six common machine learning models, with linear regression being the benchmark algorithm. Given that our goal is to predict the novelty of a paper in as close to real-time as possible, this means that the forecast horizon is a hyperparameter. \n",
    "\n",
    "For a smaller forecast horizon, we are training our model to predict papers that were published relatively recently, and vice-versa.\n",
    "\n",
    "### Predictors\n",
    "I have chosen to use a three features to predict the Novelty Score (label). They are:\n",
    "1. Author Score — We are able to reuse this metric since the information is readily available.\n",
    "2. Topic Combinations — This categorical feature would be useful since documents were grouped by Topic Combinations to find the Topic Score.\n",
    "3. Modified Topic Score — The papers included in calculating this modified Topic Score are a subset of all the papers in a topic combination, in that papers that are exceed the forecast horizon are ignored. For example, for a paper, $p$, that was published at time $t$ in a topic combination, the modified Topic Score for $p$ includes all papers that preceeded it, and as well as papers that came after $p$ so long as the subsequent papers publish date is within the forecast horizon (i.e. $\\leq t$ + forecast horizon)\n",
    "\n",
    "### Metric\n",
    "Since this is a regression problem, the mean squared error (MSE) metric was used to evaluate the models' performances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [LinearRegression(), \n",
    "          Ridge(alpha=1), \n",
    "          SVR(kernel='rbf'), \n",
    "          KNeighborsRegressor(n_neighbors=5, weights='distance'), \n",
    "          DecisionTreeRegressor(), \n",
    "          MLPRegressor(hidden_layer_sizes=(50, 50))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f03b19f1ce34d26bce8fcc66df248bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Tuning hyperparameter', max=4, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training models', max=6, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/svm/base.py:196: FutureWarning:\n",
      "\n",
      "The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
      "\n",
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:562: ConvergenceWarning:\n",
      "\n",
      "Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training models', max=6, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/svm/base.py:196: FutureWarning:\n",
      "\n",
      "The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
      "\n",
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:562: ConvergenceWarning:\n",
      "\n",
      "Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training models', max=6, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/svm/base.py:196: FutureWarning:\n",
      "\n",
      "The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
      "\n",
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:562: ConvergenceWarning:\n",
      "\n",
      "Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training models', max=6, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/svm/base.py:196: FutureWarning:\n",
      "\n",
      "The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
      "\n",
      "/anaconda3/envs/gp/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:562: ConvergenceWarning:\n",
      "\n",
      "Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def standardize(x):\n",
    "    return StandardScaler().fit_transform(x)\n",
    "\n",
    "\n",
    "months = (1, 3, 6, 12)\n",
    "train_summary = pd.DataFrame(columns=[type(model).__name__ for model in models], index=months)\n",
    "valid_summary = pd.DataFrame(columns=[type(model).__name__ for model in models], index=months)\n",
    "\n",
    "for n_month in tqdm_notebook(months, desc='Tuning hyperparameter'):\n",
    "    train = preprocessing.featurize_publish_date(train, n_month)\n",
    "    valid = preprocessing.featurize_publish_date(valid, n_month, prior=train[['top_topics', 'publish_date']])\n",
    "    \n",
    "    # Preparing and standarizing training data\n",
    "    numeric_features = standardize(train[['author_score', 'lookahead']])\n",
    "    categorical_features = pd.get_dummies(train['top_topics'].append(valid['top_topics']))[:len(train)].values\n",
    "    features = np.concatenate([numeric_features, categorical_features], axis=1)\n",
    "    labels = standardize(train[['score']])\n",
    "    \n",
    "    # Preparing and standarizing validation data\n",
    "    v_numeric_features = standardize(valid[['author_score', 'lookahead']])\n",
    "    v_categorical_features = pd.get_dummies(train['top_topics'].append(valid['top_topics']))[-len(valid):].values\n",
    "    v_features = np.concatenate([v_numeric_features, v_categorical_features], axis=1)\n",
    "    v_labels = standardize(valid[['score']])\n",
    "    \n",
    "    for model in tqdm_notebook(models, desc='Training models', leave=False):\n",
    "        model.fit(features, labels if type(model).__name__ not in ['MLPRegressor', 'SVR'] else labels.ravel())\n",
    "        y_train_pred = model.predict(features)\n",
    "        y_valid_pred = model.predict(v_features)\n",
    "        \n",
    "        train_summary.at[n_month, type(model).__name__] = mean_squared_error(y_true=labels, y_pred=y_train_pred)\n",
    "        valid_summary.at[n_month, type(model).__name__] = mean_squared_error(y_true=v_labels, y_pred=y_valid_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LinearRegression</th>\n",
       "      <th>Ridge</th>\n",
       "      <th>SVR</th>\n",
       "      <th>KNeighborsRegressor</th>\n",
       "      <th>DecisionTreeRegressor</th>\n",
       "      <th>MLPRegressor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.495284</td>\n",
       "      <td>0.496746</td>\n",
       "      <td>0.629875</td>\n",
       "      <td>0</td>\n",
       "      <td>1.26677e-09</td>\n",
       "      <td>0.105235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.484228</td>\n",
       "      <td>0.485536</td>\n",
       "      <td>0.608289</td>\n",
       "      <td>0</td>\n",
       "      <td>1.20592e-09</td>\n",
       "      <td>0.0956046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.467248</td>\n",
       "      <td>0.468383</td>\n",
       "      <td>0.580721</td>\n",
       "      <td>0</td>\n",
       "      <td>1.29126e-09</td>\n",
       "      <td>0.10631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.431757</td>\n",
       "      <td>0.432676</td>\n",
       "      <td>0.527678</td>\n",
       "      <td>0</td>\n",
       "      <td>1.2935e-09</td>\n",
       "      <td>0.0865826</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LinearRegression     Ridge       SVR KNeighborsRegressor  \\\n",
       "1          0.495284  0.496746  0.629875                   0   \n",
       "3          0.484228  0.485536  0.608289                   0   \n",
       "6          0.467248  0.468383  0.580721                   0   \n",
       "12         0.431757  0.432676  0.527678                   0   \n",
       "\n",
       "   DecisionTreeRegressor MLPRegressor  \n",
       "1            1.26677e-09     0.105235  \n",
       "3            1.20592e-09    0.0956046  \n",
       "6            1.29126e-09      0.10631  \n",
       "12            1.2935e-09    0.0865826  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LinearRegression</th>\n",
       "      <th>Ridge</th>\n",
       "      <th>SVR</th>\n",
       "      <th>KNeighborsRegressor</th>\n",
       "      <th>DecisionTreeRegressor</th>\n",
       "      <th>MLPRegressor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.74628e+20</td>\n",
       "      <td>1.52646</td>\n",
       "      <td>1.57104</td>\n",
       "      <td>1.59534</td>\n",
       "      <td>1.70524</td>\n",
       "      <td>1.71758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.29761e+21</td>\n",
       "      <td>1.53858</td>\n",
       "      <td>1.57487</td>\n",
       "      <td>1.59217</td>\n",
       "      <td>1.72501</td>\n",
       "      <td>1.63245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6.38724e+18</td>\n",
       "      <td>1.55564</td>\n",
       "      <td>1.57904</td>\n",
       "      <td>1.62778</td>\n",
       "      <td>1.72583</td>\n",
       "      <td>1.71562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>5.57115e+20</td>\n",
       "      <td>1.59641</td>\n",
       "      <td>1.59404</td>\n",
       "      <td>1.60942</td>\n",
       "      <td>1.72097</td>\n",
       "      <td>1.70309</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LinearRegression    Ridge      SVR KNeighborsRegressor  \\\n",
       "1       2.74628e+20  1.52646  1.57104             1.59534   \n",
       "3       2.29761e+21  1.53858  1.57487             1.59217   \n",
       "6       6.38724e+18  1.55564  1.57904             1.62778   \n",
       "12      5.57115e+20  1.59641  1.59404             1.60942   \n",
       "\n",
       "   DecisionTreeRegressor MLPRegressor  \n",
       "1                1.70524      1.71758  \n",
       "3                1.72501      1.63245  \n",
       "6                1.72583      1.71562  \n",
       "12               1.72097      1.70309  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_summary.to_csv('data/training_metrics/train.csv')\n",
    "valid_summary.to_csv('data/training_metrics/valid.csv')\n",
    "display(train_summary, valid_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results above, we can see that all models are overfitting to the data, as the validation MSEs are higher than the training MSEs. The reason for the poor performance is that the Topic Score label is dependent on the number of papers that came before it. Journal articles in the validation and test sets are more recent, and as such would have lower Topic Scores. This fact could not be captured by the models above. \n",
    "\n",
    "The next logical step to take in improving this modelling work is to reformulate the training procedure as a time series problem and use a Long Short Term Memory (LSTM) network to capture the sequential dependencies in the data. Unfortunately, I was not able to develop and train the model in TensorFlow given the amount of time I have. This task will be relinquished to being part of the Future Work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Future Work\n",
    "- Utilize textual information from the articles themselves to create embeddings for each paper\n",
    "- Use an LSTM-like network to model the sequential dependencies within the data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python3 (gp)",
   "language": "python",
   "name": "gp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}