4_sentiment analysis_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Emoji Sentiment Analysis with Tweets_Chinese\n",
    "        \n",
    "## step4-Sentiment analysis\n",
    "\n",
    "#### 4.1 Constructing train and test dataset\n",
    "- data cleaning\n",
    "- preparing new columns for 4.3 and 4.4\n",
    "- split the dataset\n",
    "\n",
    "#### 4.2 Classification without emojis\n",
    "- LSTM\n",
    "- DNN\n",
    "\n",
    "#### 4.3 Classification with replacing the emojis with their descriptive names\n",
    "- LSTM\n",
    "- DNN\n",
    "\n",
    "#### 4.4 Classification with replacing the emojis with the most similar 5 text tokens in Word2Vec model\n",
    "- LSTM\n",
    "- DNN\n",
    "\n",
    "#### 4.5 Classification with emojis word embedding vectors in Word2Vec model\n",
    "- LSTM\n",
    "- DNN\n",
    "\n",
    "\n",
    "\n",
    "![](https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcTigQWzoYCNiDyrz1BN4WTf2X2k9OZ_yvW-FsmcIMsdS9fppNmh)\n",
    "***Note, the dataset of labelled tweets with emojis is cloned from [DanaOshri, github](https://github.com/DanaOshri/Twitter-Sentiment-Analysis-Emoji-Embedding-and-LSTM)***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 read the dataset with sentiment labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install emoji"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>tweets</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>lmfaoo 😭 😭 😭 😭 😭</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>i hate this feeling 😢</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>ca n't believe i just went out in this cold to...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>i need a new trap house so if you really fuck ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>so very sorry for your loss 💔</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13195</th>\n",
       "      <td>13195</td>\n",
       "      <td>i love waking up skinny ahaha wish it lasted a...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13196</th>\n",
       "      <td>13196</td>\n",
       "      <td>magnificent pair of tits 😍 my cock is hard 🍆 😀</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13197</th>\n",
       "      <td>13197</td>\n",
       "      <td>soon mamsh 😘 god will give you the best among ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13198</th>\n",
       "      <td>13198</td>\n",
       "      <td>i trust u 😎</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13199</th>\n",
       "      <td>13199</td>\n",
       "      <td>aww thanks 😁</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13200 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                             tweets  labels\n",
       "0               0                                   lmfaoo 😭 😭 😭 😭 😭       0\n",
       "1               1                              i hate this feeling 😢       0\n",
       "2               2  ca n't believe i just went out in this cold to...       0\n",
       "3               3  i need a new trap house so if you really fuck ...       0\n",
       "4               4                      so very sorry for your loss 💔       0\n",
       "...           ...                                                ...     ...\n",
       "13195       13195  i love waking up skinny ahaha wish it lasted a...       1\n",
       "13196       13196     magnificent pair of tits 😍 my cock is hard 🍆 😀       1\n",
       "13197       13197  soon mamsh 😘 god will give you the best among ...       1\n",
       "13198       13198                                        i trust u 😎       1\n",
       "13199       13199                                       aww thanks 😁       1\n",
       "\n",
       "[13200 rows x 3 columns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "os.getcwd()\n",
    "df = pd.read_csv('./sentiment/raw.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 388,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweets</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>😭   😭   😭   😭   😭</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>i hate this feeling  😢</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ca believe i just went out in this cold to buy...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>i need a new trap house so if you really wit m...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>so very sorry for your loss  💔</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13175</th>\n",
       "      <td>pretty sure catch these  😂</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13176</th>\n",
       "      <td>sum kiss sound nice oh  😔</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13177</th>\n",
       "      <td>you are looking so beautiful  😘</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13178</th>\n",
       "      <td>enjoying a night with  😈</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13179</th>\n",
       "      <td>hustle hard nobody care about your inch  😁   😁...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13180 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  tweets  labels\n",
       "0                                     😭   😭   😭   😭   😭        0\n",
       "1                                i hate this feeling  😢        0\n",
       "2      ca believe i just went out in this cold to buy...       0\n",
       "3      i need a new trap house so if you really wit m...       0\n",
       "4                        so very sorry for your loss  💔        0\n",
       "...                                                  ...     ...\n",
       "13175                        pretty sure catch these  😂        1\n",
       "13176                         sum kiss sound nice oh  😔        1\n",
       "13177                   you are looking so beautiful  😘        1\n",
       "13178                          enjoying a night with  😈        1\n",
       "13179  hustle hard nobody care about your inch  😁   😁...       1\n",
       "\n",
       "[13180 rows x 2 columns]"
      ]
     },
     "execution_count": 388,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[:-20,1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.query('labels == 1')\n",
    "#0:6600\n",
    "#1:6600"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of original Tweet:\n",
      "so very sorry for your loss 💔 tweets looks cleaned\n",
      "---------------\n",
      "Lemmatizating ...\n",
      "so very sorry for your loss 💔 tweet look cleaned\n",
      "CPU times: user 2.65 s, sys: 78.6 ms, total: 2.73 s\n",
      "Wall time: 2.75 s\n"
     ]
    }
   ],
   "source": [
    "## data cleaning\n",
    "#i. lemmatization\n",
    "\n",
    "from textblob import TextBlob\n",
    "\n",
    "def lemm(text):\n",
    "    textTB = TextBlob(text)\n",
    "    words = textTB.words\n",
    "    words_lemmatized = words.lemmatize()\n",
    "    return ' '.join(words_lemmatized)\n",
    "\n",
    "example = df['tweets'][4]+' tweets'+' looks cleaned'\n",
    "print('Example of original Tweet:')\n",
    "print(example)\n",
    "print('---------------')\n",
    "print('Lemmatizating ...')\n",
    "example = lemm(example)\n",
    "print(example)\n",
    "%time df['tweets']=df['tweets'].apply(lambda x : lemm(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package words to /Users/leahtan/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of original Tweet:\n",
      "so very sorry for your loss 💔我 们 لغة عربية 123 😅😅\n",
      "---------------\n",
      "Extracting English words and emojis ...\n",
      "so very sorry for your loss\n"
     ]
    }
   ],
   "source": [
    "#ii. keep only english characters and emojis\n",
    "\n",
    "import emoji\n",
    "import nltk \n",
    "nltk.download('words')\n",
    "words = set(nltk.corpus.words.words())\n",
    "\n",
    "def keepengemoji(text):\n",
    "    ls = []\n",
    "    for w in text.split(' '):\n",
    "        if w in words:\n",
    "            ls.append(w)\n",
    "        elif w in emoji.UNICODE_EMOJI['en']:\n",
    "            w = ' '+w+' '\n",
    "            ls.append(w)\n",
    "        else:\n",
    "            continue\n",
    "    return ' '.join(ls)\n",
    "\n",
    "example = df['tweets'][4]+'我 们 لغة عربية 123 😅😅'\n",
    "print('Example of original Tweet:')\n",
    "print(example)\n",
    "print('---------------')\n",
    "print('Extracting English words and emojis ...')\n",
    "example = keepengemoji(example)\n",
    "print(example)\n",
    "df['tweets']=df['tweets'].apply(lambda x : keepengemoji(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of original Tweet:\n",
      " 😭   😭   😭   😭   😭 \n",
      "---------------\n",
      "Extracting pure text ...\n",
      "              \n"
     ]
    }
   ],
   "source": [
    "import emoji\n",
    "\n",
    "def remove_emojis(text):\n",
    "  return ''.join(c for c in text if c not in emoji.UNICODE_EMOJI['en'])\n",
    "\n",
    "example = df['tweets'][0]\n",
    "print('Example of original Tweet:')\n",
    "print(example)\n",
    "print('---------------')\n",
    "print('Extracting pure text ...')\n",
    "example = remove_emojis(example)\n",
    "print(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " 'i hate this feeling',\n",
       " 'ca believe i just went out in this cold to buy food what in the poor ca i just be rich and have people working for me',\n",
       " 'i need a new trap house so if you really wit me baby put your name on this lease',\n",
       " 'so very sorry for your loss',\n",
       " 'random',\n",
       " 'so you wan na be bad at sex and get away with it',\n",
       " 'brake failure',\n",
       " 'he is so annoying',\n",
       " 'back this because damn he fire']"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "puretext = [remove_emojis(t).strip() for t in df['tweets']]\n",
    "puretext[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text']=puretext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "# new column, replace emoji with its name, for 4.3\n",
    "import demoji\n",
    "wtemo = [demoji.replace_with_desc(df.tweets[i]).replace(':','') for i in range(len(df))]\n",
    "df['wtemo'] = wtemo\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>©</td>\n",
       "      <td>mail prince ladder cutie momo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>‼</td>\n",
       "      <td>bossy treatment dreamy urgent announcement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>⁉</td>\n",
       "      <td>predraft ser miserable spree ake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>↗</td>\n",
       "      <td>status silver elite diamond rookie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>↘</td>\n",
       "      <td>status silver elite noodle diamond</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329</th>\n",
       "      <td>🩸</td>\n",
       "      <td>rifle assault bin catwalk platinum</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>330</th>\n",
       "      <td>🩹</td>\n",
       "      <td>scratch tae knee fell attached</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>331</th>\n",
       "      <td>🫠</td>\n",
       "      <td>shaking titty kissing heel hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <td>🫣</td>\n",
       "      <td>titty prefer typo struggling confirm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>333</th>\n",
       "      <td>🫶</td>\n",
       "      <td>interact peso predraft maam happiness</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>334 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Unnamed: 0                                           0\n",
       "0            ©               mail prince ladder cutie momo\n",
       "1            ‼  bossy treatment dreamy urgent announcement\n",
       "2            ⁉            predraft ser miserable spree ake\n",
       "3            ↗          status silver elite diamond rookie\n",
       "4            ↘          status silver elite noodle diamond\n",
       "..         ...                                         ...\n",
       "329          🩸          rifle assault bin catwalk platinum\n",
       "330          🩹              scratch tae knee fell attached\n",
       "331          🫠              shaking titty kissing heel hop\n",
       "332          🫣        titty prefer typo struggling confirm\n",
       "333          🫶       interact peso predraft maam happiness\n",
       "\n",
       "[334 rows x 2 columns]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new column, replace emoji with its most similar text tokens, for 4.4\n",
    "emosimi_df = pd.read_csv(\"en_most_similar_names.csv\")  # joined str of 5 most similar text tokens of emojis\n",
    "emosimi_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "import emoji\n",
    "ls = []\n",
    "for i in df['tweets']:\n",
    "    wls = []\n",
    "    words = i.split()\n",
    "    for word in words:\n",
    "        if word not in emoji.UNICODE_EMOJI['en']:\n",
    "            wls.append(word)\n",
    "        elif word in emosimi_df['Unnamed: 0'].to_list():\n",
    "            wls.append(emosimi_df.loc[emosimi_df['Unnamed: 0'] == word, '0'].iloc[0])\n",
    "        else:\n",
    "            wls.append(demoji.replace_with_desc(word).replace(':',''))\n",
    "    simi = ' '.join(wls)\n",
    "    ls.append(simi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['simiemo'] = ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>tweets</th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "      <th>wtemo</th>\n",
       "      <th>simiemo</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>😭   😭   😭   😭   😭</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td>loudly crying face   loudly crying face   lou...</td>\n",
       "      <td>dry cringing outfit instantly had dry cringing...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>i hate this feeling  😢</td>\n",
       "      <td>0</td>\n",
       "      <td>i hate this feeling</td>\n",
       "      <td>i hate this feeling  crying face</td>\n",
       "      <td>i hate this feeling condolence fletcher rip de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>ca believe i just went out in this cold to buy...</td>\n",
       "      <td>0</td>\n",
       "      <td>ca believe i just went out in this cold to buy...</td>\n",
       "      <td>ca believe i just went out in this cold to buy...</td>\n",
       "      <td>ca believe i just went out in this cold to buy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>i need a new trap house so if you really wit m...</td>\n",
       "      <td>0</td>\n",
       "      <td>i need a new trap house so if you really wit m...</td>\n",
       "      <td>i need a new trap house so if you really wit m...</td>\n",
       "      <td>i need a new trap house so if you really wit m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>so very sorry for your loss  💔</td>\n",
       "      <td>0</td>\n",
       "      <td>so very sorry for your loss</td>\n",
       "      <td>so very sorry for your loss  broken heart</td>\n",
       "      <td>so very sorry for your loss condolence heartbr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                             tweets  labels  \\\n",
       "0           0                                 😭   😭   😭   😭   😭        0   \n",
       "1           1                            i hate this feeling  😢        0   \n",
       "2           2  ca believe i just went out in this cold to buy...       0   \n",
       "3           3  i need a new trap house so if you really wit m...       0   \n",
       "4           4                    so very sorry for your loss  💔        0   \n",
       "\n",
       "                                                text  \\\n",
       "0                                                      \n",
       "1                                i hate this feeling   \n",
       "2  ca believe i just went out in this cold to buy...   \n",
       "3  i need a new trap house so if you really wit m...   \n",
       "4                        so very sorry for your loss   \n",
       "\n",
       "                                               wtemo  \\\n",
       "0   loudly crying face   loudly crying face   lou...   \n",
       "1                  i hate this feeling  crying face    \n",
       "2  ca believe i just went out in this cold to buy...   \n",
       "3  i need a new trap house so if you really wit m...   \n",
       "4         so very sorry for your loss  broken heart    \n",
       "\n",
       "                                             simiemo  \n",
       "0  dry cringing outfit instantly had dry cringing...  \n",
       "1  i hate this feeling condolence fletcher rip de...  \n",
       "2  ca believe i just went out in this cold to buy...  \n",
       "3  i need a new trap house so if you really wit m...  \n",
       "4  so very sorry for your loss condolence heartbr...  "
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i hate this feeling condolence fletcher rip devastating heartbreaking'"
      ]
     },
     "execution_count": 378,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['simiemo'].iloc[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10560 2640\n"
     ]
    }
   ],
   "source": [
    "# split dataset\n",
    "from sklearn.model_selection import train_test_split\n",
    "train, test = train_test_split(df, test_size=0.2)\n",
    "\n",
    "y_train = train['labels']\n",
    "y_test = test['labels']\n",
    "\n",
    "print(len(y_train),len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Classification without emojis\n",
    "using text as x\n",
    "- LSTM: 0.57\n",
    "- DNN: 0.56"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10560 2640\n"
     ]
    }
   ],
   "source": [
    "# classification with pure text\n",
    "x_train42 = train['text'].to_list()\n",
    "x_test42 = test['text'].to_list()\n",
    "\n",
    "print(len(x_train42),len(x_test42))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "x_train shape: (10560, 40)\n",
      "x_train shape: (2640, 40)\n",
      "[[3, 1323, 18, 23, 629, 3, 217, 1841]]\n",
      "[[277, 796, 1039, 26, 280, 6, 308]]\n"
     ]
    }
   ],
   "source": [
    "# encode the words\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "print('Loading data...')\n",
    "def get_sequences(tokenizer, tweets):\n",
    "  sequences = tokenizer.texts_to_sequences(tweets)\n",
    "  padded = pad_sequences(sequences, truncating ='post', maxlen = maxlen)\n",
    "  return padded\n",
    "\n",
    "\n",
    "# tokenizer\n",
    "tokenizer = Tokenizer(num_words=10000, oov_token='<UNK>')\n",
    "tokenizer.fit_on_texts(x_train42)\n",
    "print(tokenizer.texts_to_sequences([x_train42[0]]))\n",
    "x_train42_seq = get_sequences(tokenizer, x_train42)\n",
    "\n",
    "tokenizer.fit_on_texts(x_test42)\n",
    "print(tokenizer.texts_to_sequences([x_test42[0]]))\n",
    "x_test42_seq = get_sequences(tokenizer, x_test42)\n",
    "\n",
    "print('x_train shape:', x_train42_seq.shape)\n",
    "print('x_train shape:', x_test42_seq.shape)\n",
    "\n",
    "get_sequences(tokenizer, x_train42[:2])\n",
    "\n",
    "# set parameters\n",
    "max_features = 60000 # cut texts after this number of words (among top max_features most common words)\n",
    "embedding_dims = 300\n",
    "maxlen = 40 #based on 2, most tweets are shorter than 40 words\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 354,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build LSTM model...\n",
      "Epoch 1/10\n",
      "330/330 [==============================] - 21s 46ms/step - loss: 0.5045 - accuracy: 0.7576 - val_loss: 0.8797 - val_accuracy: 0.5811\n",
      "Epoch 2/10\n",
      "330/330 [==============================] - 14s 42ms/step - loss: 0.2285 - accuracy: 0.9093 - val_loss: 1.1157 - val_accuracy: 0.5640\n",
      "Epoch 3/10\n",
      "330/330 [==============================] - 14s 43ms/step - loss: 0.1640 - accuracy: 0.9360 - val_loss: 1.2135 - val_accuracy: 0.5746\n",
      "83/83 [==============================] - 1s 8ms/step - loss: 1.2135 - accuracy: 0.5746\n",
      "Test accuracy: 0.5746212005615234, Test loss: 1.2135310173034668\n"
     ]
    }
   ],
   "source": [
    "# LSTM model\n",
    "print('Build LSTM model...')\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Embedding(10000, 300, input_length=maxlen),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20, return_sequences=True)),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20)),\n",
    "        tf.keras.layers.Dense(6, activation= 'softmax')\n",
    "])\n",
    "\n",
    "model.compile(\n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    optimizer = 'adam',\n",
    "    metrics = ['accuracy']\n",
    ")\n",
    "\n",
    "\n",
    "# run lstm model\n",
    "# the model, with training set, validation set\n",
    "h = model.fit(\n",
    "    x_train42_seq, y_train,\n",
    "    validation_data=( x_test42_seq, y_test,),\n",
    "    epochs=10,\n",
    "    callbacks=[\n",
    "               tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)\n",
    "    ]\n",
    ")\n",
    "\n",
    "# testing model\n",
    "score, accuracy = model.evaluate(x_test42_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build DNN model...\n",
      "Model: \"sequential_85\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_33 (Embedding)    (None, 40, 300)           18000000  \n",
      "                                                                 \n",
      " flatten_33 (Flatten)        (None, 12000)             0         \n",
      "                                                                 \n",
      " dense_200 (Dense)           (None, 512)               6144512   \n",
      "                                                                 \n",
      " dropout_108 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_201 (Dense)           (None, 512)               262656    \n",
      "                                                                 \n",
      " dropout_109 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_202 (Dense)           (None, 1)                 513       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 24,407,681\n",
      "Trainable params: 24,407,681\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "165/165 [==============================] - 22s 129ms/step - loss: 0.5237 - accuracy: 0.7035 - val_loss: 0.9727 - val_accuracy: 0.5731\n",
      "Epoch 2/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.2324 - accuracy: 0.9015 - val_loss: 1.1070 - val_accuracy: 0.5701\n",
      "Epoch 3/20\n",
      "165/165 [==============================] - 22s 132ms/step - loss: 0.1366 - accuracy: 0.9470 - val_loss: 1.4409 - val_accuracy: 0.5659\n",
      "Epoch 4/20\n",
      "165/165 [==============================] - 21s 128ms/step - loss: 0.0943 - accuracy: 0.9636 - val_loss: 1.6964 - val_accuracy: 0.5527\n",
      "Epoch 5/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0637 - accuracy: 0.9750 - val_loss: 1.9450 - val_accuracy: 0.5583\n",
      "Epoch 6/20\n",
      "165/165 [==============================] - 20s 123ms/step - loss: 0.0547 - accuracy: 0.9783 - val_loss: 2.0217 - val_accuracy: 0.5530\n",
      "Epoch 7/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0463 - accuracy: 0.9809 - val_loss: 2.0598 - val_accuracy: 0.5576\n",
      "Epoch 8/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0430 - accuracy: 0.9814 - val_loss: 2.2697 - val_accuracy: 0.5591\n",
      "Epoch 9/20\n",
      "165/165 [==============================] - 20s 123ms/step - loss: 0.0403 - accuracy: 0.9837 - val_loss: 2.2446 - val_accuracy: 0.5636\n",
      "Epoch 10/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0388 - accuracy: 0.9832 - val_loss: 2.2793 - val_accuracy: 0.5587\n",
      "Epoch 11/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0341 - accuracy: 0.9845 - val_loss: 2.3056 - val_accuracy: 0.5598\n",
      "Epoch 12/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0341 - accuracy: 0.9846 - val_loss: 2.2824 - val_accuracy: 0.5614\n",
      "Epoch 13/20\n",
      "165/165 [==============================] - 21s 124ms/step - loss: 0.0312 - accuracy: 0.9867 - val_loss: 2.5344 - val_accuracy: 0.5602\n",
      "Epoch 14/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0315 - accuracy: 0.9865 - val_loss: 2.4365 - val_accuracy: 0.5527\n",
      "Epoch 15/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0305 - accuracy: 0.9861 - val_loss: 2.5357 - val_accuracy: 0.5576\n",
      "Epoch 16/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0282 - accuracy: 0.9875 - val_loss: 2.5031 - val_accuracy: 0.5595\n",
      "Epoch 17/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0282 - accuracy: 0.9865 - val_loss: 2.4688 - val_accuracy: 0.5538\n",
      "Epoch 18/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0283 - accuracy: 0.9864 - val_loss: 2.7781 - val_accuracy: 0.5553\n",
      "Epoch 19/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0281 - accuracy: 0.9880 - val_loss: 2.6051 - val_accuracy: 0.5530\n",
      "Epoch 20/20\n",
      "165/165 [==============================] - 21s 124ms/step - loss: 0.0276 - accuracy: 0.9870 - val_loss: 2.7110 - val_accuracy: 0.5606\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7ff14aa0ba90>"
      ]
     },
     "execution_count": 366,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try a more complicated DNN since the dimension is high\n",
    "\n",
    "\n",
    "print('Build DNN model...')\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, embedding_dims, input_length=maxlen))\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "# config model\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# training model\n",
    "model.fit(x_train42_seq, y_train, batch_size=64, epochs=20, validation_data=(x_test42_seq, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 356,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "83/83 [==============================] - 0s 4ms/step - loss: 2.3907 - accuracy: 0.5587\n",
      "Test accuracy: 0.5587121248245239, Test loss: 2.3906760215759277\n"
     ]
    }
   ],
   "source": [
    "# testing DNN model\n",
    "score, accuracy = model.evaluate(x_test42_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Classification with replacing the emojis with their descriptive names\n",
    "\n",
    "using the wtemo column as x\n",
    "- LSTM: 0.77\n",
    "- DNN: 0.78"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10560 2640\n"
     ]
    }
   ],
   "source": [
    "# classification with emojis' descriptive names\n",
    "x_train43 = train['wtemo'].to_list()\n",
    "x_test43 = test['wtemo'].to_list()\n",
    "\n",
    "print(len(x_train43),len(x_test43))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[11, 1452, 21, 35, 741, 11, 280, 1973, 2, 3, 16, 7, 15]]\n",
      "[[315, 251, 1170, 3, 351, 5, 2, 14, 388]]\n",
      "Loading data...\n",
      "x_train shape: (10560, 40)\n",
      "x_train shape: (2640, 40)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,   10, 1680,   21,   35,  857,   10,\n",
       "         297, 1523,    2,    3,   16,    7,   15],\n",
       "       [   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    0, 1391,   34,  423,   14,   18,\n",
       "         698,  164,    4,   86,    8,    5,    2]], dtype=int32)"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode the words\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "# tokenizer\n",
    "tokenizer = Tokenizer(num_words=10000, oov_token='<UNK>')\n",
    "tokenizer.fit_on_texts(x_train43)\n",
    "print(tokenizer.texts_to_sequences([x_train43[0]]))\n",
    "x_train43_seq = get_sequences(tokenizer, x_train43)\n",
    "\n",
    "tokenizer.fit_on_texts(x_test43)\n",
    "print(tokenizer.texts_to_sequences([x_test43[0]]))\n",
    "x_test43_seq = get_sequences(tokenizer, x_test43)\n",
    "\n",
    "print('Loading data...')\n",
    "print('x_train shape:', x_train43_seq.shape)\n",
    "print('x_train shape:', x_test43_seq.shape)\n",
    "get_sequences(tokenizer, x_train43[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 350,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build LSTM model...\n",
      "Epoch 1/10\n",
      "330/330 [==============================] - 20s 50ms/step - loss: 0.2286 - accuracy: 0.9324 - val_loss: 0.6027 - val_accuracy: 0.7663\n",
      "Epoch 2/10\n",
      "330/330 [==============================] - 15s 45ms/step - loss: 0.0530 - accuracy: 0.9847 - val_loss: 0.7706 - val_accuracy: 0.7712\n",
      "Epoch 3/10\n",
      "330/330 [==============================] - 15s 45ms/step - loss: 0.0324 - accuracy: 0.9928 - val_loss: 0.8551 - val_accuracy: 0.7754\n",
      "Epoch 4/10\n",
      "330/330 [==============================] - 15s 44ms/step - loss: 0.0232 - accuracy: 0.9944 - val_loss: 0.9751 - val_accuracy: 0.7625\n",
      "Epoch 5/10\n",
      "330/330 [==============================] - 15s 45ms/step - loss: 0.0205 - accuracy: 0.9952 - val_loss: 0.9009 - val_accuracy: 0.7731\n",
      "83/83 [==============================] - 1s 9ms/step - loss: 0.9009 - accuracy: 0.7731\n",
      "Test accuracy: 0.773106038570404, Test loss: 0.9008961319923401\n"
     ]
    }
   ],
   "source": [
    "# LSTM model\n",
    "print('Build LSTM model...')\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Embedding(10000, 300, input_length=maxlen),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20, return_sequences=True)),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20)),\n",
    "        tf.keras.layers.Dense(6, activation= 'softmax')\n",
    "])\n",
    "\n",
    "model.compile(\n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    optimizer = 'adam',\n",
    "    metrics = ['accuracy']\n",
    ")\n",
    "\n",
    "\n",
    "# run lstm model\n",
    "# the model, with training set, validation set\n",
    "h = model.fit(\n",
    "    x_train43_seq, y_train,\n",
    "    validation_data=( x_test43_seq, y_test,),\n",
    "    epochs=10,\n",
    "    callbacks=[\n",
    "               tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)\n",
    "    ]\n",
    ")\n",
    "\n",
    "# testing model\n",
    "score, accuracy = model.evaluate(x_test43_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build DNN model...\n",
      "Model: \"sequential_84\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_32 (Embedding)    (None, 40, 300)           18000000  \n",
      "                                                                 \n",
      " flatten_32 (Flatten)        (None, 12000)             0         \n",
      "                                                                 \n",
      " dense_197 (Dense)           (None, 512)               6144512   \n",
      "                                                                 \n",
      " dropout_106 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_198 (Dense)           (None, 512)               262656    \n",
      "                                                                 \n",
      " dropout_107 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_199 (Dense)           (None, 1)                 513       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 24,407,681\n",
      "Trainable params: 24,407,681\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "165/165 [==============================] - 21s 122ms/step - loss: 0.2557 - accuracy: 0.8857 - val_loss: 0.5422 - val_accuracy: 0.7731\n",
      "Epoch 2/20\n",
      "165/165 [==============================] - 20s 122ms/step - loss: 0.0591 - accuracy: 0.9826 - val_loss: 0.6893 - val_accuracy: 0.7667\n",
      "Epoch 3/20\n",
      "165/165 [==============================] - 20s 120ms/step - loss: 0.0244 - accuracy: 0.9948 - val_loss: 0.8629 - val_accuracy: 0.7587\n",
      "Epoch 4/20\n",
      "165/165 [==============================] - 20s 122ms/step - loss: 0.0132 - accuracy: 0.9970 - val_loss: 0.9814 - val_accuracy: 0.7689\n",
      "Epoch 5/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0079 - accuracy: 0.9986 - val_loss: 1.0057 - val_accuracy: 0.7803\n",
      "Epoch 6/20\n",
      "165/165 [==============================] - 21s 127ms/step - loss: 0.0050 - accuracy: 0.9991 - val_loss: 1.0563 - val_accuracy: 0.7811\n",
      "Epoch 7/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0043 - accuracy: 0.9994 - val_loss: 1.1780 - val_accuracy: 0.7583\n",
      "Epoch 8/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0026 - accuracy: 0.9996 - val_loss: 1.2016 - val_accuracy: 0.7674\n",
      "Epoch 9/20\n",
      "165/165 [==============================] - 21s 127ms/step - loss: 0.0017 - accuracy: 0.9997 - val_loss: 1.2855 - val_accuracy: 0.7655\n",
      "Epoch 10/20\n",
      "165/165 [==============================] - 21s 128ms/step - loss: 0.0016 - accuracy: 0.9997 - val_loss: 1.3260 - val_accuracy: 0.7633\n",
      "Epoch 11/20\n",
      "165/165 [==============================] - 21s 128ms/step - loss: 0.0029 - accuracy: 0.9996 - val_loss: 1.2990 - val_accuracy: 0.7606\n",
      "Epoch 12/20\n",
      "165/165 [==============================] - 21s 127ms/step - loss: 0.0023 - accuracy: 0.9997 - val_loss: 1.2583 - val_accuracy: 0.7735\n",
      "Epoch 13/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0022 - accuracy: 0.9995 - val_loss: 1.3945 - val_accuracy: 0.7644\n",
      "Epoch 14/20\n",
      "165/165 [==============================] - 20s 121ms/step - loss: 0.0024 - accuracy: 0.9994 - val_loss: 1.4291 - val_accuracy: 0.7644\n",
      "Epoch 15/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0041 - accuracy: 0.9991 - val_loss: 1.4389 - val_accuracy: 0.7769\n",
      "Epoch 16/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0023 - accuracy: 0.9993 - val_loss: 1.5487 - val_accuracy: 0.7678\n",
      "Epoch 17/20\n",
      "165/165 [==============================] - 21s 128ms/step - loss: 2.8559e-04 - accuracy: 1.0000 - val_loss: 1.6254 - val_accuracy: 0.7663\n",
      "Epoch 18/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 1.6007e-04 - accuracy: 0.9999 - val_loss: 1.6743 - val_accuracy: 0.7723\n",
      "Epoch 19/20\n",
      "165/165 [==============================] - 22s 130ms/step - loss: 7.0549e-05 - accuracy: 1.0000 - val_loss: 1.7353 - val_accuracy: 0.7655\n",
      "Epoch 20/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 5.9393e-05 - accuracy: 1.0000 - val_loss: 1.7760 - val_accuracy: 0.7633\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7ff1b03e6f70>"
      ]
     },
     "execution_count": 365,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try a more complicated DNN since the dimension is high\n",
    "\n",
    "\n",
    "print('Build DNN model...')\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, embedding_dims, input_length=maxlen))\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "# config model\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# training model\n",
    "model.fit(x_train43_seq, y_train, batch_size=64, epochs=20, validation_data=(x_test43_seq, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 352,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "83/83 [==============================] - 0s 4ms/step - loss: 1.2321 - accuracy: 0.7754\n",
      "Test accuracy: 0.7753787636756897, Test loss: 1.2321490049362183\n"
     ]
    }
   ],
   "source": [
    "# testing DNN model\n",
    "score, accuracy = model.evaluate(x_test43_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Classification with replacing the emojis with the most similar 5 text tokens\n",
    "\n",
    "using the simiemo column as x\n",
    "\n",
    "if the emoji is not shown in our trained Word2Vec model, than replacing it with its descriptive name as 4.3\n",
    "\n",
    "- LSTM: 0.76\n",
    "- DNN: 0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10560 2640\n"
     ]
    }
   ],
   "source": [
    "# classification with emojis' descriptive names\n",
    "x_train44 = train['simiemo'].to_list()\n",
    "x_test44 = test['simiemo'].to_list()\n",
    "\n",
    "print(len(x_train44),len(x_test44))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[8, 1771, 39, 59, 1018, 8, 444, 2251, 13, 12, 16, 15, 14]]\n",
      "[[535, 477, 1494, 50, 539, 25, 47, 74, 77, 46, 11, 576]]\n",
      "Loading data...\n",
      "x_train shape: (10560, 40)\n",
      "x_train shape: (2640, 40)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    8, 1978,   40,   59, 1159,    8,\n",
       "         471, 1825,   14,   12,   16,   15,   13],\n",
       "       [   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "           0,    0,    0,    0, 1700,   58,  608,   11,   18,  974,  237,\n",
       "           2,  119,    5,    7,    4,    6,    3]], dtype=int32)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode the words\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "# tokenizer\n",
    "tokenizer = Tokenizer(num_words=10000, oov_token='<UNK>')\n",
    "tokenizer.fit_on_texts(x_train44)\n",
    "print(tokenizer.texts_to_sequences([x_train44[0]]))\n",
    "x_train44_seq = get_sequences(tokenizer, x_train44)\n",
    "\n",
    "tokenizer.fit_on_texts(x_test44)\n",
    "print(tokenizer.texts_to_sequences([x_test44[0]]))\n",
    "x_test44_seq = get_sequences(tokenizer, x_test44)\n",
    "\n",
    "print('Loading data...')\n",
    "print('x_train shape:', x_train44_seq.shape)\n",
    "print('x_train shape:', x_test44_seq.shape)\n",
    "get_sequences(tokenizer, x_train44[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build LSTM model...\n",
      "Epoch 1/10\n",
      "330/330 [==============================] - 20s 50ms/step - loss: 0.2516 - accuracy: 0.9049 - val_loss: 0.7120 - val_accuracy: 0.7443\n",
      "Epoch 2/10\n",
      "330/330 [==============================] - 15s 46ms/step - loss: 0.0591 - accuracy: 0.9819 - val_loss: 0.8236 - val_accuracy: 0.7508\n",
      "Epoch 3/10\n",
      "330/330 [==============================] - 15s 47ms/step - loss: 0.0328 - accuracy: 0.9920 - val_loss: 0.9789 - val_accuracy: 0.7504\n",
      "Epoch 4/10\n",
      "330/330 [==============================] - 15s 46ms/step - loss: 0.0243 - accuracy: 0.9944 - val_loss: 1.0831 - val_accuracy: 0.7557\n",
      "Epoch 5/10\n",
      "330/330 [==============================] - 15s 46ms/step - loss: 0.0155 - accuracy: 0.9968 - val_loss: 1.1429 - val_accuracy: 0.7564\n",
      "Epoch 6/10\n",
      "330/330 [==============================] - 15s 46ms/step - loss: 0.0155 - accuracy: 0.9962 - val_loss: 1.1703 - val_accuracy: 0.7557\n",
      "Epoch 7/10\n",
      "330/330 [==============================] - 15s 47ms/step - loss: 0.0175 - accuracy: 0.9950 - val_loss: 0.9698 - val_accuracy: 0.7648\n",
      "Epoch 8/10\n",
      "330/330 [==============================] - 15s 45ms/step - loss: 0.0102 - accuracy: 0.9975 - val_loss: 1.1643 - val_accuracy: 0.7595\n",
      "Epoch 9/10\n",
      "330/330 [==============================] - 15s 45ms/step - loss: 0.0067 - accuracy: 0.9985 - val_loss: 1.3964 - val_accuracy: 0.7617\n",
      "83/83 [==============================] - 1s 9ms/step - loss: 1.3964 - accuracy: 0.7617\n",
      "Test accuracy: 0.7617424130439758, Test loss: 1.3963693380355835\n"
     ]
    }
   ],
   "source": [
    "# LSTM model\n",
    "print('Build LSTM model...')\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Embedding(10000, 300, input_length=maxlen),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20, return_sequences=True)),\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20)),\n",
    "        tf.keras.layers.Dense(6, activation= 'softmax')\n",
    "])\n",
    "\n",
    "model.compile(\n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    optimizer = 'adam',\n",
    "    metrics = ['accuracy']\n",
    ")\n",
    "\n",
    "\n",
    "# run lstm model\n",
    "# the model, with training set, validation set\n",
    "h = model.fit(\n",
    "    x_train44_seq, y_train,\n",
    "    validation_data=( x_test44_seq, y_test,),\n",
    "    epochs=10,\n",
    "    callbacks=[\n",
    "               tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)\n",
    "    ]\n",
    ")\n",
    "\n",
    "# testing model\n",
    "score, accuracy = model.evaluate(x_test44_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build DNN model...\n",
      "Model: \"sequential_83\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_31 (Embedding)    (None, 40, 300)           18000000  \n",
      "                                                                 \n",
      " flatten_31 (Flatten)        (None, 12000)             0         \n",
      "                                                                 \n",
      " dense_194 (Dense)           (None, 512)               6144512   \n",
      "                                                                 \n",
      " dropout_104 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_195 (Dense)           (None, 512)               262656    \n",
      "                                                                 \n",
      " dropout_105 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_196 (Dense)           (None, 1)                 513       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 24,407,681\n",
      "Trainable params: 24,407,681\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "165/165 [==============================] - 21s 124ms/step - loss: 0.2526 - accuracy: 0.8832 - val_loss: 0.5042 - val_accuracy: 0.7765\n",
      "Epoch 2/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0553 - accuracy: 0.9837 - val_loss: 0.7704 - val_accuracy: 0.7595\n",
      "Epoch 3/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0242 - accuracy: 0.9943 - val_loss: 0.8685 - val_accuracy: 0.7663\n",
      "Epoch 4/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0140 - accuracy: 0.9970 - val_loss: 0.9030 - val_accuracy: 0.7723\n",
      "Epoch 5/20\n",
      "165/165 [==============================] - 20s 122ms/step - loss: 0.0100 - accuracy: 0.9983 - val_loss: 1.0011 - val_accuracy: 0.7670\n",
      "Epoch 6/20\n",
      "165/165 [==============================] - 20s 121ms/step - loss: 0.0073 - accuracy: 0.9986 - val_loss: 1.0207 - val_accuracy: 0.7678\n",
      "Epoch 7/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0042 - accuracy: 0.9992 - val_loss: 1.0599 - val_accuracy: 0.7727\n",
      "Epoch 8/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0032 - accuracy: 0.9995 - val_loss: 1.1065 - val_accuracy: 0.7720\n",
      "Epoch 9/20\n",
      "165/165 [==============================] - 21s 127ms/step - loss: 0.0022 - accuracy: 0.9995 - val_loss: 1.1976 - val_accuracy: 0.7708\n",
      "Epoch 10/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0015 - accuracy: 0.9999 - val_loss: 1.2597 - val_accuracy: 0.7731\n",
      "Epoch 11/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0012 - accuracy: 0.9999 - val_loss: 1.2797 - val_accuracy: 0.7731\n",
      "Epoch 12/20\n",
      "165/165 [==============================] - 20s 123ms/step - loss: 0.0010 - accuracy: 0.9999 - val_loss: 1.3054 - val_accuracy: 0.7723\n",
      "Epoch 13/20\n",
      "165/165 [==============================] - 20s 124ms/step - loss: 0.0019 - accuracy: 0.9997 - val_loss: 1.2415 - val_accuracy: 0.7705\n",
      "Epoch 14/20\n",
      "165/165 [==============================] - 21s 125ms/step - loss: 0.0023 - accuracy: 0.9993 - val_loss: 1.2384 - val_accuracy: 0.7663\n",
      "Epoch 15/20\n",
      "165/165 [==============================] - 21s 126ms/step - loss: 0.0015 - accuracy: 0.9994 - val_loss: 1.3241 - val_accuracy: 0.7701\n",
      "Epoch 16/20\n",
      "165/165 [==============================] - 20s 123ms/step - loss: 9.3659e-04 - accuracy: 0.9996 - val_loss: 1.5222 - val_accuracy: 0.7682\n",
      "Epoch 17/20\n",
      "165/165 [==============================] - 20s 121ms/step - loss: 0.0018 - accuracy: 0.9991 - val_loss: 1.5868 - val_accuracy: 0.7723\n",
      "Epoch 18/20\n",
      "165/165 [==============================] - 21s 124ms/step - loss: 0.0046 - accuracy: 0.9991 - val_loss: 1.4737 - val_accuracy: 0.7712\n",
      "Epoch 19/20\n",
      "165/165 [==============================] - 21s 124ms/step - loss: 0.0056 - accuracy: 0.9983 - val_loss: 1.4877 - val_accuracy: 0.7598\n",
      "Epoch 20/20\n",
      "165/165 [==============================] - 20s 122ms/step - loss: 0.0030 - accuracy: 0.9994 - val_loss: 1.5039 - val_accuracy: 0.7557\n",
      "83/83 [==============================] - 0s 4ms/step - loss: 1.5039 - accuracy: 0.7557\n",
      "Test accuracy: 0.7556818127632141, Test loss: 1.503871202468872\n"
     ]
    }
   ],
   "source": [
    "# DNN model\n",
    "\n",
    "print('Build DNN model...')\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(max_features, embedding_dims, input_length=maxlen))\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "# config model\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# training model\n",
    "model.fit(x_train44_seq, y_train, batch_size=64, epochs=20, validation_data=(x_test44_seq, y_test))\n",
    "\n",
    "# testing DNN model\n",
    "score, accuracy = model.evaluate(x_test44_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Classification with emojis word embedding vectors\n",
    "using original tweets & pre-trained model\n",
    "\n",
    "* the result is not very satisfying because many tokens are droped due to key missing in pre-trained model\n",
    "\n",
    "- LSTM: 0.74\n",
    "- DNN: 0.74"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-2.23617658e-01  3.03572536e-01 -2.34024569e-01 -2.47136518e-01\n",
      "  3.42488512e-02  2.26997703e-01 -2.13153273e-01  2.68146336e-01\n",
      "  1.67842329e-01  9.75845903e-02  1.41029686e-01  2.98320055e-01\n",
      "  2.76181161e-01 -5.50599098e-01 -3.01040024e-01 -1.99736565e-01\n",
      " -3.91080305e-02 -1.45365596e-01  3.59019399e-01  2.67562568e-01\n",
      "  8.66341814e-02  3.83522123e-01  3.28554213e-01 -3.49370912e-02\n",
      " -2.03482211e-02  1.65105611e-01 -2.13322595e-01 -4.57124531e-01\n",
      "  5.29949784e-01 -2.87918448e-01 -4.31380808e-01 -3.17272484e-01\n",
      " -2.64893621e-02  1.50817081e-01  1.24318779e-01 -6.29126579e-02\n",
      "  1.95232570e-01  2.90508717e-02 -1.99170351e-01 -4.06680286e-01\n",
      "  1.04108453e-03 -9.04468596e-02 -3.02806824e-01 -9.40951332e-02\n",
      " -1.70607924e-01  1.75087050e-01 -3.72158021e-01 -9.81016532e-02\n",
      "  4.30744514e-02 -4.16884720e-02 -1.62985206e-01 -7.64575601e-03\n",
      "  5.25135756e-01  4.74683762e-01  5.04842401e-01 -1.79820448e-01\n",
      "  3.28110069e-01 -2.36159451e-02  3.41797620e-02 -3.61506194e-02\n",
      " -8.63424167e-02 -2.08626121e-01 -4.28974897e-01  2.12110341e-01\n",
      " -4.19487715e-01  1.41099915e-01 -2.23856956e-01 -4.52091336e-01\n",
      " -2.95989543e-01  6.91810250e-02  5.35630211e-02  1.44041702e-02\n",
      " -2.40476072e-01 -7.89357722e-02  3.26384604e-02  2.31394887e-01\n",
      " -8.31204951e-02 -2.74180233e-01  4.07917202e-01 -2.94747889e-01\n",
      " -5.10340393e-01  1.08603604e-01  4.43460286e-01  5.69357634e-01\n",
      "  3.53778571e-01 -3.16926017e-02  2.22908199e-01  2.79014260e-02\n",
      " -2.86080956e-01  1.09142572e-01  1.83126718e-01 -2.70160228e-01\n",
      " -1.59829617e-01 -8.48058313e-02  2.88024545e-01 -3.36207598e-01\n",
      " -5.65250993e-01  3.89211327e-02  1.20383874e-02 -3.91801566e-01\n",
      " -4.06991422e-01 -4.10706624e-02 -5.43699265e-01 -1.90022588e-01\n",
      "  5.89590192e-01 -4.36830133e-01 -4.54423010e-01  5.93955517e-01\n",
      " -1.50434032e-01  9.57307518e-02 -6.45065904e-02  4.50280219e-01\n",
      " -1.52400434e-01  9.45198685e-02  1.10522166e-01 -7.77239203e-02\n",
      "  9.42670628e-02 -5.95719293e-02  3.69387358e-01  4.66660500e-01\n",
      "  3.98747846e-02 -3.49650502e-01  3.03898990e-01 -1.39499456e-01\n",
      "  4.04930294e-01  2.01525912e-01  2.27135181e-01  1.91256687e-01\n",
      "  2.02429369e-01  3.54770899e-01  3.36399302e-03  1.38485968e-01\n",
      " -1.09922424e-01 -3.05084586e-01  1.44450366e-01  1.26699120e-01\n",
      " -2.82681763e-01 -3.06989729e-01  2.77609676e-02  4.11545962e-01\n",
      "  1.63302913e-01 -9.25123543e-02  8.84597003e-02 -3.97149265e-01\n",
      "  1.09193269e-02  2.10716486e-01  1.89371362e-01 -7.57034868e-03\n",
      "  1.68254480e-01 -7.15897024e-01  2.50899702e-01 -3.95377517e-01\n",
      "  3.88538055e-02  1.05597162e+00  3.48172814e-01  2.27530852e-01\n",
      " -1.29716992e-02 -6.49514973e-01 -8.55180621e-02 -2.43748724e-03\n",
      "  2.17760727e-01  5.24916276e-02 -2.93483436e-01  5.31276911e-02\n",
      " -2.57775962e-01  5.28940737e-01 -2.44006127e-01 -5.59354782e-01\n",
      "  6.53303027e-01  1.03811435e-01 -2.92601824e-01  1.94283560e-01\n",
      "  8.28763843e-01  7.72296369e-01  6.70305431e-01  4.76258874e-01\n",
      " -5.24088800e-01 -4.62592691e-02  4.62247074e-01 -3.12178224e-01\n",
      " -5.74182570e-01  7.83270448e-02 -2.40525275e-01  3.16255659e-01\n",
      "  2.03068435e-01  2.47073770e-01  3.11400354e-01  6.37847483e-02\n",
      "  1.32511511e-01 -8.80229920e-02 -3.98633450e-01 -1.71538800e-01\n",
      "  3.43579412e-01 -4.65327919e-01 -4.02646869e-01  9.00762528e-02\n",
      " -2.63393968e-02 -3.32372069e-01 -2.60425657e-01  2.38782793e-01\n",
      " -4.28790033e-01  7.36200333e-01 -2.29267970e-01 -5.83295166e-01\n",
      " -7.32815266e-02 -1.67518854e-02 -2.33048409e-01  1.50632456e-01\n",
      "  1.33424878e-01 -3.96289080e-01 -6.95065260e-01  2.48038352e-01\n",
      "  8.00993800e-01 -1.97653994e-01  5.68977930e-02  2.27898099e-02\n",
      "  3.58496487e-01 -5.40912867e-01 -6.03149980e-02 -2.21866429e-01\n",
      "  5.99312603e-01 -2.58923411e-01 -3.63489836e-01 -2.25430846e-01\n",
      "  1.35035679e-01  1.45826936e-02 -2.84105420e-01 -1.45651832e-01\n",
      " -5.80859542e-01 -3.70553255e-01  1.74398124e-01 -3.24927241e-01\n",
      " -3.33231837e-01  1.25973299e-01 -3.85885715e-01  4.41772848e-01\n",
      " -3.36528420e-01 -3.96653786e-02  1.42578781e-01  5.57000935e-01\n",
      "  1.04364648e-01 -3.09821427e-01 -1.76822081e-01  1.54118672e-01\n",
      " -1.90536499e-01 -4.07581598e-01 -9.83539522e-02  3.27604972e-02\n",
      "  2.85208493e-01  3.76789868e-01  4.59334821e-01 -1.78002045e-02\n",
      " -5.55022597e-01  2.59279847e-01 -2.71476567e-01  6.09534979e-03\n",
      "  1.06961004e-01  9.52131301e-03 -3.58513921e-01 -2.01796845e-01\n",
      " -1.72335774e-01 -2.23553061e-01  6.80310428e-02 -2.66329587e-01\n",
      " -9.63847935e-02 -1.44954532e-01  5.15890121e-03  4.95136291e-01\n",
      " -1.01394296e-01  2.65441984e-01  3.93849134e-01  3.15934628e-01\n",
      "  1.76305816e-01  1.83399364e-01  2.38698721e-01 -2.14329511e-01\n",
      "  1.98480487e-02  7.23049492e-02 -3.79505754e-02 -5.13924658e-01\n",
      "  1.31960288e-01 -1.38045266e-01 -7.69306779e-01 -8.85191113e-02\n",
      "  5.10822594e-01 -3.34685415e-01 -4.46032971e-01  2.27487087e-03\n",
      "  1.67927295e-02  1.33420497e-01 -1.65024489e-01  1.00278243e-01\n",
      "  1.96767002e-01  7.16374293e-02  5.26592374e-01 -1.93517447e-01\n",
      "  9.65545177e-02 -7.43462205e-01  3.09704781e-01 -1.90565348e-01]\n"
     ]
    }
   ],
   "source": [
    "#import the trianed Word2Vec model\n",
    "# Load back with memory-mapping = read-only, shared across processes.\n",
    "wv = KeyedVectors.load(\"en_word2vec.wordvectors\", mmap='r')\n",
    "#example\n",
    "vector = wv['sun']  # Get numpy vector of a word\n",
    "print(wv['sun']+wv['hi'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-279-c93d67cb3e3b>:19: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  train['word2vec'] = vecls\n"
     ]
    }
   ],
   "source": [
    "# sum all vectors of one tweet\n",
    "\n",
    "#train set\n",
    "vecls = list()\n",
    "for tweet in train.tweets:\n",
    "    vec = []\n",
    "    for word in tweet.split():\n",
    "        if vec == []:\n",
    "            try:\n",
    "                vec = wv[word].tolist()\n",
    "            except:\n",
    "                continue\n",
    "        else:\n",
    "            try:\n",
    "                vec += wv[word].tolist()\n",
    "            except:\n",
    "                continue\n",
    "    vecls.append(vec)\n",
    "train['word2vec'] = vecls\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-276-63eff24cf4c7>:6: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.\n",
      "  if vec == []:\n",
      "<ipython-input-276-63eff24cf4c7>:17: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test['word2vec'] = vecls2\n"
     ]
    }
   ],
   "source": [
    "#test set\n",
    "vecls2 = list()\n",
    "for tweet in test.tweets:\n",
    "    vec = []\n",
    "    for word in tweet.split():\n",
    "        if vec == []:\n",
    "            try:\n",
    "                vec = wv[word]\n",
    "            except:\n",
    "                continue\n",
    "        else:\n",
    "            try:\n",
    "                vec += wv[word]\n",
    "            except:\n",
    "                continue\n",
    "    vecls2.append(vec)\n",
    "test['word2vec'] = vecls2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>tweets</th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "      <th>wtemo</th>\n",
       "      <th>simiemo</th>\n",
       "      <th>word2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10056</th>\n",
       "      <td>10056</td>\n",
       "      <td>the emotion in your match the tweet perfectly  😂</td>\n",
       "      <td>1</td>\n",
       "      <td>the emotion in your match the tweet perfectly</td>\n",
       "      <td>the emotion in your match the tweet perfectly ...</td>\n",
       "      <td>the emotion in your match the tweet perfectly ...</td>\n",
       "      <td>[0.16144225001335144, 0.32414403557777405, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4215</th>\n",
       "      <td>4215</td>\n",
       "      <td>stray are coming to my country guy i cry  😭</td>\n",
       "      <td>0</td>\n",
       "      <td>stray are coming to my country guy i cry</td>\n",
       "      <td>stray are coming to my country guy i cry  loud...</td>\n",
       "      <td>stray are coming to my country guy i cry dry c...</td>\n",
       "      <td>[-0.10359860211610794, 0.11853410303592682, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10394</th>\n",
       "      <td>10394</td>\n",
       "      <td>they are both legit giving happiness that we f...</td>\n",
       "      <td>1</td>\n",
       "      <td>they are both legit giving happiness that we find</td>\n",
       "      <td>they are both legit giving happiness that we f...</td>\n",
       "      <td>they are both legit giving happiness that we f...</td>\n",
       "      <td>[0.2070552110671997, -0.1897415965795517, -0.2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6926</th>\n",
       "      <td>6926</td>\n",
       "      <td>water too warm hand are a pleasant surprise i ...</td>\n",
       "      <td>1</td>\n",
       "      <td>water too warm hand are a pleasant surprise i ...</td>\n",
       "      <td>water too warm hand are a pleasant surprise i ...</td>\n",
       "      <td>water too warm hand are a pleasant surprise i ...</td>\n",
       "      <td>[0.02034851536154747, 0.18007014691829681, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10158</th>\n",
       "      <td>10158</td>\n",
       "      <td>with actor mother  😊</td>\n",
       "      <td>1</td>\n",
       "      <td>with actor mother</td>\n",
       "      <td>with actor mother  smiling face with smiling e...</td>\n",
       "      <td>with actor mother wishing wonderful appreciate...</td>\n",
       "      <td>[0.021692397072911263, -0.013324310071766376, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5836</th>\n",
       "      <td>5836</td>\n",
       "      <td>slut sex here to promote and slut for detail  ❤</td>\n",
       "      <td>0</td>\n",
       "      <td>slut sex here to promote and slut for detail</td>\n",
       "      <td>slut sex here to promote and slut for detail  ...</td>\n",
       "      <td>slut sex here to promote and slut for detail ️...</td>\n",
       "      <td>[-0.10501617938280106, 0.13486815989017487, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7086</th>\n",
       "      <td>7086</td>\n",
       "      <td>ya i feel glad and grateful to have a great  😊</td>\n",
       "      <td>1</td>\n",
       "      <td>ya i feel glad and grateful to have a great</td>\n",
       "      <td>ya i feel glad and grateful to have a great  s...</td>\n",
       "      <td>ya i feel glad and grateful to have a great wi...</td>\n",
       "      <td>[-0.3802816867828369, 0.14718611538410187, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4134</th>\n",
       "      <td>4134</td>\n",
       "      <td>i clapped but i show you lot the ugliness  😭</td>\n",
       "      <td>0</td>\n",
       "      <td>i clapped but i show you lot the ugliness</td>\n",
       "      <td>i clapped but i show you lot the ugliness  lou...</td>\n",
       "      <td>i clapped but i show you lot the ugliness dry ...</td>\n",
       "      <td>[-0.004160878248512745, -0.09457463771104813, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9053</th>\n",
       "      <td>9053</td>\n",
       "      <td>the only meat ill be getting today thanks to  ...</td>\n",
       "      <td>1</td>\n",
       "      <td>the only meat ill be getting today thanks to  ...</td>\n",
       "      <td>the only meat ill be getting today thanks to  ...</td>\n",
       "      <td>the only meat ill be getting today thanks to k...</td>\n",
       "      <td>[0.16144225001335144, 0.32414403557777405, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10137</th>\n",
       "      <td>10137</td>\n",
       "      <td>always doe what he is supposed to do in the mi...</td>\n",
       "      <td>1</td>\n",
       "      <td>always doe what he is supposed to do in the mi...</td>\n",
       "      <td>always doe what he is supposed to do in the mi...</td>\n",
       "      <td>always doe what he is supposed to do in the mi...</td>\n",
       "      <td>[0.028448142111301422, 0.07543884217739105, -0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10560 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                             tweets  labels  \\\n",
       "10056       10056  the emotion in your match the tweet perfectly  😂        1   \n",
       "4215         4215       stray are coming to my country guy i cry  😭        0   \n",
       "10394       10394  they are both legit giving happiness that we f...       1   \n",
       "6926         6926  water too warm hand are a pleasant surprise i ...       1   \n",
       "10158       10158                              with actor mother  😊        1   \n",
       "...           ...                                                ...     ...   \n",
       "5836         5836   slut sex here to promote and slut for detail  ❤        0   \n",
       "7086         7086    ya i feel glad and grateful to have a great  😊        1   \n",
       "4134         4134      i clapped but i show you lot the ugliness  😭        0   \n",
       "9053         9053  the only meat ill be getting today thanks to  ...       1   \n",
       "10137       10137  always doe what he is supposed to do in the mi...       1   \n",
       "\n",
       "                                                    text  \\\n",
       "10056      the emotion in your match the tweet perfectly   \n",
       "4215            stray are coming to my country guy i cry   \n",
       "10394  they are both legit giving happiness that we find   \n",
       "6926   water too warm hand are a pleasant surprise i ...   \n",
       "10158                                  with actor mother   \n",
       "...                                                  ...   \n",
       "5836        slut sex here to promote and slut for detail   \n",
       "7086         ya i feel glad and grateful to have a great   \n",
       "4134           i clapped but i show you lot the ugliness   \n",
       "9053   the only meat ill be getting today thanks to  ...   \n",
       "10137  always doe what he is supposed to do in the mi...   \n",
       "\n",
       "                                                   wtemo  \\\n",
       "10056  the emotion in your match the tweet perfectly ...   \n",
       "4215   stray are coming to my country guy i cry  loud...   \n",
       "10394  they are both legit giving happiness that we f...   \n",
       "6926   water too warm hand are a pleasant surprise i ...   \n",
       "10158  with actor mother  smiling face with smiling e...   \n",
       "...                                                  ...   \n",
       "5836   slut sex here to promote and slut for detail  ...   \n",
       "7086   ya i feel glad and grateful to have a great  s...   \n",
       "4134   i clapped but i show you lot the ugliness  lou...   \n",
       "9053   the only meat ill be getting today thanks to  ...   \n",
       "10137  always doe what he is supposed to do in the mi...   \n",
       "\n",
       "                                                 simiemo  \\\n",
       "10056  the emotion in your match the tweet perfectly ...   \n",
       "4215   stray are coming to my country guy i cry dry c...   \n",
       "10394  they are both legit giving happiness that we f...   \n",
       "6926   water too warm hand are a pleasant surprise i ...   \n",
       "10158  with actor mother wishing wonderful appreciate...   \n",
       "...                                                  ...   \n",
       "5836   slut sex here to promote and slut for detail ️...   \n",
       "7086   ya i feel glad and grateful to have a great wi...   \n",
       "4134   i clapped but i show you lot the ugliness dry ...   \n",
       "9053   the only meat ill be getting today thanks to k...   \n",
       "10137  always doe what he is supposed to do in the mi...   \n",
       "\n",
       "                                                word2vec  \n",
       "10056  [0.16144225001335144, 0.32414403557777405, -0....  \n",
       "4215   [-0.10359860211610794, 0.11853410303592682, 0....  \n",
       "10394  [0.2070552110671997, -0.1897415965795517, -0.2...  \n",
       "6926   [0.02034851536154747, 0.18007014691829681, -0....  \n",
       "10158  [0.021692397072911263, -0.013324310071766376, ...  \n",
       "...                                                  ...  \n",
       "5836   [-0.10501617938280106, 0.13486815989017487, -0...  \n",
       "7086   [-0.3802816867828369, 0.14718611538410187, -0....  \n",
       "4134   [-0.004160878248512745, -0.09457463771104813, ...  \n",
       "9053   [0.16144225001335144, 0.32414403557777405, -0....  \n",
       "10137  [0.028448142111301422, 0.07543884217739105, -0...  \n",
       "\n",
       "[10560 rows x 7 columns]"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10560 2640\n"
     ]
    }
   ],
   "source": [
    "x_train45 = train['tweets'].to_list()\n",
    "x_test45 = test['tweets'].to_list()\n",
    "\n",
    "print(len(x_train45),len(x_test45))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 315,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#example\n",
    "word_index['😂']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the words if not in word_index\n",
    "wls = []\n",
    "for i in train['tweets']:\n",
    "    ls = []\n",
    "    words = i.split(' ')\n",
    "    for w in words:\n",
    "        try:\n",
    "            word_index[w]\n",
    "            ls.append(w)\n",
    "        except:\n",
    "            continue\n",
    "    wls.append(' '.join(ls))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [],
   "source": [
    "wls2 = []\n",
    "for i in test['tweets']:\n",
    "    ls = []\n",
    "    words = i.split(' ')\n",
    "    for w in words:\n",
    "        try:\n",
    "            word_index[w]\n",
    "            ls.append(w)\n",
    "        except:\n",
    "            continue\n",
    "    wls2.append(' '.join(ls))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train45 = wls\n",
    "x_test45 = wls2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "[[4, 1471, 20, 29, 725, 4, 262, 2008, 8]]\n",
      "[[332, 907, 1171, 32, 335, 42, 7, 366]]\n",
      "x_train shape: (10560, 40)\n",
      "x_train shape: (2640, 40)\n"
     ]
    }
   ],
   "source": [
    "# encode the words\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "print('Loading data...')\n",
    "def get_sequences(tokenizer, tweets):\n",
    "  sequences = tokenizer.texts_to_sequences(tweets)\n",
    "  padded = pad_sequences(sequences, truncating ='post', maxlen = maxlen)\n",
    "  return padded\n",
    "\n",
    "# tokenizer\n",
    "tokenizer = Tokenizer(num_words=10000, oov_token='<UNK>')\n",
    "tokenizer.fit_on_texts(x_train45)\n",
    "word_index = tokenizer.word_index #get word_index\n",
    "print(tokenizer.texts_to_sequences([x_train45[0]]))\n",
    "x_train45_seq = get_sequences(tokenizer, x_train45)\n",
    "\n",
    "tokenizer.fit_on_texts(x_test45)\n",
    "print(tokenizer.texts_to_sequences([x_test45[0]]))\n",
    "x_test45_seq = get_sequences(tokenizer, x_test45)\n",
    "\n",
    "print('x_train shape:', x_train45_seq.shape)\n",
    "print('x_train shape:', x_test45_seq.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {},
   "outputs": [],
   "source": [
    "padding_type='post'\n",
    "truncation_type='post'\n",
    "\n",
    "# set parameters\n",
    "max_features = 60000 # cut texts after this number of words (among top max_features most common words)\n",
    "embedding_dims = 300\n",
    "maxlen = 40 #based on 2, most tweets are shorter than 40 words\n",
    "\n",
    "embedding_matrix = np.zeros((len(word_index) + 1, embedding_dims))\n",
    "for word, i in word_index.items():\n",
    "    try:\n",
    "        embedding_vector = wv[word]\n",
    "    except:\n",
    "        embedding_vector = None\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5813, 300)"
      ]
     },
     "execution_count": 326,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional\n",
    "\n",
    "embedding_layer = Embedding(input_dim=len(word_index)+1,\n",
    "                            output_dim= embedding_dims,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=40,\n",
    "                            trainable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build LSTM model...\n",
      "Epoch 1/10\n",
      "330/330 [==============================] - 15s 33ms/step - loss: 0.2178 - accuracy: 0.9159 - val_loss: 0.7456 - val_accuracy: 0.7458\n",
      "Epoch 2/10\n",
      "330/330 [==============================] - 7s 22ms/step - loss: 0.1237 - accuracy: 0.9555 - val_loss: 0.7662 - val_accuracy: 0.7326\n",
      "Epoch 3/10\n",
      "330/330 [==============================] - 7s 23ms/step - loss: 0.1029 - accuracy: 0.9650 - val_loss: 0.6715 - val_accuracy: 0.7583\n",
      "Epoch 4/10\n",
      "330/330 [==============================] - 7s 22ms/step - loss: 0.0882 - accuracy: 0.9681 - val_loss: 0.8753 - val_accuracy: 0.7443\n",
      "Epoch 5/10\n",
      "330/330 [==============================] - 8s 24ms/step - loss: 0.0702 - accuracy: 0.9766 - val_loss: 0.8434 - val_accuracy: 0.7417\n",
      "83/83 [==============================] - 1s 10ms/step - loss: 0.8434 - accuracy: 0.7417\n",
      "Test accuracy: 0.7416666746139526, Test loss: 0.8434050679206848\n"
     ]
    }
   ],
   "source": [
    "# LSTM model\n",
    "print('Build LSTM model...')\n",
    "\n",
    "\n",
    "from tensorflow.keras.callbacks import EarlyStopping, TensorBoard\n",
    "from tensorflow.keras.models import Sequential\n",
    "model = Sequential([\n",
    "    embedding_layer,\n",
    "    Bidirectional(LSTM(20, return_sequences=True)), \n",
    "    Bidirectional(LSTM(20)),\n",
    "    Dense(128, activation='relu'),\n",
    "   Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
    "\n",
    "# testing model\n",
    "log_folder = 'logs'\n",
    "callbacks = [\n",
    "            EarlyStopping(patience = 10),\n",
    "            TensorBoard(log_dir=log_folder)\n",
    "            ]\n",
    "num_epochs = 10\n",
    "model.fit(x_train45_seq, y_train, epochs=num_epochs, validation_data=(x_test45_seq, y_test),\n",
    "          callbacks=[\n",
    "               tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)\n",
    "    ])\n",
    "\n",
    "\n",
    "# testing model\n",
    "score, accuracy = model.evaluate(x_test45_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build DNN model...\n",
      "Model: \"sequential_82\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_21 (Embedding)    (None, 40, 300)           1743900   \n",
      "                                                                 \n",
      " flatten_30 (Flatten)        (None, 12000)             0         \n",
      "                                                                 \n",
      " dense_191 (Dense)           (None, 512)               6144512   \n",
      "                                                                 \n",
      " dropout_102 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_192 (Dense)           (None, 512)               262656    \n",
      "                                                                 \n",
      " dropout_103 (Dropout)       (None, 512)               0         \n",
      "                                                                 \n",
      " dense_193 (Dense)           (None, 1)                 513       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 8,151,581\n",
      "Trainable params: 6,407,681\n",
      "Non-trainable params: 1,743,900\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 0.3177 - accuracy: 0.8581 - val_loss: 0.5549 - val_accuracy: 0.7670\n",
      "Epoch 2/20\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.1512 - accuracy: 0.9484 - val_loss: 0.7142 - val_accuracy: 0.7322\n",
      "Epoch 3/20\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.1135 - accuracy: 0.9598 - val_loss: 0.7527 - val_accuracy: 0.7398\n",
      "83/83 [==============================] - 0s 4ms/step - loss: 0.7527 - accuracy: 0.7398\n",
      "Test accuracy: 0.7397727370262146, Test loss: 0.7526956796646118\n"
     ]
    }
   ],
   "source": [
    "# DNN model\n",
    "\n",
    "print('Build DNN model...')\n",
    "model = tf.keras.Sequential()\n",
    "model.add(embedding_layer)\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(512, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dropout(0.4))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "# config model\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# training model\n",
    "model.fit(x_train45_seq, y_train, batch_size=64, epochs=20, validation_data=(x_test45_seq, y_test),\n",
    "         callbacks=[\n",
    "               tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)\n",
    "    ])\n",
    "\n",
    "# testing DNN model\n",
    "score, accuracy = model.evaluate(x_test45_seq, y_test)\n",
    "print('Test accuracy: {}, Test loss: {}'.format(accuracy, score))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}