1_Text_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Text Data in Natural Language Processing- Part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "<img src=\"NLP-image.jpg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a Natural Language Processing? \n",
    "A field of computer science, artificial intelligence and computational linguistics concerned with \n",
    "the interactions between computers and human (natural) languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing\n",
    "\n",
    "Also known as data wrangling, Data Cleaning\n",
    "\n",
    "Process of converting data from the initial raw form into another format, in order \n",
    "to prepare data for further analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. lower case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_data='HelLO'\n",
    "sent_data='I TeaCh ProGRAmmiNG'\n",
    "para_data='ToDaY IS SaturDay. I am FEEling Hungry'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"lower_case.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lower_case(text):\n",
    "    answer=text.lower()\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lower_casing=lower_case(word_data)#---------> FUNCTION CALLING\n",
    "lower_casing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i teach programming'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lower_casing=lower_case(sent_data)\n",
    "lower_casing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'today is saturday. i am feeling hungry'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lower_casing=lower_case(para_data)\n",
    "lower_casing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "string.punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample=\"I am@ ! very happy today. I'll be fine@\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method 1\n",
    "\n",
    "def punct(text):\n",
    "    obj=str.maketrans('', '',string.punctuation)\n",
    "    answer=text.translate(obj)\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"punctuation.jpeg\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am  very happy today Ill be fine'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output=punct(sample)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method 2 \n",
    "\n",
    "# string as input\n",
    "def remove_punctuation(s):\n",
    "    list_punctuation = list(string.punctuation)\n",
    "    for i in list_punctuation:\n",
    "        s = s.replace(i,'')\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"replace_func.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am  very happy today Ill be fine'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample=\"I am@ ! very happy today. I'll be fine@\"\n",
    "remove_punctuation(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NOTE: Some imporatnt Modifications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "edited_string=string.punctuation.replace('#', '1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "q=\"I am#  very# happy, today@ Ill!! be fine.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def punct(text):\n",
    "    obj=str.maketrans('', '',edited_string)\n",
    "    answer=text.translate(obj)\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am#  very# happy today Ill be fine'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "punct(q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"punctuation1.jpeg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"word_sent_tokenize.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_word(data):\n",
    "    words=word_tokenize(data)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I',\n",
       " 'love',\n",
       " 'watching',\n",
       " 'TV',\n",
       " 'shows',\n",
       " '.',\n",
       " 'My',\n",
       " 'favourite',\n",
       " 'show',\n",
       " 'is',\n",
       " 'Silicon',\n",
       " 'Valley',\n",
       " '.']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample=\"I love watching TV shows. My favourite show is Silicon Valley.\"\n",
    "tokenize_word(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize\n",
    "# sentences=[sent_tokenize(item) for item in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenized_sent(data):\n",
    "    sentences=sent_tokenize(data)\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I love watching TV shows.', 'My favourite show is Silicon Valley.']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_sent(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"sentence_tokenization.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. appos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load appos.py\n",
    "appos = {\n",
    "\"aren't\" : \"are not\",\n",
    "\"can't\" : \"cannot\",\n",
    "\"couldn't\" : \"could not\",\n",
    "\"didn't\" : \"did not\",\n",
    "\"doesn't\" : \"does not\",\n",
    "\"don't\" : \"do not\",\n",
    "\"hadn't\" : \"had not\",\n",
    "\"hasn't\" : \"has not\",\n",
    "\"haven't\" : \"have not\",\n",
    "\"he'd\" : \"he would\",\n",
    "\"he'll\" : \"he will\",\n",
    "\"he's\" : \"he is\",\n",
    "\"i'd\" : \"I would\",\n",
    "\"i'd\" : \"I had\",\n",
    "\"i'll\" : \"I will\",\n",
    "\"i'm\" : \"I am\",\n",
    "\"isn't\" : \"is not\",\n",
    "\"it's\" : \"it is\",\n",
    "\"it'll\":\"it will\",\n",
    "\"i've\" : \"I have\",\n",
    "\"let's\" : \"let us\",\n",
    "\"mightn't\" : \"might not\",\n",
    "\"mustn't\" : \"must not\",\n",
    "\"shan't\" : \"shall not\",\n",
    "\"she'd\" : \"she would\",\n",
    "\"she'll\" : \"she will\",\n",
    "\"she's\" : \"she is\",\n",
    "\"shouldn't\" : \"should not\",\n",
    "\"that's\" : \"that is\",\n",
    "\"there's\" : \"there is\",\n",
    "\"they'd\" : \"they would\",\n",
    "\"they'll\" : \"they will\",\n",
    "\"they're\" : \"they are\",\n",
    "\"they've\" : \"they have\",\n",
    "\"we'd\" : \"we would\",\n",
    "\"we're\" : \"we are\",\n",
    "\"weren't\" : \"were not\",\n",
    "\"we've\" : \"we have\",\n",
    "\"what'll\" : \"what will\",\n",
    "\"what're\" : \"what are\",\n",
    "\"what's\" : \"what is\",\n",
    "\"what've\" : \"what have\",\n",
    "\"where's\" : \"where is\",\n",
    "\"who'd\" : \"who would\",\n",
    "\"who'll\" : \"who will\",\n",
    "\"who're\" : \"who are\",\n",
    "\"who's\" : \"who is\",\n",
    "\"who've\" : \"who have\",\n",
    "\"won't\" : \"will not\",\n",
    "\"wouldn't\" : \"would not\",\n",
    "\"you'd\" : \"you would\",\n",
    "\"you'll\" : \"you will\",\n",
    "\"you're\" : \"you are\",\n",
    "\"you've\" : \"you have\",\n",
    "\"'re\": \" are\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'll\":\" will\",\n",
    "\"didn't\": \"did not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\", \n",
    "\"'cause\": \"because\", \n",
    "\"could've\": \"could have\",\n",
    " \"couldn't\": \"could not\", \n",
    "\"didn't\": \"did not\", \n",
    " \"doesn't\": \"does not\", \n",
    "\"don't\": \"do not\",\n",
    " \"hadn't\": \"had not\", \n",
    "\"hasn't\": \"has not\",\n",
    " \"haven't\": \"have not\", \n",
    "\"he'd\": \"he would\",\n",
    "\"he'll\": \"he will\", \n",
    "\"he's\": \"he is\", \n",
    "\"how'd\": \"how did\", \n",
    "\"how'd'y\": \"how do you\", \n",
    "\"how'll\": \"how will\", \n",
    "\"how's\": \"how is\", \n",
    " \"I'd\": \"I would\",\n",
    " \"I'd've\": \"I would have\", \n",
    "\"I'll\": \"I will\", \n",
    "\"I'll've\": \"I will have\",\n",
    "\"I'm\": \"I am\", \n",
    "\"I've\": \"I have\",\n",
    "\"i'd\": \"i would\", \n",
    "\"i'd've\": \"i would have\", \n",
    "\"i'll\": \"i will\", \n",
    " \"i'll've\": \"i will have\",\n",
    "\"i'm\": \"i am\", \n",
    "\"i've\": \"i have\", \n",
    "\"isn't\": \"is not\", \n",
    "\"it'd\": \"it would\", \n",
    "\"it'd've\": \"it would have\", \n",
    "\"it'll\": \"it will\",\n",
    "\"it'll've\": \"it will have\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\", \n",
    "\"ma'am\": \"madam\",\n",
    " \"mayn't\": \"may not\", \n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"mightn't've\": \"might not have\", \n",
    "\"must've\": \"must have\", \n",
    "\"mustn't\": \"must not\", \n",
    "\"mustn't've\": \"must not have\", \n",
    "\"needn't\": \"need not\", \n",
    "\"needn't've\": \"need not have\",\n",
    "\"o'clock\": \"of the clock\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"oughtn't've\": \"ought not have\", \n",
    "\"shan't\": \"shall not\", \n",
    "\"sha'n't\": \"shall not\", \n",
    "\"shan't've\": \"shall not have\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'd've\": \"she would have\",\n",
    "\"she'll\": \"she will\", \n",
    "\"she'll've\": \"she will have\", \n",
    "\"she's\": \"she is\", \n",
    "\"should've\": \"should have\", \n",
    "\"shouldn't\": \"should not\", \n",
    "\"shouldn't've\": \"should not have\", \n",
    "\"so've\": \"so have\",\"so's\": \"so as\", \n",
    "\"this's\": \"this is\",\n",
    "\"that'd\": \"that would\", \n",
    "\"that'd've\": \"that would have\", \n",
    "\"that's\": \"that is\", \n",
    "\"there'd\": \"there would\", \n",
    "\"there'd've\": \"there would have\", \n",
    "\"there's\": \"there is\", \n",
    "\"here's\": \"here is\",\n",
    "\"they'd\": \"they would\", \n",
    "\"they'd've\": \"they would have\", \n",
    "\"they'll\": \"they will\", \n",
    "\"they'll've\": \"they will have\", \n",
    "\"they're\": \"they are\", \n",
    "\"they've\": \"they have\", \n",
    "\"to've\": \"to have\", \n",
    "\"wasn't\": \"was not\", \n",
    "\"we'd\": \"we would\", \n",
    "\"we'd've\": \"we would have\", \n",
    "\"we'll\": \"we will\", \n",
    "\"we'll've\": \"we will have\", \n",
    "\"we're\": \"we are\", \n",
    "\"we've\": \"we have\", \n",
    "\"weren't\": \"were not\", \n",
    "\"what'll\": \"what will\", \n",
    "\"what'll've\": \"what will have\", \n",
    "\"what're\": \"what are\",  \n",
    "\"what's\": \"what is\", \n",
    "\"what've\": \"what have\", \n",
    "\"when's\": \"when is\", \n",
    "\"when've\": \"when have\", \n",
    "\"where'd\": \"where did\", \n",
    "\"where's\": \"where is\", \n",
    "\"where've\": \"where have\", \n",
    "\"who'll\": \"who will\", \n",
    "\"who'll've\": \"who will have\", \n",
    "\"who's\": \"who is\",\n",
    "\"who've\": \"who have\", \n",
    "\"why's\": \"why is\", \n",
    "\"why've\": \"why have\", \n",
    "\"will've\": \"will have\", \n",
    "\"won't\": \"will not\", \n",
    "\"won't've\": \"will not have\", \n",
    "\"would've\": \"would have\", \n",
    "\"wouldn't\": \"would not\", \n",
    "\"wouldn't've\": \"would not have\", \n",
    "\"y'all\": \"you all\", \n",
    "\"y'all'd\": \"you all would\",\n",
    "\"y'all'd've\": \"you all would have\",\n",
    "\"y'all're\": \"you all are\",\n",
    "\"y'all've\": \"you all have\",\n",
    "\"you'd\": \"you would\", \n",
    "\"you'd've\": \"you would have\", \n",
    "\"you'll\": \"you will\", \n",
    "\"you'll've\": \n",
    "\"you will have\", \n",
    "\"you're\": \"you are\", \n",
    "\"you've\": \"you have\"\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from appos import appos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=\"Y'all can't expand contractions I'd think\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method 1\n",
    "\n",
    "def expand_contract(text):\n",
    "    data=text.lower()\n",
    "    tokens=data.split()\n",
    "    contract=[appos[word] if word in appos else word for word in tokens]\n",
    "    answer=' '.join(contract)\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in tokens:\n",
    "    if word in appos:\n",
    "        appos[word]\n",
    "    else:\n",
    "        word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'you all cannot expand contractions i would think'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expand_contract(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method 2\n",
    "\n",
    "# def expand_contractions(text, contraction_mapping=appos):\n",
    "    \n",
    "#     contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())), \n",
    "#                                       flags=re.IGNORECASE|re.DOTALL)\n",
    "#     def expand_match(contraction):\n",
    "#         match = contraction.group(0)\n",
    "#         first_char = match[0]\n",
    "#         expanded_contraction = contraction_mapping.get(match)\\\n",
    "#                                 if contraction_mapping.get(match)\\\n",
    "#                                 else contraction_mapping.get(match.lower())                       \n",
    "#         expanded_contraction = first_char+expanded_contraction[1:]\n",
    "#         return expanded_contraction\n",
    "    \n",
    "#     expanded_text = contractions_pattern.sub(expand_match, text)\n",
    "#     expanded_text = re.sub(\"'\", \"\", expanded_text)\n",
    "#     return expanded_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# expand_contractions(\"Y'all can't expand contractions I'd think\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Stop Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords \n",
    "from nltk.tokenize import word_tokenize "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"stopwords.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = set(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stopwords_removal(data):\n",
    "    filtered_sentence = [] \n",
    "    word_tokens = word_tokenize(data)                                                # --STEP:1\n",
    "    filtered_sentence = [w for w in word_tokens if not w in stop_words]               # --STEP:2\n",
    "#     for w in word_tokens: \n",
    "#         if w not in stop_words: \n",
    "#             filtered_sentence.append(w)\n",
    "    final=' '.join(filtered_sentence)                                                # --STEP:3\n",
    "    return final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for w in word_tokens:\n",
    "    if not w in stop_words:\n",
    "        w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This sample sentence , showing stop words filtration .'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample=\"This is a sample sentence, showing off the stop words filtration.\"\n",
    "stopwords_removal(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha-numeric words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "alphanumeric characters are those comprised of the combined set of the 26 alphabetic characters, A to Z, \n",
    "and the 10 Arabic numerals, 0 to 9.\n",
    "\n",
    "Non alpha-numeric characters:\n",
    "!\t$+$\t$<$\t[\n",
    "%\t,\t$<=$\t]\n",
    "&\t-\t$<>$\t|\n",
    "'\t.\t=\t~\n",
    "(\t/\t==\t~=\n",
    ")\t/!\t$>$\t(space)\n",
    "*\t//\t$>$=\t(return)\n",
    "*!\t{\t?\t`\n",
    "**\t}\t@\t:\n",
    ";\t^\t|=\t&=\n",
    "+=\t-=\t*=\t/=\n",
    "**=\t "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=\"Well this was fun! What do you think? 123#@!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "# def remove_special_characters(text, remove_digits=True):\n",
    "#     pattern = r'[^a-zA-z\\s]' if remove_digits else 'r[^a-zA-z0-9\\s]'\n",
    "#     text = re.sub(pattern, '', text)\n",
    "#     return text\n",
    "\n",
    "def remove_special_characters(text):\n",
    "    pattern = r'[^a-zA-z\\s]'\n",
    "    text = re.sub(pattern, '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Well this was fun What do you think '"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "remove_special_characters(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"alpha_numeric.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Morphological Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"inflection.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Applications:\n",
    "Stemming and Lemmatization are widely used in tagging systems, indexing, SEOs, Web search results,\n",
    "and information retrieval. For example, searching for fish on Google will also result in\n",
    "fishes, fishing as fish is the stem of both words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem import LancasterStemmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"stemming_steps.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_stemmer(text):\n",
    "    porter = PorterStemmer()\n",
    "    filtered=[porter.stem(word) for word in text.split()]\n",
    "    text = ' '.join(filtered)\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'My system keep crash hi crash yesterday, our crash daili'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_stemmer(\"My system keeps crashing his crashed yesterday, ours crashes daily\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lancaster_stemmer(text):\n",
    "    porter = LancasterStemmer()\n",
    "    filtered=[porter.stem(word) for word in text.split()]\n",
    "    text = ' '.join(filtered)\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my system keep crash his crash yesterday, our crash dai'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lancaster_stemmer(\"My system keeps crashing his crashed yesterday, ours crashes daily\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text=\"The first time I ate here I honestly was not that impressed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_lemma(data):\n",
    "    filtered = []\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    filtered=[lemmatizer.lemmatize(word) for word in data.split()]\n",
    "    text = ' '.join(filtered)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The first time I ate here I honestly wa not that impressed.'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_lemma(sample_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# POS Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "sample=\"I love mangoes\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tag(data):\n",
    "     x=nltk.word_tokenize(data)\n",
    "     answer=[nltk.pos_tag(x)]\n",
    "     return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('I', 'PRP'), ('love', 'VBP'), ('mangoes', 'NNS')]]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package tagsets to /home/shivangi/nltk_data...\n",
      "[nltk_data]   Package tagsets is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('tagsets')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PRP$: pronoun, possessive\n",
      "    her his mine my our ours their thy your\n"
     ]
    }
   ],
   "source": [
    "nltk.help.upenn_tagset(\"PRP$\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VBP: verb, present tense, not 3rd person singular\n",
      "    predominate wrap resort sue twist spill cure lengthen brush terminate\n",
      "    appear tend stray glisten obtain comprise detest tease attract\n",
      "    emphasize mold postpone sever return wag ...\n"
     ]
    }
   ],
   "source": [
    "nltk.help.upenn_tagset(\"VBP$\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chunking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "grammar = ('''\n",
    "    NP: {<DT>?<JJ>*<NN>} # NP\n",
    "    ''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 'DT'),\n",
       " ('little', 'JJ'),\n",
       " ('yellow', 'JJ'),\n",
       " ('dog', 'NN'),\n",
       " ('barked', 'VBD'),\n",
       " ('at', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('cat', 'NN')]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample = \"the little yellow dog barked at the cat\"\n",
    "chunkParser = nltk.RegexpParser(grammar)\n",
    "tagged = nltk.pos_tag(nltk.word_tokenize(sample))\n",
    "tagged\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = chunkParser.parse(tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for subtree in tree.subtrees():\n",
    "#     print(subtree)\n",
    "# tree.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAABTCAIAAACzjfyQAAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBAZokEykBKkDulARhAbsoYcIDfIGwqCQqFoKAnKgHKhnVARVApVQXVQE/QLdA66At2EBqGH0Dg0A/0NfYQRmATTYQVYA9aH2TAHdoV94fVwNJwK58D58F64Aq6HT8Id8BX4NjwMC+GX8BwCECLCQJQRXYSNcBEPJBiJQgTIVqQQKUfqkVakG+lD7iFCZBb5gMKgaCgmShdliXJG+aH4qFTUVlQxqgp1AtWB6kXdQ42jRKjPaDJaHq2DtkDz0IHoaHQWugBdjm5Et6OvoYfRk+h3GAyGgWFhzDDOmCBMHGYzphhzGNOGuYwZxExg5rBYrAxWB2uF9cCGYdOxBdhK7EnsJewQdhL7HkfEKeGMcI64YFwSLg9XjmvGXcQN4aZwC3hxvDreAu+Bj8BvwpfgG/Dd+Dv4SfwCQYLAIlgRfAlxhB2ECkIr4RphjPCGSCSqEM2JXsRY4nZiBfEU8QZxnPiBRCVpk7ikEFIGaS/pOOky6SHpDZlM1iDbkoPJ6eS95CbyVfJT8nsxmpieGE8sQmybWLVYh9iQ2CsKnqJO4VA2UHIo5ZQzlDuUWXG8uIY4VzxMfKt4tfg58VHxOQmahKGEh0SiRLFEs8RNiWkqlqpBdaBGUPOpx6hXqRM0hKZK49L4tJ20Bto12iQdQ2fRefQ4ehH9Z/oAXSRJlTSW9JfMlqyWvCApZCAMDQaPkcAoYZxmjDA+SilIcaQipfZItUoNSc1Ly0nbSkdKF0q3SQ9Lf5RhyjjIxMvsl+mUeSKLktWW9ZLNkj0ie012Vo4uZynHlyuUOy33SB6W15b3lt8sf0y+X35OQVHBSSFFoVLhqsKsIkPRVjFOsUzxouKMEk3JWilWqUzpktILpiSTw0xgVjB7mSJleWVn5QzlOuUB5QUVloqfSp5Km8oTVYIqWzVKtUy1R1WkpqTmrpar1qL2SB2vzlaPUT+k3qc+r8HSCNDYrdGpMc2SZvFYOawW1pgmWdNGM1WzXvO+FkaLrRWvdVjrrjasbaIdo12tfUcH1jHVidU5rDO4Cr3KfFXSqvpVo7okXY5upm6L7rgeQ89NL0+vU++Vvpp+sP5+/T79zwYmBgkGDQaPDamGLoZ5ht2GfxtpG/GNqo3uryavdly9bXXX6tfGOsaRxkeMH5jQTNxNdpv0mHwyNTMVmLaazpipmYWa1ZiNsulsT3Yx+4Y52tzOfJv5efMPFqYW6RanLf6y1LWMt2y2nF7DWhO5pmHNhJWKVZhVnZXQmmkdan3UWmijbBNmU2/zzFbVNsK20XaKo8WJ45zkvLIzsBPYtdvNcy24W7iX7RF7J/tC+wEHqoOfQ5XDU0cVx2jHFkeRk4nTZqfLzmhnV+f9zqM8BR6f18QTuZi5bHHpdSW5+rhWuT5z03YTuHW7w+4u7gfcx9aqr01a2+kBPHgeBzyeeLI8Uz1/9cJ4eXpVez33NvTO9e7zofls9Gn2eedr51vi+9hP0y/Dr8ef4h/i3+Q/H2AfUBogDNQP3BJ4O0g2KDaoKxgb7B/cGDy3zmHdwXWTISYhBSEj61nrs9ff3CC7IWHDhY2UjWEbz4SiQwNCm0MXwzzC6sPmwnnhNeEiPpd/iP8ywjaiLGIm0iqyNHIqyiqqNGo62ir6QPRMjE1MecxsLDe2KvZ1nHNcbdx8vEf88filhICEtkRcYmjiuSRqUnxSb7JicnbyYIpOSkGKMNUi9WCqSOAqaEyD0tandaXTlz/F/gzNjF0Z45nWmdWZ77P8s85kS2QnZfdv0t60Z9NUjmPOT5tRm/mbe3KVc3fkjm/hbKnbCm0N39qzTXVb/rbJ7U7bT+wg7Ijf8VueQV5p3tudATu78xXyt+dP7HLa1VIgViAoGN1tubv2B9QPsT8M7Fm9p3LP58KIwltFBkXlRYvF/OJbPxr+WPHj0t6ovQMlpiVH9mH2Je0b2W+z/0SpRGlO6cQB9wMdZcyywrK3BzcevFluXF57iHAo45Cwwq2iq1Ktcl/lYlVM1XC1XXVbjXzNnpr5wxGHh47YHmmtVagtqv14NPbogzqnuo56jfryY5hjmceeN/g39P3E/qmpUbaxqPHT8aTjwhPeJ3qbzJqamuWbS1rgloyWmZMhJ+/+bP9zV6tua10bo63oFDiVcerFL6G/jJx2Pd1zhn2m9az62Zp2WnthB9SxqUPUGdMp7ArqGjzncq6n27K7/Ve9X4+fVz5ffUHyQslFwsX8i0uXci7NXU65PHsl+spEz8aex1cDr97v9eoduOZ67cZ1x+tX+zh9l25Y3Th/0+LmuVvsW523TW939Jv0t/9m8lv7gOlAxx2zO113ze92D64ZvDhkM3Tlnv296/d5928Prx0eHPEbeTAaMip8EPFg+mHCw9ePMh8tPN4+hh4rfCL+pPyp/NP637V+bxOaCi+M24/3P/N59niCP/Hyj7Q/Fifzn5Ofl08pTTVNG02fn3Gcufti3YvJlykvF2YL/pT4s+aV5quzf9n+1S8KFE2+Frxe+rv4jcyb42+N3/bMec49fZf4bmG+8L3M+xMf2B/6PgZ8nFrIWsQuVnzS+tT92fXz2FLi0tI/QiyQvpTNDAsAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAddEVYdFNvZnR3YXJlAEdQTCBHaG9zdHNjcmlwdCA5LjI2WJButwAAEWhJREFUeJzt3U9s21aeB/DnbGZR2W0aeqDsIFusbWrRAg56aKkAezDWKUQdkl5DH4teLAHFHGck3ZLcpEwvAaYDUL202Rs110kPZAEbSIMFKuYykIEMwGcbuwgKGxCnmMotBkG1h9+ayxUpWtQfkqK+n0NAP0nke0/k+/H9ibjU7/cZAAAA+LkUdwYAAACSC2ESAABgKIRJAACAoS7HnQEAWAitVqvdbu/s7AiCIIpi3NkBGBV6kwAwc9Vq1bbtWq1mGIaqqnFnByCEJax0BYBZUxSl1WrRtmEYsizHmx+A0SFMAsDMmaapqqogCPl8XlGUuLMDEALCJABEh2YoG41G3BkBGBXmJgFg5qrVKm0oimLbdryZAQgFK10BYOYMw6BIadt2sViMOzsAIWDQFQCiYNu2aZpYvANzB2ESAABgKMxNAgAADIW5SQCIgt3rmUdH1slJfmNDWl+POzsAo0KYBIBJUQhkjNlnZ+3DQ0o0j4/ppefHx847Ly0t/dzvM8Yyv/jFu2+99UYmI62tMcbyGxvC8rKYzYrXrkWff4AAmJsEgCCjh8ABhc1N2qBA+N333z9+9uztX/3qaibz4rvvvv/xR8bYP16+/PdXrwY+eHV5WVpfp6jJGCveuMEYk9bXhZWVaRcO4GIIkwCLa1ohkJ0HM8aYfL4xQLp/n5+emvfuUX+Rn5y02u324aFxcPDXszPG2L9eu/bPgvBPV6783O/bZ2f89PTw9HRgJxvZrJjNIoJClBAmAdIpyhB4oaqmPfzqq7qiVO/c8b5qdDp6p2McHDi5upvP5zc25M1NMZulUuidDmOMn54Oi6Dvr60JKysURFdff50yP3aGARwIkwBzyeh0aEM/3zDPY8zXBwfDPkWxhE01BF6In5xIDx5I6+vGb38b/E671zMODihkUiDcyGblzc38+rpy8+ZAl5GfnFDUpJsAKr55dER9UzcqNRU5d+2amM0KKytYRgQjQpgESJxphUBaF8PiHpaUf/e7rw8O2vfuhYpM3lHZ99fW5M3N4o0bF0Z03wjqW3XUdUYEhQAIkwCRSlkIvFBzb6/85ZeV27cbOztj7yRgVDZsSDOPjuxej5+eWicnbHgEpWVE7LzCsRB3kSFMAkzNooXAC9m9nlipCCsr/OHDae1w9FHZsOjrM4+Puz/8QDOgvpO4WIi7aBAmAUaCEDiG0hdffL6/r//mN7OY+JxkVDYsdwQNWEaEhbiphDAJgBA4E0anU/z0093t7ebHH0dwrGmNyobiLCfGQtwUQ5iElEMIjIXd60kPHti9Hn/4MMrqmumobChYiJsaCJMwx2g5BjsfDWPnt/MMITBu9B8ltU8+UW7ejCsPUY7Khs0YFuLOEYRJSKiAEOh7S05ocogxRqNb7LxxoRQsU4yGeXSUf/CgsLl54X+UjExco7JhYSFuAiFMQgwQAtNt4HfpEiU5o7JhYSFuXBAmYcoQAhdc48mTWqs17HfpEiWxo7JhYSHuTCFMQggIgRCMfpdOzGbN+/fjzks48zIqGwoW4k4FwiT8L4RAmJzy2Wd/bLfD/i5doszvqGwoWIg7OoTJhYAQCBFoffvtzh/+MOHv0iVKakZlQ8FC3AEIk6nCT07UvT2EQIie87t05r17qelyufmOypZv3UpxvPQaYyFu+dateW9PECZTxe71Vn/9a+ccdebqEQIhAo0nT6S1tdSHDfeobOuTT9LXeRrPsIW43d//ft5vmxAmAQAAhroUdwYAAACS63LcGYCJmKZp27Ysy4wxwzAYY5cvX3716hW9KoqiKIpx5g8So9lsWpbVaDTG+CznvF6vl8tlSZKmckQ6V+n8pHNYEARBEDjn9AZBEOhYnHMnUZIkQRDGyP+EOOeUveC3UUFo21009+WZ+ksylS0SepNzr1gstlot2tY07c0336xWq/SnqqrOS7DgSqWSaZrjfVYURUEQnBgw+hGHvcQ51zTN+VNVVdpwTl3DMJxT10msVqtjF2ESqqqOflwnt2Tg8owlzEcsfS0SepPzTZKkQqGg67osy7Is67r+3nvvCYJAd3OUqChK3NmEpHAarJ2dHequGYah6zolOj2/ZrOp63qtVtM0zbZtJ73VammaVi6Xne6C0/TXajXacBJXV1eHZaNUKum6Th0L6mFQZtynrqIoiqJQhHYSq9VqqB5tWN7aMAyDekiUHtAdHygC87s8Ux8m09ki9WHOFQoFy7IqlUq/36d/6TTVdb1SqWiaFncGISmuXr3abrf7/X6327179+7Aq5qm6bru/FkoFOh06na7/X6fzqXd3V3LsugNzlnn3rYsa3d3lxJ1XS8UCsMyo6oqHa5erzv7pINWKhU6h51Ed67GLX047tqoVCrumgk2kEPv5Zl66WuR0JtMA7ord48L0Z2v02MAYIxJkuT0eGjmTJKkarXKORdFkXNeLpfd76eek9MBUlW1WCw6c0ucc9u2ne4pDclyznfOf1tAlmX3yOoARVHq9bosy91u1z1fRQflnJdKJadb5ph1byygNsbmvTxTL2UtEsJkSjQajVKp5DQi463UgHSjwOacJJIkmaaZy+XobLlw0qhWq3HOm80mTTrScOjAmSYIgqZpFN7cq2+8KBvNZjOfz3tfpWFYilhOommaM10AEqo2Qhm4PBdBmlqkf7g/b79QDG6GYTx+/DiTyUiSlMlkWq3W9evXHz9+/PLly8PDw3m8cYMZaTabT58+tW3bMAzDMHK5HJ0zjx49Mk3TMIxOp9PpdLa2tjKZTLVa/eabb16+fEnvtG272WxubW0pilIqlTjn+Xz++vXrh4eHqqo+ffrUMIznz59vbW1dv37dcGGMLS0t3RjygwOrq6v3799/9OgR/UknMx201WplMpmdnR13YqfTqdVqmUxmRlU0rDZyuZxTzIDLqtlsappG9ba0tCSKovfy/Oijj2aU+YRIZYuEnxcAWCyGYQz8zwpvyuhs2zZNc2B0lLqtc9om+taGbzFhQSBMAgAADIX/NwkAADAUlvCkgdHp2GdnT//yl/+0rP+27bcE4d9yua233xaWl1P/O9QAkFjNvb39Fy+233mndOtW3HkZHwZd5ww9yEY//zF+76OyLl+69Ornn90p9MwseiZc8caNVD4QDgASxeh0qq3W8+PjN1577W8//VTY3KzduTOnd+0Ik8lFD0elB9OYx8f89JSepe7493fe6f7ww391u9//+CNjbHd7e+fmTfnGDaPT+Y9nzx4/e8YYezOT+Zdf/nJ1ZWX/xQv3Z53Yufr669LaGh6wBQBTwU9OSl9++fXBwdXl5dL2du3DD+t/+lNzf/+vZ2d38/mGosxdU4MwmQh2r2ceHTkRkZ7T5n7D+2trwsqKtLZGT478208/ffXnP3++v88Y28hmy7dulba3Bx7qZvd6zf19dW+Pguvu9vbtd99947XXRj+KtL4+7w+KA4DI2L1etdWidqly+3btww+dBiTgpeRDmIwaRUR6/Dc/PbXPzgae/R3Qz/NGPuo+Bh/R6HS0b78dFlMv7LMWNjfp8c4UO+d02AQAZsfu9UbpMg50NBvnP9iUcAiTs0WLa9qHh/Q474GpxNFnDYND3ShChdjgGdCry8vS+rqYzQrLy/mNDSwUAlhkzb29+pMnh6enI05AOtOWG9ls7c6d5K/uQZicmlmElvG6j8HGjrjTCvkAkA6TBLywwTVGCJPjiGCgcvLuY7CpBOBJBpABYH5NZfh0xKHa2CFMXiDs4poJl73MovsYbOrxOOIaA4AoTX0xTvJX9yBM/p94+0az7j4Gm3V4xkIhgHk3085fklf3LG6YTMhMW/Tdx2BRRmssFAKYF9FMJSZzdc9ChMlkNsfxdh+DxRi8E3L7AgAk+tCVtNU9aQuTyR/cS1r3MVgSYjkWCgHEIsaB0ESt7pnjMDl3S0WSEHLGk8DQPnffPsAcSciymoRkYz7C5Fz3JxIYY8aW8Eif/LEEgIRLVDeOxL66J4lhMjWzUwkPKmObr8CfzJlpgARK2qSgW4yre2IOk6lswuYrikxifu8DUnMrBjAVyVxi6hVLII8uTC7CgNj8ho1JpOO2YK4H9gHGFvuQZljRDwvPJEwu2vKKdMSJyaXvLmHRzmRYKAlZIDOeKDM//TBpdDrFTz91/lyEe3Dls8/+2G6nIzBMzn3TsLu93fz447hzNGXB4yLte/cwQgtzofTFF5/v7ydknc54nK5w5fbt2fWDZ9KbrGraQs3o0AzrAnYfgxmdTirvinw5s+zJH7MCIHTDl4KGy+h0ZjqQk8SVrgAAAAlxOeA1zrkgCIIg0J+GYaiq2mq1xjsS55xzTtuSJAmCYNu2aZru98iyPN7OR2QYhq7rjLFGo+FOrNfrhmG4s+oueECi7w5H564TQRAkSRqWmBDDKnCSSvDlrYS4qqXZbFqWNWHRTNO0bZtObzrTer3eysqKO0UURVEUp5FlWCzepmmSttrbUDPGom+rg/m2xl7e604URdu2vYnBl96lgNdUVXVXjSzLtm2PXBAf1WrV2TDPOemapk2y81HIstxoNAa+b0mSBhrBgYIHJPrucHT0NVM9OF+5b2JC+JZ3wkrw5a2EuKqlVCpNZT/FYtFptjRNu3LlykBK0r5rmBfepmnCtnqgoWbn1x2LsK0O5tsa+/K9ysJeekN7k4ZhUCge6ChQNdm23Wg0aO+GYThHqtVqww4piqIgCBTDZVmuVqu1Wo3uVig9lltppyfk9E58Cz6sNnx3OEptEKdCGGNO8X0TZ4duPMvlsizLzWaz3W7XajVRFEMVZIBpms6FRJ9tNBqWZTl7do44bA/eSoi+Wqj4q6urlOItFGOs1Wq1223btovFoq7r5XLZt5srSVKhUNB1XZZlWZZ1Xf/ggw8GUhAmYQwBTdN4bbW3oZbORdBWc85VVXX+dJpfKh2ljN4ae687GogKfen1h6tUKrquu1MKhUK73e73++12u16v9/t9y7IqlQq96t72VSgULtyOgPdwAynegg9LHPh4qNpw6Lru3bNv4oxUKpVut9t35Tm4IL7fl5PY7Xbv3r3rbO/u7tIG7URVVTriKBnzVkI01WJZFmWbjlgoFHwL5X5bvV6/8OR3apL+9aYAjMG3aZqkrQ5onGfdVu/u7lJb1Pe7KDRNo5IGtMYDfK+ysJde0NzksPhM/9KdNefctm2nkz56Tz+V986haqPRaDjvDE6ctXK5XK/XG42Gqqq1Wo1N8LUyxkzT3Dlf7enMHzh9L1VVFUUJ3oO3EiKuFs65UwRZljVN8y2U+22Korjvgn3RPbh7sMibAjAtU2mrI26o3TOO7iFMzrkoipzzcrkcdp++V1moSy90mPTmgIbUQn3KNM1UrlYIVRvdbpc2OOfOSJ1v4qw5Z0wul6NzdLyv1dmbOxY6ywEYY+12u9VqlUqlYrEYsAdvJURcLYIgaJpG4060osG3UKIo1ut1etuIyyUajUapVHI3Pd4UgFkY46KOvqF2B2+67qhdomyPvYDU9yob/dILCpPUyaDx31wuR8G82WyWSiWa2qVi5HI552Crq6vD7voNw+CcO69Sr4Xmeyg97ATYGOjodDjKtjfFW/CARO/HR6wNd37Y/18n4ps4a+VyWVEU595q2NfqW10BlcA5py+aMZbL5drtNp1FFy6T81ZClNVCN+DO1I4gCHStDhRKFMVisUiJ+XzeieVedPLTtbOzs0MrqwdSZl0oSCtv0+Q+u8K21b4NNYuqrc7n8+5cNRoNuhm1LIudB1FJknxb42FlGbjKxrn0gsdku93uKEPAI75tjviWaOq1YVmWZVmjJM6aZVk0a+g2ydc64SnhrYToq8WyLJrdcQQUyrIsmgECiN7ol9tctNW6rjszlMNSoiwIfl5g0dEqMtM0vf8xBkbhdKabzSbGTgHSB2ESAABgqKCfFwAAAFhwCJMAAABDIUwCAAAMhTAJAAAw1P8AXgM98hzgxqcAAAAASUVORK5CYII=",
      "text/plain": [
       "Tree('S', [Tree('NP', [('the', 'DT'), ('little', 'JJ'), ('yellow', 'JJ'), ('dog', 'NN')]), ('barked', 'VBD'), ('at', 'IN'), Tree('NP', [('the', 'DT'), ('cat', 'NN')])])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"POS_tagging.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dependency Parsing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python -n spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                jumps                  \n",
      "  ________________|____________         \n",
      " |    |     |     |    |      over     \n",
      " |    |     |     |    |       |        \n",
      " |    |     |     |    |      dog      \n",
      " |    |     |     |    |    ___|____    \n",
      "The quick brown  fox   .  the      lazy\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "from nltk import Tree\n",
    "\n",
    "\n",
    "en_nlp = spacy.load('en')\n",
    "\n",
    "doc = en_nlp(\"The quick brown fox jumps over the lazy dog.\")\n",
    "\n",
    "def to_nltk_tree(node):\n",
    "    if node.n_lefts + node.n_rights > 0:\n",
    "        return Tree(node.orth_, [to_nltk_tree(child) for child in node.children])\n",
    "    else:\n",
    "        return node.orth_\n",
    "\n",
    "\n",
    "[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tok_format(tok):\n",
    "    return \"_\".join([tok.orth_, tok.tag_])\n",
    "\n",
    "\n",
    "def to_nltk_tree(node):\n",
    "    if node.n_lefts + node.n_rights > 0:\n",
    "        return Tree(tok_format(node), [to_nltk_tree(child) for child in node.children])\n",
    "    else:\n",
    "        return tok_format(node)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         jumps_VBZ                           \n",
      "   __________________________|___________________             \n",
      "  |       |        |         |      |         over_IN        \n",
      "  |       |        |         |      |            |            \n",
      "  |       |        |         |      |          dog_NN        \n",
      "  |       |        |         |      |     _______|_______     \n",
      "The_DT quick_JJ brown_JJ   fox_NN  ._. the_DT         lazy_JJ\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/36610179/how-to-get-the-dependency-tree-with-spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Readabilty features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: textstat in /home/shivangi/anaconda3/envs/WWC/lib/python3.7/site-packages (0.5.6)\r\n",
      "Requirement already satisfied: repoze.lru in /home/shivangi/anaconda3/envs/WWC/lib/python3.7/site-packages (from textstat) (0.7)\r\n",
      "Requirement already satisfied: pyphen in /home/shivangi/anaconda3/envs/WWC/lib/python3.7/site-packages (from textstat) (0.9.5)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install textstat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52.23"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import textstat\n",
    "\n",
    "test_data = (\n",
    "    \"Playing games has always been thought to be important to \"\n",
    "    \"the development of well-balanced and creative children; \"\n",
    "    \"however, what part, if any, they should play in the lives \"\n",
    "    \"of adults has never been researched that deeply. I believe \"\n",
    "    \"that playing games is every bit as important for adults \"\n",
    "    \"as for children. Not only is taking time out to play games \"\n",
    "    \"with our children and other adults valuable to building \"\n",
    "    \"interpersonal relationships but is also a wonderful way \"\n",
    "    \"to release built up tension.\"\n",
    ")\n",
    "\n",
    "textstat.flesch_reading_ease(test_data)\n",
    "# textstat.smog_index(test_data)\n",
    "# textstat.flesch_kincaid_grade(test_data)\n",
    "# textstat.coleman_liau_index(test_data)\n",
    "# textstat.automated_readability_index(test_data)\n",
    "# textstat.dale_chall_readability_score(test_data)\n",
    "# textstat.difficult_words(test_data)\n",
    "# textstat.linsear_write_formula(test_data)\n",
    "# textstat.gunning_fog(test_data)\n",
    "# textstat.text_standard(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://honingds.com/blog/natural-language-processing-with-python/\n",
    "# https://rstudio-pubs-static.s3.amazonaws.com/79360_850b2a69980c4488b1db95987a24867a.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc1 = \"Brocolli is good to eat. My brother likes to eat good brocolli, but not my mother.\"\n",
    "doc2 = \"My mother spends a lot of time driving my brother around to baseball practice.\"\n",
    "doc3 = \"Some health experts suggest that driving may cause increased tension and blood pressure.\"\n",
    "doc4 = \"I often feel pressure to perform well at school, but my mother never seems to drive my brother to do better.\"\n",
    "doc5 = \"Health professionals say that brocolli is good for your health.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_complete= [doc1, doc2, doc3, doc4, doc5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords \n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "import string\n",
    "stop = set(stopwords.words('english'))\n",
    "exclude = set(string.punctuation) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemma = WordNetLemmatizer()\n",
    "def clean(doc):\n",
    "    stop_free = \" \".join([i for i in doc.lower().split() if i not in stop])\n",
    "    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)\n",
    "    normalized = \" \".join(lemma.lemmatize(word) for word in punc_free.split())\n",
    "    return normalized\n",
    "\n",
    "doc_clean = [clean(doc).split() for doc in doc_complete] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim import corpora, models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# turn our tokenized documents into a id <-> term dictionary\n",
    "dictionary = corpora.Dictionary(doc_clean)\n",
    "    \n",
    "# convert tokenized documents into a document-term matrix\n",
    "corpus = [dictionary.doc2bow(doc) for doc in doc_clean]\n",
    "\n",
    "# generate LDA model\n",
    "ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word = dictionary, passes=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.058*\"health\" + 0.058*\"driving\" + 0.057*\"pressure\"'), (1, '0.067*\"mother\" + 0.067*\"brother\" + 0.067*\"brocolli\"')]\n"
     ]
    }
   ],
   "source": [
    "print(ldamodel.print_topics(num_topics=3, num_words=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# N-grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['this', 'is'], ['is', 'a'], ['a', 'sample'], ['sample', 'text']]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def generate_ngrams(text, n):\n",
    "    words = text.split()\n",
    "    output = []  \n",
    "    for i in range(len(words)-n+1):\n",
    "        output.append(words[i:i+n])\n",
    "    return output\n",
    "\n",
    "generate_ngrams('this is a sample text', 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given two strings, the task is to find out how similar they are. There are generally two methods to achieve this task:\n",
    "* Edit Distance\n",
    "* Cosine Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Also called Levenstein Method\n",
    "* compute edit distance between two words/strings\n",
    "* based on dynamic programming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"edit1.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"edit2.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"edit3.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"edit4.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "string1=\"strength\"\n",
    "string2=\"trend\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LD(s,t):\n",
    "    s = ' ' + s      #-------------------------------------STEP:1\n",
    "    t = ' ' + t      #--------------------------------------STEP:2\n",
    "    d = {}\n",
    "    S = len(s)\n",
    "    T = len(t)\n",
    "    for i in range(S):\n",
    "        d[i, 0] = i #---------------------------------------STEP:3\n",
    "    for j in range (T):\n",
    "        d[0, j] = j #---------------------------------------STEP:4\n",
    "    for j in range(1,T):\n",
    "        for i in range(1,S):\n",
    "            if s[i] == t[j]:\n",
    "                d[i, j] = d[i-1, j-1]\n",
    "            else:\n",
    "                d[i, j] = min(d[i-1, j], d[i, j-1], d[i-1, j-1]) + 1\n",
    "    return d[S-1, T-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "edit distance between two strings is: 4\n"
     ]
    }
   ],
   "source": [
    "print(\"edit distance between two strings is:\",LD(string1,string2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cosine Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Cosine similarity calculates similarity by measuring the cosine of angle between two vectors.\n",
    "\n",
    "* The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance (due to the size of the document), chances are they may still be oriented closer together. \n",
    "* The smaller the angle, higher the cosine similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"cosine1.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "from collections import Counter #A counter is a container that stores elements as dictionary keys, \n",
    "                                 # and their counts are stored as dictionary values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = 'This is an article on social media' \n",
    "text2 = 'article on social media is about natural language processing'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"cosine2.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_to_vector(text): \n",
    "    words = text.split() \n",
    "    return Counter(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector1 = text_to_vector(text1) \n",
    "vector2 = text_to_vector(text2) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'This': 1,\n",
       "         'is': 1,\n",
       "         'an': 1,\n",
       "         'article': 1,\n",
       "         'on': 1,\n",
       "         'social': 1,\n",
       "         'media': 1})"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'article': 1,\n",
       "         'on': 1,\n",
       "         'social': 1,\n",
       "         'media': 1,\n",
       "         'is': 1,\n",
       "         'about': 1,\n",
       "         'natural': 1,\n",
       "         'language': 1,\n",
       "         'processing': 1})"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"cosine3.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cosine(vec1, vec2):\n",
    "    common = set(vec1.keys()) & set(vec2.keys())\n",
    "    numerator = sum([vec1[x] * vec2[x] for x in common])\n",
    "\n",
    "    sum1 = sum([vec1[x]**2 for x in vec1.keys()]) \n",
    "    sum2 = sum([vec2[x]**2 for x in vec2.keys()]) \n",
    "    denominator = math.sqrt(sum1) * math.sqrt(sum2)\n",
    "   \n",
    "    if denominator:\n",
    "        return float(numerator) / denominator\n",
    "    else:\n",
    "        return 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.629940788348712"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_cosine(vector1,vector2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.kdnuggets.com/2018/08/practitioners-guide-processing-understanding-text-2.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assessment Time !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Write a function that performs following steps: lowecasing, punctuation and lemmatization\n",
    "* find edit distance between : execution, inspection"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (WWC_part1)",
   "language": "python",
   "name": "wwc_part1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}